flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Java API for Flume Agent
Date Fri, 23 Jan 2015 07:47:07 GMT
Can you enable debug logging for Flume? I want to see what the
configuration looks like.

On Thu, Jan 22, 2015 at 8:05 PM, Carlotta Hicks <Carlotta.Hicks@sas.com>
wrote:

>  Here is the code change in EmbeddedAgentConfiguration:
>
>
>
>   /**
>
>    * Avro sink which can send events to a downstream avro source. This is
> the
>
>    * only supported sink for Embedded Agents.
>
>    */
>
>   public static final String SINK_TYPE_AVRO = SinkType.AVRO.name();
>
>   public static final String SINK_TYPE_HBASE = SinkType.HBASE.name();
>
>   public static final String SINK_TYPE_HDFS = SinkType.HDFS.name();
>
>
>
>   private static final String[] ALLOWED_SINKS = {
>
>    SINK_TYPE_AVRO,
>
>     SINK_TYPE_HBASE,
>
>     SINK_TYPE_HDFS
>
>   };
>
>
>
> This is my code with configurations:
>
>
>
>         Map<String, String> properties = new HashMap<String, String>();
>
>
>
>         properties.put("channel.type", "file");
>
>         properties.put("sinks", "hdfs1");
>
>         properties.put("hdfs1.type", "hdfs");
>
>         properties.put("hdfs1.path", "hdfs://lax07.unx.sas.com:8020/tmp");
>
>         properties.put("processor.type", "default");
>
>
>
>          agent = new EmbeddedAgent(“myAgent”);
>
>
>
>          agent.configure(properties);             ß This statement causes
> the exception below
>
>
>
> org.apache.flume.FlumeException: Expected one source and got 0
>
>                 at
> org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:160)
>
>                 at
> org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:95)
>
>
>
>
>
>
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> *Sent:* Tuesday, January 20, 2015 3:23 PM
> *To:* user@flume.apache.org
> *Cc:* user@flume.apache.org
> *Subject:* RE: Java API for Flume Agent
>
>
>
> Can you send the code you used?
>
>
> Thanks,
>
> Hari
>
>
>
> On Tue, Jan 20, 2015 at 8:48 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
>
> I added in an HDFS sink. But, I am getting this FlumeException: Expected
> one source and got 0 .
>
> The only source for the embedded agent is 'embedded', correct? Not sure
> how to proceed from here.
>
> Need help.
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com <paliwalashish@gmail.com>]
> Sent: Saturday, January 17, 2015 7:58 AM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> Try a bit of hacking in
> org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add
> HBase Sink to allowed sinks in the section
>
> private static final String[] ALLOWED_SINKS = {
> SINK_TYPE_AVRO,
> // Add Hbase sink here <<<
>
> };
>
> See if it works and let us know.
>
> HTH!
>
> On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
> > Thanks Joey! I would love to use the embedded agent. It is exactly what
> I am looking for. However, I can't use the embedded agent because with the
> embedded agent, the sink must be avro. I need the sink to be hbase.
> > I am assuming that requirements has not changed. Please correct me if I
> am wrong.
> > ________________________________________
> > From: Joey Echeverria <joey@cloudera.com>
> > Sent: Friday, January 16, 2015 3:47 PM
> > To: user@flume.apache.org
> > Subject: Re: Java API for Flume Agent
> >
> > You could use an embedded flume agent[1]. This will do the same thing
> > that the Flume flow does but run flume inside of another Java
> > application.
> >
> > Let me know if that meets your needs.
> >
> > -Joey
> >
> > [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
> >
> >
> > On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
> >> I need to create an agent using java API to load an HBase table using
> >> a CSV file. I am able to accomplish this using flume-ng and an agent
> >> with a spooldir directory source, file channel and HBase sink.
> >>
> >>
> >>
> >> I need information on how to accomplish this via a java API.
> >>
> >>
> >>
> >> Any help would be appreciated.
> >>
> >>
> >>
> >> -CM
> >>
> >>
> >
> >
> >
> > --
> > Joey Echeverria
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>

Mime
View raw message