flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: Java API for Flume Agent
Date Tue, 20 Jan 2015 21:54:16 GMT
Also need to see the config if possible.

RE: Embedded agent not supporting hbase sink, you could avro sink from
embeedded flume to an avro source on a sperate flume "tier", which
aggregates events from your servers and then writes to hbase using hbase
sink

-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Tue, Jan 20, 2015 at 8:47 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
wrote:

> I added in an HDFS sink.  But, I am getting this FlumeException:  Expected
> one source and got 0 .
>
> The only source for the embedded agent is 'embedded', correct?  Not sure
> how to proceed from here.
>
> Need help.
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com]
> Sent: Saturday, January 17, 2015 7:58 AM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> Try a bit of hacking in
> org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add
> HBase Sink to allowed sinks in the section
>
> private static final String[] ALLOWED_SINKS = {
>   SINK_TYPE_AVRO,
>   // Add Hbase sink here <<<
>
> };
>
> See if it works and let us know.
>
> HTH!
>
> On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
> > Thanks Joey!  I would love to use the embedded agent.  It is exactly
> what I am looking for.  However, I can't use the embedded agent because
> with the embedded agent, the sink must be avro.  I need the sink to be
> hbase.
> > I am assuming that requirements has not changed.  Please correct me if I
> am wrong.
> > ________________________________________
> > From: Joey Echeverria <joey@cloudera.com>
> > Sent: Friday, January 16, 2015 3:47 PM
> > To: user@flume.apache.org
> > Subject: Re: Java API for Flume Agent
> >
> > You could use an embedded flume agent[1]. This will do the same thing
> > that the Flume flow does but run flume inside of another Java
> > application.
> >
> > Let me know if that meets your needs.
> >
> > -Joey
> >
> > [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
> >
> >
> > On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
> >> I need to create an agent using java API to load an HBase table using
> >> a CSV file.  I am able to accomplish this using flume-ng and an agent
> >> with a spooldir directory source, file channel and HBase sink.
> >>
> >>
> >>
> >> I need information on how to accomplish this via a java API.
> >>
> >>
> >>
> >> Any help would be appreciated.
> >>
> >>
> >>
> >> -CM
> >>
> >>
> >
> >
> >
> > --
> > Joey Echeverria
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Mime
View raw message