flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Flume usage with Kafka channel & HDFS sink
Date Sat, 19 Sep 2015 20:22:24 GMT
If the parameter is chosen from a fixed list, it's cumbersome but can be
done.
However if you want it to be arbitrary and create topics on demand then you
need to write your own code custom sinks and it's not trivial to manage it
efficiently

Regards,
Gonzalo
On Sep 19, 2015 7:42 PM, "Hemanth Abbina" <HemanthA@eiqnetworks.com> wrote:

> Hi,
>
>
>
> I'm new to Flume and thinking to use Flume in the below scenario.
>
>
>
> Our system receives events as HTTP POST, and we need to store them in
> Kafka(for processing) as well as HDFS(as permanent store).
>
>
>
> Can we configure Flume as below ?
>
> ·         Source:  HTTP (expecting JSON event as HTTP body, with a
> dynamic topic name in the URI)
>
> ·         Channel: KAFKA (should store the received JSON body, to a topic
> mentioned in the URI)
>
> ·         Sink:  HDFS (should store the data in a folder mentioned in the
> URI.
>
>
>
> For example, If I receive a JSON event from a HTTP source with the below
> attributes,
>
> ·         URL: https://xx.xx.xx.xx/event/*abc*
> <https://xx.xx.xx.xx/event/abc>
>
> ·         Body of POST:  { name: xyz, value=123}
>
>
>
> The event should be saved to Kafka channel – with topic ‘abc’ and written
> to HDFS to a folder as ‘abc’.
>
> This ‘abc’ will be dynamic and change from event to event.
>
>
>
> Is this possible with Flume ?
>
>
>
> Thanks in advance
>
> Hemanth
>

Mime
View raw message