flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Abbina <Heman...@eiqnetworks.com>
Subject Flume usage with Kafka channel & HDFS sink
Date Sat, 19 Sep 2015 18:42:26 GMT

I'm new to Flume and thinking to use Flume in the below scenario.

Our system receives events as HTTP POST, and we need to store them in Kafka(for processing)
as well as HDFS(as permanent store).

Can we configure Flume as below ?

*         Source:  HTTP (expecting JSON event as HTTP body, with a dynamic topic name in the

*         Channel: KAFKA (should store the received JSON body, to a topic mentioned in the

*         Sink:  HDFS (should store the data in a folder mentioned in the URI.

For example, If I receive a JSON event from a HTTP source with the below attributes,

*         URL: https://xx.xx.xx.xx/event/abc

*         Body of POST:  { name: xyz, value=123}

The event should be saved to Kafka channel - with topic 'abc' and written to HDFS to a folder
as 'abc'.
This 'abc' will be dynamic and change from event to event.

Is this possible with Flume ?

Thanks in advance

View raw message