flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Ortiz <konstt2...@gmail.com>
Subject Flow in Flume, could it make better?
Date Mon, 18 Aug 2014 16:53:20 GMT
Hi,

I have build a flow with Flume and I don't know if it's the way to do it,
or there is something better. I am spooling a directory and need those data
in three different paths in HDFS with different formats, so I have created
two interceptors.

Source(Spooling) + Replication + Interceptor1 --> to C1 and C2
C1 -> Sink1 to HDFS Path1 (It's like a historic)
C2 --> Sink2 to Avro --> Source Avro + Multiplexing + Interceptor2 --> C3
and C4
C3 --> Sink3 to HDFS Path2
C4 --> Sink4 to HDFS Path3

Interceptor1 doesn't make too much with the data, it's just to save as they
are, it's like to store an history of the original data.

Interceptor2 configure an selector and a header. It processes the data and
configure the selector to redirect to Sink3 or Sink4. But this interceptor
change the original data.

I tried to do all the process without replicating data, but I could not.
Now, it seems like too many steps just because I want to store the original
data in HDFS like a historic.

Mime
View raw message