flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutanu Das <sd2...@att.com>
Subject Creating HDFS sink directories based on LogFile Pattern - POSSIBLE with Flume?
Date Fri, 21 Aug 2015 20:44:34 GMT
Hi Team,

We are asked to create HDFS directory in HDFS Sink based on Logfile Pattern/Topic. Is it possible
with Flume Interceptors / Extractors / Serializes out-of-box ?

Example: Single Logfile has following lines:

t=1440187845 ArubaPresence op="add" sta_mac="" associated="False" ap_name="a036000000kqVoW-02i6000000T5jrU"
t=1440187845 ArubaPresence op="add" sta_mac="" associated="False" ap_name="a036000000kqVoW-02i6000000T5jrU"
t=1440189388 ArubaRadio op="update" mac="04:bd:88:80:38:d0" ap_mac="04:bd:88:c0:03:8c" type="RADIO_PHY_TYPE_A_HT"
mode="RADIO_MODE_AP"
t=1440189388 ArubaRadio op="update" mac="04:bd:88:80:38:c0" ap_mac="04:bd:88:c0:03:8c" type="RADIO_PHY_TYPE_A_HT_40"
mode="RADIO_MODE_AP"


So Is it possible to write each lines from the single sample Log above to separate HDFS Sink
Directory based on the Keywork/patter-topic ( eg Aruba Presence and ArubaRadio) ?  so it will
looks like this during Flume HDFS sink write:


Creating /prod/hadoop/Aruba Presence/2015/08/21/20/Airwave_amp_2.1440189722272.tmp

Creating /prod/hadoop/ArubaRadio/2015/08/21/20/Airwave_amp_2.1440189722272.tmp


Mime
View raw message