flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutanu Das <sd2...@att.com>
Subject HDFS hdfs.path custom config ? possible ?
Date Tue, 08 Sep 2015 16:42:15 GMT
Hi Community,

Is it possible to create /write HDFS dirs./folders every 15 mins?

Right now, we use /hdfs_sink_path/%Y/%m/%d/%H --- ie using every Hour, But, Is it possible
to write every 15 mins? - not every minute...

If this is not possible out of the box config, can DEV team please advise which code/java-file
do we need to customize so Flume creates dirs. Every 15 mins? Like /%Y/%m/%d/%H/%15M ?

Please help us, thanks.

Here is our present 1 hour HDFS dir/folder creation config:


airwave.sinks.sink1.type = hdfs
airwave.sinks.sink1.channel = channel1
airwave.sinks.sink1.hdfs.path = /prod/hadoop/smallsite/airwave_flume_ingest_hak_3_amp_1/%Y/%m/%d/%H
airwave.sinks.sink1.hdfs.fileType = DataStream
airwave.sinks.sink1.hdfs.writeFormat = Text
airwave.sinks.sink1.hdfs.filePrefix = Airwave_amp_1
airwave.sinks.sink1.hdfs.useLocalTimeStamp = true
airwave.sinks.sink1.hdfs.round = true
airwave.sinks.sink1.hdfs.roundValue = 1
airwave.sinks.sink1.hdfs.roundUnit = hour
airwave.sinks.sink1.hdfs.callTimeout = 10000000000


Mime
View raw message