flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: HDFS hdfs.path custom config ? possible ?
Date Tue, 08 Sep 2015 16:51:56 GMT
You can use roundValue as 15 and roundUnit as Minute - so a new dir would
get created every 15 mins.


Thanks,
Hari

On Tue, Sep 8, 2015 at 9:42 AM, Sutanu Das <sd2302@att.com> wrote:

> Hi Community,
>
>
>
> Is it possible to create /write HDFS dirs./folders every 15 mins?
>
>
>
> Right now, we use /hdfs_sink_path/%Y/%m/%d/%H --- ie using every Hour,
> But, Is it possible to write every 15 mins? – not every minute…
>
>
>
> If this is not possible out of the box config, can DEV team please advise
> which code/java-file do we need to customize so Flume creates dirs. Every
> 15 mins? Like /%Y/%m/%d/%H*/%15M ?*
>
>
>
> Please help us, thanks.
>
>
>
> Here is our present 1 hour HDFS dir/folder creation config:
>
>
>
>
>
> airwave.sinks.sink1.type = hdfs
>
> airwave.sinks.sink1.channel = channel1
>
> airwave.sinks.sink1.hdfs.path =
> /prod/hadoop/smallsite/airwave_flume_ingest_hak_3_amp_1*/%Y/%m/%d/%H*
>
> airwave.sinks.sink1.hdfs.fileType = DataStream
>
> airwave.sinks.sink1.hdfs.writeFormat = Text
>
> airwave.sinks.sink1.hdfs.filePrefix = Airwave_amp_1
>
> airwave.sinks.sink1.hdfs.useLocalTimeStamp = true
>
> airwave.sinks.sink1.hdfs.round = true
>
> airwave.sinks.sink1.hdfs.roundValue = 1
>
> airwave.sinks.sink1.hdfs.roundUnit = hour
>
> airwave.sinks.sink1.hdfs.callTimeout = 10000000000
>
>
>

Mime
View raw message