flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: HDFS hdfs.path custom config ? possible ?
Date Tue, 08 Sep 2015 17:22:31 GMT
No, you need the %M - else you'd still round down to the hour (since the
timestamp is rounded down, but they are still within the same hour)


Thanks,
Hari

On Tue, Sep 8, 2015 at 10:19 AM, IT CTO <goi.cto@gmail.com> wrote:

> If the path stays with %h what would be the folder name with round 15 unit
> minute?
>
> בתאריך יום ג׳, 8 בספט׳ 2015, 19:52 מאת Hari Shreedharan <
> hshreedharan@cloudera.com>:
>
>> You can use roundValue as 15 and roundUnit as Minute - so a new dir would
>> get created every 15 mins.
>>
>>
>> Thanks,
>> Hari
>>
>> On Tue, Sep 8, 2015 at 9:42 AM, Sutanu Das <sd2302@att.com> wrote:
>>
>>> Hi Community,
>>>
>>>
>>>
>>> Is it possible to create /write HDFS dirs./folders every 15 mins?
>>>
>>>
>>>
>>> Right now, we use /hdfs_sink_path/%Y/%m/%d/%H --- ie using every Hour,
>>> But, Is it possible to write every 15 mins? – not every minute…
>>>
>>>
>>>
>>> If this is not possible out of the box config, can DEV team please
>>> advise which code/java-file do we need to customize so Flume creates dirs.
>>> Every 15 mins? Like /%Y/%m/%d/%H*/%15M ?*
>>>
>>>
>>>
>>> Please help us, thanks.
>>>
>>>
>>>
>>> Here is our present 1 hour HDFS dir/folder creation config:
>>>
>>>
>>>
>>>
>>>
>>> airwave.sinks.sink1.type = hdfs
>>>
>>> airwave.sinks.sink1.channel = channel1
>>>
>>> airwave.sinks.sink1.hdfs.path =
>>> /prod/hadoop/smallsite/airwave_flume_ingest_hak_3_amp_1*/%Y/%m/%d/%H*
>>>
>>> airwave.sinks.sink1.hdfs.fileType = DataStream
>>>
>>> airwave.sinks.sink1.hdfs.writeFormat = Text
>>>
>>> airwave.sinks.sink1.hdfs.filePrefix = Airwave_amp_1
>>>
>>> airwave.sinks.sink1.hdfs.useLocalTimeStamp = true
>>>
>>> airwave.sinks.sink1.hdfs.round = true
>>>
>>> airwave.sinks.sink1.hdfs.roundValue = 1
>>>
>>> airwave.sinks.sink1.hdfs.roundUnit = hour
>>>
>>> airwave.sinks.sink1.hdfs.callTimeout = 10000000000
>>>
>>>
>>>
>>
>> --
> Eran | "You don't need eyes to see, you need vision" (Faithless)
>

Mime
View raw message