flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Philippe Caruana ...@target2sell.com>
Subject HDFS sink: "clever" routing
Date Wed, 15 Oct 2014 14:02:55 GMT

I am new to Flume (and to HDFS), so I hope my question is not stupid.

I have a multi-tenant application (about 100 different customers as for
I have 16 different data types.

(In production, we have approx. 15 million messages/day through our

I want to write to HDFS all my events, separated by tenant, data type,
and date, like this :

Is it possible with one sink definition ? I don't want to duplicate
configuration, and new client arrive every week or so

In documentation, I see
agent1.sinks.hdfs-sink1.hdfs.path = hdfs://server/events/%Y/%m/%d/%H/

Is this possible ?
agent1.sinks.hdfs-sink1.hdfs.path =

I want to write to different folder according to my incoming data.


Jean-Philippe Caruana 

View raw message