flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Shaposhnik <...@apache.org>
Subject Re: Flume and HDFS integration
Date Sat, 01 Dec 2012 00:40:21 GMT
On Fri, Nov 30, 2012 at 12:51 AM, Emile Kao <emilekao@gmx.net> wrote:
> Hello Brock,
> first of all thank you for answering my questions. I appreciate it since I am a real
newbie in Flume / Hadoop , etc...
>
> But now I am confused. According to you statement, the filetype is the key here. Now
just take a look on my flume.conf below:
> The filetype was from set to "DataStream".
> Now which is the right one now: SequenceFile, DataStream or CompressedStream?

Here's what works for me in the situation very similar to yours:

# Sink configuration
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /flume/cluster-logs
agent.sinks.sink1.hdfs.writeFormat = Text
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.filePrefix = events-
agent.sinks.sink1.hdfs.round = true
agent.sinks.sink1.hdfs.roundValue = 10
agent.sinks.sink1.hdfs.roundUnit = minute
# agent.sinks.sink1.hdfs.serializer =
org.apache.flume.serialization.BodyTextEventSerializer

Thanks,
Roman.

Mime
View raw message