flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Flume kafka channel
Date Fri, 08 Apr 2016 07:56:05 GMT
You cannot set a serializer in the channel, whatever you put in the topic
events will be stored in hdfs so you shouldn't need it
If you want to do some parsing then you can implement a sink serializer.

Gonzalo

On 8 April 2016 at 08:44, Baris Akgun (Garanti Teknoloji) <
BarisAkgu@garanti.com.tr> wrote:

> Hi
>
>
>
> I am using kafka as a channel in my flume conf file.
>
>
>
> How can I set  serializer.class" with "kafka.serializer.*StringEncoder*"
> value in flume confugration file? Or How can I set flume’s kafka consumer
> encoder?
>
>
>
> Thanks.
>
>
>
> My Conf File
>
>
>
> tier1.sinks = sink1
>
> tier1.channels = channel1
>
> #
>
> tier1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannel
>
> tier1.channels.channel1.capacity=1000000
>
> tier1.channels.channel1.transactionCapacity=100000
>
> tier1.channels.channel1.brokerList=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> tier1.channels.channel1.topic=topic_baris2
>
> tier1.channels.channel1.zookeeperConnect=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>
> tier1.channels.channel1.parseAsFlumeEvent=false
>
> #
>
> tier1.sinks.sink1.type = hdfs
>
> tier1.sinks.sink1.hdfs.useLocalTimeStamp=true
>
> tier1.sinks.sink1.hdfs.path =
> /data/stage/flume/social_media/tweet_kafka/%Y%m%d
>
> tier1.sinks.sink1.hdfs.filePrefix = events_%Y%m%d
>
> tier1.sinks.sink1.hdfs.fileSuffix = .log
>
> tier1.sinks.sink1.hdfs.inUsePrefix = .
>
> tier1.sinks.sink1.hdfs.inUseSuffix = .incomplete
>
> tier1.sinks.sink1.hdfs.round = true
>
> tier1.sinks.sink1.hdfs.roundValue = 1
>
> tier1.sinks.sink1.hdfs.roundUnit = minute
>
> tier1.sinks.sink1.hdfs.rollCount = 0
>
> tier1.sinks.sink1.hdfs.rollSize = 268435456
>
> tier1.sinks.sink1.hdfs.batchSize = 100000
>
> tier1.sinks.sink1.hdfs.fileType=DataStream
>
> tier1.sinks.sink1.hdfs.writeFormat=Text
>
> #
>
> tier1.sinks.sink1.channel = channel1
>
>
>
>
>
> Barış
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>

Mime
View raw message