flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Fri, 08 Aug 2014 20:51:03 GMT
Do you see anything in the logs? If you are configuring your serializer 
correctly it should work fine. Are you sure your events come in with the 
header key DataSrc and values either ABC or BCD?

terrey shih wrote:
>
> Hi,
>
> One more thing. I would like to know if one serializer can be used
> for two table.
>
> thanks,
> -Gary
>
> server-agent.sources = mySrc
> server-agent.sinks = hbase-sink1 hbase-sink2
> server-agent.channels = C1 C2
>
> # Describe/configure the source
> server-agent.sources.mySrc.type = avro
> server-agent.sources.mySrc.bind = 127.0.0.1
> server-agent.sources.mySrc.port = 5000
>
> # Use a channel which buffers events in memory
> server-agent.channels.C1.type = memory
> server-agent.channels.C1.capacity = 1000
> server-agent.channels.C1.transactionCapacity = 100
> server-agent.sinks.hbase-sink1.type = asynchbase
> server-agent.sinks.hbase-sink1.table = table1
> server-agent.sinks.hbase-sink1.columnFamily = fam1
> server-agent.sinks.hbase-sink1.batchSize = 1000
> server-agent.sinks.hbase-sink1.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink1.channel = C1
> server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3
>
> server-agent.channels.C2.type = memory
> server-agent.channels.C2.capacity = 1000
> server-agent.channels.C2.transactionCapacity = 100
> server-agent.sinks.hbase-sink2.type = asynchbase
> server-agent.sinks.hbase-sink2.table = table2
> server-agent.sinks.hbase-sink2.columnFamily = fam2
> server-agent.sinks.hbase-sink2.batchSize = 1000
> server-agent.sinks.hbase-sink2.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink2.channel = C2
> server-agent.sinks.hbase-sink2.serializer.columns = table2Col
>
> # Bind the source and sink to the channel
> server-agent.sources.mySrc.channels = C1 C2
> server-agent.sources.avor-Src.selector.type = multiplexing
> server-agent.sources.avor-Src.selector.header = DataSrc
> server-agent.sources.avor-Src.selector.mapping.ABC = C1
> server-agent.sources.avor-Src.selector.mapping.BCD = C2
>
>
>
> On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <terreyshih@gmail.com
> <mailto:terreyshih@gmail.com>> wrote:
>
> HI,
>
> Here is my sample config.
>
>
>
>
> On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan
> <hshreedharan@cloudera.com <mailto:hshreedharan@cloudera.com>> wrote:
>
> Can you please send your config? That would make it easier to
> understand.
>
>
> terrey shih wrote:
>>
>>
>> I have a fanning out operation where I have one source and
>> and based
>> on the event headers (headers are added from the source
>> input), I
>> would like to channel the event to different Hbase tables. I
>> am using
>> the same serializer for the hbase tables.
>
>
>

Mime
View raw message