flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Fri, 08 Aug 2014 22:14:12 GMT
Yes, it just creates new instances of the Serializer class.

terrey shih wrote:
>
> Hi,
>
> Yes. The key and values are correct because I printed them out in my
> serializer class.. Can I use the same serializer for for two
> different channels for two hbase tables ? I should be able to right ?
>
> thanks,
> -Gary
>
>
> On Fri, Aug 8, 2014 at 1:51 PM, Hari Shreedharan
> <hshreedharan@cloudera.com <mailto:hshreedharan@cloudera.com>> wrote:
>
> Do you see anything in the logs? If you are configuring your
> serializer correctly it should work fine. Are you sure your events
> come in with the header key DataSrc and values either ABC or BCD?
>
> terrey shih wrote:
>>
>>
>> Hi,
>>
>> One more thing. I would like to know if one serializer can be used
>> for two table.
>>
>> thanks,
>> -Gary
>>
>> server-agent.sources = mySrc
>> server-agent.sinks = hbase-sink1 hbase-sink2
>> server-agent.channels = C1 C2
>>
>> # Describe/configure the source
>> server-agent.sources.mySrc.type = avro
>> server-agent.sources.mySrc.bind = 127.0.0.1
>> server-agent.sources.mySrc.port = 5000
>>
>> # Use a channel which buffers events in memory
>> server-agent.channels.C1.type = memory
>> server-agent.channels.C1.capacity = 1000
>> server-agent.channels.C1.transactionCapacity = 100
>> server-agent.sinks.hbase-sink1.type = asynchbase
>> server-agent.sinks.hbase-sink1.table = table1
>> server-agent.sinks.hbase-sink1.columnFamily = fam1
>> server-agent.sinks.hbase-sink1.batchSize = 1000
>> server-agent.sinks.hbase-sink1.serializer =
>> com.test.flume.server.HBaseSinkSerializer
>> server-agent.sinks.hbase-sink1.channel = C1
>> server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3
>>
>> server-agent.channels.C2.type = memory
>> server-agent.channels.C2.capacity = 1000
>> server-agent.channels.C2.transactionCapacity = 100
>> server-agent.sinks.hbase-sink2.type = asynchbase
>> server-agent.sinks.hbase-sink2.table = table2
>> server-agent.sinks.hbase-sink2.columnFamily = fam2
>> server-agent.sinks.hbase-sink2.batchSize = 1000
>> server-agent.sinks.hbase-sink2.serializer =
>> com.test.flume.server.HBaseSinkSerializer
>> server-agent.sinks.hbase-sink2.channel = C2
>> server-agent.sinks.hbase-sink2.serializer.columns = table2Col
>>
>> # Bind the source and sink to the channel
>> server-agent.sources.mySrc.channels = C1 C2
>> server-agent.sources.avor-Src.selector.type = multiplexing
>> server-agent.sources.avor-Src.selector.header = DataSrc
>> server-agent.sources.avor-Src.selector.mapping.ABC = C1
>> server-agent.sources.avor-Src.selector.mapping.BCD = C2
>>
>>
>>
>> On Fri, Aug 8, 2014 at 11:19 AM, terrey shih
>> <terreyshih@gmail.com <mailto:terreyshih@gmail.com>
>> <mailto:terreyshih@gmail.com <mailto:terreyshih@gmail.com>>> wrote:
>>
>> HI,
>>
>> Here is my sample config.
>>
>>
>>
>>
>> On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan
>> <hshreedharan@cloudera.com <mailto:hshreedharan@cloudera.com>
>> <mailto:hshreedharan@cloudera.com
>> <mailto:hshreedharan@cloudera.com>>> wrote:
>>
>> Can you please send your config? That would make it easier to
>> understand.
>>
>>
>> terrey shih wrote:
>>>
>>>
>>>
>>> I have a fanning out operation where I have one source and
>>> and based
>>> on the event headers (headers are added from the source
>>> input), I
>>> would like to channel the event to different Hbase
>>> tables. I
>>> am using
>>> the same serializer for the hbase tables.
>>
>>
>>
>>
>

Mime
View raw message