flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@ntent.com>
Subject RE: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Sat, 09 Aug 2014 01:41:29 GMT
There is a configuration error in your multiplexing channel selector section. You are referencing
‘server-agent.sources.avor-Src.’ and it should be ‘server-agent.sources.mySrc.’. Otherwise,
the configuration looks good and should satisfy your requirements.

From: terrey shih [mailto:terreyshih@gmail.com]
Sent: Friday, August 08, 2014 1:20 PM
To: user@flume.apache.org
Subject: Re: question using multiplexing and the the same serializer for multiple sinks to
multiple hbase tables


One more thing.  I would like to know if one serializer can be used for two table.

server-agent.sources = mySrc
server-agent.sinks = hbase-sink1 hbase-sink2
server-agent.channels = C1 C2

# Describe/configure the source
server-agent.sources.mySrc.type = avro
server-agent.sources.mySrc.bind =
server-agent.sources.mySrc.port = 5000

# Use a channel which buffers events in memory
server-agent.channels.C1.type = memory
server-agent.channels.C1.capacity = 1000
server-agent.channels.C1.transactionCapacity = 100
server-agent.sinks.hbase-sink1.type = asynchbase
server-agent.sinks.hbase-sink1.table = table1
server-agent.sinks.hbase-sink1.columnFamily = fam1
server-agent.sinks.hbase-sink1.batchSize = 1000
server-agent.sinks.hbase-sink1.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink1.channel = C1
server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3

server-agent.channels.C2.type = memory
server-agent.channels.C2.capacity = 1000
server-agent.channels.C2.transactionCapacity = 100
server-agent.sinks.hbase-sink2.type = asynchbase
server-agent.sinks.hbase-sink2.table = table2
server-agent.sinks.hbase-sink2.columnFamily = fam2
server-agent.sinks.hbase-sink2.batchSize = 1000
server-agent.sinks.hbase-sink2.serializer = com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink2.channel = C2
server-agent.sinks.hbase-sink2.serializer.columns = table2Col

# Bind the source and sink to the channel
server-agent.sources.mySrc.channels = C1 C2
server-agent.sources.avor-Src.selector.type = multiplexing
server-agent.sources.avor-Src.selector.header = DataSrc
server-agent.sources.avor-Src.selector.mapping.ABC = C1
server-agent.sources.avor-Src.selector.mapping.BCD = C2

On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <terreyshih@gmail.com<mailto:terreyshih@gmail.com>>

Here is my sample config.

On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan <hshreedharan@cloudera.com<mailto:hshreedharan@cloudera.com>>
Can you please send your config? That would make it easier to understand.

terrey shih wrote:

I have a fanning out operation where I have one source and and based
on the event headers (headers are added from the source input), I
would like to channel the event to different Hbase tables.  I am using
the same serializer for the hbase tables.

View raw message