flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Suresh" <Suresh.Kum...@emc.com>
Subject Flume Source and Sink in different hosts
Date Thu, 04 Oct 2012 21:53:49 GMT
Hello:

 

I have just downloaded and build flume-ng (apache-flume-1.3.0-SNAPSHOT).

 

My goal is to collect log data from HostA (source) and send it to
HostB(sink), my initial test (sending /etc/passwd) 

from HostA to HostB worked fine, I was also able to load the passwd file
into my HBase in HostB.

 

Now, I want to load a continuous stream of log data (using tail -f), but
I was not able to replicate the above process.

Flume just started fine in HostA, but I do not see any data being
received by HostB or in my HBase.

 

What is wrong with my configuration?

 

Thanks,

Suresh

 

Here is my flume.conf in HostA

 

agent3.sources = tail

agent3.channels = MemoryChannel-1

agent3.sinks = avro-sink

 

# Define source flow

agent3.sources.tail.type = exec

agent3.sources.tail.command = tail -f /var/log/auth.log

agent3.sources.tail.channels = MemoryChannel-1

 

# What kind of channel

agent3.channels.MemoryChannel-1.type = memory

 

# avro sink properties

agent3.sinks.avro-sink.type = avro

agent3.sinks.avro-sink.channel = MemoryChannel-1

agent3.sinks.avro-sink.hostname = hostb

agent3.sinks.avro-sink.port = 41414

 

Here is my flume.conf in HostB

 

# Define a memory channel called ch1 on agent1

agent1.channels.ch1.type = memory

 

# Define an Avro source called avro-source1 on agent1 and tell it

# to bind to 0.0.0.0:41414. Connect it to channel ch1.

agent1.sources.avro-source1.channels = ch1

agent1.sources.avro-source1.type = avro

agent1.sources.avro-source1.bind = 0.0.0.0

agent1.sources.avro-source1.port = 41414

 

# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

agent1.sinks.log-sink1.channel = ch1

agent1.sinks.log-sink1.type = logger

 

# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

agent1.channels = ch1

agent1.sources = avro-source1

#agent1.sources = avro-source1

agent1.sinks = sink1

 

agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink

agent1.sinks.sink1.channel = ch1

agent1.sinks.sink1.table = flumedemo

agent1.sinks.sink1.columnFamily = testing

agent1.sinks.sink1.column = foo

agent1.sinks.sink1.serializer =
org.apache.flume.sink.hbase.SimpleHbaseEventSerializer

agent1.sinks.sink1.serializer.payloadColumn = col1

agent1.sinks.sink1.serializer.keyType = timestamp

agent1.sinks.sink1.serializer.rowPrefix = 1

agent1.sinks.sink1.serializer.suffix = timestamp

agent1.sinks.sink1.serializer.payloadColumn = pcol

agent1.sinks.sink1.serializer.incrementColumn = icol

 


Mime
View raw message