flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Suresh" <Suresh.Kum...@emc.com>
Subject Flume Source and Sink in different hosts
Date Thu, 04 Oct 2012 21:53:49 GMT


I have just downloaded and build flume-ng (apache-flume-1.3.0-SNAPSHOT).


My goal is to collect log data from HostA (source) and send it to
HostB(sink), my initial test (sending /etc/passwd) 

from HostA to HostB worked fine, I was also able to load the passwd file
into my HBase in HostB.


Now, I want to load a continuous stream of log data (using tail -f), but
I was not able to replicate the above process.

Flume just started fine in HostA, but I do not see any data being
received by HostB or in my HBase.


What is wrong with my configuration?





Here is my flume.conf in HostA


agent3.sources = tail

agent3.channels = MemoryChannel-1

agent3.sinks = avro-sink


# Define source flow

agent3.sources.tail.type = exec

agent3.sources.tail.command = tail -f /var/log/auth.log

agent3.sources.tail.channels = MemoryChannel-1


# What kind of channel

agent3.channels.MemoryChannel-1.type = memory


# avro sink properties

agent3.sinks.avro-sink.type = avro

agent3.sinks.avro-sink.channel = MemoryChannel-1

agent3.sinks.avro-sink.hostname = hostb

agent3.sinks.avro-sink.port = 41414


Here is my flume.conf in HostB


# Define a memory channel called ch1 on agent1

agent1.channels.ch1.type = memory


# Define an Avro source called avro-source1 on agent1 and tell it

# to bind to Connect it to channel ch1.

agent1.sources.avro-source1.channels = ch1

agent1.sources.avro-source1.type = avro

agent1.sources.avro-source1.bind =

agent1.sources.avro-source1.port = 41414


# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

agent1.sinks.log-sink1.channel = ch1

agent1.sinks.log-sink1.type = logger


# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

agent1.channels = ch1

agent1.sources = avro-source1

#agent1.sources = avro-source1

agent1.sinks = sink1


agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink

agent1.sinks.sink1.channel = ch1

agent1.sinks.sink1.table = flumedemo

agent1.sinks.sink1.columnFamily = testing

agent1.sinks.sink1.column = foo

agent1.sinks.sink1.serializer =

agent1.sinks.sink1.serializer.payloadColumn = col1

agent1.sinks.sink1.serializer.keyType = timestamp

agent1.sinks.sink1.serializer.rowPrefix = 1

agent1.sinks.sink1.serializer.suffix = timestamp

agent1.sinks.sink1.serializer.payloadColumn = pcol

agent1.sinks.sink1.serializer.incrementColumn = icol


View raw message