flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Configure and start a Flume sink + agent from java.
Date Tue, 20 May 2014 13:46:34 GMT
Hi flume !

I'd like to implement a simple flume sink:

agent.channels.memory-channel.type = memory
agent.channels.memory-channel.capacity = 2000
agent.channels.memory-channel.transactionCapacity = 100
agent.sources.tail-source.type = exec
agent.sources.tail-source.command = tail -F /tmp/flume-smoke
agent.sources.tail-source.channels = memory-channel

agent.sinks.log-sink.channel = memory-channel
agent.sinks.log-sink.type = logger

# Define a sink that outputs to the DFS
agent.sinks.hdfs-sink.channel = memory-channel
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = myfs:///tmp/flume-test
agent.sinks.hdfs-sink.hdfs.fileType = DataStream

# activate the channels/sinks/sources
agent.channels = memory-channel
agent.sources = tail-source
agent.sinks = log-sink hdfs-sink


But id like to do this dynamically, i.e. without having to package any
static text files.

Is there a simple way to

1) Create the above sink definition and
2) To start something like "timeout 60 flume-ng agent --conf ./flume/conf/
-f ./flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent"

All in Java, ?

-- 
Jay Vyas
http://jayunit100.blogspot.com

Mime
View raw message