flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From richard ross <richard.prion...@gmail.com>
Subject Re: Flume JMS source
Date Tue, 04 Mar 2014 14:43:19 GMT
Hello Rajesh:

I just worked on a POC for the work that I am doing with JMS --> HDFS
configuration using ActiveMQ. I was able to use everything right out of the
box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the
guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example
config that I was playing around with. Note that for ActiveMQ, you do not
want to specify the connectionFactory in the source configuration (
https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation
defect was created for this point (
https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and
4, I will let someone with more experience answer.

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe the source
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.providerURL = tcp://<activemqserver>:61616
a1.sources.r1.destinationName = TESTQUEUE
a1.sources.r1.destinationType = QUEUE

# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
a1.sinks.k1.hdfs.filePrefix = messages
a1.sinks.k1.hdfs.rollInterval = 30
#a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.rollCount = 1000
a1.sinks.k1.hdfs.fileType = DataStream

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Hope this helps,
Richard.


On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <
mrajaforu@gmail.com> wrote:

> Hello Team,
>
> I'm new to flume. Could you please help me on below query
>
>    1. How to configure JMS source(TIBCO and ActiveMQ) in flume?
>    2. Do i need to write custom source for above requirement? (Or) Out Of
>    the Box flume JMS source implementation is fine?
>    3. What is the best channel mechanism for above requirement?
>    4. Each message will be sink into HDFS? (or) Can we consolidate some
>    messages and sink into HDFS?
>
> Thank you in advance
>
> Regards,
> Rajesh
>

Mime
View raw message