flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muhammad Yaseen <yaseenaftab...@gmail.com>
Subject Re: Looking for a specific input on JMS source
Date Thu, 31 Aug 2017 07:17:27 GMT
Hello,

I think you can achieve what you want with the JMSSourceConverter. Looking
at the Flume documentation it says that "The JMS source allows pluggable
converters, though it’s likely the default converter will work for most
purposes. The default converter is able to convert Bytes, Text, and Object
messages to FlumeEvents. In all cases, the properties in the message are
added as headers to the FlumeEvent".

Perhaps you can implement the JSMMessageConverter interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConverter.java).
The default converted also implements this interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java)
and also provides a good reference point for implementing you own.

You custom code would go into plugins.d directory in flume. More details
here: https://flume.apache.org/FlumeUserGuide.html#jms-source

On Thu, Aug 31, 2017 at 12:05 PM, Kajal K Das <kajal.das@in.ibm.com> wrote:

> Hi All,
>
> We are designing a near-real time pipeline in which we will receive
> messages from IBM MQ. On the Hadoop side, we are planning to use Flume to
> ingest the messages using JMS source.
> The messages will include some custom header information, which we need to
> extract as well. We discovered that interceptors cannot be configured with
> a JMS source.
> In that case: what are the alternatives in order to extract the message
> headers using Flume? Or should we try something else (e.g. Kafka)?
>
>
>
> Regards
> Kajal Kr. Das
>
>

Mime
View raw message