flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Coluccio <roberto.coluc...@eng.it>
Subject JMS (Tibco) Source - input messages status during transaction
Date Wed, 19 Oct 2016 15:09:22 GMT

When leveraging a JMS (Tibco implementation) source to consume messages 
from a queue, are such messages immediately consumed , hence removed 
from the queue, even if the flume transaction hasn't been closed yet? 
What happens if the transaction is not committed (e.g. in case of errors)?

For instance, in a topology made by:

jms source -> memory or file channel -> hdfs sink

with a configured batch size of 100, when 99 of them are read from JMS 
and written to the channel, if an error occurs when trying to read the 
100th, to the best of my knowledge the whole batch of 100 is discarded, 
the transaction canceled, and the messages tried to be re-processed.

In such a scenario, are the first 99 messages already consumed from JMS 
to be considered as lost, supposing the ones in the channel are lost as 
well (because of other issues)?

thanks a lot in advance.

Best regards,

View raw message