Indeed it can be stripped down to the question whether jms read transactions are committed as a batch or as individual reads.

On Wednesday, 19 October 2016, Roberto Coluccio <roberto.coluccio@eng.it> wrote:
Hi,

When leveraging a JMS (Tibco implementation) source to consume messages from a queue, are such messages immediately consumed , hence removed from the queue, even if the flume transaction hasn't been closed yet? What happens if the transaction is not committed (e.g. in case of errors)?

For instance, in a topology made by:

jms source -> memory or file channel -> hdfs sink

with a configured batch size of 100, when 99 of them are read from JMS and written to the channel, if an error occurs when trying to read the 100th, to the best of my knowledge the whole batch of 100 is discarded, the transaction canceled, and the messages tried to be re-processed.

In such a scenario, are the first 99 messages already consumed from JMS to be considered as lost, supposing the ones in the channel are lost as well (because of other issues)?

thanks a lot in advance.

Best regards,
Roberto


--

Attila Simon
Software Engineer
Email:   sati@cloudera.com

 Cloudera Inc.