flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: Delete individual message from queue
Date Wed, 04 Feb 2015 19:32:51 GMT
It is not as easy, since the channel does not know what an event looks like or why the transaction
is being rolled back. That is something that is being handled by the serializer and sink.
We need to some how remove an event from the channel if it the channel is broken. We might
perhaps have to add a new interface that allows the sink/serializer to tell the channel “forget”a
specific event, and that event will be dropped from the channel and transaction.

I don’t see how we can do it outside the sink for this reason.

Thanks, Hari

On Wed, Feb 4, 2015 at 5:32 AM, Ashish <paliwalashish@gmail.com> wrote:

> Is it possible to extend File Channel Integrity tool to support
> filtering out corrupt events? Something like once we get a record and
> it's not corrupt, provide a placeholder function to validate Event
> implemented by user. I still don't have too much insight into
> FileChannel implementation.
> On Tue, Feb 3, 2015 at 12:49 AM, Hari Shreedharan
> <hshreedharan@cloudera.com> wrote:
>> Currently, no - there is no such tool, but this is a request that has come
>> up time and again. Can you file a jira for this? If someone has time, they’d
>> probably pick it up
>> Thanks,
>> Hari
>> On Mon, Feb 2, 2015 at 10:52 AM, Charles McLaughlin <charles@nextdoor.com>
>> wrote:
>>> Hello,
>>> We had a situation where one of our Flume agents got stuck on a message
>>> due to unexpected format.  To get things moving again, I stopped the Flume
>>> agent, moved the file backed channel data out of the way and re-started the
>>> Flume agent.  I'd like to pop the bad message from the queue data on disk...
>>> are there any tools or recommended ways to do this?
>>> Thanks,
>>> Charles
> -- 
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
View raw message