flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: log dump tool?
Date Tue, 09 Jun 2015 02:49:04 GMT
Somehow documentation has not been updated. Let me check the JIRA.

Look into this flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java
It has an example, you can use a simple logger or a simple file
writing utility to dump the contents.

I shall be writing a blog post on the same topic soon. Been busy lately.


On Mon, Jun 8, 2015 at 7:39 PM, Hari Shreedharan
<hshreedharan@cloudera.com> wrote:
> Flume has a tool that will allow you to run all events in the file channel through a
piece of custom code you’d supply:
> bin/flume-ng tool FCINTEGRITYTOOL
> You can see the arguments you’d need to supply when you execute this command.
> Thanks,
> Hari Shreedharan
>> On Jun 8, 2015, at 7:30 PM, Robert B Hamilton <robert.hamilton@gm.com> wrote:
>> Is there anything like a logdump tool for flume file channel?
>> Specifically I+IBk-m looking for some way to extract say the event data for the last
N puts.
>> Alternatively can the logs be modified so that the last N (sink) commits will be
ignored on restart?
>> The scenario that I+IBk-m concerned about is this:
>> 1.       server crashes, flume is restarted once the server is brought back.
>> 2.       End user sees something odd in his HiveQL and speculates that data was lost.
>> 3.       We peek into the WAL as they existed just before the restart (we saved off
a copy) and either
>> a.       Find an event corresponding to the missing data and use that to fix the
data in the destination, or
>> b.      Prove that the event corresponding to the missing data was not present at
least as far back as the logs go
>> I+IBk-m just wondering if there is a tool which makes number 3 possible+ICY.
>> Nothing in this message is intended to constitute an electronic signature unless
a specific statement to the contrary is included in this message.
>> Confidentiality Note: This message is intended only for the person or entity to which
it is addressed. It may contain confidential and/or privileged material. Any review, transmission,
dissemination or other use, or taking of any action in reliance upon this message by persons
or entities other than the intended recipient is prohibited and may be unlawful. If you received
this message in error, please contact the sender and delete it from your computer.


Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

View raw message