flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Monitoring the progress of events
Date Mon, 26 Jan 2015 22:20:00 GMT
Assuming you're using the AvroSource, the application will have to do
something like this:

1. Send all of the events to the source
2. Send a specially crafted canary event to the source
3. Monitor the destination where the sink writes looking for the canary event

You might also be able to monitor the counters that the agent
maintains to see if the expected number of events have been processed
by the sink.


On Mon, Jan 26, 2015 at 2:17 PM, Carlotta Hicks <Carlotta.Hicks@sas.com> wrote:
> I've created a simple agent with a source, a channel and a sink.  How can an application
know when all events sent to the source have been processed by the sink?
> -----Original Message-----
> From: Joey Echeverria [mailto:joey@cloudera.com]
> Sent: Monday, January 26, 2015 5:06 PM
> To: user@flume.apache.org
> Subject: Re: Monitoring the progress of events
> @Seshu: If you emailed user-unsubscribe@flume.apache.org and it still didn't unsubscribe
you you should file a JIRA with Apache's infrastructure team:
> https://issues.apache.org/jira/browse/INFRA/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel
> @Carlotta:
> What kind of monitoring are you looking for? I don't believe there is event-level monitoring
as this would be very expensive. What I've seen some folks do is send a canary event to verify
when processing of some batch has completed. This won't work if you have fan out/fan in in
the flow as the ordering between multiple Flume agent's isn't guaranteed.
> -Joey
> On Mon, Jan 26, 2015 at 12:52 PM, Seshu V <sesh12@gmail.com> wrote:
>> Please take my id off of this group.  Unsubscribe doesn't seem to be
>> working.
>> thanks.
>> On Mon, Jan 26, 2015 at 2:48 PM, Carlotta Hicks
>> <Carlotta.Hicks@sas.com>
>> wrote:
>>> Is there a way to monitor the progress of events from the time they
>>> are sent to the agent until they have been processed by the sink?  Is
>>> there a status for an agent?
> --
> Joey Echeverria

Joey Echeverria

View raw message