flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Monitoring the progress of events
Date Mon, 26 Jan 2015 22:31:40 GMT
You should be able to use the channelsize

On Mon, Jan 26, 2015 at 2:29 PM, Carlotta Hicks <Carlotta.Hicks@sas.com>
wrote:

> Are these the counters from MonitoredCounterGroup?  What is the scope of
> these counters?  Can you reset these counters?
>
> -----Original Message-----
> From: Joey Echeverria [mailto:joey@cloudera.com]
> Sent: Monday, January 26, 2015 5:20 PM
> To: user@flume.apache.org
> Subject: Re: Monitoring the progress of events
>
> Assuming you're using the AvroSource, the application will have to do
> something like this:
>
> 1. Send all of the events to the source
> 2. Send a specially crafted canary event to the source 3. Monitor the
> destination where the sink writes looking for the canary event
>
> You might also be able to monitor the counters that the agent maintains to
> see if the expected number of events have been processed by the sink.
>
> -Joey
>
> On Mon, Jan 26, 2015 at 2:17 PM, Carlotta Hicks <Carlotta.Hicks@sas.com>
> wrote:
> > I've created a simple agent with a source, a channel and a sink.  How
> can an application know when all events sent to the source have been
> processed by the sink?
> >
> > -----Original Message-----
> > From: Joey Echeverria [mailto:joey@cloudera.com]
> > Sent: Monday, January 26, 2015 5:06 PM
> > To: user@flume.apache.org
> > Subject: Re: Monitoring the progress of events
> >
> > @Seshu: If you emailed user-unsubscribe@flume.apache.org and it still
> didn't unsubscribe you you should file a JIRA with Apache's infrastructure
> team:
> >
> > https://issues.apache.org/jira/browse/INFRA/?selectedTab=com.atlassian
> > .jira.jira-projects-plugin:summary-panel
> >
> > @Carlotta:
> >
> > What kind of monitoring are you looking for? I don't believe there is
> event-level monitoring as this would be very expensive. What I've seen some
> folks do is send a canary event to verify when processing of some batch has
> completed. This won't work if you have fan out/fan in in the flow as the
> ordering between multiple Flume agent's isn't guaranteed.
> >
> > -Joey
> >
> > On Mon, Jan 26, 2015 at 12:52 PM, Seshu V <sesh12@gmail.com> wrote:
> >> Please take my id off of this group.  Unsubscribe doesn't seem to be
> >> working.
> >>
> >> thanks.
> >>
> >> On Mon, Jan 26, 2015 at 2:48 PM, Carlotta Hicks
> >> <Carlotta.Hicks@sas.com>
> >> wrote:
> >>>
> >>> Is there a way to monitor the progress of events from the time they
> >>> are sent to the agent until they have been processed by the sink?
> >>> Is there a status for an agent?
> >>>
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Joey Echeverria
>
>
>
> --
> Joey Echeverria
>

Mime
View raw message