flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: HTTP Source
Date Tue, 06 Nov 2012 18:55:56 GMT
Hi Nathaniel, 

When I wrote this (and while it was being reviewed), we concluded it was not a good idea to
expose the response object to the custom handler which can be plugged in because the handler
can flush the buffer of the response object (which also can be flushed just because it is
full) and then we could hit some exception (like channel being full) causing inconsistencies
in the response. This source is not meant to be used for HTTP based RPC etc, this is to be
used to send the data to the source and then wait for success or failure - on success send
next event, on failure - back off for a while and retry (or similar logic).

For RPC etc, it might be best to write a source specific to your use-case. In case of this
source bundled with Flume, we had to make it general enough to be useful but cautious enough
to not make it confuse applications sending the data into Flume.

Hope this helps.

Thanks,
Hari

-- 
Hari Shreedharan


On Monday, November 5, 2012 at 10:09 AM, Nathaniel Auvil wrote:

> One thing i do not see with this HTTPSource is any way to customize the response.
> 
> 
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)>
wrote:
> > Very soon. I am going to wait for the latest rev of this patch
> > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > committed and either way I will build the RC.
> > 
> > Brock
> > 
> > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > > is there a target date for 1.3?
> > >
> > >
> > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)>
wrote:
> > >>
> > >> There is actually an HTTP Source committed to the 1.3 branch:
> > >>
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > >>
> > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > >> conflict.
> > >>
> > >> Brock
> > >>
> > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > >> <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > >> > embedding
> > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > >> > work?
> > >> > It looks like my latest issue is a servlet-api conflict due to an
> > >> > ancient
> > >> > Jetty 6 version on the Flume classpath
> > >>
> > >>
> > >>
> > >> --
> > >> Apache MRUnit - Unit testing MapReduce -
> > >> http://incubator.apache.org/mrunit/
> > >
> > >
> > 
> > 
> > 
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> 


Mime
View raw message