flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: Getting Data from S3
Date Fri, 19 Aug 2011 07:48:39 GMT

This isn't available out-of-the-box currently but one could create a custom
source that can grab data from s3.

This would be an interesting source plugin. (I've thought about building
something similar that uses an hdfs dir as a source).


On Thu, Aug 18, 2011 at 11:24 AM, Jonathan <jonny2112@gmail.com> wrote:

> Hi,
> I was wondering if Flume supports retrieving data from an S3 bucket and
> sinking into HDFS. The s3 bucket is getting updated on a regular basis so I
> would like for flume to catch any changes on s3 and move them accordingly.
>  I have seen a lot of documentation on using S3 as a sink but I haven't been
> able to find anything going the other way. Is this even possible?
> Thanks
> Jonathan

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

View raw message