flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Moore <m...@crowdmob.com>
Subject Re: S3 Sink in FlumeNG Configuration?
Date Fri, 29 Mar 2013 15:47:48 GMT
Hey,

Thanks for the links to the Jiras.  It seems like someone implemented
an S3BufferedWriter which might be helpful in the future.

However, I'm still not sure what to set the configuration (flume.conf) to
use s3 as a sink?  Has anyone done that?

Best,
Matt
--
Matthew Moore
Co-Founder & CTO, CrowdMob Inc.
Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!
matt@crowdmob.com


On Fri, Mar 29, 2013 at 7:49 AM, Brock Noland <brock@cloudera.com> wrote:

> Sorry, I don't know much about this, but here are two relevant JIRA's:
>
> https://issues.apache.org/jira/browse/FLUME-1228
> https://issues.apache.org/jira/browse/FLUME-951
>
>
> On Fri, Mar 29, 2013 at 9:44 AM, Matthew Moore <matt@crowdmob.com> wrote:
>
>> Hey there,
>>
>> I know this is a really newbish question, but I'm hoping to get a little
>> assistance here so I'm not stuck guess-and-checking.
>>
>> I'm trying to figure out how to configure FlumeNG (1.3.1), but I couldn't
>> figure out how to setup the hdfs sink to use the s3 implementations.
>>
>> I'm keeping track of my progress on this gist I made:
>> https://gist.github.com/crowdmatt/5256881
>>
>> From what I've gathered, I should be using the hdfs type, which I'm
>> setting up as such:
>>
>> agent.sinks = s3Sink
>> agent.sinks.s3Sink.type = hdfs
>> agent.sinks.s3Sink.channel = recoverableMemoryChannel
>>
>> ... but that's where I end up hitting my head against the wall.  I know I
>> should be specifying my s3 access key, secret, and bucket in this format:
>> s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@my-hdfs/
>>
>> However, I don't know where to specify that, or what dot notation to use.
>>
>> Can anyone point me in the right direction?
>>
>> Best,
>> Matt
>> --
>> Matthew Moore
>> Co-Founder & CTO, CrowdMob Inc.
>> Mobile: (650) 888-5962
>>
>> Need to schedule a meeting?  Invite me via Google Calendar!
>> matt@crowdmob.com
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>

Mime
View raw message