flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Analysis of Data
Date Fri, 08 Feb 2013 08:48:03 GMT
Another thought - for streaming analytics you'd need a system which scales
so in retrospective how about something like a STORM SINK which internally
can use FLUME again to write the processed event to a persistent SINK.

- Inder

On Fri, Feb 8, 2013 at 10:25 AM, Nitin Pawar <nitinpawar432@gmail.com>wrote:

> Hi Steve,
> I can understand  the idea of having data processed inside flume by
> streaming it to another flume agent. But do we really need to re-engineer
> something inside flume is what I am thinking? Core flume dev team may have
> better ideas on this but currently for streaming data processing storm is a
> huge candidate.
> flume does have have an open jira on this integration FLUME-1286<https://issues.apache.org/jira/browse/FLUME-1286>
> It will be interesting to draw up the comparisons in performance if the
> data processing logic is added to to flume. We do see currently people
> having a little bit of pre-processing of their data (they have their own
> custom channel types where they modify the data and sink it)
> On Fri, Feb 8, 2013 at 8:52 AM, Steve Yates <syates@stevendyates.com>wrote:
>> Thanks for your feedback Mike, I have been thinking about this a little
>> more and just using Mahout as an example I was considering the concept of
>> somehow developing an enriched 'sink' so to speak where it would accept
>> input streams / msgs from a flume channel and onforward specifically to a
>> 'service' i.e Mahout service which would subsequently deliver the results
>> to the configured sink. So yes it would behave as an
>> intercept->filter->process->sink for applicable data items.
>> I apologise if that is still vague. It would be great to receive further
>> feedback from the user group.
>> -Steve
>> Mike Percy <mpercy@apache.org> wrote:
>> Hi Steven,
>> Thanks for chiming in! Please see my responses inline:
>> On Thu, Feb 7, 2013 at 3:04 PM, Steven Yates <syates@stevendyates.com>wrote:
>>> The only missing link within the Flume architecture I see in this
>>> conversation is the actual channel's and brokers themselves which
>>> orchestrate this lovely undertaking of data collection.
>> Can you define what you mean by channels and brokers in this context?
>> Since channel is a synonym for queueing event buffer in Flume parlance.
>> Also, can you elaborate more on what you mean by orchestration? I think I
>> know where you're going but I don't want to put words in your mouth.
>> One opportunity I do see (and I may be wrong) is for the data to
>>> offloaded into a system such as Apache Mahout  before being sent to the
>>> sink. Perhaps the concept of a ChannelAdapter of sorts? I.e Mahout Adapter
>>> ? Just thinking out loud and it may be well out of the question.
>> Why not a Mahout sink? Since Mahout often wants sequence files in a
>> particular format to begin its MapReduce processing (e.g. its k-Means
>> clustering implementation), Flume is already a good fit with its HDFS sink
>> and EventSerializers allowing for writing a plugin to format your data
>> however it needs to go in. In fact that works today if you have a batch
>> (even 5-minute batch) use case. With today's functionality, you could use
>> Oozie to coordinate kicking off the Mahout M/R job periodically, as new
>> data becomes available and the files are rolled.
>> Perhaps even more interestingly, I can see a use case where you might
>> want to use Mahout to do streaming / realtime updates driven by Flume in
>> the form of an interceptor or a Mahout sink. If online machine learning
>> (e.g. stochastic gradient descent or something else online) was what you
>> were thinking, I wonder if there are any folks on this list who might have
>> an interest in helping to work on putting such a thing together.
>> In any case, I'd like to hear more about specific use cases for streaming
>> analytics. :)
>> Regards,
>> Mike
> --
> Nitin Pawar

- Inder
"You are average of the 5 people you spend the most time with"

View raw message