flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sri Ramya <ramya.1...@gmail.com>
Subject Re: flume-cassandra
Date Fri, 01 Feb 2013 14:32:28 GMT
hi

I am unable to talk with cassandra using flume-ng sink. Can you tell me
what is the configuration file to give to flume agent.


On Thu, Jan 31, 2013 at 4:26 PM, Sri Ramya <ramya.1255@gmail.com> wrote:

> thanks for ur support,
>  i will get back to u for farther help
>
>
> On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers <ralph.goers@dslextreme.com>wrote:
>
>> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its
>> own Cluster of hosts is correct.  Just as all the other Flume Sinks, the
>> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to
>> Cassandra using one of the supported protocols. I believe we are using
>> Hector but you could use Astyanax, Thrift or I suppose even CQL.  You could
>> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor
>> it to meet your needs.
>>
>> Ralph
>>
>>
>> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>>
>>
>>  I am running flume-ng on one host and Cassandra  on another host. How i
>> have to connect it. Which host i have to install flume-casandra sink? Do i
>> need to run flume-ng on the host which is having Cassandra?
>>
>> Please reply me...
>>
>>
>> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <ralph.goers@dslextreme.com
>> > wrote:
>>
>>> We ended up writing our own custom sink for Cassandra so that we could
>>> customize the fields, write to Cassandra indexes and multiple column
>>> families.  We primarily use Cassandra to store log records and we insure
>>> they can't be duplicated by generating a guid in the Appender that writes
>>> to Flume (via Avro).  The guid is then used as the row key.
>>>
>>> What do you mean by separation?
>>>
>>> We use Log4j 2's Flume Appender with the RFC5424Layout and
>>> StructuredDataMessages (see EventLogger).  All the individual items in the
>>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
>>> the FlumeEvent data map.  The Cassandra Sink then writes these to
>>> individual columns in Cassandra.  The Cassandra Sink also writes these
>>> items to the appropriate indexes so that we can perform whatever queries
>>> that perform well.  The ipAddress is definitely one of the items we are
>>> interested in and capture in our indexes.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>>>
>>> hi Ralph
>>>
>>> I am able to integrate flume with Cassandra. But the problem is ' if i
>>> send a single event to Cassandra to store, it is storing multiple row in
>>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>>> i am staring work with Flume-ng.
>>>
>>> Make me clear in one thing, Is there any separation of storing data in
>>> Cassandra based on flume-ng agent(IP address)???
>>>
>>> Thank you
>>>
>>>
>>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <
>>> ralph.goers@dslextreme.com> wrote:
>>>
>>>> This is absolutely NOT correct.   We are using Flume NG to log to
>>>> Cassandra.   We fought with Flume OG for months trying to get it to work
>>>> but it was never stable.  The architecture of Flume OG had serious
>>>> problems, which is why Flume NG was created.
>>>>
>>>> Ralph
>>>>
>>>>
>>>>
>>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>>
>>>> Hi ,
>>>> Can you give details like your input format and all ...
>>>> I think going with flume 0.9.4 for cassandra is better.
>>>>
>>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mpercy@cloudera.com>wrote:
>>>>
>>>>> What do you mean by "collector"?
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ramya.1255@gmail.com>wrote:
>>>>>
>>>>>>  Thank you very much. But I need a collector in my application,
>>>>>> flume-ng  does not have any collector. Thats why i started with Cloudeera
>>>>>> flum. Now i am getting a problem with integrating flume collector
with
>>>>>> cassandra.  If any body tried it before please help me.
>>>>>> thank in advance.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mpercy@cloudera.com>wrote:
>>>>>>
>>>>>>> Hi Sri,
>>>>>>> Cloudera originally created Flume, then contributed it to the
Apache
>>>>>>> Software Foundation (ASF), and continues to invest heavily into
it under
>>>>>>> the auspices of the ASF. The current generation of Flume is called
Flume
>>>>>>> NG. I encourage you to use the latest "NG" generation of Flume
(version
>>>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mike
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ramya.1255@gmail.com>wrote:
>>>>>>>
>>>>>>>>  Thank you very much.
>>>>>>>> Can u tell me whats the difference between cloudera flume
and
>>>>>>>> flume-ng . And which is the better option to use.
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You can use develop your own customized sink which can
do
>>>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Som
>>>>>>>>>
>>>>>>>>>  On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rcamp@ebay.com>wrote:
>>>>>>>>>
>>>>>>>>>> You may also want to check out
>>>>>>>>>>
>>>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>>>
>>>>>>>>>> Roy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>>>> To: user@flume.apache.org
>>>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>>>
>>>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration?
How did that go?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya
wrote:
>>>>>>>>>> >    hi
>>>>>>>>>> >
>>>>>>>>>> >    How to intigrate flume with cassandra by
using logsandra
>>>>>>>>>> plugin.
>>>>>>>>>> >
>>>>>>>>>> >    thank you.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Mime
View raw message