flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shadyxu <shad...@gmail.com>
Subject Re: How to separate json message into different columns when using ES sink?
Date Wed, 19 Nov 2014 09:02:19 GMT
Hi Nickolay,

Thank you and please upload the repo on Github.

2014-11-19 16:41 GMT+08:00 Nickolay Kolev <lem0na2@gmail.com>:

> Hi,
>
> It looks like ES sink is oriented to handle text payload and not json and
> there are problems with serialization.
> We have the same problem and I have developed patch and custom version
> based on sources (two months ago)
> I am not java developer so probably it is not the best solution but it
> works.
> Now it is hosted on company's internal repo but if you have interest I can
> upload it on GitHub or Bitbucket
>
> best regards,
> Nickolay Kolev
>
>
> On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <shadyxu@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I am now using Flume to collect log into ElasticSearch. And the logs are
>> in json format. However, when I check them in ES, it seems that Flume has
>> put the entire json log in the @message attribute. Is there any config to
>> do or need I do some coding to separate them into different columns as ES
>> always does?
>>
>> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
>> 1.5.0.1.
>>
>> Any clue shall be appreciated.
>>
>>
>>
>
>

Mime
View raw message