flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nickolay Kolev <lem0...@gmail.com>
Subject Re: How to separate json message into different columns when using ES sink?
Date Wed, 19 Nov 2014 08:41:00 GMT

It looks like ES sink is oriented to handle text payload and not json and
there are problems with serialization.
We have the same problem and I have developed patch and custom version
based on sources (two months ago)
I am not java developer so probably it is not the best solution but it
Now it is hosted on company's internal repo but if you have interest I can
upload it on GitHub or Bitbucket

best regards,
Nickolay Kolev

On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <shadyxu@gmail.com> wrote:

> Hi everyone,
> I am now using Flume to collect log into ElasticSearch. And the logs are
> in json format. However, when I check them in ES, it seems that Flume has
> put the entire json log in the @message attribute. Is there any config to
> do or need I do some coding to separate them into different columns as ES
> always does?
> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
> Any clue shall be appreciated.

View raw message