flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Subhramanian <deepak.subhraman...@gmail.com>
Subject Re: Flume ElasticSearch Sink not indexing JSON data
Date Fri, 11 Apr 2014 12:52:29 GMT
Is there any additional serializers in 1.5SNAPSHOT which can resolve the
JSON issue. I noticed that with the 1.5 SNAPSHOT it was not storing
timestamp field in elasticsearch. Is the default serializer changed from
Logstash or is it a bug. I was getting timestamp in 1.4 version .


On Fri, Apr 11, 2014 at 12:56 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:

>
>  I tried using the latest flume elastic search sink jar  build from
> 1.5SNAPSHOT ,but still no luck. I will try to see if it is an issue with
> elastic search api . When I loaded json data using hive it loaded JSON
> properly. But we have to pass a property es.input.json in hive.  Is there a
> way to pass a property in Flume.There is another user also posted the same
> issue. Any inputs ?
>
>
> On Wed, Apr 9, 2014 at 2:18 PM, Ashish <paliwalashish@gmail.com> wrote:
>
>> IMHO, checkout the trunk, build it and then just use the agent config
>> from old setup. Keep the old setup as is.
>>
>> thanks
>> ashish
>>
>>
>> On Wed, Apr 9, 2014 at 6:17 PM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> Thanks Otis. I will give it a try. Do I have to replace the flume jars
>>> or add the flume jar to the plugins directory to override the current flume
>>> version.
>>>
>>>
>>> On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <
>>> otis.gospodnetic@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The ES Sink has been modified quite a bit in the trunk.  Would it make
>>>> sense to give that a try?
>>>>
>>>> Otis
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>>
>>>> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
>>>> deepak.subhramanian@gmail.com> wrote:
>>>>
>>>>> I am posting JSON data to ElasticSearch using Flume . But for some
>>>>> reason it is not loading the correct data in the message. Instead of
JSON
>>>>> message it is storing message data as
>>>>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>>>>
>>>>> I tried using the DynamicSerializer which determines the content-type.
>>>>> But no luck.
>>>>>
>>>>> Here is the sample data posted on Elastic Search.
>>>>>
>>>>> {
>>>>>
>>>>>   "_index": "test_flume-2014-04-07",
>>>>>
>>>>>   "_type": "logs",
>>>>>
>>>>>   "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>>>>
>>>>>   "_score": null,
>>>>>
>>>>>   "_source": {
>>>>>
>>>>>     "@message": "
>>>>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>>>>
>>>>>     "@timestamp": "2014-04-07T09:49:26.490Z",
>>>>>
>>>>>     "@fields": {
>>>>>
>>>>>       "timestamp": "1396864166490"
>>>>>
>>>>>     }
>>>>>
>>>>>   },
>>>>>
>>>>>   "sort": [
>>>>>
>>>>>     1396864166490,
>>>>>
>>>>>     1396864166490
>>>>>
>>>>>   ]
>>>>>
>>>>> }
>>>>>
>>>>> org.elasticsearch.common.xcontent.XContentBuilder
>>>>> --
>>>>> Deepak Subhramanian
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak Subhramanian
>>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
>
> --
> Deepak Subhramanian
>



-- 
Deepak Subhramanian

Mime
View raw message