flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Json over netcat source
Date Sun, 13 Apr 2014 15:56:30 GMT
little more on the issue

builder.field(fieldName, tmp); calls the XContentBuilder API where class
type is determined and appropriate method is called. Since tmp, which is
instance of XContentBuilder, doesn't match any of the defined if conditions
it goes to final else where the tmp.toString() is called, and field(String,
String) method is called so we get object address in index.

Replacing
builder.field(fieldName, tmp);
with
builder.field(fieldName, tmp.string());

shall make things work, but I am not sure if this would be the best way to
use the API.

Got the answer from ES user list :)
http://elasticsearch-users.115913.n3.nabble.com/Issue-with-posting-json-data-to-elastic-search-via-Flume-td4054017.html

Can ES experts comment on the best way forward?



On Sun, Apr 13, 2014 at 8:10 PM, Ashish <paliwalashish@gmail.com> wrote:

> Have been able to reproduce the problem locally using the existing test
> cases inside ES Sink. The problem does exist.
>
> Did some initial investigation, the framework is able to detect the JSON
> content and tries to add it as complex field.
> timestamp is added only if present in header.
>
> In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
>
> public static void addComplexField(XContentBuilder builder, String
> fieldName,
>       XContentType contentType, byte[] data) throws IOException {
>     XContentParser parser = null;
>     try {
>       XContentBuilder tmp = jsonBuilder();
>       parser = XContentFactory.xContent(contentType).createParser(data);
>       parser.nextToken();
>       tmp.copyCurrentStructure(parser);
>       builder.field(fieldName, tmp); <<<< This is where the we might have
> an issue (real action is happening inside this method
>                                       call)
>
> Can someone familiar with this part look further into this? I shall debug
> further as soon as I have free cycles.
>
> thanks
> ashish
>
>
>
> On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>>  Thanks Simon. I am also struggling with no luck. I tried using the
>> latest flume elastic search sink jar  build from 1.5SNAPSHOT ,but still no
>> luck. I will try to see if it is an issue with elastic search api . When I
>> loaded json using hive it loaded JSON properly. But we have to pass a
>> property es.input.json in hive.  Is there a way to pass the same in Flume.
>>
>> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>>
>>
>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>> TBLPROPERTIES('es.resource' = '...',
>>
>>
>>               'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message