flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Json over netcat source
Date Sun, 13 Apr 2014 14:40:00 GMT
Have been able to reproduce the problem locally using the existing test
cases inside ES Sink. The problem does exist.

Did some initial investigation, the framework is able to detect the JSON
content and tries to add it as complex field.
timestamp is added only if present in header.

In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil

public static void addComplexField(XContentBuilder builder, String
      XContentType contentType, byte[] data) throws IOException {
    XContentParser parser = null;
    try {
      XContentBuilder tmp = jsonBuilder();
      parser = XContentFactory.xContent(contentType).createParser(data);
      builder.field(fieldName, tmp); <<<< This is where the we might have
an issue (real action is happening inside this method

Can someone familiar with this part look further into this? I shall debug
further as soon as I have free cycles.


On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:

>  Thanks Simon. I am also struggling with no luck. I tried using the latest
> flume elastic search sink jar  build from 1.5SNAPSHOT ,but still no luck. I
> will try to see if it is an issue with elastic search api . When I loaded
> json using hive it loaded JSON properly. But we have to pass a property
> es.input.json in hive.  Is there a way to pass the same in Flume.
> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = '...',
>               'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);


Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

View raw message