flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mo Jia <life.130...@gmail.com>
Subject Re: write custom HTTPSource json handler problem
Date Tue, 12 May 2015 17:18:34 GMT
I find out the json the source get is like this :  JAONHandler::getEvents print

{"ts":"1970-01-01T00:00:00.000000Z","peer":"bro","src_name":"parent","level":"info","message":"raised
pipe\u0027s socket buffer size from 208K to
1024K"}{"ts":"1970-01-01T00:00:00.000000Z","peer":"bro","src_name":"parent","level":"info","message":"communication
started, parent pid is 8972, child pid is
8976"}{"ts":"2015-05-12T15:04:17.859868Z","peer":"bro","src_name":"child","level":"info","message":"listening
on 0.0.0.0:47760
(clear)"}{"ts":"2015-05-12T15:04:17.859868Z","peer":"bro","src_name":"child","level":"info","message":"listening
on [::]:47760 (clear)"}

something like {}{}{}{}

But what my custom client send is {} one by one , Is http handler
cached the source and handle it together?

So the  getEevents fun must handle jsons like {}{}{} , which type
should I send to `gson.fromJson(reader, type);`?



2015-05-12 3:26 GMT+08:00 Mo Jia <life.130815@gmail.com>:
> The offical handle like this:
>
>      List<Event> eventList = new ArrayList<Event>(0);
>       JSONEvent event=new JSONEvent();
>     try {
>         eventList = gson.fromJson(reader, listType);
>     } catch (JsonSyntaxException ex) {
>       throw new HTTPBadRequestException("Request has invalid JSON Syntax.", ex);
>     }
>
>   for (Event e : eventList) {
>      ((JSONEvent) e).setCharset(charset);
>     }
>
> Post must like this:
>  curl -X POST -d '[{  "headers" : { "timestamp" : "110434324343",
> "host" :"random_host.example.com", "field1" : "val1", "m_user" :
> "m_user", "m_year" : "m_year", "m_month" : "m_month", "m_day" :
> "m_day" },  "body" : "random_body"  }]' 127.0.0.1:8181
>
> For me , my post always is a single json, so I change the code to this:
>
>     List<Event> eventList = new ArrayList<Event>(0);
>       Type type = new TypeToken<JSONEvent>(){}.getType();
>       JSONEvent event=new JSONEvent();
>     try {
>         //eventList = gson.fromJson(reader, listType);
>         event = gson.fromJson(reader, type);
>     } catch (JsonSyntaxException ex) {
>         LOG.info("{}",reader);
>       throw new HTTPBadRequestException("Request has invalid JSON Syntax.", ex);
>     }
>
> //    for (Event e : eventList) {
> //      ((JSONEvent) e).setCharset(charset);
> //    }
>       ((JSONEvent) event).setCharset(charset);
>      eventList.add(((Event)event));
>
> When test like this :
>  curl -X POST -d '{"headers" : { "timestamp" : "140434324343", "host"
> :"random_host.example.com", "field1" : "val1", "m_user" : "m_user"}}'
> 127.0.0.1:8181
>
> It works fine.
>
> However , my c++ client using libcurl,
>
> My c++ client send POST like this :
> curl_easy_setopt(handle, CURLOPT_URL, bulk_url.c_str());
> curl_easy_setopt(handle, CURLOPT_POST, 1);
> curl_easy_setopt(handle, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)buffer.Len());
> curl_easy_setopt(handle, CURLOPT_POSTFIELDS, buffer.Bytes());
> curl_easy_setopt(handle, CURLOPT_HTTPHEADER, http_headers);
> curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION,
> &logging::writer::Flume::HTTPReceive); // This gets called with the
> result.
> curl_easy_setopt(handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
> curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1);
> curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT, transfer_timeout);
> curl_easy_setopt(handle, CURLOPT_TIMEOUT, transfer_timeout);
> curl_easy_setopt(handle, CURLOPT_DNS_CACHE_TIMEOUT, 60*60);
> CURLcode return_code = curl_easy_perform(handle);
>
> and get error like this:
>
> 15/05/12 03:17:02 INFO http.JSONHandler: org.mortbay.jetty.Request$1@43d06911
> 15/05/12 03:17:02 WARN http.HTTPSource: Received bad request from client.
> org.apache.flume.source.http.HTTPBadRequestException: Request has
> invalid JSON Syntax.
>
>
> from the error, it seem curl don't send a data in a single json.
>
>
> My question is :
> 1 how can I print http.JSONHandler:
> org.mortbay.jetty.Request$1@43d06911 to show what logs I get ?
> 2 This time I modify the core source , how can I write a simple custom
> file  and can use it in conf file like this:
>    a1.sources.r1.handler = org.xxx.xxx.my_custom_json
> 3 Is there a better way don't need write handler to handle my
> situation here?  [ A single json post]

Mime
View raw message