flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mo Jia <life.130...@gmail.com>
Subject write custom HTTPSource json handler problem
Date Mon, 11 May 2015 19:26:49 GMT
The offical handle like this:

     List<Event> eventList = new ArrayList<Event>(0);
      JSONEvent event=new JSONEvent();
    try {
        eventList = gson.fromJson(reader, listType);
    } catch (JsonSyntaxException ex) {
      throw new HTTPBadRequestException("Request has invalid JSON Syntax.", ex);

  for (Event e : eventList) {
     ((JSONEvent) e).setCharset(charset);

Post must like this:
 curl -X POST -d '[{  "headers" : { "timestamp" : "110434324343",
"host" :"random_host.example.com", "field1" : "val1", "m_user" :
"m_user", "m_year" : "m_year", "m_month" : "m_month", "m_day" :
"m_day" },  "body" : "random_body"  }]'

For me , my post always is a single json, so I change the code to this:

    List<Event> eventList = new ArrayList<Event>(0);
      Type type = new TypeToken<JSONEvent>(){}.getType();
      JSONEvent event=new JSONEvent();
    try {
        //eventList = gson.fromJson(reader, listType);
        event = gson.fromJson(reader, type);
    } catch (JsonSyntaxException ex) {
      throw new HTTPBadRequestException("Request has invalid JSON Syntax.", ex);

//    for (Event e : eventList) {
//      ((JSONEvent) e).setCharset(charset);
//    }
      ((JSONEvent) event).setCharset(charset);

When test like this :
 curl -X POST -d '{"headers" : { "timestamp" : "140434324343", "host"
:"random_host.example.com", "field1" : "val1", "m_user" : "m_user"}}'

It works fine.

However , my c++ client using libcurl,

My c++ client send POST like this :
curl_easy_setopt(handle, CURLOPT_URL, bulk_url.c_str());
curl_easy_setopt(handle, CURLOPT_POST, 1);
curl_easy_setopt(handle, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)buffer.Len());
curl_easy_setopt(handle, CURLOPT_POSTFIELDS, buffer.Bytes());
curl_easy_setopt(handle, CURLOPT_HTTPHEADER, http_headers);
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION,
&logging::writer::Flume::HTTPReceive); // This gets called with the
curl_easy_setopt(handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT, transfer_timeout);
curl_easy_setopt(handle, CURLOPT_TIMEOUT, transfer_timeout);
curl_easy_setopt(handle, CURLOPT_DNS_CACHE_TIMEOUT, 60*60);
CURLcode return_code = curl_easy_perform(handle);

and get error like this:

15/05/12 03:17:02 INFO http.JSONHandler: org.mortbay.jetty.Request$1@43d06911
15/05/12 03:17:02 WARN http.HTTPSource: Received bad request from client.
org.apache.flume.source.http.HTTPBadRequestException: Request has
invalid JSON Syntax.

from the error, it seem curl don't send a data in a single json.

My question is :
1 how can I print http.JSONHandler:
org.mortbay.jetty.Request$1@43d06911 to show what logs I get ?
2 This time I modify the core source , how can I write a simple custom
file  and can use it in conf file like this:
   a1.sources.r1.handler = org.xxx.xxx.my_custom_json
3 Is there a better way don't need write handler to handle my
situation here?  [ A single json post]

View raw message