flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shushuai zhu <ss...@yahoo.com>
Subject How to use Flume Client perl API to send data to ElasticSearch?
Date Thu, 13 Jun 2013 03:47:20 GMT
I am sending data in JSON format using Flume Client perl API to Flume Avro source then ElasticSearchSink
to an ElasticSearch engine. I could send the data to ElasticSearch engine since I saw new
indexes created in the engine, but the data are not formatted correctly or partially lost
since they could not be viewed via Kibana.
An example of the data in JSON format is:
{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host name","@type":"flume-input","@fields":{"FIELD_PRIORITY":"DEBUG","FIELD_CLASS":"procedure.engine","FIELD_FILE":"/scratch/abc/abc.log","FIELD_MESSAGE":"The
debug message","FIELD_TIME":"1369785558004"}}
Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and the LogStash conf
file looks like:
input {
    redis {
        host => "abc"
        type => "redis-input"
        data_type => "list"
        key => "logstash"
        format => "json"
output {
    elasticsearch {
        embedded => false
        bind_host => "abc"
        host => "abc"
        port => 9300
The data were correctly input to ElasticSearch and could be viewed via Kibana. 
The sample perl API call for FlumeNG shown in Flume Client doc is: 
use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host name', port =>
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client => $ng_client);
 my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers => {},
body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n";    # response will be 'OK' on success
I tried putting the JSON data in "body" or "headers" or some other ways in the above example,
but could not get the right results in ElasticSearch. Could someone tell me how to use the
API to send the above example data to ElasticSearch?
View raw message