flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Sargisson <esa...@pobox.com>
Subject How to use Flume Client perl API to send data to ElasticSearch?
Date Fri, 14 Jun 2013 15:53:19 GMT
Hi Shushuai,
The Flume event must have a header with the name timestamp (all lower case)
which is Unix milliseconds since 1970.

Without this, the event gets sent to elasticsearch but Kibana can never
read it.

The ElasticSearchSink converts the timestamp (millis) header into the
@timestamp for Kibana.



I am sending data in JSON format using Flume Client perl API to Flume Avro
source then ElasticSearchSink to an ElasticSearch engine. I could send the
data to ElasticSearch engine since I saw new indexes created in the engine,
but the data are not formatted correctly or partially lost since they could
not be viewed via Kibana.

An example of the data in JSON format is:

{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host
debug message","FIELD_TIME":"1369785558004"}}

Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and
the LogStash conf file looks like:

input {
    redis {
        host => "abc"
        type => "redis-input"
        data_type => "list"
        key => "logstash"
        format => "json"
output {
    elasticsearch {
        embedded => false
        bind_host => "abc"
        host => "abc"
        port => 9300

The data were correctly input to ElasticSearch and could be viewed via

The sample perl API call for FlumeNG shown in Flume Client doc is:

use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host
name', port => 41414);
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client =>
 my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers
=> {}, body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n";    # response will be 'OK' on success

I tried putting the JSON data in "body" or "headers" or some other ways in
the above example, but could not get the right results in ElasticSearch.
Could someone tell me how to use the API to send the above example data to


View raw message