flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Sargisson <esa...@pobox.com>
Subject Re: How to use Flume Client perl API to send data to ElasticSearch?
Date Tue, 18 Jun 2013 21:48:16 GMT
Hi Shushuai,
Your index naming isn't matching what Kibana is looking for.
Note that flume is writing to flume-2013-06-14 while logstash is writing to
The first part is a setting kin your Kibana config. I've never seen
logstash write with periods as delimiters so I'm not sure why that would

The other thing to be aware of is that the machine running
ElasticSearchSink needs to be in the UTC Timezone. There is a fixed defect
where ElasticSearchSink uses the local timezone to decide which index to
write to while Kibana always uses UTC.



Thanks for the reply. I tried using both epoch time and date for timestamp
but either of them makes the ElasticSearch to be viewed by Kibana. The perl
call looks like:

my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers
=> {"\@source" => "default", "\@timestamp" => 1369769253000,
"\@source_host" => "null", "\@source_path" => "default", "\@message" =>
"abc", "\@type" => "flume-input"}, body => "hello, this is sent from perl
(using FlumeNG)"}]);
print "$response\n";    # response will be 'OK' on success
I could always query the ElasticSearch with REST API. A returned example
result in JSON looks like:

_index: "flume-2013-06-14"
_type: "logs"
_id: "iug8xyVWRK6Qm7Z22ohCXw"
_score: 1
_source: {
body: "hello, this is sent from perl (using FlumeNG)"
@timestamp: "2013-05-28T19:27:49.947Z"
@message: "abc"
@source: "default"
@type: "flume-input"
@source_host: "null"
@source_path: "default"

A returned example result from the ElasticSearch built with Redis+LogStash
looks like:

_index: "logstash-2013.05.28"
_type: "redis-input"
_id: "I3AJHKzhQ1GXWc1Lb8WlFQ"
_score: 1
_source: {
@source: "default"
@tags: [0]
@fields: {
OBS_LOG_ENTRY_CONTENT: "2013-05-28 12:34:21,224
[#SYSTEM_POOL#:WorkManagerSvc] DEBUG workmanager.WorkManagerStats logp.251
- WeightedAvg.summarize[RunningThreads] : Avg : 1.0, min : 1, max : 1,
lastVal : 1, dur : 60014812861, totdur : 60014812861 "
OBS_FIELD_CLASS: "workmanager.WorkManagerStats"
OBS_LOG_FILE: "/scratch/work/system/log/abc.trc"
OBS_FIELD_MESSAGE: "WeightedAvg.summarize[RunningThreads] : Avg : 1.0, min
: 1, max : 1, lastVal : 1, dur : 60014812861, totdur : 60014812861 "
OBS_FIELD_METHOD: "logp.251"
OBS_FIELD_TIME: "1369769661224"
@timestamp: "2013-05-28T19:34:50.672Z"
@source_host: null
@source_path: "default"
@type: "redis-input"

Not sure what exactly in ElasticSearch causes the issue for Kibana. The
"body" name/value seems incompatible with others, but I could not control
it (it still shows with an empty string even if I removed it from the API

Another major issue I have with the perl API is, how could I add the
@fields data? When I have another level of JSON in "hearders", it became an
object string when querying with REST.


View raw message