flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com>
Subject Re: Flume ElasticSearchSink - how to customize index name
Date Thu, 16 Apr 2015 12:58:14 GMT
Additions to my earlier question:
1. It looks like I can customize which index to write into in the prepareIndexREquest(...)
method - when implementing my custom event serializer. I could probably do something like
this (very pseudo-code-ish):-- get customerId from the Event [actually, where is the org.apache.flume.Event
class? I could not find it in the  apache-flume-1.5.2-src.tar ...]-- create String myIndexName
= 'indexPrefix-from-config' + "_" + customerId-- in [repareIndexRequest(....) do:     
indexRequestBuilder.setIndex(myIndexName). ....would this work?
2. However, I am not sure how and when this index is created.... ? I understand that by default
a new index with the configured indexPrefix will be created every day - is it possible to
customize this behavior?
3. And one more question: I see that it is the IndexREquestBuilder class that is used for
indexing, and not BulkRequestBuilder. Does it mean that Flume indexes one event at a time
only? without any batching?
      From: Marina <ppine7@yahoo.com>
 To: "user@flume.apache.org" <user@flume.apache.org> 
 Sent: Wednesday, April 15, 2015 4:06 PM
 Subject: Flume ElasticSearchSink - how to customize index name
Hi,I'm researching am option of populating ElasticSearch cluster from Flume for Apache-like
logs. One of the requirements is to add log events to a specific index - the index name has
to be determined based on a 'customerId'-like property available in the log message, that
can be parsed out.I see that I can potentially customize event parsing logic by writing a
custom SErializer by implementing the ElasticSearchIndexRequestBuilderFactory interface.Any
suggestion on how I could also customize index selection?

View raw message