Additions to my earlier question:
1. It looks like I can customize which index to write into in the prepareIndexREquest(...) method - when implementing my custom event serializer.
I could probably do something like this (very pseudo-code-ish):
-- get customerId from the Event [actually, where is the org.apache.flume.Event class? I could not find it in the apache-flume-1.5.2-src.tar ...]
-- create String myIndexName = 'indexPrefix-from-config' + "_" + customerId
-- in [repareIndexRequest(....) do:
would this work?
2. However, I am not sure how and when this index is created.... ? I understand that by default a new index with the configured indexPrefix will be created every day - is it possible to customize this behavior?
3. And one more question:
I see that it is the IndexREquestBuilder class that is used for indexing, and not BulkRequestBuilder. Does it mean that Flume indexes one event at a time only? without any batching?
From: Marina <email@example.com>
To: "firstname.lastname@example.org" <email@example.com>
Sent: Wednesday, April 15, 2015 4:06 PM
Subject: Flume ElasticSearchSink - how to customize index name
I'm researching am option of populating ElasticSearch cluster from Flume for Apache-like logs. One of the requirements is to add log events to a specific index - the index name has to be determined based on a 'customerId'-like property available in the log message, that can be parsed out.
I see that I can potentially customize event parsing logic by writing a custom SErializer by implementing the ElasticSearchIndexRequestBuilderFactory interface.
Any suggestion on how I could also customize index selection?