flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Javi Roman <javiro...@redoop.org>
Subject Re: Flume/Avro schema manipulation
Date Sat, 30 Aug 2014 11:59:07 GMT
It could be useful for you this project [1] the side where the source
is converted to avro using a custom parser.

[1] https://github.com/danieltardon/flume-ng-kafka-avro-sink


On Fri, Aug 29, 2014 at 5:18 PM, Ed Judge <ejudgie@gmail.com> wrote:
> From initial reading of what it does, it look promising.  So, I could
> ultimately leverage something like the readLine and toAvro commands
>
> Thanks,
> Ed
>
>
> On Aug 29, 2014, at 8:25 AM, Gaspar Muñoz <gmunoz@stratio.com> wrote:
>
> Hi Ed,
>
> maybe after read logs from your file, you can apply
> org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
> interceptor and then apply to Avro morphline.
>
> Is that what you are looking for?
>
> Regards.
>
>
> 2014-08-29 13:43 GMT+02:00 Ed Judge <ejudgie@gmail.com>:
>>
>> I am looking for some good documentation that would explain how I can pull
>> logs from a file and apply a certain schema to that log file format.  I have
>> gone threw the exercise of sending a file using a Avro source/sink with the
>> flume_ng avro_client to a flume_ng agent but my understanding is that this
>> is just using a "default Avro schema”.  I want to be able build the schema a
>> certain way.  Assuming it is doable, is there anything out that documents
>> this process?
>>
>> Thanks,
>> Ed
>
>
>

Mime
View raw message