flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: flume problem
Date Wed, 02 Mar 2016 08:56:47 GMT
Could it be that you are serializing avro instead of json?

On 2 March 2016 at 08:25, Baris Akgun (Garanti Teknoloji) <
BarisAkgu@garanti.com.tr> wrote:

> Hi,
>
>
>
> When I send json data to flume with using http post, flume adds
> Co**ntent-Typeapplication/json** for each json post.
>
>
>
> In my http post java code,  I give the content-type with using
>
>
>
> **con.setRequestProperty("Content-Type", "application/json");** function.
>
>
>
>
>
> I am using blob handler.
>
>
>
> **In flume conf file**
>
>
>
> *tier1.sources.source1.type = org.apache.flume.source.http.HTTPSource
>
> tier1.sources.source1.handler =
> org.apache.flume.sink.solr.morphline.BlobHandler*
>
>
>
> In flume channel, flume adds content type for each post as you see. After
> HDFS sink, The content type word causes a problem when ı try to parse json
> with spark sql or hive serDe.
>
>
>
> **The flume channel log data**
>
>
>
> *^LContent-Typeapplication/jsonú{"id":"+ag:_ea_ch.++i++e_.c-
>
> ^LContentTypeapplication/json‘{"id":"tag:search.twitter.com
> ,2005:642913165047648*
>
>
>
> Is there any idea for that problem?
>
>
>
> Thank a lot.
>
>
>
> *Barış Akgün*
> Analitik Veri Ambarı ve Büyük Veri Yönetimi
> Uzman
>
> Tel
>
> :
>
> Dahili
>
> :
>
> Faks
>
> :
>
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>

Mime
View raw message