flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: set flume send logs to hdfs error
Date Thu, 14 May 2015 03:55:15 GMT
You need to use Flume’s client API to send data to the Avro Source. Use the thrift source,
and generate thrift client for Python using this IDL file: https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift
<https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift>

You can use that to send data to the thrift source at that point.

Thanks,
Hari Shreedharan




> On May 13, 2015, at 8:06 PM, 鹰 <980548079@qq.com> wrote:
> 
> I send data by python scripts use socket send the code like this:
> 
> import sys 
> from socket import *
> 
> 
> HOST = '192.168.1.117'                                                              
                                                                                         
                                                                
> PORT =44444
> BUFSIZ = 1024
> ADDR = (HOST, PORT)
> 
> tcpCliSock = socket(AF_INET, SOCK_STREAM)
> tcpCliSock.connect(ADDR)
> i=0
> for x in range(3):
>     print x, "xx"
>     n=tcpCliSock.send("test datas from flume")
> tcpCliSock.close()
> 
> 
> 
> ------------------ 原始邮件 ------------------
> 发件人: "Hari Shreedharan";<hshreedharan@cloudera.com>;
> 发送时间: 2015年5月14日(星期四) 上午10:53
> 收件人: "user@flume.apache.org"<user@flume.apache.org>;
> 主题: Re: set flume send logs to hdfs error
> 
> How are you sending data to the Avro Source?
> 
> 
> Thanks,
> Hari
> 
> On Wed, May 13, 2015 at 7:38 PM, 鹰 <980548079@qq.com <mailto:980548079@qq.com>>
wrote:
> hi all ,
>  i'm want set flume send data to hdfs my configure file is lile this :
> tier1.sources=source1  
> tier1.channels=channel1  
> tier1.sinks=sink1  
> 
> tier1.sources.source1.type=avro  
> tier1.sources.source1.bind=0.0.0.0  
> tier1.sources.source1.port=44444  
> tier1.sources.source1.channels=channel1  
> 
> tier1.channels.channel1.type=memory  
> tier1.channels.channel1.capacity=10000  
> tier1.channels.channel1.transactionCapacity=1000  
> tier1.channels.channel1.keep-alive=30  
> 
> tier1.sinks.sink1.type=hdfs  
> tier1.sinks.sink1.channel=channel1  
> tier1.sinks.sink1.hdfs.path=hdfs://hadoop-home.com:9000/user/hadoop/ <http://hadoop-home.com:9000/user/hadoop/>

> tier1.sinks.sink1.hdfs.fileType=DataStream  
> tier1.sinks.sink1.hdfs.writeFormat=Text  
> tier1.sinks.sink1.hdfs.rollInterval=0  
> tier1.sinks.sink1.hdfs.rollSize=10240  
> tier1.sinks.sink1.hdfs.rollCount=0  
> tier1.sinks.sink1.hdfs.idleTimeout=60  
> 
> when I start the flume by this configure file and send data to the port 44444 I get an
error :
> org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected:
154218761 items! Connection closed;
> dose anybody can help me ,thanks.
> 


Mime
View raw message