flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "鹰" <980548...@qq.com>
Subject 回复: set flume send logs to hdfs error
Date Fri, 15 May 2015 02:04:23 GMT
thanks Hari,
                    when i use avor client send data to the flume port I get an kerberos error
,it looks like this :
[ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:561)] Hadoop
running in secure mode, but Flume config doesn't specify a principal to use for Kerberos auth.
 [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:273)] Failed
to authenticate!
and then I add the hdfs.kerberosPrincipal to the configure file but it dosen't work , what
can I do now? 
thanks for any help .




------------------ 原始邮件 ------------------
发件人: "Hari Shreedharan";<hshreedharan@cloudera.com>;
发送时间: 2015年5月14日(星期四) 中午11:55
收件人: "user"<user@flume.apache.org>; 

主题: Re: set flume send logs to hdfs error



You need to use Flume’s client API to send data to the Avro Source. Use the thrift source,
and generate thrift client for Python using this IDL file: https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift

You can use that to send data to the thrift source at that point.
 
Thanks,
Hari Shreedharan





 
 
On May 13, 2015, at 8:06 PM, 鹰 <980548079@qq.com> wrote:

I send data by python scripts use socket send the code like this:

import sys 
from socket import *


HOST = '192.168.1.117'                                                                   
                                                                                         
                                                           
PORT =44444
BUFSIZ = 1024
ADDR = (HOST, PORT)

tcpCliSock = socket(AF_INET, SOCK_STREAM)
tcpCliSock.connect(ADDR)
i=0
for x in range(3):
    print x, "xx"
    n=tcpCliSock.send("test datas from flume")
tcpCliSock.close()





------------------ 原始邮件 ------------------
发件人: "Hari Shreedharan";<hshreedharan@cloudera.com>;
发送时间: 2015年5月14日(星期四) 上午10:53
收件人: "user@flume.apache.org"<user@flume.apache.org>; 

主题: Re: set flume send logs to hdfs error



How are you sending data to the Avro Source?


Thanks,
Hari



 
On Wed, May 13, 2015 at 7:38 PM, 鹰 <980548079@qq.com> wrote:
hi all ,
 i'm want set flume send data to hdfs my configure file is lile this :
tier1.sources=source1  
tier1.channels=channel1  
tier1.sinks=sink1  

tier1.sources.source1.type=avro  
tier1.sources.source1.bind=0.0.0.0  
tier1.sources.source1.port=44444  
tier1.sources.source1.channels=channel1  

tier1.channels.channel1.type=memory  
tier1.channels.channel1.capacity=10000  
tier1.channels.channel1.transactionCapacity=1000  
tier1.channels.channel1.keep-alive=30  

tier1.sinks.sink1.type=hdfs  
tier1.sinks.sink1.channel=channel1  
tier1.sinks.sink1.hdfs.path=hdfs://hadoop-home.com:9000/user/hadoop/ 
tier1.sinks.sink1.hdfs.fileType=DataStream  
tier1.sinks.sink1.hdfs.writeFormat=Text  
tier1.sinks.sink1.hdfs.rollInterval=0  
tier1.sinks.sink1.hdfs.rollSize=10240  
tier1.sinks.sink1.hdfs.rollCount=0  
tier1.sinks.sink1.hdfs.idleTimeout=60  

when I start the flume by this configure file and send data to the port 44444 I get an error
:
org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected:
154218761 items! Connection closed;
dose anybody can help me ,thanks.
Mime
View raw message