flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johny Rufus <jru...@cloudera.com>
Subject Re: set flume send logs to hdfs error
Date Fri, 15 May 2015 16:35:12 GMT
Are you running your Hadoop cluster in Kerberos mode ? If so, is your
kerberos principal/keytab combination  correct ? You can try to login into
the KDC server of your hadoop cluster independently using the
specified principal/keytab to make sure the combination can login to the
KDC/cluster and use the above combination in your flume config.

Thanks,
Rufus

On Thu, May 14, 2015 at 7:04 PM, 鹰 <980548079@qq.com> wrote:

>
> thanks Hari,
>                     when i use avor client send data to the flume port I
> get an kerberos error ,it looks like this :
> [ERROR -
> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:561)]
> Hadoop running in secure mode, but Flume config doesn't specify a principal
> to use for Kerberos auth.
>  [ERROR -
> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:273)]
> Failed to authenticate!
> and then I add the hdfs.kerberosPrincipal to the configure file but it
> dosen't work , what can I do now?
> thanks for any help .
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Hari Shreedharan";<hshreedharan@cloudera.com>;
> *发送时间:* 2015年5月14日(星期四) 中午11:55
> *收件人:* "user"<user@flume.apache.org>;
> *主题:* Re: set flume send logs to hdfs error
>
> You need to use Flume’s client API to send data to the Avro Source. Use
> the thrift source, and generate thrift client for Python using this IDL
> file:
> https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift
>
> You can use that to send data to the thrift source at that point.
>
> Thanks,
> Hari Shreedharan
>
>
>
>
> On May 13, 2015, at 8:06 PM, 鹰 <980548079@qq.com> wrote:
>
> I send data by python scripts use socket send the code like this:
>
> import sys
> from socket import *
>
>
> HOST =
> '192.168.1.117'
>
> PORT =44444
> BUFSIZ = 1024
> ADDR = (HOST, PORT)
>
> tcpCliSock = socket(AF_INET, SOCK_STREAM)
> tcpCliSock.connect(ADDR)
> i=0
> for x in range(3):
>     print x, "xx"
>     n=tcpCliSock.send("test datas from flume")
> tcpCliSock.close()
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Hari Shreedharan";<hshreedharan@cloudera.com>;
> *发送时间:* 2015年5月14日(星期四) 上午10:53
> *收件人:* "user@flume.apache.org"<user@flume.apache.org>;
> *主题:* Re: set flume send logs to hdfs error
>
> How are you sending data to the Avro Source?
>
>
> Thanks,
> Hari
>
> On Wed, May 13, 2015 at 7:38 PM, 鹰 <980548079@qq.com> wrote:
>
>> hi all ,
>>  i'm want set flume send data to hdfs my configure file is lile this :
>> tier1.sources=source1
>> tier1.channels=channel1
>> tier1.sinks=sink1
>>
>> tier1.sources.source1.type=avro
>> tier1.sources.source1.bind=0.0.0.0
>> tier1.sources.source1.port=44444
>> tier1.sources.source1.channels=channel1
>>
>> tier1.channels.channel1.type=memory
>> tier1.channels.channel1.capacity=10000
>> tier1.channels.channel1.transactionCapacity=1000
>> tier1.channels.channel1.keep-alive=30
>>
>> tier1.sinks.sink1.type=hdfs
>> tier1.sinks.sink1.channel=channel1
>> tier1.sinks.sink1.hdfs.path=hdfs://hadoop-home.com:9000/user/hadoop/
>> tier1.sinks.sink1.hdfs.fileType=DataStream
>> tier1.sinks.sink1.hdfs.writeFormat=Text
>> tier1.sinks.sink1.hdfs.rollInterval=0
>> tier1.sinks.sink1.hdfs.rollSize=10240
>> tier1.sinks.sink1.hdfs.rollCount=0
>> tier1.sinks.sink1.hdfs.idleTimeout=60
>>
>> when I start the flume by this configure file and send data to the port
>> 44444 I get an error :
>> org.apache.avro.AvroRuntimeException: Excessively large list allocation
>> request detected: 154218761 items! Connection closed;
>> dose anybody can help me ,thanks.
>>
>
>
>

Mime
View raw message