flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: flume hive sink not work
Date Thu, 22 Oct 2015 06:13:41 GMT
Are you able to telnet to bigdata1 on port 9083?
On Oct 21, 2015 7:27 PM, "lizhenmxcz@163.com" <lizhenmxcz@163.com> wrote:

>
> hi all:
> I use flume to import data from syslog to hive,but encount the follow
> errors.
>
>
> 2015-10-22 10:05:05,115 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:324)]
k2 : Failed connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb',
table='dns_request', partitionVals=[] }
>
> org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint
{metaStoreUri='thrift://bigdata1:9083', database='dnsdb', table='dns_request', partitionVals=[]
}
>         at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
>
>         at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
>
>         at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
>         at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
>
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting
to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', table='dns_request',
partitionVals=[] }
>
>         at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
>         at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
>         ... 6 more
> Caused by: java.util.concurrent.TimeoutException
>         at java.util.concurrent.FutureTask.get(FutureTask.java:201)
>
>         at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:431)
>
>         at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:373)
>         ... 7 more
>
>
> my configuration is:
>
>
> a1.sources = r1
> a1.channels = c1 c2
> a1.sinks = k1 k2
>
> a1.sources.r1.type = syslogudp
> a1.sources.r1.port = 514
> a1.sources.r1.host = 192.168.55.246
>
> a1.sources.r1.channels = c1 c2
> a1.sources.r1.interceptors = i1
> a1.sources.r1.interceptors.i1.type = regex_extractor
> a1.sources.r1.interceptors.i1.regex = Dns(.*)\\[
> a1.sources.r1.interceptors.i1.serializers = t1
> a1.sources.r1.interceptors.i1.serializers.t1.name = type
>
> a1.sources.r1.selector.type = multiplexing
> a1.sources.r1.selector.header = type
> a1.sources.r1.selector.mapping.Request = c1
> a1.sources.r1.selector.mapping.Answer = c2
>
> a1.sinks.k2.type = hive
> a1.sinks.k2.channel = c1
> a1.sinks.k2.hive.metastore = thrift://bigdata1:9083
> a1.sinks.k2.hive.database = dnsdb
> a1.sinks.k2.hive.table = dns_request
> a1.sinks.k2.hive.partiton = %Y,%m,%d,%H
> a1.sinks.k2.hive.txnsPerBatchAsk = 2
> a1.sinks.k2.batchSize = 10
> a1.sinks.k2.serializer = delimited
> a1.sinks.k2.serializer.delimiter = ,
> a1.sinks.k2.serializer.fieldnames = timepoint,random,sip,dip,spt,type,name
>
> a1.sinks.k1.type = hive
> a1.sinks.k1.channel = c2
> a1.sinks.k1.hive.metastore = thrift://bigdata1:9083
> a1.sinks.k1.hive.database = Dnsdb
> a1.sinks.k1.hive.table = dns_answer
> a1.sinks.k1.hive.partiton = %Y,%m,%d,%H
> a1.sinks.k1.hive.txnsPerBatchAsk = 2
> a1.sinks.k1.batchSize = 10
> a1.sinks.k1.serializer = delimited
> a1.sinks.k1.serializer.delimiter = ,
>
> a1.sinks.k1.serializer.fieldnames = timepoint,random,sip,dip,dpt,name,nosuchname,typemax,typecname,typeaddr,authservername,additionalrecords
>
> help me please,thanks.
>
> ------------------------------
> lizhenmxcz@163.com
>

Mime
View raw message