flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Kumar <delhiam...@gmail.com>
Subject Hive-Sink: org.apache.hive.hcatalog.streaming.HeartBeatFailure: Heart beat error
Date Wed, 08 Nov 2017 16:14:15 GMT
Hi,

I am using flume 1.7.0 on AWS EMR cluster with hive version 2.1.1. Flume is
configured for hive-sink to write data to hive metastore using hive
streaming. flume hive-sink commit transactions for few minutes but after
some time (10-15 minutes), transactions aborted and sink runner stopped. In
order to restart the hive sink runner again, I need to reload the flume.

2017-11-08 15:38:44,826 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:236)] Sending
heartbeat on batch TxnIds=[1491330
...1491339] on endPoint = {metaStoreUri='thrift://localhost:9083',
database='bsfc', table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[WARN -
org.apache.flume.sink.hive.HiveWriter.heartBeat(HiveWriter.java:244)]
Unable to send hea
rtbeat on Txn Batch TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }
org.apache.hive.hcatalog.streaming.HeartBeatFailure: Heart beat error.
InvalidTxns: []. AbortedTxns: [1491334, 1491335, 1491338, 1491339, 1491336,
1491337]
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.heartbeat(HiveEndPoint.java:953)
        at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:237)
        at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:233)
        at
org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO -
org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:337)]
Committing Txn 149
1334 on EndPoint: {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,833 (hive-sink3-call-runner-0) [ERROR -
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:756)]
Fatal
error on TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }; cause Unable t
o abort invalid transaction id : 1491334: No such transaction txnid:1491334
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort
invalid transaction id : 1491334: No such transaction txnid:1491334
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:894)
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:753)
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:853)
        at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:343)
        at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:340)
        at
org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491334)
        at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
        at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
        at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
        at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
        at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
        at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
        at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:923)
        ... 10 more


Attached is the flume error file

Thanks & Regards,

Amit Kumar,
Mob: 9910611621

Mime
View raw message