From user-return-7155-apmail-flume-user-archive=flume.apache.org@flume.apache.org Wed Sep 30 09:39:31 2015 Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B615F176F4 for ; Wed, 30 Sep 2015 09:39:31 +0000 (UTC) Received: (qmail 60278 invoked by uid 500); 30 Sep 2015 09:39:26 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 60226 invoked by uid 500); 30 Sep 2015 09:39:26 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 60212 invoked by uid 99); 30 Sep 2015 09:39:26 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2015 09:39:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E386418099D for ; Wed, 30 Sep 2015 09:39:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id TPziQRCRxtm7 for ; Wed, 30 Sep 2015 09:39:21 +0000 (UTC) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 7A6EA42F38 for ; Wed, 30 Sep 2015 09:39:20 +0000 (UTC) Received: by wicfx3 with SMTP id fx3so188129188wic.1 for ; Wed, 30 Sep 2015 02:39:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=FvS1Ga/Ft1U51WpEPfThblFiaLAZpFOql2/u378VMZs=; b=V1keK0RV1xdZTEsV4I5LgiMQFbtq40zespP41VklOcf21DxQVXGQSSGKHBVOJLy0qw zSzfBaF0lHryLqpLovladyKm2a/Py5b7Kg+02OAaKb76x+e7DIya4UeTjqsWE1UjLTtz XF83EuvhrGbpJYytC6ADei4JTnijA5ZnHMrS8ai//2HNDnQT2zQdBnWtGtHvQbYk9TLP v3s+3QAqEBsVRv+Ypt7LKkD6GNTj0zA6+kzAiFxtzy0coso8AHnB1ffBC5nYxi4mH6h1 mPADGscpf7+YrLu8nFIn6q1aYD3hsPF4dtrBOWQjlAui66w7cX0A/Lf+ArT5utdffhQl F0pw== MIME-Version: 1.0 X-Received: by 10.180.106.196 with SMTP id gw4mr32076534wib.63.1443605959620; Wed, 30 Sep 2015 02:39:19 -0700 (PDT) Received: by 10.194.18.134 with HTTP; Wed, 30 Sep 2015 02:39:19 -0700 (PDT) In-Reply-To: <3AE89209CA2BC849B34BACBE09E7F9A01D44A7@DEERLM99EX5MSX.ww931.my-it-solutions.net> References: <3AE89209CA2BC849B34BACBE09E7F9A01D3F71@DEERLM99EX5MSX.ww931.my-it-solutions.net> <3AE89209CA2BC849B34BACBE09E7F9A01D44A7@DEERLM99EX5MSX.ww931.my-it-solutions.net> Date: Wed, 30 Sep 2015 10:39:19 +0100 Message-ID: Subject: Re: No columns to insert for #1 item From: Gonzalo Herreros To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e89a8f3baebf6b42070520f3b4fa --e89a8f3baebf6b42070520f3b4fa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The column family is a required field for the sink. You should have an error in the log explaining that (unless you are running a very old Flume version or something). You need something like this: agent.sinks.hbaseSink.columnFamily=3Dcf Regards, Gonzalo On 30 September 2015 at 09:21, Tinte garcia, Miguel Angel < miguel.tinte@atos.net> wrote: > Hi Gonzalo, > > Thanks for the answer. My flume agent configuration is the following: > > agent.sources=3DavroSource > > agent.channels=3Dchannel1 > > agent.sinks=3DhbaseSink > > > > agent.sources.avroSource.type=3Davro > > agent.sources.avroSource.channels=3Dchannel1 > > agent.sources.avroSource.bind=3D0.0.0.0 > > agent.sources.avroSource.port=3D61621 > > agent.sources.avroSource.interceptors=3Di1 > > agent.sources.avroSource.interceptors.i1.type=3Dtimestamp > > #agent.sources.avroSource.interceptors.i1.payloadColumn=3D > > > > agent.channels.channel1.type=3Dmemory > > agent.channels.channel1.capacity=3D1000000 > > agent.channels.channel1.transactionCapacity=3D100000 > > #agent.channels.channel1.byteCapacityBufferPercentage=3D20 > > agent.channels.channel1.byteCapacity=3D134217728 > > > > agent.sinks.hbaseSink.type=3Dhbase > > agent.sinks.hbaseSink.channel=3Dchannel1 > > agent.sinks.hbaseSink.channel.capacity=3D1000000 > > agent.sinks.hbaseSink.channel.transactionCapacity=3D100000 > > agent.sinks.hbaseSink.table=3DTest_Simulations_Data > > > > > > PayloadColumn is commented. Should I uncomment it and fill with any > specific value? > > > > Thanks > > > > *From:* Gonzalo Herreros [mailto:gherreros@gmail.com] > *Sent:* Wednesday, September 30, 2015 9:56 AM > *To:* user@flume.apache.org > *Subject:* Re: No columns to insert for #1 item > > > > My guess is that the HBase serializer is not filling the payload column > correctly. i.e. an empty value in the property "payloadColumn" > > Can you share the sink configuration? > > > > Regards, > > Gonzalo > > > > On 29 September 2015 at 23:40, Tinte garcia, Miguel Angel < > miguel.tinte@atos.net> wrote: > > Hi, > > I am trying write data into HBase sending it through Flume agents, but > some error appears in the during transaction: > > > > 2015-09-30 00:34:55,745 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:353= )] > Failed to commit transaction.Transaction rolled back. > > java.lang.IllegalArgumentException: No columns to insert for #1 item > > at > org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:4= 68) > > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati= on.processBatchCallback(HConnectionManager.java:2332) > > at > org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:842) > > at > org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:857) > > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:832) > > at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:386= ) > > at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374= ) > > at > org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427) > > at > org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:3= 74) > > at > org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344) > > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.j= ava:68) > > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > > > Does anybody know what it could be caused for? > > > > Thanks in advance > > > > This e-mail and the documents attached are confidential and intended > solely for the addressee; it may also be privileged. If you receive this > e-mail in error, please notify the sender immediately and destroy it. > As its integrity cannot be secured on the Internet, the Atos group > liability cannot be triggered for the message content. Although the sende= r > endeavors to maintain a computer virus-free network, the sender does not > warrant that this transmission is virus-free and will not be liable for a= ny > damages resulting from any virus transmitted. > > Este mensaje y los ficheros adjuntos pueden contener informaci=C3=B3n > confidencial destinada solamente a la(s) persona(s) mencionadas > anteriormente y pueden estar protegidos por secreto profesional. > Si usted recibe este correo electr=C3=B3nico por error, gracias por infor= mar > inmediatamente al remitente y destruir el mensaje. > Al no estar asegurada la integridad de este mensaje sobre la red, Atos no > se hace responsable por su contenido. Su contenido no constituye ning=C3= =BAn > compromiso para el grupo Atos, salvo ratificaci=C3=B3n escrita por ambas = partes. > Aunque se esfuerza al m=C3=A1ximo por mantener su red libre de virus, el = emisor > no puede garantizar nada al respecto y no ser=C3=A1 responsable de cuales= quiera > da=C3=B1os que puedan resultar de una transmisi=C3=B3n de virus. > > > This e-mail and the documents attached are confidential and intended > solely for the addressee; it may also be privileged. If you receive this > e-mail in error, please notify the sender immediately and destroy it. > As its integrity cannot be secured on the Internet, the Atos group > liability cannot be triggered for the message content. Although the sende= r > endeavors to maintain a computer virus-free network, the sender does not > warrant that this transmission is virus-free and will not be liable for a= ny > damages resulting from any virus transmitted. > > Este mensaje y los ficheros adjuntos pueden contener informaci=C3=B3n > confidencial destinada solamente a la(s) persona(s) mencionadas > anteriormente y pueden estar protegidos por secreto profesional. > Si usted recibe este correo electr=C3=B3nico por error, gracias por infor= mar > inmediatamente al remitente y destruir el mensaje. > Al no estar asegurada la integridad de este mensaje sobre la red, Atos no > se hace responsable por su contenido. Su contenido no constituye ning=C3= =BAn > compromiso para el grupo Atos, salvo ratificaci=C3=B3n escrita por ambas = partes. > Aunque se esfuerza al m=C3=A1ximo por mantener su red libre de virus, el = emisor > no puede garantizar nada al respecto y no ser=C3=A1 responsable de cuales= quiera > da=C3=B1os que puedan resultar de una transmisi=C3=B3n de virus. > --e89a8f3baebf6b42070520f3b4fa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The column family is a required field for the sink. You sh= ould have an error in the log explaining that (unless you are running a ver= y old Flume version or something).
You need something like this:
<= div>
agent.sinks.hbaseSink.columnFamily= =3Dcf

Regards,
Gonzalo

On 30 September 2015 at 09:= 21, Tinte garcia, Miguel Angel <miguel.tinte@atos.net> w= rote:

Hi Gonzalo,=

Thanks for the answer. My= flume agent configuration is the following:

agent.sources= =3DavroSource

agent.channel= s=3Dchannel1

agent.sinks= =3DhbaseSink

=C2=A0=

agent.sources= .avroSource.type=3Davro

agent.sources= .avroSource.channels=3Dchannel1

agent.sources= .avroSource.bind=3D0.0.0.0

agent.sources= .avroSource.port=3D61621

agent.sources= .avroSource.interceptors=3Di1

agent.sources= .avroSource.interceptors.i1.type=3Dtimestamp

#agent.source= s.avroSource.interceptors.i1.payloadColumn=3D

=C2=A0=

agent.channels.channel1.t= ype=3Dmemory

agent.channels.channel1.c= apacity=3D1000000

agent.channels.channel1.t= ransactionCapacity=3D100000

#agent.channels.channel1.= byteCapacityBufferPercentage=3D20

agent.channels.channel1.b= yteCapacity=3D134217728

=C2=A0

agent.sinks.hbaseSink.typ= e=3Dhbase

agent.sinks.hbaseSink.cha= nnel=3Dchannel1

agent.sinks.hbaseSink.cha= nnel.capacity=3D1000000

agent.sinks.hbaseSink.cha= nnel.transactionCapacity=3D100000

agent.sinks.hbaseSink.tab= le=3DTest_Simulations_Data

=C2=A0

=C2=A0

PayloadColumn is commente= d. Should I uncomment it and fill with any specific value?

=C2=A0

Thanks

=C2=A0

From: Gonzalo = Herreros [mailto:g= herreros@gmail.com]
Sent: Wednesday, September 30, 2015 9:56 AM
To: user@flume.apache.org
Subject: Re: No columns to insert for #1 item

=C2=A0

My guess is that the HBase serializer is not filling= the payload column correctly. i.e. an empty value in the property "pa= yloadColumn"

Can you share the sink configuration?<= /p>

=C2=A0

Regards,

Gonzalo

=C2=A0

On 29 September 2015 at 23:40, Tinte garcia, Miguel = Angel <miguel= .tinte@atos.net> wrote:

Hi,

I am trying write data into HBase sendi= ng it through Flume agents, but some error appears in the during transactio= n:

=C2=A0

2015-09-30 00:34:55,745 (SinkRunner-Pol= lingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hbase.HBase= Sink.process(HBaseSink.java:353)] Failed to commit transaction.Transaction rolled back.

java.lang.IllegalArgumentException: No = columns to insert for #1 item

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.j= ava:468)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpleme= ntation.processBatchCallback(HConnectionManager.java:2332)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:842)=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:857)=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:832)<= u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:386)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:4= 27)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.j= ava:374)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProces= sor.java:68)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

=C2=A0

Does anybody know what it could be caus= ed for?

=C2=A0

Thanks in advance<= /p>

=C2=A0

This e-mail and the documents attached are confident= ial and intended solely for the addressee; it may also be privileged. If yo= u receive this e-mail in error, please notify the sender immediately and de= stroy it.
As its integrity cannot be secured on the Internet, the Atos group liabilit= y cannot be triggered for the message content. Although the sender endeavor= s to maintain a computer virus-free network, the sender does not warrant th= at this transmission is virus-free and will not be liable for any damages resulting from any virus transmitte= d.

Este mensaje y los ficheros adjuntos pueden contener informaci=C3=B3n confi= dencial destinada solamente a la(s) persona(s) mencionadas anteriormente y = pueden estar protegidos por secreto profesional.
Si usted recibe este correo electr=C3=B3nico por error, gracias por informa= r inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no s= e hace responsable por su contenido. Su contenido no constituye ning=C3=BAn= compromiso para el grupo Atos, salvo ratificaci=C3=B3n escrita por ambas p= artes.
Aunque se esfuerza al m=C3=A1ximo por mantener su red libre de virus, el em= isor no puede garantizar nada al respecto y no ser=C3=A1 responsable de cua= lesquiera da=C3=B1os que puedan resultar de una transmisi=C3=B3n de virus.

=C2=A0

This e-mail and the documents attached are confidential and intended solely= for the addressee; it may also be privileged. If you receive this e-mail i= n error, please notify the sender immediately and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liabilit= y cannot be triggered for the message content. Although the sender endeavor= s to maintain a computer virus-free network, the sender does not warrant th= at this transmission is virus-free and will not be liable for any damages resulting from any virus transmitte= d.

Este mensaje y los ficheros adjuntos pueden contener informaci=C3=B3n confi= dencial destinada solamente a la(s) persona(s) mencionadas anteriormente=C2= =A0y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electr=C3=B3nico por error, gracias por informa= r inmediatamente al remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no s= e hace responsable por su contenido. Su contenido no constituye ning=C3=BAn= compromiso para el grupo Atos, salvo ratificaci=C3=B3n escrita por ambas p= artes.
Aunque se esfuerza al m=C3=A1ximo por mantener su red libre de virus, el em= isor no puede garantizar nada al respecto y no ser=C3=A1 responsable de cua= lesquiera da=C3=B1os que puedan resultar de una transmisi=C3=B3n de virus.

--e89a8f3baebf6b42070520f3b4fa--