flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Khanna <khannap...@yahoo.com>
Subject Re: HBase and Flume
Date Mon, 06 Feb 2012 18:29:20 GMT
Hi Alex,

I got that working. I had to define the FLUME_CLASSPATHto  point to the HBase config.



________________________________
 From: Alex Baranau <alex.baranov.v@gmail.com>
To: flume-user@incubator.apache.org; Gaurav Khanna <khannapost@yahoo.com> 
Sent: Monday, February 6, 2012 10:15 AM
Subject: Re: HBase and Flume
 

Hi,

Hm. Looks like there are too many open connections to the ZK from that host. How many flume
nodes which write to Hbase did you start?

Also, make sure your ZK allows proper number of connections from one host (I remember there
was some issue with having default less than expected(30) in one of ZK versions). You can
add this to zoo.cfg (and restart ZK) manually  to be sure about how many is allowed:
maxClientCnxns=30

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase


On Thu, Feb 2, 2012 at 2:59 PM, Gaurav Khanna <khannapost@yahoo.com> wrote:


>
>Got the jar for the hbase plugin and updated the flume-site.xml with the plugin configuration.
Now I can see the plugin and use it as well. But here is the problem:
>
>
>This is on a client box that is not part of the hbase or hadoop cluster and am getting
the following exception in the flume node. These are the env variables:
># HADOOP RELATED STUFF 
>export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
>export HBASE_CONF_DIR=/app/hadoop/hbase-conf
>export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
>export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
>export ZOOKEEPER_CONF=$HBASE_CONF_DIR
>export
 HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar
>
>
>
>And this is the exception that I get:
>
>
>012-02-02 13:52:08,721 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR
connector.DirectDriver: Closing down due to exception on open calls
>2012-02-02 13:52:08,722 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO
connector.DirectDriver: Connector logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22
exited with error: HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration
as often as you can. See HTable's javadoc for more
 information.
>org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper
but the connection closes immediately. This could be a sign that the server has too many connections
(30 is the default). Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more
information.
>        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
>        at
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
>        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
>        at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
>        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
>Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase
>        at
 org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
>
>
>
>________________________________
> From: Gaurav Khanna <khannapost@yahoo.com>
>To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>; Gaurav
Khanna <khannapost@yahoo.com> 
>Sent: Thursday, February 2, 2012 10:57 AM
>Subject: Re: HBase and Flume
> 
>
>
>Got the information at:
>https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
>
>
>Thanks
>Gaurav
>
>
>
>________________________________
> From: Gaurav Khanna <khannapost@yahoo.com>
>To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org> 
>Sent: Thursday, February 2, 2012 10:54 AM
>Subject: HBase and Flume
> 
>
>Hi,
>Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink. 
>
>
>Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")
>
>
>and got the following in the master logs (now I know that I am missing some configuration
steps - please can anybody point me to where I can get the Hbase related plugin and how to
configure that?):
>
>
>2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config
[POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable",
"%{nanos}", "colfam1", "q1", "%{body}")]
>2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception
during cmdid:0 config
 [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable",
"%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid
sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
>java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid
sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
>        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
>        at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
>        at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
>        at
 com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
>        at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
>        at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
>        at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
>Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable",
"%{nanos}", "colfam1", "q1", "%{body}" )
>        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
>        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
>        at
 com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
>        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
>
>
>
>
>
>
Mime
View raw message