Hi Alex,

I got that working. I had to define the FLUME_CLASSPATH to  point to the HBase config.


From: Alex Baranau <alex.baranov.v@gmail.com>
To: flume-user@incubator.apache.org; Gaurav Khanna <khannapost@yahoo.com>
Sent: Monday, February 6, 2012 10:15 AM
Subject: Re: HBase and Flume

Hi,

Hm. Looks like there are too many open connections to the ZK from that host. How many flume nodes which write to Hbase did you start?

Also, make sure your ZK allows proper number of connections from one host (I remember there was some issue with having default less than expected(30) in one of ZK versions). You can add this to zoo.cfg (and restart ZK) manually  to be sure about how many is allowed:
maxClientCnxns=30

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

On Thu, Feb 2, 2012 at 2:59 PM, Gaurav Khanna <khannapost@yahoo.com> wrote:

Got the jar for the hbase plugin and updated the flume-site.xml with the plugin configuration. Now I can see the plugin and use it as well. But here is the problem:

This is on a client box that is not part of the hbase or hadoop cluster and am getting the following exception in the flume node. These are the env variables:
# HADOOP RELATED STUFF
export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
export HBASE_CONF_DIR=/app/hadoop/hbase-conf
export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
export ZOOKEEPER_CONF=$HBASE_CONF_DIR
export HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar

And this is the exception that I get:

012-02-02 13:52:08,721 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR connector.DirectDriver: Closing down due to exception on open calls
2012-02-02 13:52:08,722 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO connector.DirectDriver: Connector logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22 exited with error: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
        at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)


From: Gaurav Khanna <khannapost@yahoo.com>
To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>; Gaurav Khanna <khannapost@yahoo.com>
Sent: Thursday, February 2, 2012 10:57 AM
Subject: Re: HBase and Flume

Got the information at:

Thanks
Gaurav


From: Gaurav Khanna <khannapost@yahoo.com>
To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>
Sent: Thursday, February 2, 2012 10:54 AM
Subject: HBase and Flume

Hi,
Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink. 

Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")

and got the following in the master logs (now I know that I am missing some configuration steps - please can anybody point me to where I can get the Hbase related plugin and how to configure that?):

2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")]
2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during cmdid:0 config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
        at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
        at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
        at com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
        at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
        at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
        at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
        at com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)