flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: Including Hbase dependencies
Date Mon, 08 Oct 2012 06:57:34 GMT
Thanks very much for the help & advice Hari

Respectfully,

-- 
Iain Wright

<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.


On Sun, Oct 7, 2012 at 11:42 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> s/org.apache.flume.sink.hbase.SimpleHBaseEventSerializer/org.apache.flume.sink.hbase.SimpleHbaseEventSerializer.
> I am sorry for not following the same naming convention.
>
> I also suggest using the AsyncHBaseSink. The AsyncHBaseSink provides way
> better performance. The only reason HBaseSink itself is not deprecated is
> that it uses the "official" HBase client API. But as of now, the
> AsyncHBaseSink is recommended over HBaseSink.
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
> On Sunday, October 7, 2012 at 11:15 PM, iain wright wrote:
>
> Thanks again Mike -
>
> HADOOP_HOME & HBASE_HOME in the env script did the trick.
> *
> Now we're getting stuck on the class for the serializer not being found,
> which i see present, I'm guessing its just another PATH im missing:*
> [ERROR -
> org.apache.flume.sink.hbase.HBaseSink.configure(HBaseSink.java:176)] Could
> not instantiate event serializer.
> java.lang.ClassNotFoundException:
> org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:186)
>     at org.apache.flume.sink.hbase.HBaseSink.configure(HBaseSink.java:170)
>     at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>     at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
>
> *sink exists:
> *
> $/app/flume/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase]$
> ls -lah
> total 58
> drwxrwxrwx  2 root  wheel    11B Oct  3 16:36 .
> drwxrwxrwx  3 root  wheel     3B Oct  3 16:36 ..
> -rwxrwxrwx  1 root  wheel    15k Oct  3 16:36 AsyncHBaseSink.java
> -rwxrwxrwx  1 root  wheel   2.6k Oct  3 16:36
> AsyncHbaseEventSerializer.java
> -rwxrwxrwx  1 root  wheel   9.1k Oct  3 16:36 HBaseSink.java
> -rwxrwxrwx  1 root  wheel   1.8k Oct  3 16:36
> HBaseSinkConfigurationConstants.java
> -rwxrwxrwx  1 root  wheel   2.2k Oct  3 16:36 HbaseEventSerializer.java
> -rwxrwxrwx  1 root  wheel   5.9k Oct  3 16:36
> RegexHbaseEventSerializer.java
> -rwxrwxrwx  1 root  wheel   4.8k Oct  3 16:36
> SimpleAsyncHbaseEventSerializer.java
> -rwxrwxrwx  1 root  wheel   4.6k Oct  3 16:36
> SimpleHbaseEventSerializer.java
> -rwxrwxrwx  1 root  wheel   1.8k Oct  3 16:36 SimpleRowKeyGenerator.java
>
> *
> Startup used:*
>  $/app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf
> -f conf/brian.properties  -Dflume.root.logger=INFO,console
>
> *Relevant config:*
>  $cat conf/brian.properties
> #example.conf: A single-node Flume configuration
>
> # Name the components on this agent
> agent1.sources = source1
> agent1.sinks = sink1
> agent1.channels = channel1
>
> # Describe/configure source1
> agent1.sources.source1.type = exec
> agent1.sources.source1.command = tail -F /log.log
> agent1.sources.source1.batchSize = 1
> # Describe sink1
> #agent1.sinks.sink1.type = logger
> agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> agent1.sinks.sink1.table = brian_test
> agent1.sinks.sink1.columnFamily = f1
> agent1.sinks.sink1.serializer =
> org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
>
>
> # Use a channel which buffers events in memory
> agent1.channels.channel1.type = memory
> agent1.channels.channel1.capacity = 1000
> agent1.channels.channel1.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.source1.channels = channel1
> agent1.sinks.sink1.channel = channel1
> *
> flume-env.sh*
>
> $ cat flume-env.sh
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements.  See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership.  The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License.  You may obtain a copy of the License at
> #
> #     http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be
> sourced
> # during Flume startup.
>
> # Enviroment variables can be set here.
>
> #JAVA_HOME=/usr/lib/jvm/java-6-sun
>
> # Give Flume more memory and pre-allocate, enable remote monitoring via JMX
> #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
>
> # Note that the Flume conf directory is always included in the classpath.
> #FLUME_CLASSPATH=""
>
> HBASE_HOME="/app/hbase"
>
> HADOOP_HOME="/app/hadoop"
>
> I've tried copying the sink from /flume-ng-sinks/blah/blah/hbase-blah.sink
> to /app/flume/lib/ and setting my CLASSPATH & FLUME_CLASSPATH to
> /app/flume/lib but same result.
>
> Appreciate any help
>
> Cheers,
>
> --
> Iain Wright
>
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
> On Sat, Oct 6, 2012 at 9:34 PM, Mike Percy <mpercy@apache.org> wrote:
>
> Iain, you have to set HBASE_HOME in flume-env.sh (which is in the Flume
> conf directory) if Flume can't find the HBase classes.
>
> Regards,
> Mike
>
>
> On Fri, Oct 5, 2012 at 11:51 AM, iain wright <iainwrig@gmail.com> wrote:
>
> Hi guys,
>
>
> May someone please advise on the best way to include the Hbase depencies
> for a brand new install of flume, 1.3.0-SNAPSHOT? We are using HBase
> 0.94.0, r1332822.
>
>
> 2012-10-05 11:38:00,084 (conf-file-poller-0) [ERROR -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:207)]
> Failed to start agent because dependencies were not found in classpath.
> Error follows.
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
>
>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>
>         at java.lang.Class.privateGetDeclaredConstructors(Class.java:2404)
>
>         at java.lang.Class.getConstructor0(Class.java:2714)
>
>         at java.lang.Class.newInstance0(Class.java:343)
>
>         at java.lang.Class.newInstance(Class.java:325)
>
>         at
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:103)
>
>         at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:371)
>
>         at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
>
>         at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>
>         at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>
>         at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
>         at java.lang.Thread.run(Thread.java:722)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.conf.Configuration
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>         ... 19 more
>
> Thank you,
>
> --
> Iain Wright
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
>
>

Mime
View raw message