flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Percy <mpe...@cloudera.com>
Subject Re: Flume agent failure
Date Thu, 05 Jul 2012 18:41:22 GMT
Vijay, I agree with Alex & Mohammed, your Hadoop installation seems messed
up. Either that, or you have a copy & paste error in your output...

Also, apparently you are trying to run Hadoop 0.20.204? It would be much
better for you to either use a stable version of Hadoop, such as 1.0.3, or
use the Hadoop version from CDH 3 or 4. I think 0.20.204 may even be
missing hsync (I know it's there in 0.20.205), and if so then the Flume
HDFS Sink will not work.

At a minimum, ensure that your HADOOP_HOME is pointing to a location so
that you can execute "$HADOOP_HOME/bin/hadoop classpath" and it will output
some jars, etc. The output you showed below looks like it should be the
output of "which hadoop"... if it's not then the hadoop classpath command
is broken in your version of hadoop and since Flume depends on it working,
you are kind of out of luck without hacking up the start script or putting
jars and config files into the Flume lib directory.

Regards,
Mike

On Thu, Jul 5, 2012 at 5:08 AM, Mohammad Tariq <dontariq@gmail.com> wrote:

> If you are using CDH then it's always better to uninstall the previous
> version. If it is Apache Hadoop then it's just a matter of changing
> the directory.
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Jul 5, 2012 at 5:28 PM, vijay k <k.vijay52@gmail.com> wrote:
> > Hi,
> >
> > with out removing older version, shall i install hadoop 0.20.204 in
> another
> > folder?
> >
> > Thanks,
> > vijay
> >
> > On Wed, Jul 4, 2012 at 7:26 PM, alo alt <wget.null@gmail.com> wrote:
> >>
> >> Yes.
> >>
> >> Follow these link:
> >> http://hadoop.apache.org/common/docs/r0.20.204.0/
> >>
> >> to install a single node or cluster. Note, you need on the sink side the
> >> same hdfs version as you've in your cluster.
> >>
> >> - Alex
> >>
> >>
> >> On Jul 4, 2012, at 3:51 PM, vijay k wrote:
> >>
> >> > I have already downloaded the hadoop-0.20.204.0.tar.gz,
> >> >
> >> > shall i go ahead  with hadoop 0.20.204 ver installation. please
> confirm
> >> > it
> >> > is compatible with flume 1.2.0 version?
> >> >
> >> > Thanks,
> >> > Vijay
> >> >
> >> > On Wed, Jul 4, 2012 at 7:03 PM, alo alt <wget.null@gmail.com> wrote:
> >> >
> >> >> 1. Flume uses the libs from the classpath
> >> >> 2. remove the installed hadoop version (apt-get remove hadoop*)
> >> >> 3. http://hadoop.apache.org/common/docs/   => click at the version
> you
> >> >> want to install and follow the steps described there
> >> >>
> >> >> - Alex
> >> >>
> >> >> On Jul 4, 2012, at 2:36 PM, vijay k wrote:
> >> >>
> >> >>> Hi Alex,
> >> >>>
> >> >>> I have installed hadoop via tar xzf hadoop-0.20.2.zip in Ubuntu,
> but i
> >> >>> could not able to find the *hadoop-install* directory.
> >> >>> Please let me know the following query's
> >> >>>
> >> >>> 1. which latest HADOOP version is compatible with FLUME 1.2.0 .
> >> >>> 2. if need to remove my hadoop 0.20.0 version, how to remove the
> older
> >> >>> version.
> >> >>> 3. and provide the hadoop latest version installation steps.
> >> >>>
> >> >>> Thanks a lot for your comments.
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Wed, Jul 4, 2012 at 5:28 PM, alo alt <wget.null@gmail.com>
> wrote:
> >> >>>
> >> >>>> U're on Ubuntu / Debian?
> >> >>>>
> >> >>>> If yes, remove the hadoop installation via apt, you have an
really
> >> >>>> old
> >> >> one
> >> >>>> installed. Take a look into the /usr/local/hadoop/lib directory
(as
> >> >>>> you
> >> >>>> posted before):
> >> >>>>
> >> >>>> -rw-r--r-- 1 hduser hadoop   15010 2010-02-19 13:25 xmlenc-0.52.jar
> >> >>>> -rw-r--r-- 1 hduser hadoop    8601 2010-02-19 13:25
> >> >> slf4j-log4j12-1.4.3.jar
> >> >>>>
> >> >>>> and so on. From 2010 - pretty old.
> >> >>>>
> >> >>>> How do you installed hadoop? I think you've done already from
the
> >> >> tarball
> >> >>>> into /hadoop-install/hadoop - there should be a ../lib dir
too.
> >> >>>> Export
> >> >>>> these and try again.
> >> >>>>
> >> >>>> - Alex
> >> >>>>
> >> >>>>
> >> >>>> On Jul 4, 2012, at 1:50 PM, vijay k wrote:
> >> >>>>
> >> >>>>> Hi Alex,
> >> >>>>>
> >> >>>>> I have set the HADOOP_CLASSPATH=/usr/local/hadoop/lib only,
but
> >> >>>>> still
> >> >> am
> >> >>>>> unable to proceed further.
> >> >>>>> Please find the attached flume-ng script. Please guide
me if i did
> >> >>>> anything
> >> >>>>> wrong in the script
> >> >>>>>
> >> >>>>> Thanks a lot for your helping to proceed further.
> >> >>>>>
> >> >>>>>
> >> >>>>> On Wed, Jul 4, 2012 at 5:08 PM, alo alt <wget.null@gmail.com>
> wrote:
> >> >>>>>
> >> >>>>>> Hi Vijay,
> >> >>>>>>
> >> >>>>>> but you've set the path to:
> >> >>>>>>> hadoop classpath        -- /usr/local/hadoop/bin/hadoop
> >> >>>>>>
> >> >>>>>> should be: /usr/local/hadoop/lib
> >> >>>>>>
> >> >>>>>> Flume reads out the variable and look into that path
for all
> needed
> >> >>>>>> classes / jars. But in /usr/local/hadoop/bin/hadoop
flume found
> >> >> nothing.
> >> >>>>>>
> >> >>>>>> - Alex
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Jul 4, 2012, at 1:33 PM, vijay k wrote:
> >> >>>>>>
> >> >>>>>>> Hi Alex,
> >> >>>>>>>
> >> >>>>>>> Hadoop lib are installed following path
> >> >>>>>>>
> >> >>>>>>> root@md-trngpoc1
> >> >>>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/bin#
> >> >>>>>>> cd /usr/local/hadoop/lib
> >> >>>>>>> root@md-trngpoc1:/usr/local/hadoop/lib# ls -lrt
> >> >>>>>>> total 8428
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>> Alexander Alten-Lorenz
> >> >>>>>> http://mapredit.blogspot.com
> >> >>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >> >>>>>>
> >> >>>>>>
> >> >>>>> <flume-ng script.txt>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Alexander Alten-Lorenz
> >> >>>> http://mapredit.blogspot.com
> >> >>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >> >>>>
> >> >>>>
> >> >>
> >> >>
> >> >> --
> >> >> Alexander Alten-Lorenz
> >> >> http://mapredit.blogspot.com
> >> >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Alexander Alten-Lorenz
> >> http://mapredit.blogspot.com
> >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >>
> >
>

Mime
View raw message