flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gumnaam Sur <gumnaam....@gmail.com>
Subject Re: Installation of full flume on window OS
Date Mon, 30 Jul 2012 16:19:01 GMT
Hi Mardan,
Please try with the attached, flume-ng script,
I've modified it to be able to run under cygwin.

Put the file under where-ever you've extracted flume dist, under bin
directory.

I've tested it , and it works, if you can confirm, then I can submit a
patch to flume devs.

cheers

On Mon, Jul 30, 2012 at 11:48 AM, mardan Khan <mardan8310@gmail.com> wrote:

> yeah, you right.
>
> After successfully build the project through mvn package -DskipTest.
>
> I have got two directories in flume source: flume-ng-dist/target/
> apache-flume-1.2.0-dist.tar and apache-flume-1.2.0-src.tar.
>
> I have copied the apache-flume-1.2.0-dist.tar and unpacked into
> d:\apache-flume.
>
> Thanks
>
>
>
> On Mon, Jul 30, 2012 at 3:58 PM, alo alt <wget.null@gmail.com> wrote:
>
>> Ah, okay.
>>
>> For non-cygwin:
>>
>> You have done git-checkout and build flume successfully (mvn -DSkipTests)?
>>
>> After you've build you've copied the binaries, located in flume's source
>> directory (flume-ng-dist\target\flume-ng-dist-1.3.0-SNAPSHOT-dist.tar.gz)
>> onto another directory and unpacked them? And that directory is
>> apache-flume?
>>
>> cheers,
>>  Alex
>>
>>
>>
>>   mardan Khan <mardan8310@gmail.com>
>>  July 30, 2012 4:44 PM
>> HI,
>>
>> Actually i am doing conversation with Inder regarding the Cygwin. As I am
>> trying to run on cygwin and with you on MS-DOS command prompt.
>>
>> I am just telling you important location
>>
>>
>> 1). My flume installation directory: D:\apache-flume
>> 2). Lib path:                                 D:\apache-flume\lib
>> 3). Configuration File:
>> D:\apache-flume\conf\flume-conf.properties.template (I have copied the
>> configuration file from your previous post).
>> 4). My Java path:                          D:\program
>> files\java\jdk1.7.0_03
>>
>> My Command:
>>
>> D:\apache-flume>"c:\Program Files\Java\jdk1.7.0_03\bin\java.exe" -Xmx20m
>> -Dlog4j
>>
>> .configuration=file:///%CD%\conf\log4j.properties -cp
>> "d:\apache-flume\lib\*" org.apache.flume.node.Application
>>
>> -f d:\apache-flume\conf\flume-conf.properties.template -n syslog-agent
>>
>>
>>
>> Error: Could not find or load main class
>> d:\apache-flume\lib\asynchbase-1.2.0.jar
>>
>>
>> Now please let me know where I am doing mistake.
>>
>>
>>
>>
>>   alo alt <wget.null@gmail.com>
>>  July 30, 2012 4:20 PM
>>  I think we have two different conversations about the same topic. I
>> reply here.
>>
>> Thats not an issue of a env script in windows, because we haven't one so
>> far. You may need to cp (Classpath) the flume - libs as Inder says.
>> As I wrote in my howto:
>>
>> As example, starting from inside of your flume-installation (the
>> directory where you unzip'ed the build), lets assume D:\apache-flume:
>> d:\apache flume > "PATH\to\java.exe" -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties
>> -cp "d:\apache-flume\flume-1.3.0-SNAPSHOT\lib\*"
>> org.apache.flume.node.Application -f d:\flume\flume-1.3.0-SNAPSHOT\conf\YOUR_CONFIG.conf
>> -n agent
>>
>> Please be sure, that the lib-directory from flume is in place as well in
>> the CLASSPATH (cp).
>>
>> - Alex
>>   Inder Pall <inder.pall@gmail.com>
>>  July 30, 2012 4:02 PM
>> That's the problem. I am not well versed with this script to fix on WIN
>> but i guess as a workaround you might want to add $FLUME_INSTALL_DIR/lib/.
>> this lib has all the required jars and should get you moving forward.
>>
>> Mind, filing a JIRA  for the same.
>>
>> - inder
>>
>>
>>
>>
>> --
>> Thanks,
>> - Inder
>>   Tech Platforms @Inmobi
>>   Linkedin - http://goo.gl/eR4Ub
>>    mardan Khan <mardan8310@gmail.com>
>>  July 30, 2012 3:53 PM
>> Thanks Inder,
>>
>> when I am running the $echo $CLASSPATH, I am just getting . (dot) and
>> nothing more.
>>
>> Please see below the flume-ng files. Can you just pointout where I should
>> put echo$CLASSPATH.
>>
>> Thanks
>>
>>
>> CLEAN_FLAG=1
>> ################################
>> # functions
>> ################################
>>
>> info() {
>>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>>     local msg=$1
>>     echo "Info: $msg" >&2
>>   fi
>> }
>>
>> warn() {
>>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>>     local msg=$1
>>     echo "Warning: $msg" >&2
>>   fi
>> }
>>
>> error() {
>>   local msg=$1
>>   local exit_code=$2
>>
>>   echo "Error: $msg" >&2
>>
>>   if [ -n "$exit_code" ] ; then
>>     exit $exit_code
>>   fi
>> }
>>
>> # If avail, add Hadoop paths to the FLUME_CLASSPATH and to the
>> # FLUME_JAVA_LIBRARY_PATH env vars.
>> # Requires Flume jars to already be on FLUME_CLASSPATH.
>> add_hadoop_paths() {
>>   local
>> HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" \
>>       which hadoop 2>/dev/null)
>>
>>   if [ -f "${HADOOP_IN_PATH}" ]; then
>>     info "Including Hadoop libraries found via ($HADOOP_IN_PATH) for HDFS
>> access"
>>
>>     # determine hadoop java.library.path and use that for flume
>>     local HADOOP_CLASSPATH=""
>>     local HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \
>>         ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \
>>         java.library.path 2>/dev/null)
>>
>>     # look for the line that has the desired property value
>>     # (considering extraneous output from some GC options that write to
>> stdout)
>>     # IFS = InternalFieldSeparator (set to recognize only newline char as
>> delimiter)
>>     IFS=$'\n'
>>     for line in $HADOOP_JAVA_LIBRARY_PATH; do
>>       if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
>>         HADOOP_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
>>         break
>>       fi
>>     done
>>     unset IFS
>>
>>     if [ -n "${HADOOP_JAVA_LIBRARY_PATH}" ]; then
>>
>> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH"
>>     fi
>>
>>     # determine hadoop classpath
>>     HADOOP_CLASSPATH=$($HADOOP_IN_PATH classpath 2>/dev/null)
>>
>>     # hack up and filter hadoop classpath
>>     local ELEMENTS=$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH})
>>     local ELEMENT
>>     for ELEMENT in $ELEMENTS; do
>>       local PIECE
>>       for PIECE in $(echo $ELEMENT); do
>>         if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
>>           info "Excluding $PIECE from classpath"
>>           continue
>>         else
>>           FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
>>         fi
>>       done
>>     done
>>
>>   fi
>> }
>> add_HBASE_paths() {
>>   local HBASE_IN_PATH=$(PATH="${HBASE_HOME}/bin:$PATH" \
>>       which hbase 2>/dev/null)
>>
>>   if [ -f "${HBASE_IN_PATH}" ]; then
>>     info "Including HBASE libraries found via ($HBASE_IN_PATH) for HBASE
>> access"
>>
>>     # determine HBASE java.library.path and use that for flume
>>     local HBASE_CLASSPATH=""
>>     local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
>>         ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
>>         java.library.path 2>/dev/null)
>>
>>     # look for the line that has the desired property value
>>     # (considering extraneous output from some GC options that write to
>> stdout)
>>     # IFS = InternalFieldSeparator (set to recognize only newline char as
>> delimiter)
>>     IFS=$'\n'
>>     for line in $HBASE_JAVA_LIBRARY_PATH; do
>>       if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
>>         HBASE_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
>>         break
>>       fi
>>     done
>>     unset IFS
>>
>>     if [ -n "${HBASE_JAVA_LIBRARY_PATH}" ]; then
>>
>> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HBASE_JAVA_LIBRARY_PATH"
>>     fi
>>
>>     # determine HBASE classpath
>>     HBASE_CLASSPATH=$($HBASE_IN_PATH classpath 2>/dev/null)
>>
>>     # hack up and filter HBASE classpath
>>     local ELEMENTS=$(sed -e 's/:/ /g' <<<${HBASE_CLASSPATH})
>>     local ELEMENT
>>     for ELEMENT in $ELEMENTS; do
>>       local PIECE
>>       for PIECE in $(echo $ELEMENT); do
>>         if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
>>           info "Excluding $PIECE from classpath"
>>           continue
>>         else
>>           FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
>>         fi
>>       done
>>     done
>>     FLUME_CLASSPATH="$FLUME_CLASSPATH:$HBASE_HOME/conf"
>>
>>   fi
>> }
>>
>> display_help() {
>>   cat <<EOF
>> Usage: $0 <command> [options]...
>>
>> commands:
>>   help                  display this help text
>>   agent                 run a Flume agent
>>   avro-client           run an avro Flume client
>>   version               show Flume version info
>>
>> global options:
>>   --conf,-c <conf>      use configs in <conf> directory
>>   --classpath,-C <cp>   append to the classpath
>>   --dryrun,-d           do not actually start Flume, just print the
>> command
>>   -Dproperty=value      sets a JDK system property value
>>
>> agent options:
>>   --conf-file,-f <file> specify a config file (required)
>>   --name,-n <name>      the name of this agent (required)
>>   --help,-h             display help text
>>
>> avro-client options:
>>   --host,-H <host>      hostname to which events will be sent (required)
>>   --port,-p <port>      port of the avro source (required)
>>   --filename,-F <file>  text file to stream to avro source [default: std
>> input]
>>   --headerFile,-R <file> headerFile containing headers as key/value pairs
>> on each new line
>>   --help,-h             display help text
>>
>> Note that if <conf> directory is specified, then it is always included
>> first
>> in the classpath.
>>
>> EOF
>> }
>>
>> run_flume() {
>>   local FLUME_APPLICATION_CLASS
>>
>>   if [ "$#" -gt 0 ]; then
>>     FLUME_APPLICATION_CLASS=$1
>>     shift
>>   else
>>     error "Must specify flume application class" 1
>>   fi
>>
>>   if [ ${CLEAN_FLAG} -ne 0 ]; then
>>     set -x
>>   fi
>>   $EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPATH" \
>>       -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH
>> "$FLUME_APPLICATION_CLASS" $*
>> }
>>
>> ################################
>> # main
>> ################################
>>
>> # set default params
>> FLUME_CLASSPATH=""
>> FLUME_JAVA_LIBRARY_PATH=""
>> JAVA_OPTS="-Xmx20m"
>>
>> opt_conf=""
>> opt_classpath=""
>> opt_java_props=""
>> opt_dryrun=""
>>
>> mode=$1
>> shift
>>
>> case "$mode" in
>>   help)
>>     display_help
>>     exit 0
>>     ;;
>>   agent)
>>     opt_agent=1
>>     ;;
>>   node)
>>     opt_agent=1
>>     warn "The \"node\" command is deprecated. Please use \"agent\"
>> instead."
>>     ;;
>>   avro-client)
>>     opt_avro_client=1
>>     ;;
>>   version)
>>    opt_version=1
>>    CLEAN_FLAG=0
>>    ;;
>>   *)
>>     error "Unknown or unspecified command '$mode'"
>>     echo
>>     display_help
>>     exit 1
>>     ;;
>> esac
>>
>> while [ -n "$*" ] ; do
>>   arg=$1
>>   shift
>>
>>   case "$arg" in
>>     --conf|-c)
>>       [ -n "$1" ] || error "Option --conf requires an argument" 1
>>       opt_conf=$1
>>       shift
>>       ;;
>>     --classpath|-C)
>>       [ -n "$1" ] || error "Option --classpath requires an argument" 1
>>       opt_classpath=$1
>>       shift
>>       ;;
>>     --dryrun|-d)
>>       opt_dryrun="1"
>>       ;;
>>     -D*)
>>       opt_java_props="$opt_java_props $arg"
>>       ;;
>>     *)
>>       args="$args $arg"
>>       ;;
>>   esac
>> done
>>
>> # make opt_conf absolute
>> if [[ -n "$opt_conf" && -d "$opt_conf" ]]; then
>>   opt_conf=$(cd $opt_conf; pwd)
>> fi
>>
>> # allow users to override the default env vars via conf/flume-env.sh
>> if [ -z "$opt_conf" ]; then
>>   warn "No configuration directory set! Use --conf <dir> to override."
>> elif [ -f "$opt_conf/flume-env.sh" ]; then
>>   info "Sourcing environment configuration script $opt_conf/flume-env.sh"
>>   source "$opt_conf/flume-env.sh"
>> fi
>>
>> # append command-line java options to stock or env script JAVA_OPTS
>> if [ -n "${opt_java_props}" ]; then
>>   JAVA_OPTS="${JAVA_OPTS} ${opt_java_props}"
>> fi
>>
>> # prepend command-line classpath to env script classpath
>> if [ -n "${opt_classpath}" ]; then
>>   if [ -n "${FLUME_CLASSPATH}" ]; then
>>     FLUME_CLASSPATH="${opt_classpath}:${FLUME_CLASSPATH}"
>>   else
>>     FLUME_CLASSPATH="${opt_classpath}"
>>   fi
>> fi
>>
>> if [ -z "${FLUME_HOME}" ]; then
>>   FLUME_HOME=$(cd $(dirname $0)/..; pwd)
>> fi
>>
>> # prepend $FLUME_HOME/lib jars to the specified classpath (if any)
>> if [ -n "${FLUME_CLASSPATH}" ] ; then
>>   FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH"
>> else
>>   FLUME_CLASSPATH="${FLUME_HOME}/lib/*"
>> fi
>>
>> # find java
>> if [ -z "${JAVA_HOME}" ] ; then
>>   warn "JAVA_HOME is not set!"
>>   # Try to use Bigtop to autodetect JAVA_HOME if it's available
>>   if [ -e /usr/libexec/bigtop-detect-javahome ] ; then
>>     . /usr/libexec/bigtop-detect-javahome
>>   elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ] ; then
>>     . /usr/lib/bigtop-utils/bigtop-detect-javahome
>>   fi
>>
>>   # Using java from path if bigtop is not installed or couldn't find it
>>   if [ -z "${JAVA_HOME}" ] ; then
>>     JAVA_DEFAULT=$(type -p java)
>>     [ -n "$JAVA_DEFAULT" ] || error "Unable to find java executable. Is
>> it in your PATH?" 1
>>     JAVA_HOME=$(cd $(dirname $JAVA_DEFAULT)/..; pwd)
>>   fi
>> fi
>>
>> # look for hadoop libs
>> add_hadoop_paths
>> add_HBASE_paths
>>
>> # prepend conf dir to classpath
>> if [ -n "$opt_conf" ]; then
>>   FLUME_CLASSPATH="$opt_conf:$FLUME_CLASSPATH"
>> fi
>>
>> # allow dryrun
>> EXEC="exec"
>> if [ -n "${opt_dryrun}" ]; then
>>   warn "Dryrun mode enabled (will not actually initiate startup)"
>>   EXEC="echo"
>> fi
>>
>> # finally, invoke the appropriate command
>> if [ -n "$opt_agent" ] ; then
>>   run_flume $FLUME_AGENT_CLASS $args
>> elif [ -n "$opt_avro_client" ] ; then
>>   run_flume $FLUME_AVRO_CLIENT_CLASS $args
>> elif [ -n "${opt_version}" ] ; then
>>   run_flume $FLUME_VERSION_CLASS $args
>> else
>>   error "This message should never appear" 1
>> fi
>>
>> exit 0
>>
>>
>>
>>
>>
>>
>>
>>
>>   Inder Pall <inder.pall@gmail.com>
>>  July 30, 2012 3:40 PM
>> Hey if you edit flume-ng script and just before java is invoked put an
>>
>> echo $CLASSPATH that should help in figuring what all dir's are there in
>> CLASSPATH.
>> I am sorry but i am a big fan of windows hence don't run that O.S. to try
>> it out.
>>
>> Thanks,
>> - inder
>>
>>
>>
>>
>> --
>> Thanks,
>> - Inder
>>   Tech Platforms @Inmobi
>>   Linkedin - http://goo.gl/eR4Ub
>>
>>
>

Mime
View raw message