flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Gheorghe <radu.gheor...@sematext.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Fri, 18 Sep 2015 10:29:09 GMT
Thanks, Gonzalo, for all your replies. I'm pretty sure the solution is
within reach, I'm just running out of time. I did build flume from the repo
you indicated but again it fails to start with some missing dependencies. I
guess that indeed I should look at what are the required dependencies and
provide them and only them.

I will try that later on and come back if I have more issues.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Sep 17, 2015 at 4:28 PM, Gonzalo Herreros <gherreros@gmail.com>
wrote:

> Yes, it sounds like a dependency library version error.
> You can take their the Cloudera Flume binaries, build from their branch of
> Flume or just see how the manage the dependencies and do the same.
>
> https://github.com/cloudera/flume-ng  (look at their poms)
> You can download binaries here:
> https://repository.cloudera.com/cloudera/cloudera-repos/
>
> Gonzalo
>
>
> On 17 September 2015 at 14:21, Radu Gheorghe <radu.gheorghe@sematext.com>
> wrote:
>
>> Thanks, Gonzalo! That actually got me further. A Solr exception later
>> (got around it by removing 4.10 Solr libs which I assume don't work with my
>> 5.3 for which I already have libs in CP), then I got another error pointing
>> the same sort of issue.
>>
>> So I just went ahead and removed all duplicated jars (though many were
>> din different versions) from kite. With something as ugly as this:
>> for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
>> /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
>> $lib*; fi; done
>>
>> Then Flume complained about some codahale metrics class missing, so I
>> moved back the metrics-core library to kite. And now I get this:
>>
>> java.lang.VerifyError: Cannot inherit from final class
>>
>> I'm not sure if you need the whole stacktrace (I will paste it below) but
>> I have again the suspicion of duplicated stuff or mismatched versions.
>>
>> I'm not sure I'm on the right path, so you were pointing me to different
>> ones:
>> - add the integration with Solr. Can you tell me which are the jars
>> needed? From which subproject?
>> - look at the version of Flume distributed by Cloudera. I can't find just
>> Flume, I suppose I need to get the whole CDH or something like that?
>>
>> Thanks again for all your help!
>>
>> Best regards,
>> Radu
>>
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gherreros@gmail.com>
>> wrote:
>>
>>> That's it.
>>> Remove that jar from the Flume classpath so it doesn't interfere with
>>> the one in Flume (version 2.5)
>>>
>>> Note Flume already has the kite libraries. I would just add the
>>> integration with Solr (I suggest you have a look at the version of Flume
>>> distributed by Cloudera)
>>>
>>> Gonzalo
>>>
>>>
>>> On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheorghe@sematext.com
>>> > wrote:
>>>
>>>> Hi Gonzalo,
>>>>
>>>> Thanks for your reply. Unfortunately, I've already went that route with
>>>> no success. I can only find out one jar holding that, located in Morphlines:
>>>>
>>>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>>>> grep -RHls javax.servlet.AsyncContext .
>>>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>>>
>>>> grepping in the Solr and Flume directories got me no result. Do you
>>>> have more ideas?
>>>>
>>>> Best regards,
>>>> Radu
>>>>
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherreros@gmail.com>
>>>> wrote:
>>>>
>>>>> Usually that means you are loading different version of the servlet
>>>>> API.
>>>>> You need to do a bit of classpath troubleshooting to find which jars
>>>>> contain javax.servlet.AsyncContext and keep just the one included in
>>>>> Flume/lib
>>>>>
>>>>> Regards,
>>>>> Gonzalo
>>>>>
>>>>> On 17 September 2015 at 11:31, Radu Gheorghe <
>>>>> radu.gheorghe@sematext.com> wrote:
>>>>>
>>>>>> Hello flume users!
>>>>>>
>>>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink.
I
>>>>>> did that with 1.5 but now I seem to fail :(
>>>>>>
>>>>>> I got this error in the log (full exception below):
>>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>>> signer information does not match signer information of other classes
in
>>>>>> the same package
>>>>>>
>>>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars
to
>>>>>> Flume's classpath. Here's the full command line:
>>>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>>>> conf/flume-conf.properties
>>>>>>
>>>>>> Here's my Java version, if it matters:
>>>>>> java version "1.7.0_79"
>>>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>>>
>>>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>>>> this may be caused by multiple jars having different versions of
the same
>>>>>> thing. But I can only find it in Morphlines (via grep -RHls in all
the
>>>>>> classpaths). Maybe I'm bumping into
>>>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>>>> Too many files with unapproved license: 2 See RAT report....
>>>>>>
>>>>>> So then I compiled only morphlines and that worked, only to bump
into
>>>>>> the same error.
>>>>>>
>>>>>> As promised, below is the full exception. For completeness, I'm also
>>>>>> including the main config and the morphline config:
>>>>>>
>>>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)
 -
>>>>>> Unable to start SinkRunner: {
>>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>>> signer information does not match signer information of other classes
in
>>>>>> the same package
>>>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> at
>>>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>>>> at
>>>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>>>> at
>>>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>>>> at
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>>>> at
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>>>> at
>>>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>>>> at
>>>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> > cat conf/flume-conf.properties
>>>>>> agent.sources = spoolDir
>>>>>> agent.channels = spillableMem
>>>>>> agent.sinks = solr
>>>>>>
>>>>>> agent.sources.spoolDir.type = spooldir
>>>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>>>> agent.sources.spoolDir.batchSize = 1000
>>>>>> agent.sources.spoolDir.channels = spillableMem
>>>>>>
>>>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>>>
>>>>>> agent.sinks.solr.type =
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>>>> agent.sinks.solr.channel = spillableMem
>>>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>>>> agent.sinks.solr.morphlineId = 1
>>>>>> agent.sinks.solr.batchSize = 10000
>>>>>>
>>>>>> > cat conf/morphline.conf
>>>>>> SOLR_LOCATOR : {
>>>>>>   collection : gettingstarted
>>>>>>   solrUrl : "http://192.168.1.3:8983/solr/"
>>>>>>   batchSize : 5000
>>>>>> }
>>>>>>
>>>>>> morphlines : [
>>>>>> { id : 1
>>>>>>  commands : [
>>>>>>   { readLine { charset : UTF-8 } }
>>>>>>   {
>>>>>>     grok {
>>>>>>      dictionaryFiles : [conf/grok-patterns]
>>>>>>      expressions : {
>>>>>>        message : """%{COMBINEDAPACHELOG}"""
>>>>>>      }
>>>>>>     }
>>>>>>   }
>>>>>> #  { generateUUID {
>>>>>> #      field : id
>>>>>> #    }
>>>>>> #  }
>>>>>>   {
>>>>>>     loadSolr {
>>>>>>       solrLocator : ${SOLR_LOCATOR}
>>>>>>     }
>>>>>>   }
>>>>>>  ]
>>>>>> }
>>>>>> ]
>>>>>>
>>>>>> I also have a grok-patterns file there, but I don't think it's
>>>>>> relevant.
>>>>>>
>>>>>> Any ideas are much appreciated!
>>>>>>
>>>>>> Thanks and best regards,
>>>>>> Radu
>>>>>> --
>>>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message