flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 13:28:21 GMT
Yes, it sounds like a dependency library version error.
You can take their the Cloudera Flume binaries, build from their branch of
Flume or just see how the manage the dependencies and do the same.

https://github.com/cloudera/flume-ng  (look at their poms)
You can download binaries here:
https://repository.cloudera.com/cloudera/cloudera-repos/

Gonzalo


On 17 September 2015 at 14:21, Radu Gheorghe <radu.gheorghe@sematext.com>
wrote:

> Thanks, Gonzalo! That actually got me further. A Solr exception later (got
> around it by removing 4.10 Solr libs which I assume don't work with my 5.3
> for which I already have libs in CP), then I got another error pointing the
> same sort of issue.
>
> So I just went ahead and removed all duplicated jars (though many were din
> different versions) from kite. With something as ugly as this:
> for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
> /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
> $lib*; fi; done
>
> Then Flume complained about some codahale metrics class missing, so I
> moved back the metrics-core library to kite. And now I get this:
>
> java.lang.VerifyError: Cannot inherit from final class
>
> I'm not sure if you need the whole stacktrace (I will paste it below) but
> I have again the suspicion of duplicated stuff or mismatched versions.
>
> I'm not sure I'm on the right path, so you were pointing me to different
> ones:
> - add the integration with Solr. Can you tell me which are the jars
> needed? From which subproject?
> - look at the version of Flume distributed by Cloudera. I can't find just
> Flume, I suppose I need to get the whole CDH or something like that?
>
> Thanks again for all your help!
>
> Best regards,
> Radu
>
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gherreros@gmail.com>
> wrote:
>
>> That's it.
>> Remove that jar from the Flume classpath so it doesn't interfere with the
>> one in Flume (version 2.5)
>>
>> Note Flume already has the kite libraries. I would just add the
>> integration with Solr (I suggest you have a look at the version of Flume
>> distributed by Cloudera)
>>
>> Gonzalo
>>
>>
>> On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheorghe@sematext.com>
>> wrote:
>>
>>> Hi Gonzalo,
>>>
>>> Thanks for your reply. Unfortunately, I've already went that route with
>>> no success. I can only find out one jar holding that, located in Morphlines:
>>>
>>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>>> grep -RHls javax.servlet.AsyncContext .
>>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>>
>>> grepping in the Solr and Flume directories got me no result. Do you have
>>> more ideas?
>>>
>>> Best regards,
>>> Radu
>>>
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherreros@gmail.com>
>>> wrote:
>>>
>>>> Usually that means you are loading different version of the servlet API.
>>>> You need to do a bit of classpath troubleshooting to find which jars
>>>> contain javax.servlet.AsyncContext and keep just the one included in
>>>> Flume/lib
>>>>
>>>> Regards,
>>>> Gonzalo
>>>>
>>>> On 17 September 2015 at 11:31, Radu Gheorghe <
>>>> radu.gheorghe@sematext.com> wrote:
>>>>
>>>>> Hello flume users!
>>>>>
>>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>>>> that with 1.5 but now I seem to fail :(
>>>>>
>>>>> I got this error in the log (full exception below):
>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>> signer information does not match signer information of other classes
in
>>>>> the same package
>>>>>
>>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars
to
>>>>> Flume's classpath. Here's the full command line:
>>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>>> conf/flume-conf.properties
>>>>>
>>>>> Here's my Java version, if it matters:
>>>>> java version "1.7.0_79"
>>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>>
>>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>>> this may be caused by multiple jars having different versions of the
same
>>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the
>>>>> classpaths). Maybe I'm bumping into
>>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>>> Too many files with unapproved license: 2 See RAT report....
>>>>>
>>>>> So then I compiled only morphlines and that worked, only to bump into
>>>>> the same error.
>>>>>
>>>>> As promised, below is the full exception. For completeness, I'm also
>>>>> including the main config and the morphline config:
>>>>>
>>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)
 -
>>>>> Unable to start SinkRunner: {
>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>> signer information does not match signer information of other classes
in
>>>>> the same package
>>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> at
>>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>>> at
>>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>>> at
>>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>>> at
>>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>>> at
>>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>>> at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>>> at
>>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> > cat conf/flume-conf.properties
>>>>> agent.sources = spoolDir
>>>>> agent.channels = spillableMem
>>>>> agent.sinks = solr
>>>>>
>>>>> agent.sources.spoolDir.type = spooldir
>>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>>> agent.sources.spoolDir.batchSize = 1000
>>>>> agent.sources.spoolDir.channels = spillableMem
>>>>>
>>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>>
>>>>> agent.sinks.solr.type =
>>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>>> agent.sinks.solr.channel = spillableMem
>>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>>> agent.sinks.solr.morphlineId = 1
>>>>> agent.sinks.solr.batchSize = 10000
>>>>>
>>>>> > cat conf/morphline.conf
>>>>> SOLR_LOCATOR : {
>>>>>   collection : gettingstarted
>>>>>   solrUrl : "http://192.168.1.3:8983/solr/"
>>>>>   batchSize : 5000
>>>>> }
>>>>>
>>>>> morphlines : [
>>>>> { id : 1
>>>>>  commands : [
>>>>>   { readLine { charset : UTF-8 } }
>>>>>   {
>>>>>     grok {
>>>>>      dictionaryFiles : [conf/grok-patterns]
>>>>>      expressions : {
>>>>>        message : """%{COMBINEDAPACHELOG}"""
>>>>>      }
>>>>>     }
>>>>>   }
>>>>> #  { generateUUID {
>>>>> #      field : id
>>>>> #    }
>>>>> #  }
>>>>>   {
>>>>>     loadSolr {
>>>>>       solrLocator : ${SOLR_LOCATOR}
>>>>>     }
>>>>>   }
>>>>>  ]
>>>>> }
>>>>> ]
>>>>>
>>>>> I also have a grok-patterns file there, but I don't think it's
>>>>> relevant.
>>>>>
>>>>> Any ideas are much appreciated!
>>>>>
>>>>> Thanks and best regards,
>>>>> Radu
>>>>> --
>>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message