flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Gheorghe <radu.gheor...@sematext.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 13:21:22 GMT
Thanks, Gonzalo! That actually got me further. A Solr exception later (got
around it by removing 4.10 Solr libs which I assume don't work with my 5.3
for which I already have libs in CP), then I got another error pointing the
same sort of issue.

So I just went ahead and removed all duplicated jars (though many were din
different versions) from kite. With something as ugly as this:
for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
/opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
$lib*; fi; done

Then Flume complained about some codahale metrics class missing, so I moved
back the metrics-core library to kite. And now I get this:

java.lang.VerifyError: Cannot inherit from final class

I'm not sure if you need the whole stacktrace (I will paste it below) but I
have again the suspicion of duplicated stuff or mismatched versions.

I'm not sure I'm on the right path, so you were pointing me to different
ones:
- add the integration with Solr. Can you tell me which are the jars needed?
>From which subproject?
- look at the version of Flume distributed by Cloudera. I can't find just
Flume, I suppose I need to get the whole CDH or something like that?

Thanks again for all your help!

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gherreros@gmail.com>
wrote:

> That's it.
> Remove that jar from the Flume classpath so it doesn't interfere with the
> one in Flume (version 2.5)
>
> Note Flume already has the kite libraries. I would just add the
> integration with Solr (I suggest you have a look at the version of Flume
> distributed by Cloudera)
>
> Gonzalo
>
>
> On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheorghe@sematext.com>
> wrote:
>
>> Hi Gonzalo,
>>
>> Thanks for your reply. Unfortunately, I've already went that route with
>> no success. I can only find out one jar holding that, located in Morphlines:
>>
>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>> grep -RHls javax.servlet.AsyncContext .
>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>
>> grepping in the Solr and Flume directories got me no result. Do you have
>> more ideas?
>>
>> Best regards,
>> Radu
>>
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherreros@gmail.com>
>> wrote:
>>
>>> Usually that means you are loading different version of the servlet API.
>>> You need to do a bit of classpath troubleshooting to find which jars
>>> contain javax.servlet.AsyncContext and keep just the one included in
>>> Flume/lib
>>>
>>> Regards,
>>> Gonzalo
>>>
>>> On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheorghe@sematext.com
>>> > wrote:
>>>
>>>> Hello flume users!
>>>>
>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>>> that with 1.5 but now I seem to fail :(
>>>>
>>>> I got this error in the log (full exception below):
>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>> signer information does not match signer information of other classes in
>>>> the same package
>>>>
>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>>> Flume's classpath. Here's the full command line:
>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>> conf/flume-conf.properties
>>>>
>>>> Here's my Java version, if it matters:
>>>> java version "1.7.0_79"
>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>
>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>> this may be caused by multiple jars having different versions of the same
>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the
>>>> classpaths). Maybe I'm bumping into
>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>> [ERROR] Failed to execute goal
>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>> Too many files with unapproved license: 2 See RAT report....
>>>>
>>>> So then I compiled only morphlines and that worked, only to bump into
>>>> the same error.
>>>>
>>>> As promised, below is the full exception. For completeness, I'm also
>>>> including the main config and the morphline config:
>>>>
>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)
 -
>>>> Unable to start SinkRunner: {
>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>> signer information does not match signer information of other classes in
>>>> the same package
>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at
>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>> at
>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>> at
>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>> at
>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>> at
>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>> at
>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> > cat conf/flume-conf.properties
>>>> agent.sources = spoolDir
>>>> agent.channels = spillableMem
>>>> agent.sinks = solr
>>>>
>>>> agent.sources.spoolDir.type = spooldir
>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>> agent.sources.spoolDir.batchSize = 1000
>>>> agent.sources.spoolDir.channels = spillableMem
>>>>
>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>
>>>> agent.sinks.solr.type =
>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>> agent.sinks.solr.channel = spillableMem
>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>> agent.sinks.solr.morphlineId = 1
>>>> agent.sinks.solr.batchSize = 10000
>>>>
>>>> > cat conf/morphline.conf
>>>> SOLR_LOCATOR : {
>>>>   collection : gettingstarted
>>>>   solrUrl : "http://192.168.1.3:8983/solr/"
>>>>   batchSize : 5000
>>>> }
>>>>
>>>> morphlines : [
>>>> { id : 1
>>>>  commands : [
>>>>   { readLine { charset : UTF-8 } }
>>>>   {
>>>>     grok {
>>>>      dictionaryFiles : [conf/grok-patterns]
>>>>      expressions : {
>>>>        message : """%{COMBINEDAPACHELOG}"""
>>>>      }
>>>>     }
>>>>   }
>>>> #  { generateUUID {
>>>> #      field : id
>>>> #    }
>>>> #  }
>>>>   {
>>>>     loadSolr {
>>>>       solrLocator : ${SOLR_LOCATOR}
>>>>     }
>>>>   }
>>>>  ]
>>>> }
>>>> ]
>>>>
>>>> I also have a grok-patterns file there, but I don't think it's relevant.
>>>>
>>>> Any ideas are much appreciated!
>>>>
>>>> Thanks and best regards,
>>>> Radu
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>
>>>
>>
>

Mime
View raw message