flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Gheorghe <radu.gheor...@sematext.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 11:39:38 GMT
Hi Gonzalo,

Thanks for your reply. Unfortunately, I've already went that route with no
success. I can only find out one jar holding that, located in Morphlines:

radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> grep
-RHls javax.servlet.AsyncContext .
./target/lib/javax.servlet-3.0.0.v201112011016.jar

grepping in the Solr and Flume directories got me no result. Do you have
more ideas?

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherreros@gmail.com>
wrote:

> Usually that means you are loading different version of the servlet API.
> You need to do a bit of classpath troubleshooting to find which jars
> contain javax.servlet.AsyncContext and keep just the one included in
> Flume/lib
>
> Regards,
> Gonzalo
>
> On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheorghe@sematext.com>
> wrote:
>
>> Hello flume users!
>>
>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>> that with 1.5 but now I seem to fail :(
>>
>> I got this error in the log (full exception below):
>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>> information does not match signer information of other classes in the same
>> package
>>
>> Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
>> Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
>> classpath. Here's the full command line:
>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>> conf/flume-conf.properties
>>
>> Here's my Java version, if it matters:
>> java version "1.7.0_79"
>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>
>> From what I can understand out off the Internet (thanks, Internet!) this
>> may be caused by multiple jars having different versions of the same thing.
>> But I can only find it in Morphlines (via grep -RHls in all the
>> classpaths). Maybe I'm bumping into
>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>> verify this by compiling Kite 0.12.0 but it failed with:
>> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
>> (default) on project kite-data: Too many files with unapproved license: 2
>> See RAT report....
>>
>> So then I compiled only morphlines and that worked, only to bump into the
>> same error.
>>
>> As promised, below is the full exception. For completeness, I'm also
>> including the main config and the morphline config:
>>
>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  -
>> Unable to start SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>> counterGroup:{ name:null counters:{} } } - Exception follows.
>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>> information does not match signer information of other classes in the same
>> package
>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at
>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>> at
>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>> at
>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>> at
>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>> at
>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>> at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> > cat conf/flume-conf.properties
>> agent.sources = spoolDir
>> agent.channels = spillableMem
>> agent.sinks = solr
>>
>> agent.sources.spoolDir.type = spooldir
>> agent.sources.spoolDir.spoolDir = /opt/logs
>> agent.sources.spoolDir.batchSize = 1000
>> agent.sources.spoolDir.channels = spillableMem
>>
>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>> agent.channels.spillableMem.memoryCapacity = 100000
>> agent.channels.spillableMem.overflowCapacity = 1000000
>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>
>> agent.sinks.solr.type =
>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>> agent.sinks.solr.channel = spillableMem
>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>> agent.sinks.solr.morphlineId = 1
>> agent.sinks.solr.batchSize = 10000
>>
>> > cat conf/morphline.conf
>> SOLR_LOCATOR : {
>>   collection : gettingstarted
>>   solrUrl : "http://192.168.1.3:8983/solr/"
>>   batchSize : 5000
>> }
>>
>> morphlines : [
>> { id : 1
>>  commands : [
>>   { readLine { charset : UTF-8 } }
>>   {
>>     grok {
>>      dictionaryFiles : [conf/grok-patterns]
>>      expressions : {
>>        message : """%{COMBINEDAPACHELOG}"""
>>      }
>>     }
>>   }
>> #  { generateUUID {
>> #      field : id
>> #    }
>> #  }
>>   {
>>     loadSolr {
>>       solrLocator : ${SOLR_LOCATOR}
>>     }
>>   }
>>  ]
>> }
>> ]
>>
>> I also have a grok-patterns file there, but I don't think it's relevant.
>>
>> Any ideas are much appreciated!
>>
>> Thanks and best regards,
>> Radu
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>
>

Mime
View raw message