flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Baris Akgun (Garanti Teknoloji)" <BarisA...@garanti.com.tr>
Subject RE: Flume MorphlineSolrSink
Date Wed, 18 May 2016 08:55:16 GMT
Hi


The version of solr in cdh is solr-4.10.3-cdh5.4.0

And the files in flume lib directory;

[gbbdap36]/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib$ls
activation-1.1.jar                                 hive-metastore.jar
ant-1.8.1.jar                                      hive-serde.jar
ant-launcher-1.8.1.jar                             hive-shims-0.23.jar
antlr-2.7.7.jar                                    hive-shims-common.jar
antlr-runtime-3.4.jar                              hive-shims.jar
aopalliance-1.0.jar                                hive-shims-scheduler.jar
apacheds-i18n-2.0.0-M15.jar                        htrace-core-3.0.4.jar
apacheds-kerberos-codec-2.0.0-M15.jar              httpclient-4.2.5.jar
apache-log4j-extras-1.1.jar                        httpcore-4.2.5.jar
api-asn1-api-1.0.0-M20.jar                         irclib-1.10.jar
api-util-1.0.0-M20.jar                             jackson-annotations-2.3.0.jar
asm-3.2.jar                                        jackson-core-2.3.1.jar
async-1.4.0.jar                                    jackson-core-asl-1.8.8.jar
asynchbase-1.5.0.jar                               jackson-databind-2.3.1.jar
avro-ipc.jar                                       jackson-jaxrs-1.8.8.jar
avro.jar                                           jackson-mapper-asl-1.8.8.jar
bonecp-0.7.1.RELEASE.jar                           jackson-xc-1.8.8.jar
calcite-avatica-1.0.0-incubating.jar               janino-2.7.6.jar
calcite-core-1.0.0-incubating.jar                  jasper-compiler-5.5.23.jar
calcite-linq4j-1.0.0-incubating.jar                jasper-runtime-5.5.23.jar
commons-beanutils-1.7.0.jar                        javax.inject-1.jar
commons-beanutils-core-1.8.0.jar                   jaxb-api-2.2.2.jar
commons-cli-1.2.jar                                jaxb-impl-2.2.3-1.jar
commons-codec-1.8.jar                              jdo-api-3.0.1.jar
commons-collections-3.2.1.jar                      jersey-client-1.9.jar
commons-compiler-2.7.6.jar                         jersey-core-1.8.jar
commons-compress-1.4.1.jar                         jersey-guice-1.9.jar
commons-configuration-1.6.jar                      jersey-json-1.9.jar
commons-dbcp-1.4.jar                               jersey-server-1.9.jar
commons-digester-1.8.jar                           jersey-servlet-1.14.jar
commons-el-1.0.jar                                 jets3t-0.6.1.jar
commons-httpclient-3.1.jar                         jettison-1.1.jar
commons-io-2.4.jar                                 jetty-6.1.26.cloudera.4.jar
commons-jexl-2.1.1.jar                             jetty-util-6.1.26.cloudera.4.jar
commons-lang-2.6.jar                               jline-2.11.jar
commons-logging-1.1.3.jar                          joda-time-2.1.jar
commons-math3-3.1.1.jar                            jopt-simple-3.2.jar
commons-net-3.1.jar                                jsch-0.1.42.jar
commons-pool-1.5.4.jar                             jsp-api-2.1.jar
curator-client-2.6.0.jar                           jsr305-1.3.9.jar
curator-framework-2.6.0.jar                        jta-1.1.jar
curator-recipes-2.6.0.jar                          kafka_2.10-0.8.1.1.jar
datanucleus-api-jdo-3.2.1.jar                      kite-data-core.jar
datanucleus-core-3.2.2.jar                         kite-data-hbase.jar
datanucleus-rdbms-3.2.1.jar                        kite-data-hive.jar
derby-10.8.2.2.jar                                 kite-hadoop-compatibility.jar
eigenbase-properties-1.1.4.jar                     leveldbjni-all-1.8.jar
flume-avro-source-1.5.0-cdh5.4.0.jar               libfb303-0.9.2.jar
flume-dataset-sink-1.5.0-cdh5.4.0.jar              libthrift-0.9.2.jar
flume-file-channel-1.5.0-cdh5.4.0.jar              log4j-1.2.17.jar
flume-hdfs-sink-1.5.0-cdh5.4.0.jar                 logredactor-1.0.2.jar
flume-hive-sink-1.5.0-cdh5.4.0.jar                 mapdb-0.9.9.jar
flume-irc-sink-1.5.0-cdh5.4.0.jar                  metrics-core-2.2.0.jar
flume-jdbc-channel-1.5.0-cdh5.4.0.jar              mina-core-2.0.4.jar
flume-jms-source-1.5.0-cdh5.4.0.jar                netty-3.6.2.Final.jar
flume-kafka-channel-1.5.0-cdh5.4.0.jar             opencsv-2.3.jar
flume-kafka-source-1.5.0-cdh5.4.0.jar              paranamer-2.3.jar
flume-ng-auth-1.5.0-cdh5.4.0.jar                   parquet-avro.jar
flume-ng-configuration-1.5.0-cdh5.4.0.jar          parquet-column.jar
flume-ng-core-1.5.0-cdh5.4.0.jar                   parquet-common.jar
flume-ng-elasticsearch-sink-1.5.0-cdh5.4.0.jar     parquet-encoding.jar
flume-ng-embedded-agent-1.5.0-cdh5.4.0.jar         parquet-format.jar
flume-ng-hbase-sink-1.5.0-cdh5.4.0.jar             parquet-generator.jar
flume-ng-kafka-sink-1.5.0-cdh5.4.0.jar             parquet-hadoop-bundle.jar
flume-ng-log4jappender-1.5.0-cdh5.4.0.jar          parquet-hadoop.jar
flume-ng-morphline-solr-sink-1.5.0-cdh5.4.0.jar    parquet-jackson.jar
flume-ng-node-1.5.0-cdh5.4.0.jar                   pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
flume-ng-sdk-1.5.0-cdh5.4.0.jar                    protobuf-java-2.5.0.jar
flume-scribe-source-1.5.0-cdh5.4.0.jar             scala-library-2.10.1.jar
flume-spillable-memory-channel-1.5.0-cdh5.4.0.jar  serializer-2.7.1.jar
flume-thrift-source-1.5.0-cdh5.4.0.jar             servlet-api-2.5-20110124.jar
flume-tools-1.5.0-cdh5.4.0.jar                     servlet-api-2.5.jar
flume-twitter-source-1.5.0-cdh5.4.0.jar            slf4j-api-1.7.5.jar
groovy-all-2.1.6.jar                               slf4j-log4j12-1.7.5.jar
gson-2.2.4.jar                                     snappy-java-1.0.4.1.jar
guava-11.0.2.jar                                   ST4-4.0.4.jar
guice-3.0.jar                                      stax-api-1.0.1.jar
guice-servlet-3.0.jar                              stax-api-1.0-2.jar
hadoop-annotations.jar                             stringtemplate-3.2.1.jar
hadoop-auth.jar                                    twitter4j-core-3.0.3.jar
hadoop-common.jar                                  twitter4j-media-support-3.0.3.jar
hadoop-yarn-api.jar                                twitter4j-stream-3.0.3.jar
hadoop-yarn-common.jar                             velocity-1.7.jar
hadoop-yarn-server-applicationhistoryservice.jar   xalan-2.7.1.jar
hadoop-yarn-server-common.jar                      xercesImpl-2.9.1.jar
hadoop-yarn-server-resourcemanager.jar             xml-apis-1.3.04.jar
hadoop-yarn-server-web-proxy.jar                   xmlenc-0.52.jar
hive-ant.jar                                       xz-1.0.jar
hive-common.jar                                    zkclient-0.3.jar
hive-exec.jar


Thanks

From: أنس الليثي [mailto:dev.fanooos@gmail.com]
Sent: Wednesday, May 18, 2016 11:28 AM
To: user@flume.apache.org
Subject: Re: Flume MorphlineSolrSink

I am not sure the issue you shared is the same issue you are facing.

NoClassDefFoundError is totally different than NoSuchMethodError.

I think solving this problem without upgrading is easy.

Could you please tell what is the version of Solr you installed? and also put a list of the
files in the flume lib directory.

Best regards

On 18 May 2016 at 11:11, Baris Akgun (Garanti Teknoloji) <BarisAkgu@garanti.com.tr<mailto:BarisAkgu@garanti.com.tr>>
wrote:
Hi

You re right. I also find some jira for that problem. (https://issues.apache.org/jira/browse/FLUME-2392)
The problem is solved in 1.6.0 version.

Unfortunately, we are using oracle bda with cdh 5.4 ,therefore it is not easy to upgrade any
service (flume, hive etc.) . We depended the oracle for upgrading.

How can we solve that problem without upgrading the flume? It must be some other solutions.

Thanks..

From: Gonzalo Herreros [mailto:gherreros@gmail.com<mailto:gherreros@gmail.com>]
Sent: Wednesday, May 18, 2016 10:48 AM
To: user
Subject: Re: Flume MorphlineSolrSink

Flume 1.5.0 is pretty old,
Why don't you use version 1.6.0 included in CDH, that will ensure the library compatibility.

On 18 May 2016 at 08:43, Baris Akgun (Garanti Teknoloji) <BarisAkgu@garanti.com.tr<mailto:BarisAkgu@garanti.com.tr>>
wrote:
Hi,

I am trying to make real time indexing with using flume 1.5.0 and morphline solr sink. I am
using packed base solr in my CDH.


tier1.sinks = solrSink
tier1.channels = channel1
#
tier1.channels.channel1.type=memory
tier1.channels.channel1.capacity=1000000
#
tier1.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
tier1.sinks.solrSink.batchSize = 100
tier1.sinks.solrSink.batchDurationMillis = 10000
tier1.sinks.solrSink.morphlineFile = morphlines.conf
tier1.sinks.solrSink.morphlineId = morphline1
#
tier1.sinks.solrSink.channel = channel1


The flume throw error in sink step.  The error is given below. What is your ideas to solve
the problem?

Thanks.


ERROR
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@748016d1 counterGroup:{
name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.apache.lucene.util.fst.FST$BytesReader.skipBytes(J)V
         at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1117)
         at org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:1030)
         at org.apache.lucene.util.fst.FST.readRootArcs(FST.java:487)
         at org.apache.lucene.util.fst.FST.cacheRootArcs(FST.java:475)
         at org.apache.lucene.util.fst.FST.finish(FST.java:458)
         at org.apache.lucene.util.fst.Builder.finish(Builder.java:464)
         at org.apache.lucene.analysis.synonym.SynonymMap$Builder.build(SynonymMap.java:284)
         at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.loadSynonyms(FSTSynonymFilterFactory.java:145)
         at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.inform(FSTSynonymFilterFactory.java:112)
         at org.apache.lucene.analysis.synonym.SynonymFilterFactory.inform(SynonymFilterFactory.java:89)
         at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:675)
         at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:167)
         at org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:55)
         at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:69)
         at org.kitesdk.morphline.solr.SolrLocator.getIndexSchema(SolrLocator.java:181)
         at org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder$SanitizeUnknownSolrFields.<init>(SanitizeUnknownSolrFieldsBuilder.java:70)
         at org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder.build(SanitizeUnknownSolrFieldsBuilder.java:52)
         at org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:302)
         at org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:249)
         at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:46)
         at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
         at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
         at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
         at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
         at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
         at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
         at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
         at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)





Barış Akgün
Analitik Veri Ambarı ve Büyük Veri Yönetimi
Uzman

Tel

:

Dahili

:

Faks

:



Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu
mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi
ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve
eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin
herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige
ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden
ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated
in this message. If you received this message although you are not the addressee, you are
responsible to keep the message confidential. The sender has no responsibility for the accuracy
or correctness of the information in the message and its attachments. Our company shall have
no liability for any changes or late receiving, loss of integrity and confidentiality, viruses
and any damages caused in anyway to your computer system.

Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu
mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi
ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve
eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin
herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige
ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden
ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated
in this message. If you received this message although you are not the addressee, you are
responsible to keep the message confidential. The sender has no responsibility for the accuracy
or correctness of the information in the message and its attachments. Our company shall have
no liability for any changes or late receiving, loss of integrity and confidentiality, viruses
and any damages caused in anyway to your computer system.



--
Anas Rabei
Senior Software Developer
Mubasher.info
anas.rabei@mubasher.info<mailto:anas.rabei@mubasher.info>
Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu
mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi
ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve
eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin
herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige
ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden
ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated
in this message. If you received this message although you are not the addressee, you are
responsible to keep the message confidential. The sender has no responsibility for the accuracy
or correctness of the information in the message and its attachments. Our company shall have
no liability for any changes or late receiving, loss of integrity and confidentiality, viruses
and any damages caused in anyway to your computer system.
Mime
View raw message