livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject How to run multiple Dstream transformations using Livy without creating multiple streaming contexts?
Date Wed, 16 May 2018 02:22:56 GMT
Hi All,

I am wondering how to run multiple DStream transformations using Livy
without creating multiple streaming contexts? This seems be tricky compared
to running multiple transformations on structured streaming.

So every time I spawn a job via Livy I want to a new Dstream transformation
to run. is that possible? I am running into two errors

1) Cannot create multiple streaming contexts. Not that I wish to but I want
to be able to run multiple dstream transformations so whatever efficient
way exists I am fine

2)  Whenever I spawn a job via livy I hit this line and it would throw an
illegal state exception.  I am not sure how to get around this

KafkaUtils.createDirectStream(
                        jssc,
                        LocationStrategies.PreferConsistent(),
                        ConsumerStrategies.<String, byte
[]>Subscribe(topics, consumerConfig)
                );

Thanks!

Mime
View raw message