flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutanu Das <sd2...@att.com>
Subject How to increase NUMBER of Spark Executors ?
Date Thu, 25 Feb 2016 02:22:08 GMT
Community,

How can I increase the NUMBER OF EXECUTORS for my Streaming job Local ?

We have tried spark.master = local[4] but It is not starting 4 executors and our job keeps
getting Queued - do we need to make a code change to increase number of executors?

This job - jar file read from Kafka Stream with 2 partitions and send data to Cassandra -

Please help advise - thanks again community

Here is How we start the job:

nohup spark-submit --properties-file /hadoop_common/airwaveApList.properties --class airwaveApList
/hadoop_common/airwaveApList-1.0.jar

Properties file for the Steaming Job:

spark.cassandra.connection.host       cass_host
spark.cassandra.auth.username         cass_app
spark.cassandra.auth.password         xxxxx
spark.topic                           ap_list_spark_streaming
spark.app.name                        ap-status
spark.metadata.broker.list            server.corp.net:6667
spark.zookeeper.connect               server.net:2181
spark.group.id                        airwave_activation_status
spark.zookeeper.connection.timeout.ms 1000
spark.cassandra.sql.keyspace          enterprise
spark.master                          local[4]
spark.batch.size.seconds              120
spark.driver.memory                   12G
spark.executor.memory                 12G
spark.akka.frameSize                  512
spark.local.dir                       /prod/hadoop/spark/airwaveApList_temp
spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.history.ui.port 18080
spark.yarn.historyServer.address has-dal-0001.corp.wayport.net:18080
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService


Mime
View raw message