Clash Royale CLAN TAG#URR8PPP

Is it possible to get the current spark context settings in PySpark?

I'm trying to get the path to spark.worker.dir for the current sparkcontext.

spark.worker.dir

sparkcontext

If I explicitly set it as a config param, I can read it back out of SparkConf, but is there anyway to access the complete config (including all defaults) using PySpark?

config param

SparkConf

config

PySpark

SparkConf

spark.worker.dir

10 Answers
10

Yes: sc._conf.getAll()

Which uses the method:

SparkConf.getAll()


as accessed by

SparkContext.sc._conf


Note the Underscore: that makes this tricky. I had to look at the spark source code to figure it out ;)

But it does work:

In [4]: sc._conf.getAll()
Out[4]:
[(u'spark.master', u'local'),
(u'spark.rdd.compress', u'True'),
(u'spark.serializer.objectStreamReset', u'100'),
(u'spark.app.name', u'PySparkShell')]


Spark 2.1+

spark.sparkContext.getConf().getAll() where spark is your sparksession (gives you a dict with all configured settings)

spark.sparkContext.getConf().getAll()

spark

sparksession

dict

Spark 1.6+

sc.getConf.getAll.foreach(println)


sc.getConf.getAll.foreach(println)

For a complete overview of your Spark environment and configuration I found the following code snippets useful:

SparkContext:

for item in sorted(sc._conf.getAll()): print(item)


Hadoop Configuration:

hadoopConf =
iterator = sc._jsc.hadoopConfiguration().iterator()
while iterator.hasNext():
prop = iterator.next()
hadoopConf[prop.getKey()] = prop.getValue()
for item in sorted(hadoopConf.items()): print(item)


Environment variables:

import os
for item in sorted(os.environ.items()): print(item)


You can use:

ssc.sparkContext.getConf.getAll


For example, I often have the following at the top of my Spark programs:

logger.info(ssc.sparkContext.getConf.getAll.mkString("n"))


update configuration in Spark 2.3.1

To change the default spark configurations you can follow these steps:

Import the required classes

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession


Get the default configurations

spark.sparkContext._conf.getAll()


Update the default configurations

conf = spark.sparkContext._conf.setAll([('spark.executor.memory', '4g'), ('spark.app.name', 'Spark Updated Conf'), ('spark.executor.cores', '4'), ('spark.cores.max', '4'), ('spark.driver.memory','4g')])


Stop the current Spark Session

spark.sparkContext.stop()


Create a Spark Session

spark = SparkSession.builder.config(conf=conf).getOrCreate()


Not sure if you can get all the default settings easily, but specifically for the worker dir, it's quite straigt-forward:

from pyspark import SparkFiles
print SparkFiles.getRootDirectory()


Just for the records the analogous java version:

Tuple2<String, String> sc = sparkConf.getAll();
for (int i = 0; i < sc.length; i++)
System.out.println(sc[i]);



For Spark 2+ you can also use when using scala

spark.conf.getAll; //spark as spark session


Unfortunately, no, the Spark platform as of version 2.3.1 does not provide any way to programmatically access the value of every property at run time. It provides several methods to access the values of properties that were explicitly set through a configuration file (like spark-defaults.conf), set through the SparkConf object when you created the session, or set through the command line when you submitted the job, but none of these methods will show the default value for a property that was not explicitly set. For completeness, the best options are:

spark-defaults.conf

SparkConf

http://<driver>:4040

SparkContext

getAll

spark.sparkContext._conf.getAll()

SET

spark.sql("SET").toPandas()

SET -v

(These three methods all return the same data on my cluster.)

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.