What's the best way to set credentials securely? Is there a way to pass variables to the %%configure cell?

[flowy0]

Is there a way to pass variables into the %%configure cell?

What's the best way to set credentials for the spark config in the spark session?

Eg:

%%configure -f
{"conf": {
        "spark.hadoop.fs.s3a.access.key":"myusername",
        "spark.hadoop.fs.s3a.secret.key": "mypassword",
}

[devstein]

Hey @flowy0 -- there is no way to parameterize the `%%configure`` cell. There are two common approaches to setting credentials

1. Set it in your config.json. If you have access to the Jupyter environment, it's fairly common to generate a custom config.json in an init script
2. Use configuration.py to override it in Python, which is what %%configure is doing under the hood

Hope this helps!

[flowy0]

I did try to set config in a separate cell, but it did not seem to take effect, is there something specifically about how the spark context is created (so this method does not work)?

in the cell below %%configure, I created variables and sourced to the spark kernel using %%send_to_spark.

spark._jsc.hadoopConfiguration().set("fs.s3a.access.key",AWS_ACCESS_KEY_ID)
spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key",AWS_SECRET_ACCESS_KEY)

After running this, I couldn't find the updated spark context by running spark.sparkContext.getConf().getAll()

[flowy0]

Sorry, managed to fix this issue by using the command below instead.

spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID)
spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.secret.key", AWS_SECRET_ACCESS_KEY)

Check Config is applied

spark._jsc.hadoopConfiguration().get('fs.s3a.access.key')