EMR cluster bootstrap + setting environment variables cluster-wise - pyspark

Question

I am trying to create an EMR cluster (through the command line) and give it some bootstrap actions and configurations file.
The aim is setting some SPARK/Yarn vars, and some other environment variables that should be used cluster-wise (so these env vars should be available on the master AND the slaves).
I am giving it a configurations file that looks like this:
[
{
"Classification": "yarn-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"appMasterEnv.SOME_VAR": "123",
"nodemanager.vmem-check-enabled": "false",
"executor.memoryOverhead": "5g"
},
"Configurations": [
]
}
]
},
{
"Classification": "spark-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"appMasterEnv.SOME_VAR": "123",
"PYSPARK_DRIVER_PYTHON": "python36",
"PYSPARK_PYTHON": "python36",
"driver.memoryOverhead": "14g",
"driver.memory": "14g",
"executor.memory": "14g"
},
"Configurations": [
]
}
]
}
]
However when I try to add some steps to the cluster, the step fails claiming it does not know about the environment variable SOME_VAR.
Traceback (most recent call last):
File "..", line 9, in <module>.
..
raise EnvironmentError
OSError
(The line number is where I am trying to use the environment var SOME_VAR)
Am I doing it the right way both for SOME_VAR and the other Spark/Yarn vars?
Thank you

Answer 1

Remove appMasterEnv in front of appMasterEnv.SOME_VAR, as user lenin suggested.
Use classification yarn-env to pass environment variables to the worker nodes.
Use classification spark-env to pass environment variables to the driver, with deploy mode client. When using deploy mode cluster, use yarn-env.