[3-spark-fundamentals] Tests failing to initialise a Spark session

[alexgawrilow]

When running pytest or python -m pytest, I get the following error:

pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

For the spark part I am using the setup with docker where I run docker compose up. For the python part I created a virtual environment and installed the requirements there.
I also tried to set the PYSPARK_SUBMIT_ARGS environment variable as suggested here.
Am I missing something in the setup with running spark through docker and having a local virtual python environment?\

The first part with running Spark in a Jupyter Notebook works fine.

[peterbonnesoeur]

This error typically occurs when the PySpark runtime cannot establish communication with the Java Spark backend.

For this exercise, spark needs to be set up locally (as said in the note in bootcamp/materials/3-spark-fundamentals/README.md)

For this, you first need to install java on your machine (which is the part that is likely failing) and then reinstall pyspark (that is in the requirements.txt)

Let me know if that helped