PySparkProcessor and spark.dynamicAllocation.maxExecutors conf

[andre-marcos-perez]

Hello, I notice the spark.dynamicAllocation.maxExecutors is limited to the number of instances defined in the PySparkProcessor object:

• The object:

processor = PySparkProcessor(
    ...
    instance_type="ml-<type>"
    instance_count=10
    ...
)

• The trigger:

processor.run(
      ...,
      configuration=[
          {
              "Classification": "spark-defaults",
              "Properties": {
                  "spark.dynamicAllocation.maxExecutors": 40,
              },
          }
      ],
)

So I guess the idea is to do the math to make sure Spark can scale the workers up to the instance count, given the instance type hardware configs and define workers configs. Is there a way to scale the amount of instances beyond the instance_count param?