Spark Integration Hub K8s

Learn about configurations >

• driver-pod-template | string

  Define K8s driver pod from a file accessible to the spark-submit process. Ex.: "s3a://bucket/template.yml". Is equivalent to the following Spark configuration:

  spark.kubernetes.driver.podTemplateFile=<driver-pod-template>

• enable-dynamic-allocation | boolean

  Enable dynamic allocation of pods for autoscaling Spark jobs. Is equivalent to the following Spark configurations:

  spark.dynamicAllocation.enabled=true spark.dynamicAllocation.shuffleTracking.enabled=true spark.dynamicAllocation.minExecutors=1

• executor-pod-template | string

  Define K8s executor pods from a file accessible to the spark-submit process. Ex.: "s3a://bucket/template.yml". Is equivalent to the following Spark configuration:

  spark.kubernetes.executor.podTemplateFile=<executor-pod-template>

• monitored-service-accounts | string

  Comma-separated patterns for namespaces and service accounts to monitor and update in addition to the ones managed by the charm's relations. Ex.: "namespace1:sa1,namespace2:*".

• spark-image | string

  Define a custom Spark image to be used for driver and executor pods. If this option is set, the integration hub will supply the related application with an additional Spark property to use this image for drivers and executors. If not set, it's up to the related application to choose what image to use for the driver and executor pods.