Publish to my blog (weekly)
Running Spark on Kubernetes - Spark 3.3.0 Documentation If you run your driver inside a Kubernetes pod, you can use a headless service to allow your driver pod to be routable from the executors by a stable hostname. When deploying your headless service, ensure that the service’s label selector will only match the driver pod and no other pods; it is recommended to assign your driver pod a sufficiently unique label and to use that label in the label selector of the headless service. Specify the driver’s hostname via spark.driver.host and your spark driver’s port to spark.driver.port . Posted from Diigo . The rest of my favorite links are here .