Publish to my blog (weekly)
-
Running Spark on Kubernetes - Spark 3.3.0 Documentation
- If you run your driver inside a Kubernetes pod, you can use a headless service to allow your driver pod to be routable from the executors by a stable hostname.
- When deploying your headless service, ensure that the service’s label selector will only match the driver pod and no other pods; it is recommended to assign your driver pod a sufficiently unique label and to use that label in the label selector of the headless service.
- Specify the driver’s hostname via
spark.driver.host
and your spark driver’s port tospark.driver.port
.
-
댓글