SageMaker HyperPod Gets Managed Ray on EKS, Replacing Manual Kubectl Setup
In this article
AWS announced on 24 August 2026 that SageMaker HyperPod now carries a managed Ray integration on Amazon EKS, collapsing what was previously a multi-tool setup — YAML manifests, manual kubectl port-forwarding, hand-configured Prometheus and Grafana — into a console-driven workflow inside SageMaker Studio. The integration wires KubeRay's cluster lifecycle management directly into HyperPod's node health monitoring, tiered storage, and IAM-authenticated endpoint infrastructure without requiring changes to existing Ray scripts or APIs.
As teams push more ML pipeline work onto managed Kubernetes rather than bespoke training clusters — a direction consistent with the argument that pipeline architecture, not raw model improvements, drives the biggest efficiency gains in 2026 — infrastructure that reduces the Kubernetes expertise tax on data scientists becomes a genuine platform differentiator.
Cluster Architecture and the Operator Stack
The integration layers four components on top of a HyperPod EKS cluster. The SageMaker Spaces EKS add-on enables JupyterLab and Code Editor workspaces that attach directly to Ray clusters. The HyperPod Observability EKS add-on handles metric scraping and Grafana dashboard provisioning. The KubeRay operator manages RayCluster, RayJob, and RayService as native Kubernetes custom resources. A fourth piece, the HyperPod Ray Endpoint Operator deployed via Helm, generates IAM-authenticated public endpoints, eliminating the need for local kubectl port-forward to reach the Ray Dashboard or submit remote jobs.
Cluster creation through Studio prompts for a cluster name, head and worker instance types, worker count, and container image. The default image is the SageMaker Distribution, which ships with Ray pre-installed and receives ongoing vulnerability patching from AWS. Custom images are supported for workloads with additional dependencies. An inline YAML editor exposes the full Kubernetes manifest for teams that need advanced customisation or prefer kubectl. Compute quotas and scheduling priorities integrate with HyperPod task governance alongside other training jobs.
Remote job submission uses the toolkit-for-ray-on-sagemaker-ai Python package, which resolves endpoints and generates EKS API credentials through IAM. The submission command targets a sagemaker_ray://<ray-cluster-name>/<namespace> address scheme on top of standard Ray job APIs, making it usable from CI/CD pipelines without bespoke credential handling.
Observability: Four Pre-Built Dashboards, Zero Manual Configuration
Previously, standing up Ray observability on Kubernetes with Amazon Managed Service for Prometheus and Amazon Managed Grafana required installing Helm charts, creating PodMonitors and ServiceMonitors, configuring IAM roles for SigV4 signing, and manually importing dashboard JSON files. The HyperPod Observability EKS add-on now handles all of this automatically. It discovers Ray head and worker pods, scrapes their metrics endpoints, and provisions four dashboards in Amazon Managed Grafana: Ray Core, Ray Data, Ray Train, and Ray Serve. All four are grouped under a Ray folder and support per-cluster filtering, appearing alongside existing HyperPod infrastructure dashboards covering GPU utilisation, EFA networking, and task governance.
Resilience Layers for Training Workloads
| Resilience Layer | Mechanism | Required Code Changes | User Notification Path |
|---|---|---|---|
| Automatic node recovery | HyperPod health monitoring replaces faulty nodes; Ray reschedules worker pods onto replacement node | None — requires sufficient retries in RayJob FailureConfig and periodic checkpoint logic | N/A — transparent recovery |
| Hung job detection | Per-node Job Monitoring Agent watches node-level and job-level signals for Ray Train workloads | None for default detection; custom log patterns and timeout thresholds via toolkit-for-ray-on-sagemaker-ai | CloudWatch log group + Ray Train Grafana dashboard |
| Tiered checkpointing | amzn-sagemaker-checkpointing writes to local disk then asynchronously uploads to S3; recovery checks HyperPod Tiered Storage before S3 | Library integration required | N/A — transparent faster recovery |
The hung job detection case is operationally significant. A single pod failure in a distributed training run can cause every other pod to block indefinitely at the next collective operation, keeping GPUs allocated with memory loaded while producing no useful compute. Without automated detection, teams typically discover the problem only when they manually check job progress. When the configured action is cancel, HyperPod terminates the hung worker process and Ray Train's FailureConfig restarts workers from the last checkpoint.
Inference: JumpStart Integration and Tiered KV Cache
The toolkit-for-ray-on-sagemaker-ai library now includes a JumpStart model loader that pulls weights from the SageMaker JumpStart catalog and deploys them into Ray Serve on HyperPod without requiring manual weight download, model configuration, or container setup. Ray Serve on HyperPod supports multi-model composition, autoscaling, and serving engines including vLLM.
For long-context workloads, HyperPod introduces Managed Tiered KV Cache. The system stores attention key-value vectors in CPU memory on each node as an L1 cache, and on HyperPod Tiered Storage for cross-instance sharing as an L2 cache. Ray Serve deployments access this capability with minimal code changes, reducing time-to-first-token for multi-turn conversations and long-document requests.
The integration is available now in all AWS Regions where SageMaker HyperPod EKS is supported. Managed Kubernetes for ML is maturing past raw compute provisioning toward operational tooling — fault detection, tiered storage, authenticated dashboards — that previously required substantial platform engineering investment to assemble from open-source components.