Salesforce Gets Multi-AZ HA on SageMaker With SchedulingConfig

Salesforce's Agentforce platform runs more than 50 production models on SageMaker, and the team had already achieved an 8x reduction in GPU infrastructure costs by co-hosting multiple models on shared accelerators using SageMaker Inference Components (ICs). The problem: SageMaker's default IC placement algorithm optimises each deployment operation independently, distributing copies evenly across instances without considering Availability Zone balance. Even a properly configured multi-AZ endpoint can accumulate all copies of a given model in a single AZ — a silent compliance violation under Salesforce's mandate requiring 2-AZ support for every production model. The fix is SchedulingConfig, a parameter added to the CreateInferenceComponent API that gives operators explicit, per-IC control over cross-AZ distribution and within-AZ packing strategy.

This matters beyond Salesforce. As pipeline architecture increasingly drives production AI gains, infrastructure teams need concrete primitives — not just endpoint-level HA — to meet uptime guarantees.

The Default Placement Gap

SageMaker's original IC scheduler treats each CreateInferenceComponent call as an isolated optimisation problem. It spreads copies evenly across available instances but does not track cumulative AZ distribution across multiple IC deployments on the same endpoint. In a 4-instance endpoint with 2 instances in AZ-1 and 2 in AZ-2, deploying three ICs sequentially — IC1 with 4 copies, IC2 with 2 copies, IC3 with 2 copies — can leave any one of them concentrated in a single zone. Two failure modes follow: an instance crash takes out all copies of a model simultaneously; an AZ outage makes the model entirely unavailable. Neither is detectable from endpoint-level health metrics alone.

SchedulingConfig: Two Parameters, Two Levels of Control

SchedulingConfig surfaces two sub-parameters inside the Specification block of CreateInferenceComponent:

• AvailabilityZoneBalance — controls cross-AZ distribution with a configurable MaxImbalance tolerance (maximum permitted copy-count difference between any two AZs) and an EnforcementMode (currently PERMISSIVE only, meaning best-effort placement rather than hard failure when balance cannot be achieved).
• PlacementStrategy — controls within-AZ instance-level distribution. SPREAD distributes copies across as many instances as possible; BINPACK packs copies onto fewer instances to maximise GPU utilisation.

Salesforce chose SPREAD across the board, accepting lower packing density in exchange for fault isolation. For a 4-copy IC on a 4-instance, 2-AZ endpoint:

'SchedulingConfig': {
    'PlacementStrategy': 'SPREAD',
    'AvailabilityZoneBalance': {
        'EnforcementMode': 'PERMISSIVE',
        'MaxImbalance': 1
    }
}

Setting MaxImbalance to 1 tolerates at most a 1-copy difference between AZs. For lighter models with only 2 copies, MaxImbalance: 0 enforces strict 1-copy-per-AZ balance, preserving availability through an entire AZ failure. The source is explicit: never set CopyCount to 1 for HA-critical models — a single copy can only reside in one AZ and immediately breaks 2-AZ compliance.

Configuration Parameters at a Glance

DataCacheConfig and RoutingConfig are endpoint-level features independent of SchedulingConfig; they complement HA deployments but are not part of the IC placement algorithm.

Scaling, Consolidation, and Observability

SchedulingConfig governs placement for each individual scale operation. When CopyCount increases, SageMaker places new copies to restore AZ balance; symmetric scale-in removes copies evenly across zones. For long-running endpoints that undergo repeated scale cycles, set the endpoint's ScaleInPolicy to CONSOLIDATION, which enables a background sweeper to periodically consolidate IC copies and release idle instances while honouring AZ balance constraints.

AWS also recommends On-Demand Capacity Reservations (ODCR) in each target AZ. Without them, on-demand capacity constraints in high-demand regions can prevent balanced placement even when SchedulingConfig requests it. The placement algorithm will still attempt partial deployment rather than failing the operation outright, but the resulting balance may not meet MaxImbalance targets.

Ongoing validation runs through SageMaker AI Insights: the Reliability tab exposes AZ skew as a distribution imbalance percentage, per-IC copy counts broken down by AZ, rebalancing event frequency and duration, and Insufficient Capacity Error counts per AZ and instance type — useful for determining whether ODCR allocations need adjustment. These metrics are also accessible through Amazon CloudWatch.

Endpoint-level HA has always been table stakes; IC-level placement control is what closes the gap between endpoint health and model availability. Salesforce's pattern — SPREAD placement with MaxImbalance: 0 for 2-copy models and MaxImbalance: 1 for 4-copy models — is a reproducible template any team can adapt directly to their own SageMaker fleet, without surrendering the GPU cost efficiency that co-hosting delivers.