Deployment and Orchestration of ML and AI Workflows

P-family instances (for example, p4d, p5) provide high-end NVIDIA GPUs for large training jobs; G-family instances (for example, g5, g6) provide cost-effective GPUs for inference and lighter training.

Inf1/Inf2 instances use AWS Inferentia chips built for high-throughput, low-cost inference; models are compiled with the AWS Neuron SDK.

The AWS Neuron SDK, which compiles and runs models (PyTorch, TensorFlow, JAX) on Neuron devices.

For small or classical models (linear, tree ensembles, small NLP models) with modest latency needs, where GPUs would sit mostly idle and cost more.

SageMaker AI endpoints (real-time, serverless, async) and batch transform, Amazon Bedrock for FMs, containers on Amazon EKS or ECS, AWS Lambda for small models, and EC2 for full control.

When the organization standardizes on Kubernetes, needs custom serving stacks or fine-grained control over scheduling and networking, and has the skills to operate it.