inference service
A hosted service that runs AI models, receiving inputs and returning outputs in an agent system.
Learn
When to use it
Use an inference service when running AI models locally is impractical due to resource constraints. Inference services provide scalable, managed environments for executing models, enabling tasks like real-time data processing and complex computations without local infrastructure.
Quick example
In Amazon Bedrock, you want to deploy a large language model to handle customer queries without setting up your own servers. You use Bedrock's inference service to host the model, which manages the compute resources and scales automatically with demand. Bedrock itself is the inference service, offering a seamless way to run models in a hosted environment.
prompt → inference service → output
Ecosystem
Inference services sit between the client application and the model, handling requests and responses. They often integrate with data sources and storage for comprehensive AI solutions.
client app → inference service → model
Misconceptions
| Misconception | Rebuttal |
|---|---|
| Inference services train models | They only run pre-trained models |
| They require local hardware | They operate on cloud infrastructure |
| Inference is always real-time | It can be batch-processed as well |
Trade-offs
- Scalability — may incur higher costs with increased usage
- Resource management — offloads complexity but reduces direct control
- Performance — depends on network latency and service optimization