Together AI described a compiler that searches GPU kernel and parallelism configurations for model inference. Its search incorporates measurements from the model workload, since traffic patterns such as prompt length and expert usage affect which configuration performs well.
Together AI
Also known as: Together Inference
Plain English
Together AI hosts open-weight models behind APIs. It offers shared access for variable workloads as well as reserved capacity and custom deployments for teams with production requirements.
Technical Overview
The platform offers shared serverless inference, token-based reserved capacity, and dedicated endpoints. Dedicated deployments can route requests between versions behind a stable endpoint. The provider also optimizes GPU kernels and parallelism configurations in its serving stack.
Chronological Entries (3)
Together AI introduced support for multiple deployments behind a stable endpoint. Operators could direct some traffic to a new version and use rollback thresholds to limit the impact of a rollout that performs poorly.
Together AI introduced reserved inference capacity measured in tokens per minute. Customers could reserve a rate for a supported model without choosing the GPU count or running a model server themselves.
Direct Relations (2)
- Part ofInference Provider
- Alternative toFireworks AI