Documentation

Deploy your own models and serve them over an OpenAI-compatible API. Bring your weights — open-weight or your own fine-tune — and ModelCloud handles the rest.

Quickstart

From weights to a live endpoint in about a minute. The whole flow:

bash

# 1. install the CLI
curl -fsSL https://modelcloud.cloud/install.sh | sh

# 2. log in (creates an API key)
modelcloud login

# 3. deploy a model
modelcloud deploy ./qwen2.5-7b --name qwen2.5-7b \
  --gpu h100 --min 0 --max 8 --quantize fp8

# 4. call it
curl https://api.modelcloud.cloud/v1/chat/completions \
  -H "Authorization: Bearer $MODELCLOUD_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-7b","messages":[{"role":"user","content":"Hello"}]}'

Install the CLI

The modelcloud CLI handles login, deploy, logs, and local testing. It's a single static binary with no runtime dependencies.

bash

# macOS / Linux
curl -fsSL https://modelcloud.cloud/install.sh | sh

# verify
modelcloud --version

Note: the installer never writes to your shell profile without asking. Download the binary directly from the releases page if you prefer.

Python SDK

The modelcloud package wraps the REST API so you can deploy, wait for a healthy endpoint, and chat without leaving Python. It reads MODELCLOUD_KEY from the environment by default.

python

import modelcloud

mc = modelcloud.Client()

# deploy and block until healthy
dep = mc.deploy(
    model="./qwen2.5-7b",
    name="qwen2.5-7b",
    gpu="h100",
    quantization="fp8",
    min_replicas=0,
    max_replicas=8,
).wait()

out = mc.chat.completions.create(
    model="qwen2.5-7b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(out.choices[0].message.content)

Authentication

All API requests are authenticated with a bearer token. Create a key from the dashboard, or run modelcloud login to generate one interactively.

http

Authorization: Bearer MODELCLOUD_KEY

Keys are scoped to a workspace and can be revoked at any time. Never commit them to source control — use an environment variable or secret store.

Chat completions

The inference API is OpenAI-compatible, so existing SDKs work unchanged — just swap the base URL.

python

from openai import OpenAI

client = OpenAI(
    base_url="https://api.modelcloud.cloud/v1",
    api_key="MODELCLOUD_KEY",
)

resp = client.chat.completions.create(
    model="qwen2.5-7b",   # your deployed model name
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain cold starts."},
    ],
    temperature=0.7,
)
print(resp.choices[0].message.content)

Request parameters

Parameter	Type	Description
`model`	string	The name of your deployed model.
`messages`	array	A list of chat messages in OpenAI format.
`temperature`	number	Sampling temperature, 0–2.
`max_tokens`	integer	Upper bound on generated tokens for this request.
`stream`	boolean	When true, returns server-sent events.

Deploy a model

Deploy from a local directory, Hugging Face, or a private registry. Declarative deploys use a modelcloud.yaml config file.

modelcloud.yaml

# modelcloud.yaml
name: code-assistant
model: hf://acme/llama-3-8b-code
runtime: vllm
quantization: fp8
gpu:
  type: a10g
  count: 1
autoscale:
  min: 0
  max: 8
  target: queue-depth
  scale_to_zero: true
  cooldown_seconds: 60

Apply it with modelcloud deploy -f modelcloud.yaml. Common fields:

Field	Type	Description
`name`	string	Deployment name, used as the model ID on the API.
`model`	string	Weight source — a local path, `hf://`, or a private registry.
`runtime`	string	`vllm`, `trt-llm`, `sglang`, or `llama-cpp`.
`quantization`	string	`bf16`, `fp8`, `int8`, or `int4`.
`gpu.count`	integer	Number of accelerators. >1 enables tensor parallelism.
`autoscale.target`	string	`request-rate`, `queue-depth`, or `gpu-utilization`.

Tensor parallelism

Models too big for one accelerator are sharded automatically. Set gpu.count and we choose the tensor-parallel and pipeline-parallel layout from the model size and context length.

modelcloud.yaml

# 70B model sharded across 4× A100 80GB
name: llama-3-70b
model: hf://meta-llama/Llama-3-70B-Instruct
gpu:
  type: a100-80gb
  count: 4
  tensor_parallel: 4

Billing is per accelerator, so a four-way shard runs at four times the single-GPU rate — with no parallelism overhead fee on top.

List models

List every model deployed in your workspace, with status and endpoint.

json

{
  "object": "list",
  "data": [
    {
      "id": "qwen2.5-7b",
      "status": "serving",
      "replicas": { "current": 2, "min": 0, "max": 8 }
    }
  ]
}

Autoscaling

Autoscaling controls how many replicas serve your model. You can target request rate, queue depth, or GPU saturation.

request-rate — scale when requests per second exceeds a threshold
queue-depth — scale when requests pile up behind a saturated replica
gpu-utilization — scale when a replica's GPU stays hot

Set scale_to_zero: true to stop billing entirely during idle periods. Warm pools resume from cache without the full cold-start penalty.

Observability

Every request emits a trace with time-to-first-token, inter-token latency, and GPU utilization. Export to your own stack over OTLP, or stream with the CLI:

bash

modelcloud logs qwen2.5-7b --follow

req_7f2c  ttft=142ms  itl=31ms  tokens=412  gpu_util=78%