Running models locally

Overview

wriftai model start runs a model locally, and wriftai model serve runs a model server for local development. Both currently use Docker to do this.

Requirements

• Docker, version 29.6.0 or later

GPU support

Both commands can additionally expose GPUs to a model. This depends on Docker's NVIDIA GPU integration.

Prerequisites

To run a model locally with GPU acceleration, you also need:

• An NVIDIA GPU
• NVIDIA drivers installed on the host
• The NVIDIA Container Toolkit installed and configured for Docker

If any of these are missing, the model runs on CPU instead of failing outright.

You can confirm your setup is working independently of wriftai by running:

docker run -it --rm --gpus all ubuntu nvidia-smi

If this prints a GPU table, your host is ready. If it errors, the issue is in your Docker/NVIDIA setup rather than in wriftai.

Choosing which GPUs to expose

Both wriftai model start and wriftai model serve accept a --gpus flag. All available GPUs are exposed by default.

For the full list of accepted --gpus values and usage examples, see the model start and model serve reference pages.