handbook/docs/deploy.md

# Deployment Guide

> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.

!!! tip "What can run here"
    APIs, websites, applets, bots, monitors.

!!! danger "Your compose names nothing repo-specific"
    **No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
    deploy layer injects identity and host paths at deploy time — your compose stays
    generic so it can't collide with any other service on the fleet.

    - service key is always **`svc`**
    - named volumes use **bare names** (`cache`, not `myapp_cache`)
    - host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**

    Naming things after your repo used to cause **container-name collisions** at
    fleet scale — two repos shipping the same name clashed. The generic compose
    below fixes that; copy it verbatim.

## Docker — the services account

Every service runs containerized as the shared **`services`** account:
**uid/gid 1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it
runs cleanly as that account.

```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp                                  # (1)!

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*            # (2)!

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # (3)!

COPY . .
RUN chmod -R a+rwX /app                        # (4)!
CMD ["python", "-m", "yourapp"]
```

1.  `HOME=/tmp` — the `services` account has no home dir; anything writing to
    `$HOME` (caches, configs) needs a writable target.
2.  Include `git` **only if the container itself needs it** — e.g. you
    `pip install` from git, or the app shells out to git at runtime. The build and
    host always have git; this line is about what's *inside* the image.
3.  Install deps **before** copying the code (see [layer
    caching](#layer-caching)).
4.  `chmod -R a+rwX /app` makes the app tree writable by **any** uid — that's what
    "uid-agnostic" means.

!!! tip "Faster builds with uv (optional)"
    [uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
    `pyproject.toml` — no lockfile needed in the image. Swap the deps layer and add
    `UV_COMPILE_BYTECODE` so containers don't pay the first-import `.pyc` compile
    cost:

    ```dockerfile
    FROM python:3.12-slim
    COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # (1)!
    ENV HOME=/tmp
    ENV UV_COMPILE_BYTECODE=1                              # (2)!

    WORKDIR /app
    COPY pyproject.toml .
    RUN uv pip install --system .                          # (3)!
    COPY . .
    RUN chmod -R a+rwX /app
    CMD ["python", "-m", "yourapp"]
    ```

    1.  Pull the `uv` binary from its published image — no pip-installing uv itself.
    2.  Compile bytecode at build time so the container doesn't eat the first-import
        `.pyc` compile cost on every cold start.
    3.  `--system` installs into the image's Python (no venv needed — the container
        *is* the isolation); reads `pyproject.toml`, no `uv.lock` required.

### Getting your files into the image

`COPY . .` grabs the whole repo, but be explicit about anything that needs its own
place — assets, templates, a config the app reads at runtime. `COPY <src> <dest>`:
`<src>` is relative to the build context (your repo), `<dest>` is a path in the
image.

=== "A single file"

    ```dockerfile
    COPY ./config.toml /app/config.toml   # (1)!
    ```

    1.  One file into a specific path. The app reads it at `/app/config.toml`.

=== "A directory"

    ```dockerfile
    COPY ./assets /app/assets   # (1)!
    ```

    1.  A whole tree. Trailing paths are dirs — `/app/assets/` mirrors `./assets/`.

=== "Deps first (layer caching)"

    ```dockerfile
    COPY requirements.txt .     # (1)!
    RUN pip install --no-cache-dir -r requirements.txt
    COPY . .                    # (2)!
    ```

    1.  Copy just the deps file first, install, **then** copy the code.
    2.  See [layer caching](#layer-caching) — this ordering is why.

!!! warning "Don't `COPY` secrets into the image"
    Anything sensitive stays **out** of the image — no `COPY ./secrets.env`. Secrets
    live on the host and are injected read-only at runtime (see [Secrets](#secrets)).
    Add them to `.dockerignore` so a blanket `COPY . .` can't sweep them in.

## Your `compose.yaml`

Copy this verbatim. It names nothing repo-specific — the deploy layer fills in
identity and host paths.

```yaml
services:
  svc:                                       # (1)!
    build: .
    user: "1337:1337"                        # (2)!
    restart: unless-stopped                  # (3)!
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # (4)!
      - ${CONFIG_DIR:-./config}:/app/config  # (5)!
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # (6)!
      - cache:/app/cache                     # (7)!

volumes:
  cache:                                     # (8)!
```

1.  The service key is always **`svc`** — never a repo-specific name, never a
    `container_name`. The ops tooling keys off `svc`.
2.  Run as the shared **`services`** account (**uid/gid 1337**, fixed fleet-wide).
    **No** in-container `user`/`useradd` — set it here. This is the one identity
    line that stays; it's not repo-specific.
3.  **Required.** The host-side service is oneshot; Docker's own restart policy is
    what recovers a crashed container.
4.  Logs → host (**output**), path **injected** by the deploy layer. The `:-./logs`
    default lets you `docker compose up` locally with nothing set and still work.
5.  Config → host (**input you edit**), likewise injected.
6.  **Optional, commented by default.** A host dir for arbitrary read/write data.
    The deploy layer injects `MOUNTS_DIR` and auto-creates the dir — but you
    **uncomment** the line and pick the container path (`/app/data` here), since
    that's app-specific and can't be auto-mounted.
7.  Named volume with a **bare name** — ephemeral, auto-namespaced per service at
    deploy time so it can't collide.
8.  Declare bare (`cache`, not `myapp_cache`). Deploy auto-prefixes it.

!!! tip "Three tiers of storage"
    - **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
      injected and managed at deploy time.
    - **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy
      layer **injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is
      **opt-in**: uncomment it and choose the container path yourself (it's
      app-specific, so it can't be auto-mounted).
    - **Named volumes** (`cache`, …) — **yours**: anything else your service
      persists. Docker owns them, no host paths to manage.

!!! tip "The `${VAR:-./default}` pattern"
    Host paths use a variable with a local fallback. Locally, `docker compose up`
    needs nothing set — it uses `./logs`, `./config` (and `./mounts` if you enable
    it). When deployed, the ops tooling sets the real values, so the same file works
    both places.

## Deployment is handled centrally

!!! note "You don't run any deploy commands"
    All services are deployed centrally by the ops tooling. You don't pick a host,
    manage keys, or run anything — just make your repo follow the compose convention
    above and it's deployable.

At deploy time the tooling **generates** the environment your compose reads, so the
`${...}` variables resolve without anything from you. You can rely on these being
set — that's why your compose works unchanged on the fleet:

```
LOGS_DIR      # your host log dir      -> ${LOGS_DIR}
CONFIG_DIR    # your host config dir   -> ${CONFIG_DIR}
MOUNTS_DIR    # optional host data dir -> ${MOUNTS_DIR} (if you enable the mount)
```

Write your logs and config to those mount points and you're set. If a service won't
come up or its logs aren't persisting, that's a host-side detail on our end — flag it
and we'll sort it.

## Layer caching

Copy the deps file and install **before** `COPY . .`. Docker caches layers in
order, so deps only reinstall when the deps file changes — not on every code edit.
Get this backwards and every one-line change triggers a full dependency reinstall.
Same principle whether you use pip or uv:

```dockerfile
COPY requirements.txt .                              # pip baseline
RUN pip install --no-cache-dir -r requirements.txt   # cached until deps change
COPY . .                                             # changes every build
```

```dockerfile
COPY pyproject.toml .                                # uv path
RUN uv pip install --system .                        # cached until deps change
COPY . .                                             # changes every build
```

## Subprocess and browser workloads

Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra
knobs in compose:

```yaml
services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # (1)!
    shm_size: "2gb"         # (2)!
    mem_limit: "4g"         # (3)!
```

1.  Runs **tini** as PID 1 to reap zombie subprocesses and forward signals.
    Without it, spawned Chrome/Xvfb processes leak as zombies.
2.  Chrome and most headless browsers crash on Docker's default **64 MB**
    `/dev/shm`. Bump it for any browser workload.
3.  Bound memory — especially when each worker spawns a browser. Raise it as
    worker count grows.

!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
    If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
    be PID 1, or the real process dies on startup. `init: true` is exactly what
    fixes this — tini takes PID 1, and your wrapper runs as a normal child.

## What your compose / Dockerfile needs

**Compose**

- service key `svc` — **no `container_name`**, no repo-specific names
- `user: "1337:1337"` — the shared `services` account
- `restart: unless-stopped`
- host paths via `${LOGS_DIR}` / `${CONFIG_DIR}` (and optional `${MOUNTS_DIR}`) —
  never hardcoded
- named volumes with **bare names** (`cache`, not `myapp_cache`)
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`

**Dockerfile**

- `HOME=/tmp`
- `chmod -R a+rwX /app` (uid-agnostic; runs as the `services` account, 1337)
- deps installed **before** the code copy (layer caching) — pip or `uv pip install`
- `git` in the image **if the container needs it**
- using uv? add `ENV UV_COMPILE_BYTECODE=1`

## Secrets

!!! warning "Secrets never go in the image"
    We do **not** commit secrets (usually, lol). They stay **gitignored** and live
    on the host in your config dir, reaching your container read-only via
    `${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
    into a layer.

Rotating a secret is a host-side edit — update the file and the service picks it up
on restart. No rebuild, and nothing you run: flag it and we handle the restart.