handbook/docs/deploy.md

# Deployment Guide

> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.

!!! tip "What can run here"
    APIs, websites, applets, bots, monitors.

!!! danger "Your compose names nothing repo-specific"
    **No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
    deploy layer injects identity and host paths at deploy time — your compose stays
    generic so it can't collide with any other service on the fleet.

    - service key is always **`svc`**
    - named volumes use **bare names** (`cache`, not `myapp_cache`)
    - host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**

    Naming things after your repo used to cause **container-name collisions** at
    fleet scale — two repos shipping the same name clashed. The generic compose
    below fixes that; copy it verbatim.

## Docker — the services account

Every service runs containerized as the shared **`services`** account:
**uid/gid 1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it
runs cleanly as that account.

```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp                                  # (1)!

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*            # (2)!

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # (3)!

COPY . .
RUN chmod -R a+rwX /app                        # (4)!
CMD ["python", "-m", "yourapp"]
```

1.  `HOME=/tmp` — the `services` account has no home dir; anything writing to
    `$HOME` (caches, configs) needs a writable target.
2.  Include `git` **only if the container itself needs it** — e.g. you
    `pip install` from git, or the app shells out to git at runtime. The build and
    host always have git; this line is about what's *inside* the image.
3.  Install deps **before** copying the code (see [layer
    caching](#layer-caching)).
4.  `chmod -R a+rwX /app` makes the app tree writable by **any** uid — that's what
    "uid-agnostic" means.

!!! tip "Faster builds with uv (optional)"
    [uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
    `pyproject.toml` — no lockfile needed in the image. Swap the deps layer and add
    `UV_COMPILE_BYTECODE` so containers don't pay the first-import `.pyc` compile
    cost:

    ```dockerfile
    FROM python:3.12-slim
    COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # (1)!
    ENV HOME=/tmp
    ENV UV_COMPILE_BYTECODE=1                              # (2)!

    WORKDIR /app
    COPY pyproject.toml .
    RUN uv pip install --system .                          # (3)!
    COPY . .
    RUN chmod -R a+rwX /app
    CMD ["python", "-m", "yourapp"]
    ```

    1.  Pull the `uv` binary from its published image — no pip-installing uv itself.
    2.  Compile bytecode at build time so the container doesn't eat the first-import
        `.pyc` compile cost on every cold start.
    3.  `--system` installs into the image's Python (no venv needed — the container
        *is* the isolation); reads `pyproject.toml`, no `uv.lock` required.

### Getting your files into the image

`COPY . .` grabs the whole repo, but be explicit about anything that needs its own
place — assets, templates, a config the app reads at runtime. `COPY <src> <dest>`:
`<src>` is relative to the build context (your repo), `<dest>` is a path in the
image.

=== "A single file"

    ```dockerfile
    COPY ./config.toml /app/config.toml   # (1)!
    ```

    1.  One file into a specific path. The app reads it at `/app/config.toml`.

=== "A directory"

    ```dockerfile
    COPY ./assets /app/assets   # (1)!
    ```

    1.  A whole tree. Trailing paths are dirs — `/app/assets/` mirrors `./assets/`.

=== "Deps first (layer caching)"

    ```dockerfile
    COPY requirements.txt .     # (1)!
    RUN pip install --no-cache-dir -r requirements.txt
    COPY . .                    # (2)!
    ```

    1.  Copy just the deps file first, install, **then** copy the code.
    2.  See [layer caching](#layer-caching) — this ordering is why.

!!! warning "Don't `COPY` secrets into the image"
    Anything sensitive stays **out** of the image — no `COPY ./secrets.env`. Secrets
    live on the host and are injected read-only at runtime (see [Secrets](#secrets)).
    Add them to `.dockerignore` so a blanket `COPY . .` can't sweep them in.

## Your `compose.yaml`

Copy this verbatim. It names nothing repo-specific — the deploy layer fills in
identity and host paths.

```yaml
services:
  svc:                                       # (1)!
    build: .
    user: "1337:1337"                        # (2)!
    restart: unless-stopped                  # (3)!
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # (4)!
      - ${CONFIG_DIR:-./config}:/app/config  # (5)!
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # (6)!
      - cache:/app/cache                     # (7)!

volumes:
  cache:                                     # (8)!
```

1.  The service key is always **`svc`** — never a repo-specific name, never a
    `container_name`. Our tooling keys off `svc` (the `svc` command, the
    `svc-<workspace>-<name>` unit).
2.  Run as the shared **`services`** account (**uid/gid 1337**, fixed fleet-wide).
    **No** in-container `user`/`useradd` — set it here. This is the one identity
    line that stays; it's not repo-specific.
3.  **Required.** The host-side service is oneshot; Docker's own restart policy is
    what recovers a crashed container.
4.  Logs → host (**output**), path **injected** by the deploy layer. The `:-./logs`
    default lets you `docker compose up` locally with nothing set and still work.
5.  Config → host (**input you edit**), likewise injected.
6.  **Optional, commented by default.** A host dir for arbitrary read/write data.
    The deploy layer injects `MOUNTS_DIR` and auto-creates the dir — but you
    **uncomment** the line and pick the container path (`/app/data` here), since
    that's app-specific and can't be auto-mounted.
7.  Named volume with a **bare name** — ephemeral, auto-namespaced per service at
    deploy time so it can't collide.
8.  Declare bare (`cache`, not `myapp_cache`). Deploy auto-prefixes it.

!!! tip "Three tiers of storage"
    - **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
      injected and managed at deploy time.
    - **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy
      layer **injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is
      **opt-in**: uncomment it and choose the container path yourself (it's
      app-specific, so it can't be auto-mounted).
    - **Named volumes** (`cache`, …) — **yours**: anything else your service
      persists. Docker owns them, no host paths to manage.

!!! tip "The `${VAR:-./default}` pattern"
    Host paths use a variable with a local fallback. Locally, `docker compose up`
    needs nothing set — it uses `./logs`, `./config` (and `./mounts` if you enable
    it). On the fleet, the deploy layer sets the real values (below), so the same
    file works both places.

## How deploy fills it in

At deploy time we set the variables your compose reads, so identity and host paths
are injected — not baked into your file. You can rely on these being present:

```
COMPOSE_PROJECT_NAME=<workspace>-<name>
LOGS_DIR=/srv/logs/<workspace>/<name>
CONFIG_DIR=/srv/config/<workspace>/<name>
MOUNTS_DIR=/srv/mounts/<workspace>/<name>   # dir auto-created; mount line opt-in
```

Which makes ownership obvious and collisions impossible:

| item | value |
| --- | --- |
| container | `<workspace>-<name>-svc-1` |
| volume | `<workspace>-<name>_cache` |
| network | `<workspace>-<name>_default` |
| logs | `/srv/logs/<workspace>/<name>/` |

!!! warning "What `<workspace>` and `<name>` are"
    - **`<workspace>`** — who owns it: an individual dev (`ricky`, `xattam`, …) or a
      category (`tpv`, `web`, `bots`, …).
    - **`<name>`** — the git repo name, lowercased, by default. Overridable at
      deploy time.

Host paths live under `/srv/<kind>/<workspace>/<name>/` (config, logs, and mounts if
enabled), all owned by the `services` user (uid/gid **1337**). You never write these
paths in your compose — you read the injected `${...}` variables.

## Onboarding a service

One command registers and starts a service:

```
deploy <host> <workspace> <git-url> [name]
```

It registers the service, generates a per-repo **read-only deploy key on the host**
(the private key never leaves the box), pushes the public key to the repo's Gitea
deploy keys automatically, clones the code, and installs and starts the service's
unit. No manual key paste, no host setup on your end.

!!! note "If your service won't start or its logs aren't persisting"
    That's usually a host-side bind-mount **ownership** thing — the kind of detail
    **we sort out at deploy time**, not something you need to chown or provision.
    If a bot won't come up or logs/caches keep vanishing, flag it and we'll fix it.
    Stick to the generic `compose.yaml` above and let us handle the host.

## Layer caching

Copy the deps file and install **before** `COPY . .`. Docker caches layers in
order, so deps only reinstall when the deps file changes — not on every code edit.
Get this backwards and every one-line change triggers a full dependency reinstall.
Same principle whether you use pip or uv:

```dockerfile
COPY requirements.txt .                              # pip baseline
RUN pip install --no-cache-dir -r requirements.txt   # cached until deps change
COPY . .                                             # changes every build
```

```dockerfile
COPY pyproject.toml .                                # uv path
RUN uv pip install --system .                        # cached until deps change
COPY . .                                             # changes every build
```

## Subprocess and browser workloads

Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra
knobs in compose:

```yaml
services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # (1)!
    shm_size: "2gb"         # (2)!
    mem_limit: "4g"         # (3)!
```

1.  Runs **tini** as PID 1 to reap zombie subprocesses and forward signals.
    Without it, spawned Chrome/Xvfb processes leak as zombies.
2.  Chrome and most headless browsers crash on Docker's default **64 MB**
    `/dev/shm`. Bump it for any browser workload.
3.  Bound memory — especially when each worker spawns a browser. Raise it as
    worker count grows.

!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
    If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
    be PID 1, or the real process dies on startup. `init: true` is exactly what
    fixes this — tini takes PID 1, and your wrapper runs as a normal child.

## What your compose / Dockerfile needs

**Compose**

- service key `svc` — **no `container_name`**, no repo-specific names
- `user: "1337:1337"` — the shared `services` account
- `restart: unless-stopped`
- host paths via `${LOGS_DIR}` / `${CONFIG_DIR}` (and optional `${MOUNTS_DIR}`) —
  never hardcoded
- named volumes with **bare names** (`cache`, not `myapp_cache`)
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`

**Dockerfile**

- `HOME=/tmp`
- `chmod -R a+rwX /app` (uid-agnostic; runs as the `services` account, 1337)
- deps installed **before** the code copy (layer caching) — pip or `uv pip install`
- `git` in the image **if the container needs it**
- using uv? add `ENV UV_COMPILE_BYTECODE=1`

## Secrets

!!! warning "Secrets never go in the image"
    We do **not** commit secrets (usually, lol). They stay **gitignored** and live
    on the host under the config dir (`/srv/config/<workspace>/<name>/`), reaching
    your container read-only via `${CONFIG_DIR}`. Add them to `.dockerignore` so a
    `COPY . .` can't sweep them into a layer.

Rotating a secret is a host-side edit — update the file and the service picks it up
on restart. No rebuild, and nothing you run: flag it and we handle the restart.