handbook/docs/deploy.md

Deployment Guide

How a project gets onto rethink-net — our Ubuntu 26.x servers. Get your container to follow a few consistent rules and deploying is mostly handing us a compose.yaml.

!!! success "What can run here" APIs, websites, applets, bots, monitors.

Preparing for Deploy

You build and test locally — leaning on our libraries, AI, and this handbook — then push to git. From there the ops tooling takes over: staging (if you run gitflow), then it builds the Docker image and runs it on the fleet. You don't run any deploy commands — your only job is the three pieces in the tabs below.

flowchart LR
    subgraph build ["you build"]
        direction TB
        local["local testing"]
        libs["libs<br/>(rethink-public, pip)"]
        ai["AI-assisted<br/>(Claude Code)"]
        docs["reading the<br/>handbook"]
        local --- libs
        libs --- ai
        ai --- docs
        docs --- local
    end

    build -->|push| git["git<br/>(Gitea)"]
    git -.->|develop, if gitflow| stage["staging"]
    git --> deploy["deployment<br/>(ops tooling)"]
    stage --> deploy
    deploy --> docker["docker<br/>(built + run<br/>on the fleet)"]

    classDef ship fill:#061541,stroke:#569bcc,color:#eef1f6;
    classDef work fill:#0e1530,stroke:#294274,color:#eef1f6;
    class git,stage,deploy,docker ship;
    class local,libs,ai,docs work;

!!! danger "Your compose names nothing repo-specific" No container_name, no hardcoded names/paths. The deploy layer injects identity and host paths so your compose can't collide with any other service on the fleet:

- the service key is **always `svc`** — it never changes, in any repo
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**

=== "Compose"

The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.

```yaml
services:
  svc:                                     # generic service key — ALWAYS svc, no container_name
    build: .
    user: "1337:1337"                      # the shared services account (required)
    restart: unless-stopped                # required — host unit is oneshot; this recovers crashes
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # output   — host log dir, injected at deploy
      - ${CONFIG_DIR:-./config}:/app/config  # input     — host config dir, injected at deploy
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # optional  — arbitrary host data (opt-in)
      - cache:/app/cache                     # your data — ephemeral named volume

volumes:
  cache:                                   # bare name — deploy auto-prefixes it per service
```

!!! example "The container path is where your code reads and writes"
    `WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
    your code targets. Write to `./cache` (i.e. `/app/cache`) → the `cache` volume;
    logs go to `/app/logs`, config is read from `/app/config`.

**Storage, three tiers:**

- **`logs` + `config`** — we inject and manage these at deploy time.
- **`mounts`** (optional) — host dir for arbitrary data. We inject `MOUNTS_DIR` and
  create the dir; you **uncomment** the line and pick the container path.
- **Named volumes** (`cache`, …) — yours; Docker owns them, no host paths to manage.

!!! warning "Two kinds of logs — and crashes go to the other one"
    `${LOGS_DIR}` holds **only** what your app writes to disk through its logger
    (e.g. `log_setup` writing a file) — your own structured logging. It does **not**
    capture the process's stdout/stderr, and that's where **startup crashes and
    uncaught exceptions land** — a traceback from a failed import or a missing file
    never reaches your logger. So if a service dies on startup, or you don't see the
    error in your log files, it's in the process output, not `${LOGS_DIR}`. Make
    fatal errors visible — and to route uncaught exceptions into your log file too,
    install a top-level hook:

    ```python
    import sys, logging
    sys.excepthook = lambda *exc: logging.getLogger().critical("uncaught", exc_info=exc)
    ```

!!! success "Same file, both places"
    Locally, `docker compose up` needs nothing set — the `${VAR:-./default}`
    fallbacks use `./logs` / `./config`. When deployed, the ops tooling sets the
    real values. One `compose.yaml` works everywhere.

**Subprocess and browser workloads** (bots that spawn Chrome, Xvfb, ffmpeg) need
three extra knobs:

```yaml
services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # tini as PID 1 — reaps zombie subprocesses, forwards signals
    shm_size: "2gb"         # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
    mem_limit: "4g"         # bound memory — raise as worker/browser count grows
```

!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
    If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
    be PID 1, or the real process dies on startup. `init: true` fixes this — tini
    takes PID 1, your wrapper runs as a normal child.

=== "Dockerfile"

Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build the image to be **uid-agnostic** so it runs cleanly
as that account.

```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp                                  # services account has no home dir; $HOME must be writable

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*             # include git ONLY if the container itself needs it

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # deps before code — layer caching

COPY . .
RUN chmod -R a+rwX /app                         # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```

**Layer caching:** copy the deps file and install **before** `COPY . .` — Docker
caches layers in order, so deps only reinstall when the deps file changes, not on
every code edit.

**Getting files in:** `COPY . .` grabs the whole repo; be explicit about anything
that needs its own place. In `COPY <src> <dest>`, `<src>` is relative to the build
context (your repo), `<dest>` is a path in the image.

```dockerfile
COPY ./config.toml /app/config.toml   # a single file into a specific path
COPY ./assets /app/assets             # a whole directory (tree mirrors ./assets/)
```

!!! tip "Faster builds with uv (optional)"
    [uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
    `pyproject.toml` — no lockfile needed in the image. Add `UV_COMPILE_BYTECODE`
    so containers don't pay the first-import `.pyc` compile cost:

    ```dockerfile
    FROM python:3.12-slim
    COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # pull the uv binary from its image
    ENV HOME=/tmp
    ENV UV_COMPILE_BYTECODE=1                                # compile bytecode at build, not cold start

    WORKDIR /app
    COPY pyproject.toml .
    RUN uv pip install --system .                            # into the image's Python, no venv/lock
    COPY . .
    RUN chmod -R a+rwX /app
    CMD ["python", "-m", "yourapp"]
    ```

=== "Secrets"

!!! warning "Secrets never go in the image"
    We do **not** commit secrets (usually, lol). They stay **gitignored** and live
    on the host in your config dir, reaching your container read-only via
    `${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
    into a layer.

- Secrets live on the **host**, in your config dir — never in git, never in the
  image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount.
- Keep them out of the build context: list them in `.dockerignore` so a blanket
  `COPY . .` can't pull them into a layer.

**Rotating a secret** is a host-side edit — update the file and the service picks
it up on restart. No rebuild, and nothing you run: flag it and we handle the
restart.