handbook/docs/deploy.md

Deployment Guide

How a project gets onto rethink-net — our Ubuntu 26.x servers. Get your container to follow a few consistent rules and deploying is mostly handing us a compose.yaml.

!!! success "What can run here" APIs, websites, applets, bots, monitors.

Preparing for Deploy

You build and test locally — leaning on our libraries, AI, and this handbook — then push to git. From there the ops tooling takes over: staging (if you run gitflow), then it builds the Docker image and runs it on the fleet. You don't run any deploy commands — your only job is the three pieces in the tabs below.

flowchart LR
    subgraph build ["you build"]
        direction TB
        local["local testing"]
        libs["libs<br/>(rethink-public, pip)"]
        ai["AI-assisted<br/>(Claude Code)"]
        docs["reading the<br/>handbook"]
        local --- libs
        libs --- ai
        ai --- docs
        docs --- local
    end

    build -->|push| git["git<br/>(Gitea)"]
    git -.->|develop, if gitflow| stage["staging"]
    git --> deploy["deployment<br/>(ops tooling)"]
    stage --> deploy
    deploy --> docker["docker<br/>(built + run<br/>on the fleet)"]

    classDef ship fill:#061541,stroke:#569bcc,color:#eef1f6;
    classDef work fill:#0e1530,stroke:#294274,color:#eef1f6;
    class git,stage,deploy,docker ship;
    class local,libs,ai,docs work;

!!! danger "Your compose names nothing repo-specific" No container_name, no hardcoded names/paths. The deploy layer injects identity and host paths so your compose can't collide with any other service on the fleet:

- the service key is **always `svc`** — it never changes, in any repo
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**

=== "Compose"

The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.

```yaml
services:
  svc:                                     # generic service key — ALWAYS svc, no container_name
    build: .
    user: "1337:1337"                      # the shared services account (required)
    restart: unless-stopped                # required — host unit is oneshot; this recovers crashes
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # output   — host log dir, injected at deploy
      - ${CONFIG_DIR:-./config}:/app/config  # input     — host config dir, injected at deploy
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # optional  — arbitrary host data (opt-in)
      - cache:/app/cache                     # your data — ephemeral named volume

volumes:
  cache:                                   # bare name — deploy auto-prefixes it per service
```

!!! example "The container path is where your code reads and writes"
    `WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
    your code targets. Write to `./cache` (i.e. `/app/cache`) → the `cache` volume;
    logs go to `/app/logs`, config is read from `/app/config`.

**Storage, three tiers:**

- **`logs` + `config`** — we inject and manage these at deploy time.
- **`mounts`** (optional) — host dir for arbitrary data. We inject `MOUNTS_DIR` and
  create the dir; you **uncomment** the line and pick the container path.
- **Named volumes** (`cache`, …) — yours; Docker owns them, no host paths to manage.

!!! warning "Two kinds of logs — and crashes go to the other one"
    `${LOGS_DIR}` holds **only** what your app writes to disk through its logger
    (e.g. `log_setup` writing a file) — your own structured logging. It does **not**
    capture the process's stdout/stderr, and that's where **startup crashes and
    uncaught exceptions land** — a traceback from a failed import or a missing file
    never reaches your logger. So if a service dies on startup, or you don't see the
    error in your log files, it's in the process output, not `${LOGS_DIR}`. Make
    fatal errors visible — and to route uncaught exceptions into your log file too,
    install a top-level hook:

    ```python
    import sys, logging
    sys.excepthook = lambda *exc: logging.getLogger().critical("uncaught", exc_info=exc)
    ```

!!! success "Same file, both places"
    Locally, `docker compose up` needs nothing set — the `${VAR:-./default}`
    fallbacks use `./logs` / `./config`. When deployed, the ops tooling sets the
    real values. One `compose.yaml` works everywhere.

**Subprocess and browser workloads** (bots that spawn Chrome, Xvfb, ffmpeg) need
three extra knobs:

```yaml
services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # tini as PID 1 — reaps zombie subprocesses, forwards signals
    shm_size: "2gb"         # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
    mem_limit: "4g"         # bound memory — raise as worker/browser count grows
```

!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
    If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
    be PID 1, or the real process dies on startup. `init: true` fixes this — tini
    takes PID 1, your wrapper runs as a normal child.

=== "Dockerfile"

Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build the image to be **uid-agnostic** so it runs cleanly
as that account.

```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp                                  # services account has no home dir; $HOME must be writable

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*             # include git ONLY if the container itself needs it

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # deps before code — layer caching

COPY . .
RUN chmod -R a+rwX /app                         # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```

**Layer caching:** copy the deps file and install **before** `COPY . .` — Docker
caches layers in order, so deps only reinstall when the deps file changes, not on
every code edit.

**Getting files in:** `COPY . .` grabs the whole repo; be explicit about anything
that needs its own place. In `COPY <src> <dest>`, `<src>` is relative to the build
context (your repo), `<dest>` is a path in the image.

```dockerfile
COPY ./config.toml /app/config.toml   # a single file into a specific path
COPY ./assets /app/assets             # a whole directory (tree mirrors ./assets/)
```

!!! tip "Faster builds with uv (optional)"
    [uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
    `pyproject.toml` — no lockfile needed in the image. Add `UV_COMPILE_BYTECODE`
    so containers don't pay the first-import `.pyc` compile cost:

    ```dockerfile
    FROM python:3.12-slim
    COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # pull the uv binary from its image
    ENV HOME=/tmp
    ENV UV_COMPILE_BYTECODE=1                                # compile bytecode at build, not cold start

    WORKDIR /app
    COPY pyproject.toml .
    RUN uv pip install --system .                            # into the image's Python, no venv/lock
    COPY . .
    RUN chmod -R a+rwX /app
    CMD ["python", "-m", "yourapp"]
    ```

=== "Secrets"

!!! warning "Secrets never go in the image"
    We do **not** commit secrets (usually, lol). They stay **gitignored** and live
    on the host in your config dir, reaching your container read-only via
    `${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
    into a layer.

- Secrets live on the **host**, in your config dir — never in git, never in the
  image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount.
- Keep them out of the build context: list them in `.dockerignore` so a blanket
  `COPY . .` can't pull them into a layer.

**Rotating a secret** is a host-side edit — update the file and the service picks
it up on restart. No rebuild, and nothing you run: flag it and we handle the
restart.

Supporting services live in your compose, not the fleet

Need Redis? Declare it in your own compose.yaml, as a sidecar. There is no shared Redis — nothing fleet-wide, nothing per-workspace, nothing ops provisions for you. A repo that needs Redis brings its own; a repo that doesn't adds nothing.

That's the whole point of the deploy model. Every service already runs in its own compose project on its own network so that one service falling over can't touch another. A shared Redis puts that coupling straight back: one process everything depends on, whose OOM, stray FLUSHALL, single-threaded stall, or restart becomes everyone's outage. A sidecar shares its owning repo's fate and nobody else's — and the isolation is free, because it rides the per-project network you already get. No ACLs, no key-prefix discipline, no shared credentials to manage.

Your app talks to it with the redis lib from the suite (async, config-free, kv/hash/ttl/pubsub), pointed at redis://redis:6379 — the compose service name, not a host port. The sidecar comes up auto-namespaced on your project's network like every other container, exactly as the naming convention above describes.

Ephemeral or persistent — pick deliberately

A sidecar Redis is ephemeral by default: restart it and the data is gone. That's correct for some workloads and quietly destructive for others, so make the call on purpose. Ask one question — if this data vanished on a restart, would anything be lost?

• No → ephemeral. A scratch cache, a dedupe set, rate-limit counters, transient data you can just re-fetch. Nothing to back up, nothing to grow.
• Yes → persistent. An outbound webhook or notification queue, a job queue, anything that could be mid-flight when the process dies. Losing it drops real work.

=== "Ephemeral (cache / throwaway)"

Fine to lose on restart — no volume, no persistence, by design.

```yaml
redis:
  image: redis:7-alpine
  restart: unless-stopped
  command: redis-server --save "" --maxmemory 256mb --maxmemory-policy allkeys-lru
  # no volume: throwaway by design
```

=== "Persistent (durable queue / state)"

Survives restart, rebuild, and reboot — the append-only file lives on the host mounts
dir injected at deploy, the same `${MOUNTS_DIR}` mechanism described above. Redis just
uses it as its backing store.

```yaml
redis:
  image: redis:7-alpine
  restart: unless-stopped
  command: redis-server --appendonly yes --appendfsync everysec
  volumes:
    - ${MOUNTS_DIR:-./mounts}/redis:/data   # AOF persists on the host mounts dir
```

The app connects the same way in both modes — REDIS_URL: redis://redis:6379. Only the durability changes.

!!! warning "Two things to know before you rely on a persistent sidecar" --appendfsync everysec can lose ~1 second of the newest entries on a hard crash. For a webhook queue that's an acceptable trade — just know it's there. Use --appendfsync always if you genuinely cannot drop a single entry (safer, slower).

**Data under the mounts dir is not backed up.** Mounts are excluded from the backup
pipeline, and that's *right* for a queue: a lost queue means some notifications didn't
fire, not that business data is gone. **That's the dividing line.** If losing this data
would actually hurt, it isn't queue or cache state — it's a system of record, and it
belongs in Postgres (which *is* backed up), not a local mount.

!!! tip "Sometimes you don't need Redis at all" For the dead-simple case — one process draining its own queue — a SQLite file under the mounts dir does the job with no extra container: atomic commits, crash-safe, zero infra. Reach for Redis when it's actually doing queue things (multiple workers, blocking pops, streams / consumer groups, pub/sub) or when it's already in your compose for something else.

The rules

!!! danger "Don't do these" - Don't map Redis to a host port. No ports: - "6379:6379". Two repos both grabbing host 6379 on the same box collide. Keep it internal to the compose network — nothing exposed, nothing to collide. - Don't stand up a shared or fleet-wide Redis. Per-repo means per-need. One Redis per project, shared by that project's containers if a repo runs several — never one per fleet. - Don't treat persistent Redis as a database. Queues and caches, yes. A durable system of record, no — that's Postgres, and unlike a mount it's backed up.

You can't reach another repo's Redis — different project, different network. That's not a restriction you have to work around; it's the isolation working for you. Nobody else's service can touch your cache or drain your queue either, and you never have to think about whose keys are whose.

!!! quote "What about ACLs?" A shared Redis can be secured — Redis 6+ ACLs scope users by command, key pattern, and channel. But ACLs don't solve resource contention or the noisy-neighbour problem, and they add real management burden, so the fleet uses per-repo sidecars instead. Reserve ACLs for the rare case of a deliberately shared, durable, backed-up Redis run as actual infrastructure.