handbook/docs/deploy.md

Deployment Guide

How a project gets onto rethink-net — our Ubuntu 26.x servers. Get your container to follow a few consistent rules and deploying is mostly handing us a compose.yaml.

!!! tip "What can run here" APIs, websites, applets, bots, monitors.

!!! danger "Your compose names nothing repo-specific" No container_name. No hardcoded names, workspace, or /srv paths. The deploy layer injects identity and host paths at deploy time — your compose stays generic so it can't collide with any other service on the fleet.

- service key is always **`svc`**
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**

Naming things after your repo used to cause **container-name collisions** at
fleet scale — two repos shipping the same name clashed. The generic compose
below fixes that; copy it verbatim.

Docker — the services account

Every service runs containerized as the shared services account: uid/gid 1337, fixed fleet-wide. Build your image to be uid-agnostic so it runs cleanly as that account.

FROM python:3.12-slim
ENV HOME=/tmp                                  # (1)!

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*            # (2)!

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # (3)!

COPY . .
RUN chmod -R a+rwX /app                        # (4)!
CMD ["python", "-m", "yourapp"]

1. HOME=/tmp — the services account has no home dir; anything writing to $HOME (caches, configs) needs a writable target.
2. Include git only if the container itself needs it — e.g. you pip install from git, or the app shells out to git at runtime. The build and host always have git; this line is about what's inside the image.
3. Install deps before copying the code (see layer caching).
4. chmod -R a+rwX /app makes the app tree writable by any uid — that's what "uid-agnostic" means.

!!! tip "Faster builds with uv (optional)" uv is a drop-in for pip that reads the same pyproject.toml — no lockfile needed in the image. Swap the deps layer and add UV_COMPILE_BYTECODE so containers don't pay the first-import .pyc compile cost:

```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # (1)!
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1                              # (2)!

WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system .                          # (3)!
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```

1.  Pull the `uv` binary from its published image — no pip-installing uv itself.
2.  Compile bytecode at build time so the container doesn't eat the first-import
    `.pyc` compile cost on every cold start.
3.  `--system` installs into the image's Python (no venv needed — the container
    *is* the isolation); reads `pyproject.toml`, no `uv.lock` required.

Getting your files into the image

COPY . . grabs the whole repo, but be explicit about anything that needs its own place — assets, templates, a config the app reads at runtime. COPY <src> <dest>: <src> is relative to the build context (your repo), <dest> is a path in the image.

=== "A single file"

```dockerfile
COPY ./config.toml /app/config.toml   # (1)!
```

1.  One file into a specific path. The app reads it at `/app/config.toml`.

=== "A directory"

```dockerfile
COPY ./assets /app/assets   # (1)!
```

1.  A whole tree. Trailing paths are dirs — `/app/assets/` mirrors `./assets/`.

=== "Deps first (layer caching)"

```dockerfile
COPY requirements.txt .     # (1)!
RUN pip install --no-cache-dir -r requirements.txt
COPY . .                    # (2)!
```

1.  Copy just the deps file first, install, **then** copy the code.
2.  See [layer caching](#layer-caching) — this ordering is why.

!!! warning "Don't COPY secrets into the image" Anything sensitive stays out of the image — no COPY ./secrets.env. Secrets live on the host and are injected read-only at runtime (see Secrets). Add them to .dockerignore so a blanket COPY . . can't sweep them in.

Your compose.yaml

Copy this verbatim. It names nothing repo-specific — the deploy layer fills in identity and host paths.

services:
  svc:                                       # (1)!
    build: .
    user: "1337:1337"                        # (2)!
    restart: unless-stopped                  # (3)!
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # (4)!
      - ${CONFIG_DIR:-./config}:/app/config  # (5)!
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # (6)!
      - cache:/app/cache                     # (7)!

volumes:
  cache:                                     # (8)!

1. The service key is always svc — never a repo-specific name, never a container_name. Our tooling keys off svc (the svc command, the svc-<workspace>-<name> unit).
2. Run as the shared services account (uid/gid 1337, fixed fleet-wide). No in-container user/useradd — set it here. This is the one identity line that stays; it's not repo-specific.
3. Required. The host-side service is oneshot; Docker's own restart policy is what recovers a crashed container.
4. Logs → host (output), path injected by the deploy layer. The :-./logs default lets you docker compose up locally with nothing set and still work.
5. Config → host (input you edit), likewise injected.
6. Optional, commented by default. A host dir for arbitrary read/write data. The deploy layer injects MOUNTS_DIR and auto-creates the dir — but you uncomment the line and pick the container path (/app/data here), since that's app-specific and can't be auto-mounted.
7. Named volume with a bare name — ephemeral, auto-namespaced per service at deploy time so it can't collide.
8. Declare bare (cache, not myapp_cache). Deploy auto-prefixes it.

!!! tip "Three tiers of storage" - logs (output) and config (input you edit) — we handle these: injected and managed at deploy time. - mounts (optional) — a host dir for arbitrary read/write data. The deploy layer injects MOUNTS_DIR and auto-creates the dir, but the mount line is opt-in: uncomment it and choose the container path yourself (it's app-specific, so it can't be auto-mounted). - Named volumes (cache, …) — yours: anything else your service persists. Docker owns them, no host paths to manage.

!!! tip "The ${VAR:-./default} pattern" Host paths use a variable with a local fallback. Locally, docker compose up needs nothing set — it uses ./logs, ./config (and ./mounts if you enable it). On the fleet, the deploy layer sets the real values (below), so the same file works both places.

How deploy fills it in

At deploy time we set the variables your compose reads, so identity and host paths are injected — not baked into your file. You can rely on these being present:

COMPOSE_PROJECT_NAME=<workspace>-<name>
LOGS_DIR=/srv/logs/<workspace>/<name>
CONFIG_DIR=/srv/config/<workspace>/<name>
MOUNTS_DIR=/srv/mounts/<workspace>/<name>   # dir auto-created; mount line opt-in

Which makes ownership obvious and collisions impossible:

item	value
container	<workspace>-<name>-svc-1
volume	<workspace>-<name>_cache
network	<workspace>-<name>_default
logs	/srv/logs/<workspace>/<name>/

!!! warning "What <workspace> and <name> are" - <workspace> — who owns it: an individual dev (ricky, xattam, …) or a category (tpv, web, bots, …). - <name> — the git repo name, lowercased, by default. Overridable at deploy time.

Host paths live under /srv/<kind>/<workspace>/<name>/ (config, logs, and mounts if enabled), all owned by the services user (uid/gid 1337). You never write these paths in your compose — you read the injected ${...} variables.

Onboarding a service

One command registers and starts a service:

deploy <host> <workspace> <git-url> [name]

It registers the service, generates a per-repo read-only deploy key on the host (the private key never leaves the box), pushes the public key to the repo's Gitea deploy keys automatically, clones the code, and installs and starts the service's unit. No manual key paste, no host setup on your end.

!!! note "If your service won't start or its logs aren't persisting" That's usually a host-side bind-mount ownership thing — the kind of detail we sort out at deploy time, not something you need to chown or provision. If a bot won't come up or logs/caches keep vanishing, flag it and we'll fix it. Stick to the generic compose.yaml above and let us handle the host.

Layer caching

Copy the deps file and install before COPY . .. Docker caches layers in order, so deps only reinstall when the deps file changes — not on every code edit. Get this backwards and every one-line change triggers a full dependency reinstall. Same principle whether you use pip or uv:

COPY requirements.txt .                              # pip baseline
RUN pip install --no-cache-dir -r requirements.txt   # cached until deps change
COPY . .                                             # changes every build

COPY pyproject.toml .                                # uv path
RUN uv pip install --system .                        # cached until deps change
COPY . .                                             # changes every build

Subprocess and browser workloads

Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra knobs in compose:

services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # (1)!
    shm_size: "2gb"         # (2)!
    mem_limit: "4g"         # (3)!

1. Runs tini as PID 1 to reap zombie subprocesses and forward signals. Without it, spawned Chrome/Xvfb processes leak as zombies.
2. Chrome and most headless browsers crash on Docker's default 64 MB /dev/shm. Bump it for any browser workload.
3. Bound memory — especially when each worker spawns a browser. Raise it as worker count grows.

!!! warning "The PID-1 gotcha with shell-wrapper CMDs" If your CMD is a shell-script wrapper (e.g. xvfb-run ...), it must not be PID 1, or the real process dies on startup. init: true is exactly what fixes this — tini takes PID 1, and your wrapper runs as a normal child.

What your compose / Dockerfile needs

Compose

• service key svc — no container_name, no repo-specific names
• user: "1337:1337" — the shared services account
• restart: unless-stopped
• host paths via ${LOGS_DIR} / ${CONFIG_DIR} (and optional ${MOUNTS_DIR}) — never hardcoded
• named volumes with bare names (cache, not myapp_cache)
• for browser/subprocess workloads: init: true, shm_size, mem_limit

Dockerfile

• HOME=/tmp
• chmod -R a+rwX /app (uid-agnostic; runs as the services account, 1337)
• deps installed before the code copy (layer caching) — pip or uv pip install
• git in the image if the container needs it
• using uv? add ENV UV_COMPILE_BYTECODE=1

Secrets

!!! warning "Secrets never go in the image" We do not commit secrets (usually, lol). They stay gitignored and live on the host under the config dir (/srv/config/<workspace>/<name>/), reaching your container read-only via ${CONFIG_DIR}. Add them to .dockerignore so a COPY . . can't sweep them into a layer.

Rotating a secret is a host-side edit — update the file and the service picks it up on restart. No rebuild, and nothing you run: flag it and we handle the restart.