rethink-public/handbook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

split Deploy into a hub + 3 sub-pages; inline comments over annotations

Browse Source 
The Deploy page got long and the code annotations rendered inconsistently
(some as (n) numbers, some as clickable +, and a marker on a fully-commented
line made that line vanish). Fix both:

- Deploy is now a HUB (docs/deploy/index.md): the intro + collision rule +
  central-deploy note, then card links to three focused sub-pages. Deploy
  stays the single global-nav entry; sub-pages are not_in_nav, reached from
  the hub cards.
  - deploy/compose.md — compose convention, storage tiers, how deploy fills
    it in, subprocess/browser knobs, checklist
  - deploy/dockerfile.md — services-account image, COPY, layer caching, uv
  - deploy/secrets.md — keeping secrets out of the image
- Replace code annotations with INLINE COMMENTS on the compose/Dockerfile
  examples: everything visible at once, no + to click, and the commented
  MOUNTS_DIR line no longer disappears.
- Update inbound links (index card, standards, workflow, environments) to
  deploy/ and deploy/compose.md; nav Deploy -> deploy/index.md with
  not_in_nav for the sub-pages.

Verified in-browser: hub cards link correctly, sub-pages render with visible
inline comments (0 annotation markers), left nav shows only Deploy;
mkdocs build --strict clean (validates not_in_nav + all cross-links).

Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
dsql
2026-07-01 02:32:16 -04:00
parent c335887dd6
commit f9e5e4f24d
10 changed files with 281 additions and 285 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploy.md
-280
View File
@@ -1,280 +0,0 @@
# Deployment Guide
> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.
!!! tip "What can run here"
APIs, websites, applets, bots, monitors.
!!! danger "Your compose names nothing repo-specific"
**No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
deploy layer injects identity and host paths at deploy time — your compose stays
generic so it can't collide with any other service on the fleet.
- service key is always **`svc`**
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**
Naming things after your repo used to cause **container-name collisions** at
fleet scale — two repos shipping the same name clashed. The generic compose
below fixes that; copy it verbatim.
## Docker — the services account
Every service runs containerized as the shared **`services`** account:
**uid/gid 1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it
runs cleanly as that account.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # (1)!
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # (2)!
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # (3)!
COPY . .
RUN chmod -R a+rwX /app # (4)!
CMD ["python", "-m", "yourapp"]
```
1. `HOME=/tmp` — the `services` account has no home dir; anything writing to
`$HOME` (caches, configs) needs a writable target.
2. Include `git` **only if the container itself needs it** — e.g. you
`pip install` from git, or the app shells out to git at runtime. The build and
host always have git; this line is about what's *inside* the image.
3. Install deps **before** copying the code (see [layer
caching](#layer-caching)).
4. `chmod -R a+rwX /app` makes the app tree writable by **any** uid — that's what
"uid-agnostic" means.
!!! tip "Faster builds with uv (optional)"
[uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
`pyproject.toml` — no lockfile needed in the image. Swap the deps layer and add
`UV_COMPILE_BYTECODE` so containers don't pay the first-import `.pyc` compile
cost:
```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # (1)!
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1 # (2)!
WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system . # (3)!
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```
1. Pull the `uv` binary from its published image — no pip-installing uv itself.
2. Compile bytecode at build time so the container doesn't eat the first-import
`.pyc` compile cost on every cold start.
3. `--system` installs into the image's Python (no venv needed — the container
*is* the isolation); reads `pyproject.toml`, no `uv.lock` required.
### Getting your files into the image
`COPY . .` grabs the whole repo, but be explicit about anything that needs its own
place — assets, templates, a config the app reads at runtime. `COPY <src> <dest>`:
`<src>` is relative to the build context (your repo), `<dest>` is a path in the
image.
=== "A single file"
```dockerfile
COPY ./config.toml /app/config.toml # (1)!
```
1. One file into a specific path. The app reads it at `/app/config.toml`.
=== "A directory"
```dockerfile
COPY ./assets /app/assets # (1)!
```
1. A whole tree. Trailing paths are dirs — `/app/assets/` mirrors `./assets/`.
=== "Deps first (layer caching)"
```dockerfile
COPY requirements.txt . # (1)!
RUN pip install --no-cache-dir -r requirements.txt
COPY . . # (2)!
```
1. Copy just the deps file first, install, **then** copy the code.
2. See [layer caching](#layer-caching) — this ordering is why.
!!! warning "Don't `COPY` secrets into the image"
Anything sensitive stays **out** of the image — no `COPY ./secrets.env`. Secrets
live on the host and are injected read-only at runtime (see [Secrets](#secrets)).
Add them to `.dockerignore` so a blanket `COPY . .` can't sweep them in.
## Your `compose.yaml`
Copy this verbatim. It names nothing repo-specific — the deploy layer fills in
identity and host paths.
```yaml
services:
svc: # (1)!
build: .
user: "1337:1337" # (2)!
restart: unless-stopped # (3)!
environment:
HOME: /tmp
volumes:
- ${LOGS_DIR:-./logs}:/app/logs # (4)!
- ${CONFIG_DIR:-./config}:/app/config # (5)!
# - ${MOUNTS_DIR:-./mounts}:/app/data # optional — see below
- cache:/app/cache # (6)!
volumes:
cache: # (7)!
```
1. The service key is always **`svc`** — never a repo-specific name, never a
`container_name`. The ops tooling keys off `svc`.
2. Run as the shared **`services`** account (**uid/gid 1337**, fixed fleet-wide).
**No** in-container `user`/`useradd` — set it here. This is the one identity
line that stays; it's not repo-specific.
3. **Required.** The host-side service is oneshot; Docker's own restart policy is
what recovers a crashed container.
4. Logs → host (**output**), path **injected** by the deploy layer. The `:-./logs`
default lets you `docker compose up` locally with nothing set and still work.
5. Config → host (**input you edit**), likewise injected.
6. Named volume mounted at **`/app/cache`** — ephemeral scratch. The **right-hand
path is where your code writes**: `WORKDIR` is `/app`, so if your app writes to
`./cache` (or `/app/cache`), that's this mount. Same rule for logs (`/app/logs`)
and config (`/app/config`) — match the container path to where your code reads
and writes. The volume name is a bare name, auto-namespaced per service at deploy
so it can't collide.
7. Declare bare (`cache`, not `myapp_cache`). Deploy auto-prefixes it.
The commented **`${MOUNTS_DIR}`** line is the optional third tier — a host dir for
arbitrary read/write data. The deploy layer injects `MOUNTS_DIR` and auto-creates the
dir, but the mount line is **opt-in**: uncomment it and pick the container path
(`/app/data` above) yourself, since that's app-specific and can't be auto-mounted.
!!! tip "Three tiers of storage"
- **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
injected and managed at deploy time.
- **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy
layer **injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is
**opt-in**: uncomment it and choose the container path yourself (it's
app-specific, so it can't be auto-mounted).
- **Named volumes** (`cache`, …) — **yours**: anything else your service
persists. Docker owns them, no host paths to manage.
!!! tip "The `${VAR:-./default}` pattern"
Host paths use a variable with a local fallback. Locally, `docker compose up`
needs nothing set — it uses `./logs`, `./config` (and `./mounts` if you enable
it). When deployed, the ops tooling sets the real values, so the same file works
both places.
## Deployment is handled centrally
!!! note "You don't run any deploy commands"
All services are deployed centrally by the ops tooling. You don't pick a host,
manage keys, or run anything — just make your repo follow the compose convention
above and it's deployable.
At deploy time the tooling **generates** the environment your compose reads, so the
`${...}` variables resolve without anything from you. You can rely on these being
set — that's why your compose works unchanged on the fleet:
```
LOGS_DIR # your host log dir -> ${LOGS_DIR}
CONFIG_DIR # your host config dir -> ${CONFIG_DIR}
MOUNTS_DIR # optional host data dir -> ${MOUNTS_DIR} (if you enable the mount)
```
Write your logs and config to those mount points and you're set. If a service won't
come up or its logs aren't persisting, that's a host-side detail on our end — flag it
and we'll sort it.
## Layer caching
Copy the deps file and install **before** `COPY . .`. Docker caches layers in
order, so deps only reinstall when the deps file changes — not on every code edit.
Get this backwards and every one-line change triggers a full dependency reinstall.
Same principle whether you use pip or uv:
```dockerfile
COPY requirements.txt . # pip baseline
RUN pip install --no-cache-dir -r requirements.txt # cached until deps change
COPY . . # changes every build
```
```dockerfile
COPY pyproject.toml . # uv path
RUN uv pip install --system . # cached until deps change
COPY . . # changes every build
```
## Subprocess and browser workloads
Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra
knobs in compose:
```yaml
services:
svc:
build: .
user: "1337:1337"
restart: unless-stopped
init: true # (1)!
shm_size: "2gb" # (2)!
mem_limit: "4g" # (3)!
```
1. Runs **tini** as PID 1 to reap zombie subprocesses and forward signals.
Without it, spawned Chrome/Xvfb processes leak as zombies.
2. Chrome and most headless browsers crash on Docker's default **64 MB**
`/dev/shm`. Bump it for any browser workload.
3. Bound memory — especially when each worker spawns a browser. Raise it as
worker count grows.
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
be PID 1, or the real process dies on startup. `init: true` is exactly what
fixes this — tini takes PID 1, and your wrapper runs as a normal child.
## What your compose / Dockerfile needs
**Compose**
- service key `svc` — **no `container_name`**, no repo-specific names
- `user: "1337:1337"` — the shared `services` account
- `restart: unless-stopped`
- host paths via `${LOGS_DIR}` / `${CONFIG_DIR}` (and optional `${MOUNTS_DIR}`) —
never hardcoded
- named volumes with **bare names** (`cache`, not `myapp_cache`)
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`
**Dockerfile**
- `HOME=/tmp`
- `chmod -R a+rwX /app` (uid-agnostic; runs as the `services` account, 1337)
- deps installed **before** the code copy (layer caching) — pip or `uv pip install`
- `git` in the image **if the container needs it**
- using uv? add `ENV UV_COMPILE_BYTECODE=1`
## Secrets
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored** and live
on the host in your config dir, reaching your container read-only via
`${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
into a layer.
Rotating a secret is a host-side edit — update the file and the service picks it up
on restart. No rebuild, and nothing you run: flag it and we handle the restart.
docs/deploy/compose.md
+106
View File
@@ -0,0 +1,106 @@
# Compose convention
The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.
```yaml
services:
svc: # generic service key — ALWAYS svc, no container_name
build: .
user: "1337:1337" # the shared services account (required)
restart: unless-stopped # required — the host unit is oneshot; this recovers crashes
environment:
HOME: /tmp
volumes:
- ${LOGS_DIR:-./logs}:/app/logs # output — host log dir, injected at deploy
- ${CONFIG_DIR:-./config}:/app/config # input — host config dir, injected at deploy
# - ${MOUNTS_DIR:-./mounts}:/app/data # optional — arbitrary host data (opt-in, see below)
- cache:/app/cache # your data — ephemeral named volume
volumes:
cache: # bare name — deploy auto-prefixes it per service
```
## The rules
- **Service key is always `svc`.** Never a repo-specific name, never a
`container_name` — the ops tooling keys off `svc`. This is what makes services
collision-proof on the fleet.
- **`user: "1337:1337"` and `restart: unless-stopped` are required.** The first runs
as the shared `services` account (uid/gid 1337, fixed fleet-wide); the second lets
Docker recover a crashed container (the host-side unit is oneshot).
- **Host paths come from `${...}` variables, never hardcoded.** Write to `${LOGS_DIR}`
and `${CONFIG_DIR}`; `${MOUNTS_DIR}` is optional.
- **Named volumes use bare names** (`cache`, not `myapp_cache`) — deploy auto-prefixes
them per service so they can't collide.
!!! tip "The container path is where your code reads and writes"
`WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
your code targets. If your app writes to `./cache` (i.e. `/app/cache`), that's the
`cache` mount; logs go to `/app/logs`, config is read from `/app/config`. Match
the container path to what your code actually uses.
## Three tiers of storage
- **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
injected and managed at deploy time.
- **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy layer
**injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is **opt-in**:
uncomment it and choose the container path yourself (it's app-specific, so it can't
be auto-mounted).
- **Named volumes** (`cache`, …) — **yours**: anything else your service persists.
Docker owns them, no host paths to manage.
!!! tip "The `${VAR:-./default}` pattern"
Host paths use a variable with a local fallback. Locally, `docker compose up` needs
nothing set — it uses `./logs`, `./config` (and `./mounts` if you enable it). When
deployed, the ops tooling sets the real values, so the same file works both places.
## How deploy fills it in
At deploy time the tooling **generates** the environment your compose reads, so the
`${...}` variables resolve without anything from you:
```
LOGS_DIR # your host log dir -> ${LOGS_DIR}
CONFIG_DIR # your host config dir -> ${CONFIG_DIR}
MOUNTS_DIR # optional host data dir -> ${MOUNTS_DIR} (if you enable the mount)
```
Write your logs and config to those mount points and you're set. If a service won't
come up or its logs aren't persisting, that's a host-side detail on our end — flag it
and we'll sort it.
## Subprocess and browser workloads
Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra knobs:
```yaml
services:
svc:
build: .
user: "1337:1337"
restart: unless-stopped
init: true # tini as PID 1 — reaps zombie subprocesses, forwards signals
shm_size: "2gb" # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
mem_limit: "4g" # bound memory — raise as worker/browser count grows
```
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not** be
PID 1, or the real process dies on startup. `init: true` is exactly what fixes this
— tini takes PID 1, and your wrapper runs as a normal child.
## Checklist
- service key `svc` — **no `container_name`**, no repo-specific names
- `user: "1337:1337"` — the shared `services` account
- `restart: unless-stopped`
- host paths via `${LOGS_DIR}` / `${CONFIG_DIR}` (and optional `${MOUNTS_DIR}`) — never
hardcoded
- named volumes with **bare names** (`cache`, not `myapp_cache`)
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`
See **[Dockerfile & build](dockerfile.md)** for the image, and **[Secrets](secrets.md)**
for keeping credentials out of it.
docs/deploy/dockerfile.md
+91
View File
@@ -0,0 +1,91 @@
# Dockerfile & build
Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it runs cleanly
as that account.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # services account has no home dir; $HOME must be writable
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # include git ONLY if the container itself needs it
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # deps before code — see layer caching
COPY . .
RUN chmod -R a+rwX /app # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```
- **`HOME=/tmp`** — the `services` account has no home dir; anything writing to `$HOME`
(caches, configs) needs a writable target.
- **`git`** — include it **only if the container itself needs it** (e.g. you
`pip install` from git, or the app shells out to git). The build and host always have
git; this is about what's *inside* the image.
- **`chmod -R a+rwX /app`** — makes the app tree writable by any uid, so it runs as
1337.
## Layer caching
Copy the deps file and install **before** `COPY . .`. Docker caches layers in order, so
deps only reinstall when the deps file changes — not on every code edit. Get this
backwards and every one-line change triggers a full dependency reinstall.
```dockerfile
COPY requirements.txt . # pip baseline
RUN pip install --no-cache-dir -r requirements.txt # cached until deps change
COPY . . # changes every build
```
## Getting your files into the image
`COPY . .` grabs the whole repo, but be explicit about anything that needs its own
place — assets, templates, a config the app reads at runtime. In `COPY <src> <dest>`,
`<src>` is relative to the build context (your repo) and `<dest>` is a path in the
image.
```dockerfile
COPY ./config.toml /app/config.toml # a single file into a specific path
COPY ./assets /app/assets # a whole directory (tree mirrors ./assets/)
```
!!! warning "Don't `COPY` secrets into the image"
Anything sensitive stays **out** of the image — no `COPY ./secrets.env`. Secrets
live on the host and are injected read-only at runtime (see
**[Secrets](secrets.md)**). Add them to `.dockerignore` so a blanket `COPY . .`
can't sweep them in.
## Faster builds with uv (optional)
[uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
`pyproject.toml` — no lockfile needed in the image. Swap the deps layer and add
`UV_COMPILE_BYTECODE` so containers don't pay the first-import `.pyc` compile cost:
```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # pull the uv binary from its image
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1 # compile bytecode at build, not first cold start
WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system . # --system: into the image's Python, no venv/lock
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```
## Checklist
- `HOME=/tmp`
- `chmod -R a+rwX /app` (uid-agnostic; runs as the `services` account, 1337)
- deps installed **before** the code copy (layer caching) — pip or `uv pip install`
- `git` in the image **if the container needs it**
- using uv? add `ENV UV_COMPILE_BYTECODE=1`
See **[Compose convention](compose.md)** for the `compose.yaml`, and
**[Secrets](secrets.md)** for credentials.
docs/deploy/index.md
+54
View File
@@ -0,0 +1,54 @@
# Deployment Guide
> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.
!!! tip "What can run here"
APIs, websites, applets, bots, monitors.
!!! danger "Your compose names nothing repo-specific"
**No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
deploy layer injects identity and host paths at deploy time — your compose stays
generic so it can't collide with any other service on the fleet.
- service key is always **`svc`**
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**
Naming things after your repo used to cause **container-name collisions** at
fleet scale — two repos shipping the same name clashed. The generic compose
convention fixes that.
!!! note "You don't run any deploy commands"
All services are deployed centrally by the ops tooling. You don't pick a host,
manage keys, or run anything — just make your repo follow the compose convention
and it's deployable. At deploy time the tooling **generates** the environment your
compose reads (`${LOGS_DIR}`, `${CONFIG_DIR}`, optional `${MOUNTS_DIR}`), so the
same file works locally and on the fleet.
## The three things your repo needs
<div class="grid cards" markdown>
- :material-file-cog: __[Compose convention](compose.md)__
---
The `compose.yaml` your repo ships — generic `svc` service, the `${...}` host
mounts, storage tiers, and how deploy fills it in.
- :material-docker: __[Dockerfile & build](dockerfile.md)__
---
The uid-agnostic image (services account 1337), getting files in with `COPY`,
layer caching, uv, and subprocess/browser workloads.
- :material-key: __[Secrets](secrets.md)__
---
How secrets stay out of the image and reach the container read-only at runtime.
</div>
docs/deploy/secrets.md
+19
View File
@@ -0,0 +1,19 @@
# Secrets
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored** and live on
the host in your config dir, reaching your container read-only via `${CONFIG_DIR}`.
Add them to `.dockerignore` so a `COPY . .` can't sweep them into a layer.
## How it works
- Secrets live on the **host**, in your config dir — never in git, never in the image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount (see
**[Compose convention](compose.md)**).
- Keep them out of the build context: list them in `.dockerignore` so a blanket
`COPY . .` can't pull them into a layer.
## Rotating a secret
A host-side edit — update the file and the service picks it up on restart. No rebuild,
and nothing you run: flag it and we handle the restart.
docs/environments.md
+1 -1
View File
@@ -67,7 +67,7 @@ happens, depending on where the project runs:
COPY . .
```
This is how things run in production — see the [Deploy guide](deploy.md) for the
This is how things run in production — see the [Deploy guide](deploy/) for the
full container standard (uid 1337, the compose convention, layer caching, and the
uv image setup).
docs/index.md
+1 -1
View File
@@ -66,7 +66,7 @@ out of it; examples use placeholders like `<workspace>`, `<project>`, and
Project-based Python isolation — local `.venv`, Makefile, or Docker — and
local version management with pyenv.
- :material-rocket-launch: __[Deploy](deploy.md)__
- :material-rocket-launch: __[Deploy](deploy/)__
---
docs/standards.md
+1 -1
View File
@@ -134,7 +134,7 @@ TimeoutError: request timed out after 30s
A deployable service ships a `compose.yaml` that names **nothing repo-specific** —
the deploy layer injects identity and host paths. See the
[Deploy guide](deploy.md#your-composeyaml) for the full convention and the variables
[Deploy guide](deploy/compose.md) for the full convention and the variables
you can rely on. The short version:
=== "Right"
docs/workflow.md
+1 -1
View File
@@ -44,7 +44,7 @@ Our code lives on **Gitea** at
2. Add your **SSH public key** under *Settings → SSH / GPG Keys* so you can clone
and push over SSH.
3. For servers, we use a **per-repo deploy-key** model rather than your personal
key — see the [Deploy guide](deploy.md) for how a box gets read access to just
key — see the [Deploy guide](deploy/) for how a box gets read access to just
the repos it needs.
## Our git vs. public git (GitHub / GitLab)