rethink-public/handbook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

deploy: mermaid flow diagram, single page with tabs, section rework

Browse Source 
Collapse the deploy sub-pages back into one deploy.md (tabs, not separate
pages), add a working mermaid flow diagram, and rework the top per feedback.

- Mermaid: enable via superfences fence_div_format (the fence_code_format
  <pre><code> wrapper broke mermaid's render — div format fixes it); vendor
  mermaid.min.js locally (no CDN dependency for a self-hosted site) + a small
  init that renders each block once and survives instant-nav.
- Content tabs Compose / Dockerfile / Secrets hold the three pieces together
  on one page (inline comments in the code, no annotation +).
- 'What can run here' tip moved above the flow; section renamed
  'Preparing for Deploy' with the 'you don't run deploy commands' point folded
  sparsely into the intro (standalone note removed); 'svc never changes'
  called out.
- Flow diagram (below the tabs): build group now includes our libraries; git
  -> staging shown as develop-if-gitflow (dotted); deployment (no MR-to-main)
  -> docker image built + run on the fleet.
- Revert inbound links to deploy.md.

Verified in-browser: diagram renders with all nodes, tabs switch, section
order correct; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
dsql
2026-07-01 02:53:02 -04:00
parent f9e5e4f24d
commit 823aa7708a
12 changed files with 3837 additions and 282 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploy.md
+207
View File
@@ -0,0 +1,207 @@
# Deployment Guide
> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.
!!! tip "What can run here"
APIs, websites, applets, bots, monitors.
## Preparing for Deploy
Get these three right and your repo is deployable — you don't run any deploy commands
yourself, the ops tooling takes it from git. Each tab is one piece.
!!! danger "Your compose names nothing repo-specific"
**No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
deploy layer injects identity and host paths at deploy time — your compose stays
generic so it can't collide with any other service on the fleet.
- the service key is **always `svc`** — it never changes, in any repo
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**
Naming things after your repo used to cause **container-name collisions** at
fleet scale — two repos shipping the same name clashed. The generic compose in
the **Compose** tab fixes that.
=== "Compose"
The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.
```yaml
services:
svc: # generic service key — ALWAYS svc, no container_name
build: .
user: "1337:1337" # the shared services account (required)
restart: unless-stopped # required — host unit is oneshot; this recovers crashes
environment:
HOME: /tmp
volumes:
- ${LOGS_DIR:-./logs}:/app/logs # output — host log dir, injected at deploy
- ${CONFIG_DIR:-./config}:/app/config # input — host config dir, injected at deploy
# - ${MOUNTS_DIR:-./mounts}:/app/data # optional — arbitrary host data (opt-in)
- cache:/app/cache # your data — ephemeral named volume
volumes:
cache: # bare name — deploy auto-prefixes it per service
```
**The rules:**
- **Service key is always `svc`** — never a repo-specific name, never a
`container_name`. The ops tooling keys off `svc`; this is what makes services
collision-proof on the fleet.
- **`user: "1337:1337"` and `restart: unless-stopped` are required** — the shared
`services` account (uid/gid 1337, fixed fleet-wide), and Docker's restart policy
recovering a crashed container (the host unit is oneshot).
- **Host paths come from `${...}` variables, never hardcoded.** Write to
`${LOGS_DIR}` and `${CONFIG_DIR}`; `${MOUNTS_DIR}` is optional.
- **Named volumes use bare names** (`cache`, not `myapp_cache`) — deploy
auto-prefixes them per service so they can't collide.
!!! tip "The container path is where your code reads and writes"
`WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
your code targets. If your app writes to `./cache` (i.e. `/app/cache`), that's
the `cache` mount; logs go to `/app/logs`, config is read from `/app/config`.
**Three tiers of storage:**
- **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
injected and managed at deploy time.
- **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy
layer **injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is
**opt-in**: uncomment it and choose the container path yourself (it's
app-specific, so it can't be auto-mounted).
- **Named volumes** (`cache`, …) — **yours**: anything else your service persists.
Docker owns them, no host paths to manage.
Locally, `docker compose up` needs nothing set — the `${VAR:-./default}` fallbacks
use `./logs`, `./config` (and `./mounts` if you enable it). When deployed, the ops
tooling sets the real values, so the same file works both places.
**Subprocess and browser workloads** (bots that spawn Chrome, Xvfb, ffmpeg) need
three extra knobs:
```yaml
services:
svc:
build: .
user: "1337:1337"
restart: unless-stopped
init: true # tini as PID 1 — reaps zombie subprocesses, forwards signals
shm_size: "2gb" # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
mem_limit: "4g" # bound memory — raise as worker/browser count grows
```
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
be PID 1, or the real process dies on startup. `init: true` fixes this — tini
takes PID 1, your wrapper runs as a normal child.
=== "Dockerfile"
Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build the image to be **uid-agnostic** so it runs cleanly
as that account.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # services account has no home dir; $HOME must be writable
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # include git ONLY if the container itself needs it
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # deps before code — layer caching
COPY . .
RUN chmod -R a+rwX /app # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```
**Layer caching:** copy the deps file and install **before** `COPY . .` — Docker
caches layers in order, so deps only reinstall when the deps file changes, not on
every code edit.
**Getting files in:** `COPY . .` grabs the whole repo; be explicit about anything
that needs its own place. In `COPY <src> <dest>`, `<src>` is relative to the build
context (your repo), `<dest>` is a path in the image.
```dockerfile
COPY ./config.toml /app/config.toml # a single file into a specific path
COPY ./assets /app/assets # a whole directory (tree mirrors ./assets/)
```
!!! tip "Faster builds with uv (optional)"
[uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
`pyproject.toml` — no lockfile needed in the image. Add `UV_COMPILE_BYTECODE`
so containers don't pay the first-import `.pyc` compile cost:
```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # pull the uv binary from its image
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1 # compile bytecode at build, not cold start
WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system . # into the image's Python, no venv/lock
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```
=== "Secrets"
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored** and live
on the host in your config dir, reaching your container read-only via
`${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
into a layer.
- Secrets live on the **host**, in your config dir — never in git, never in the
image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount.
- Keep them out of the build context: list them in `.dockerignore` so a blanket
`COPY . .` can't pull them into a layer.
**Rotating a secret** is a host-side edit — update the file and the service picks
it up on restart. No rebuild, and nothing you run: flag it and we handle the
restart.
## The path to deploy
You build and test locally — leaning on our libraries, AI, and this handbook — then
push to git. From there it's staging (if you're running gitflow) and, once it's good,
deployment: the ops tooling builds the Docker image and starts it on the fleet.
```mermaid
flowchart LR
subgraph build ["you build"]
direction TB
local["local testing"]
libs["our libraries<br/>(rethink-public)"]
ai["AI-assisted<br/>(Claude Code)"]
docs["reading the<br/>handbook"]
local --- libs
libs --- ai
ai --- docs
docs --- local
end
build -->|push| git["git<br/>(Gitea)"]
git -.->|develop, if gitflow| stage["staging"]
git --> deploy["deployment<br/>(ops tooling)"]
stage --> deploy
deploy --> docker["docker<br/>(image built + run on the fleet)"]
classDef ship fill:#061541,stroke:#569bcc,color:#eef1f6;
classDef work fill:#0e1530,stroke:#294274,color:#eef1f6;
class git,stage,deploy,docker ship;
class local,libs,ai,docs work;
```
docs/deploy/compose.md
-106
View File
@@ -1,106 +0,0 @@
# Compose convention
The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.
```yaml
services:
svc: # generic service key — ALWAYS svc, no container_name
build: .
user: "1337:1337" # the shared services account (required)
restart: unless-stopped # required — the host unit is oneshot; this recovers crashes
environment:
HOME: /tmp
volumes:
- ${LOGS_DIR:-./logs}:/app/logs # output — host log dir, injected at deploy
- ${CONFIG_DIR:-./config}:/app/config # input — host config dir, injected at deploy
# - ${MOUNTS_DIR:-./mounts}:/app/data # optional — arbitrary host data (opt-in, see below)
- cache:/app/cache # your data — ephemeral named volume
volumes:
cache: # bare name — deploy auto-prefixes it per service
```
## The rules
- **Service key is always `svc`.** Never a repo-specific name, never a
`container_name` — the ops tooling keys off `svc`. This is what makes services
collision-proof on the fleet.
- **`user: "1337:1337"` and `restart: unless-stopped` are required.** The first runs
as the shared `services` account (uid/gid 1337, fixed fleet-wide); the second lets
Docker recover a crashed container (the host-side unit is oneshot).
- **Host paths come from `${...}` variables, never hardcoded.** Write to `${LOGS_DIR}`
and `${CONFIG_DIR}`; `${MOUNTS_DIR}` is optional.
- **Named volumes use bare names** (`cache`, not `myapp_cache`) — deploy auto-prefixes
them per service so they can't collide.
!!! tip "The container path is where your code reads and writes"
`WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
your code targets. If your app writes to `./cache` (i.e. `/app/cache`), that's the
`cache` mount; logs go to `/app/logs`, config is read from `/app/config`. Match
the container path to what your code actually uses.
## Three tiers of storage
- **`logs`** (output) and **`config`** (input you edit) — **we handle these**:
injected and managed at deploy time.
- **`mounts`** (optional) — a host dir for arbitrary read/write data. The deploy layer
**injects `MOUNTS_DIR` and auto-creates the dir**, but the mount line is **opt-in**:
uncomment it and choose the container path yourself (it's app-specific, so it can't
be auto-mounted).
- **Named volumes** (`cache`, …) — **yours**: anything else your service persists.
Docker owns them, no host paths to manage.
!!! tip "The `${VAR:-./default}` pattern"
Host paths use a variable with a local fallback. Locally, `docker compose up` needs
nothing set — it uses `./logs`, `./config` (and `./mounts` if you enable it). When
deployed, the ops tooling sets the real values, so the same file works both places.
## How deploy fills it in
At deploy time the tooling **generates** the environment your compose reads, so the
`${...}` variables resolve without anything from you:
```
LOGS_DIR # your host log dir -> ${LOGS_DIR}
CONFIG_DIR # your host config dir -> ${CONFIG_DIR}
MOUNTS_DIR # optional host data dir -> ${MOUNTS_DIR} (if you enable the mount)
```
Write your logs and config to those mount points and you're set. If a service won't
come up or its logs aren't persisting, that's a host-side detail on our end — flag it
and we'll sort it.
## Subprocess and browser workloads
Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra knobs:
```yaml
services:
svc:
build: .
user: "1337:1337"
restart: unless-stopped
init: true # tini as PID 1 — reaps zombie subprocesses, forwards signals
shm_size: "2gb" # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
mem_limit: "4g" # bound memory — raise as worker/browser count grows
```
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not** be
PID 1, or the real process dies on startup. `init: true` is exactly what fixes this
— tini takes PID 1, and your wrapper runs as a normal child.
## Checklist
- service key `svc` — **no `container_name`**, no repo-specific names
- `user: "1337:1337"` — the shared `services` account
- `restart: unless-stopped`
- host paths via `${LOGS_DIR}` / `${CONFIG_DIR}` (and optional `${MOUNTS_DIR}`) — never
hardcoded
- named volumes with **bare names** (`cache`, not `myapp_cache`)
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`
See **[Dockerfile & build](dockerfile.md)** for the image, and **[Secrets](secrets.md)**
for keeping credentials out of it.
docs/deploy/dockerfile.md
-91
View File
@@ -1,91 +0,0 @@
# Dockerfile & build
Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it runs cleanly
as that account.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # services account has no home dir; $HOME must be writable
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # include git ONLY if the container itself needs it
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # deps before code — see layer caching
COPY . .
RUN chmod -R a+rwX /app # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```
- **`HOME=/tmp`** — the `services` account has no home dir; anything writing to `$HOME`
(caches, configs) needs a writable target.
- **`git`** — include it **only if the container itself needs it** (e.g. you
`pip install` from git, or the app shells out to git). The build and host always have
git; this is about what's *inside* the image.
- **`chmod -R a+rwX /app`** — makes the app tree writable by any uid, so it runs as
1337.
## Layer caching
Copy the deps file and install **before** `COPY . .`. Docker caches layers in order, so
deps only reinstall when the deps file changes — not on every code edit. Get this
backwards and every one-line change triggers a full dependency reinstall.
```dockerfile
COPY requirements.txt . # pip baseline
RUN pip install --no-cache-dir -r requirements.txt # cached until deps change
COPY . . # changes every build
```
## Getting your files into the image
`COPY . .` grabs the whole repo, but be explicit about anything that needs its own
place — assets, templates, a config the app reads at runtime. In `COPY <src> <dest>`,
`<src>` is relative to the build context (your repo) and `<dest>` is a path in the
image.
```dockerfile
COPY ./config.toml /app/config.toml # a single file into a specific path
COPY ./assets /app/assets # a whole directory (tree mirrors ./assets/)
```
!!! warning "Don't `COPY` secrets into the image"
Anything sensitive stays **out** of the image — no `COPY ./secrets.env`. Secrets
live on the host and are injected read-only at runtime (see
**[Secrets](secrets.md)**). Add them to `.dockerignore` so a blanket `COPY . .`
can't sweep them in.
## Faster builds with uv (optional)
[uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
`pyproject.toml` — no lockfile needed in the image. Swap the deps layer and add
`UV_COMPILE_BYTECODE` so containers don't pay the first-import `.pyc` compile cost:
```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # pull the uv binary from its image
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1 # compile bytecode at build, not first cold start
WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system . # --system: into the image's Python, no venv/lock
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```
## Checklist
- `HOME=/tmp`
- `chmod -R a+rwX /app` (uid-agnostic; runs as the `services` account, 1337)
- deps installed **before** the code copy (layer caching) — pip or `uv pip install`
- `git` in the image **if the container needs it**
- using uv? add `ENV UV_COMPILE_BYTECODE=1`
See **[Compose convention](compose.md)** for the `compose.yaml`, and
**[Secrets](secrets.md)** for credentials.
docs/deploy/index.md
-54
View File
@@ -1,54 +0,0 @@
# Deployment Guide
> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.
!!! tip "What can run here"
APIs, websites, applets, bots, monitors.
!!! danger "Your compose names nothing repo-specific"
**No `container_name`. No hardcoded names, workspace, or `/srv` paths.** The
deploy layer injects identity and host paths at deploy time — your compose stays
generic so it can't collide with any other service on the fleet.
- service key is always **`svc`**
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**
Naming things after your repo used to cause **container-name collisions** at
fleet scale — two repos shipping the same name clashed. The generic compose
convention fixes that.
!!! note "You don't run any deploy commands"
All services are deployed centrally by the ops tooling. You don't pick a host,
manage keys, or run anything — just make your repo follow the compose convention
and it's deployable. At deploy time the tooling **generates** the environment your
compose reads (`${LOGS_DIR}`, `${CONFIG_DIR}`, optional `${MOUNTS_DIR}`), so the
same file works locally and on the fleet.
## The three things your repo needs
<div class="grid cards" markdown>
- :material-file-cog: __[Compose convention](compose.md)__
---
The `compose.yaml` your repo ships — generic `svc` service, the `${...}` host
mounts, storage tiers, and how deploy fills it in.
- :material-docker: __[Dockerfile & build](dockerfile.md)__
---
The uid-agnostic image (services account 1337), getting files in with `COPY`,
layer caching, uv, and subprocess/browser workloads.
- :material-key: __[Secrets](secrets.md)__
---
How secrets stay out of the image and reach the container read-only at runtime.
</div>
docs/deploy/secrets.md
-19
View File
@@ -1,19 +0,0 @@
# Secrets
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored** and live on
the host in your config dir, reaching your container read-only via `${CONFIG_DIR}`.
Add them to `.dockerignore` so a `COPY . .` can't sweep them into a layer.
## How it works
- Secrets live on the **host**, in your config dir — never in git, never in the image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount (see
**[Compose convention](compose.md)**).
- Keep them out of the build context: list them in `.dockerignore` so a blanket
`COPY . .` can't pull them into a layer.
## Rotating a secret
A host-side edit — update the file and the service picks it up on restart. No rebuild,
and nothing you run: flag it and we handle the restart.
docs/environments.md
+1 -1
View File
@@ -67,7 +67,7 @@ happens, depending on where the project runs:
COPY . .
```
This is how things run in production — see the [Deploy guide](deploy/) for the
This is how things run in production — see the [Deploy guide](deploy.md) for the
full container standard (uid 1337, the compose convention, layer caching, and the
uv image setup).
docs/index.md
+1 -1
View File
@@ -66,7 +66,7 @@ out of it; examples use placeholders like `<workspace>`, `<project>`, and
Project-based Python isolation — local `.venv`, Makefile, or Docker — and
local version management with pyenv.
- :material-rocket-launch: __[Deploy](deploy/)__
- :material-rocket-launch: __[Deploy](deploy.md)__
---
docs/javascripts/mermaid-init.js
+29
View File
@@ -0,0 +1,29 @@
// render mermaid diagrams (emitted as <div class="mermaid">SOURCE</div>).
// render(id, src) is used directly and each block is processed once.
(function () {
var inited = false;
var seq = 0;
function boot() {
if (typeof mermaid === "undefined") return;
if (!inited) {
mermaid.initialize({ startOnLoad: false, theme: "dark", securityLevel: "loose" });
inited = true;
}
document.querySelectorAll("div.mermaid").forEach(function (el) {
if (el.dataset.mmdDone) return;
var src = el.textContent.trim();
if (!src) return;
el.dataset.mmdDone = "1";
mermaid.render("mmd-" + seq++, src).then(function (out) {
el.innerHTML = out.svg;
}).catch(function () {
delete el.dataset.mmdDone;
});
});
}
if (window.document$ && typeof window.document$.subscribe === "function") {
window.document$.subscribe(boot);
} else {
document.addEventListener("DOMContentLoaded", boot);
}
})();
docs/javascripts/mermaid.min.js
Vendored
+3587
View File
File diff suppressed because one or more lines are too long
docs/standards.md
+1 -1
View File
@@ -134,7 +134,7 @@ TimeoutError: request timed out after 30s
A deployable service ships a `compose.yaml` that names **nothing repo-specific** —
the deploy layer injects identity and host paths. See the
[Deploy guide](deploy/compose.md) for the full convention and the variables
[Deploy guide](deploy.md) for the full convention and the variables
you can rely on. The short version:
=== "Right"
docs/workflow.md
+1 -1
View File
@@ -44,7 +44,7 @@ Our code lives on **Gitea** at
2. Add your **SSH public key** under *Settings → SSH / GPG Keys* so you can clone
and push over SSH.
3. For servers, we use a **per-repo deploy-key** model rather than your personal
key — see the [Deploy guide](deploy/) for how a box gets read access to just
key — see the [Deploy guide](deploy.md) for how a box gets read access to just
the repos it needs.
## Our git vs. public git (GitHub / GitLab)
mkdocs.yml
+10 -8
View File
@@ -6,6 +6,10 @@ copyright: rethink development (handbook)
extra_css:
- stylesheets/extra.css
extra_javascript:
- javascripts/mermaid.min.js
- javascripts/mermaid-init.js
theme:
name: material
language: en
@@ -35,7 +39,11 @@ markdown_extensions:
- toc:
permalink: true
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_div_format
- pymdownx.tabbed:
alternate_style: true
- pymdownx.highlight:
@@ -55,10 +63,4 @@ nav:
- Standards: standards.md
- Workflow: workflow.md
- Virtual environments: environments.md
- Deploy: deploy/index.md
# Deploy sub-pages are reached from the Deploy hub's cards, not the global nav.
not_in_nav: |
/deploy/compose.md
/deploy/dockerfile.md
/deploy/secrets.md
- Deploy: deploy.md