Compare commits

...
21 Commits
Author SHA1 Message Date
dsql 9fb5067124 docs: drop the SQLite alternative from the Redis sidecar section
Redis-only now. The persist-or-not decision stays ephemeral (--save "")
vs persistent (--appendonly yes + ${MOUNTS_DIR}), and both caveats — the
~1s appendfsync everysec window and the not-backed-up dividing line —
are unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-14 15:55:16 -04:00
dsql a248879451 docs: document Redis as a per-repo compose sidecar
New Deploy section 'Supporting services live in your compose, not the
fleet': Redis is declared per-repo, never shared fleet-wide — a shared
instance would re-couple services the per-project network deliberately
isolates.

- ephemeral vs persistent as a conscious choice, both compose snippets in
  content tabs (--save "" + no volume, vs --appendonly yes + ${MOUNTS_DIR})
- connect by compose service name (redis://redis:6379), never a host port
- caveats given weight: appendfsync everysec ~1s loss window, and mounts
  are not backed up — the dividing line between queue state and Postgres
- SQLite noted as the no-container alternative for a single drainer
- the don'ts, plus one line on why ACLs aren't the answer here
- cross-links the redis lib on the Libraries page

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-14 15:52:55 -04:00
dsql b3acd57e32 docs: use dedicated purple lambda favicon
Split favicon off from the logo — new docs/assets/favicon.svg (purple
gradient lambda), mkdocs favicon key repointed to it; logo unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-10 21:45:52 -04:00
dsql 6151cb2e4e docs: add paste service to Workflow; clarify app-log vs stdout in Deploy
- Workflow: paste.rethinkstudios.io pastebin section — pb() function
  (recommended) + alias alternative in content tabs, usage, expires note,
  creation rate-limit warning.
- Deploy: warning admonition distinguishing ${LOGS_DIR} (app logger, to
  disk) from container stdout/stderr where startup crashes and uncaught
  exceptions land; sys.excepthook snippet to route them into the log file.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 22:13:59 -04:00
dsql 350884c2b3 deploy: whole-flow as plain text, restore v2 diagram shape; admonition variety
- 'The whole flow' is now plain prose, not an abstract box.
- Restore the v2 diagram shape (git -> staging via develop-if-gitflow AND
  git -> deployment, converging on docker) instead of the linear main-only
  one, keeping the simplified 'libs (rethink-public, pip)' wording.
- Spread admonition types for variety across pages: index agent-callout
  tip -> example; environments uv.lock note -> info; standards logging
  note -> info. (deploy already mixes success/danger/example/warning/tip.)

Verified in-browser: whole-flow reads as text, diagram renders in the v2
shape; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 03:04:51 -04:00
dsql 5797248931 deploy: move diagram into Preparing section, vary admonitions, simplify
- Move the flow diagram up into 'Preparing for Deploy' (was a separate
  section at the bottom); drop that trailing section.
- Use more admonition types for visual variety: success (what can run),
  abstract (the whole flow), example (container-path), success (same file
  both places) — alongside the existing danger/warning/tip.
- Simplify the diagram: libs -> 'libs (rethink-public, pip)', the git edge
  -> 'branch', staging -> 'main'; abstract intro reworded to branch+main.
- Trim the Compose tab: drop the rules list that duplicated the inline
  comments + danger box; tighten the storage tiers.

Verified in-browser: diagram renders with simplified nodes near the top,
six admonition types present; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:58:22 -04:00
dsql 823aa7708a deploy: mermaid flow diagram, single page with tabs, section rework
Collapse the deploy sub-pages back into one deploy.md (tabs, not separate
pages), add a working mermaid flow diagram, and rework the top per feedback.

- Mermaid: enable via superfences fence_div_format (the fence_code_format
  <pre><code> wrapper broke mermaid's render — div format fixes it); vendor
  mermaid.min.js locally (no CDN dependency for a self-hosted site) + a small
  init that renders each block once and survives instant-nav.
- Content tabs Compose / Dockerfile / Secrets hold the three pieces together
  on one page (inline comments in the code, no annotation +).
- 'What can run here' tip moved above the flow; section renamed
  'Preparing for Deploy' with the 'you don't run deploy commands' point folded
  sparsely into the intro (standalone note removed); 'svc never changes'
  called out.
- Flow diagram (below the tabs): build group now includes our libraries; git
  -> staging shown as develop-if-gitflow (dotted); deployment (no MR-to-main)
  -> docker image built + run on the fleet.
- Revert inbound links to deploy.md.

Verified in-browser: diagram renders with all nodes, tabs switch, section
order correct; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:53:02 -04:00
dsql f9e5e4f24d split Deploy into a hub + 3 sub-pages; inline comments over annotations
The Deploy page got long and the code annotations rendered inconsistently
(some as (n) numbers, some as clickable +, and a marker on a fully-commented
line made that line vanish). Fix both:

- Deploy is now a HUB (docs/deploy/index.md): the intro + collision rule +
  central-deploy note, then card links to three focused sub-pages. Deploy
  stays the single global-nav entry; sub-pages are not_in_nav, reached from
  the hub cards.
  - deploy/compose.md — compose convention, storage tiers, how deploy fills
    it in, subprocess/browser knobs, checklist
  - deploy/dockerfile.md — services-account image, COPY, layer caching, uv
  - deploy/secrets.md — keeping secrets out of the image
- Replace code annotations with INLINE COMMENTS on the compose/Dockerfile
  examples: everything visible at once, no + to click, and the commented
  MOUNTS_DIR line no longer disappears.
- Update inbound links (index card, standards, workflow, environments) to
  deploy/ and deploy/compose.md; nav Deploy -> deploy/index.md with
  not_in_nav for the sub-pages.

Verified in-browser: hub cards link correctly, sub-pages render with visible
inline comments (0 annotation markers), left nav shows only Deploy;
mkdocs build --strict clean (validates not_in_nav + all cross-links).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:32:16 -04:00
dsql c335887dd6 deploy: fix vanishing commented mounts line in compose annotations
A '# (6)!' annotation marker on a fully-commented YAML line made Material's
annotation JS swallow the whole line into the tooltip — so the commented
'# - ${MOUNTS_DIR...}' line disappeared, leaving an orphan marker with no
visible code (which is why '(6)' seemed missing).

Fix: drop the marker from the commented line, keep it as plain visible YAML
('# optional — see below'), and move its explanation to prose below the
annotation list. Renumber cache/volume markers 7->6, 8->7.

Verified in-browser: commented mounts line now renders visibly; no orphan
marker. build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:22:46 -04:00
dsql fe707d5f24 deploy: clarify container path = where your code reads/writes
Make the volume mapping explicit: the right-hand mount path is where the
app writes. WORKDIR is /app, so code writing to ./cache lands in /app/cache
(this mount); same for /app/logs and /app/config. Removes ambiguity about
which side of the volume line the code targets.

Verified: mkdocs build --strict clean.
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:14:50 -04:00
dsql 0ebd6f86b0 deploy: scope page to the dev's job — drop operator commands
Devs never deploy — that's a central ops tool. Remove operator-facing
content and keep only what a dev is responsible for (a conforming compose).

- remove the 'Onboarding a service' section (deploy <host> <workspace>
  <git-url> command + deploy-key internals) and the 'How deploy fills it in'
  docker-ps collision table + <workspace>/<name> naming internals
- replace with 'Deployment is handled centrally': a note that devs run no
  deploy commands, plus the kept dev-facing context that the env is
  GENERATED at deploy time so ${LOGS_DIR}/${CONFIG_DIR}/${MOUNTS_DIR}
  resolve without them (less ambiguous than just 'it's handled')
- trim operator leftovers: svc-<ws>-<name> unit ref, dangling '(below)',
  and the /srv/config/<ws>/<name> path in the secrets note

Compose convention, uid 1337, restart, mounts tier, layer caching, uv,
subprocess knobs all unchanged. Verified in-browser; build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 02:04:17 -04:00
dsql 08ad37eddc deploy: re-add mounts as optional (commented) third storage tier
The deploy scripts already handle mounts (dir creation + MOUNTS_DIR
injection); only the docs had dropped it. Re-add it as opt-in so scripts
and docs agree:

- compose: a commented '# - ${MOUNTS_DIR:-./mounts}:/app/data' line — the
  dir is auto-created and MOUNTS_DIR injected, but the mount line stays
  dev-opted (container-side path is app-specific, can't be auto-mounted)
- reframe the storage tip as 'Three tiers of storage': logs+config (ours),
  mounts (optional, injected + auto-created, mount line opt-in), named
  volumes (yours)
- re-add MOUNTS_DIR to the injected env vars (noted opt-in) and mention it
  in the local-fallback tip, paths line, and checklist as optional

Verified in-browser; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 01:39:52 -04:00
dsql 8c56fd98a8 deploy: add COPY-into-image tabs, clarify volume ownership, drop mounts
- deploy.md: new 'Getting your files into the image' subsection — a
  multi-tab annotated block (single file / directory / deps-first) showing
  COPY <src> <dest> patterns, plus a warning not to COPY secrets.
- clarify volume ownership: 'Logs and config are ours; the rest is yours' —
  we inject/manage the logs + config host mounts; devs put everything else
  in named volumes Docker owns.
- remove all /srv/mounts: dropped the MOUNTS_DIR volume + env var and every
  mounts reference from the compose block, injected-vars, paths line, and
  checklist. environments.md + index.md mounts mentions cleaned too.

Verified in-browser: COPY tabs switch, volume tip renders, zero mounts refs
remain; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 01:36:25 -04:00
dsql 7ed388c69c new compose + deploy standard: generic compose, deploy injects identity
Supersedes the prior 'name everything after the repo' guidance, which caused
container-name collisions at fleet scale. The dev's compose now names nothing
repo-specific; the deploy layer injects identity and host paths.

deploy.md:
- replace the 'One name everywhere' danger callout with 'Your compose names
  nothing repo-specific' (no container_name, no hardcoded names/paths)
- rewrite the compose example to the standard: svc service key,
  restart: unless-stopped (required — host unit is oneshot), host paths via
  ${LOGS_DIR}/${CONFIG_DIR}/${MOUNTS_DIR} with :-./ local fallbacks, bare
  'cache' volume. Keep user: "1337:1337" + HOME=/tmp (the services account
  identity — not repo-specific)
- add 'How deploy fills it in' (injected env vars a dev can rely on + the
  resulting docker ps names) and 'Onboarding a service' (deploy <host>
  <workspace> <git-url> [name], deploy key handled via Gitea API)
- update paths to <name>, subprocess example to svc, checklist to the new
  rules; drop docker CLI from secret rotation (host-side edit, we restart)

standards.md:
- add a 'Service compose' section: short convention + Right/Wrong tabs
  (the nova before/after), linking to the Deploy guide for the full detail

Kept the services-account (1337) section and uid-agnostic Dockerfile notes.
Verified in-browser; mkdocs build --strict clean (cross-ref anchor resolves).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 01:04:44 -04:00
dsql 17b2888c1f deploy: red callout — repo name must match container + volume names
Add a prominent danger admonition ('One name everywhere — repo = container =
volumes') right under the intro: the repo name is the name; the compose
service key, container_name, and every named volume must all use it. Spell
out the consequence — mismatched names break deploy, monitoring, and log
scraping, which key off the repo name.

Make the compose example teach it: rename yourapp -> myrepo throughout
(service key, container_name: myrepo, myrepo-data volume) with annotations
calling out each spot where the repo name must appear.

Verified in-browser; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-30 23:13:35 -04:00
dsql c53d67da2f add uv as a parallel option alongside pip
uv is presented as the recommended faster, standards-compliant drop-in;
pip stays the baseline/fallback. uv command syntax verified against
docs.astral.sh/uv before writing.

environments.md:
- new 'uv (optional, faster)' section: install from pyproject
  (uv pip install . / -e . / '.[dev]'), the uv sync managed-venv flow,
  and a note that uv.lock is local-only/gitignored (not committed)
- pinning subsection: pin direct deps in pyproject via == or git @ref,
  with a warning that this pins direct deps only — transitive deps still
  float at build time (documented tradeoff)
- uv equivalents added beside pip in the Local .venv and Docker tabs

deploy.md:
- 'Faster builds with uv' tip: uv-from-ghcr COPY, ENV UV_COMPILE_BYTECODE=1,
  uv pip install --system . (reads pyproject, no lock)
- layer-caching shows the uv variant beside the pip one
- checklist notes uv pip install + UV_COMPILE_BYTECODE; fix stray 'configs'
  plural -> 'config'

.gitignore: ignore uv.lock (local-only, never committed).

Verified in-browser; mkdocs build --strict clean (anchors resolve).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-30 04:40:10 -04:00
dsql dafc1dcacd fix path: /srv/configs -> /srv/config (singular); fix broken anchor
- deploy.md: /srv/configs -> /srv/config (singular is canonical) across the
  compose mount, paths table, secrets path, and rotation snippet. Add the
  repo+compose row (/srv/docker/<workspace>/<project>, created by the clone,
  not pre-provisioned) and a note that all /srv paths are owned by the
  services user (1337) — rounding out the canonical layout.
- workflow.md: the first 'per-project git identity' link pointed at a
  nonexistent #per-project-git-identity anchor; point it at #handy-shell-setup
  (where the gitsetup alias lives), matching the other link to the same spot.

Did not touch pip/requirements, chmod a+rwX + user 1337, HOME=/tmp, init,
layer caching, or the git-install caveat — all intentional standards.

Verified: mkdocs build --strict clean (validates anchors); table renders.
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-30 00:57:24 -04:00
dsql 6498c5b6d5 rename <dev> path segment to <workspace>, unify /srv paths
Host paths now follow one shape: /srv/<kind>/<workspace>/<project>/ across
configs, logs, and mounts (mounts added as its own row; configs gains the
workspace segment it was missing).

- deploy.md: compose volumes, paths table, secrets path + rotation snippet
  all use <workspace>; add a yellow (warning) admonition ABOVE the table
  explaining it — individual dev = lowercase username (ricky, xattam),
  shared/official project = workspace (bots, web, apis) — so it stands out
  from the all-blue palette and reads before the table.
- index.md: placeholder example uses <workspace>.

Verified in-browser; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 23:50:42 -04:00
dsql 5f91d73e1f agent callout: read from git direct, don't clone into the project
The 'from git' tab now tells the agent to read docs/ straight from the
repo URL instead of git-cloning it into the working project — clone to
/tmp only if it needs a local copy.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:30:35 -04:00
dsql 75ad61b06a add 'point your coding agent here' callout on the landing page
A tip admonition under the intro telling devs to point their coding agent
at the handbook for awareness of our libs/standards/deploy rules, with
copy-pasteable agent prompts in two tabs: from the live site, or from a
git clone of the repo.

Verified in-browser: tabs switch, prompts + clone command render, copy
buttons present; mkdocs build --strict clean.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:30:07 -04:00
dsql ed4e558dca tone down advertisey copy
- index.md: 'building and shipping with Rethink Studios' -> 'building and
  shipping on our network' (it's the network, not the company)
- workflow.md + its index card: 'how we actually work day to day' ->
  'get hands on with how we dev'
- deploy.md: drop the 'see the light / ready to host whatever you've built'
  pitch for a plain 'how a project gets onto rethink-net'; trim the now-
  duplicate eligibility tip

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:23:34 -04:00
10 changed files with 4081 additions and 135 deletions
+1
View File
@@ -9,6 +9,7 @@ __pycache__/
*.py[cod]
.venv/
venv/
uv.lock
# Playwright MCP run artifacts
.playwright-mcp/
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.0 KiB

+262 -116
View File
@@ -1,147 +1,293 @@
# Deployment Guide
> Ready for your project to see the light? You may be eligible for deployment on
> **rethink-net** — our fleet of Ubuntu 26.x servers, ready to host whatever
> you've built.
> How a project gets onto **rethink-net** — our Ubuntu 26.x servers. Get your
> container to follow a few consistent rules and deploying is mostly handing us a
> `compose.yaml`.
!!! tip "Eligible"
APIs, websites, applets, bots, monitors. The whole network runs on a few
simple, consistent rules — get your container to follow them and deploying is
mostly handing us a `compose.yaml`.
!!! success "What can run here"
APIs, websites, applets, bots, monitors.
## Docker — the services account
## Preparing for Deploy
Every service runs containerized as the shared **`services`** account:
**uid/gid 1337**, fixed fleet-wide. Build your image to be **uid-agnostic** so it
runs cleanly as that account.
You build and test locally — leaning on our libraries, AI, and this handbook — then
push to git. From there the ops tooling takes over: staging (if you run gitflow), then
it builds the Docker image and runs it on the fleet. **You don't run any deploy
commands** — your only job is the three pieces in the tabs below.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # (1)!
```mermaid
flowchart LR
subgraph build ["you build"]
direction TB
local["local testing"]
libs["libs<br/>(rethink-public, pip)"]
ai["AI-assisted<br/>(Claude Code)"]
docs["reading the<br/>handbook"]
local --- libs
libs --- ai
ai --- docs
docs --- local
end
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # (2)!
build -->|push| git["git<br/>(Gitea)"]
git -.->|develop, if gitflow| stage["staging"]
git --> deploy["deployment<br/>(ops tooling)"]
stage --> deploy
deploy --> docker["docker<br/>(built + run<br/>on the fleet)"]
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # (3)!
COPY . .
RUN chmod -R a+rwX /app # (4)!
CMD ["python", "-m", "yourapp"]
classDef ship fill:#061541,stroke:#569bcc,color:#eef1f6;
classDef work fill:#0e1530,stroke:#294274,color:#eef1f6;
class git,stage,deploy,docker ship;
class local,libs,ai,docs work;
```
1. `HOME=/tmp` — the `services` account has no home dir; anything writing to
`$HOME` (caches, configs) needs a writable target.
2. Include `git` **only if the container itself needs it** — e.g. you
`pip install` from git, or the app shells out to git at runtime. The build and
host always have git; this line is about what's *inside* the image.
3. Install deps **before** copying the code (see [layer
caching](#layer-caching)).
4. `chmod -R a+rwX /app` makes the app tree writable by **any** uid — that's what
"uid-agnostic" means.
!!! danger "Your compose names nothing repo-specific"
**No `container_name`, no hardcoded names/paths.** The deploy layer injects
identity and host paths so your compose can't collide with any other service on
the fleet:
- the service key is **always `svc`** — it never changes, in any repo
- named volumes use **bare names** (`cache`, not `myapp_cache`)
- host paths come from **`${LOGS_DIR}` / `${CONFIG_DIR}`**
=== "Compose"
The `compose.yaml` your repo ships. Copy it verbatim — it names nothing
repo-specific, so the deploy layer can inject identity and host paths without
collisions.
```yaml
services:
svc: # generic service key — ALWAYS svc, no container_name
build: .
user: "1337:1337" # the shared services account (required)
restart: unless-stopped # required — host unit is oneshot; this recovers crashes
environment:
HOME: /tmp
volumes:
- ${LOGS_DIR:-./logs}:/app/logs # output — host log dir, injected at deploy
- ${CONFIG_DIR:-./config}:/app/config # input — host config dir, injected at deploy
# - ${MOUNTS_DIR:-./mounts}:/app/data # optional — arbitrary host data (opt-in)
- cache:/app/cache # your data — ephemeral named volume
```yaml
services:
yourapp:
build: .
user: "1337:1337" # (1)!
environment:
HOME: /tmp
volumes:
- /srv/configs/<project>:/app/config:ro # (2)!
- /srv/logs/<dev>/<project>:/app/logs # (3)!
- yourapp-data:/app/data # (4)!
cache: # bare name — deploy auto-prefixes it per service
```
volumes:
yourapp-data:
```
!!! example "The container path is where your code reads and writes"
`WORKDIR` is `/app`, so the **right-hand side** of each volume line is the path
your code targets. Write to `./cache` (i.e. `/app/cache`) → the `cache` volume;
logs go to `/app/logs`, config is read from `/app/config`.
1. Run as the shared account. **No** in-container `user`/`useradd` — don't bake a
user into the image; set it here.
2. Configs: host-managed bind mount, mounted **read-only**.
3. Logs: bind mount — live and rolled, scraped for monitoring.
4. Everything else: a **named volume**. Docker owns it, so there are no host
permissions to fiddle with.
**Storage, three tiers:**
## Paths and mounts
- **`logs` + `config`** — we inject and manage these at deploy time.
- **`mounts`** (optional) — host dir for arbitrary data. We inject `MOUNTS_DIR` and
create the dir; you **uncomment** the line and pick the container path.
- **Named volumes** (`cache`, …) — yours; Docker owns them, no host paths to manage.
| What | Where | How |
| --- | --- | --- |
| Configs | `/srv/configs/<project>/` | bind mount, host-managed, read-only |
| Logs | `/srv/logs/<dev>/<project>/` | bind mount; live + rolled, scraped |
| Caches, profiles, scratch | named volume | Docker manages ownership |
!!! warning "Two kinds of logs — and crashes go to the other one"
`${LOGS_DIR}` holds **only** what your app writes to disk through its logger
(e.g. `log_setup` writing a file) — your own structured logging. It does **not**
capture the process's stdout/stderr, and that's where **startup crashes and
uncaught exceptions land** — a traceback from a failed import or a missing file
never reaches your logger. So if a service dies on startup, or you don't see the
error in your log files, it's in the process output, not `${LOGS_DIR}`. Make
fatal errors visible — and to route uncaught exceptions into your log file too,
install a top-level hook:
!!! note "If your service won't start or its logs aren't persisting"
That's usually a host-side bind-mount **ownership** thing — the kind of detail
**we sort out at deploy time**, not something you need to chown or provision.
If a bot won't come up or logs/caches keep vanishing, flag it and we'll fix
the mount perms. Stick to a clean `compose.yaml` and let us handle the host.
```python
import sys, logging
sys.excepthook = lambda *exc: logging.getLogger().critical("uncaught", exc_info=exc)
```
## Layer caching
!!! success "Same file, both places"
Locally, `docker compose up` needs nothing set — the `${VAR:-./default}`
fallbacks use `./logs` / `./config`. When deployed, the ops tooling sets the
real values. One `compose.yaml` works everywhere.
Copy `requirements.txt` and `pip install` **before** `COPY . .`. Docker caches
layers in order, so deps only reinstall when `requirements.txt` changes — not on
every code edit. Get this backwards and every one-line change triggers a full
dependency reinstall.
**Subprocess and browser workloads** (bots that spawn Chrome, Xvfb, ffmpeg) need
three extra knobs:
```dockerfile
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # cached until deps change
COPY . . # changes every build
```
```yaml
services:
svc:
build: .
user: "1337:1337"
restart: unless-stopped
init: true # tini as PID 1 — reaps zombie subprocesses, forwards signals
shm_size: "2gb" # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
mem_limit: "4g" # bound memory — raise as worker/browser count grows
```
## Subprocess and browser workloads
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
be PID 1, or the real process dies on startup. `init: true` fixes this — tini
takes PID 1, your wrapper runs as a normal child.
Bots that spawn Chrome, Xvfb, ffmpeg, or other child processes need three extra
knobs in compose:
=== "Dockerfile"
```yaml
services:
yourbot:
build: .
user: "1337:1337"
init: true # (1)!
shm_size: "2gb" # (2)!
mem_limit: "4g" # (3)!
```
Every service runs containerized as the shared **`services`** account: **uid/gid
1337**, fixed fleet-wide. Build the image to be **uid-agnostic** so it runs cleanly
as that account.
1. Runs **tini** as PID 1 to reap zombie subprocesses and forward signals.
Without it, spawned Chrome/Xvfb processes leak as zombies.
2. Chrome and most headless browsers crash on Docker's default **64 MB**
`/dev/shm`. Bump it for any browser workload.
3. Bound memory — especially when each worker spawns a browser. Raise it as
worker count grows.
```dockerfile
FROM python:3.12-slim
ENV HOME=/tmp # services account has no home dir; $HOME must be writable
!!! warning "The PID-1 gotcha with shell-wrapper CMDs"
If your `CMD` is a shell-script wrapper (e.g. `xvfb-run ...`), it must **not**
be PID 1, or the real process dies on startup. `init: true` is exactly what
fixes this — tini takes PID 1, and your wrapper runs as a normal child.
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/* # include git ONLY if the container itself needs it
## What your compose / Dockerfile needs
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt # deps before code — layer caching
- `user: "1337:1337"`
- bind mounts for **configs + logs**
- named volumes for **the rest**
- secrets bind-mounted **`:ro`**
- `HOME=/tmp`
- `chmod -R a+rwX /app`
- deps installed **before** the code copy (layer caching)
- `git` in the image **if the container needs it**
- for browser/subprocess workloads: `init: true`, `shm_size`, `mem_limit`
COPY . .
RUN chmod -R a+rwX /app # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]
```
## Secrets
**Layer caching:** copy the deps file and install **before** `COPY . .` — Docker
caches layers in order, so deps only reinstall when the deps file changes, not on
every code edit.
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored**, live on
the host at `/srv/configs/<project>/`, and are bind-mounted **read-only** at
runtime. Add them to `.dockerignore` so a `COPY . .` can't sweep them into a
layer.
**Getting files in:** `COPY . .` grabs the whole repo; be explicit about anything
that needs its own place. In `COPY <src> <dest>`, `<src>` is relative to the build
context (your repo), `<dest>` is a path in the image.
Rotating a secret = edit the host file and restart. No rebuild.
```dockerfile
COPY ./config.toml /app/config.toml # a single file into a specific path
COPY ./assets /app/assets # a whole directory (tree mirrors ./assets/)
```
```bash
vim /srv/configs/<project>/secrets.env # edit on the host
docker compose restart yourapp # pick up the change — no rebuild
```
!!! tip "Faster builds with uv (optional)"
[uv](https://docs.astral.sh/uv/) is a drop-in for pip that reads the same
`pyproject.toml` — no lockfile needed in the image. Add `UV_COMPILE_BYTECODE`
so containers don't pay the first-import `.pyc` compile cost:
```dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # pull the uv binary from its image
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1 # compile bytecode at build, not cold start
WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system . # into the image's Python, no venv/lock
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]
```
=== "Secrets"
!!! warning "Secrets never go in the image"
We do **not** commit secrets (usually, lol). They stay **gitignored** and live
on the host in your config dir, reaching your container read-only via
`${CONFIG_DIR}`. Add them to `.dockerignore` so a `COPY . .` can't sweep them
into a layer.
- Secrets live on the **host**, in your config dir — never in git, never in the
image.
- They reach the container **read-only** via the injected `${CONFIG_DIR}` mount.
- Keep them out of the build context: list them in `.dockerignore` so a blanket
`COPY . .` can't pull them into a layer.
**Rotating a secret** is a host-side edit — update the file and the service picks
it up on restart. No rebuild, and nothing you run: flag it and we handle the
restart.
## Supporting services live in your compose, not the fleet
Need Redis? **Declare it in your own `compose.yaml`, as a sidecar.** There is no shared
Redis — nothing fleet-wide, nothing per-workspace, nothing ops provisions for you. A repo
that needs Redis brings its own; a repo that doesn't adds nothing.
That's the whole point of the deploy model. Every service already runs in its own compose
project on its own network so that one service falling over can't touch another. A shared
Redis puts that coupling straight back: one process everything depends on, whose OOM, stray
`FLUSHALL`, single-threaded stall, or restart becomes *everyone's* outage. A sidecar shares
its owning repo's fate and nobody else's — and the isolation is free, because it rides the
per-project network you already get. No ACLs, no key-prefix discipline, no shared
credentials to manage.
Your app talks to it with the [`redis` lib](libraries.md) from the suite (async,
config-free, kv/hash/ttl/pubsub), pointed at **`redis://redis:6379`** — the compose
**service name**, not a host port. The sidecar comes up auto-namespaced on your project's
network like every other container, exactly as the naming convention above describes.
### Ephemeral or persistent — pick deliberately
A sidecar Redis is **ephemeral by default**: restart it and the data is gone. That's
correct for some workloads and quietly destructive for others, so make the call on purpose.
Ask one question — *if this data vanished on a restart, would anything be lost?*
- **No → ephemeral.** A scratch cache, a dedupe set, rate-limit counters, transient data
you can just re-fetch. Nothing to back up, nothing to grow.
- **Yes → persistent.** An outbound webhook or notification queue, a job queue, anything
that could be mid-flight when the process dies. Losing it drops real work.
=== "Ephemeral (cache / throwaway)"
Fine to lose on restart — no volume, no persistence, by design.
```yaml
redis:
image: redis:7-alpine
restart: unless-stopped
command: redis-server --save "" --maxmemory 256mb --maxmemory-policy allkeys-lru
# no volume: throwaway by design
```
=== "Persistent (durable queue / state)"
Survives restart, rebuild, and reboot — the append-only file lives on the host mounts
dir injected at deploy, the same `${MOUNTS_DIR}` mechanism described above. Redis just
uses it as its backing store.
```yaml
redis:
image: redis:7-alpine
restart: unless-stopped
command: redis-server --appendonly yes --appendfsync everysec
volumes:
- ${MOUNTS_DIR:-./mounts}/redis:/data # AOF persists on the host mounts dir
```
The app connects the same way in both modes — `REDIS_URL: redis://redis:6379`. Only the
durability changes.
!!! warning "Two things to know before you rely on a persistent sidecar"
**`--appendfsync everysec` can lose ~1 second of the newest entries** on a hard crash.
For a webhook queue that's an acceptable trade — just know it's there. Use
`--appendfsync always` if you genuinely cannot drop a single entry (safer, slower).
**Data under the mounts dir is not backed up.** Mounts are excluded from the backup
pipeline, and that's *right* for a queue: a lost queue means some notifications didn't
fire, not that business data is gone. **That's the dividing line.** If losing this data
would actually hurt, it isn't queue or cache state — it's a system of record, and it
belongs in Postgres (which *is* backed up), not a local mount.
### The rules
!!! danger "Don't do these"
- **Don't map Redis to a host port.** No `ports: - "6379:6379"`. Two repos both
grabbing host 6379 on the same box collide. Keep it internal to the compose network —
nothing exposed, nothing to collide.
- **Don't stand up a shared or fleet-wide Redis.** Per-repo means per-need. One Redis
per *project*, shared by that project's containers if a repo runs several — never one
per fleet.
- **Don't treat persistent Redis as a database.** Queues and caches, yes. A durable
system of record, no — that's Postgres, and unlike a mount it's backed up.
**You can't reach another repo's Redis** — different project, different network. That's not
a restriction you have to work around; it's the isolation working *for* you. Nobody else's
service can touch your cache or drain your queue either, and you never have to think about
whose keys are whose.
!!! quote "What about ACLs?"
A shared Redis *can* be secured — Redis 6+ ACLs scope users by command, key pattern, and
channel. But ACLs don't solve resource contention or the noisy-neighbour problem, and
they add real management burden, so the fleet uses per-repo sidecars instead. Reserve
ACLs for the rare case of a deliberately shared, durable, backed-up Redis run as actual
infrastructure.
+57 -4
View File
@@ -21,7 +21,14 @@ happens, depending on where the project runs:
```bash
python -m venv .venv # create it (once)
source .venv/bin/activate # activate for this shell
pip install -r requirements.txt
pip install -e . # install the project from pyproject.toml
```
Or with [uv](#uv-optional-faster) — same `pyproject.toml`, much faster:
```bash
uv venv # create .venv
uv pip install -e . # install from pyproject.toml
```
Keep `.venv/` **gitignored** — it's per-machine, never committed.
@@ -55,13 +62,14 @@ happens, depending on where the project runs:
```dockerfile
FROM python:3.12-slim
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY pyproject.toml .
RUN pip install --no-cache-dir . # or: uv pip install .
COPY . .
```
This is how things run in production — see the [Deploy guide](deploy.md) for the
full container standard (uid 1337, mounts, layer caching).
full container standard (uid 1337, the compose convention, layer caching, and the
uv image setup).
!!! tip "Which one?"
**Local `.venv`** for quick iteration, **Makefile** when you want repeatable
@@ -69,6 +77,51 @@ happens, depending on where the project runs:
exclusive — a project often has a `.venv` for local dev *and* a Dockerfile for
deploy.
## uv (optional, faster)
!!! tip "uv is the recommended fast path; pip stays the baseline"
[uv](https://docs.astral.sh/uv/) is a faster, standards-compliant drop-in for
pip. It reads the **same `pyproject.toml`** — no workflow change required, and
`pip` keeps working exactly as before. Use it wherever you'd reach for pip; the
rest of this handbook shows the pip command with the uv equivalent beside it.
Install deps straight from `pyproject.toml` (no `requirements.txt` needed):
```bash
uv pip install . # install the project + its deps
uv pip install -e . # editable (dev) install
uv pip install '.[dev]' # with an extras group, e.g. dev
```
If you'd rather have uv manage the venv for you, use the managed-venv flow:
```bash
uv sync # create/refresh .venv from pyproject.toml + uv.lock
uv run python -m yourapp # run inside the managed env, no manual activate
```
!!! info "`uv.lock` is local-only — never committed"
`uv sync` writes a `uv.lock` for your machine's resolved environment. It is
**gitignored**, not committed — we don't ship a lockfile. Pinning happens in
`pyproject.toml` (below), not the lock.
### Pinning deps
Pin **direct** dependencies in `pyproject.toml` with `==` or a git `@ref`:
```toml
[project]
dependencies = [
"requests==2.31.0",
"mylib @ git+https://git.rethinkstudios.io/rethink-public/mylib.git@<sha>",
]
```
!!! warning "This pins direct deps only — transitive deps still float"
`==` / `@ref` pins the packages **you** list. Their dependencies still resolve
fresh at build time. That's an accepted tradeoff — documented on purpose — not
an oversight: we pin what we depend on directly and let the rest float.
## Local dev with pyenv
For local work you also need the right **Python version**, not just isolated deps.
+33 -6
View File
@@ -1,11 +1,38 @@
# rethink development
The public reference for building and shipping with Rethink Studios: our shared
libraries, our coding standards, and how to deploy a project on our network.
The reference for building and shipping on our network: the shared libraries, the
coding standards, and how to get a project deployed.
This is a public site — it documents generic patterns and conventions. Real
infrastructure specifics (hostnames, internal IPs, exact topology, secrets) stay
out of it; examples use placeholders like `<dev>`, `<project>`, and `/srv/...`.
out of it; examples use placeholders like `<workspace>`, `<project>`, and
`/srv/...`.
!!! example "Point your coding agent here"
Want your agent aware of our libraries, standards, and deploy rules before it
writes a line? Tell it to read this handbook — so it reaches for an existing
`rethink-public` lib instead of reinventing it, follows our conventions, and
builds a deploy-ready container.
=== "From the live site"
```text
Read https://docs.rethinkstudios.io and follow it: prefer our
rethink-public libraries, match our coding standards, and make anything
deployable per the deploy guide.
```
=== "From git"
Point it straight at the repo — no need to clone into your project:
```text
Read the markdown under docs/ in
https://git.rethinkstudios.io/rethink-public/handbook and follow it:
prefer our rethink-public libraries, match our coding standards, and make
anything deployable per the deploy guide. Clone to /tmp if you need it
local.
```
## Sections
@@ -29,7 +56,7 @@ out of it; examples use placeholders like `<dev>`, `<project>`, and `/srv/...`.
---
How we actually work day to day — our Gitea, git habits, and the
Get hands on with how we dev — our Gitea, git habits, and the
plan-in-chat / build-in-Claude-Code flow, plus shell setup.
- :material-language-python: __[Virtual environments](environments.md)__
@@ -43,7 +70,7 @@ out of it; examples use placeholders like `<dev>`, `<project>`, and `/srv/...`.
---
How to get a project running on **rethink-net**containers, paths and
mounts, permissions, and secrets.
How to get a project running on **rethink-net**the compose convention,
the one-command deploy, and secrets.
</div>
+29
View File
@@ -0,0 +1,29 @@
// render mermaid diagrams (emitted as <div class="mermaid">SOURCE</div>).
// render(id, src) is used directly and each block is processed once.
(function () {
var inited = false;
var seq = 0;
function boot() {
if (typeof mermaid === "undefined") return;
if (!inited) {
mermaid.initialize({ startOnLoad: false, theme: "dark", securityLevel: "loose" });
inited = true;
}
document.querySelectorAll("div.mermaid").forEach(function (el) {
if (el.dataset.mmdDone) return;
var src = el.textContent.trim();
if (!src) return;
el.dataset.mmdDone = "1";
mermaid.render("mmd-" + seq++, src).then(function (out) {
el.innerHTML = out.svg;
}).catch(function () {
delete el.dataset.mmdDone;
});
});
}
if (window.document$ && typeof window.document$.subscribe === "function") {
window.document$.subscribe(boot);
} else {
document.addEventListener("DOMContentLoaded", boot);
}
})();
+3587
View File
File diff suppressed because one or more lines are too long
+41 -1
View File
@@ -125,7 +125,47 @@ TimeoutError: request timed out after 30s
2026-06-29 14:03:11,204 WARNING aioweb.session fetch timed out: https://example.test/feed
```
!!! note "Logging belongs to the app, not the library"
!!! info "Logging belongs to the app, not the library"
Libraries **emit only** — `log = logging.getLogger(__name__)` and nothing
else. Handlers, levels, and formatting are configured once at the
application entry point, so a lib never dictates how its host logs.
## Service compose
A deployable service ships a `compose.yaml` that names **nothing repo-specific** —
the deploy layer injects identity and host paths. See the
[Deploy guide](deploy.md) for the full convention and the variables
you can rely on. The short version:
=== "Right"
```yaml
services:
svc:
user: "1337:1337"
restart: unless-stopped
volumes:
- ${LOGS_DIR:-./logs}:/app/logs
- profile:/app/profile
volumes:
profile:
```
=== "Wrong (causes collisions)"
```yaml
services:
nova:
container_name: nova # repo-specific name -> collides
volumes:
- /srv/logs/ricky/nova:/app/logs # hardcoded host path
- nova_profile:/app/profile # repo-prefixed volume
volumes:
nova_profile:
```
Generic service key `svc`, no `container_name`, host paths from `${...}` variables,
and **bare** volume names — that's what makes a service collision-proof on the
fleet.
+51 -6
View File
@@ -1,10 +1,10 @@
# Workflow
How we actually work day to day at Rethink Studios — where code lives, how we use
git, the AI-assisted dev flow we recommend, and the shell setup that ties it
together. This is the **our-flavored** version: why *we* do it this way and how
*our* setup is wired. For the truly generic parts (installing WSL, learning git),
we link the official docs rather than reteach them.
Get hands on with how we dev — where code lives, how we use git, the AI-assisted
flow we recommend, and the shell setup that ties it together. This is the
**our-flavored** version: why *we* do it this way and how *our* setup is wired. For
the truly generic parts (installing WSL, learning git), we link the official docs
rather than reteach them.
!!! info "Public, sanitized"
Examples use placeholders — `<you>`, `<key>`, `dev@<you>`, `/mnt/c/<your>/...`.
@@ -56,7 +56,7 @@ with the right one per project.
We solve that with **per-repo local git config** — run a small alias inside a repo
to set its local user and the SSH key it pushes with (see
[per-project git identity](#per-project-git-identity) below). No global identity
[per-project git identity](#handy-shell-setup) below). No global identity
juggling.
Our conventions, in short:
@@ -241,3 +241,48 @@ gitcs() {
- **`gl`** — a readable branch graph for understanding history at a glance.
- **`pyenv`** — per-project Python versions, so each repo builds against the
version it targets.
## Paste service
A shared, self-hosted pastebin at
[paste.rethinkstudios.io](https://paste.rethinkstudios.io) — for quickly sharing
logs, snippets, or command output when you're pairing, filing an issue, or handing
output to a coding agent. Pastes are **unlisted** (random URL), **expire** by
default, and support **burn-after-read**. Anonymous — no login.
The easy way: pipe anything into it and get back a URL. Drop this into your
`.zshrc` / `.bashrc` — a **function** is preferred over a plain alias because it
reads stdin cleanly and has room to grow options (needs `jq` + `curl`):
=== "Function (recommended)"
```bash
# paste stdin to the rethink paste service, print the URL. usage: cat file | pb
pb() {
jq -Rns '{text: inputs, expires: 259200}' \
| curl -s -H 'Content-Type: application/json' --data-binary @- https://paste.rethinkstudios.io/ \
| jq -r '"https://paste.rethinkstudios.io" + .path'
}
```
=== "Alias (alternative)"
Same behaviour as a one-liner — note the extra escaping the alias form needs:
```bash
alias pb="jq -Rns '{text: inputs, expires: 259200}' | curl -s -H 'Content-Type: application/json' --data-binary @- https://paste.rethinkstudios.io/ | jq -r '\"https://paste.rethinkstudios.io\" + .path'"
```
Usage — pipe any file or command output straight in:
```bash
cat latest.log | pb # -> https://paste.rethinkstudios.io/xxxxxxx
mycommand 2>&1 | pb # pipe any command's output (stderr too)
```
`expires` is in **seconds** — `259200` = 72h (the default). Change it (e.g.
`86400` for 24h) or drop the field entirely.
!!! info "Rate-limited on creation"
The service rate-limits paste **creation** (not viewing), so it's built for
occasional shares — not bulk or automated posting.
+10 -2
View File
@@ -6,11 +6,15 @@ copyright: rethink development (handbook)
extra_css:
- stylesheets/extra.css
extra_javascript:
- javascripts/mermaid.min.js
- javascripts/mermaid-init.js
theme:
name: material
language: en
logo: assets/logo.svg
favicon: assets/logo.svg
favicon: assets/favicon.svg
palette:
scheme: slate
primary: custom
@@ -35,7 +39,11 @@ markdown_extensions:
- toc:
permalink: true
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_div_format
- pymdownx.tabbed:
alternate_style: true
- pymdownx.highlight: