# CLAUDE.md — Profit Lounge monitor Orientation for working in this repo. This is **one monitor (or the Discord bot) in a polyrepo system** that scrapes a retailer, normalizes its products into a shared `Product` model, writes them to MongoDB, and lets a Discord bot consume them and post deals. Shared code lives in the installable `monitor_utils` package; each repo pins its own version of it. This file is portable — most of it is true in **every** repo. Only the **Per-repo facts** block at the top changes between monitors. ## Per-repo facts (the only section that varies) - **Monitor name:** `` (the string passed to `configure_logging()` and `setup_config()`). - **Entry script:** `_monitor.py` (e.g. `lg_monitor.py`). - **`category_ids` shape:** list of `` — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses category paths, Woot uses category names). - **Pagination mode:** ``. - **Site-specific quirks:** `` ## Architecture (settled — do not relitigate) - **Polyrepo, not monorepo.** Each retailer monitor and the Discord bot is its own repo. Anything shared across them lives in **`monitor_utils`**, an installable package consumed as a pinned dependency. `monitor_utils` is **public** (the Gitea repo is public), so no token is needed to install it. - **`monitor_utils` is pinned per-repo and upgraded manually.** A repo gets a specific pinned version via `uv.lock`. To pick up changes, bump it deliberately: `uv lock --upgrade-package monitor-utils` then rebuild. There is no auto-latest — pinning is intentional for reproducible builds. - **Each repo builds its own Docker image** that installs the pinned `monitor_utils`. The Dockerfile installs from `uv.lock`; Docker never reads local editable installs (those are dev-only — see Local dev). - **No `.env` files in any form.** Not committed, not copied into images, not mounted, not via `env_file`. Configuration that varies by environment is resolved through `monitor_utils.config` and the `DevelopmentMode` enum, not env files. (One narrow exception used in dev only: the `log_dir` argument to `configure_logging`.) ## `monitor_utils` (the shared package) - **`runner.py`** — the monitor execution contract. Exposes: - `setup_config(monitor_name, development_mode)` → returns the triple `(category_identifiers, channel_ids, collection)`. It reads the JSON config via `get_config` and returns `config["categories"]`, `config["channels"]`, and the Mongo collection. **`categories` can be a list OR a dict** — when it's a dict (e.g. Best Buy), `execute_monitor_logic` iterates `.items()` and passes the value as `query_category_id`. The collection is the **monitor name in production, but `f"{monitor_name}_testing"` in development** — dev writes to a separate `_testing` collection. - `execute_monitor_logic(category_identifiers, task_func, headers=None, cookies=None, pre_task_func=None, use_tls=False, **kwargs)` — **runs forever in a `while True` loop**, recreating the session each pass and sleeping 0.30s between passes. Each pass: opens a session (`get_tls_session` if `use_tls` else `get_base_session`), calls optional `pre_task_func(session=session, **kwargs)` once, then builds per-category tasks and `asyncio.gather`s them. If `category_identifiers` is a **dict**, each task gets `task_func(category_id, session, query_category_id=query, **kwargs)`; if a **list**, `task_func(category_identifier, session, **kwargs)`. Results: `EXPECTED_TASK_ERRORS` → `LOGGER.warning` + continue; any other `Exception` → `LOGGER.error` + **raise** (kills the loop). `task_func` is called with the category as the **first positional arg** and `session` second. - `execute_pagination(category_identifier, total_products, products_per_page, func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode= "offset", **kwargs)` — paginates pages 2..N concurrently under a semaphore. **`pagination_mode="offset"`** calls `func(session=session, offset=..., **kwargs)` with `offset = start_offset + page_index * products_per_page`; **`"page"`** calls `func(session=session, page_number=page_index, **kwargs)`. Failed pages are retried (`retry_failed_pages`, exp backoff, 3 attempts). `stop_event` is **internal** to this function (early-stop plumbing) — it is NOT a parameter you pass in. - **`build_product_details(product_dict, channel_ids, min_percentage, min_price_drop=None)`** (consumer side) — takes a **Mongo doc** (camelCase keys) and returns a `Product` or `None`. Computes `percentage`, applies the two-rule gate (`percentage >= min_percentage` OR price-drop ≥ `min_price_drop`), and sets `metadata.channel_id` via `get_channel_id`. Returns `None` when the product fails the gate or has no/zero `originalPrice`. - **`get_channel_id(product, channel_ids, use_price_drop_bucket=False)`** — routing precedence: app-exclusive → sold-out → ending-within-1-day event → `FLAG_WEBHOOK_MAP` flags (member pick/deal) → percentage bucket (`BUCKET_WEBHOOK_MAP`, capped at 90). Used by the consumer, not the scrapers. - **`config.py`** — configuration + environment. Holds: `get_config` (reads JSON configs — **production path `PROJECT_ROOT/configs/.json`, dev path `PROJECT_ROOT/tests/configs/.json`**), `get_bot_token` (by `is_production` bool), **the logging setup (`configure_logging`, the `JSONFormatter`) — logging lives HERE, there is no `observability.py`**, `get_proxies`, the `DevelopmentMode` enum (a `str` Enum: `PRODUCTION`/`DEVELOPMENT`), and `is_production`. - `configure_logging(monitor_name, log_dir="/logs")` clears root handlers, adds a `StreamHandler` **and** a `RotatingFileHandler` (`/.jsonl`, 10MB × 5 backups), both using `JSONFormatter`, and pins `aiohttp`/`pymongo` to WARNING. A filter injects **`monitor_name`** onto every record. - **The `JSONFormatter` emits `time`, `level`, `logger`, `function`, `message`, then merges any non-reserved `extra` keys**, and adds `exc` on exceptions. So the injected field is literally **`monitor_name`** (not `monitor`), and your `extra={...}` keys appear at the top level of the JSON. - `get_proxies` parses `proxies.txt`: a line `host:port:user:password` → `http://user:password@host:port`; otherwise the line is used as `http://`. Cached; empty list raises. - **`database.py`** — Mongo controls. `add_products_to_database(products, collection, category_identifier)` upserts with a **composite `_id` of `f"{product_id}_{category_identifier}"`** (NOT bare product_id — the same product under two categories is two docs). It compares new vs existing `currentPrice`: a **new product or a price drop** sets `status=pending` (and `priceBeforeDrop`); `discoveredDate` is `$setOnInsert` only. `bulk_write(..., ordered=False)`. `get_product_from_database(collection, product_id)` is a bare `find_one({"_id": product_id})` — note it does NOT apply the composite-id scheme, so callers using it (HP) must pass whatever id matches their stored `_id`. `convert_product_to_mongo` maps the dataclass (snake_case) to the stored Mongo doc (camelCase: `currentPrice`, `imageUrl`, `appExclusive`, …) — this snake→camel mapping is the boundary; the `Product` model is snake_case, Mongo is camelCase. - **`dsql_shortlinks.py`** — affiliate short-link client (`generate_short_link` async / `generate_short_link_sync`). Posts to the Profit Lounge affiliate API; on any failure it **returns the original url** (`{"short_url": url}`) rather than raising — link generation never blocks a post. - **`sessions.py`** — HTTP session creation + the `send_request` wrapper. `get_base_session` (aiohttp) and `get_tls_session` (curl_cffi `AsyncSession` with `impersonate=random.choice(CLIENT_IDENTIFIERS)`) are async context managers; `execute_monitor_logic` picks via `use_tls`. `curl_cffi` replaced the old `noble_tls` for Python 3.13-compatible TLS fingerprint spoofing. - `send_request(session, url, method, params=None, json_data=None, data=None, reraise_json_error=False, development_proxy=None)` — tenacity-retried (`RETRY_LIMIT`× on network/timeout errors). Picks a **random proxy** from `get_proxies()` unless `development_proxy` is set (which also disables TLS verify — that's the local-mitmproxy path, e.g. HP's `http://localhost:8001`). Returns parsed JSON, or the raw text on `JSONDecodeError` **unless `reraise_json_error=True`** (then it raises — used where a non-JSON response must be treated as a failure). Branches on session type internally; callers just pass the session. **`use_tls` is a parameter of `execute_monitor_logic`, not of `send_request`.** - **`models.py`** — the shared product schema (see Models). - **`constants.py`** — fixed values + **secrets currently live here**: `MONGO_URL`, `PRODUCTION_BOT_TOKEN`/`DEVELOPMENT_BOT_TOKEN`, `PROFIT_API_KEY`, plus `EXPECTED_TASK_ERRORS` (the known-error tuple: curl_cffi `RequestsError`, `aiohttp.ClientError`, `asyncio.TimeoutError`, `requests.Timeout/HTTPError`, `PyMongoError`, `JSONDecodeError`), status constants (`STATUS_PENDING/SENT/SKIPPED/FAILED`), delays (`BACKOFF_DELAY`/`FAILURE_DELAY` — both `0.001`), `RETRY_LIMIT`/`RETRY_WAIT`, `LOGGING_LEVEL`, `CLIENT_IDENTIFIERS`, `FLAG_WEBHOOK_MAP`, `BUCKET_WEBHOOK_MAP`, and `PROJECT_ROOT`/`PROXIES_PATH`. ## Models (`monitor_utils.models`, pydantic dataclasses) `Product` is the normalized shape every monitor produces and the consumer reads. Built with `pydantic.dataclasses.dataclass` (validation on construction — a bad field raises at build time, which is the boundary where bad data should be caught). - `Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…)` — `metadata`/`timing`/`availability` default via `field(default_factory=…)`. - `PricingInfo(original_price, current_price, price_before_drop=None, edu_price=None)`. - `AvailabilityInfo(available=None, condition=None, app_exclusive=None, member_pick=None, member_deal=None)`. - `ProductMetadata(image_url, channel_id, customization_index, member_non_member_comparison, overstock_text, percentage, affiliate_url — all optional)`. - `ProductTiming(discovered_date, start_date, end_date — all optional)`. - `ConsumerConfig(monitor_name, collection, channel_ids, min_percentage, ping_threshold=None)` — one per retailer for the Discord consumer. `ping_threshold` is the price floor for the secondary 40% rule (Woot pings on 40%+ if the item is ≥ this price); `None` disables it. ## Standard monitor flow (the canonical shape — every monitor follows it) ``` configure_logging(monitor_name="") # logging first, from config.py DEVELOPMENT_MODE = DevelopmentMode. # set in THIS script (see below) category_ids, _, collection = setup_config( # second arg is positional monitor_name="", development_mode=DEVELOPMENT_MODE ) async def main(): await execute_monitor_logic( # runs forever (while True) category_ids, # list OR dict (dict → query_category_id) task_func=get_category_products, pre_task_func=, # e.g. exclusive deals, called once/pass headers=..., cookies=..., use_tls=..., ) # category first, session second — matches how execute_monitor_logic calls it. # pagination passes offset= or page_number= as a kwarg, so accept it with a default. async def get_category_products(category_id, session, offset=): # or page_number= response = await send_request(session, url=..., method=...) products = await extract_products(response, ...) # parse → build Product(s) await add_products_to_database(products, collection, category_identifier=category_id) if : # only kick pagination once await execute_pagination( category_identifier=category_id, total_products=response[...], products_per_page=, func=get_category_products, session=session, pagination_mode="page"|"offset", category_id=category_id, # flows through **kwargs ) asyncio.run(main()) ``` Note: `execute_monitor_logic` already loops forever and recreates the session each pass — the monitor does NOT add its own outer loop. A category task raising a non-`EXPECTED_TASK_ERRORS` exception kills the whole loop (intentional fail-loud). **Where `DevelopmentMode` is set:** - **Monitors:** in the monitor's own entry script (`_monitor.py`). - **Discord bot:** in `discord_bot/helpers/config.py` (multiple cogs depend on it, so it's defined once there, not per-cog). ## Critical invariants (do not break) - **Structured logging only — never f-string interpolation in log calls.** Use `LOGGER.info("message", extra={...})`. Standard `extra` keys, kept consistent for queryability: `product_id`, `category`, `page`/`offset`, `count`, `config_id`. The message is a short static string; variable data goes in `extra`. `configure_logging(monitor_name)` injects **`monitor_name`** on every record via a filter (so don't also pass `monitor_name` in `extra`) and emits JSON. The `JSONFormatter` reserves the standard `LogRecord` attrs and merges everything else, so your `extra` keys land at the JSON top level. Pick one name per value (`product_id`, not a mix of `sku`/`product_id`). - **Two-tier error handling.** Known/expected failures → `LOGGER.warning` (or `.exception` for tracebacks) **and continue**. Unknown failures → `LOGGER.exception` **and raise**. `EXPECTED_TASK_ERRORS` (in `constants.py`) is the known-error tuple caught throughout. - **One bad product must never kill a category.** Per-product build is wrapped in `try/except EXPECTED_TASK_ERRORS` (log with `product_id`, `continue`). Guard indexing into possibly-empty lists (`if not items:` before `items[0]`) — a single malformed item silently skips, it does not crash the run. (This is why the Woot `IndexError` on empty `Attributes` took the monitor down before the guard was added.) - **Validate implausible data at the boundary, skip per-item, don't emit.** Bad upstream values (negative/zero prices) are rejected where they're read by raising `ValueError`, caught per-item in the loop so other items still process — never let a false deal reach the consumer. (See HP negative-cart note.) - **Mongo `_id` is composite: `f"{product_id}_{category_identifier}"`.** The same product under two categories is two documents. `add_products_to_database` builds this; anything querying products by id must account for it. (`Product.product_id` itself stays the bare retailer id — the composite only exists in the stored doc.) - **Dev writes to a separate `_testing` collection.** `setup_config` returns the `_testing` collection in development mode and `` in production, and reads configs from `tests/configs/` vs `configs/`. So running in development never touches production data. - **`Product` is snake_case; the Mongo doc is camelCase.** `convert_product_to_mongo` is the only translation point. Don't hand-write camelCase into a `Product` or snake_case into a Mongo filter. - **Logs write to `/logs/.jsonl`** inside the container, host-mounted to a shared folder. The host folder must be owned by the container user (UID 1000) — bind mounts override the image's ownership, so `chown -R 1000:1000 ` is required on the host, NOT fixable in the Dockerfile alone. ## Coding conventions - **Python 3.10+**, `src/` layout where applicable, type hints on public funcs. - **Functional-first.** Avoid OOP unless a framework requires it (discord.py cogs). Prefer plain functions and module-level helpers. - **Direct dict access `[]` over `.get()`** for controlled schemas where the key is guaranteed; use `.get()` only where a key is genuinely optional. - **Required dataclass fields over optional-with-defaults** when the field is always present. - **Doc-strings only — do not over-comment.** No narrating obvious code; a doc-string on a function is enough. Doc-strings start lowercase, no trailing period. - **No trailing whitespace; trailing newline on every file; 4-space indent.** - **No hardcoded paths or secrets in code** — tunable/env-varying state goes through `config.py`; fixed values go in `constants.py`. - **Terse, direct edits.** When changing code, return the change without surrounding explanation unless options are being compared. ## Local dev (debugger uses local `monitor_utils`, Docker does not) - To make the debugger / interpreter use your **local** `monitor_utils` working copy instead of the pinned site-packages build, editable-install it into the repo's venv: `uv pip install -e `. This touches only the local venv — it does NOT modify `pyproject.toml` / `uv.lock`, so the Docker build still installs the pinned published version. - Verify which copy is loaded: `python -c "import monitor_utils.config as c; print(c.__file__)"` — should point at the local clone, not `site-packages`. - VS Code: select the repo's `.venv` interpreter; run via a launch config with `"module": "_monitor"` (or the bot's `module`) and `"cwd": "${workspaceFolder}"`, not the editor's run-file button (which runs the file directly and breaks package imports). - `log_dir` for `configure_logging` defaults to `/logs` (container). Locally, either create the dir or pass a local `log_dir`; do not commit a local override into the entry script. ## Discord bot (when this repo is the bot) - **Cogs architecture.** `controller.py` is the entry; cogs live in `discord_bot/cogs/` (e.g. `channels.py`, `consumer.py`). Commands are cog methods; `DevelopmentMode`/`IS_PRODUCTION` come from `discord_bot/helpers/config.py`. - **Consumer** reads pending products per monitor from Mongo (`find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)`), builds embeds, posts, and updates status. A compound index on `(status, discoveredDate)` backs this query (ESR: equality field first, sort field second) — create it at startup; it is idempotent. - **Stop/start control is via a DB flag, NOT the Docker socket.** Server hardening blocks Docker socket access, so the consumer (and, where wired, the scraper loops) checks a `monitor_control` collection (one doc per monitor, keyed by name) each pass and goes idle on `stop: true` without exiting. The bot flips the flag through a command; readers and writers communicate only through Mongo. Do not reintroduce Docker-socket control. - **Owner/admin gating.** Use a check that passes for the bot owner (global, independent of server perms — works even in servers where you have none) OR a guild admin. Prefer a `app_commands.check` decorator routing failures through `on_app_command_error` over inline per-command guards. ## Known data hazards (record per-repo as you find them) - **HP — negative/zero cart totals on CTO products.** HP's cart API returns genuinely negative `grandTotal` for some configure-to-order configs (promo components exceeding base, or a config that doesn't register a valid BOM). Guard `grand_total <= 0` at the cart boundary; raise and skip that config in the pricing loop so other configs still price. Also: configurator picks must include every option's path (don't drop `display: false` options from the path map) and the base SKU, or HP misprices. - **Woot — empty `Attributes`/`Items` lists.** `parse_condition` must guard `if not items` / `if not attributes` before indexing `[0]`; an empty list otherwise raises `IndexError` and (without the per-product try/except) kills the category.