diff --git a/.gitignore b/.gitignore index e1ce79c..ccce2ff 100644 --- a/.gitignore +++ b/.gitignore @@ -3,5 +3,4 @@ __pycache__ *.pyc .vscode .idea -*.md monitor_utils.egg-info \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..859ec05 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,328 @@ +# CLAUDE.md — Profit Lounge monitor + +Orientation for working in this repo. This is **one monitor (or the Discord bot) +in a polyrepo system** that scrapes a retailer, normalizes its products into a +shared `Product` model, writes them to MongoDB, and lets a Discord bot consume +them and post deals. Shared code lives in the installable `monitor_utils` +package; each repo pins its own version of it. + +This file is portable — most of it is true in **every** repo. Only the +**Per-repo facts** block at the top changes between monitors. + +## Per-repo facts (the only section that varies) + + + +- **Monitor name:** `` (the + string passed to `configure_logging()` and `setup_config()`). +- **Entry script:** `_monitor.py` (e.g. `lg_monitor.py`). +- **`category_ids` shape:** list of `` + — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses + category paths, Woot uses category names). +- **Pagination mode:** ``. +- **Site-specific quirks:** `` + +## Architecture (settled — do not relitigate) + +- **Polyrepo, not monorepo.** Each retailer monitor and the Discord bot is its + own repo. Anything shared across them lives in **`monitor_utils`**, an + installable package consumed as a pinned dependency. `monitor_utils` is + **public** (the Gitea repo is public), so no token is needed to install it. +- **`monitor_utils` is pinned per-repo and upgraded manually.** A repo gets a + specific pinned version via `uv.lock`. To pick up changes, bump it + deliberately: `uv lock --upgrade-package monitor-utils` then rebuild. There is + no auto-latest — pinning is intentional for reproducible builds. +- **Each repo builds its own Docker image** that installs the pinned + `monitor_utils`. The Dockerfile installs from `uv.lock`; Docker never reads + local editable installs (those are dev-only — see Local dev). +- **No `.env` files in any form.** Not committed, not copied into images, not + mounted, not via `env_file`. Configuration that varies by environment is + resolved through `monitor_utils.config` and the `DevelopmentMode` enum, not + env files. (One narrow exception used in dev only: the `log_dir` argument to + `configure_logging`.) + +## `monitor_utils` (the shared package) + +- **`runner.py`** — the monitor execution contract. Exposes: + - `setup_config(monitor_name, development_mode)` → returns the triple + `(category_identifiers, channel_ids, collection)`. It reads the JSON config + via `get_config` and returns `config["categories"]`, `config["channels"]`, + and the Mongo collection. **`categories` can be a list OR a dict** — when it's + a dict (e.g. Best Buy), `execute_monitor_logic` iterates `.items()` and passes + the value as `query_category_id`. The collection is the **monitor name in + production, but `f"{monitor_name}_testing"` in development** — dev writes to a + separate `_testing` collection. + - `execute_monitor_logic(category_identifiers, task_func, headers=None, + cookies=None, pre_task_func=None, use_tls=False, **kwargs)` — **runs forever in + a `while True` loop**, recreating the session each pass and sleeping 0.30s + between passes. Each pass: opens a session (`get_tls_session` if `use_tls` else + `get_base_session`), calls optional `pre_task_func(session=session, **kwargs)` + once, then builds per-category tasks and `asyncio.gather`s them. If + `category_identifiers` is a **dict**, each task gets + `task_func(category_id, session, query_category_id=query, **kwargs)`; if a + **list**, `task_func(category_identifier, session, **kwargs)`. Results: + `EXPECTED_TASK_ERRORS` → `LOGGER.warning` + continue; any other `Exception` → + `LOGGER.error` + **raise** (kills the loop). `task_func` is called with the + category as the **first positional arg** and `session` second. + - `execute_pagination(category_identifier, total_products, products_per_page, + func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode= + "offset", **kwargs)` — paginates pages 2..N concurrently under a semaphore. + **`pagination_mode="offset"`** calls `func(session=session, offset=..., + **kwargs)` with `offset = start_offset + page_index * products_per_page`; + **`"page"`** calls `func(session=session, page_number=page_index, **kwargs)`. + Failed pages are retried (`retry_failed_pages`, exp backoff, 3 attempts). + `stop_event` is **internal** to this function (early-stop plumbing) — it is + NOT a parameter you pass in. + - **`build_product_details(product_dict, channel_ids, min_percentage, + min_price_drop=None)`** (consumer side) — takes a **Mongo doc** (camelCase + keys) and returns a `Product` or `None`. Computes `percentage`, applies the + two-rule gate (`percentage >= min_percentage` OR price-drop ≥ `min_price_drop`), + and sets `metadata.channel_id` via `get_channel_id`. Returns `None` when the + product fails the gate or has no/zero `originalPrice`. + - **`get_channel_id(product, channel_ids, use_price_drop_bucket=False)`** — + routing precedence: app-exclusive → sold-out → ending-within-1-day event → + `FLAG_WEBHOOK_MAP` flags (member pick/deal) → percentage bucket + (`BUCKET_WEBHOOK_MAP`, capped at 90). Used by the consumer, not the scrapers. +- **`config.py`** — configuration + environment. Holds: `get_config` (reads JSON + configs — **production path `PROJECT_ROOT/configs/.json`, dev path + `PROJECT_ROOT/tests/configs/.json`**), `get_bot_token` (by + `is_production` bool), **the logging setup (`configure_logging`, the + `JSONFormatter`) — logging lives HERE, there is no `observability.py`**, + `get_proxies`, the `DevelopmentMode` enum (a `str` Enum: + `PRODUCTION`/`DEVELOPMENT`), and `is_production`. + - `configure_logging(monitor_name, log_dir="/logs")` clears root handlers, adds + a `StreamHandler` **and** a `RotatingFileHandler` + (`/.jsonl`, 10MB × 5 backups), both using + `JSONFormatter`, and pins `aiohttp`/`pymongo` to WARNING. A filter injects + **`monitor_name`** onto every record. + - **The `JSONFormatter` emits `time`, `level`, `logger`, `function`, `message`, + then merges any non-reserved `extra` keys**, and adds `exc` on exceptions. So + the injected field is literally **`monitor_name`** (not `monitor`), and your + `extra={...}` keys appear at the top level of the JSON. + - `get_proxies` parses `proxies.txt`: a line `host:port:user:password` → + `http://user:password@host:port`; otherwise the line is used as + `http://`. Cached; empty list raises. +- **`database.py`** — Mongo controls. `add_products_to_database(products, + collection, category_identifier)` upserts with a **composite `_id` of + `f"{product_id}_{category_identifier}"`** (NOT bare product_id — the same + product under two categories is two docs). It compares new vs existing + `currentPrice`: a **new product or a price drop** sets `status=pending` (and + `priceBeforeDrop`); `discoveredDate` is `$setOnInsert` only. `bulk_write(..., + ordered=False)`. `get_product_from_database(collection, product_id)` is a bare + `find_one({"_id": product_id})` — note it does NOT apply the composite-id scheme, + so callers using it (HP) must pass whatever id matches their stored `_id`. + `convert_product_to_mongo` maps the dataclass (snake_case) to the stored Mongo + doc (camelCase: `currentPrice`, `imageUrl`, `appExclusive`, …) — this snake→camel + mapping is the boundary; the `Product` model is snake_case, Mongo is camelCase. +- **`dsql_shortlinks.py`** — affiliate short-link client (`generate_short_link` + async / `generate_short_link_sync`). Posts to the Profit Lounge affiliate API; + on any failure it **returns the original url** (`{"short_url": url}`) rather than + raising — link generation never blocks a post. +- **`sessions.py`** — HTTP session creation + the `send_request` wrapper. + `get_base_session` (aiohttp) and `get_tls_session` (curl_cffi `AsyncSession` + with `impersonate=random.choice(CLIENT_IDENTIFIERS)`) are async context + managers; `execute_monitor_logic` picks via `use_tls`. `curl_cffi` replaced the + old `noble_tls` for Python 3.13-compatible TLS fingerprint spoofing. + - `send_request(session, url, method, params=None, json_data=None, data=None, + reraise_json_error=False, development_proxy=None)` — tenacity-retried + (`RETRY_LIMIT`× on network/timeout errors). Picks a **random proxy** from + `get_proxies()` unless `development_proxy` is set (which also disables TLS + verify — that's the local-mitmproxy path, e.g. HP's `http://localhost:8001`). + Returns parsed JSON, or the raw text on `JSONDecodeError` **unless + `reraise_json_error=True`** (then it raises — used where a non-JSON response + must be treated as a failure). Branches on session type internally; callers + just pass the session. **`use_tls` is a parameter of `execute_monitor_logic`, + not of `send_request`.** +- **`models.py`** — the shared product schema (see Models). +- **`constants.py`** — fixed values + **secrets currently live here**: + `MONGO_URL`, `PRODUCTION_BOT_TOKEN`/`DEVELOPMENT_BOT_TOKEN`, `PROFIT_API_KEY`, + plus `EXPECTED_TASK_ERRORS` (the known-error tuple: curl_cffi `RequestsError`, + `aiohttp.ClientError`, `asyncio.TimeoutError`, `requests.Timeout/HTTPError`, + `PyMongoError`, `JSONDecodeError`), status constants + (`STATUS_PENDING/SENT/SKIPPED/FAILED`), delays (`BACKOFF_DELAY`/`FAILURE_DELAY` + — both `0.001`), `RETRY_LIMIT`/`RETRY_WAIT`, `LOGGING_LEVEL`, `CLIENT_IDENTIFIERS`, + `FLAG_WEBHOOK_MAP`, `BUCKET_WEBHOOK_MAP`, and `PROJECT_ROOT`/`PROXIES_PATH`. + +## Models (`monitor_utils.models`, pydantic dataclasses) + +`Product` is the normalized shape every monitor produces and the consumer reads. +Built with `pydantic.dataclasses.dataclass` (validation on construction — a bad +field raises at build time, which is the boundary where bad data should be +caught). + +- `Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…)` + — `metadata`/`timing`/`availability` default via `field(default_factory=…)`. +- `PricingInfo(original_price, current_price, price_before_drop=None, + edu_price=None)`. +- `AvailabilityInfo(available=None, condition=None, app_exclusive=None, + member_pick=None, member_deal=None)`. +- `ProductMetadata(image_url, channel_id, customization_index, + member_non_member_comparison, overstock_text, percentage, affiliate_url — + all optional)`. +- `ProductTiming(discovered_date, start_date, end_date — all optional)`. +- `ConsumerConfig(monitor_name, collection, channel_ids, min_percentage, + ping_threshold=None)` — one per retailer for the Discord consumer. + `ping_threshold` is the price floor for the secondary 40% rule (Woot pings on + 40%+ if the item is ≥ this price); `None` disables it. + +## Standard monitor flow (the canonical shape — every monitor follows it) + +``` +configure_logging(monitor_name="") # logging first, from config.py +DEVELOPMENT_MODE = DevelopmentMode. # set in THIS script (see below) +category_ids, _, collection = setup_config( # second arg is positional + monitor_name="", development_mode=DEVELOPMENT_MODE +) + +async def main(): + await execute_monitor_logic( # runs forever (while True) + category_ids, # list OR dict (dict → query_category_id) + task_func=get_category_products, + pre_task_func=, # e.g. exclusive deals, called once/pass + headers=..., cookies=..., use_tls=..., + ) + +# category first, session second — matches how execute_monitor_logic calls it. +# pagination passes offset= or page_number= as a kwarg, so accept it with a default. +async def get_category_products(category_id, session, offset=): # or page_number= + response = await send_request(session, url=..., method=...) + products = await extract_products(response, ...) # parse → build Product(s) + await add_products_to_database(products, collection, category_identifier=category_id) + if : # only kick pagination once + await execute_pagination( + category_identifier=category_id, + total_products=response[...], + products_per_page=, + func=get_category_products, + session=session, + pagination_mode="page"|"offset", + category_id=category_id, # flows through **kwargs + ) + +asyncio.run(main()) +``` + +Note: `execute_monitor_logic` already loops forever and recreates the session each +pass — the monitor does NOT add its own outer loop. A category task raising a +non-`EXPECTED_TASK_ERRORS` exception kills the whole loop (intentional fail-loud). + +**Where `DevelopmentMode` is set:** +- **Monitors:** in the monitor's own entry script (`_monitor.py`). +- **Discord bot:** in `discord_bot/helpers/config.py` (multiple cogs depend on + it, so it's defined once there, not per-cog). + +## Critical invariants (do not break) + +- **Structured logging only — never f-string interpolation in log calls.** Use + `LOGGER.info("message", extra={...})`. Standard `extra` keys, kept consistent + for queryability: `product_id`, `category`, `page`/`offset`, `count`, + `config_id`. The message is a short static string; variable data goes in + `extra`. `configure_logging(monitor_name)` injects **`monitor_name`** on every + record via a filter (so don't also pass `monitor_name` in `extra`) and emits + JSON. The `JSONFormatter` reserves the standard `LogRecord` attrs and merges + everything else, so your `extra` keys land at the JSON top level. Pick one name + per value (`product_id`, not a mix of `sku`/`product_id`). +- **Two-tier error handling.** Known/expected failures → `LOGGER.warning` (or + `.exception` for tracebacks) **and continue**. Unknown failures → + `LOGGER.exception` **and raise**. `EXPECTED_TASK_ERRORS` (in `constants.py`) is + the known-error tuple caught throughout. +- **One bad product must never kill a category.** Per-product build is wrapped in + `try/except EXPECTED_TASK_ERRORS` (log with `product_id`, `continue`). Guard + indexing into possibly-empty lists (`if not items:` before `items[0]`) — a + single malformed item silently skips, it does not crash the run. (This is why + the Woot `IndexError` on empty `Attributes` took the monitor down before the + guard was added.) +- **Validate implausible data at the boundary, skip per-item, don't emit.** Bad + upstream values (negative/zero prices) are rejected where they're read by + raising `ValueError`, caught per-item in the loop so other items still process + — never let a false deal reach the consumer. (See HP negative-cart note.) +- **Mongo `_id` is composite: `f"{product_id}_{category_identifier}"`.** The same + product under two categories is two documents. `add_products_to_database` builds + this; anything querying products by id must account for it. (`Product.product_id` + itself stays the bare retailer id — the composite only exists in the stored doc.) +- **Dev writes to a separate `_testing` collection.** `setup_config` returns the + `_testing` collection in development mode and `` in production, and + reads configs from `tests/configs/` vs `configs/`. So running in development + never touches production data. +- **`Product` is snake_case; the Mongo doc is camelCase.** `convert_product_to_mongo` + is the only translation point. Don't hand-write camelCase into a `Product` or + snake_case into a Mongo filter. +- **Logs write to `/logs/.jsonl`** inside the container, host-mounted to a + shared folder. The host folder must be owned by the container user (UID 1000) — + bind mounts override the image's ownership, so `chown -R 1000:1000 ` is required on the host, NOT fixable in the Dockerfile alone. + +## Coding conventions + +- **Python 3.10+**, `src/` layout where applicable, type hints on public funcs. +- **Functional-first.** Avoid OOP unless a framework requires it (discord.py + cogs). Prefer plain functions and module-level helpers. +- **Direct dict access `[]` over `.get()`** for controlled schemas where the key + is guaranteed; use `.get()` only where a key is genuinely optional. +- **Required dataclass fields over optional-with-defaults** when the field is + always present. +- **Doc-strings only — do not over-comment.** No narrating obvious code; a + doc-string on a function is enough. Doc-strings start lowercase, no trailing + period. +- **No trailing whitespace; trailing newline on every file; 4-space indent.** +- **No hardcoded paths or secrets in code** — tunable/env-varying state goes + through `config.py`; fixed values go in `constants.py`. +- **Terse, direct edits.** When changing code, return the change without + surrounding explanation unless options are being compared. + +## Local dev (debugger uses local `monitor_utils`, Docker does not) + +- To make the debugger / interpreter use your **local** `monitor_utils` working + copy instead of the pinned site-packages build, editable-install it into the + repo's venv: `uv pip install -e `. This touches + only the local venv — it does NOT modify `pyproject.toml` / `uv.lock`, so the + Docker build still installs the pinned published version. +- Verify which copy is loaded: + `python -c "import monitor_utils.config as c; print(c.__file__)"` — should point + at the local clone, not `site-packages`. +- VS Code: select the repo's `.venv` interpreter; run via a launch config with + `"module": "_monitor"` (or the bot's `module`) and `"cwd": + "${workspaceFolder}"`, not the editor's run-file button (which runs the file + directly and breaks package imports). +- `log_dir` for `configure_logging` defaults to `/logs` (container). Locally, + either create the dir or pass a local `log_dir`; do not commit a local override + into the entry script. + +## Discord bot (when this repo is the bot) + +- **Cogs architecture.** `controller.py` is the entry; cogs live in + `discord_bot/cogs/` (e.g. `channels.py`, `consumer.py`). Commands are cog + methods; `DevelopmentMode`/`IS_PRODUCTION` come from + `discord_bot/helpers/config.py`. +- **Consumer** reads pending products per monitor from Mongo + (`find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)`), + builds embeds, posts, and updates status. A compound index on + `(status, discoveredDate)` backs this query (ESR: equality field first, sort + field second) — create it at startup; it is idempotent. +- **Stop/start control is via a DB flag, NOT the Docker socket.** Server + hardening blocks Docker socket access, so the consumer (and, where wired, the + scraper loops) checks a `monitor_control` collection (one doc per monitor, + keyed by name) each pass and goes idle on `stop: true` without exiting. The bot + flips the flag through a command; readers and writers communicate only through + Mongo. Do not reintroduce Docker-socket control. +- **Owner/admin gating.** Use a check that passes for the bot owner (global, + independent of server perms — works even in servers where you have none) OR a + guild admin. Prefer a `app_commands.check` decorator routing failures through + `on_app_command_error` over inline per-command guards. + +## Known data hazards (record per-repo as you find them) + +- **HP — negative/zero cart totals on CTO products.** HP's cart API returns + genuinely negative `grandTotal` for some configure-to-order configs (promo + components exceeding base, or a config that doesn't register a valid BOM). + Guard `grand_total <= 0` at the cart boundary; raise and skip that config in + the pricing loop so other configs still price. Also: configurator picks must + include every option's path (don't drop `display: false` options from the + path map) and the base SKU, or HP misprices. +- **Woot — empty `Attributes`/`Items` lists.** `parse_condition` must guard + `if not items` / `if not attributes` before indexing `[0]`; an empty list + otherwise raises `IndexError` and (without the per-product try/except) kills + the category. + +