monitor-shared-utils/CLAUDE.md

# CLAUDE.md — Profit Lounge monitor

Orientation for working in this repo. This is **one monitor (or the Discord bot)
in a polyrepo system** that scrapes a retailer, normalizes its products into a
shared `Product` model, writes them to MongoDB, and lets a Discord bot consume
them and post deals. Shared code lives in the installable `monitor_utils`
package; each repo pins its own version of it.

This file is portable — most of it is true in **every** repo. Only the
**Per-repo facts** block at the top changes between monitors.

## Per-repo facts (the only section that varies)

<!-- TODO per repo: fill these in. Everything below this block is shared. -->

- **Monitor name:** `<bestbuy | hp | lg | samsung | woot | northerntool>` (the
  string passed to `configure_logging()` and `setup_config()`).
- **Entry script:** `<name>_monitor.py` (e.g. `lg_monitor.py`).
- **`category_ids` shape:** list of `<ints | category codes | category paths>`
  — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses
  category paths, Woot uses category names).
- **Pagination mode:** `<page | offset>`.
- **Site-specific quirks:** `<record known data hazards here — see HP/Woot notes
  in "Known data hazards" for the pattern.>`

## Architecture (settled — do not relitigate)

- **Polyrepo, not monorepo.** Each retailer monitor and the Discord bot is its
  own repo. Anything shared across them lives in **`monitor_utils`**, an
  installable package consumed as a pinned dependency. `monitor_utils` is
  **public** (the Gitea repo is public), so no token is needed to install it.
- **`monitor_utils` is pinned per-repo and upgraded manually.** A repo gets a
  specific pinned version via `uv.lock`. To pick up changes, bump it
  deliberately: `uv lock --upgrade-package monitor-utils` then rebuild. There is
  no auto-latest — pinning is intentional for reproducible builds.
- **Each repo builds its own Docker image** that installs the pinned
  `monitor_utils`. The Dockerfile installs from `uv.lock`; Docker never reads
  local editable installs (those are dev-only — see Local dev).
- **No `.env` files in any form.** Not committed, not copied into images, not
  mounted, not via `env_file`. Configuration that varies by environment is
  resolved through `monitor_utils.config` and the `DevelopmentMode` enum, not
  env files. (One narrow exception used in dev only: the `log_dir` argument to
  `configure_logging`.)

## `monitor_utils` (the shared package)

- **`runner.py`** — the monitor execution contract. Exposes:
  - `setup_config(monitor_name, development_mode)` → returns the triple
    `(category_identifiers, channel_ids, collection)`. It reads the JSON config
    via `get_config` and returns `config["categories"]`, `config["channels"]`,
    and the Mongo collection. **`categories` can be a list OR a dict** — when it's
    a dict (e.g. Best Buy), `execute_monitor_logic` iterates `.items()` and passes
    the value as `query_category_id`. The collection is the **monitor name in
    production, but `f"{monitor_name}_testing"` in development** — dev writes to a
    separate `_testing` collection.
  - `execute_monitor_logic(category_identifiers, task_func, headers=None,
    cookies=None, pre_task_func=None, use_tls=False, **kwargs)` — **runs forever in
    a `while True` loop**, recreating the session each pass and sleeping 0.30s
    between passes. Each pass: opens a session (`get_tls_session` if `use_tls` else
    `get_base_session`), calls optional `pre_task_func(session=session, **kwargs)`
    once, then builds per-category tasks and `asyncio.gather`s them. If
    `category_identifiers` is a **dict**, each task gets
    `task_func(category_id, session, query_category_id=query, **kwargs)`; if a
    **list**, `task_func(category_identifier, session, **kwargs)`. Results:
    `EXPECTED_TASK_ERRORS` → `LOGGER.warning` + continue; any other `Exception` →
    `LOGGER.error` + **raise** (kills the loop). `task_func` is called with the
    category as the **first positional arg** and `session` second.
  - `execute_pagination(category_identifier, total_products, products_per_page,
    func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode=
    "offset", **kwargs)` — paginates pages 2..N concurrently under a semaphore.
    **`pagination_mode="offset"`** calls `func(session=session, offset=...,
    **kwargs)` with `offset = start_offset + page_index * products_per_page`;
    **`"page"`** calls `func(session=session, page_number=page_index, **kwargs)`.
    Failed pages are retried (`retry_failed_pages`, exp backoff, 3 attempts).
    `stop_event` is **internal** to this function (early-stop plumbing) — it is
    NOT a parameter you pass in.
  - **`build_product_details(product_dict, channel_ids, min_percentage,
    min_price_drop=None)`** (consumer side) — takes a **Mongo doc** (camelCase
    keys) and returns a `Product` or `None`. Computes `percentage`, applies the
    two-rule gate (`percentage >= min_percentage` OR price-drop ≥ `min_price_drop`),
    and sets `metadata.channel_id` via `get_channel_id`. Returns `None` when the
    product fails the gate or has no/zero `originalPrice`.
  - **`get_channel_id(product, channel_ids, use_price_drop_bucket=False)`** —
    routing precedence: app-exclusive → sold-out → ending-within-1-day event →
    `FLAG_WEBHOOK_MAP` flags (member pick/deal) → percentage bucket
    (`BUCKET_WEBHOOK_MAP`, capped at 90). Used by the consumer, not the scrapers.
- **`config.py`** — configuration + environment. Holds: `get_config` (reads JSON
  configs — **production path `PROJECT_ROOT/configs/<name>.json`, dev path
  `PROJECT_ROOT/tests/configs/<name>.json`**), `get_bot_token` (by
  `is_production` bool), **the logging setup (`configure_logging`, the
  `JSONFormatter`) — logging lives HERE, there is no `observability.py`**,
  `get_proxies`, the `DevelopmentMode` enum (a `str` Enum:
  `PRODUCTION`/`DEVELOPMENT`), and `is_production`.
  - `configure_logging(monitor_name, log_dir="/logs")` clears root handlers, adds
    a `StreamHandler` **and** a `RotatingFileHandler`
    (`<log_dir>/<monitor_name>.jsonl`, 10MB × 5 backups), both using
    `JSONFormatter`, and pins `aiohttp`/`pymongo` to WARNING. A filter injects
    **`monitor_name`** onto every record.
  - **The `JSONFormatter` emits `time`, `level`, `logger`, `function`, `message`,
    then merges any non-reserved `extra` keys**, and adds `exc` on exceptions. So
    the injected field is literally **`monitor_name`** (not `monitor`), and your
    `extra={...}` keys appear at the top level of the JSON.
  - `get_proxies` parses `proxies.txt`: a line `host:port:user:password` →
    `http://user:password@host:port`; otherwise the line is used as
    `http://<line>`. Cached; empty list raises.
- **`database.py`** — Mongo controls. `add_products_to_database(products,
  collection, category_identifier)` upserts with a **composite `_id` of
  `f"{product_id}_{category_identifier}"`** (NOT bare product_id — the same
  product under two categories is two docs). It compares new vs existing
  `currentPrice`: a **new product or a price drop** sets `status=pending` (and
  `priceBeforeDrop`); `discoveredDate` is `$setOnInsert` only. `bulk_write(...,
  ordered=False)`. `get_product_from_database(collection, product_id)` is a bare
  `find_one({"_id": product_id})` — note it does NOT apply the composite-id scheme,
  so callers using it (HP) must pass whatever id matches their stored `_id`.
  `convert_product_to_mongo` maps the dataclass (snake_case) to the stored Mongo
  doc (camelCase: `currentPrice`, `imageUrl`, `appExclusive`, …) — this snake→camel
  mapping is the boundary; the `Product` model is snake_case, Mongo is camelCase.
- **`dsql_shortlinks.py`** — affiliate short-link client (`generate_short_link`
  async / `generate_short_link_sync`). Posts to the Profit Lounge affiliate API;
  on any failure it **returns the original url** (`{"short_url": url}`) rather than
  raising — link generation never blocks a post.
- **`sessions.py`** — HTTP session creation + the `send_request` wrapper.
  `get_base_session` (aiohttp) and `get_tls_session` (curl_cffi `AsyncSession`
  with `impersonate=random.choice(CLIENT_IDENTIFIERS)`) are async context
  managers; `execute_monitor_logic` picks via `use_tls`. `curl_cffi` replaced the
  old `noble_tls` for Python 3.13-compatible TLS fingerprint spoofing.
  - `send_request(session, url, method, params=None, json_data=None, data=None,
    reraise_json_error=False, development_proxy=None)` — tenacity-retried
    (`RETRY_LIMIT`× on network/timeout errors). Picks a **random proxy** from
    `get_proxies()` unless `development_proxy` is set (which also disables TLS
    verify — that's the local-mitmproxy path, e.g. HP's `http://localhost:8001`).
    Returns parsed JSON, or the raw text on `JSONDecodeError` **unless
    `reraise_json_error=True`** (then it raises — used where a non-JSON response
    must be treated as a failure). Branches on session type internally; callers
    just pass the session. **`use_tls` is a parameter of `execute_monitor_logic`,
    not of `send_request`.**
- **`models.py`** — the shared product schema (see Models).
- **`constants.py`** — fixed values + **secrets currently live here**:
  `MONGO_URL`, `PRODUCTION_BOT_TOKEN`/`DEVELOPMENT_BOT_TOKEN`, `PROFIT_API_KEY`,
  plus `EXPECTED_TASK_ERRORS` (the known-error tuple: curl_cffi `RequestsError`,
  `aiohttp.ClientError`, `asyncio.TimeoutError`, `requests.Timeout/HTTPError`,
  `PyMongoError`, `JSONDecodeError`), status constants
  (`STATUS_PENDING/SENT/SKIPPED/FAILED`), delays (`BACKOFF_DELAY`/`FAILURE_DELAY`
  — both `0.001`), `RETRY_LIMIT`/`RETRY_WAIT`, `LOGGING_LEVEL`, `CLIENT_IDENTIFIERS`,
  `FLAG_WEBHOOK_MAP`, `BUCKET_WEBHOOK_MAP`, and `PROJECT_ROOT`/`PROXIES_PATH`.

## Models (`monitor_utils.models`, pydantic dataclasses)

`Product` is the normalized shape every monitor produces and the consumer reads.
Built with `pydantic.dataclasses.dataclass` (validation on construction — a bad
field raises at build time, which is the boundary where bad data should be
caught).

- `Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…)`
  — `metadata`/`timing`/`availability` default via `field(default_factory=…)`.
- `PricingInfo(original_price, current_price, price_before_drop=None,
  edu_price=None)`.
- `AvailabilityInfo(available=None, condition=None, app_exclusive=None,
  member_pick=None, member_deal=None)`.
- `ProductMetadata(image_url, channel_id, customization_index,
  member_non_member_comparison, overstock_text, percentage, affiliate_url —
  all optional)`.
- `ProductTiming(discovered_date, start_date, end_date — all optional)`.
- `ConsumerConfig(monitor_name, collection, channel_ids, min_percentage,
  ping_threshold=None)` — one per retailer for the Discord consumer.
  `ping_threshold` is the price floor for the secondary 40% rule (Woot pings on
  40%+ if the item is ≥ this price); `None` disables it.

## Standard monitor flow (the canonical shape — every monitor follows it)

```
configure_logging(monitor_name="<name>")          # logging first, from config.py
DEVELOPMENT_MODE = DevelopmentMode.<MODE>          # set in THIS script (see below)
category_ids, _, collection = setup_config(        # second arg is positional
    monitor_name="<name>", development_mode=DEVELOPMENT_MODE
)

async def main():
    await execute_monitor_logic(                    # runs forever (while True)
        category_ids,                               # list OR dict (dict → query_category_id)
        task_func=get_category_products,
        pre_task_func=<optional>,                   # e.g. exclusive deals, called once/pass
        headers=..., cookies=..., use_tls=...,
    )

# category first, session second — matches how execute_monitor_logic calls it.
# pagination passes offset= or page_number= as a kwarg, so accept it with a default.
async def get_category_products(category_id, session, offset=<n>):   # or page_number=<n>
    response = await send_request(session, url=..., method=...)
    products = await extract_products(response, ...)   # parse → build Product(s)
    await add_products_to_database(products, collection, category_identifier=category_id)
    if <first page/offset>:                            # only kick pagination once
        await execute_pagination(
            category_identifier=category_id,
            total_products=response[...],
            products_per_page=<n>,
            func=get_category_products,
            session=session,
            pagination_mode="page"|"offset",
            category_id=category_id,                    # flows through **kwargs
        )

asyncio.run(main())
```

Note: `execute_monitor_logic` already loops forever and recreates the session each
pass — the monitor does NOT add its own outer loop. A category task raising a
non-`EXPECTED_TASK_ERRORS` exception kills the whole loop (intentional fail-loud).

**Where `DevelopmentMode` is set:**
- **Monitors:** in the monitor's own entry script (`<name>_monitor.py`).
- **Discord bot:** in `discord_bot/helpers/config.py` (multiple cogs depend on
  it, so it's defined once there, not per-cog).

## Critical invariants (do not break)

- **Structured logging only — never f-string interpolation in log calls.** Use
  `LOGGER.info("message", extra={...})`. Standard `extra` keys, kept consistent
  for queryability: `product_id`, `category`, `page`/`offset`, `count`,
  `config_id`. The message is a short static string; variable data goes in
  `extra`. `configure_logging(monitor_name)` injects **`monitor_name`** on every
  record via a filter (so don't also pass `monitor_name` in `extra`) and emits
  JSON. The `JSONFormatter` reserves the standard `LogRecord` attrs and merges
  everything else, so your `extra` keys land at the JSON top level. Pick one name
  per value (`product_id`, not a mix of `sku`/`product_id`).
- **Two-tier error handling.** Known/expected failures → `LOGGER.warning` (or
  `.exception` for tracebacks) **and continue**. Unknown failures →
  `LOGGER.exception` **and raise**. `EXPECTED_TASK_ERRORS` (in `constants.py`) is
  the known-error tuple caught throughout.
- **One bad product must never kill a category.** Per-product build is wrapped in
  `try/except EXPECTED_TASK_ERRORS` (log with `product_id`, `continue`). Guard
  indexing into possibly-empty lists (`if not items:` before `items[0]`) — a
  single malformed item silently skips, it does not crash the run. (This is why
  the Woot `IndexError` on empty `Attributes` took the monitor down before the
  guard was added.)
- **Validate implausible data at the boundary, skip per-item, don't emit.** Bad
  upstream values (negative/zero prices) are rejected where they're read by
  raising `ValueError`, caught per-item in the loop so other items still process
  — never let a false deal reach the consumer. (See HP negative-cart note.)
- **Mongo `_id` is composite: `f"{product_id}_{category_identifier}"`.** The same
  product under two categories is two documents. `add_products_to_database` builds
  this; anything querying products by id must account for it. (`Product.product_id`
  itself stays the bare retailer id — the composite only exists in the stored doc.)
- **Dev writes to a separate `_testing` collection.** `setup_config` returns the
  `<name>_testing` collection in development mode and `<name>` in production, and
  reads configs from `tests/configs/` vs `configs/`. So running in development
  never touches production data.
- **`Product` is snake_case; the Mongo doc is camelCase.** `convert_product_to_mongo`
  is the only translation point. Don't hand-write camelCase into a `Product` or
  snake_case into a Mongo filter.
- **Logs write to `/logs/<name>.jsonl`** inside the container, host-mounted to a
  shared folder. The host folder must be owned by the container user (UID 1000) —
  bind mounts override the image's ownership, so `chown -R 1000:1000 <host log
  dir>` is required on the host, NOT fixable in the Dockerfile alone.

## Coding conventions

- **Python 3.10+**, `src/` layout where applicable, type hints on public funcs.
- **Functional-first.** Avoid OOP unless a framework requires it (discord.py
  cogs). Prefer plain functions and module-level helpers.
- **Direct dict access `[]` over `.get()`** for controlled schemas where the key
  is guaranteed; use `.get()` only where a key is genuinely optional.
- **Required dataclass fields over optional-with-defaults** when the field is
  always present.
- **Doc-strings only — do not over-comment.** No narrating obvious code; a
  doc-string on a function is enough. Doc-strings start lowercase, no trailing
  period.
- **No trailing whitespace; trailing newline on every file; 4-space indent.**
- **No hardcoded paths or secrets in code** — tunable/env-varying state goes
  through `config.py`; fixed values go in `constants.py`.
- **Terse, direct edits.** When changing code, return the change without
  surrounding explanation unless options are being compared.

## Local dev (debugger uses local `monitor_utils`, Docker does not)

- To make the debugger / interpreter use your **local** `monitor_utils` working
  copy instead of the pinned site-packages build, editable-install it into the
  repo's venv: `uv pip install -e <path to monitor-shared-utils>`. This touches
  only the local venv — it does NOT modify `pyproject.toml` / `uv.lock`, so the
  Docker build still installs the pinned published version.
- Verify which copy is loaded:
  `python -c "import monitor_utils.config as c; print(c.__file__)"` — should point
  at the local clone, not `site-packages`.
- VS Code: select the repo's `.venv` interpreter; run via a launch config with
  `"module": "<name>_monitor"` (or the bot's `module`) and `"cwd":
  "${workspaceFolder}"`, not the editor's run-file button (which runs the file
  directly and breaks package imports).
- `log_dir` for `configure_logging` defaults to `/logs` (container). Locally,
  either create the dir or pass a local `log_dir`; do not commit a local override
  into the entry script.

## Discord bot (when this repo is the bot)

- **Cogs architecture.** `controller.py` is the entry; cogs live in
  `discord_bot/cogs/` (e.g. `channels.py`, `consumer.py`). Commands are cog
  methods; `DevelopmentMode`/`IS_PRODUCTION` come from
  `discord_bot/helpers/config.py`.
- **Consumer** reads pending products per monitor from Mongo
  (`find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)`),
  builds embeds, posts, and updates status. A compound index on
  `(status, discoveredDate)` backs this query (ESR: equality field first, sort
  field second) — create it at startup; it is idempotent.
- **Stop/start control is via a DB flag, NOT the Docker socket.** Server
  hardening blocks Docker socket access, so the consumer (and, where wired, the
  scraper loops) checks a `monitor_control` collection (one doc per monitor,
  keyed by name) each pass and goes idle on `stop: true` without exiting. The bot
  flips the flag through a command; readers and writers communicate only through
  Mongo. Do not reintroduce Docker-socket control.
- **Owner/admin gating.** Use a check that passes for the bot owner (global,
  independent of server perms — works even in servers where you have none) OR a
  guild admin. Prefer a `app_commands.check` decorator routing failures through
  `on_app_command_error` over inline per-command guards.

## Known data hazards (record per-repo as you find them)

- **HP — negative/zero cart totals on CTO products.** HP's cart API returns
  genuinely negative `grandTotal` for some configure-to-order configs (promo
  components exceeding base, or a config that doesn't register a valid BOM).
  Guard `grand_total <= 0` at the cart boundary; raise and skip that config in
  the pricing loop so other configs still price. Also: configurator picks must
  include every option's path (don't drop `display: false` options from the
  path map) and the base SKU, or HP misprices.
- **Woot — empty `Attributes`/`Items` lists.** `parse_condition` must guard
  `if not items` / `if not attributes` before indexing `[0]`; an empty list
  otherwise raises `IndexError` and (without the per-product try/except) kills
  the category.

<!-- TODO per repo: add this monitor's own observed hazards here. -->