added CLAUDE.md

.gitignore

@@ -3,5 +3,4 @@ __pycache__
 *.pyc
 .vscode
 .idea
-*.md
 monitor_utils.egg-info

CLAUDE.md

@@ -0,0 +1,328 @@
+# CLAUDE.md — Profit Lounge monitor
+
+Orientation for working in this repo. This is **one monitor (or the Discord bot)
+in a polyrepo system** that scrapes a retailer, normalizes its products into a
+shared `Product` model, writes them to MongoDB, and lets a Discord bot consume
+them and post deals. Shared code lives in the installable `monitor_utils`
+package; each repo pins its own version of it.
+
+This file is portable — most of it is true in **every** repo. Only the
+**Per-repo facts** block at the top changes between monitors.
+
+## Per-repo facts (the only section that varies)
+
+<!-- TODO per repo: fill these in. Everything below this block is shared. -->
+
+- **Monitor name:** `<bestbuy | hp | lg | samsung | woot | northerntool>` (the
+  string passed to `configure_logging()` and `setup_config()`).
+- **Entry script:** `<name>_monitor.py` (e.g. `lg_monitor.py`).
+- **`category_ids` shape:** list of `<ints | category codes | category paths>`
+  — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses
+  category paths, Woot uses category names).
+- **Pagination mode:** `<page | offset>`.
+- **Site-specific quirks:** `<record known data hazards here — see HP/Woot notes
+  in "Known data hazards" for the pattern.>`
+
+## Architecture (settled — do not relitigate)
+
+- **Polyrepo, not monorepo.** Each retailer monitor and the Discord bot is its
+  own repo. Anything shared across them lives in **`monitor_utils`**, an
+  installable package consumed as a pinned dependency. `monitor_utils` is
+  **public** (the Gitea repo is public), so no token is needed to install it.
+- **`monitor_utils` is pinned per-repo and upgraded manually.** A repo gets a
+  specific pinned version via `uv.lock`. To pick up changes, bump it
+  deliberately: `uv lock --upgrade-package monitor-utils` then rebuild. There is
+  no auto-latest — pinning is intentional for reproducible builds.
+- **Each repo builds its own Docker image** that installs the pinned
+  `monitor_utils`. The Dockerfile installs from `uv.lock`; Docker never reads
+  local editable installs (those are dev-only — see Local dev).
+- **No `.env` files in any form.** Not committed, not copied into images, not
+  mounted, not via `env_file`. Configuration that varies by environment is
+  resolved through `monitor_utils.config` and the `DevelopmentMode` enum, not
+  env files. (One narrow exception used in dev only: the `log_dir` argument to
+  `configure_logging`.)
+
+## `monitor_utils` (the shared package)
+
+- **`runner.py`** — the monitor execution contract. Exposes:
+  - `setup_config(monitor_name, development_mode)` → returns the triple
+    `(category_identifiers, channel_ids, collection)`. It reads the JSON config
+    via `get_config` and returns `config["categories"]`, `config["channels"]`,
+    and the Mongo collection. **`categories` can be a list OR a dict** — when it's
+    a dict (e.g. Best Buy), `execute_monitor_logic` iterates `.items()` and passes
+    the value as `query_category_id`. The collection is the **monitor name in
+    production, but `f"{monitor_name}_testing"` in development** — dev writes to a
+    separate `_testing` collection.
+  - `execute_monitor_logic(category_identifiers, task_func, headers=None,
+    cookies=None, pre_task_func=None, use_tls=False, **kwargs)` — **runs forever in
+    a `while True` loop**, recreating the session each pass and sleeping 0.30s
+    between passes. Each pass: opens a session (`get_tls_session` if `use_tls` else
+    `get_base_session`), calls optional `pre_task_func(session=session, **kwargs)`
+    once, then builds per-category tasks and `asyncio.gather`s them. If
+    `category_identifiers` is a **dict**, each task gets
+    `task_func(category_id, session, query_category_id=query, **kwargs)`; if a
+    **list**, `task_func(category_identifier, session, **kwargs)`. Results:
+    `EXPECTED_TASK_ERRORS` → `LOGGER.warning` + continue; any other `Exception` →
+    `LOGGER.error` + **raise** (kills the loop). `task_func` is called with the
+    category as the **first positional arg** and `session` second.
+  - `execute_pagination(category_identifier, total_products, products_per_page,
+    func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode=
+    "offset", **kwargs)` — paginates pages 2..N concurrently under a semaphore.
+    **`pagination_mode="offset"`** calls `func(session=session, offset=...,
+    **kwargs)` with `offset = start_offset + page_index * products_per_page`;
+    **`"page"`** calls `func(session=session, page_number=page_index, **kwargs)`.
+    Failed pages are retried (`retry_failed_pages`, exp backoff, 3 attempts).
+    `stop_event` is **internal** to this function (early-stop plumbing) — it is
+    NOT a parameter you pass in.
+  - **`build_product_details(product_dict, channel_ids, min_percentage,
+    min_price_drop=None)`** (consumer side) — takes a **Mongo doc** (camelCase
+    keys) and returns a `Product` or `None`. Computes `percentage`, applies the
+    two-rule gate (`percentage >= min_percentage` OR price-drop ≥ `min_price_drop`),
+    and sets `metadata.channel_id` via `get_channel_id`. Returns `None` when the
+    product fails the gate or has no/zero `originalPrice`.
+  - **`get_channel_id(product, channel_ids, use_price_drop_bucket=False)`** —
+    routing precedence: app-exclusive → sold-out → ending-within-1-day event →
+    `FLAG_WEBHOOK_MAP` flags (member pick/deal) → percentage bucket
+    (`BUCKET_WEBHOOK_MAP`, capped at 90). Used by the consumer, not the scrapers.
+- **`config.py`** — configuration + environment. Holds: `get_config` (reads JSON
+  configs — **production path `PROJECT_ROOT/configs/<name>.json`, dev path
+  `PROJECT_ROOT/tests/configs/<name>.json`**), `get_bot_token` (by
+  `is_production` bool), **the logging setup (`configure_logging`, the
+  `JSONFormatter`) — logging lives HERE, there is no `observability.py`**,
+  `get_proxies`, the `DevelopmentMode` enum (a `str` Enum:
+  `PRODUCTION`/`DEVELOPMENT`), and `is_production`.
+  - `configure_logging(monitor_name, log_dir="/logs")` clears root handlers, adds
+    a `StreamHandler` **and** a `RotatingFileHandler`
+    (`<log_dir>/<monitor_name>.jsonl`, 10MB × 5 backups), both using
+    `JSONFormatter`, and pins `aiohttp`/`pymongo` to WARNING. A filter injects
+    **`monitor_name`** onto every record.
+  - **The `JSONFormatter` emits `time`, `level`, `logger`, `function`, `message`,
+    then merges any non-reserved `extra` keys**, and adds `exc` on exceptions. So
+    the injected field is literally **`monitor_name`** (not `monitor`), and your
+    `extra={...}` keys appear at the top level of the JSON.
+  - `get_proxies` parses `proxies.txt`: a line `host:port:user:password` →
+    `http://user:password@host:port`; otherwise the line is used as
+    `http://<line>`. Cached; empty list raises.
+- **`database.py`** — Mongo controls. `add_products_to_database(products,
+  collection, category_identifier)` upserts with a **composite `_id` of
+  `f"{product_id}_{category_identifier}"`** (NOT bare product_id — the same
+  product under two categories is two docs). It compares new vs existing
+  `currentPrice`: a **new product or a price drop** sets `status=pending` (and
+  `priceBeforeDrop`); `discoveredDate` is `$setOnInsert` only. `bulk_write(...,
+  ordered=False)`. `get_product_from_database(collection, product_id)` is a bare
+  `find_one({"_id": product_id})` — note it does NOT apply the composite-id scheme,
+  so callers using it (HP) must pass whatever id matches their stored `_id`.
+  `convert_product_to_mongo` maps the dataclass (snake_case) to the stored Mongo
+  doc (camelCase: `currentPrice`, `imageUrl`, `appExclusive`, …) — this snake→camel
+  mapping is the boundary; the `Product` model is snake_case, Mongo is camelCase.
+- **`dsql_shortlinks.py`** — affiliate short-link client (`generate_short_link`
+  async / `generate_short_link_sync`). Posts to the Profit Lounge affiliate API;
+  on any failure it **returns the original url** (`{"short_url": url}`) rather than
+  raising — link generation never blocks a post.
+- **`sessions.py`** — HTTP session creation + the `send_request` wrapper.
+  `get_base_session` (aiohttp) and `get_tls_session` (curl_cffi `AsyncSession`
+  with `impersonate=random.choice(CLIENT_IDENTIFIERS)`) are async context
+  managers; `execute_monitor_logic` picks via `use_tls`. `curl_cffi` replaced the
+  old `noble_tls` for Python 3.13-compatible TLS fingerprint spoofing.
+  - `send_request(session, url, method, params=None, json_data=None, data=None,
+    reraise_json_error=False, development_proxy=None)` — tenacity-retried
+    (`RETRY_LIMIT`× on network/timeout errors). Picks a **random proxy** from
+    `get_proxies()` unless `development_proxy` is set (which also disables TLS
+    verify — that's the local-mitmproxy path, e.g. HP's `http://localhost:8001`).
+    Returns parsed JSON, or the raw text on `JSONDecodeError` **unless
+    `reraise_json_error=True`** (then it raises — used where a non-JSON response
+    must be treated as a failure). Branches on session type internally; callers
+    just pass the session. **`use_tls` is a parameter of `execute_monitor_logic`,
+    not of `send_request`.**
+- **`models.py`** — the shared product schema (see Models).
+- **`constants.py`** — fixed values + **secrets currently live here**:
+  `MONGO_URL`, `PRODUCTION_BOT_TOKEN`/`DEVELOPMENT_BOT_TOKEN`, `PROFIT_API_KEY`,
+  plus `EXPECTED_TASK_ERRORS` (the known-error tuple: curl_cffi `RequestsError`,
+  `aiohttp.ClientError`, `asyncio.TimeoutError`, `requests.Timeout/HTTPError`,
+  `PyMongoError`, `JSONDecodeError`), status constants
+  (`STATUS_PENDING/SENT/SKIPPED/FAILED`), delays (`BACKOFF_DELAY`/`FAILURE_DELAY`
+  — both `0.001`), `RETRY_LIMIT`/`RETRY_WAIT`, `LOGGING_LEVEL`, `CLIENT_IDENTIFIERS`,
+  `FLAG_WEBHOOK_MAP`, `BUCKET_WEBHOOK_MAP`, and `PROJECT_ROOT`/`PROXIES_PATH`.
+
+## Models (`monitor_utils.models`, pydantic dataclasses)
+
+`Product` is the normalized shape every monitor produces and the consumer reads.
+Built with `pydantic.dataclasses.dataclass` (validation on construction — a bad
+field raises at build time, which is the boundary where bad data should be
+caught).
+
+- `Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…)`
+  — `metadata`/`timing`/`availability` default via `field(default_factory=…)`.
+- `PricingInfo(original_price, current_price, price_before_drop=None,
+  edu_price=None)`.
+- `AvailabilityInfo(available=None, condition=None, app_exclusive=None,
+  member_pick=None, member_deal=None)`.
+- `ProductMetadata(image_url, channel_id, customization_index,
+  member_non_member_comparison, overstock_text, percentage, affiliate_url —
+  all optional)`.
+- `ProductTiming(discovered_date, start_date, end_date — all optional)`.
+- `ConsumerConfig(monitor_name, collection, channel_ids, min_percentage,
+  ping_threshold=None)` — one per retailer for the Discord consumer.
+  `ping_threshold` is the price floor for the secondary 40% rule (Woot pings on
+  40%+ if the item is ≥ this price); `None` disables it.
+
+## Standard monitor flow (the canonical shape — every monitor follows it)
+
+```
+configure_logging(monitor_name="<name>")          # logging first, from config.py
+DEVELOPMENT_MODE = DevelopmentMode.<MODE>          # set in THIS script (see below)
+category_ids, _, collection = setup_config(        # second arg is positional
+    monitor_name="<name>", development_mode=DEVELOPMENT_MODE
+)
+
+async def main():
+    await execute_monitor_logic(                    # runs forever (while True)
+        category_ids,                               # list OR dict (dict → query_category_id)
+        task_func=get_category_products,
+        pre_task_func=<optional>,                   # e.g. exclusive deals, called once/pass
+        headers=..., cookies=..., use_tls=...,
+    )
+
+# category first, session second — matches how execute_monitor_logic calls it.
+# pagination passes offset= or page_number= as a kwarg, so accept it with a default.
+async def get_category_products(category_id, session, offset=<n>):   # or page_number=<n>
+    response = await send_request(session, url=..., method=...)
+    products = await extract_products(response, ...)   # parse → build Product(s)
+    await add_products_to_database(products, collection, category_identifier=category_id)
+    if <first page/offset>:                            # only kick pagination once
+        await execute_pagination(
+            category_identifier=category_id,
+            total_products=response[...],
+            products_per_page=<n>,
+            func=get_category_products,
+            session=session,
+            pagination_mode="page"|"offset",
+            category_id=category_id,                    # flows through **kwargs
+        )
+
+asyncio.run(main())
+```
+
+Note: `execute_monitor_logic` already loops forever and recreates the session each
+pass — the monitor does NOT add its own outer loop. A category task raising a
+non-`EXPECTED_TASK_ERRORS` exception kills the whole loop (intentional fail-loud).
+
+**Where `DevelopmentMode` is set:**
+- **Monitors:** in the monitor's own entry script (`<name>_monitor.py`).
+- **Discord bot:** in `discord_bot/helpers/config.py` (multiple cogs depend on
+  it, so it's defined once there, not per-cog).
+
+## Critical invariants (do not break)
+
+- **Structured logging only — never f-string interpolation in log calls.** Use
+  `LOGGER.info("message", extra={...})`. Standard `extra` keys, kept consistent
+  for queryability: `product_id`, `category`, `page`/`offset`, `count`,
+  `config_id`. The message is a short static string; variable data goes in
+  `extra`. `configure_logging(monitor_name)` injects **`monitor_name`** on every
+  record via a filter (so don't also pass `monitor_name` in `extra`) and emits
+  JSON. The `JSONFormatter` reserves the standard `LogRecord` attrs and merges
+  everything else, so your `extra` keys land at the JSON top level. Pick one name
+  per value (`product_id`, not a mix of `sku`/`product_id`).
+- **Two-tier error handling.** Known/expected failures → `LOGGER.warning` (or
+  `.exception` for tracebacks) **and continue**. Unknown failures →
+  `LOGGER.exception` **and raise**. `EXPECTED_TASK_ERRORS` (in `constants.py`) is
+  the known-error tuple caught throughout.
+- **One bad product must never kill a category.** Per-product build is wrapped in
+  `try/except EXPECTED_TASK_ERRORS` (log with `product_id`, `continue`). Guard
+  indexing into possibly-empty lists (`if not items:` before `items[0]`) — a
+  single malformed item silently skips, it does not crash the run. (This is why
+  the Woot `IndexError` on empty `Attributes` took the monitor down before the
+  guard was added.)
+- **Validate implausible data at the boundary, skip per-item, don't emit.** Bad
+  upstream values (negative/zero prices) are rejected where they're read by
+  raising `ValueError`, caught per-item in the loop so other items still process
+  — never let a false deal reach the consumer. (See HP negative-cart note.)
+- **Mongo `_id` is composite: `f"{product_id}_{category_identifier}"`.** The same
+  product under two categories is two documents. `add_products_to_database` builds
+  this; anything querying products by id must account for it. (`Product.product_id`
+  itself stays the bare retailer id — the composite only exists in the stored doc.)
+- **Dev writes to a separate `_testing` collection.** `setup_config` returns the
+  `<name>_testing` collection in development mode and `<name>` in production, and
+  reads configs from `tests/configs/` vs `configs/`. So running in development
+  never touches production data.
+- **`Product` is snake_case; the Mongo doc is camelCase.** `convert_product_to_mongo`
+  is the only translation point. Don't hand-write camelCase into a `Product` or
+  snake_case into a Mongo filter.
+- **Logs write to `/logs/<name>.jsonl`** inside the container, host-mounted to a
+  shared folder. The host folder must be owned by the container user (UID 1000) —
+  bind mounts override the image's ownership, so `chown -R 1000:1000 <host log
+  dir>` is required on the host, NOT fixable in the Dockerfile alone.
+
+## Coding conventions
+
+- **Python 3.10+**, `src/` layout where applicable, type hints on public funcs.
+- **Functional-first.** Avoid OOP unless a framework requires it (discord.py
+  cogs). Prefer plain functions and module-level helpers.
+- **Direct dict access `[]` over `.get()`** for controlled schemas where the key
+  is guaranteed; use `.get()` only where a key is genuinely optional.
+- **Required dataclass fields over optional-with-defaults** when the field is
+  always present.
+- **Doc-strings only — do not over-comment.** No narrating obvious code; a
+  doc-string on a function is enough. Doc-strings start lowercase, no trailing
+  period.
+- **No trailing whitespace; trailing newline on every file; 4-space indent.**
+- **No hardcoded paths or secrets in code** — tunable/env-varying state goes
+  through `config.py`; fixed values go in `constants.py`.
+- **Terse, direct edits.** When changing code, return the change without
+  surrounding explanation unless options are being compared.
+
+## Local dev (debugger uses local `monitor_utils`, Docker does not)
+
+- To make the debugger / interpreter use your **local** `monitor_utils` working
+  copy instead of the pinned site-packages build, editable-install it into the
+  repo's venv: `uv pip install -e <path to monitor-shared-utils>`. This touches
+  only the local venv — it does NOT modify `pyproject.toml` / `uv.lock`, so the
+  Docker build still installs the pinned published version.
+- Verify which copy is loaded:
+  `python -c "import monitor_utils.config as c; print(c.__file__)"` — should point
+  at the local clone, not `site-packages`.
+- VS Code: select the repo's `.venv` interpreter; run via a launch config with
+  `"module": "<name>_monitor"` (or the bot's `module`) and `"cwd":
+  "${workspaceFolder}"`, not the editor's run-file button (which runs the file
+  directly and breaks package imports).
+- `log_dir` for `configure_logging` defaults to `/logs` (container). Locally,
+  either create the dir or pass a local `log_dir`; do not commit a local override
+  into the entry script.
+
+## Discord bot (when this repo is the bot)
+
+- **Cogs architecture.** `controller.py` is the entry; cogs live in
+  `discord_bot/cogs/` (e.g. `channels.py`, `consumer.py`). Commands are cog
+  methods; `DevelopmentMode`/`IS_PRODUCTION` come from
+  `discord_bot/helpers/config.py`.
+- **Consumer** reads pending products per monitor from Mongo
+  (`find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)`),
+  builds embeds, posts, and updates status. A compound index on
+  `(status, discoveredDate)` backs this query (ESR: equality field first, sort
+  field second) — create it at startup; it is idempotent.
+- **Stop/start control is via a DB flag, NOT the Docker socket.** Server
+  hardening blocks Docker socket access, so the consumer (and, where wired, the
+  scraper loops) checks a `monitor_control` collection (one doc per monitor,
+  keyed by name) each pass and goes idle on `stop: true` without exiting. The bot
+  flips the flag through a command; readers and writers communicate only through
+  Mongo. Do not reintroduce Docker-socket control.
+- **Owner/admin gating.** Use a check that passes for the bot owner (global,
+  independent of server perms — works even in servers where you have none) OR a
+  guild admin. Prefer a `app_commands.check` decorator routing failures through
+  `on_app_command_error` over inline per-command guards.
+
+## Known data hazards (record per-repo as you find them)
+
+- **HP — negative/zero cart totals on CTO products.** HP's cart API returns
+  genuinely negative `grandTotal` for some configure-to-order configs (promo
+  components exceeding base, or a config that doesn't register a valid BOM).
+  Guard `grand_total <= 0` at the cart boundary; raise and skip that config in
+  the pricing loop so other configs still price. Also: configurator picks must
+  include every option's path (don't drop `display: false` options from the
+  path map) and the base SKU, or HP misprices.
+- **Woot — empty `Attributes`/`Items` lists.** `parse_condition` must guard
+  `if not items` / `if not attributes` before indexing `[0]`; an empty list
+  otherwise raises `IndexError` and (without the per-product try/except) kills
+  the category.
+
+<!-- TODO per repo: add this monitor's own observed hazards here. -->