added claude folder and removed project root constant

.gitignore

@@ -3,4 +3,6 @@ __pycache__
 *.pyc
 .vscode
 .idea
-monitor_utils.egg-info
+monitor_utils.egg-info
+uv.lock
+.claude

CLAUDE.md

@@ -1,328 +0,0 @@
-# CLAUDE.md — Profit Lounge monitor
-
-Orientation for working in this repo. This is **one monitor (or the Discord bot)
-in a polyrepo system** that scrapes a retailer, normalizes its products into a
-shared `Product` model, writes them to MongoDB, and lets a Discord bot consume
-them and post deals. Shared code lives in the installable `monitor_utils`
-package; each repo pins its own version of it.
-
-This file is portable — most of it is true in **every** repo. Only the
-**Per-repo facts** block at the top changes between monitors.
-
-## Per-repo facts (the only section that varies)
-
-<!-- TODO per repo: fill these in. Everything below this block is shared. -->
-
-- **Monitor name:** `<bestbuy | hp | lg | samsung | woot | northerntool>` (the
-  string passed to `configure_logging()` and `setup_config()`).
-- **Entry script:** `<name>_monitor.py` (e.g. `lg_monitor.py`).
-- **`category_ids` shape:** list of `<ints | category codes | category paths>`
-  — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses
-  category paths, Woot uses category names).
-- **Pagination mode:** `<page | offset>`.
-- **Site-specific quirks:** `<record known data hazards here — see HP/Woot notes
-  in "Known data hazards" for the pattern.>`
-
-## Architecture (settled — do not relitigate)
-
-- **Polyrepo, not monorepo.** Each retailer monitor and the Discord bot is its
-  own repo. Anything shared across them lives in **`monitor_utils`**, an
-  installable package consumed as a pinned dependency. `monitor_utils` is
-  **public** (the Gitea repo is public), so no token is needed to install it.
-- **`monitor_utils` is pinned per-repo and upgraded manually.** A repo gets a
-  specific pinned version via `uv.lock`. To pick up changes, bump it
-  deliberately: `uv lock --upgrade-package monitor-utils` then rebuild. There is
-  no auto-latest — pinning is intentional for reproducible builds.
-- **Each repo builds its own Docker image** that installs the pinned
-  `monitor_utils`. The Dockerfile installs from `uv.lock`; Docker never reads
-  local editable installs (those are dev-only — see Local dev).
-- **No `.env` files in any form.** Not committed, not copied into images, not
-  mounted, not via `env_file`. Configuration that varies by environment is
-  resolved through `monitor_utils.config` and the `DevelopmentMode` enum, not
-  env files. (One narrow exception used in dev only: the `log_dir` argument to
-  `configure_logging`.)
-
-## `monitor_utils` (the shared package)
-
-- **`runner.py`** — the monitor execution contract. Exposes:
-  - `setup_config(monitor_name, development_mode)` → returns the triple
-    `(category_identifiers, channel_ids, collection)`. It reads the JSON config
-    via `get_config` and returns `config["categories"]`, `config["channels"]`,
-    and the Mongo collection. **`categories` can be a list OR a dict** — when it's
-    a dict (e.g. Best Buy), `execute_monitor_logic` iterates `.items()` and passes
-    the value as `query_category_id`. The collection is the **monitor name in
-    production, but `f"{monitor_name}_testing"` in development** — dev writes to a
-    separate `_testing` collection.
-  - `execute_monitor_logic(category_identifiers, task_func, headers=None,
-    cookies=None, pre_task_func=None, use_tls=False, **kwargs)` — **runs forever in
-    a `while True` loop**, recreating the session each pass and sleeping 0.30s
-    between passes. Each pass: opens a session (`get_tls_session` if `use_tls` else
-    `get_base_session`), calls optional `pre_task_func(session=session, **kwargs)`
-    once, then builds per-category tasks and `asyncio.gather`s them. If
-    `category_identifiers` is a **dict**, each task gets
-    `task_func(category_id, session, query_category_id=query, **kwargs)`; if a
-    **list**, `task_func(category_identifier, session, **kwargs)`. Results:
-    `EXPECTED_TASK_ERRORS` → `LOGGER.warning` + continue; any other `Exception` →
-    `LOGGER.error` + **raise** (kills the loop). `task_func` is called with the
-    category as the **first positional arg** and `session` second.
-  - `execute_pagination(category_identifier, total_products, products_per_page,
-    func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode=
-    "offset", **kwargs)` — paginates pages 2..N concurrently under a semaphore.
-    **`pagination_mode="offset"`** calls `func(session=session, offset=...,
-    **kwargs)` with `offset = start_offset + page_index * products_per_page`;
-    **`"page"`** calls `func(session=session, page_number=page_index, **kwargs)`.
-    Failed pages are retried (`retry_failed_pages`, exp backoff, 3 attempts).
-    `stop_event` is **internal** to this function (early-stop plumbing) — it is
-    NOT a parameter you pass in.
-  - **`build_product_details(product_dict, channel_ids, min_percentage,
-    min_price_drop=None)`** (consumer side) — takes a **Mongo doc** (camelCase
-    keys) and returns a `Product` or `None`. Computes `percentage`, applies the
-    two-rule gate (`percentage >= min_percentage` OR price-drop ≥ `min_price_drop`),
-    and sets `metadata.channel_id` via `get_channel_id`. Returns `None` when the
-    product fails the gate or has no/zero `originalPrice`.
-  - **`get_channel_id(product, channel_ids, use_price_drop_bucket=False)`** —
-    routing precedence: app-exclusive → sold-out → ending-within-1-day event →
-    `FLAG_WEBHOOK_MAP` flags (member pick/deal) → percentage bucket
-    (`BUCKET_WEBHOOK_MAP`, capped at 90). Used by the consumer, not the scrapers.
-- **`config.py`** — configuration + environment. Holds: `get_config` (reads JSON
-  configs — **production path `PROJECT_ROOT/configs/<name>.json`, dev path
-  `PROJECT_ROOT/tests/configs/<name>.json`**), `get_bot_token` (by
-  `is_production` bool), **the logging setup (`configure_logging`, the
-  `JSONFormatter`) — logging lives HERE, there is no `observability.py`**,
-  `get_proxies`, the `DevelopmentMode` enum (a `str` Enum:
-  `PRODUCTION`/`DEVELOPMENT`), and `is_production`.
-  - `configure_logging(monitor_name, log_dir="/logs")` clears root handlers, adds
-    a `StreamHandler` **and** a `RotatingFileHandler`
-    (`<log_dir>/<monitor_name>.jsonl`, 10MB × 5 backups), both using
-    `JSONFormatter`, and pins `aiohttp`/`pymongo` to WARNING. A filter injects
-    **`monitor_name`** onto every record.
-  - **The `JSONFormatter` emits `time`, `level`, `logger`, `function`, `message`,
-    then merges any non-reserved `extra` keys**, and adds `exc` on exceptions. So
-    the injected field is literally **`monitor_name`** (not `monitor`), and your
-    `extra={...}` keys appear at the top level of the JSON.
-  - `get_proxies` parses `proxies.txt`: a line `host:port:user:password` →
-    `http://user:password@host:port`; otherwise the line is used as
-    `http://<line>`. Cached; empty list raises.
-- **`database.py`** — Mongo controls. `add_products_to_database(products,
-  collection, category_identifier)` upserts with a **composite `_id` of
-  `f"{product_id}_{category_identifier}"`** (NOT bare product_id — the same
-  product under two categories is two docs). It compares new vs existing
-  `currentPrice`: a **new product or a price drop** sets `status=pending` (and
-  `priceBeforeDrop`); `discoveredDate` is `$setOnInsert` only. `bulk_write(...,
-  ordered=False)`. `get_product_from_database(collection, product_id)` is a bare
-  `find_one({"_id": product_id})` — note it does NOT apply the composite-id scheme,
-  so callers using it (HP) must pass whatever id matches their stored `_id`.
-  `convert_product_to_mongo` maps the dataclass (snake_case) to the stored Mongo
-  doc (camelCase: `currentPrice`, `imageUrl`, `appExclusive`, …) — this snake→camel
-  mapping is the boundary; the `Product` model is snake_case, Mongo is camelCase.
-- **`dsql_shortlinks.py`** — affiliate short-link client (`generate_short_link`
-  async / `generate_short_link_sync`). Posts to the Profit Lounge affiliate API;
-  on any failure it **returns the original url** (`{"short_url": url}`) rather than
-  raising — link generation never blocks a post.
-- **`sessions.py`** — HTTP session creation + the `send_request` wrapper.
-  `get_base_session` (aiohttp) and `get_tls_session` (curl_cffi `AsyncSession`
-  with `impersonate=random.choice(CLIENT_IDENTIFIERS)`) are async context
-  managers; `execute_monitor_logic` picks via `use_tls`. `curl_cffi` replaced the
-  old `noble_tls` for Python 3.13-compatible TLS fingerprint spoofing.
-  - `send_request(session, url, method, params=None, json_data=None, data=None,
-    reraise_json_error=False, development_proxy=None)` — tenacity-retried
-    (`RETRY_LIMIT`× on network/timeout errors). Picks a **random proxy** from
-    `get_proxies()` unless `development_proxy` is set (which also disables TLS
-    verify — that's the local-mitmproxy path, e.g. HP's `http://localhost:8001`).
-    Returns parsed JSON, or the raw text on `JSONDecodeError` **unless
-    `reraise_json_error=True`** (then it raises — used where a non-JSON response
-    must be treated as a failure). Branches on session type internally; callers
-    just pass the session. **`use_tls` is a parameter of `execute_monitor_logic`,
-    not of `send_request`.**
-- **`models.py`** — the shared product schema (see Models).
-- **`constants.py`** — fixed values + **secrets currently live here**:
-  `MONGO_URL`, `PRODUCTION_BOT_TOKEN`/`DEVELOPMENT_BOT_TOKEN`, `PROFIT_API_KEY`,
-  plus `EXPECTED_TASK_ERRORS` (the known-error tuple: curl_cffi `RequestsError`,
-  `aiohttp.ClientError`, `asyncio.TimeoutError`, `requests.Timeout/HTTPError`,
-  `PyMongoError`, `JSONDecodeError`), status constants
-  (`STATUS_PENDING/SENT/SKIPPED/FAILED`), delays (`BACKOFF_DELAY`/`FAILURE_DELAY`
-  — both `0.001`), `RETRY_LIMIT`/`RETRY_WAIT`, `LOGGING_LEVEL`, `CLIENT_IDENTIFIERS`,
-  `FLAG_WEBHOOK_MAP`, `BUCKET_WEBHOOK_MAP`, and `PROJECT_ROOT`/`PROXIES_PATH`.
-
-## Models (`monitor_utils.models`, pydantic dataclasses)
-
-`Product` is the normalized shape every monitor produces and the consumer reads.
-Built with `pydantic.dataclasses.dataclass` (validation on construction — a bad
-field raises at build time, which is the boundary where bad data should be
-caught).
-
-- `Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…)`
-  — `metadata`/`timing`/`availability` default via `field(default_factory=…)`.
-- `PricingInfo(original_price, current_price, price_before_drop=None,
-  edu_price=None)`.
-- `AvailabilityInfo(available=None, condition=None, app_exclusive=None,
-  member_pick=None, member_deal=None)`.
-- `ProductMetadata(image_url, channel_id, customization_index,
-  member_non_member_comparison, overstock_text, percentage, affiliate_url —
-  all optional)`.
-- `ProductTiming(discovered_date, start_date, end_date — all optional)`.
-- `ConsumerConfig(monitor_name, collection, channel_ids, min_percentage,
-  ping_threshold=None)` — one per retailer for the Discord consumer.
-  `ping_threshold` is the price floor for the secondary 40% rule (Woot pings on
-  40%+ if the item is ≥ this price); `None` disables it.
-
-## Standard monitor flow (the canonical shape — every monitor follows it)
-
-```
-configure_logging(monitor_name="<name>")          # logging first, from config.py
-DEVELOPMENT_MODE = DevelopmentMode.<MODE>          # set in THIS script (see below)
-category_ids, _, collection = setup_config(        # second arg is positional
-    monitor_name="<name>", development_mode=DEVELOPMENT_MODE
-)
-
-async def main():
-    await execute_monitor_logic(                    # runs forever (while True)
-        category_ids,                               # list OR dict (dict → query_category_id)
-        task_func=get_category_products,
-        pre_task_func=<optional>,                   # e.g. exclusive deals, called once/pass
-        headers=..., cookies=..., use_tls=...,
-    )
-
-# category first, session second — matches how execute_monitor_logic calls it.
-# pagination passes offset= or page_number= as a kwarg, so accept it with a default.
-async def get_category_products(category_id, session, offset=<n>):   # or page_number=<n>
-    response = await send_request(session, url=..., method=...)
-    products = await extract_products(response, ...)   # parse → build Product(s)
-    await add_products_to_database(products, collection, category_identifier=category_id)
-    if <first page/offset>:                            # only kick pagination once
-        await execute_pagination(
-            category_identifier=category_id,
-            total_products=response[...],
-            products_per_page=<n>,
-            func=get_category_products,
-            session=session,
-            pagination_mode="page"|"offset",
-            category_id=category_id,                    # flows through **kwargs
-        )
-
-asyncio.run(main())
-```
-
-Note: `execute_monitor_logic` already loops forever and recreates the session each
-pass — the monitor does NOT add its own outer loop. A category task raising a
-non-`EXPECTED_TASK_ERRORS` exception kills the whole loop (intentional fail-loud).
-
-**Where `DevelopmentMode` is set:**
-- **Monitors:** in the monitor's own entry script (`<name>_monitor.py`).
-- **Discord bot:** in `discord_bot/helpers/config.py` (multiple cogs depend on
-  it, so it's defined once there, not per-cog).
-
-## Critical invariants (do not break)
-
-- **Structured logging only — never f-string interpolation in log calls.** Use
-  `LOGGER.info("message", extra={...})`. Standard `extra` keys, kept consistent
-  for queryability: `product_id`, `category`, `page`/`offset`, `count`,
-  `config_id`. The message is a short static string; variable data goes in
-  `extra`. `configure_logging(monitor_name)` injects **`monitor_name`** on every
-  record via a filter (so don't also pass `monitor_name` in `extra`) and emits
-  JSON. The `JSONFormatter` reserves the standard `LogRecord` attrs and merges
-  everything else, so your `extra` keys land at the JSON top level. Pick one name
-  per value (`product_id`, not a mix of `sku`/`product_id`).
-- **Two-tier error handling.** Known/expected failures → `LOGGER.warning` (or
-  `.exception` for tracebacks) **and continue**. Unknown failures →
-  `LOGGER.exception` **and raise**. `EXPECTED_TASK_ERRORS` (in `constants.py`) is
-  the known-error tuple caught throughout.
-- **One bad product must never kill a category.** Per-product build is wrapped in
-  `try/except EXPECTED_TASK_ERRORS` (log with `product_id`, `continue`). Guard
-  indexing into possibly-empty lists (`if not items:` before `items[0]`) — a
-  single malformed item silently skips, it does not crash the run. (This is why
-  the Woot `IndexError` on empty `Attributes` took the monitor down before the
-  guard was added.)
-- **Validate implausible data at the boundary, skip per-item, don't emit.** Bad
-  upstream values (negative/zero prices) are rejected where they're read by
-  raising `ValueError`, caught per-item in the loop so other items still process
-  — never let a false deal reach the consumer. (See HP negative-cart note.)
-- **Mongo `_id` is composite: `f"{product_id}_{category_identifier}"`.** The same
-  product under two categories is two documents. `add_products_to_database` builds
-  this; anything querying products by id must account for it. (`Product.product_id`
-  itself stays the bare retailer id — the composite only exists in the stored doc.)
-- **Dev writes to a separate `_testing` collection.** `setup_config` returns the
-  `<name>_testing` collection in development mode and `<name>` in production, and
-  reads configs from `tests/configs/` vs `configs/`. So running in development
-  never touches production data.
-- **`Product` is snake_case; the Mongo doc is camelCase.** `convert_product_to_mongo`
-  is the only translation point. Don't hand-write camelCase into a `Product` or
-  snake_case into a Mongo filter.
-- **Logs write to `/logs/<name>.jsonl`** inside the container, host-mounted to a
-  shared folder. The host folder must be owned by the container user (UID 1000) —
-  bind mounts override the image's ownership, so `chown -R 1000:1000 <host log
-  dir>` is required on the host, NOT fixable in the Dockerfile alone.
-
-## Coding conventions
-
-- **Python 3.10+**, `src/` layout where applicable, type hints on public funcs.
-- **Functional-first.** Avoid OOP unless a framework requires it (discord.py
-  cogs). Prefer plain functions and module-level helpers.
-- **Direct dict access `[]` over `.get()`** for controlled schemas where the key
-  is guaranteed; use `.get()` only where a key is genuinely optional.
-- **Required dataclass fields over optional-with-defaults** when the field is
-  always present.
-- **Doc-strings only — do not over-comment.** No narrating obvious code; a
-  doc-string on a function is enough. Doc-strings start lowercase, no trailing
-  period.
-- **No trailing whitespace; trailing newline on every file; 4-space indent.**
-- **No hardcoded paths or secrets in code** — tunable/env-varying state goes
-  through `config.py`; fixed values go in `constants.py`.
-- **Terse, direct edits.** When changing code, return the change without
-  surrounding explanation unless options are being compared.
-
-## Local dev (debugger uses local `monitor_utils`, Docker does not)
-
-- To make the debugger / interpreter use your **local** `monitor_utils` working
-  copy instead of the pinned site-packages build, editable-install it into the
-  repo's venv: `uv pip install -e <path to monitor-shared-utils>`. This touches
-  only the local venv — it does NOT modify `pyproject.toml` / `uv.lock`, so the
-  Docker build still installs the pinned published version.
-- Verify which copy is loaded:
-  `python -c "import monitor_utils.config as c; print(c.__file__)"` — should point
-  at the local clone, not `site-packages`.
-- VS Code: select the repo's `.venv` interpreter; run via a launch config with
-  `"module": "<name>_monitor"` (or the bot's `module`) and `"cwd":
-  "${workspaceFolder}"`, not the editor's run-file button (which runs the file
-  directly and breaks package imports).
-- `log_dir` for `configure_logging` defaults to `/logs` (container). Locally,
-  either create the dir or pass a local `log_dir`; do not commit a local override
-  into the entry script.
-
-## Discord bot (when this repo is the bot)
-
-- **Cogs architecture.** `controller.py` is the entry; cogs live in
-  `discord_bot/cogs/` (e.g. `channels.py`, `consumer.py`). Commands are cog
-  methods; `DevelopmentMode`/`IS_PRODUCTION` come from
-  `discord_bot/helpers/config.py`.
-- **Consumer** reads pending products per monitor from Mongo
-  (`find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)`),
-  builds embeds, posts, and updates status. A compound index on
-  `(status, discoveredDate)` backs this query (ESR: equality field first, sort
-  field second) — create it at startup; it is idempotent.
-- **Stop/start control is via a DB flag, NOT the Docker socket.** Server
-  hardening blocks Docker socket access, so the consumer (and, where wired, the
-  scraper loops) checks a `monitor_control` collection (one doc per monitor,
-  keyed by name) each pass and goes idle on `stop: true` without exiting. The bot
-  flips the flag through a command; readers and writers communicate only through
-  Mongo. Do not reintroduce Docker-socket control.
-- **Owner/admin gating.** Use a check that passes for the bot owner (global,
-  independent of server perms — works even in servers where you have none) OR a
-  guild admin. Prefer a `app_commands.check` decorator routing failures through
-  `on_app_command_error` over inline per-command guards.
-
-## Known data hazards (record per-repo as you find them)
-
-- **HP — negative/zero cart totals on CTO products.** HP's cart API returns
-  genuinely negative `grandTotal` for some configure-to-order configs (promo
-  components exceeding base, or a config that doesn't register a valid BOM).
-  Guard `grand_total <= 0` at the cart boundary; raise and skip that config in
-  the pricing loop so other configs still price. Also: configurator picks must
-  include every option's path (don't drop `display: false` options from the
-  path map) and the base SKU, or HP misprices.
-- **Woot — empty `Attributes`/`Items` lists.** `parse_condition` must guard
-  `if not items` / `if not attributes` before indexing `[0]`; an empty list
-  otherwise raises `IndexError` and (without the per-product try/except) kills
-  the category.
-
-<!-- TODO per repo: add this monitor's own observed hazards here. -->

src/monitor_utils/config.py

@@ -1,12 +1,9 @@
 import logging
 import json
 import pathlib
-from functools import cache
 from enum import Enum
 
 from monitor_utils.constants import (
-    PROXIES_PATH,
-    PROJECT_ROOT,
     PRODUCTION_BOT_TOKEN,
     DEVELOPMENT_BOT_TOKEN,
     LOGGERS,