monitor-shared-utils/CLAUDE.md

CLAUDE.md — Profit Lounge monitor

Orientation for working in this repo. This is one monitor (or the Discord bot) in a polyrepo system that scrapes a retailer, normalizes its products into a shared Product model, writes them to MongoDB, and lets a Discord bot consume them and post deals. Shared code lives in the installable monitor_utils package; each repo pins its own version of it.

This file is portable — most of it is true in every repo. Only the Per-repo facts block at the top changes between monitors.

Per-repo facts (the only section that varies)

• Monitor name: <bestbuy | hp | lg | samsung | woot | northerntool> (the string passed to configure_logging() and setup_config()).
• Entry script: <name>_monitor.py (e.g. lg_monitor.py).
• category_ids shape: list of <ints | category codes | category paths> — varies by retailer (e.g. Best Buy uses numeric category ids, HP uses category paths, Woot uses category names).
• Pagination mode: <page | offset>.
• Site-specific quirks: <record known data hazards here — see HP/Woot notes in "Known data hazards" for the pattern.>

Architecture (settled — do not relitigate)

• Polyrepo, not monorepo. Each retailer monitor and the Discord bot is its own repo. Anything shared across them lives in monitor_utils, an installable package consumed as a pinned dependency. monitor_utils is public (the Gitea repo is public), so no token is needed to install it.
• monitor_utils is pinned per-repo and upgraded manually. A repo gets a specific pinned version via uv.lock. To pick up changes, bump it deliberately: uv lock --upgrade-package monitor-utils then rebuild. There is no auto-latest — pinning is intentional for reproducible builds.
• Each repo builds its own Docker image that installs the pinned monitor_utils. The Dockerfile installs from uv.lock; Docker never reads local editable installs (those are dev-only — see Local dev).
• No .env files in any form. Not committed, not copied into images, not mounted, not via env_file. Configuration that varies by environment is resolved through monitor_utils.config and the DevelopmentMode enum, not env files. (One narrow exception used in dev only: the log_dir argument to configure_logging.)

monitor_utils (the shared package)

• runner.py — the monitor execution contract. Exposes:
  • setup_config(monitor_name, development_mode) → returns the triple (category_identifiers, channel_ids, collection). It reads the JSON config via get_config and returns config["categories"], config["channels"], and the Mongo collection. categories can be a list OR a dict — when it's a dict (e.g. Best Buy), execute_monitor_logic iterates .items() and passes the value as query_category_id. The collection is the monitor name in production, but f"{monitor_name}_testing" in development — dev writes to a separate _testing collection.
  • execute_monitor_logic(category_identifiers, task_func, headers=None, cookies=None, pre_task_func=None, use_tls=False, **kwargs) — runs forever in a while True loop, recreating the session each pass and sleeping 0.30s between passes. Each pass: opens a session (get_tls_session if use_tls else get_base_session), calls optional pre_task_func(session=session, **kwargs) once, then builds per-category tasks and asyncio.gathers them. If category_identifiers is a dict, each task gets task_func(category_id, session, query_category_id=query, **kwargs); if a list, task_func(category_identifier, session, **kwargs). Results: EXPECTED_TASK_ERRORS → LOGGER.warning + continue; any other Exception → LOGGER.error + raise (kills the loop). task_func is called with the category as the first positional arg and session second.
  • execute_pagination(category_identifier, total_products, products_per_page, func, session, max_concurrent_tasks=5, start_offset=24, pagination_mode= "offset", **kwargs) — paginates pages 2..N concurrently under a semaphore. pagination_mode="offset" calls func(session=session, offset=..., **kwargs) with offset = start_offset + page_index * products_per_page; "page" calls func(session=session, page_number=page_index, **kwargs). Failed pages are retried (retry_failed_pages, exp backoff, 3 attempts). stop_event is internal to this function (early-stop plumbing) — it is NOT a parameter you pass in.
  • build_product_details(product_dict, channel_ids, min_percentage, min_price_drop=None) (consumer side) — takes a Mongo doc (camelCase keys) and returns a Product or None. Computes percentage, applies the two-rule gate (percentage >= min_percentage OR price-drop ≥ min_price_drop), and sets metadata.channel_id via get_channel_id. Returns None when the product fails the gate or has no/zero originalPrice.
  • get_channel_id(product, channel_ids, use_price_drop_bucket=False) — routing precedence: app-exclusive → sold-out → ending-within-1-day event → FLAG_WEBHOOK_MAP flags (member pick/deal) → percentage bucket (BUCKET_WEBHOOK_MAP, capped at 90). Used by the consumer, not the scrapers.
• config.py — configuration + environment. Holds: get_config (reads JSON configs — production path PROJECT_ROOT/configs/<name>.json, dev path PROJECT_ROOT/tests/configs/<name>.json), get_bot_token (by is_production bool), the logging setup (configure_logging, the JSONFormatter) — logging lives HERE, there is no observability.py, get_proxies, the DevelopmentMode enum (a str Enum: PRODUCTION/DEVELOPMENT), and is_production.
  • configure_logging(monitor_name, log_dir="/logs") clears root handlers, adds a StreamHandler and a RotatingFileHandler (<log_dir>/<monitor_name>.jsonl, 10MB × 5 backups), both using JSONFormatter, and pins aiohttp/pymongo to WARNING. A filter injects monitor_name onto every record.
  • The JSONFormatter emits time, level, logger, function, message, then merges any non-reserved extra keys, and adds exc on exceptions. So the injected field is literally monitor_name (not monitor), and your extra={...} keys appear at the top level of the JSON.
  • get_proxies parses proxies.txt: a line host:port:user:password → http://user:password@host:port; otherwise the line is used as http://<line>. Cached; empty list raises.
• database.py — Mongo controls. add_products_to_database(products, collection, category_identifier) upserts with a composite _id of f"{product_id}_{category_identifier}" (NOT bare product_id — the same product under two categories is two docs). It compares new vs existing currentPrice: a new product or a price drop sets status=pending (and priceBeforeDrop); discoveredDate is $setOnInsert only. bulk_write(..., ordered=False). get_product_from_database(collection, product_id) is a bare find_one({"_id": product_id}) — note it does NOT apply the composite-id scheme, so callers using it (HP) must pass whatever id matches their stored _id. convert_product_to_mongo maps the dataclass (snake_case) to the stored Mongo doc (camelCase: currentPrice, imageUrl, appExclusive, …) — this snake→camel mapping is the boundary; the Product model is snake_case, Mongo is camelCase.
• dsql_shortlinks.py — affiliate short-link client (generate_short_link async / generate_short_link_sync). Posts to the Profit Lounge affiliate API; on any failure it returns the original url ({"short_url": url}) rather than raising — link generation never blocks a post.
• sessions.py — HTTP session creation + the send_request wrapper. get_base_session (aiohttp) and get_tls_session (curl_cffi AsyncSession with impersonate=random.choice(CLIENT_IDENTIFIERS)) are async context managers; execute_monitor_logic picks via use_tls. curl_cffi replaced the old noble_tls for Python 3.13-compatible TLS fingerprint spoofing.
  • send_request(session, url, method, params=None, json_data=None, data=None, reraise_json_error=False, development_proxy=None) — tenacity-retried (RETRY_LIMIT× on network/timeout errors). Picks a random proxy from get_proxies() unless development_proxy is set (which also disables TLS verify — that's the local-mitmproxy path, e.g. HP's http://localhost:8001). Returns parsed JSON, or the raw text on JSONDecodeError unless reraise_json_error=True (then it raises — used where a non-JSON response must be treated as a failure). Branches on session type internally; callers just pass the session. use_tls is a parameter of execute_monitor_logic, not of send_request.
• models.py — the shared product schema (see Models).
• constants.py — fixed values + secrets currently live here: MONGO_URL, PRODUCTION_BOT_TOKEN/DEVELOPMENT_BOT_TOKEN, PROFIT_API_KEY, plus EXPECTED_TASK_ERRORS (the known-error tuple: curl_cffi RequestsError, aiohttp.ClientError, asyncio.TimeoutError, requests.Timeout/HTTPError, PyMongoError, JSONDecodeError), status constants (STATUS_PENDING/SENT/SKIPPED/FAILED), delays (BACKOFF_DELAY/FAILURE_DELAY — both 0.001), RETRY_LIMIT/RETRY_WAIT, LOGGING_LEVEL, CLIENT_IDENTIFIERS, FLAG_WEBHOOK_MAP, BUCKET_WEBHOOK_MAP, and PROJECT_ROOT/PROXIES_PATH.

Models (monitor_utils.models, pydantic dataclasses)

Product is the normalized shape every monitor produces and the consumer reads. Built with pydantic.dataclasses.dataclass (validation on construction — a bad field raises at build time, which is the boundary where bad data should be caught).

• Product(product_id, title, url, pricing, metadata=…, timing=…, availability=…) — metadata/timing/availability default via field(default_factory=…).
• PricingInfo(original_price, current_price, price_before_drop=None, edu_price=None).
• AvailabilityInfo(available=None, condition=None, app_exclusive=None, member_pick=None, member_deal=None).
• ProductMetadata(image_url, channel_id, customization_index, member_non_member_comparison, overstock_text, percentage, affiliate_url — all optional).
• ProductTiming(discovered_date, start_date, end_date — all optional).
• ConsumerConfig(monitor_name, collection, channel_ids, min_percentage, ping_threshold=None) — one per retailer for the Discord consumer. ping_threshold is the price floor for the secondary 40% rule (Woot pings on 40%+ if the item is ≥ this price); None disables it.

Standard monitor flow (the canonical shape — every monitor follows it)

configure_logging(monitor_name="<name>")          # logging first, from config.py
DEVELOPMENT_MODE = DevelopmentMode.<MODE>          # set in THIS script (see below)
category_ids, _, collection = setup_config(        # second arg is positional
    monitor_name="<name>", development_mode=DEVELOPMENT_MODE
)

async def main():
    await execute_monitor_logic(                    # runs forever (while True)
        category_ids,                               # list OR dict (dict → query_category_id)
        task_func=get_category_products,
        pre_task_func=<optional>,                   # e.g. exclusive deals, called once/pass
        headers=..., cookies=..., use_tls=...,
    )

# category first, session second — matches how execute_monitor_logic calls it.
# pagination passes offset= or page_number= as a kwarg, so accept it with a default.
async def get_category_products(category_id, session, offset=<n>):   # or page_number=<n>
    response = await send_request(session, url=..., method=...)
    products = await extract_products(response, ...)   # parse → build Product(s)
    await add_products_to_database(products, collection, category_identifier=category_id)
    if <first page/offset>:                            # only kick pagination once
        await execute_pagination(
            category_identifier=category_id,
            total_products=response[...],
            products_per_page=<n>,
            func=get_category_products,
            session=session,
            pagination_mode="page"|"offset",
            category_id=category_id,                    # flows through **kwargs
        )

asyncio.run(main())

Note: execute_monitor_logic already loops forever and recreates the session each pass — the monitor does NOT add its own outer loop. A category task raising a non-EXPECTED_TASK_ERRORS exception kills the whole loop (intentional fail-loud).

Where DevelopmentMode is set:

• Monitors: in the monitor's own entry script (<name>_monitor.py).
• Discord bot: in discord_bot/helpers/config.py (multiple cogs depend on it, so it's defined once there, not per-cog).

Critical invariants (do not break)

• Structured logging only — never f-string interpolation in log calls. Use LOGGER.info("message", extra={...}). Standard extra keys, kept consistent for queryability: product_id, category, page/offset, count, config_id. The message is a short static string; variable data goes in extra. configure_logging(monitor_name) injects monitor_name on every record via a filter (so don't also pass monitor_name in extra) and emits JSON. The JSONFormatter reserves the standard LogRecord attrs and merges everything else, so your extra keys land at the JSON top level. Pick one name per value (product_id, not a mix of sku/product_id).
• Two-tier error handling. Known/expected failures → LOGGER.warning (or .exception for tracebacks) and continue. Unknown failures → LOGGER.exception and raise. EXPECTED_TASK_ERRORS (in constants.py) is the known-error tuple caught throughout.
• One bad product must never kill a category. Per-product build is wrapped in try/except EXPECTED_TASK_ERRORS (log with product_id, continue). Guard indexing into possibly-empty lists (if not items: before items[0]) — a single malformed item silently skips, it does not crash the run. (This is why the Woot IndexError on empty Attributes took the monitor down before the guard was added.)
• Validate implausible data at the boundary, skip per-item, don't emit. Bad upstream values (negative/zero prices) are rejected where they're read by raising ValueError, caught per-item in the loop so other items still process — never let a false deal reach the consumer. (See HP negative-cart note.)
• Mongo _id is composite: f"{product_id}_{category_identifier}". The same product under two categories is two documents. add_products_to_database builds this; anything querying products by id must account for it. (Product.product_id itself stays the bare retailer id — the composite only exists in the stored doc.)
• Dev writes to a separate _testing collection. setup_config returns the <name>_testing collection in development mode and <name> in production, and reads configs from tests/configs/ vs configs/. So running in development never touches production data.
• Product is snake_case; the Mongo doc is camelCase. convert_product_to_mongo is the only translation point. Don't hand-write camelCase into a Product or snake_case into a Mongo filter.
• Logs write to /logs/<name>.jsonl inside the container, host-mounted to a shared folder. The host folder must be owned by the container user (UID 1000) — bind mounts override the image's ownership, so chown -R 1000:1000 <host log dir> is required on the host, NOT fixable in the Dockerfile alone.

Coding conventions

• Python 3.10+, src/ layout where applicable, type hints on public funcs.
• Functional-first. Avoid OOP unless a framework requires it (discord.py cogs). Prefer plain functions and module-level helpers.
• Direct dict access [] over .get() for controlled schemas where the key is guaranteed; use .get() only where a key is genuinely optional.
• Required dataclass fields over optional-with-defaults when the field is always present.
• Doc-strings only — do not over-comment. No narrating obvious code; a doc-string on a function is enough. Doc-strings start lowercase, no trailing period.
• No trailing whitespace; trailing newline on every file; 4-space indent.
• No hardcoded paths or secrets in code — tunable/env-varying state goes through config.py; fixed values go in constants.py.
• Terse, direct edits. When changing code, return the change without surrounding explanation unless options are being compared.

Local dev (debugger uses local monitor_utils, Docker does not)

• To make the debugger / interpreter use your local monitor_utils working copy instead of the pinned site-packages build, editable-install it into the repo's venv: uv pip install -e <path to monitor-shared-utils>. This touches only the local venv — it does NOT modify pyproject.toml / uv.lock, so the Docker build still installs the pinned published version.
• Verify which copy is loaded: python -c "import monitor_utils.config as c; print(c.__file__)" — should point at the local clone, not site-packages.
• VS Code: select the repo's .venv interpreter; run via a launch config with "module": "<name>_monitor" (or the bot's module) and "cwd": "${workspaceFolder}", not the editor's run-file button (which runs the file directly and breaks package imports).
• log_dir for configure_logging defaults to /logs (container). Locally, either create the dir or pass a local log_dir; do not commit a local override into the entry script.

Discord bot (when this repo is the bot)

• Cogs architecture. controller.py is the entry; cogs live in discord_bot/cogs/ (e.g. channels.py, consumer.py). Commands are cog methods; DevelopmentMode/IS_PRODUCTION come from discord_bot/helpers/config.py.
• Consumer reads pending products per monitor from Mongo (find({"status": STATUS_PENDING}).sort("discoveredDate", 1).limit(...)), builds embeds, posts, and updates status. A compound index on (status, discoveredDate) backs this query (ESR: equality field first, sort field second) — create it at startup; it is idempotent.
• Stop/start control is via a DB flag, NOT the Docker socket. Server hardening blocks Docker socket access, so the consumer (and, where wired, the scraper loops) checks a monitor_control collection (one doc per monitor, keyed by name) each pass and goes idle on stop: true without exiting. The bot flips the flag through a command; readers and writers communicate only through Mongo. Do not reintroduce Docker-socket control.
• Owner/admin gating. Use a check that passes for the bot owner (global, independent of server perms — works even in servers where you have none) OR a guild admin. Prefer a app_commands.check decorator routing failures through on_app_command_error over inline per-command guards.

Known data hazards (record per-repo as you find them)

• HP — negative/zero cart totals on CTO products. HP's cart API returns genuinely negative grandTotal for some configure-to-order configs (promo components exceeding base, or a config that doesn't register a valid BOM). Guard grand_total <= 0 at the cart boundary; raise and skip that config in the pricing loop so other configs still price. Also: configurator picks must include every option's path (don't drop display: false options from the path map) and the base SKU, or HP misprices.
• Woot — empty Attributes/Items lists. parse_condition must guard if not items / if not attributes before indexing [0]; an empty list otherwise raises IndexError and (without the per-product try/except) kills the category.