pl_devs/monitor-shared-utils
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: pl_devs/monitor-shared-utils:69484c5ae0d7f487350712f04ffef9331a49df9a
pl_devs/monitor-shared-utils:main pl_devs/monitor-shared-utils:development
..
compare: pl_devs/monitor-shared-utils:main
pl_devs/monitor-shared-utils:main pl_devs/monitor-shared-utils:development
17 Commits
69484c5ae0 .. main
Author SHA1 Message Date
dsql 1120083e45 Merge pull request 'build_product_details: skip non-finite prices instead of raising OverflowError' (#26) from development into main 2026-10-10 03:24:57 +00:00
dsqlandClaude Code Opus 5.5 6ea3187b33 build_product_details: skip non-finite prices instead of raising OverflowError 
PR #25 guarded calculate_priority but missed the same calculation here.
On 2026-10-09 06:25:50Z the consumer hit a product with an infinite price;
runner.py:201

    percentage = int(((original_price - current_price) / original_price) * 100)

raised `OverflowError: cannot convert float infinity to integer`. The
Discord leg dropped that one alert, but the API leg's caller does not
catch OverflowError, so the exception escaped and the consumer logged
CRITICAL "api deal loop stopped" -- no deals reached the API from then on
(nothing sent at all after 22:00Z), while the bot itself stayed "alive".

build_product_details now returns None (with a warning, like the
existing missing-price skip) when either price is a non-finite float, on
BOTH paths: the MSRP-less path never computes a percentage and would
otherwise carry inf into the Product and the API payload. A finite pair
whose ratio still overflows (a subnormal original price) is skipped too.
Ints and other types keep their existing behaviour.

Tests (tests/test_build_product_details.py): five new cases fail on the
old code -- OverflowError, ValueError for nan, and the MSRP-less case
building a Product with inf -- and all 25 pass now. Ruff findings on the
two files are unchanged from main (5, all pre-existing).

Rebuild every service that installs monitor-utils with --no-cache to pick
this up (it is pulled unpinned from main).

Co-Authored-By: Claude Code Opus 5.5 (ansible-ops, auto-repair) <noreply@anthropic.com>
Signed-off-by: disqualifier <dev@disqualifier.me>
 2026-10-09 23:22:36 -04:00
dsql 7d514a5401 Merge pull request 'calculate_priority: resolve non-finite prices to 0 instead of crashing the monitor' (#25) from development into main 2026-10-08 22:48:57 +00:00
dsqlandClaude Code Opus 5.5 f51aed2c9f calculate_priority: resolve non-finite prices to 0 instead of crashing the monitor 
tpv-academysports died on 2026-10-08 06:27 UTC and stayed down: six
identical crashes in 8 minutes (each ~2 min after start), then docker's
on-failure:5 gave up. Every one was

    database.py, in calculate_priority
        percentage = int(((original_price - current_price) / original_price) * 100)
    OverflowError: cannot convert float infinity to integer

A monitor builds prices with float() on the retailer's raw field, and
float() accepts "Infinity" / "1e999", so a non-finite price reaches here.
The `not original_price` guard catches None and 0, not inf or nan, and the
exception escapes add_products_to_database -> run_monitor_cycle and kills
the whole process -- one bad product takes the monitor down, and it
crashes again on every restart while that product is listed. Every
monitor that uses this lib has the same exposure.

calculate_priority now returns 0 (the existing "broken prices" bucket)
for a non-finite current/original price, and for a finite pair whose
ratio still overflows (a subnormal original price). Tests cover both;
they fail on the old code (OverflowError, plus ValueError for nan) and
pass now (tests/test_alert_priority.py: 10 passed).

Not changed: the monitor-side float() parsing, which could reject these
prices before they are stored -- worth a look by the monitor owners.

Co-Authored-By: Claude Code Opus 5.5 (ansible-ops, auto-repair) <noreply@anthropic.com>
Signed-off-by: disqualifier <dev@disqualifier.me>
 2026-10-08 18:46:59 -04:00
TPV bd5df3a4c2 Merge pull request 'small cleanup of comments/added optional impersonate into run_bounded_cycle for monitors like bhphoto that need different clients than what we support/dont need to rotate' (#24) from development into main 
Reviewed-on: #24
 2026-09-30 01:56:40 +00:00
TPV 6720592685 small cleanup of comments/added optional impersonate into run_bounded_cycle for monitors like bhphoto that need different clients than what we support/dont need to rotate 2026-09-29 18:55:52 -07:00
TPV 1b3f860ff3 Merge pull request 'updated tls profiles' (#23) from development into main 
Reviewed-on: #23
 2026-09-27 04:11:53 +00:00
TPV 4563cdf920 updated tls profiles 2026-09-26 21:11:37 -07:00
TPV 7032f5b9d6 Merge pull request 'added get_response_json function to return json from aioweb' (#22) from development into main 
Reviewed-on: #22
 2026-09-26 03:00:57 +00:00
TPV b9fbb64364 added get_response_json function to return json from aioweb 2026-09-25 20:00:19 -07:00
TPV ae637a134f rename monitor_name parameters to monitor 
Applies to setup_monitor_logging, init_monitor, setup_config, get_config,
add_products_to_database and build_alert. Breaking for callers passing
monitor_name by keyword; every monitor repo is updated alongside.
 2026-09-25 19:02:09 -07:00
TPV 4729a62ae2 Merge pull request 'renamed monitor_name to monitor' (#21) from development into main 
Reviewed-on: #21
 2026-09-26 00:29:32 +00:00
TPV 401258f52a renamed monitor_name to monitor 2026-09-25 17:29:08 -07:00
TPV 2bf2c613a9 Merge pull request 'added asins to build_product_details' (#20) from development into main 
Reviewed-on: #20
 2026-09-10 13:39:02 +00:00
TPV 9a34ae5ca6 added asins to build_product_details 2026-09-10 06:38:44 -07:00
TPV 0fceafb364 Merge pull request 'added price drops to alerts/fixed race condition where late claimed docs got their tags wiped and sent to the api without tags' (#19) from development into main 
Reviewed-on: #19
 2026-09-09 16:52:30 +00:00
TPV 0275f94500 added price drops to alerts/fixed race condition where late claimed docs got their tags wiped and sent to the api without tags 2026-09-09 09:52:15 -07:00
31 changed files with 311 additions and 1609 deletions
Expand all files Collapse all files

No files matched your search

scripts/bootstrap_db.py
-319
View File
@@ -1,319 +0,0 @@
"""Bootstrap the shared Mongo database: collections, indexes, and config seeds.
Rebuilds what the monitors and the discord consumer expect to find: the
configs/control/alerts collections in both modes, the alerts indexes the
consumer relies on for claim ordering and dedupe, one empty product
collection per monitor and mode, the per-monitor config documents from
scripts/db_seed/<mode>/<monitor>.json, and the control docs from
scripts/db_seed/control.json (the monitors registry; log_channel and
bot_owner hold Discord ids only the bot's commands know, and migrations
must stay absent so the consumer re-runs them on a fresh database).
Idempotent: existing collections and indexes are left alone and seed
documents use $setOnInsert, so re-running never clobbers operator edits.
Usage:
python scripts/bootstrap_db.py [--dry-run] [--url URL] [--database NAME]
"""
import argparse
import json
import logging
import sys
from pathlib import Path
from pymongo import ASCENDING, DESCENDING, MongoClient
from pymongo.collection import Collection
from pymongo.database import Database
from monitor_utils.constants import MONGO_DATABASE_NAME, MONGO_URL
LOGGER = logging.getLogger("bootstrap_db")
SEED_ROOT = Path(__file__).resolve().parent / "db_seed"
# collection suffix -> seed directory
MODES = {"": "production", "_testing": "testing"}
# monitors that call init_monitor and so own a product collection per mode
MONITORS = [
"apple",
"bestbuy",
"corsair",
"hp",
"lg",
"northerntool",
"samsung",
"samsung_shop",
"woot",
]
SEED_KEYS = {"categories", "channels"}
# categories shape each monitor's task function accepts: run_monitor_cycle
# passes query_category_id= only for a mapping, and only bestbuy (required
# query) and apple (node -> route) take that argument; the rest take a list
CATEGORY_SHAPES = {
"apple": dict,
"bestbuy": dict,
"corsair": list,
"hp": list,
"lg": list,
"northerntool": list,
"samsung": list,
"samsung_shop": list,
"woot": list,
}
CONTROL_SEED = SEED_ROOT / "control.json"
# shape setup_config (runner.py) reads: _id is the monitor name, categories is
# a list of names or a name -> query mapping, channels maps a bucket to ONE
# channel id as a digit string — the consumer's find_invalid_channel_ids
# rejects lists (the pre-07-16 spreading shape). Extra fields stay allowed.
CHANNEL_ID = {"bsonType": "string", "pattern": "^[0-9]+$"}
CONFIG_VALIDATOR = {
"$jsonSchema": {
"bsonType": "object",
"required": ["_id", "categories", "channels"],
"properties": {
"_id": {"bsonType": "string"},
"categories": {
"anyOf": [
{"bsonType": "array", "items": {"bsonType": "string"}},
{
"bsonType": "object",
"additionalProperties": {"bsonType": "string"},
},
]
},
"channels": {
"bsonType": "object",
"additionalProperties": CHANNEL_ID,
},
},
}
}
# identical to the consumer's cog_load create_index calls; a differing spec
# under the same key pattern would make its startup fail
ALERT_INDEXES = [
([("claimed", ASCENDING), ("priority", DESCENDING), ("createdAt", ASCENDING)], {}),
(
[("product_id", ASCENDING)],
{"unique": True, "partialFilterExpression": {"claimed": False}},
),
]
def ensure_collection(
database: Database, name: str, dry_run: bool = False, **options
) -> bool:
"""Create a collection when absent; return True when it was created."""
if name in database.list_collection_names():
LOGGER.info("collection exists", extra={"collection": name})
return False
LOGGER.info("creating collection", extra={"collection": name, "dry_run": dry_run})
if not dry_run:
database.create_collection(name, **options)
return True
def ensure_config_validator(
database: Database, name: str, dry_run: bool = False
) -> bool:
"""Apply CONFIG_VALIDATOR to an existing collection; True when it changed."""
info = next(iter(database.list_collections(filter={"name": name})), None)
if info is None or info["options"].get("validator") == CONFIG_VALIDATOR:
return False
LOGGER.info("updating validator", extra={"collection": name, "dry_run": dry_run})
if not dry_run:
database.command("collMod", name, validator=CONFIG_VALIDATOR)
return True
def ensure_alert_indexes(collection: Collection, dry_run: bool = False) -> list[str]:
"""Create the consumer's alert indexes; return the index names."""
names = []
for keys, options in ALERT_INDEXES:
if dry_run:
names.append("+".join(f"{field}_{direction}" for field, direction in keys))
continue
names.append(collection.create_index(keys, **options))
LOGGER.info(
"alert indexes ensured",
extra={"collection": collection.name, "indexes": names, "dry_run": dry_run},
)
return names
def load_seed_configs(seed_dir: Path) -> dict[str, dict]:
"""Load <monitor>.json seed docs keyed by monitor name.
Raises:
ValueError: When a seed file lacks categories or channels.
"""
docs = {}
for path in sorted(seed_dir.glob("*.json")):
with path.open() as seed_file:
doc = json.load(seed_file)
missing = SEED_KEYS - set(doc)
if missing:
raise ValueError(f"{path}: missing {sorted(missing)}")
bad_channels = {
bucket: value
for bucket, value in doc["channels"].items()
if not (isinstance(value, str) and value.isdigit())
}
if bad_channels:
raise ValueError(
f"{path}: channels must be single id strings {bad_channels}"
)
expected = CATEGORY_SHAPES.get(path.stem)
if expected is not None and not isinstance(doc["categories"], expected):
raise ValueError(
f"{path}: categories must be {expected.__name__} for {path.stem}"
)
docs[path.stem] = {key: doc[key] for key in SEED_KEYS}
return docs
def seed_configs(
collection: Collection, docs: dict[str, dict], dry_run: bool = False
) -> list[str]:
"""Insert absent config docs via $setOnInsert; return the ids inserted."""
inserted = []
for monitor_name, doc in docs.items():
if dry_run:
if collection.find_one({"_id": monitor_name}, {"_id": 1}) is None:
inserted.append(monitor_name)
continue
result = collection.update_one(
{"_id": monitor_name}, {"$setOnInsert": doc}, upsert=True
)
if result.upserted_id is not None:
inserted.append(monitor_name)
LOGGER.info(
"configs seeded",
extra={
"collection": collection.name,
"inserted": inserted,
"skipped": sorted(set(docs) - set(inserted)),
"dry_run": dry_run,
},
)
return inserted
def load_control_seed(path: Path) -> dict[str, dict]:
"""Load control docs keyed by _id from one JSON file."""
with path.open() as seed_file:
docs = json.load(seed_file)
for doc_id, doc in docs.items():
if not isinstance(doc, dict):
raise ValueError(f"{path}: {doc_id} must be an object")
return docs
def seed_control(
collection: Collection, docs: dict[str, dict], dry_run: bool = False
) -> list[str]:
"""Insert absent control docs via $setOnInsert; return the ids inserted."""
inserted = []
for doc_id, doc in docs.items():
if dry_run:
if collection.find_one({"_id": doc_id}, {"_id": 1}) is None:
inserted.append(doc_id)
continue
result = collection.update_one(
{"_id": doc_id}, {"$setOnInsert": doc}, upsert=True
)
if result.upserted_id is not None:
inserted.append(doc_id)
LOGGER.info(
"control seeded",
extra={
"collection": collection.name,
"inserted": inserted,
"skipped": sorted(set(docs) - set(inserted)),
"dry_run": dry_run,
},
)
return inserted
def bootstrap(
database: Database, seed_root: Path = SEED_ROOT, dry_run: bool = False
) -> dict:
"""Run every step for both modes and return a summary of what changed."""
summary = {"created": [], "validators": [], "indexes": {}, "seeded": {}}
for suffix, seed_mode in MODES.items():
configs_name = f"configs{suffix}"
if ensure_collection(
database, configs_name, dry_run, validator=CONFIG_VALIDATOR
):
summary["created"].append(configs_name)
elif ensure_config_validator(database, configs_name, dry_run):
summary["validators"].append(configs_name)
for base in ("control", "alerts", *MONITORS):
name = f"{base}{suffix}"
if ensure_collection(database, name, dry_run):
summary["created"].append(name)
alerts_name = f"alerts{suffix}"
summary["indexes"][alerts_name] = ensure_alert_indexes(
database[alerts_name], dry_run
)
docs = load_seed_configs(seed_root / seed_mode)
summary["seeded"][configs_name] = seed_configs(
database[configs_name], docs, dry_run
)
control_name = f"control{suffix}"
summary["seeded"][control_name] = seed_control(
database[control_name],
load_control_seed(seed_root / "control.json"),
dry_run,
)
return summary
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--url", default=MONGO_URL, help="Mongo connection url")
parser.add_argument("--database", default=MONGO_DATABASE_NAME)
parser.add_argument(
"--dry-run", action="store_true", help="report actions without writing"
)
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
client = MongoClient(args.url, serverSelectionTimeoutMS=30_000)
LOGGER.info("connected", extra={"server": client.server_info()["version"]})
summary = bootstrap(client[args.database], dry_run=args.dry_run)
print(json.dumps(summary, indent=2))
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/db_seed/control.json
-53
View File
@@ -1,53 +0,0 @@
{
"monitors": {
"names": [
"bestbuy",
"woot",
"lg",
"hp",
"samsung",
"northerntool",
"corsair",
"apple",
"samsung_shop"
],
"settings": {
"bestbuy": {
"min_percentage": 50,
"ping_threshold": null
},
"woot": {
"min_percentage": 50,
"ping_threshold": 800
},
"lg": {
"min_percentage": 40,
"ping_threshold": null
},
"hp": {
"min_percentage": 50,
"ping_threshold": null
},
"samsung": {
"min_percentage": 40,
"ping_threshold": null
},
"northerntool": {
"min_percentage": 40,
"ping_threshold": null
},
"corsair": {
"min_percentage": 40,
"ping_threshold": null
},
"apple": {
"min_percentage": 40,
"ping_threshold": null
},
"samsung_shop": {
"min_percentage": 40,
"ping_threshold": null
}
}
}
}
scripts/db_seed/production/apple.json
-74
View File
@@ -1,74 +0,0 @@
{
"categories": {
"home/shop_iphone/family/iphone_17_pro": "html",
"home/shop_iphone/family/iphone_air": "html",
"home/shop_iphone/family/iphone_17": "html",
"home/shop_iphone/family/iphone_17e": "html",
"home/shop_iphone/family/iphone_16": "html",
"home/shop_mac/family/macbook_neo": "html",
"home/shop_mac/family/macbook_air_13_inch": "html",
"home/shop_mac/family/macbook_air_15_inch": "html",
"home/shop_mac/family/macbook_pro_14_inch": "html",
"home/shop_mac/family/macbook_pro_16_inch": "html",
"home/shop_mac/family/imac": "html",
"home/shop_mac/family/mac_mini": "html",
"home/shop_mac/family/mac_studio": "html",
"home/shop_mac/family/studio_display": "html",
"home/shop_mac/family/studio_display_xdr": "html",
"/shop/accessories/all/accessibility": "grid",
"/shop/accessories/all/cases-protection": "grid",
"home/shop_ipad/family/ipad_pro": "html",
"home/shop_ipad/family/ipad_air": "html",
"home/shop_ipad/family/ipad": "html",
"home/shop_ipad/family/ipad_mini": "html",
"home/shop_watch/family/apple_watch": "html",
"home/shop_watch/family/apple_watch_se": "html",
"home/shop_watch/family/apple_watch_ultra": "html",
"home/shop_watch/family/apple_watch_hermes": "html",
"home/shop_watch/family/apple_watch_hermes_ultra": "html",
"home/shop_vision/family/apple_vision_pro": "html",
"home/shop_airpods/family/airpods_4": "html",
"home/shop_airpods/family/airpods_pro_3": "html",
"home/shop_airpods/family/airpods_max_2": "html",
"home/shop_homepod/family/homepod": "html",
"home/shop_homepod/family/homepod_mini": "html",
"home/shop_tv/family/apple_tv_4k": "html",
"/shop/accessories/all/chargers-adapters": "grid",
"/shop/accessories/all/travel-essentials": "grid",
"/shop/accessories/all/seasonal-colors": "grid",
"/shop/airtag/accessories/airtag-essentials": "grid",
"/shop/ipad/accessories/content-creation": "grid",
"/shop/mac/accessories/content-creation": "grid",
"/shop/vision/accessories/apple-vision-pro-essentials": "grid",
"/shop/accessories/all/college-essentials": "grid",
"/shop/accessories/all/content-creation": "grid",
"/shop/accessories/all/drives-storage": "grid",
"/shop/accessories/all/gaming": "grid",
"/shop/accessories/all/headphones-speakers": "grid",
"/shop/accessories/all/health-fitness": "grid",
"/shop/accessories/all/homekit": "grid",
"/shop/accessories/all/made-by-apple": "grid",
"/shop/accessories/all/mice-keyboards": "grid",
"/shop/accessories/all/new-arrivals": "grid",
"/shop/accessories/all/office": "grid",
"/shop/accessories/all/software": "grid",
"/shop/refurbished/mac": "grid",
"/shop/refurbished/ipad": "grid",
"/shop/refurbished/iphone": "grid",
"/shop/refurbished/watch": "grid",
"/shop/refurbished/appletv": "grid",
"/shop/refurbished/accessories": "grid",
"/shop/refurbished/airpods": "grid",
"/shop/refurbished/homepod": "grid"
},
"channels": {
"40-49": "1542905433235267694",
"50-60": "1542905485722779738",
"60-70": "1542905516047732947",
"70-80": "1542905555402629151",
"80-90": "1542905590832037932",
"90-100": "1542905643671756820",
"soldOut": "1542905707781816340",
"newListing": "1542905774601281677"
}
}
scripts/db_seed/production/bestbuy.json
-144
View File
@@ -1,144 +0,0 @@
{
"categories": {
"pcmcat1764872919906": "categoryid$pcmcat1764872919906",
"abcat0715000": "categoryid$abcat0715000",
"pcmcat1662993575810": "categoryid$pcmcat1662993575810",
"pcmcat287600050002": "categoryid$pcmcat287600050002",
"pcmcat287600050003": "categoryid$pcmcat287600050003",
"pcmcat1763397583811": "pcmcat1763397583811_categoryid$pcmcat241600050001",
"abcat0208000": "categoryid$abcat0208000",
"pcmcat1591132221892": "pcmcat1591132221892_categoryid$abcat0500000",
"pcmcat304600050011": "categoryid$pcmcat304600050011",
"abcat0507000": "categoryid$abcat0507000",
"pcmcat243800050005": "categoryid$pcmcat243800050005",
"abcat0401000": "categoryid$abcat0401000",
"abcat0410000": "categoryid$abcat0410000",
"pcmcat369900050001": "categoryid$pcmcat369900050001",
"pcmcat1762875326350": "pcmcat1762875326350_categoryid$abcat0915000",
"abcat0915001": "categoryid$abcat0915001",
"pcmcat1762453391793": "pcmcat1762453391793_categoryid$abcat0915004",
"pcmcat1717008056314": "categoryid$pcmcat1717008056314",
"pcmcat244100050013": "categoryid$pcmcat244100050013",
"pcmcat333300050027": "categoryid$pcmcat333300050027",
"abcat0901004": "categoryid$abcat0901004",
"abcat0901005": "categoryid$abcat0901005",
"abcat0901006": "categoryid$abcat0901006",
"abcat0901001": "categoryid$abcat0901001",
"abcat0901003": "categoryid$abcat0901003",
"pcmcat313100050031": "categoryid$pcmcat313100050031",
"pcmcat144700050004": "categoryid$pcmcat144700050004",
"pcmcat308100050020": "categoryid$pcmcat308100050020",
"pcmcat748301598672": "categoryid$pcmcat748301598672",
"pcmcat1477674238305": "categoryid$pcmcat1477674238305",
"pcmcat1602773246733": "categoryid$abcat0503000",
"pcmcat345400050002": "categoryid$pcmcat345400050002",
"pcmcat308100050014": "categoryid$pcmcat308100050014",
"pcmcat308100050018": "categoryid$pcmcat308100050018",
"pcmcat308100050017": "categoryid$pcmcat308100050017",
"pcmcat1515445788580": "categoryid$pcmcat1515445788580",
"pcmcat254000050007": "categoryid$pcmcat254000050007",
"pcmcat1586207767722": "categoryid$pcmcat1586207767722",
"pcmcat179200050010": "categoryid$pcmcat179200050010",
"pcmcat1651764083616": "categoryid$pcmcat1651764083616",
"pcmcat1615397403616": "categoryid$pcmcat1615397403616",
"pcmcat270300050004": "categoryid$pcmcat270300050004",
"pcmcat1651764174444": "categoryid$pcmcat1651764174444",
"pcmcat179200050004": "categoryid$pcmcat179200050004",
"pcmcat270300050005": "categoryid$pcmcat270300050005",
"pcmcat179200050007": "categoryid$pcmcat179200050007",
"pcmcat1631303385635": "categoryid$pcmcat1631303385635",
"pcmcat179200050008": "categoryid$pcmcat179200050008",
"pcmcat1597939198513": "categoryid$pcmcat1597939198513",
"pcmcat179200050003": "categoryid$pcmcat179200050003",
"pcmcat1491416706156": "categoryid$pcmcat1491416706156",
"pcmcat1592600995496": "categoryid$pcmcat1592600995496",
"pcmcat334800050011": "categoryid$pcmcat334800050011",
"abcat0205006": "categoryid$abcat0205006",
"pcmcat273800050032": "categoryid$pcmcat273800050032",
"pcmcat1545417879756": "pcmcat1545417879756_categoryid$pcmcat250300050003",
"pcmcat250300050008": "categoryid$pcmcat250300050008",
"pcmcat265100050011": "categoryid$pcmcat265100050011",
"abcat0302005": "categoryid$abcat0302005",
"abcat0302012": "categoryid$abcat0302012",
"abcat0302034": "categoryid$abcat0302034",
"abcat0302001": "categoryid$abcat0302001",
"abcat0302011": "categoryid$abcat0302011",
"pcmcat326300050013": "categoryid$pcmcat326300050013",
"abcat0302019": "categoryid$abcat0302019",
"pcmcat1493671011172": "categoryid$pcmcat1493671011172",
"pcmcat165900050023": "categoryid$pcmcat165900050023",
"pcmcat330300050001": "categoryid$pcmcat330300050001",
"abcat0305000": "categoryid$abcat0305000",
"pcmcat1762288382250": "pcmcat1762288382250_categoryid$abcat0301000",
"pcmcat748301308133": "categoryid$pcmcat748301308133",
"pcmcat1496180344205": "categoryid$pcmcat1496180344205",
"pcmcat1754489958914": "categoryid$pcmcat1754489958914",
"pcmcat1754490913800": "categoryid$pcmcat1754490913800",
"pcmcat156300050006": "categoryid$pcmcat156300050006",
"pcmcat311200050005": "categoryid$pcmcat311200050005",
"pcmcat384300050013": "categoryid$pcmcat384300050013",
"pcmcat214700050000": "categoryid$pcmcat214700050000",
"pcmcat171900050026": "categoryid$pcmcat171900050026",
"pcmcat214700050002": "categoryid$pcmcat214700050002",
"pcmcat326000050010": "categoryid$pcmcat326000050010",
"pcmcat326000050008": "categoryid$pcmcat326000050008",
"wallets-grips-straps-mounts-travel-accessories": "wallets-grips-straps-mounts-travel-accessories",
"pcmcat1631123340977": "pcmcat1631123340977_categoryid$pcmcat748301221690",
"magsafe-multiple-station-chargers": "magsafe-multiple-station-chargers",
"pcmcat326000050006": "categoryid$pcmcat326000050006",
"pcmcat326300050011": "categoryid$pcmcat326300050011",
"pcmcat367400050001": "categoryid$pcmcat367400050001",
"pcmcat1484596902850": "categoryid$pcmcat1484596902850",
"abcat0916004": "categoryid$abcat0916004",
"abcat0905001": "categoryid$abcat0905001",
"abcat0916001": "categoryid$abcat0916001",
"pcmcat1684266033737": "categoryid$pcmcat1684266033737",
"pcmcat1707935982082": "pcmcat1707935982082_categoryid$abcat0905000",
"pcmcat1710866322171": "categoryid$pcmcat1710866322171",
"pcmcat196400050016": "pcmcat196400050016_categoryid$abcat0904003",
"pcmcat196400050015": "pcmcat196400050015_categoryid$abcat0904003",
"pcmcat1660745100989": "pcmcat1660745100989_categoryid$abcat0904003",
"pcmcat196400050017": "pcmcat196400050017_categoryid$abcat0904003",
"pcmcat1710865851607": "categoryid$pcmcat1710865851607",
"pcmcat1680897792908": "pcmcat1680897792908_categoryid$abcat0904003",
"abcat0904003": "categoryid$abcat0904003",
"pcmcat1564413660184": "categoryid$pcmcat1564413660184",
"pcmcat272800050011": "categoryid$pcmcat272800050011",
"pcmcat1597762271403": "categoryid$pcmcat1597762271403",
"pcmcat249700050007": "categoryid$pcmcat249700050007",
"pcmcat748300655035": "categoryid$pcmcat748300655035",
"abcat0106001": "categoryid$abcat0106001",
"pcmcat748300652083": "categoryid$pcmcat748300652083",
"pcmcat748300654236": "categoryid$pcmcat748300654236",
"pcmcat272800050000": "categoryid$pcmcat272800050000",
"pcmcat312500050006": "categoryid$pcmcat312500050006",
"pcmcat138200050001": "categoryid$pcmcat138200050001",
"abcat0809004": "categoryid$abcat0809004",
"abcat0809002": "categoryid$abcat0809002",
"abcat0106021": "categoryid$abcat0106021",
"pcmcat1760724127414": "categoryid$pcmcat1760724127414",
"pcmcat1688401595784": "categoryid$pcmcat1688401595784",
"pcmcat219100050010": "categoryid$pcmcat219100050010",
"pcmcat748300353430": "categoryid$pcmcat748300353430",
"pcmcat286300050020": "pcmcat286300050020",
"pcmcat1563557714651": "categoryid$pcmcat1563557714651",
"pcmcat1578693425373": "categoryid$pcmcat1578693425373",
"pcmcat367400050002": "categoryid$pcmcat367400050002",
"pcmcat1547762619778": "categoryid$pcmcat1547762619778",
"abcat0912011": "categoryid$abcat0912011",
"abcat0912022": "categoryid$abcat0912022",
"abcat0912013": "categoryid$abcat0912013",
"abcat0912012": "categoryid$abcat0912012"
},
"channels": {
"event": "1452471895047012535",
"memberPick": "1452811066823278696",
"memberDeal": "1452811121776922794",
"soldOut": "1451799526540050512",
"60-70": "1451747974156914799",
"50-60": "1451747992297013318",
"70-80": "1451747949313921094",
"80-90": "1451747927923101737",
"90-100": "1451747737501564938"
}
}
scripts/db_seed/production/corsair.json
-15
View File
@@ -1,15 +0,0 @@
{
"categories": [
"pc-cases",
"wireless-keyboards"
],
"channels": {
"50-60": "1525179737700306944",
"60-70": "1525179526898913280",
"70-80": "1525179285382369412",
"80-90": "1525180350035394590",
"90-100": "1525178672510664894",
"40-49": "1537283810452832357",
"newListing": "1537179654849568950"
}
}
scripts/db_seed/production/hp.json
-35
View File
@@ -1,35 +0,0 @@
{
"categories": [
"vwa/business-solutions/bizcat=Workstation",
"vwa/business-solutions/bizcat=Laptop",
"vwa/business-solutions/form=Mobile-workstation",
"vwa/laptops/usage=Gaming",
"vwa/accessories-88342--1/newprod=Yes",
"vwa/business-solutions/bizcat=Thin-Client",
"vwa/business-solutions/bizcat=Desktop",
"vwa/desktops/usage=Gaming",
"vwa/accessories-88342--1/subcat=Audio-Video-Multimedia&useaudvid=Microphones,Webcams-and-cameras",
"vwa/printers/segm=Business,Enterprise,Small-medium-business",
"vwa/printers/segm=Home,Home-and-home-office,Small-Office",
"vwa/laptops/segm=Home",
"vwa/accessories-88342--1/subcat=Headsets",
"vwa/accessories-88342--1/subcat=Bags-Cases",
"vwa/accessories-88342--1/subcat=Docking-Stations-Hubs",
"vwa/gaming-3074457345617980168--1/gamingcat=Accessories,Monitors",
"vwa/ink--toner---paper/inktyp=High-Yield-Ink-Cartridges",
"vwa/accessories-88342--1/typemicekbd=Stylus-Pens&subcat=Mice-Keyboards",
"vwa/accessories-88342--1/subcat=Home-Office-Connectivity",
"vwa/accessories-88342--1/subcat=PC-Components",
"vwa/desktops/segm=Home",
"vwa/accessories-88342--1/brandaccessories=OMEN",
"vwa/accessories-88342--1/subcat=Microphones-Webcams"
],
"channels": {
"50-60": "1440486150723604551",
"60-70": "1440486181329703013",
"70-80": "1440486200929419335",
"80-90": "1440486221221724160",
"90-100": "1440486248895611103",
"soldOut": "1440486290490523700"
}
}
scripts/db_seed/production/lg.json
-30
View File
@@ -1,30 +0,0 @@
{
"categories": [
"window_air_conditioners",
"portable_air_conditioners",
"tvs",
"4k_uhd_tvs",
"monitors",
"speakers",
"microwaves",
"refrigerators",
"washers",
"dryers",
"ovens",
"dishwashers",
"lg_gram_pro_2_in_1_laptops",
"sound_bars",
"cooking_appliances",
"tv_and_home_theater_accessories",
"computing_accessories"
],
"channels": {
"soldOut": "1497368064126095671",
"40-49": "1495254588520005683",
"50-60": "1495254526784049242",
"60-70": "1495254479891595304",
"70-80": "1497367798882504945",
"80-90": "1495254455644455012",
"90-100": "1495254418533388399"
}
}
scripts/db_seed/production/northerntool.json
-46
View File
@@ -1,46 +0,0 @@
{
"categories": [
"1733",
"1381",
"6970",
"1354",
"1385",
"1382",
"1357",
"1352",
"1378",
"1387",
"1380",
"1362",
"1353",
"1377",
"1355",
"1369",
"1384",
"1363",
"1376",
"1388",
"1383",
"1367",
"1371",
"1366",
"1360",
"1356",
"1368",
"1389",
"1358",
"1365",
"1386",
"1361",
"1364"
],
"channels": {
"soldOut": "1510403235632906262",
"40-49": "1510403793576136814",
"50-60": "1510403590504841326",
"60-70": "1510266064305651752",
"70-80": "1510266139018531048",
"80-90": "1513214343620329572",
"90-100": "1510403299596173374"
}
}
scripts/db_seed/production/samsung.json
-44
View File
@@ -1,44 +0,0 @@
{
"categories": [
"01010000",
"01020000",
"01030000",
"01040000",
"01050000",
"01090000",
"01120000",
"02010000",
"03010000",
"04010000",
"04020000",
"04030000",
"04050000",
"04060000",
"04080000",
"05010000",
"05020000",
"05030000",
"07010000",
"07020000",
"07030000",
"07040000",
"08010000",
"08030000",
"08070000",
"08080000",
"08090000",
"08110000",
"08120000",
"09010000",
"11010000"
],
"channels": {
"40-49": "1470224395514417276",
"50-60": "1467755134514761728",
"60-70": "1467755157805596692",
"70-80": "1467755178731114528",
"80-90": "1467755178731114528",
"90-100": "1467755197148172484",
"soldOut": "1467757497023729755"
}
}
scripts/db_seed/production/samsung_shop.json
-36
View File
@@ -1,36 +0,0 @@
{
"categories": [
"01010000",
"01020000",
"01030000",
"01040000",
"01050000",
"01090000",
"01120000",
"02010000",
"03010000",
"04010000",
"04020000",
"04030000",
"04050000",
"04060000",
"04080000",
"05010000",
"05020000",
"05030000",
"07010000",
"07020000",
"07030000",
"07040000",
"08010000",
"08030000",
"08070000",
"08080000",
"08090000",
"08110000",
"08120000",
"09010000",
"11010000"
],
"channels": {}
}
scripts/db_seed/production/woot.json
-21
View File
@@ -1,21 +0,0 @@
{
"categories": [
"tools",
"sport",
"home",
"grocery",
"pc",
"tech"
],
"channels": {
"50-60": "1428512471723016283",
"60-70": "1428512958081798254",
"70-80": "1428512446678827099",
"80-90": "1428512446678827099",
"90-100": "1428512424176259124",
"event": "1428512391775387789",
"appExclusive": "1428512332446695434",
"soldOut": "1428512362763128933",
"40-49": "1479229158411468890"
}
}
scripts/db_seed/testing/apple.json
-65
View File
@@ -1,65 +0,0 @@
{
"categories": {
"home/shop_iphone/family/iphone_17_pro": "html",
"home/shop_iphone/family/iphone_air": "html",
"home/shop_iphone/family/iphone_17": "html",
"home/shop_iphone/family/iphone_17e": "html",
"home/shop_iphone/family/iphone_16": "html",
"home/shop_mac/family/macbook_neo": "html",
"home/shop_mac/family/macbook_air_13_inch": "html",
"home/shop_mac/family/macbook_air_15_inch": "html",
"home/shop_mac/family/macbook_pro_14_inch": "html",
"home/shop_mac/family/macbook_pro_16_inch": "html",
"home/shop_mac/family/imac": "html",
"home/shop_mac/family/mac_mini": "html",
"home/shop_mac/family/mac_studio": "html",
"home/shop_mac/family/studio_display": "html",
"home/shop_mac/family/studio_display_xdr": "html",
"/shop/accessories/all/accessibility": "grid",
"/shop/accessories/all/cases-protection": "grid",
"home/shop_ipad/family/ipad_pro": "html",
"home/shop_ipad/family/ipad_air": "html",
"home/shop_ipad/family/ipad": "html",
"home/shop_ipad/family/ipad_mini": "html",
"home/shop_watch/family/apple_watch": "html",
"home/shop_watch/family/apple_watch_se": "html",
"home/shop_watch/family/apple_watch_ultra": "html",
"home/shop_watch/family/apple_watch_hermes": "html",
"home/shop_watch/family/apple_watch_hermes_ultra": "html",
"home/shop_vision/family/apple_vision_pro": "html",
"home/shop_airpods/family/airpods_4": "html",
"home/shop_airpods/family/airpods_pro_3": "html",
"home/shop_airpods/family/airpods_max_2": "html",
"home/shop_homepod/family/homepod": "html",
"home/shop_homepod/family/homepod_mini": "html",
"home/shop_tv/family/apple_tv_4k": "html",
"/shop/accessories/all/chargers-adapters": "grid",
"/shop/accessories/all/travel-essentials": "grid",
"/shop/accessories/all/seasonal-colors": "grid",
"/shop/airtag/accessories/airtag-essentials": "grid",
"/shop/ipad/accessories/content-creation": "grid",
"/shop/mac/accessories/content-creation": "grid",
"/shop/vision/accessories/apple-vision-pro-essentials": "grid",
"/shop/accessories/all/college-essentials": "grid",
"/shop/accessories/all/content-creation": "grid",
"/shop/accessories/all/drives-storage": "grid",
"/shop/accessories/all/gaming": "grid",
"/shop/accessories/all/headphones-speakers": "grid",
"/shop/accessories/all/health-fitness": "grid",
"/shop/accessories/all/homekit": "grid",
"/shop/accessories/all/made-by-apple": "grid",
"/shop/accessories/all/mice-keyboards": "grid",
"/shop/accessories/all/new-arrivals": "grid",
"/shop/accessories/all/office": "grid",
"/shop/accessories/all/software": "grid",
"/shop/refurbished/mac": "grid",
"/shop/refurbished/ipad": "grid",
"/shop/refurbished/iphone": "grid",
"/shop/refurbished/watch": "grid",
"/shop/refurbished/appletv": "grid",
"/shop/refurbished/accessories": "grid",
"/shop/refurbished/airpods": "grid",
"/shop/refurbished/homepod": "grid"
},
"channels": {}
}
scripts/db_seed/testing/bestbuy.json
-144
View File
@@ -1,144 +0,0 @@
{
"categories": {
"pcmcat1764872919906": "categoryid$pcmcat1764872919906",
"abcat0715000": "categoryid$abcat0715000",
"pcmcat1662993575810": "categoryid$pcmcat1662993575810",
"pcmcat287600050002": "categoryid$pcmcat287600050002",
"pcmcat287600050003": "categoryid$pcmcat287600050003",
"pcmcat1763397583811": "pcmcat1763397583811_categoryid$pcmcat241600050001",
"abcat0208000": "categoryid$abcat0208000",
"pcmcat1591132221892": "pcmcat1591132221892_categoryid$abcat0500000",
"pcmcat304600050011": "categoryid$pcmcat304600050011",
"abcat0507000": "categoryid$abcat0507000",
"pcmcat243800050005": "categoryid$pcmcat243800050005",
"abcat0401000": "categoryid$abcat0401000",
"abcat0410000": "categoryid$abcat0410000",
"pcmcat369900050001": "categoryid$pcmcat369900050001",
"pcmcat1762875326350": "pcmcat1762875326350_categoryid$abcat0915000",
"abcat0915001": "categoryid$abcat0915001",
"pcmcat1762453391793": "pcmcat1762453391793_categoryid$abcat0915004",
"pcmcat1717008056314": "categoryid$pcmcat1717008056314",
"pcmcat244100050013": "categoryid$pcmcat244100050013",
"pcmcat333300050027": "categoryid$pcmcat333300050027",
"abcat0901004": "categoryid$abcat0901004",
"abcat0901005": "categoryid$abcat0901005",
"abcat0901006": "categoryid$abcat0901006",
"abcat0901001": "categoryid$abcat0901001",
"abcat0901003": "categoryid$abcat0901003",
"pcmcat313100050031": "categoryid$pcmcat313100050031",
"pcmcat144700050004": "categoryid$pcmcat144700050004",
"pcmcat308100050020": "categoryid$pcmcat308100050020",
"pcmcat748301598672": "categoryid$pcmcat748301598672",
"pcmcat1477674238305": "categoryid$pcmcat1477674238305",
"pcmcat1602773246733": "categoryid$abcat0503000",
"pcmcat345400050002": "categoryid$pcmcat345400050002",
"pcmcat308100050014": "categoryid$pcmcat308100050014",
"pcmcat308100050018": "categoryid$pcmcat308100050018",
"pcmcat308100050017": "categoryid$pcmcat308100050017",
"pcmcat1515445788580": "categoryid$pcmcat1515445788580",
"pcmcat254000050007": "categoryid$pcmcat254000050007",
"pcmcat1586207767722": "categoryid$pcmcat1586207767722",
"pcmcat179200050010": "categoryid$pcmcat179200050010",
"pcmcat1651764083616": "categoryid$pcmcat1651764083616",
"pcmcat1615397403616": "categoryid$pcmcat1615397403616",
"pcmcat270300050004": "categoryid$pcmcat270300050004",
"pcmcat1651764174444": "categoryid$pcmcat1651764174444",
"pcmcat179200050004": "categoryid$pcmcat179200050004",
"pcmcat270300050005": "categoryid$pcmcat270300050005",
"pcmcat179200050007": "categoryid$pcmcat179200050007",
"pcmcat1631303385635": "categoryid$pcmcat1631303385635",
"pcmcat179200050008": "categoryid$pcmcat179200050008",
"pcmcat1597939198513": "categoryid$pcmcat1597939198513",
"pcmcat179200050003": "categoryid$pcmcat179200050003",
"pcmcat1491416706156": "categoryid$pcmcat1491416706156",
"pcmcat1592600995496": "categoryid$pcmcat1592600995496",
"pcmcat334800050011": "categoryid$pcmcat334800050011",
"abcat0205006": "categoryid$abcat0205006",
"pcmcat273800050032": "categoryid$pcmcat273800050032",
"pcmcat1545417879756": "pcmcat1545417879756_categoryid$pcmcat250300050003",
"pcmcat250300050008": "categoryid$pcmcat250300050008",
"pcmcat265100050011": "categoryid$pcmcat265100050011",
"abcat0302005": "categoryid$abcat0302005",
"abcat0302012": "categoryid$abcat0302012",
"abcat0302034": "categoryid$abcat0302034",
"abcat0302001": "categoryid$abcat0302001",
"abcat0302011": "categoryid$abcat0302011",
"pcmcat326300050013": "categoryid$pcmcat326300050013",
"abcat0302019": "categoryid$abcat0302019",
"pcmcat1493671011172": "categoryid$pcmcat1493671011172",
"pcmcat165900050023": "categoryid$pcmcat165900050023",
"pcmcat330300050001": "categoryid$pcmcat330300050001",
"abcat0305000": "categoryid$abcat0305000",
"pcmcat1762288382250": "pcmcat1762288382250_categoryid$abcat0301000",
"pcmcat748301308133": "categoryid$pcmcat748301308133",
"pcmcat1496180344205": "categoryid$pcmcat1496180344205",
"pcmcat1754489958914": "categoryid$pcmcat1754489958914",
"pcmcat1754490913800": "categoryid$pcmcat1754490913800",
"pcmcat156300050006": "categoryid$pcmcat156300050006",
"pcmcat311200050005": "categoryid$pcmcat311200050005",
"pcmcat384300050013": "categoryid$pcmcat384300050013",
"pcmcat214700050000": "categoryid$pcmcat214700050000",
"pcmcat171900050026": "categoryid$pcmcat171900050026",
"pcmcat214700050002": "categoryid$pcmcat214700050002",
"pcmcat326000050010": "categoryid$pcmcat326000050010",
"pcmcat326000050008": "categoryid$pcmcat326000050008",
"wallets-grips-straps-mounts-travel-accessories": "wallets-grips-straps-mounts-travel-accessories",
"pcmcat1631123340977": "pcmcat1631123340977_categoryid$pcmcat748301221690",
"magsafe-multiple-station-chargers": "magsafe-multiple-station-chargers",
"pcmcat326000050006": "categoryid$pcmcat326000050006",
"pcmcat326300050011": "categoryid$pcmcat326300050011",
"pcmcat367400050001": "categoryid$pcmcat367400050001",
"pcmcat1484596902850": "categoryid$pcmcat1484596902850",
"abcat0916004": "categoryid$abcat0916004",
"abcat0905001": "categoryid$abcat0905001",
"abcat0916001": "categoryid$abcat0916001",
"pcmcat1684266033737": "categoryid$pcmcat1684266033737",
"pcmcat1707935982082": "pcmcat1707935982082_categoryid$abcat0905000",
"pcmcat1710866322171": "categoryid$pcmcat1710866322171",
"pcmcat196400050016": "pcmcat196400050016_categoryid$abcat0904003",
"pcmcat196400050015": "pcmcat196400050015_categoryid$abcat0904003",
"pcmcat1660745100989": "pcmcat1660745100989_categoryid$abcat0904003",
"pcmcat196400050017": "pcmcat196400050017_categoryid$abcat0904003",
"pcmcat1710865851607": "categoryid$pcmcat1710865851607",
"pcmcat1680897792908": "pcmcat1680897792908_categoryid$abcat0904003",
"abcat0904003": "categoryid$abcat0904003",
"pcmcat1564413660184": "categoryid$pcmcat1564413660184",
"pcmcat272800050011": "categoryid$pcmcat272800050011",
"pcmcat1597762271403": "categoryid$pcmcat1597762271403",
"pcmcat249700050007": "categoryid$pcmcat249700050007",
"pcmcat748300655035": "categoryid$pcmcat748300655035",
"abcat0106001": "categoryid$abcat0106001",
"pcmcat748300652083": "categoryid$pcmcat748300652083",
"pcmcat748300654236": "categoryid$pcmcat748300654236",
"pcmcat272800050000": "categoryid$pcmcat272800050000",
"pcmcat312500050006": "categoryid$pcmcat312500050006",
"pcmcat138200050001": "categoryid$pcmcat138200050001",
"abcat0809004": "categoryid$abcat0809004",
"abcat0809002": "categoryid$abcat0809002",
"abcat0106021": "categoryid$abcat0106021",
"pcmcat1760724127414": "categoryid$pcmcat1760724127414",
"pcmcat1688401595784": "categoryid$pcmcat1688401595784",
"pcmcat219100050010": "categoryid$pcmcat219100050010",
"pcmcat748300353430": "categoryid$pcmcat748300353430",
"pcmcat286300050020": "pcmcat286300050020",
"pcmcat1563557714651": "categoryid$pcmcat1563557714651",
"pcmcat1578693425373": "categoryid$pcmcat1578693425373",
"pcmcat367400050002": "categoryid$pcmcat367400050002",
"pcmcat1547762619778": "categoryid$pcmcat1547762619778",
"abcat0912011": "categoryid$abcat0912011",
"abcat0912022": "categoryid$abcat0912022",
"abcat0912013": "categoryid$abcat0912013",
"abcat0912012": "categoryid$abcat0912012"
},
"channels": {
"event": "1524040238064865393",
"memberPick": "1524040239369420841",
"memberDeal": "1524040240246030336",
"soldOut": "1524040240879374397",
"60-70": "1524040241902518414",
"50-60": "1524040242632458412",
"70-80": "1524040243450482779",
"80-90": "1524040244406517811",
"90-100": "1524040245136592957"
}
}
scripts/db_seed/testing/corsair.json
-22
View File
@@ -1,22 +0,0 @@
{
"categories": [
"pc-cases",
"keyboards",
"gaming-headsets",
"mousepads",
"leverless-controllers",
"ambient-lights",
"gaming-gear-accessories",
"cpu-coolers",
"gaming-computers",
"icue-link-components",
"gaming-furniture"
],
"channels": {
"50-60": "1525179737700306944",
"60-70": "1525179526898913280",
"70-80": "1525179285382369412",
"80-90": "1525180350035394590",
"90-100": "1525178672510664894"
}
}
scripts/db_seed/testing/hp.json
-35
View File
@@ -1,35 +0,0 @@
{
"categories": [
"vwa/business-solutions/bizcat=Workstation",
"vwa/business-solutions/bizcat=Laptop",
"vwa/business-solutions/form=Mobile-workstation",
"vwa/laptops/usage=Gaming",
"vwa/accessories-88342--1/newprod=Yes",
"vwa/business-solutions/bizcat=Thin-Client",
"vwa/business-solutions/bizcat=Desktop",
"vwa/desktops/usage=Gaming",
"vwa/accessories-88342--1/subcat=Audio-Video-Multimedia&useaudvid=Microphones,Webcams-and-cameras",
"vwa/printers/segm=Business,Enterprise,Small-medium-business",
"vwa/printers/segm=Home,Home-and-home-office,Small-Office",
"vwa/laptops/segm=Home",
"vwa/accessories-88342--1/subcat=Headsets",
"vwa/accessories-88342--1/subcat=Bags-Cases",
"vwa/accessories-88342--1/subcat=Docking-Stations-Hubs",
"vwa/gaming-3074457345617980168--1/gamingcat=Accessories,Monitors",
"vwa/ink--toner---paper/inktyp=High-Yield-Ink-Cartridges",
"vwa/accessories-88342--1/typemicekbd=Stylus-Pens&subcat=Mice-Keyboards",
"vwa/accessories-88342--1/subcat=Home-Office-Connectivity",
"vwa/accessories-88342--1/subcat=PC-Components",
"vwa/desktops/segm=Home",
"vwa/accessories-88342--1/brandaccessories=OMEN",
"vwa/accessories-88342--1/subcat=Microphones-Webcams"
],
"channels": {
"soldOut": "1523153716566560859",
"60-70": "1523153305897926776",
"50-60": "1523153047579136010",
"70-80": "1523153377587105892",
"80-90": "1523153568054513765",
"90-100": "1523153638971801710"
}
}
scripts/db_seed/testing/lg.json
-29
View File
@@ -1,29 +0,0 @@
{
"categories": [
"window_air_conditioners",
"portable_air_conditioners",
"tvs",
"4k_uhd_tvs",
"monitors",
"speakers",
"microwaves",
"refrigerators",
"washers",
"dryers",
"ovens",
"dishwashers",
"lg_gram_pro_2_in_1_laptops",
"sound_bars",
"cooking_appliances",
"tv_and_home_theater_accessories",
"computing_accessories"
],
"channels": {
"soldOut": "1524233737431421028",
"60-70": "1524233738748428328",
"50-60": "1524233739503407174",
"70-80": "1524233740417630238",
"80-90": "1524233741185187950",
"90-100": "1524233742036635852"
}
}
scripts/db_seed/testing/northerntool.json
-46
View File
@@ -1,46 +0,0 @@
{
"categories": [
"1733",
"1381",
"6970",
"1354",
"1385",
"1382",
"1357",
"1352",
"1378",
"1387",
"1380",
"1362",
"1353",
"1377",
"1355",
"1369",
"1384",
"1363",
"1376",
"1388",
"1383",
"1367",
"1371",
"1366",
"1360",
"1356",
"1368",
"1389",
"1358",
"1365",
"1386",
"1361",
"1364"
],
"channels": {
"soldOut": "1524233828661727292",
"60-70": "1524233830301433926",
"50-60": "1524233832838987827",
"70-80": "1524233834655387738",
"80-90": "1524233836714791003",
"90-100": "1524233837846991072",
"40-49": "1524839581055324160"
}
}
scripts/db_seed/testing/samsung.json
-43
View File
@@ -1,43 +0,0 @@
{
"categories": [
"01010000",
"01020000",
"01030000",
"01040000",
"01050000",
"01090000",
"01120000",
"02010000",
"03010000",
"04010000",
"04020000",
"04030000",
"04050000",
"04060000",
"04080000",
"05010000",
"05020000",
"05030000",
"07010000",
"07020000",
"07030000",
"07040000",
"08010000",
"08030000",
"08070000",
"08080000",
"08090000",
"08110000",
"08120000",
"09010000",
"11010000"
],
"channels": {
"soldOut": "1524233898702278741",
"60-70": "1524233899897520272",
"50-60": "1524233900866539642",
"70-80": "1524233901961117816",
"80-90": "1524233902611234856",
"90-100": "1524233903945154794"
}
}
scripts/db_seed/testing/samsung_shop.json
-36
View File
@@ -1,36 +0,0 @@
{
"categories": [
"01010000",
"01020000",
"01030000",
"01040000",
"01050000",
"01090000",
"01120000",
"02010000",
"03010000",
"04010000",
"04020000",
"04030000",
"04050000",
"04060000",
"04080000",
"05010000",
"05020000",
"05030000",
"07010000",
"07020000",
"07030000",
"07040000",
"08010000",
"08030000",
"08070000",
"08080000",
"08090000",
"08110000",
"08120000",
"09010000",
"11010000"
],
"channels": {}
}
scripts/db_seed/testing/woot.json
-21
View File
@@ -1,21 +0,0 @@
{
"categories": [
"tools",
"sport",
"home",
"grocery",
"pc",
"tech"
],
"channels": {
"event": "1523804538702794892",
"appExclusive": "1523804539302449342",
"priceDrop800": "1523804540271333530",
"soldOut": "1523804541164978266",
"60-70": "1523804541802385484",
"50-60": "1523804542783717652",
"70-80": "1523804545342242838",
"80-90": "1523804546286223482",
"90-100": "1523804547003453574"
}
}
src/monitor_utils/config.py
+4 -4
View File
@@ -29,7 +29,7 @@ def is_production(development_mode: DevelopmentMode) -> bool:
async def get_config( async def get_config(
monitor_name: str, monitor: str,
use_production: bool = False, use_production: bool = False,
mongo_client: AsyncIOMotorClient = None, mongo_client: AsyncIOMotorClient = None,
) -> dict: ) -> dict:
@@ -39,7 +39,7 @@ async def get_config(
development, so a dev run never reads production routing. development, so a dev run never reads production routing.
Args: Args:
monitor_name: Monitor identifier (the config document _id). monitor: Monitor identifier (the config document _id).
use_production: Read the production configs collection instead of use_production: Read the production configs collection instead of
configs_testing. configs_testing.
mongo_client: Client to read through; the shared cluster client when mongo_client: Client to read through; the shared cluster client when
@@ -58,10 +58,10 @@ async def get_config(
else f"{CONFIGS_COLLECTION_NAME}_testing" else f"{CONFIGS_COLLECTION_NAME}_testing"
) )
config = await get_collection(collection_name, mongo_client=mongo_client).find_one( config = await get_collection(collection_name, mongo_client=mongo_client).find_one(
{"_id": monitor_name} {"_id": monitor}
) )
if config is None: if config is None:
raise ConfigNotFoundError(monitor_name) raise ConfigNotFoundError(monitor)
return config return config
src/monitor_utils/constants.py
+9 -9
View File
@@ -33,17 +33,17 @@ FAILURE_DELAY = 3
TIMEOUT_SECONDS = 30 TIMEOUT_SECONDS = 30
CYCLE_TIMEOUT_SECONDS = 600 CYCLE_TIMEOUT_SECONDS = 600
# curl_cffi's AsyncSession defaults to 10 handles and blocks on an untimed
# pool.get() past that, so a monitor fanning out wider silently serialises with
# no timeout covering the wait. Handles are allocated lazily (the pool holds
# None placeholders until popped), so a generous ceiling costs a low-concurrency
# monitor nothing. Monitors fanning out past this pass max_clients explicitly.
MAX_CURL_HANDLES = 64 MAX_CURL_HANDLES = 64
# Budget for closing a session. curl_cffi's close awaits the multi handle with
# no timeout of its own, so a wedged transfer would otherwise park whatever is
# closing the session, including an `async with` exit.
SESSION_CLOSE_TIMEOUT_SECONDS = 10 SESSION_CLOSE_TIMEOUT_SECONDS = 10
CLIENT_IDENTIFIERS = ["chrome", "chrome_android", "safari", "safari_ios"] CLIENT_IDENTIFIERS = [
"chrome119",
"chrome120",
"chrome124",
"chrome131",
"chrome131_android",
"safari",
"safari_ios",
]
CLIENT_TIMEOUT = aiohttp.ClientTimeout( CLIENT_TIMEOUT = aiohttp.ClientTimeout(
total=TIMEOUT_SECONDS, sock_connect=30, sock_read=30 total=TIMEOUT_SECONDS, sock_connect=30, sock_read=30
) )
src/monitor_utils/database.py
+31 -16
View File
@@ -7,6 +7,7 @@ alert references for new products and price changes.
""" """
import logging import logging
import math
from dataclasses import asdict from dataclasses import asdict
from datetime import datetime, timezone from datetime import datetime, timezone
from functools import cache from functools import cache
@@ -112,7 +113,7 @@ async def add_products_to_database(
products: list[Product], products: list[Product],
collection: AsyncIOMotorCollection, collection: AsyncIOMotorCollection,
category_identifier: str, category_identifier: str,
monitor_name: str, monitor: str,
use_composite: bool = False, use_composite: bool = False,
): ):
"""Upsert products, tracking deduplicated price history and emitting alerts. """Upsert products, tracking deduplicated price history and emitting alerts.
@@ -138,7 +139,7 @@ async def add_products_to_database(
products: Products to upsert. products: Products to upsert.
collection: Target MongoDB collection. collection: Target MongoDB collection.
category_identifier: Category suffix appended to composite ids. category_identifier: Category suffix appended to composite ids.
monitor_name: Monitor name stamped onto emitted alerts for consumer monitor: Monitor name stamped onto emitted alerts for consumer
routing. routing.
use_composite: Build composite ids so multi-variant monitors (Woot) use_composite: Build composite ids so multi-variant monitors (Woot)
keep one doc per variant; other monitors collapse to one doc per keep one doc per variant; other monitors collapse to one doc per
@@ -180,7 +181,7 @@ async def add_products_to_database(
never_seen_price = new_price is not None and new_price not in recorded_prices never_seen_price = new_price is not None and new_price not in recorded_prices
set_on_insert = {"discoveredDate": now} set_on_insert = {"discoveredDate": now}
if existing_doc is None or never_seen_price: if existing_doc is None or never_seen_price:
alert = build_alert(monitor_name, document, now) alert = build_alert(monitor, document, now)
alert_ops.append( alert_ops.append(
UpdateOne( UpdateOne(
{"product_id": doc_id, "claimed": False}, {"product_id": doc_id, "claimed": False},
@@ -279,19 +280,22 @@ def extract_recorded_prices(existing_doc: dict | None) -> set[float]:
return {entry["price"] for entry in existing_doc.get("pastPrices", [])} return {entry["price"] for entry in existing_doc.get("pastPrices", [])}
def build_alert(monitor_name: str, document: dict, now: datetime) -> dict: def build_alert(monitor: str, document: dict, now: datetime) -> dict:
"""Build an alert referencing a product for consumer delivery. """Build an alert referencing a product for consumer delivery.
The alert carries its own copy of the document's newListing flag: the The alert carries its own copy of the document's newListing flag and
product doc is rewritten every cycle, so its flag flips False one cycle priceBeforeDrop: the product doc is rewritten every cycle, so its flag
after discovery, while a discovery alert may sit in a backed-up queue flips False one cycle after discovery and its priceBeforeDrop is
longer than that. The alert's copy is written once and cannot rot. cleared by the next cycle that sees no drop, while an alert may sit in
a backed-up queue longer than that. The alert's copies are written once
and cannot rot.
Args: Args:
monitor_name: Monitor that produced the product, used by the monitor: Monitor that produced the product, used by the
consumer to resolve the owning collection and thresholds. consumer to resolve the owning collection and thresholds.
document: Product document supplying the id, pricing for the document: Product document supplying the id, pricing for the
priority calculation, and the stamped newListing flag. priority calculation, and the stamped newListing flag and
priceBeforeDrop.
now: Detection timestamp stored as createdAt. now: Detection timestamp stored as createdAt.
Returns: Returns:
@@ -299,13 +303,14 @@ def build_alert(monitor_name: str, document: dict, now: datetime) -> dict:
""" """
return { return {
"monitor": monitor_name, "monitor": monitor,
"product_id": document["_id"], "product_id": document["_id"],
"priority": calculate_priority(document), "priority": calculate_priority(document),
"claimed": False, "claimed": False,
"attempts": 0, "attempts": 0,
"createdAt": now, "createdAt": now,
"newListing": document.get("newListing", False), "newListing": document.get("newListing", False),
"priceBeforeDrop": document.get("priceBeforeDrop"),
} }
@@ -314,10 +319,13 @@ def calculate_priority(document: dict) -> int:
Always recalculated from current and original price so the priority Always recalculated from current and original price so the priority
reflects the prices at alert time rather than any stale stored reflects the prices at alert time rather than any stale stored
percentage. Missing prices, zero original prices, and price increases percentage. Missing prices, zero original prices, non-finite prices, a
resolve to priority 0. A product with no MSRP has no percentage to rank discount too large to represent, and price increases resolve to priority
on and takes NO_MSRP_PRIORITY instead, since a real 0 would leave it 0, so one product's bad pricing can never raise out of the whole cycle (a
permanently behind every other alert in the claim query's sort. A new retailer field like "Infinity" survives float() and used to crash the
monitor with an OverflowError). A product with no MSRP has no percentage
to rank on and takes NO_MSRP_PRIORITY instead, since a real 0 would leave
it permanently behind every other alert in the claim query's sort. A new
listing is floored at NEW_LISTING_PRIORITY for the same reason: it may listing is floored at NEW_LISTING_PRIORITY for the same reason: it may
carry any discount at all, and its percentage rank still wins when higher. carry any discount at all, and its percentage rank still wins when higher.
@@ -333,10 +341,17 @@ def calculate_priority(document: dict) -> int:
if current_price is None or not original_price: if current_price is None or not original_price:
return 0 return 0
if not (math.isfinite(current_price) and math.isfinite(original_price)):
return 0
if not document.get("hasMsrp", True): if not document.get("hasMsrp", True):
return NO_MSRP_PRIORITY return NO_MSRP_PRIORITY
percentage = int(((original_price - current_price) / original_price) * 100) discount = ((original_price - current_price) / original_price) * 100
if not math.isfinite(discount):
return 0
percentage = int(discount)
priority = min(max(percentage // 10, 0), 10) priority = min(max(percentage // 10, 0), 10)
if document.get("newListing"): if document.get("newListing"):
return max(priority, NEW_LISTING_PRIORITY) return max(priority, NEW_LISTING_PRIORITY)
src/monitor_utils/models.py
+2 -2
View File
@@ -29,7 +29,7 @@ class ConsumerConfig:
"""Configuration for a single retailer's consumer. """Configuration for a single retailer's consumer.
Attributes: Attributes:
monitor_name: Retailer name (matches setup_config key). monitor: Retailer name.
collection: MongoDB collection for this retailer's products. collection: MongoDB collection for this retailer's products.
channel_ids: Discord webhook/channel routing config. channel_ids: Discord webhook/channel routing config.
min_percentage: Minimum discount % to trigger a ping. min_percentage: Minimum discount % to trigger a ping.
@@ -38,7 +38,7 @@ class ConsumerConfig:
the secondary rule. the secondary rule.
""" """
monitor_name: str monitor: str
collection: object collection: object
channel_ids: dict channel_ids: dict
min_percentage: int min_percentage: int
src/monitor_utils/runner.py
+44 -16
View File
@@ -1,11 +1,13 @@
import asyncio import asyncio
import logging import logging
import math
from collections.abc import Awaitable, Callable, Iterable from collections.abc import Awaitable, Callable, Iterable
from contextlib import asynccontextmanager, nullcontext from contextlib import asynccontextmanager, nullcontext
import aiohttp import aiohttp
from aioproxies import AioProxies from aioproxies import AioProxies
from curl_cffi.requests import AsyncSession from curl_cffi.requests import AsyncSession
from curl_cffi.requests.impersonate import BrowserTypeLiteral
from log_setup import setup_logging from log_setup import setup_logging
from motor.motor_asyncio import AsyncIOMotorCollection from motor.motor_asyncio import AsyncIOMotorCollection
from motor.motor_asyncio import AsyncIOMotorClient from motor.motor_asyncio import AsyncIOMotorClient
@@ -37,22 +39,22 @@ from monitor_utils.models import (
LOGGER = logging.getLogger(__name__) LOGGER = logging.getLogger(__name__)
def setup_monitor_logging(monitor_name: str, logging_level=LOGGING_LEVEL): def setup_monitor_logging(monitor: str, logging_level=LOGGING_LEVEL):
"""Configure JSON logging for a monitor and quiet noisy libraries. """Configure JSON logging for a monitor and quiet noisy libraries.
Call once at module import, at global scope, before creating module loggers, Call once at module import, at global scope, before creating module loggers,
so logging is configured synchronously at startup rather than inside the so logging is configured synchronously at startup rather than inside the
async init_monitor. Logs go to logs/latest.log with rotated history keyed by async init_monitor. Logs go to logs/latest.log with rotated history keyed by
monitor_name. monitor.
Args: Args:
monitor_name: Monitor name used as the rotated-history base. monitor: Monitor name used as the rotated-history base.
logging_level: Root logging level. logging_level: Root logging level.
""" """
setup_logging( setup_logging(
name="logs/latest.log", name="logs/latest.log",
history_name=monitor_name, history_name=monitor,
level=logging_level, level=logging_level,
output="json", output="json",
) )
@@ -60,7 +62,7 @@ def setup_monitor_logging(monitor_name: str, logging_level=LOGGING_LEVEL):
async def init_monitor( async def init_monitor(
monitor_name: str, monitor: str,
development_mode: DevelopmentMode, development_mode: DevelopmentMode,
production_proxy_path: str = "proxies.txt", production_proxy_path: str = "proxies.txt",
testing_proxy_path: str = "tests/configs/proxies.txt", testing_proxy_path: str = "tests/configs/proxies.txt",
@@ -72,7 +74,7 @@ async def init_monitor(
by setup_monitor_logging at module scope and is not touched here. by setup_monitor_logging at module scope and is not touched here.
Args: Args:
monitor_name: Monitor identifier used for config and collection lookup. monitor: Monitor identifier used for config and collection lookup.
development_mode: Selects production or testing config, collection, development_mode: Selects production or testing config, collection,
and proxy path. and proxy path.
production_proxy_path: Proxy file path used in production mode. production_proxy_path: Proxy file path used in production mode.
@@ -84,7 +86,7 @@ async def init_monitor(
use_production = is_production(development_mode) use_production = is_production(development_mode)
category_names, _, collection = await setup_config( category_names, _, collection = await setup_config(
monitor_name=monitor_name, development_mode=development_mode monitor=monitor, development_mode=development_mode
) )
proxy_path = production_proxy_path if use_production else testing_proxy_path proxy_path = production_proxy_path if use_production else testing_proxy_path
proxies = AioProxies.from_file(path=proxy_path) proxies = AioProxies.from_file(path=proxy_path)
@@ -92,7 +94,7 @@ async def init_monitor(
async def setup_config( async def setup_config(
monitor_name: str, monitor: str,
development_mode: DevelopmentMode, development_mode: DevelopmentMode,
mongo_client: AsyncIOMotorClient = None, mongo_client: AsyncIOMotorClient = None,
) -> tuple[ ) -> tuple[
@@ -105,7 +107,7 @@ async def setup_config(
during development. during development.
Args: Args:
monitor_name: Monitor identifier used for config and collection lookup. monitor: Monitor identifier used for config and collection lookup.
development_mode: Selects the production or testing config and development_mode: Selects the production or testing config and
collection. collection.
mongo_client: Client the config read and the returned collection use; mongo_client: Client the config read and the returned collection use;
@@ -117,14 +119,14 @@ async def setup_config(
use_production = is_production(development_mode) use_production = is_production(development_mode)
config = await get_config( config = await get_config(
monitor_name=monitor_name, monitor=monitor,
use_production=use_production, use_production=use_production,
mongo_client=mongo_client, mongo_client=mongo_client,
) )
category_names = config["categories"] category_names = config["categories"]
collection = get_collection( collection = get_collection(
monitor_name if use_production else f"{monitor_name}_testing", monitor if use_production else f"{monitor}_testing",
mongo_client=mongo_client, mongo_client=mongo_client,
) )
return category_names, config["channels"], collection return category_names, config["channels"], collection
@@ -159,7 +161,10 @@ def build_product_details(
) -> Product | None: ) -> Product | None:
"""Build a Product from a Mongo document if it passes discount thresholds. """Build a Product from a Mongo document if it passes discount thresholds.
Skips documents with a missing or zero original price. Passing requires Skips documents with a missing or zero original price, a non-finite
price, or a discount too large to represent: a retailer field like
"Infinity" survives float(), and the percentage math used to raise
OverflowError out of every caller. Passing requires
meeting min_percentage, or min_price_drop when provided. The caller may meeting min_percentage, or min_price_drop when provided. The caller may
bypass both thresholds (new listings send at any discount for monitors bypass both thresholds (new listings send at any discount for monitors
that opt in); which alerts qualify is the caller's decision, not read that opt in); which alerts qualify is the caller's decision, not read
@@ -172,7 +177,7 @@ def build_product_details(
min_percentage: Minimum discount percentage to pass. min_percentage: Minimum discount percentage to pass.
min_price_drop: Optional minimum absolute price drop to pass. min_price_drop: Optional minimum absolute price drop to pass.
bypass_thresholds: Build the product even when it fails both bypass_thresholds: Build the product even when it fails both
thresholds. The missing-price skip still applies. thresholds. The missing- and non-finite-price skips still apply.
Returns: Returns:
The built Product, or None when skipped or below thresholds. The built Product, or None when skipped or below thresholds.
@@ -190,6 +195,19 @@ def build_product_details(
) )
return return
# Skipped on BOTH paths, not just the percentage one: an MSRP-less product
# would otherwise carry the infinite price into its payload. Only floats can
# be non-finite, so ints and other types keep their existing behaviour.
if any(
isinstance(price, float) and not math.isfinite(price)
for price in (current_price, original_price)
):
LOGGER.warning(
"non-finite price, skipping",
extra={"product_id": product_id},
)
return
percentage = 0 percentage = 0
# Defaulted, not indexed: documents written before the field existed have no # Defaulted, not indexed: documents written before the field existed have no
# hasMsrp at all, and stale ones that drop out of a monitor's feed are never # hasMsrp at all, and stale ones that drop out of a monitor's feed are never
@@ -197,7 +215,15 @@ def build_product_details(
# pre-existing threshold path rather than silently routing them as MSRP-less. # pre-existing threshold path rather than silently routing them as MSRP-less.
has_msrp = product.get("hasMsrp", True) has_msrp = product.get("hasMsrp", True)
if has_msrp: if has_msrp:
percentage = int(((original_price - current_price) / original_price) * 100) discount = ((original_price - current_price) / original_price) * 100
if not math.isfinite(discount):
LOGGER.warning(
"discount not representable, skipping",
extra={"product_id": product_id},
)
return
percentage = int(discount)
price_drop = original_price - current_price price_drop = original_price - current_price
# A percentage code compounds onto the sale price, so an item can clear the # A percentage code compounds onto the sale price, so an item can clear the
# threshold on the combined discount even when the base discount would not. # threshold on the combined discount even when the base discount would not.
@@ -211,7 +237,7 @@ def build_product_details(
if not bypass_thresholds and not passes_percentage and not passes_price_drop: if not bypass_thresholds and not passes_percentage and not passes_price_drop:
return return
built = Product( built_product = Product(
product_id=product_id, product_id=product_id,
title=product["title"], title=product["title"],
url=product["url"], url=product["url"],
@@ -227,6 +253,7 @@ def build_product_details(
promo_dollar_off=product.get("promoDollarOff"), promo_dollar_off=product.get("promoDollarOff"),
promo_min_spend=product.get("promoMinSpend"), promo_min_spend=product.get("promoMinSpend"),
new_listing=new_listing, new_listing=new_listing,
asin=product.get("asin"),
), ),
timing=ProductTiming( timing=ProductTiming(
discovered_date=product["discoveredDate"], discovered_date=product["discoveredDate"],
@@ -249,7 +276,7 @@ def build_product_details(
member_deal=product["memberDeal"], member_deal=product["memberDeal"],
), ),
) )
return built return built_product
async def run_monitor_cycle( async def run_monitor_cycle(
@@ -376,6 +403,7 @@ async def run_bounded_cycle(
session_timeout: int = TIMEOUT_SECONDS, session_timeout: int = TIMEOUT_SECONDS,
max_clients: int = MAX_CURL_HANDLES, max_clients: int = MAX_CURL_HANDLES,
cycle_timeout: int = CYCLE_TIMEOUT_SECONDS, cycle_timeout: int = CYCLE_TIMEOUT_SECONDS,
impersonate: BrowserTypeLiteral | None = None,
**kwargs, **kwargs,
) -> CycleResult | None: ) -> CycleResult | None:
"""Run one monitor cycle on a fresh session under a hard time budget. """Run one monitor cycle on a fresh session under a hard time budget.
src/monitor_utils/sessions.py
+47 -1
View File
@@ -13,6 +13,7 @@ import random
from aioproxies import AioProxies from aioproxies import AioProxies
from aioweb import Response from aioweb import Response
from aioweb_tls import CurlCffi, TLSSession from aioweb_tls import CurlCffi, TLSSession
from curl_cffi.requests.impersonate import BrowserTypeLiteral
from monitor_utils.constants import ( from monitor_utils.constants import (
CLIENT_IDENTIFIERS, CLIENT_IDENTIFIERS,
MAX_CURL_HANDLES, MAX_CURL_HANDLES,
@@ -70,6 +71,7 @@ def get_default_session(
headers: dict | None = None, headers: dict | None = None,
timeout: int = TIMEOUT_SECONDS, timeout: int = TIMEOUT_SECONDS,
max_clients: int = MAX_CURL_HANDLES, max_clients: int = MAX_CURL_HANDLES,
impersonate: BrowserTypeLiteral | None = None,
) -> TLSSession: ) -> TLSSession:
"""Return a TLSSession impersonating a random client fingerprint. """Return a TLSSession impersonating a random client fingerprint.
@@ -98,13 +100,17 @@ def get_default_session(
checkout. Keep it at or above the monitor's real fan-out (the sum checkout. Keep it at or above the monitor's real fan-out (the sum
of its pagination semaphores plus any unsemaphored per-cycle of its pagination semaphores plus any unsemaphored per-cycle
requests) so the wait is never reachable. requests) so the wait is never reachable.
impersonate: A curl_cffi profile to pin instead of drawing one from
CLIENT_IDENTIFIERS, for a retailer whose bot protection only
tolerates some profiles from a given IP (B&H's Cloudflare refuses
every chrome profile on a flagged IP but still accepts safari).
Returns: Returns:
A BoundedCloseSession (a TLSSession whose close is bounded) over a A BoundedCloseSession (a TLSSession whose close is bounded) over a
curl_cffi backend with a random impersonation, with the proxy pool curl_cffi backend with a random impersonation, with the proxy pool
attached as proxy_pool. attached as proxy_pool.
""" """
impersonate = random.choice(CLIENT_IDENTIFIERS) impersonate = impersonate or random.choice(CLIENT_IDENTIFIERS)
session = BoundedCloseSession( session = BoundedCloseSession(
backend=CurlCffi(impersonate=impersonate), backend=CurlCffi(impersonate=impersonate),
headers=headers, headers=headers,
@@ -171,3 +177,43 @@ async def send_request(
) )
response.raise_for_status() response.raise_for_status()
return response return response
def get_response_json(
response: Response,
*,
logger: logging.Logger = LOGGER,
expected: type | tuple[type, ...] = dict,
**context,
) -> dict | list | None:
"""Parse a response body as JSON, or log it and return None.
aioweb's response.json() yields None for a non-JSON body, so a single
isinstance check covers both the empty/invalid case and a body of the
wrong shape. Callers stop on None the same way they did with the inline
check this replaces.
Args:
response: A 2xx aioweb Response from send_request.
logger: Logger the warning is emitted on; pass the monitor's own so
the record stays attributed to that monitor.
expected: Type(s) the parsed body must be an instance of.
**context: Extra fields for the warning record (category, page, ...).
Returns:
The parsed body when it matches `expected`, else None.
"""
data = response.json()
if isinstance(data, expected):
return data
logger.warning(
"unexpected response",
extra={
**context,
"body_type": type(data).__name__,
"body": str(data)[:200],
},
)
return None
tests/test_alert_priority.py
+17
View File
@@ -50,6 +50,23 @@ def test_broken_prices_still_outrank_nothing():
assert calculate_priority(document(None, 100.0, hasMsrp=False)) == 0 assert calculate_priority(document(None, 100.0, hasMsrp=False)) == 0
def test_non_finite_prices_resolve_to_zero_instead_of_raising():
"""float("Infinity") from a retailer field once crashed the whole monitor."""
inf, nan = float("inf"), float("nan")
assert calculate_priority(document(50.0, inf)) == 0
assert calculate_priority(document(inf, 100.0)) == 0
assert calculate_priority(document(-inf, 100.0)) == 0
assert calculate_priority(document(nan, 100.0)) == 0
assert calculate_priority(document(50.0, nan)) == 0
assert calculate_priority(document(50.0, inf, newListing=True)) == 0
def test_a_discount_too_large_to_represent_resolves_to_zero():
"""Finite prices can still overflow the ratio (a subnormal original)."""
assert calculate_priority(document(-1e308, 5e-324)) == 0
assert calculate_priority(document(1e308, 5e-324)) == 0
def test_new_listing_is_floored_not_buried(): def test_new_listing_is_floored_not_buried():
"""A low-discount new listing sends regardless, so it must not sort last.""" """A low-discount new listing sends regardless, so it must not sort last."""
assert ( assert (
tests/test_bootstrap_db.py
-302
View File
@@ -1,302 +0,0 @@
"""Unit tests for scripts/bootstrap_db.py with a fake Mongo at the leaf.
The $jsonSchema validator's runtime behavior is server-side and is not
covered here; it is verified against the live cluster when the script runs.
"""
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
import bootstrap_db # noqa: E402
class FakeCollection:
def __init__(self, name):
self.name = name
self.docs = {}
self.indexes = []
def find_one(self, query, projection=None):
return self.docs.get(query["_id"])
def update_one(self, query, update, upsert=False):
class Result:
upserted_id = None
result = Result()
if query["_id"] not in self.docs and upsert:
self.docs[query["_id"]] = {"_id": query["_id"], **update["$setOnInsert"]}
result.upserted_id = query["_id"]
return result
def create_index(self, keys, **options):
self.indexes.append((keys, options))
return "_".join(f"{field}_{direction}" for field, direction in keys)
class FakeDatabase:
def __init__(self, existing=()):
self.collections = {name: FakeCollection(name) for name in existing}
self.options = {name: {} for name in existing}
self.created = []
self.commands = []
def list_collection_names(self):
return list(self.collections)
def list_collections(self, filter=None):
names = [filter["name"]] if filter else list(self.collections)
return [
{"name": n, "options": self.options[n]}
for n in names
if n in self.collections
]
def create_collection(self, name, **options):
self.collections[name] = FakeCollection(name)
self.options[name] = options
self.created.append((name, options))
def command(self, name, collection, **kwargs):
self.commands.append((name, collection, kwargs))
self.options[collection].update(kwargs)
def __getitem__(self, name):
return self.collections.setdefault(name, FakeCollection(name))
@pytest.fixture
def seed_root(tmp_path):
for mode in ("production", "testing"):
(tmp_path / mode).mkdir()
(tmp_path / mode / "woot.json").write_text(
json.dumps({"categories": ["tech"], "channels": {"50-60": "1"}})
)
(tmp_path / "production" / "bestbuy.json").write_text(
json.dumps(
{
"categories": {"a": "categoryid$a"},
"channels": {"60-70": "2"},
"extra": "ignored",
}
)
)
(tmp_path / "control.json").write_text(
json.dumps({"monitors": {"names": ["woot"], "settings": {}}})
)
return tmp_path
def test_ensure_collection_creates_absent_with_options():
db = FakeDatabase()
assert bootstrap_db.ensure_collection(db, "configs", validator={"x": 1}) is True
assert db.created == [("configs", {"validator": {"x": 1}})]
def test_ensure_collection_skips_existing():
db = FakeDatabase(existing=["configs"])
assert bootstrap_db.ensure_collection(db, "configs") is False
assert db.created == []
def test_ensure_collection_dry_run_writes_nothing():
db = FakeDatabase()
assert bootstrap_db.ensure_collection(db, "configs", dry_run=True) is True
assert db.created == []
def test_ensure_config_validator_applies_collmod_once():
db = FakeDatabase(existing=["configs"])
assert bootstrap_db.ensure_config_validator(db, "configs") is True
assert db.commands == [
("collMod", "configs", {"validator": bootstrap_db.CONFIG_VALIDATOR})
]
assert bootstrap_db.ensure_config_validator(db, "configs") is False
assert len(db.commands) == 1
def test_ensure_config_validator_dry_run_and_missing_collection():
db = FakeDatabase(existing=["configs"])
assert bootstrap_db.ensure_config_validator(db, "configs", dry_run=True) is True
assert db.commands == []
assert bootstrap_db.ensure_config_validator(db, "nope") is False
def test_load_seed_configs_rejects_list_channels(tmp_path):
(tmp_path / "woot.json").write_text(
json.dumps({"categories": [], "channels": {"50-60": ["1", "2"]}})
)
with pytest.raises(ValueError, match="single id strings"):
bootstrap_db.load_seed_configs(tmp_path)
def test_alert_indexes_match_consumer_spec():
alerts = FakeCollection("alerts")
bootstrap_db.ensure_alert_indexes(alerts)
assert alerts.indexes == [
([("claimed", 1), ("priority", -1), ("createdAt", 1)], {}),
(
[("product_id", 1)],
{"unique": True, "partialFilterExpression": {"claimed": False}},
),
]
def test_load_seed_configs_keys_by_stem_and_drops_extra_fields(seed_root):
docs = bootstrap_db.load_seed_configs(seed_root / "production")
assert set(docs) == {"bestbuy", "woot"}
assert docs["bestbuy"] == {
"categories": {"a": "categoryid$a"},
"channels": {"60-70": "2"},
}
def test_load_seed_configs_rejects_missing_keys(tmp_path):
(tmp_path / "bad.json").write_text(json.dumps({"categories": []}))
with pytest.raises(ValueError, match="channels"):
bootstrap_db.load_seed_configs(tmp_path)
def test_load_seed_configs_rejects_wrong_shape_for_monitor(tmp_path):
(tmp_path / "bestbuy.json").write_text(
json.dumps({"categories": ["abcat0"], "channels": {}})
)
with pytest.raises(ValueError, match="categories must be dict for bestbuy"):
bootstrap_db.load_seed_configs(tmp_path)
(tmp_path / "bestbuy.json").unlink()
(tmp_path / "woot.json").write_text(
json.dumps({"categories": {"tech": "tech"}, "channels": {}})
)
with pytest.raises(ValueError, match="categories must be list for woot"):
bootstrap_db.load_seed_configs(tmp_path)
def test_load_seed_configs_empty_dir(tmp_path):
assert bootstrap_db.load_seed_configs(tmp_path) == {}
def test_seed_configs_inserts_absent_and_preserves_existing():
configs = FakeCollection("configs")
configs.docs["woot"] = {"_id": "woot", "categories": ["edited"], "channels": {}}
docs = {
"woot": {"categories": ["tech"], "channels": {}},
"hp": {"categories": [], "channels": {}},
}
assert bootstrap_db.seed_configs(configs, docs) == ["hp"]
assert configs.docs["woot"]["categories"] == ["edited"]
assert configs.docs["hp"] == {"_id": "hp", "categories": [], "channels": {}}
def test_seed_configs_dry_run_reports_without_writing():
configs = FakeCollection("configs")
docs = {"hp": {"categories": [], "channels": {}}}
assert bootstrap_db.seed_configs(configs, docs, dry_run=True) == ["hp"]
assert configs.docs == {}
def test_load_control_seed_keys_by_id(seed_root):
docs = bootstrap_db.load_control_seed(seed_root / "control.json")
assert docs == {"monitors": {"names": ["woot"], "settings": {}}}
def test_load_control_seed_rejects_non_object_doc(tmp_path):
path = tmp_path / "control.json"
path.write_text(json.dumps({"monitors": ["woot"]}))
with pytest.raises(ValueError, match="monitors"):
bootstrap_db.load_control_seed(path)
def test_seed_control_inserts_absent_and_preserves_existing():
control = FakeCollection("control")
control.docs["monitors"] = {"_id": "monitors", "names": ["edited"]}
docs = {"monitors": {"names": ["woot"]}, "other": {"x": 1}}
assert bootstrap_db.seed_control(control, docs) == ["other"]
assert control.docs["monitors"]["names"] == ["edited"]
assert control.docs["other"] == {"_id": "other", "x": 1}
def test_seed_control_dry_run_reports_without_writing():
control = FakeCollection("control")
assert bootstrap_db.seed_control(control, {"monitors": {}}, dry_run=True) == [
"monitors"
]
assert control.docs == {}
def test_bootstrap_builds_both_modes(seed_root):
db = FakeDatabase()
summary = bootstrap_db.bootstrap(db, seed_root=seed_root)
created = dict(db.created)
for suffix in ("", "_testing"):
assert created[f"configs{suffix}"] == {
"validator": bootstrap_db.CONFIG_VALIDATOR
}
assert f"alerts{suffix}" in created
assert f"control{suffix}" in created
for monitor in bootstrap_db.MONITORS:
assert f"{monitor}{suffix}" in created
assert len(db[f"alerts{suffix}"].indexes) == 2
assert summary["seeded"]["configs"] == ["bestbuy", "woot"]
assert summary["seeded"]["configs_testing"] == ["woot"]
assert summary["seeded"]["control"] == ["monitors"]
assert summary["seeded"]["control_testing"] == ["monitors"]
assert db["control"].docs["monitors"]["names"] == ["woot"]
assert len(summary["created"]) == 2 * (3 + len(bootstrap_db.MONITORS))
def test_bootstrap_rerun_is_noop(seed_root):
db = FakeDatabase()
bootstrap_db.bootstrap(db, seed_root=seed_root)
db.created.clear()
summary = bootstrap_db.bootstrap(db, seed_root=seed_root)
assert db.created == []
assert db.commands == []
assert summary["created"] == []
assert summary["validators"] == []
assert summary["seeded"] == {
"configs": [],
"configs_testing": [],
"control": [],
"control_testing": [],
}
def test_shipped_control_seed_matches_registry_shape():
"""monitors doc: names list and per-name {min_percentage, ping_threshold}"""
docs = bootstrap_db.load_control_seed(bootstrap_db.CONTROL_SEED)
monitors = docs["monitors"]
assert set(monitors["settings"]) == set(monitors["names"])
for name, settings in monitors["settings"].items():
assert set(settings) == {"min_percentage", "ping_threshold"}, name
assert isinstance(settings["min_percentage"], int), name
assert 40 <= settings["min_percentage"] <= 100, name
assert settings["ping_threshold"] is None or isinstance(
settings["ping_threshold"], int
), name
def test_shipped_seed_files_load_and_match_validator_shape():
"""every checked-in seed doc has the shape the validator and setup_config want"""
for mode in ("production", "testing"):
docs = bootstrap_db.load_seed_configs(bootstrap_db.SEED_ROOT / mode)
assert docs, mode
for name, doc in docs.items():
categories, channels = doc["categories"], doc["channels"]
assert isinstance(categories, bootstrap_db.CATEGORY_SHAPES[name]), name
if isinstance(categories, dict):
assert all(isinstance(v, str) for v in categories.values()), name
else:
assert all(isinstance(v, str) for v in categories), name
for value in channels.values():
assert isinstance(value, str) and value.isdigit(), (name, value)
tests/test_build_product_details.py
+49
View File
@@ -78,6 +78,17 @@ def test_promo_fields_read_through():
assert product.metadata.percentage == 20 assert product.metadata.percentage == 20
def test_asin_reads_through():
"""asin round-trips: convert_product_to_mongo writes it, so the consumer
must read it back for the SellerAmp lookup"""
product = build_product_details(_doc(asin="B0CHX3QBCH"), min_percentage=0)
assert product.metadata.asin == "B0CHX3QBCH"
def test_missing_asin_defaults_to_none():
assert build_product_details(_doc(), min_percentage=0).metadata.asin is None
def test_missing_promo_keys_default_to_none(): def test_missing_promo_keys_default_to_none():
"""a doc from a lagging monitor lacks promo keys -> None, not KeyError""" """a doc from a lagging monitor lacks promo keys -> None, not KeyError"""
product = build_product_details(_doc(), min_percentage=0) product = build_product_details(_doc(), min_percentage=0)
@@ -192,3 +203,41 @@ def test_msrpless_doc_with_prices_builds_unthresholded():
product = build_product_details(doc, min_percentage=40) product = build_product_details(doc, min_percentage=40)
assert product is not None assert product is not None
assert product.pricing.has_msrp is False assert product.pricing.has_msrp is False
# ------------------------------------------------------------ non-finite prices
def test_infinite_price_skips_instead_of_raising():
"""Regression (2026-10-09 06:25Z): float("Infinity") from a retailer field
raised OverflowError here and stopped the consumer's API deal loop."""
inf = float("inf")
assert build_product_details(_doc(originalPrice=inf), min_percentage=40) is None
assert build_product_details(_doc(currentPrice=inf), min_percentage=40) is None
assert build_product_details(_doc(currentPrice=-inf), min_percentage=40) is None
def test_nan_price_skips_instead_of_raising():
nan = float("nan")
assert build_product_details(_doc(currentPrice=nan), min_percentage=40) is None
assert build_product_details(_doc(originalPrice=nan), min_percentage=40) is None
def test_msrpless_infinite_price_skips_rather_than_reaching_the_payload():
"""Without the skip the MSRP-less path would carry inf into the Product."""
doc = _doc(hasMsrp=False, currentPrice=float("inf"))
assert build_product_details(doc, min_percentage=40) is None
def test_bypass_does_not_rescue_a_non_finite_price():
doc = _doc(originalPrice=float("inf"))
assert build_product_details(doc, min_percentage=40, bypass_thresholds=True) is None
def test_unrepresentable_discount_skips_instead_of_raising():
"""Finite prices can still overflow the ratio (a subnormal original)."""
doc = _doc(currentPrice=-1e308, originalPrice=5e-324)
assert build_product_details(doc, min_percentage=40) is None
tests/test_get_response_json.py
+80
View File
@@ -0,0 +1,80 @@
"""unit tests: sessions.get_response_json parses a body or logs and returns None
run from the repo root:
uv run pytest tests/test_response_json.py
offline — the aioweb Response is faked at the leaf; nothing is sent.
"""
import logging
import pytest
from monitor_utils.sessions import get_response_json
class FakeResponse:
def __init__(self, body):
self.body = body
def json(self):
return self.body
def warnings(caplog):
return [r for r in caplog.records if r.levelno == logging.WARNING]
def test_dict_body_is_returned_without_logging(caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
assert get_response_json(FakeResponse({"a": 1})) == {"a": 1}
assert warnings(caplog) == []
def test_none_body_logs_and_returns_none(caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
assert get_response_json(FakeResponse(None), category="c1", page=2) is None
(record,) = warnings(caplog)
assert record.message == "unexpected response"
assert record.body_type == "NoneType"
assert record.body == "None"
assert (record.category, record.page) == ("c1", 2)
@pytest.mark.parametrize("body", [[1, 2], "text", 42, b"bytes"])
def test_wrong_shape_logs_type_and_truncated_body(body, caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
assert get_response_json(FakeResponse(body)) is None
(record,) = warnings(caplog)
assert record.body_type == type(body).__name__
assert record.body == str(body)[:200]
def test_body_preview_is_capped_at_200_chars(caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
get_response_json(FakeResponse("x" * 1000))
(record,) = warnings(caplog)
assert len(record.body) == 200
def test_expected_accepts_other_types(caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
assert get_response_json(FakeResponse([1]), expected=list) == [1]
assert get_response_json(FakeResponse([1]), expected=(dict, list)) == [1]
assert get_response_json(FakeResponse({}), expected=list) is None
assert len(warnings(caplog)) == 1
def test_warning_goes_to_the_passed_logger(caplog):
with caplog.at_level(logging.WARNING):
get_response_json(
FakeResponse(None), logger=logging.getLogger("woot.woot_monitor")
)
(record,) = warnings(caplog)
assert record.name == "woot.woot_monitor"
def test_empty_dict_counts_as_a_dict(caplog):
with caplog.at_level(logging.WARNING, logger="monitor_utils.sessions"):
assert get_response_json(FakeResponse({})) == {}
assert warnings(caplog) == []
tests/test_new_listing_stamp.py
+28 -1
View File
@@ -99,6 +99,32 @@ def test_existing_doc_with_new_price_stamps_false():
assert alert_doc(collection)["newListing"] is False assert alert_doc(collection)["newListing"] is False
def test_drop_alert_carries_price_before_drop():
"""the doc's priceBeforeDrop is cleared by the next no-drop cycle, so the
alert keeps its own copy from emit time — a drop claimed late still knows
what it dropped from"""
existing = {"_id": "p1", "currentPrice": 100.0, "pastPrices": []}
collection = FakeCollection([existing])
run(collection, [_product(current=95.0)])
assert alert_doc(collection)["priceBeforeDrop"] == 100.0
def test_non_drop_alert_carries_none():
"""a never-seen higher price alerts without a drop; discoveries have no
prior price at all"""
existing = {"_id": "p1", "currentPrice": 100.0, "pastPrices": []}
collection = FakeCollection([existing])
run(collection, [_product(current=120.0)])
assert alert_doc(collection)["priceBeforeDrop"] is None
discovery = FakeCollection()
run(discovery, [_product()])
assert alert_doc(discovery)["priceBeforeDrop"] is None
def test_existing_doc_with_known_price_emits_no_alert(): def test_existing_doc_with_known_price_emits_no_alert():
existing = { existing = {
"_id": "p1", "_id": "p1",
@@ -112,8 +138,9 @@ def test_existing_doc_with_known_price_emits_no_alert():
assert collection.alerts is None assert collection.alerts is None
def test_build_alert_defaults_missing_flag_to_false(): def test_build_alert_defaults_missing_fields():
"""Direct callers passing an unstamped document must not KeyError.""" """Direct callers passing an unstamped document must not KeyError."""
alert = build_alert("bestbuy", {"_id": "p1"}, NOW) alert = build_alert("bestbuy", {"_id": "p1"}, NOW)
assert alert["newListing"] is False assert alert["newListing"] is False
assert alert["priceBeforeDrop"] is None