Compare commits
4
Commits
v0.1.0
...
2e837da7df
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2e837da7df | ||
|
|
50388de22c | ||
|
|
a46c4b13fc | ||
|
|
2372b1ecd1 |
@@ -11,24 +11,26 @@ wire-compatible and share the driver, so this covers both.
|
||||
`requirements.txt`:
|
||||
|
||||
```
|
||||
mysql @ git+ssh://git@git.rethinkstudios.io/rethink-public/mysql.git@v0.1.0
|
||||
mysql @ git+ssh://git@git.rethinkstudios.io/rethink-public/mysql.git@v0.1.4
|
||||
```
|
||||
|
||||
Direct:
|
||||
|
||||
```bash
|
||||
pip install "mysql @ git+ssh://git@git.rethinkstudios.io/rethink-public/mysql.git@v0.1.0"
|
||||
pip install "mysql @ git+ssh://git@git.rethinkstudios.io/rethink-public/mysql.git@v0.1.4"
|
||||
```
|
||||
|
||||
Pulls `aiomysql` (which pulls `PyMySQL`).
|
||||
|
||||
Drop the `@v0.1.0` suffix from the line above to install the latest unpinned.
|
||||
Drop the `@v0.1.3` suffix from the line above to install the latest unpinned.
|
||||
|
||||
## The two-layer API
|
||||
|
||||
**Layer 1 — friendly verbs** for simple single-table CRUD. These hide the dialect and are
|
||||
**identical to the `psql` lib** — same method names, signatures, return shapes — so you
|
||||
swap psql↔mysql with zero call-site changes.
|
||||
**signature-identical to the `psql` lib** — same method names, arguments, and return
|
||||
*types* — so you swap psql↔mysql with zero call-site changes. One value-level difference:
|
||||
`insert()` returns the new row id (`lastrowid`) here vs. the inserted rowcount in psql
|
||||
(both `int`); everything else returns the same shape.
|
||||
|
||||
**Layer 2 — raw escape hatch** for the complex ~20% (joins, aggregates, subqueries). You
|
||||
write the SQL with `%s` placeholders + a params sequence; the wrapper still gives pooling,
|
||||
@@ -97,7 +99,12 @@ For anything even Layer 2 doesn't model (`executemany`, server-side cursors), us
|
||||
- **Values are always parameterized.** Layer 1 builds `%s` internally; Layer 2 takes your
|
||||
`%s` placeholders + a params sequence. Never f-string/`%`-format a value into SQL (the
|
||||
`%s` is the DBAPI placeholder, not Python string formatting). Only identifiers
|
||||
(table/column names) are interpolated, and they're backtick-quoted.
|
||||
(table/column names) are interpolated, and they're backtick-quoted — Layer 1 also
|
||||
escapes a literal `%` inside an identifier for you.
|
||||
- **A literal `%` in Layer-2 SQL text must be written `%%`.** PyMySQL builds the query via
|
||||
`query % args` whenever you pass params, so `"... LIKE '%foo%'"` with params raises
|
||||
`TypeError`/`ValueError` — write `"... LIKE '%%foo%%'"` instead. Calls with no params are
|
||||
unaffected (no substitution happens), and Layer 1 handles this for you internally.
|
||||
|
||||
## Dialect vs psql
|
||||
|
||||
@@ -111,6 +118,11 @@ Layer 1 is portable — you never see these. A Layer-2 raw-SQL author does:
|
||||
`fetchone` is the unified Layer-2 name in both libs (psql maps it onto asyncpg's
|
||||
`fetchrow` internally; here it's the native DBAPI verb).
|
||||
|
||||
`upsert()`'s returned rowcount follows MySQL's `ON DUPLICATE KEY UPDATE` convention (1 for
|
||||
an insert, 2 for an update, 0 for a no-op) — a per-row count that differs from psql's; treat
|
||||
it as "affected", not "rows matched". It emits the `col = VALUES(col)` update form, which
|
||||
MySQL 8.0.20+ deprecates (warning 1287, still functional; MariaDB is unaffected).
|
||||
|
||||
## Error contract — fail loud
|
||||
|
||||
Unlike the `mongo` lib (which log-and-swallows), **this lib re-raises.** Every method
|
||||
@@ -128,7 +140,10 @@ result (no row, empty table) — never a swallowed failure.
|
||||
|
||||
`get`/`delete`/`exists`/`get_one` take **equality** conditions only (`col = val AND ...`);
|
||||
anything richer goes through Layer 2. The pool runs with `autocommit=True`; `transaction()`
|
||||
opens an explicit transaction for atomic multi-statement blocks.
|
||||
opens an explicit transaction for atomic multi-statement blocks. `cursorclass` is
|
||||
overridable via a constructor kwarg (default `DictCursor`); `fetchval`/`exists` work under
|
||||
either a dict or tuple cursorclass, everything else stays dict-shaped only under the
|
||||
default `DictCursor`.
|
||||
|
||||
## Versioning
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "mysql"
|
||||
version = "0.1.0"
|
||||
version = "0.1.4"
|
||||
description = "async mysql/mariadb wrapper over aiomysql: two-layer API (friendly verbs + raw escape hatch), fail-loud, config-free"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
from .mysql import MysqlDB
|
||||
|
||||
__all__ = ["MysqlDB"]
|
||||
__version__ = "0.1.4"
|
||||
|
||||
__all__ = ["MysqlDB", "__version__"]
|
||||
|
||||
+91
-29
@@ -33,18 +33,17 @@ call-site changes; only the dialect below differs, hidden inside the wrapper):
|
||||
|
||||
dialect (mysql-specific; visible only in raw layer-2 SQL you write):
|
||||
- placeholders are `%s` (here) vs `$1` (psql). the `%s` is the DBAPI placeholder the
|
||||
driver consumes, NOT python `%` string formatting.
|
||||
- upsert emits `ON DUPLICATE KEY UPDATE` (here) vs `ON CONFLICT` (psql); layer-1
|
||||
upsert() hides this.
|
||||
driver consumes, NOT python `%` string formatting — but PyMySQL builds it via
|
||||
`query % args` whenever params are passed, so any OTHER literal `%` in the query
|
||||
text (e.g. `LIKE '%foo%'`) must be written `%%` or it raises. no params -> no
|
||||
substitution -> a lone `%` is fine. layer 1 escapes `%` in identifiers for you.
|
||||
- upsert emits `ON DUPLICATE KEY UPDATE` (here) vs `ON CONFLICT` (psql), via the
|
||||
deprecated `col = VALUES(col)` form (MySQL 8.0.20+ warns; still works, MariaDB is
|
||||
unaffected). layer-1 upsert() hides both.
|
||||
|
||||
rows:
|
||||
layer 1 and fetch/fetchone return plain dicts ({column: value}) via aiomysql's
|
||||
DictCursor — identical shape to the psql lib.
|
||||
|
||||
placeholders / injection safety:
|
||||
values are ALWAYS parameterized — layer 1 builds `%s` internally; layer 2 takes your
|
||||
`%s` placeholders + a params sequence. never f-string/%-format a value into SQL. only
|
||||
identifiers (table/column names) are interpolated, and they are backtick-quoted.
|
||||
layer 1 and fetch/fetchone return plain dicts via aiomysql's DictCursor (identical
|
||||
shape to psql). fetchval/exists tolerate a caller-overridden tuple cursorclass too.
|
||||
|
||||
errors (FAIL LOUD — unlike the mongo lib's swallow-and-default):
|
||||
every method catches the driver error (pymysql.err.MySQLError, OSError on connection
|
||||
@@ -53,6 +52,7 @@ errors (FAIL LOUD — unlike the mongo lib's swallow-and-default):
|
||||
wrapped, use the raw `.pool` property (the aiomysql.Pool).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional, Sequence
|
||||
|
||||
@@ -64,13 +64,27 @@ log = logging.getLogger(__name__)
|
||||
_DRIVER_ERRORS = (MySQLError, OSError)
|
||||
|
||||
|
||||
def _first_value(row) -> Any:
|
||||
"""return the first column's value from a dict or sequence row
|
||||
|
||||
fetchval/exists read a single scalar column and work under either cursorclass: the
|
||||
default DictCursor (dict row) or a caller override to a tuple/sequence cursor via
|
||||
pool_kwargs — the only two row shapes aiomysql cursors produce.
|
||||
"""
|
||||
return next(iter(row.values())) if isinstance(row, dict) else row[0]
|
||||
|
||||
|
||||
def _quote_ident(identifier: str) -> str:
|
||||
"""backtick-quote a sql identifier (table/column), escaping embedded backticks
|
||||
"""backtick-quote a sql identifier (table/column), escaping backticks and `%`
|
||||
|
||||
identifiers can't be parameterized, so they are interpolated — backtick-quoting +
|
||||
doubling any embedded backtick is the mysql-safe way to do that for caller names.
|
||||
doubling any embedded backtick is the mysql-safe way to do that for caller names. an
|
||||
embedded `%` is also doubled: PyMySQL applies `query % args` whenever a call passes
|
||||
params, and an unescaped `%` inside an interpolated identifier would hit that same
|
||||
substitution and raise, not just literal `%` written by the caller (see the module
|
||||
docstring's `%%`-escaping rule).
|
||||
"""
|
||||
return "`" + identifier.replace("`", "``") + "`"
|
||||
return "`" + identifier.replace("`", "``").replace("%", "%%") + "`"
|
||||
|
||||
|
||||
class MysqlDB:
|
||||
@@ -92,9 +106,22 @@ class MysqlDB:
|
||||
|
||||
host/port/db/user/password/minsize/maxsize are injected by the caller. extra
|
||||
pool_kwargs pass through to aiomysql.create_pool (unix_socket, ssl, charset, ...).
|
||||
rows come back as dicts (DictCursor) and autocommit is on by default; both can be
|
||||
overridden via pool_kwargs.
|
||||
rows come back as dicts (DictCursor), overridable via pool_kwargs to a tuple
|
||||
cursorclass — fetchval/exists work either way, everything else stays dict-shaped
|
||||
only under DictCursor. autocommit is always on: layer-1 verbs and raw execute()
|
||||
rely on it to commit their own writes without an explicit commit, so
|
||||
`autocommit=False` in pool_kwargs is rejected outright — silently accepting it
|
||||
would mean every write sits uncommitted until something else commits or the
|
||||
connection is returned to the pool and rolled back, losing data with no error.
|
||||
use transaction() for an atomic multi-statement block instead (it disables
|
||||
autocommit for just that block).
|
||||
"""
|
||||
if pool_kwargs.get("autocommit") is False:
|
||||
raise ValueError(
|
||||
"mysql: autocommit=False is not supported via pool_kwargs — layer-1 verbs and "
|
||||
"raw execute() need autocommit to persist their writes; use db.transaction() "
|
||||
"for an atomic multi-statement block instead"
|
||||
)
|
||||
self._config = dict(
|
||||
host=host,
|
||||
port=port,
|
||||
@@ -108,21 +135,33 @@ class MysqlDB:
|
||||
self._config.setdefault("cursorclass", aiomysql.cursors.DictCursor)
|
||||
self._config.update(pool_kwargs)
|
||||
self._pool: Optional[aiomysql.Pool] = None
|
||||
self._connect_lock = asyncio.Lock()
|
||||
|
||||
async def connect(self) -> "MysqlDB":
|
||||
"""build the pool and validate it with SELECT 1; fail loud on bad config
|
||||
|
||||
returns self so callers can write `db = await MysqlDB(...).connect()`.
|
||||
returns self so callers can write `db = await MysqlDB(...).connect()`. if called
|
||||
again on an already-connected instance the previous pool is closed first (no
|
||||
orphaned pool); if the SELECT-1 validation fails the freshly-built pool is torn
|
||||
down before re-raising, so a failed connect() never leaks a live pool. concurrent
|
||||
callers serialize on an internal lock so racing connect() calls build exactly one
|
||||
pool instead of each leaking the losers' live pools.
|
||||
"""
|
||||
async with self._connect_lock:
|
||||
if self._pool is not None:
|
||||
await self.close()
|
||||
pool = await aiomysql.create_pool(**self._config)
|
||||
try:
|
||||
self._pool = await aiomysql.create_pool(**self._config)
|
||||
async with self._pool.acquire() as conn:
|
||||
async with pool.acquire() as conn:
|
||||
async with conn.cursor() as cur:
|
||||
await cur.execute("SELECT 1")
|
||||
await cur.fetchone()
|
||||
except _DRIVER_ERRORS:
|
||||
log.exception("mysql.connect() failed")
|
||||
except BaseException:
|
||||
log.exception("mysql.connect() validation failed")
|
||||
pool.close()
|
||||
await pool.wait_closed()
|
||||
raise
|
||||
self._pool = pool
|
||||
return self
|
||||
|
||||
async def close(self) -> None:
|
||||
@@ -135,6 +174,8 @@ class MysqlDB:
|
||||
except _DRIVER_ERRORS:
|
||||
log.exception("mysql.close()")
|
||||
raise
|
||||
finally:
|
||||
self._pool = None
|
||||
|
||||
async def __aenter__(self) -> "MysqlDB":
|
||||
return await self.connect()
|
||||
@@ -218,7 +259,7 @@ class MysqlDB:
|
||||
raise
|
||||
if not row:
|
||||
return None
|
||||
return next(iter(row.values()))
|
||||
return _first_value(row)
|
||||
|
||||
def transaction(self):
|
||||
"""async context manager running a block of statements atomically
|
||||
@@ -333,16 +374,19 @@ class MysqlDB:
|
||||
except _DRIVER_ERRORS:
|
||||
log.exception("mysql.exists(%s)", table)
|
||||
raise
|
||||
return bool(next(iter(row.values()))) if row else False
|
||||
return bool(_first_value(row)) if row else False
|
||||
|
||||
async def upsert(self, table: str, values: Dict[str, Any], conflict: Sequence[str]) -> int:
|
||||
"""INSERT ... ON DUPLICATE KEY UPDATE — insert or update on unique/pk clash
|
||||
|
||||
`conflict` is the list of columns forming the unique/pk constraint. mysql upserts
|
||||
on ANY unique-key clash (it does not name the constraint the way postgres does), so
|
||||
`conflict` is accepted for a call-signature identical to psql but the constraint is
|
||||
whatever unique/pk key the row violates. the wrapper emits ON DUPLICATE KEY UPDATE
|
||||
(psql emits ON CONFLICT for the identical call). returns the affected rowcount.
|
||||
`conflict` is accepted for call-signature parity with psql but the constraint is
|
||||
whatever unique/pk key the row violates. returns the affected rowcount. emits the
|
||||
`col = VALUES(col)` update form, deprecated since MySQL 8.0.20 (warning 1287, still
|
||||
functional) and unaffected on MariaDB; a future MySQL-only build could probe
|
||||
`SELECT VERSION()` and switch to alias syntax, not done here as a documented cost/
|
||||
benefit call, not an oversight.
|
||||
"""
|
||||
cols = list(values.keys())
|
||||
placeholders = ", ".join(["%s"] * len(cols))
|
||||
@@ -373,7 +417,14 @@ class _Transaction:
|
||||
|
||||
async def __aenter__(self):
|
||||
self._conn = await self._pool.acquire()
|
||||
try:
|
||||
await self._conn.begin()
|
||||
except BaseException:
|
||||
# __aexit__ never runs if __aenter__ raises, so begin() failing after acquire
|
||||
# would otherwise leak the pooled connection — release it here instead.
|
||||
await self._pool.release(self._conn)
|
||||
self._conn = None
|
||||
raise
|
||||
return self._conn
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb) -> None:
|
||||
@@ -383,15 +434,26 @@ class _Transaction:
|
||||
else:
|
||||
await self._conn.rollback()
|
||||
finally:
|
||||
self._pool.release(self._conn)
|
||||
# release() returns a task (schedules a _wakeup); await it to avoid leaving
|
||||
# it un-awaited each transaction, matching psql's and aiomysql's own idiom.
|
||||
await self._pool.release(self._conn)
|
||||
|
||||
|
||||
def _where(conditions: Optional[Dict[str, Any]]) -> tuple:
|
||||
"""build a parameterized `WHERE col = %s AND ...` clause + the params list
|
||||
|
||||
returns ("", []) when there are no conditions. equality only.
|
||||
returns ("", []) when there are no conditions. equality only. a None value renders as
|
||||
`col IS NULL` (not `col = %s` bound to NULL, which sql never matches) and does not
|
||||
consume a placeholder.
|
||||
"""
|
||||
if not conditions:
|
||||
return "", []
|
||||
clause = " AND ".join(f"{_quote_ident(c)} = %s" for c in conditions)
|
||||
return f" WHERE {clause}", list(conditions.values())
|
||||
parts = []
|
||||
params = []
|
||||
for col, val in conditions.items():
|
||||
if val is None:
|
||||
parts.append(f"{_quote_ident(col)} IS NULL")
|
||||
else:
|
||||
params.append(val)
|
||||
parts.append(f"{_quote_ident(col)} = %s")
|
||||
return f" WHERE {' AND '.join(parts)}", params
|
||||
|
||||
Reference in New Issue
Block a user