Add new check for indexing with atomic=True
Briefly, this check helps prevent a long-running index creation from preventing all reads and writes to a table. This supplements an existing check ================================== There is an existing check, `CREATE_INDEX` which warns about creating indices without the `CONCURRENTLY` argument. It's a helpful check! However, there are some valid reasons to prefer non-concurrent index creation. For example, concurrent index creation may struggle to complete when applied to write-heavy tables (since Postgres will have to make repeated passes on the table to accommodate the new reads since the last round). The danger of `EXCLUSIVE` locks =============================== If you *do* want to create indices nonconcurrently, there is still a **huge footgun** to be aware of -- locks which are held in a transaction are not released until the transaction closes. Dangerous Django defaults ========================= By default, any time you add a new Django field with `db_index=True` or `unique=True`, the automatically generated `Migration` has: - implicit setting of `atomic=True` (transactions are on) - an `ALTER TABLE...ADD COLUMN` call (exclusively locks the table) - a `CREATE INDEX` invocation (*hopefully* quick, due to nullable columns) It's even more dangerous if you combine an `ALTER TABLE` call with `CREATE INDEX` of an existing, non-nullable column. Favor `atomic=False` if you must index ====================================== Index-creating migrations should have few `operations`. Since index creation does not modify row-level data (and only requires locking against writes), there's no reason to use transactions. It's prudent to bypass the default `atomic=True` for index creation.
This commit is contained in:
1 parent
9fdfa06845
commit
0c5519d9dc
11 files changed
+119
No files matched your search
@@ -5,6 +5,36 @@ import re
|
||||
from .base import BaseAnalyser, Check, CheckMode, CheckType
|
||||
|
||||
|
||||
def has_create_index_in_transaction(sql_statements: list[str], **kwargs) -> bool:
|
||||
"""Return if a migration opens a transaction, acquires EXCLUSIVE lock, then indexes.
|
||||
|
||||
Any locks that are obtained after a transaction is opened will not be released
|
||||
until that transaction either commits or rolls back.
|
||||
|
||||
Accordingly, it's extremely important that a migration avoid any long-running
|
||||
queries while exclusive locks are held.
|
||||
|
||||
Building an index is generally a long-running operation. A standard index
|
||||
creation (that is, avoiding use of `CONCURRENTLY`) can be safe to run on
|
||||
tables which are small in size and/or read-heavy. (`CREATE INDEX` only
|
||||
prevents writes to a table; reads are allowed during index creation).
|
||||
|
||||
This check is a stricter version of `CREATE_INDEX` -- if a team wishes
|
||||
to build indices nonconcurrently, it's imperative to be mindful of locks.
|
||||
"""
|
||||
if not (sql_statements and sql_statements[0].startswith("BEGIN")):
|
||||
return False
|
||||
|
||||
for i, sql in enumerate(sql_statements):
|
||||
# If any statements acquire an exclusive lock, complain about index creation later.
|
||||
# Nearly every single `ALTER TABLE` command requires an exclusive lock:
|
||||
# https://www.postgresql.org/docs/current/sql-altertable.html
|
||||
# (Most common example is `ALTER TABLE... ADD COLUMN`, then later `CREATE INDEX`)
|
||||
if sql.startswith("ALTER TABLE"):
|
||||
return has_create_index(sql_statements[i + 1 :])
|
||||
return False
|
||||
|
||||
|
||||
def has_create_index(sql_statements: list[str], **kwargs) -> bool:
|
||||
regex_result = None
|
||||
for sql in sql_statements:
|
||||
@@ -32,6 +62,13 @@ class PostgresqlAnalyser(BaseAnalyser):
|
||||
mode=CheckMode.TRANSACTION,
|
||||
type=CheckType.WARNING,
|
||||
),
|
||||
Check(
|
||||
code="CREATE_INDEX_EXCLUSIVE",
|
||||
fn=has_create_index_in_transaction,
|
||||
message="CREATE INDEX prolongs transaction, delaying lock release",
|
||||
mode=CheckMode.TRANSACTION,
|
||||
type=CheckType.ERROR,
|
||||
),
|
||||
Check(
|
||||
code="DROP_INDEX",
|
||||
fn=lambda sql, **kw: re.search("DROP INDEX", sql)
|
||||
|
||||
Reference in new issue
Block a user