fix(schema): serialize CREATE SCHEMA so concurrent sessions cannot collide

[axellpadilla]

Closes #839

The bug

Four macros create a schema on demand, each with its own copy of the same guard:

IF NOT EXISTS (SELECT * FROM sys.schemas WHERE name = '<schema>')
BEGIN
  EXEC('CREATE SCHEMA [<schema>]')
END

That is check-then-act. With threads > 1 — or two dbt processes pointed at one database — several sessions pass the check together and all but one fail their create with Msg 2714, There is already an object named '<schema>' in the database. The run dies on a schema that, by the time the error surfaces, exists and is perfectly usable, which is why a rerun succeeds and why CI that builds a schema per pull request only sees it on a branch's first run.

sqlserver__get_test_sql is the one reported, but the same guard is in sqlserver__get_unit_test_sql, sqlserver__create_schema and sqlserver__create_schema_with_authorization. All four are fixed.

Reproduced on SQL Server 2022 with 8 sessions released from a barrier: 15 failures across 48 attempts, with the reporter's exact message.

Why not TRY/CATCH

Swallowing 2714 looks like the obvious fix and is actively worse. Every connection runs SET XACT_ABORT ON (#718), under which the failed create dooms the enclosing transaction. Measured, same server:

caught error: 2759 | XACT_STATE: -1   (doomed, uncommittable)
COMMIT FAILED: The COMMIT TRANSACTION request has no corresponding BEGIN TRANSACTION.

The transaction is already rolled back by the time the batch reaches its COMMIT, so a TRY/CATCH would convert a loud, recoverable error into a silently discarded transaction. SQL Server has no CREATE SCHEMA IF NOT EXISTS, so the remaining option is to not race at all.

The fix

One shared create_schema_if_not_exists macro, used by all four sites, that serializes the check and the create behind a database-scoped sp_getapplock. The lock is held for the microseconds the create takes and released in the same batch. Same race, after: 0 failures across 48 attempts, schema present exactly once each round.

Design points, all verified against SQL Server 2022:

• Session-scoped lock, not transaction-scoped. These callers run both inside dbt's transaction (dbt_sqlserver_use_dbt_transactions defaults on in 1.12) and in autocommit, and @LockOwner = 'Transaction' errors when there is no transaction. Confirmed the session-scoped lock commits cleanly inside an explicit transaction and is released afterwards (APPLOCK_MODE → NoLock).
• No stray result set. EXEC @rc = sp_getapplock returns a return code, not a result set. This matters because sqlserver__get_test_sql and sqlserver__get_unit_test_sql emit it ahead of the SELECT whose rows dbt fetches; confirmed the caller's SELECT is still the first result set.
• Degrades to today's behaviour. A lock request that times out (30s) returns negative and falls through to the bare check — no worse than before — and releases nothing it did not take.

Also escapes schema and authorization names where they are interpolated into string literals, since the new macro rewrites every one of those literals anyway; a name containing an apostrophe no longer breaks the statement.

Tests

New tests/functional/adapter/mssql/test_concurrent_schema_creation.py:

• TestConcurrentCreateSchema races 8 real adapter connections through adapter.create_schema, three rounds, barrier-synchronised, and asserts every session succeeds and the schema exists exactly once.
• TestDataTestSchemaGuardIsConcurrencySafe asserts the statement sqlserver__get_test_sql actually emits is the serialized form — racing dbt test deterministically is not practical, so this pins the reported macro to the safe guard.

Both are genuine regression tests: reverted to the current macros they fail, the first with the reporter's exact error (There is already an object named '..._race' in the database), the second on the missing sp_getapplock.

Full suite on SQL Server 2022:

• tests/functional — 377 passed, 48 skipped, 2 xfailed (run with -n 8, which exercises the concurrent path itself)
• tests/unit — 526 passed

Note

This targets master (1.12). The same four macros carry the same race on release/v1.11; happy to backport if you want it in 1.11.2.

🤖 Generated with Claude Code

[Benjamin-Knight]

Seems very edge case and perhaps just CI related under normal conditions. See the comment, I'm concerned we could block.

[Benjamin-Knight]

I assume the lock time out is a fall back in case the lock is not released? That is fine but right now there is no try finally here so if the schema creation fails, perhaps something transient, then the resource is locked for 30 seconds?