diff --git a/.env.example b/.env.example index e6ead732..3b53b9ae 100644 --- a/.env.example +++ b/.env.example @@ -40,6 +40,16 @@ TRANSFER_NGWMN_VIEWS=True TRANSFER_WATERLEVELS_PRESSURE_DAILY=True TRANSFER_WEATHER_DATA=True TRANSFER_MINOR_TRACE_CHEMISTRY=True +# NM_Wells (geothermal) migration: run `python -m transfers.transfer_geothermal` +# (separate from the deprecated transfers/transfer.py NM_Aquifer driver). +TRANSFER_GEOTHERMAL_REFERENCE=True # load ref_* lookups into the lexicon +TRANSFER_NMW_MIRROR=True # load the NMW_* 1:1 staging mirror +# Optional: path to a NM_Wells SQL Server data-dump .sql file (INSERT statements). +# When set, the mirror parses it to a CSV per table (sqlparse) and bulk-loads via +# Postgres COPY; otherwise it falls back to CSV exports + row inserts. +# NMW_SQL_DUMP=/path/to/NMWells_data.sql +# Optional: dir for the per-table CSVs written from the dump (default: temp dir). +# NMW_CSV_DIR=/path/to/nmw_csv # asset storage GCS_BUCKET_NAME= diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 00000000..3731fa02 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,87 @@ +# SPEC — BDMS-826 NM_Wells migrations core + +Distilled from branch `feature/BDMS-826-NMW-migrations-core` (PR #738), POC of #686. +Status: Phase-1 impl landed in branch, NOT reviewed/tested. Spec captures built state + remaining work. + +## §G — goal + +1:1 staging mirror of legacy NM_Wells (SQL Server) into Postgres `NMW_*` tables. +Plus geothermal OGC API layers over mirror data. Phase-1 only: faithful copy, no transform to Ocotillo model. + +## §C — constraints + +- C1. Mirror = column-for-column copy. Original col names, types, PKs preserved. `SSMA_TimeStamp` dropped. +- C2. Phase split: P1 = land `NMW_*` mirror + OGC views (this branch). P2 = transform mirror → Ocotillo model (Location→Thing→Event→Sample→Observation). P2 NOT built; per-col targets + lexicon maps flagged inline. +- C3. Follow `db/nma_legacy.py` (NM_Aquifer) mirror convention. +- C4. Standalone orchestrator. Do NOT extend deprecated `transfers/transfer.py` (NM_Aquifer driver). +- C5. Tables load parent→child (FK order). Mirror load truncate+COPY (dump) or INSERT ON CONFLICT DO NOTHING (CSV). No upsert. +- C6. Coords/geom stay WGS84 4326 per project std. +- C7. Migrations idempotent, reversible (alembic up/down). + +## §I — interfaces + +- I.cli — `python -m transfers.transfer_geothermal` — runs ref→lexicon then NMW mirror load then refresh matviews. +- I.env — `NMW_SQL_DUMP` (dump path; else CSV), `NMW_CSV_DIR`, `TRANSFER_LIMIT`, `TRANSFER_GEOTHERMAL_REFERENCE` (def 1), `TRANSFER_NMW_MIRROR` (def 1). Export: `NMW_HOST/USER/PASSWORD/PORT/DATABASE`. +- I.export — `transfers/export_nmw_csvs.py` — pymssql dump SQL Server → `transfers/data/nma_csv_cache/.csv`. +- I.db — 18 `NMW_*` mirror tables (`db/nmw_legacy.py`), PK verified vs dump DDL. +- I.migrations — `c0d1e2f3a4b5` (tables+FK), `d1e2f3a4b5c6` (per-well views), `e2f3a4b5c6d7` (measurement views). Linear chain (rebased onto staging head): x2y3z4a5b6c7 → c0 → d1 → e2. +- I.ogc — 6 new collections in `core/pygeoapi-config.yml`: geothermal_wells_bht, geothermal_wells_temperature_profile (MATVIEW), bht_measurements, temp_depth_measurements, heat_flow, dst. +- I.views — DB: ogc_geothermal_wells_bht, ogc_geothermal_wells_temperature_profile (MAT), ogc_geothermal_wells_summary_heat_flow, ogc_geothermal_wells_interval_heat_flow, ogc_bht_measurements, ogc_temp_depth_measurements, ogc_heat_flow, ogc_dst. +- I.lexicon — `reference_lexicon_transfer.py` maps 46 `ref_*` → `nmw_ref_*` LexiconCategory + terms. +- I.jira — BDMS-826 "Geothermal Migration Planning" (Story, In Progress) under epic BDMS-843 "Geothermal Migration". 6 linked tasks tracked in §T (T18-T23). + +## §V — invariants + +- V1. `NMW_*` cols match legacy NM_Wells DDL exactly (name+type+PK). No renames except dropped `SSMA_TimeStamp`. +- V2. Mirror load respects parent→child order in `NMW_MIRROR_SPECS`; child never loads before parent. +- V3. `alembic upgrade head` then `downgrade` cleanly creates+drops all 18 tables + 8 views, no orphans. +- V4. Re-running mirror load is non-destructive at row level (truncate+COPY full reload OR ON CONFLICT skip) — no dup rows, no partial-state corruption. +- V5. After mirror load, `ogc_geothermal_wells_temperature_profile` matview refreshed; stale matview never served. +- V6. Each of 6 OGC collections resolves to existing backing view; pygeoapi config view name == migration view name. +- V7. `core/pygeoapi.py` unchanged from staging (reviewer note: reverted to original). +- V8. `NMW_WellRecords.SourceID` joined as TEXT (free-text citation), not numeric FK. +- V9. P2 lexicon mapping complete: every coded NMW col either in `LEXICON_REF_BY_COLUMN` (28, has ref_*) or `LEXICON_CANDIDATES_NO_REF` (11, needs new enum). +- V10. ORM `NMW_*` models declare index only (`index=True`, no ORM `ForeignKey`); FK enforcement lives in migration `c0d1e2f3a4b5` (`op.create_foreign_key`). Keep both in sync. +- V11. SQL-dump parser unwraps `CAST(expr AS )` for parameterised types too (`Decimal(18,2)`, `nvarchar(10)`); never store the literal `CAST(...)` string in a mirror column. +- V12. Every "Migrate First" table in the NM_Wells inventory (planning workbook) has a `NMW_*` mirror in `NMW_MIRROR_SPECS`. Verified 18/18 (2026-06-23). Subsurface Library is a separate source DB — NOT covered by this invariant (see T24). +- V13. Mirror dump-reload truncates with `CASCADE` (mirror tables carry FK constraints; bare TRUNCATE of a referenced parent is rejected). Safe because parents load before children (V2). +- V14. Per-well OGC views dedup `NMW_WellLocations` by `WellDataID` (`DISTINCT ON ... ORDER BY "WellDataID","OBJECTID"`); one feature per well, counts not multiplied by duplicate location rows. Applies to all 4 per-well geothermal views. + +## §T — tasks + +id|status|task|cites +T1|x|18 NMW_* mirror tables in db/nmw_legacy.py|V1,I.db +T2|x|migration c0d1e2f3a4b5 tables+FK|V3,I.migrations +T3|x|migration d1e2f3a4b5c6 per-well OGC views|I.views +T4|x|migration e2f3a4b5c6d7 measurement OGC views|I.views +T5|x|nmw_sql_dump.py SSMS dump parser|I.cli +T6|x|nmw_mirror_transfer.py loader (dump+CSV)|V2,V4,I.cli +T7|x|reference_lexicon_transfer.py ref_*→lexicon|I.lexicon +T8|x|export_nmw_csvs.py pymssql export|I.export +T9|x|transfer_geothermal.py orchestrator|I.cli +T10|x|6 OGC collections in pygeoapi-config.yml|V6,I.ogc +T11|x|FK enforced via migration op.create_foreign_key; model index-only (resolved)|V2,V10 +T12|x|add NMW_* mirror/loader/migration/OGC tests (tests/test_nmw_mirror.py, 19 tests); found+fixed CAST-unwrap bug B1|V1,V2,V3,V5,V6,V10,V11 +T13|.|verify alembic down path drops all views+tables (V3) on real db|V3 +T14|.|run end-to-end load vs real dump, capture row counts per table|V2,V4 +T15|.|finish PR #738 body (truncated at "- I ") + reviewer notes|- +T16|.|P2 (later): transform NMW_* → Ocotillo model; build new enums for 11 LEXICON_CANDIDATES_NO_REF|C2,V9 +T24|.|Subsurface Library "Migrate First" tables NOT mirrored (separate source DB, out of NM_Wells scope): dst_scan, log_scanned, Well_Header, well_operators. Workbook lacks field map/DDL — needs own ticket|V12 +T17|x|landed docs/nm_wells-migration.md (commit ccf566d9; force-add, docs/ gitignored). Referenced 4x: nmw_legacy.py:75,81,759; nmw_mirror_transfer.py:19|- + +### BDMS-826 linked tasks (six; status mirrors Jira) + +id|status|task|cites +T18|x|BDMS-827 read/review Geothermal Data Discovery Report (Jira Done)|I.jira +T19|x|BDMS-846 technical review complete (Jira Done)|I.jira +T20|x|BDMS-845 Geothermal Report finalized (Jira Done)|I.jira +T21|x|BDMS-847 update data-migration tracking mechanism (Jira Done)|I.jira +T22|x|BDMS-907 stakeholder engagement follow-up (Jira Done; blocks BDMS-826)|I.jira +T23|~|BDMS-848 Geothermal Migration Technical Implementation Plan (Jira In Progress) — umbrella for code tasks T1-T16|I.jira,T1,T16 + +## §B — bugs + +id|date|cause|fix +B1|2026-06-23|_CAST_RE in transfers/nmw_sql_dump.py matched AS-type without parens only; parenthesised types (nvarchar(10), Decimal(18,2)) left value as literal "CAST(...)" string|V11; widened regex to allow one paren level +B2|2026-06-23|dump-load reload used bare TRUNCATE; FK from NMW_WellLocations/NMW_WellRecords to NMW_WellHeaders makes Postgres reject it, aborting load before COPY (PR#740 P1)|V13; TRUNCATE ... CASCADE +B3|2026-06-23|per-well geothermal OGC views (bht, summary_heat_flow, interval_heat_flow) joined NMW_WellLocations directly; multiple OBJECTID rows per WellDataID multiplied counts / emitted >1 feature per well (PR#740 P2)|V14; DISTINCT ON loc CTE in all 4 views diff --git a/alembic/versions/c0d1e2f3a4b5_nmw_mirror_tables.py b/alembic/versions/c0d1e2f3a4b5_nmw_mirror_tables.py new file mode 100644 index 00000000..f59a760c --- /dev/null +++ b/alembic/versions/c0d1e2f3a4b5_nmw_mirror_tables.py @@ -0,0 +1,723 @@ +"""NMW staging mirror tables and FK constraints + +Revision ID: c0d1e2f3a4b5 +Revises: t6u7v8w9x0y1 +Create Date: 2026-06-22 + +1:1 staging mirror of the legacy NM_Wells SQL Server tables needed for the +geothermal OGC collections (see db/nmw_legacy.py and docs/nm_wells-migration.md). +Faithful, column-for-column copies; the transform into the Ocotillo data model +is a later phase. + +Core well tables: + tbl_well_locations -> NMW_WellLocations + tbl_well_headers -> NMW_WellHeaders + tbl_well_records -> NMW_WellRecords + tbl_well_z_datum -> NMW_WellZDatum + tbl_well_samples -> NMW_WellSamples + +Geothermal tables: + tbl_gt_bht_headers -> NMW_GtBhtHeaders + tbl_gt_bht_data -> NMW_GtBhtData + tbl_ws_intervals -> NMW_WsIntervals + tbl_gt_conductivity -> NMW_GtConductivity + tbl_gt_heat_flow -> NMW_GtHeatFlow + tbl_gt_sum_heat_flow -> NMW_GtSumHeatFlow + tbl_gt_temp_depths -> NMW_GtTempDepths + +Drill Stem Test tables: + tbl_ws_dst_headers -> NMW_WsDstHeaders + tbl_ws_dst_intervals -> NMW_WsDstIntervals + tbl_ws_dst_flow_history -> NMW_WsDstFlowHistory + tbl_ws_dst_fluid_properties -> NMW_WsDstFluidProperties + tbl_ws_dst_pressure -> NMW_WsDstPressure + +Publication/source registry: + tbl_sources -> NMW_Sources + (Keyed by free-text SourceID; joined into ogc_heat_flow for attribution.) +""" + +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +revision: str = "c0d1e2f3a4b5" +down_revision: Union[str, Sequence[str], None] = "x2y3z4a5b6c7" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ------------------------------------------------------------------ + # Core well tables + # ------------------------------------------------------------------ + op.create_table( + "NMW_WellLocations", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("WellDataID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Well_ID", sa.String(), nullable=True), + sa.Column("Import_ID", sa.Integer(), nullable=True), + sa.Column("Township", sa.Float(), nullable=True), + sa.Column("NorS_TDir", sa.String(), nullable=True), + sa.Column("Range", sa.Float(), nullable=True), + sa.Column("EorW_RDir", sa.String(), nullable=True), + sa.Column("Sectn", sa.SmallInteger(), nullable=True), + sa.Column("SectnPart", sa.String(), nullable=True), + sa.Column("UnitLetter", sa.String(), nullable=True), + sa.Column("UTM_zone", sa.String(), nullable=True), + sa.Column("State", sa.String(), nullable=True), + sa.Column("County", sa.String(), nullable=True), + sa.Column("Basin", sa.String(), nullable=True), + sa.Column("Footage_NS", sa.Float(), nullable=True), + sa.Column("NorS_FDir", sa.String(), nullable=True), + sa.Column("Footage_EW", sa.Float(), nullable=True), + sa.Column("EorW_FDir", sa.String(), nullable=True), + sa.Column("Lat_min", sa.SmallInteger(), nullable=True), + sa.Column("Lat_sec", sa.Float(), nullable=True), + sa.Column("Long_deg", sa.SmallInteger(), nullable=True), + sa.Column("Long_min", sa.SmallInteger(), nullable=True), + sa.Column("Long_sec", sa.Float(), nullable=True), + sa.Column("Lat_dd27", sa.Float(), nullable=True), + sa.Column("Long_dd27", sa.Float(), nullable=True), + sa.Column("Lat_dd83", sa.Float(), nullable=True), + sa.Column("Long_dd83", sa.Float(), nullable=True), + sa.Column("SourceID", sa.String(), nullable=True), + sa.Column("SourceDatum", sa.String(), nullable=True), + sa.Column("SourceUnits", sa.String(), nullable=True), + sa.Column("LocAccType", sa.String(), nullable=True), + sa.Column("LocAccMeas", sa.String(), nullable=True), + sa.Column("LocAccVal", sa.Float(), nullable=True), + sa.Column("Duplicated", sa.SmallInteger(), nullable=True), + sa.Column("Exclude", sa.SmallInteger(), nullable=True), + sa.Column("Comments", sa.String(), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("API", sa.String(), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_WellLocations_WellDataID", "NMW_WellLocations", ["WellDataID"] + ) + + op.create_table( + "NMW_WellHeaders", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("WellDataID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("WellSpotID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("API", sa.String(), nullable=True), + sa.Column("WellClass", sa.String(), nullable=True), + sa.Column("WellType", sa.String(), nullable=True), + sa.Column("WellOrient", sa.String(), nullable=True), + sa.Column("CurWellNam", sa.String(), nullable=True), + sa.Column("CurWellNum", sa.String(), nullable=True), + sa.Column("CurStatus", sa.String(), nullable=True), + sa.Column("PrdPoolCnt", sa.SmallInteger(), nullable=True), + sa.Column("CurOperatr", sa.String(), nullable=True), + sa.Column("CurOwner", sa.String(), nullable=True), + sa.Column("TotalDepth", sa.Float(), nullable=True), + sa.Column("Well_TVD", sa.Float(), nullable=True), + sa.Column("Fm_TD", sa.String(), nullable=True), + sa.Column("Age_TD", sa.String(), nullable=True), + sa.Column("SpudDate", sa.DateTime(), nullable=True), + sa.Column("ComplDate", sa.DateTime(), nullable=True), + sa.Column("PlugDate", sa.DateTime(), nullable=True), + sa.Column("PlugBack", sa.Float(), nullable=True), + sa.Column("BridgePlug", sa.String(), nullable=True), + sa.Column("ScoutTickt", sa.SmallInteger(), nullable=True), + sa.Column("DwnHoleSur", sa.SmallInteger(), nullable=True), + sa.Column("GeolLog", sa.SmallInteger(), nullable=True), + sa.Column("Geophyslog", sa.SmallInteger(), nullable=True), + sa.Column("GthrmExist", sa.SmallInteger(), nullable=True), + sa.Column("PetroData", sa.SmallInteger(), nullable=True), + sa.Column("CoreExists", sa.SmallInteger(), nullable=True), + sa.Column("Cuttings", sa.SmallInteger(), nullable=True), + sa.Column("SampleData", sa.SmallInteger(), nullable=True), + sa.Column("Comments", sa.String(), nullable=True), + sa.Column("Import_ID", sa.String(), nullable=True), + sa.Column("Import_DB", sa.String(), nullable=True), + sa.PrimaryKeyConstraint("WellDataID"), + ) + + op.create_table( + "NMW_WellRecords", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("RecrdSetID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("WellDataID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("RecrdClass", sa.String(), nullable=True), + sa.Column("SourceID", sa.String(), nullable=True), + sa.Column("ActionDate", sa.DateTime(), nullable=True), + sa.Column("WellName", sa.String(), nullable=True), + sa.Column("WellNumber", sa.String(), nullable=True), + sa.Column("API_suffix", sa.String(), nullable=True), + sa.Column("EnteredBy", sa.String(), nullable=True), + sa.Column("EntryDate", sa.DateTime(), nullable=True), + sa.Column("Comments", sa.String(), nullable=True), + sa.PrimaryKeyConstraint("RecrdSetID"), + ) + op.create_index("ix_NMW_WellRecords_WellDataID", "NMW_WellRecords", ["WellDataID"]) + + op.create_table( + "NMW_WellZDatum", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("RecrdsetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Elev_GL", sa.Float(), nullable=True), + sa.Column("Elev_DF", sa.Float(), nullable=True), + sa.Column("Elev_KB", sa.Float(), nullable=True), + sa.Column("Elev_unspc", sa.Float(), nullable=True), + sa.Column("DatumElev", sa.Float(), nullable=True), + sa.Column("DepthDatum", sa.String(), nullable=True), + sa.Column("DepthUnits", sa.String(), nullable=True), + sa.Column("Z_datum", sa.String(), nullable=True), + sa.Column("Z_units", sa.String(), nullable=True), + sa.Column("ElevSource", sa.String(), nullable=True), + sa.Column("ElvAccType", sa.String(), nullable=True), + sa.Column("ElvAccMeas", sa.String(), nullable=True), + sa.Column("ElvAccVal", sa.Float(), nullable=True), + sa.Column("Comments", sa.String(), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index("ix_NMW_WellZDatum_RecrdsetID", "NMW_WellZDatum", ["RecrdsetID"]) + + op.create_table( + "NMW_WellSamples", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("RecrdsetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("SmpSetName", sa.String(), nullable=True), + sa.Column("SamplClass", sa.String(), nullable=True), + sa.Column("SampleType", sa.String(), nullable=True), + sa.Column("SampleFm", sa.String(), nullable=True), + sa.Column("SampleLoc", sa.String(), nullable=True), + sa.Column("SampleDate", sa.DateTime(), nullable=True), + sa.Column("From_Depth", sa.Float(), nullable=True), + sa.Column("To_Depth", sa.Float(), nullable=True), + sa.Column("SmpDpUnt", sa.String(), nullable=True), + sa.Column("From_TVD", sa.Float(), nullable=True), + sa.Column("To_TVD", sa.Float(), nullable=True), + sa.Column("From_Elev", sa.Float(), nullable=True), + sa.Column("To_Elev", sa.Float(), nullable=True), + sa.Column("Porosity", sa.SmallInteger(), nullable=True), + sa.Column("Permeablty", sa.SmallInteger(), nullable=True), + sa.Column("Density", sa.SmallInteger(), nullable=True), + sa.Column("DST_Tests", sa.SmallInteger(), nullable=True), + sa.Column("ThinSect", sa.SmallInteger(), nullable=True), + sa.Column("Geochron", sa.SmallInteger(), nullable=True), + sa.Column("Geochem", sa.SmallInteger(), nullable=True), + sa.Column("Geothermal", sa.SmallInteger(), nullable=True), + sa.Column("WholeRock", sa.SmallInteger(), nullable=True), + sa.Column("Paleontlgy", sa.SmallInteger(), nullable=True), + sa.Column("EnteredBy", sa.String(), nullable=True), + sa.Column("EntryDate", sa.DateTime(), nullable=True), + sa.Column("Notes", sa.String(), nullable=True), + sa.PrimaryKeyConstraint("SamplSetID"), + ) + op.create_index("ix_NMW_WellSamples_RecrdsetID", "NMW_WellSamples", ["RecrdsetID"]) + + # ------------------------------------------------------------------ + # Geothermal tables + # ------------------------------------------------------------------ + op.create_table( + "NMW_GtBhtHeaders", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("BHTGUID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("BoreDia", sa.Float(), nullable=True), + sa.Column("BoreUnits", sa.String(length=16), nullable=True), + sa.Column("DrillFluid", sa.String(length=16), nullable=True), + sa.Column("TempUnit", sa.String(length=1), nullable=True), + sa.Column("FldSalinity", sa.Float(), nullable=True), + sa.Column("FldRstvity", sa.Float(), nullable=True), + sa.Column("Fluid_pH", sa.Float(), nullable=True), + sa.Column("FldDensity", sa.Float(), nullable=True), + sa.Column("FldLevel", sa.Float(), nullable=True), + sa.Column("FldViscsty", sa.Float(), nullable=True), + sa.Column("FluidLoss", sa.String(length=50), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("BHTGUID"), + ) + op.create_index( + "ix_NMW_GtBhtHeaders_SamplSetID", "NMW_GtBhtHeaders", ["SamplSetID"] + ) + + op.create_table( + "NMW_GtBhtData", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("BHTGUID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Depth", sa.Float(), nullable=True), + sa.Column("BHT", sa.Float(), nullable=True), + sa.Column("TempUnit", sa.String(length=5), nullable=True), + sa.Column("HrsSnceCir", sa.Float(), nullable=True), + sa.Column("DateMeasrd", sa.DateTime(), nullable=True), + sa.Column("Comments", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index("ix_NMW_GtBhtData_BHTGUID", "NMW_GtBhtData", ["BHTGUID"]) + + op.create_table( + "NMW_WsIntervals", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("IntrvlGUID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("SampleID", sa.String(length=128), nullable=True), + sa.Column("From_Depth", sa.Float(), nullable=True), + sa.Column("To_Depth", sa.Float(), nullable=True), + sa.Column("From_TVD", sa.Float(), nullable=True), + sa.Column("To_TVD", sa.Float(), nullable=True), + sa.Column("From_Elev", sa.Float(), nullable=True), + sa.Column("To_Elev", sa.Float(), nullable=True), + sa.Column("Intv_Notes", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("IntrvlGUID"), + ) + op.create_index("ix_NMW_WsIntervals_SamplSetID", "NMW_WsIntervals", ["SamplSetID"]) + + op.create_table( + "NMW_GtConductivity", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("IntrvlGUID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Cnductvity", sa.Float(), nullable=True), + sa.Column("CnductUnit", sa.String(length=3), nullable=True), + sa.Column("Comments", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_GtConductivity_IntrvlGUID", "NMW_GtConductivity", ["IntrvlGUID"] + ) + + op.create_table( + "NMW_GtHeatFlow", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("IntrvlGUID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Gradient", sa.Float(), nullable=True), + sa.Column("Ka", sa.Float(), nullable=True), + sa.Column("Ka_unit", sa.String(length=3), nullable=True), + sa.Column("Pm", sa.Float(), nullable=True), + sa.Column("Kpr", sa.Float(), nullable=True), + sa.Column("Kpr_unit", sa.String(length=3), nullable=True), + sa.Column("Q", sa.Float(), nullable=True), + sa.Column("Q_unit", sa.String(length=3), nullable=True), + sa.Column("Comments", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index("ix_NMW_GtHeatFlow_IntrvlGUID", "NMW_GtHeatFlow", ["IntrvlGUID"]) + + op.create_table( + "NMW_GtSumHeatFlow", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("RecrdSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("LithClass", sa.String(length=50), nullable=True), + sa.Column("UnitBasis", sa.String(length=16), nullable=True), + sa.Column("UnitName", sa.String(length=128), nullable=True), + sa.Column("GeoID", sa.String(length=16), nullable=True), + sa.Column("FromDepth", sa.Float(), nullable=True), + sa.Column("ToDepth", sa.Float(), nullable=True), + sa.Column("DepthUnit", sa.String(length=8), nullable=True), + sa.Column("From_Elev", sa.Float(), nullable=True), + sa.Column("To_Elev", sa.Float(), nullable=True), + sa.Column("ThermlGrad", sa.Float(), nullable=True), + sa.Column("TGError", sa.Float(), nullable=True), + sa.Column("GradUnit", sa.String(length=3), nullable=True), + sa.Column("TGradRange", sa.String(length=15), nullable=True), + sa.Column("SampleType", sa.String(length=50), nullable=True), + sa.Column("NumSamples", sa.SmallInteger(), nullable=True), + sa.Column("ThermlCond", sa.Float(), nullable=True), + sa.Column("TCondError", sa.Float(), nullable=True), + sa.Column("TCondUnit", sa.String(length=3), nullable=True), + sa.Column("TCondRange", sa.String(length=15), nullable=True), + sa.Column("HeatFlow", sa.Float(), nullable=True), + sa.Column("HtFlowErr", sa.Float(), nullable=True), + sa.Column("HtFlowUnit", sa.String(length=3), nullable=True), + sa.Column("HtFlowEst", sa.Float(), nullable=True), + sa.Column("Quality", sa.String(length=50), nullable=True), + sa.Column("Comments", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_GtSumHeatFlow_RecrdSetID", "NMW_GtSumHeatFlow", ["RecrdSetID"] + ) + op.create_index( + "ix_NMW_GtSumHeatFlow_SamplSetID", "NMW_GtSumHeatFlow", ["SamplSetID"] + ) + + op.create_table( + "NMW_GtTempDepths", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Depth", sa.Float(), nullable=True), + sa.Column("Temp", sa.Float(), nullable=True), + sa.Column("TempUnit", sa.String(length=1), nullable=True), + sa.Column("IntrvlGrad", sa.Float(), nullable=True), + sa.Column("Comments", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_GtTempDepths_SamplSetID", "NMW_GtTempDepths", ["SamplSetID"] + ) + + # ------------------------------------------------------------------ + # Drill Stem Test tables + # ------------------------------------------------------------------ + op.create_table( + "NMW_WsDstHeaders", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("DSTGUID", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("SamplSetID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("TestType", sa.String(length=50), nullable=True), + sa.Column("DSTOprator", sa.String(length=50), nullable=True), + sa.Column("PressUnits", sa.String(length=8), nullable=True), + sa.Column("TempUnit", sa.String(length=1), nullable=True), + sa.Column("PipeDiaUnt", sa.String(length=8), nullable=True), + sa.Column("PipeLenUnt", sa.String(length=8), nullable=True), + sa.Column("ChokeSizUn", sa.String(length=8), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("DSTGUID"), + ) + op.create_index( + "ix_NMW_WsDstHeaders_SamplSetID", "NMW_WsDstHeaders", ["SamplSetID"] + ) + + op.create_table( + "NMW_WsDstIntervals", + sa.Column("OBJECTID", sa.Integer(), nullable=True), + sa.Column("DSTInterval", postgresql.UUID(as_uuid=True), nullable=False), + sa.Column("DSTGUID", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("DSTName", sa.String(length=128), nullable=True), + sa.Column("TargetFm", sa.String(length=16), nullable=True), + sa.Column("DSTDate", sa.DateTime(), nullable=True), + sa.Column("DSTNumber", sa.SmallInteger(), nullable=True), + sa.Column("Status", sa.String(length=255), nullable=True), + sa.Column("StatusDate", sa.DateTime(), nullable=True), + sa.Column("PackrFrom", sa.Float(), nullable=True), + sa.Column("PackerTo", sa.Float(), nullable=True), + sa.Column("SrfChokeSz", sa.Float(), nullable=True), + sa.Column("BotChokeSz", sa.Float(), nullable=True), + sa.Column("PipeDia", sa.Float(), nullable=True), + sa.Column("PipeLength", sa.Float(), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("DSTInterval"), + ) + op.create_index("ix_NMW_WsDstIntervals_DSTGUID", "NMW_WsDstIntervals", ["DSTGUID"]) + + op.create_table( + "NMW_WsDstFlowHistory", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("DSTInterval", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("Operation", sa.String(length=255), nullable=True), + sa.Column("StartTime", sa.DateTime(), nullable=True), + sa.Column("EndTime", sa.DateTime(), nullable=True), + sa.Column("Duration", sa.Float(), nullable=True), + sa.Column("Pressure", sa.Float(), nullable=True), + sa.Column("Temp", sa.Float(), nullable=True), + sa.Column("RecovColmn", sa.Float(), nullable=True), + sa.Column("RecovType", sa.String(length=255), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_WsDstFlowHistory_DSTInterval", "NMW_WsDstFlowHistory", ["DSTInterval"] + ) + + op.create_table( + "NMW_WsDstFluidProperties", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("DSTInterval", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("SourceLoc", sa.String(length=255), nullable=True), + sa.Column("Resistivty", sa.Float(), nullable=True), + sa.Column("Temp", sa.Float(), nullable=True), + sa.Column("Chlorides", sa.Float(), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_WsDstFluidProperties_DSTInterval", + "NMW_WsDstFluidProperties", + ["DSTInterval"], + ) + + op.create_table( + "NMW_WsDstPressure", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("DSTInterval", postgresql.UUID(as_uuid=True), nullable=True), + sa.Column("PrsGageDpt", sa.Float(), nullable=True), + sa.Column("BlankedOff", sa.SmallInteger(), nullable=True), + sa.Column("InShtInMin", sa.Float(), nullable=True), + sa.Column("FlwPrsInMin", sa.Float(), nullable=True), + sa.Column("PrsInShtIn", sa.Float(), nullable=True), + sa.Column("PrsInitClsdIn", sa.Float(), nullable=True), + sa.Column("FnShtInMin", sa.Float(), nullable=True), + sa.Column("FlwPrsFinMin", sa.Float(), nullable=True), + sa.Column("PrsFnShtIn", sa.Float(), nullable=True), + sa.Column("ShtInPrMth", sa.String(length=255), nullable=True), + sa.Column("HydrostPrsIn", sa.Float(), nullable=True), + sa.Column("HydStPrsFl", sa.Float(), nullable=True), + sa.Column("HydstPrMth", sa.String(length=255), nullable=True), + sa.Column("EquilPress", sa.Float(), nullable=True), + sa.Column("EqlPrsMth", sa.String(length=255), nullable=True), + sa.Column("FlowPrsMin", sa.Float(), nullable=True), + sa.Column("FlowPrsMax", sa.Float(), nullable=True), + sa.Column("FlowPrsMth", sa.String(length=255), nullable=True), + sa.Column("DSTFluid", sa.String(length=128), nullable=True), + sa.Column("FmTemp", sa.Float(), nullable=True), + sa.Column("TempCorrtn", sa.Float(), nullable=True), + sa.Column("TempFlowng", sa.Float(), nullable=True), + sa.Column("TempUnit", sa.String(length=5), nullable=True), + sa.Column("Notes", sa.String(length=255), nullable=True), + sa.Column("GlobalID", postgresql.UUID(as_uuid=True), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index( + "ix_NMW_WsDstPressure_DSTInterval", "NMW_WsDstPressure", ["DSTInterval"] + ) + + # ------------------------------------------------------------------ + # Publication/source registry + # ------------------------------------------------------------------ + op.create_table( + "NMW_Sources", + sa.Column("OBJECTID", sa.Integer(), nullable=False), + sa.Column("SourceID", sa.String(), nullable=True), + sa.Column("FirstAuth", sa.String(), nullable=True), + sa.Column("PubYear", sa.String(), nullable=True), + sa.Column("Title", sa.String(), nullable=True), + sa.Column("Journal", sa.String(), nullable=True), + sa.Column("Volume", sa.String(), nullable=True), + sa.Column("PageNo", sa.String(), nullable=True), + sa.Column("ReportNo", sa.String(), nullable=True), + sa.Column("Publisher", sa.String(), nullable=True), + sa.Column("City", sa.String(), nullable=True), + sa.Column("URL", sa.String(), nullable=True), + sa.Column("Comments", sa.String(), nullable=True), + sa.PrimaryKeyConstraint("OBJECTID"), + ) + op.create_index("ix_NMW_Sources_SourceID", "NMW_Sources", ["SourceID"]) + + # ------------------------------------------------------------------ + # FK constraints + # ------------------------------------------------------------------ + op.create_foreign_key( + "fk_nmw_welllocations_welldataid", + "NMW_WellLocations", + "NMW_WellHeaders", + ["WellDataID"], + ["WellDataID"], + ) + op.create_foreign_key( + "fk_nmw_wellrecords_welldataid", + "NMW_WellRecords", + "NMW_WellHeaders", + ["WellDataID"], + ["WellDataID"], + ) + op.create_foreign_key( + "fk_nmw_wellzdatum_recrdsetid", + "NMW_WellZDatum", + "NMW_WellRecords", + ["RecrdsetID"], + ["RecrdSetID"], + ) + op.create_foreign_key( + "fk_nmw_wellsamples_recrdsetid", + "NMW_WellSamples", + "NMW_WellRecords", + ["RecrdsetID"], + ["RecrdSetID"], + ) + op.create_foreign_key( + "fk_nmw_gtbhtheaders_samplsetid", + "NMW_GtBhtHeaders", + "NMW_WellSamples", + ["SamplSetID"], + ["SamplSetID"], + ) + op.create_foreign_key( + "fk_nmw_gtbhtdata_bhtguid", + "NMW_GtBhtData", + "NMW_GtBhtHeaders", + ["BHTGUID"], + ["BHTGUID"], + ) + op.create_foreign_key( + "fk_nmw_wsintervals_samplsetid", + "NMW_WsIntervals", + "NMW_WellSamples", + ["SamplSetID"], + ["SamplSetID"], + ) + op.create_foreign_key( + "fk_nmw_gtconductivity_intrvlguid", + "NMW_GtConductivity", + "NMW_WsIntervals", + ["IntrvlGUID"], + ["IntrvlGUID"], + ) + op.create_foreign_key( + "fk_nmw_gtheatflow_intrvlguid", + "NMW_GtHeatFlow", + "NMW_WsIntervals", + ["IntrvlGUID"], + ["IntrvlGUID"], + ) + op.create_foreign_key( + "fk_nmw_gtsumheatflow_recrdsetid", + "NMW_GtSumHeatFlow", + "NMW_WellRecords", + ["RecrdSetID"], + ["RecrdSetID"], + ) + op.create_foreign_key( + "fk_nmw_gtsumheatflow_samplsetid", + "NMW_GtSumHeatFlow", + "NMW_WellSamples", + ["SamplSetID"], + ["SamplSetID"], + ) + op.create_foreign_key( + "fk_nmw_gttempdepths_samplsetid", + "NMW_GtTempDepths", + "NMW_WellSamples", + ["SamplSetID"], + ["SamplSetID"], + ) + op.create_foreign_key( + "fk_nmw_wsdstheaders_samplsetid", + "NMW_WsDstHeaders", + "NMW_WellSamples", + ["SamplSetID"], + ["SamplSetID"], + ) + op.create_foreign_key( + "fk_nmw_wsdstintervals_dstguid", + "NMW_WsDstIntervals", + "NMW_WsDstHeaders", + ["DSTGUID"], + ["DSTGUID"], + ) + op.create_foreign_key( + "fk_nmw_wsdstflowhistory_dstinterval", + "NMW_WsDstFlowHistory", + "NMW_WsDstIntervals", + ["DSTInterval"], + ["DSTInterval"], + ) + op.create_foreign_key( + "fk_nmw_wsdstfluidproperties_dstinterval", + "NMW_WsDstFluidProperties", + "NMW_WsDstIntervals", + ["DSTInterval"], + ["DSTInterval"], + ) + op.create_foreign_key( + "fk_nmw_wsdstpressure_dstinterval", + "NMW_WsDstPressure", + "NMW_WsDstIntervals", + ["DSTInterval"], + ["DSTInterval"], + ) + + +def downgrade() -> None: + # Drop FKs before tables (reverse order of creation) + op.drop_constraint( + "fk_nmw_wsdstpressure_dstinterval", "NMW_WsDstPressure", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wsdstfluidproperties_dstinterval", + "NMW_WsDstFluidProperties", + type_="foreignkey", + ) + op.drop_constraint( + "fk_nmw_wsdstflowhistory_dstinterval", + "NMW_WsDstFlowHistory", + type_="foreignkey", + ) + op.drop_constraint( + "fk_nmw_wsdstintervals_dstguid", "NMW_WsDstIntervals", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wsdstheaders_samplsetid", "NMW_WsDstHeaders", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_gttempdepths_samplsetid", "NMW_GtTempDepths", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_gtsumheatflow_samplsetid", "NMW_GtSumHeatFlow", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_gtsumheatflow_recrdsetid", "NMW_GtSumHeatFlow", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_gtheatflow_intrvlguid", "NMW_GtHeatFlow", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_gtconductivity_intrvlguid", "NMW_GtConductivity", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wsintervals_samplsetid", "NMW_WsIntervals", type_="foreignkey" + ) + op.drop_constraint("fk_nmw_gtbhtdata_bhtguid", "NMW_GtBhtData", type_="foreignkey") + op.drop_constraint( + "fk_nmw_gtbhtheaders_samplsetid", "NMW_GtBhtHeaders", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wellsamples_recrdsetid", "NMW_WellSamples", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wellzdatum_recrdsetid", "NMW_WellZDatum", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_wellrecords_welldataid", "NMW_WellRecords", type_="foreignkey" + ) + op.drop_constraint( + "fk_nmw_welllocations_welldataid", "NMW_WellLocations", type_="foreignkey" + ) + + # Drop tables in child-first order + op.drop_index("ix_NMW_Sources_SourceID", table_name="NMW_Sources") + op.drop_table("NMW_Sources") + op.drop_index("ix_NMW_WsDstPressure_DSTInterval", table_name="NMW_WsDstPressure") + op.drop_table("NMW_WsDstPressure") + op.drop_index( + "ix_NMW_WsDstFluidProperties_DSTInterval", table_name="NMW_WsDstFluidProperties" + ) + op.drop_table("NMW_WsDstFluidProperties") + op.drop_index( + "ix_NMW_WsDstFlowHistory_DSTInterval", table_name="NMW_WsDstFlowHistory" + ) + op.drop_table("NMW_WsDstFlowHistory") + op.drop_index("ix_NMW_WsDstIntervals_DSTGUID", table_name="NMW_WsDstIntervals") + op.drop_table("NMW_WsDstIntervals") + op.drop_index("ix_NMW_WsDstHeaders_SamplSetID", table_name="NMW_WsDstHeaders") + op.drop_table("NMW_WsDstHeaders") + op.drop_index("ix_NMW_GtConductivity_IntrvlGUID", table_name="NMW_GtConductivity") + op.drop_table("NMW_GtConductivity") + op.drop_index("ix_NMW_GtHeatFlow_IntrvlGUID", table_name="NMW_GtHeatFlow") + op.drop_table("NMW_GtHeatFlow") + op.drop_index("ix_NMW_WsIntervals_SamplSetID", table_name="NMW_WsIntervals") + op.drop_table("NMW_WsIntervals") + op.drop_index("ix_NMW_GtBhtData_BHTGUID", table_name="NMW_GtBhtData") + op.drop_table("NMW_GtBhtData") + op.drop_index("ix_NMW_GtBhtHeaders_SamplSetID", table_name="NMW_GtBhtHeaders") + op.drop_table("NMW_GtBhtHeaders") + op.drop_index("ix_NMW_GtSumHeatFlow_SamplSetID", table_name="NMW_GtSumHeatFlow") + op.drop_index("ix_NMW_GtSumHeatFlow_RecrdSetID", table_name="NMW_GtSumHeatFlow") + op.drop_table("NMW_GtSumHeatFlow") + op.drop_index("ix_NMW_GtTempDepths_SamplSetID", table_name="NMW_GtTempDepths") + op.drop_table("NMW_GtTempDepths") + op.drop_index("ix_NMW_WellSamples_RecrdsetID", table_name="NMW_WellSamples") + op.drop_table("NMW_WellSamples") + op.drop_index("ix_NMW_WellZDatum_RecrdsetID", table_name="NMW_WellZDatum") + op.drop_table("NMW_WellZDatum") + op.drop_index("ix_NMW_WellRecords_WellDataID", table_name="NMW_WellRecords") + op.drop_table("NMW_WellRecords") + op.drop_index("ix_NMW_WellLocations_WellDataID", table_name="NMW_WellLocations") + op.drop_table("NMW_WellLocations") + op.drop_table("NMW_WellHeaders") diff --git a/alembic/versions/d1e2f3a4b5c6_nmw_per_well_geothermal_ogc_views.py b/alembic/versions/d1e2f3a4b5c6_nmw_per_well_geothermal_ogc_views.py new file mode 100644 index 00000000..696b8118 --- /dev/null +++ b/alembic/versions/d1e2f3a4b5c6_nmw_per_well_geothermal_ogc_views.py @@ -0,0 +1,262 @@ +"""NMW per-well geothermal OGC views + +Revision ID: d1e2f3a4b5c6 +Revises: c0d1e2f3a4b5 +Create Date: 2026-06-22 + +Four pygeoapi point-layer views that aggregate geothermal data to one +feature per well. All geometry is built from NMW_WellLocations Lat/Long_dd83 +(WGS84). All views use integer id (row_number() OVER ()) as the pygeoapi +id_field; pygeoapi's PostgreSQL provider requires an integer PK column. + + ogc_geothermal_wells_bht (VIEW) + One feature per well with BHT aggregate stats. Links via: + NMW_GtBhtData → NMW_GtBhtHeaders.BHTGUID → NMW_WellSamples.SamplSetID + → NMW_WellRecords.RecrdSetID → NMW_WellLocations/Headers.WellDataID + + ogc_geothermal_wells_temperature_profile (MATERIALIZED VIEW) + One feature per well with temperature-depth JSON series. Materialized + because the source NMW_GtTempDepths is large (~370k rows) and the + json_agg is too heavy to recompute per request. NMW_WellLocations is + deduped via DISTINCT ON before joining because a well can have multiple + location rows (OBJECTID is the PK, not WellDataID). REFRESH after a + data reload. + + ogc_geothermal_wells_summary_heat_flow (VIEW) + One feature per well with summary heat-flow stats and a measurements + JSON series. Links via NMW_GtSumHeatFlow.RecrdSetID. + + ogc_geothermal_wells_interval_heat_flow (VIEW) + One feature per well with per-interval heat-flow stats. Links via + NMW_GtHeatFlow.IntrvlGUID → NMW_WsIntervals → NMW_WellSamples. +""" + +from alembic import op +from sqlalchemy import text + +revision = "d1e2f3a4b5c6" +down_revision = "c0d1e2f3a4b5" +branch_labels = None +depends_on = None + +_BHT_VIEW = "ogc_geothermal_wells_bht" +_PROFILE_VIEW = "ogc_geothermal_wells_temperature_profile" +_SUM_HF_VIEW = "ogc_geothermal_wells_summary_heat_flow" +_INT_HF_VIEW = "ogc_geothermal_wells_interval_heat_flow" + + +def upgrade() -> None: + # ogc_geothermal_wells_bht + op.execute(text(f'DROP VIEW IF EXISTS "{_BHT_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_BHT_VIEW}" AS + WITH loc AS ( + SELECT DISTINCT ON ("WellDataID") + "WellDataID", "Lat_dd83", "Long_dd83" + FROM "NMW_WellLocations" + WHERE "Lat_dd83" IS NOT NULL + AND "Long_dd83" IS NOT NULL + ORDER BY "WellDataID", "OBJECTID" + ) + SELECT + row_number() OVER () AS id, + r."WellDataID"::text AS well_data_id, + hdr."CurWellNam" AS well_name, + hdr."API" AS api, + hdr."TotalDepth" AS total_depth, + count(d.*) AS bht_count, + max(d."BHT") AS max_bht, + min(d."BHT") AS min_bht, + max(d."Depth") AS max_bht_depth, + max(d."TempUnit") AS temp_unit, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtBhtData" AS d + JOIN "NMW_GtBhtHeaders" AS h ON h."BHTGUID" = d."BHTGUID" + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = h."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + GROUP BY + r."WellDataID", + loc."Lat_dd83", + loc."Long_dd83", + hdr."CurWellNam", + hdr."API", + hdr."TotalDepth" + """)) + + # ogc_geothermal_wells_temperature_profile (materialized) + op.execute(text(f'DROP MATERIALIZED VIEW IF EXISTS "{_PROFILE_VIEW}"')) + op.execute(text(f""" + CREATE MATERIALIZED VIEW "{_PROFILE_VIEW}" AS + WITH loc AS ( + SELECT DISTINCT ON ("WellDataID") + "WellDataID", "Lat_dd83", "Long_dd83" + FROM "NMW_WellLocations" + WHERE "Lat_dd83" IS NOT NULL + AND "Long_dd83" IS NOT NULL + ORDER BY "WellDataID", "OBJECTID" + ) + SELECT + row_number() OVER () AS id, + r."WellDataID"::text AS well_data_id, + hdr."CurWellNam" AS well_name, + hdr."API" AS api, + count(td.*) AS reading_count, + min(td."Depth") AS min_depth, + max(td."Depth") AS max_depth, + min(td."Temp") AS min_temp, + max(td."Temp") AS max_temp, + max(td."TempUnit") AS temp_unit, + json_agg( + json_build_object('depth', td."Depth", 'temp', td."Temp") + ORDER BY td."Depth" + ) AS series, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtTempDepths" AS td + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = td."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + WHERE td."Depth" IS NOT NULL + AND td."Temp" IS NOT NULL + GROUP BY + r."WellDataID", + loc."Lat_dd83", + loc."Long_dd83", + hdr."CurWellNam", + hdr."API" + """)) + op.execute( + text(f'CREATE UNIQUE INDEX ux_{_PROFILE_VIEW}_id ON "{_PROFILE_VIEW}" (id)') + ) + op.execute( + text( + f'CREATE INDEX ix_{_PROFILE_VIEW}_geom ON "{_PROFILE_VIEW}" USING GIST (geom)' + ) + ) + + # ogc_geothermal_wells_summary_heat_flow + op.execute(text(f'DROP VIEW IF EXISTS "{_SUM_HF_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_SUM_HF_VIEW}" AS + WITH loc AS ( + SELECT DISTINCT ON ("WellDataID") + "WellDataID", "Lat_dd83", "Long_dd83" + FROM "NMW_WellLocations" + WHERE "Lat_dd83" IS NOT NULL + AND "Long_dd83" IS NOT NULL + ORDER BY "WellDataID", "OBJECTID" + ) + SELECT + row_number() OVER () AS id, + r."WellDataID"::text AS well_data_id, + hdr."CurWellNam" AS well_name, + hdr."API" AS api, + count(shf.*) AS heat_flow_count, + max(shf."HeatFlow") AS max_heat_flow, + avg(shf."HeatFlow") AS avg_heat_flow, + max(shf."HtFlowUnit") AS heat_flow_unit, + max(shf."ThermlGrad") AS max_thermal_gradient, + max(shf."GradUnit") AS gradient_unit, + max(shf."ThermlCond") AS max_thermal_conductivity, + max(shf."TCondUnit") AS conductivity_unit, + max(shf."Quality") AS quality, + json_agg( + json_build_object( + 'from_depth', shf."FromDepth", + 'to_depth', shf."ToDepth", + 'depth_unit', shf."DepthUnit", + 'heat_flow', shf."HeatFlow", + 'heat_flow_error', shf."HtFlowErr", + 'heat_flow_unit', shf."HtFlowUnit", + 'thermal_gradient', shf."ThermlGrad", + 'gradient_unit', shf."GradUnit", + 'thermal_conductivity', shf."ThermlCond", + 'conductivity_unit', shf."TCondUnit", + 'quality', shf."Quality" + ) + ORDER BY shf."FromDepth" + ) AS measurements, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtSumHeatFlow" AS shf + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = shf."RecrdSetID" + JOIN loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + GROUP BY + r."WellDataID", + loc."Lat_dd83", + loc."Long_dd83", + hdr."CurWellNam", + hdr."API" + """)) + + # ogc_geothermal_wells_interval_heat_flow + op.execute(text(f'DROP VIEW IF EXISTS "{_INT_HF_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_INT_HF_VIEW}" AS + WITH loc AS ( + SELECT DISTINCT ON ("WellDataID") + "WellDataID", "Lat_dd83", "Long_dd83" + FROM "NMW_WellLocations" + WHERE "Lat_dd83" IS NOT NULL + AND "Long_dd83" IS NOT NULL + ORDER BY "WellDataID", "OBJECTID" + ) + SELECT + row_number() OVER () AS id, + r."WellDataID"::text AS well_data_id, + hdr."CurWellNam" AS well_name, + hdr."API" AS api, + count(hf.*) AS interval_count, + max(hf."Q") AS max_heat_flow, + avg(hf."Q") AS avg_heat_flow, + max(hf."Q_unit") AS heat_flow_unit, + max(hf."Gradient") AS max_gradient, + max(hf."Kpr") AS max_thermal_conductivity, + max(hf."Kpr_unit") AS conductivity_unit, + max(hf."Ka") AS max_diffusivity, + max(hf."Ka_unit") AS diffusivity_unit, + json_agg( + json_build_object( + 'from_depth', i."From_Depth", + 'to_depth', i."To_Depth", + 'heat_flow', hf."Q", + 'heat_flow_unit', hf."Q_unit", + 'gradient', hf."Gradient", + 'thermal_conductivity', hf."Kpr", + 'conductivity_unit', hf."Kpr_unit", + 'diffusivity', hf."Ka", + 'diffusivity_unit', hf."Ka_unit" + ) + ORDER BY i."From_Depth" + ) AS measurements, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtHeatFlow" AS hf + JOIN "NMW_WsIntervals" AS i ON i."IntrvlGUID" = hf."IntrvlGUID" + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = i."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + GROUP BY + r."WellDataID", + loc."Lat_dd83", + loc."Long_dd83", + hdr."CurWellNam", + hdr."API" + """)) + + +def downgrade() -> None: + op.execute(text(f'DROP VIEW IF EXISTS "{_INT_HF_VIEW}"')) + op.execute(text(f'DROP VIEW IF EXISTS "{_SUM_HF_VIEW}"')) + op.execute(text(f'DROP MATERIALIZED VIEW IF EXISTS "{_PROFILE_VIEW}"')) + op.execute(text(f'DROP VIEW IF EXISTS "{_BHT_VIEW}"')) diff --git a/alembic/versions/e2f3a4b5c6d7_nmw_measurement_ogc_views.py b/alembic/versions/e2f3a4b5c6d7_nmw_measurement_ogc_views.py new file mode 100644 index 00000000..f54ac694 --- /dev/null +++ b/alembic/versions/e2f3a4b5c6d7_nmw_measurement_ogc_views.py @@ -0,0 +1,279 @@ +"""NMW individual-measurement and analytical OGC views + +Revision ID: e2f3a4b5c6d7 +Revises: d1e2f3a4b5c6 +Create Date: 2026-06-22 + +Four pygeoapi point-layer views exposing individual measurement rows (not +aggregated per well) and analytical summaries from the NMW staging mirror. +All translated from legacy MSSQL queries against NM_Aquifer. Integer OBJECTID +columns are used as the pygeoapi id_field directly. + + ogc_bht_measurements (VIEW) + One row per BHT measurement with well header and location. + Translated from the legacy MSSQL BHT query. + + ogc_temp_depth_measurements (VIEW) + One row per downhole temperature reading with well header, location, + and elevation. Translated from TempDepth2_SortedWellName query. + Locations with Exclude=1 are filtered out. + + ogc_heat_flow (VIEW) + One row per NMW_GtSumHeatFlow record with well header, location, + elevation, and publication attribution from NMW_Sources. Unit + conversions (ft→m, HFU→mW/m², TCU→W/m·K) applied inline via + CASE WHEN. Translated from the legacy MSSQL HeatFlow query. + + ogc_dst (VIEW) + One row per DST interval with pressure, flow history, and well + header. The original Access query referenced DST_flwHstryConcat + (a broken saved query); replaced with a string_agg() CTE over + NMW_WsDstFlowHistory. DISTINCT used in place of the original + GROUP BY with no aggregate functions. Translated from the legacy + MSSQL DST query. +""" + +from alembic import op +from sqlalchemy import text + +revision = "e2f3a4b5c6d7" +down_revision = "d1e2f3a4b5c6" +branch_labels = None +depends_on = None + +_BHT_MEAS_VIEW = "ogc_bht_measurements" +_TEMP_DEPTH_VIEW = "ogc_temp_depth_measurements" +_HEAT_FLOW_VIEW = "ogc_heat_flow" +_DST_VIEW = "ogc_dst" + + +def upgrade() -> None: + # ogc_bht_measurements + op.execute(text(f'DROP VIEW IF EXISTS "{_BHT_MEAS_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_BHT_MEAS_VIEW}" AS + SELECT + d."OBJECTID" AS id, + hdr."API" AS api, + hdr."CurWellNam" AS well_name, + hdr."CurWellNum" AS well_num, + hdr."CurOperatr" AS operator, + hdr."WellType" AS well_type, + hdr."Well_TVD" AS well_tvd, + hdr."ComplDate" AS completion_date, + hdr."CurStatus" AS current_status, + hdr."TotalDepth" AS total_depth, + hdr."Cuttings" AS cuttings, + hdr."CoreExists" AS core_exists, + loc."County" AS county, + d."Depth" AS bht_depth, + d."BHT" AS bht, + d."HrsSnceCir" AS hours_since_circulation, + d."DateMeasrd" AS date_measured, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtBhtData" AS d + JOIN "NMW_GtBhtHeaders" AS bh ON bh."BHTGUID" = d."BHTGUID" + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = bh."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN "NMW_WellZDatum" AS z ON z."RecrdsetID" = r."RecrdSetID" + JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + JOIN "NMW_WellLocations" AS loc ON loc."WellDataID" = r."WellDataID" + WHERE loc."Lat_dd83" IS NOT NULL + AND loc."Long_dd83" IS NOT NULL + """)) + + # ogc_temp_depth_measurements + op.execute(text(f'DROP VIEW IF EXISTS "{_TEMP_DEPTH_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_TEMP_DEPTH_VIEW}" AS + SELECT + td."OBJECTID" AS id, + hdr."CurWellNam" AS well_name, + hdr."CurWellNum" AS well_num, + hdr."API" AS api, + r."SourceID" AS source_id, + s."SampleFm" AS sample_fm, + loc."County" AS county, + loc."State" AS state, + loc."Lat_dd27" AS lat_dd27, + loc."Long_dd27" AS long_dd27, + loc."Lat_dd83" AS lat_dd83, + loc."Long_dd83" AS long_dd83, + loc."LocAccVal" AS loc_acc_val, + s."EnteredBy" AS entered_by, + s."EntryDate" AS entry_date, + td."Depth" AS depth, + s."SmpDpUnt" AS depth_unit, + td."Temp" AS temp, + td."TempUnit" AS temp_unit, + z."Elev_GL" AS elev_gl, + z."Elev_unspc" AS elev_unspc, + z."Elev_KB" AS elev_kb, + s."SampleDate" AS sample_date, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtTempDepths" AS td + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = td."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN "NMW_WellZDatum" AS z ON z."RecrdsetID" = r."RecrdSetID" + JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + JOIN "NMW_WellLocations" AS loc ON loc."WellDataID" = r."WellDataID" + WHERE loc."Exclude" = 0 + AND loc."Lat_dd83" IS NOT NULL + AND loc."Long_dd83" IS NOT NULL + """)) + + # ogc_heat_flow + op.execute(text(f'DROP VIEW IF EXISTS "{_HEAT_FLOW_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_HEAT_FLOW_VIEW}" AS + SELECT + shf."OBJECTID" AS id, + hdr."CurWellNam" AS well_name, + hdr."CurWellNum" AS well_num, + hdr."API" AS api, + loc."County" AS county, + loc."State" AS state, + loc."Lat_dd27" AS lat_dd27, + loc."Long_dd27" AS long_dd27, + loc."Lat_dd83" AS lat_dd83, + loc."Long_dd83" AS long_dd83, + r."SourceID" AS source_id, + z."Elev_GL" AS elev_gl, + z."Elev_KB" AS elev_kb, + z."Elev_unspc" AS elev_unspc, + CASE WHEN z."DepthUnits" = 'ft' + THEN 0.3048 * z."Elev_unspc" + ELSE z."Elev_unspc" + END AS elevation_m, + z."DepthUnits" AS depth_units, + hdr."TotalDepth" AS total_depth, + CASE WHEN z."DepthUnits" = 'ft' + THEN 0.3048 * hdr."TotalDepth" + ELSE hdr."TotalDepth" + END AS total_depth_m, + shf."FromDepth" AS from_depth, + shf."ToDepth" AS to_depth, + shf."ThermlCond" AS therml_cond, + shf."TCondRange" AS tcond_range, + shf."TCondError" AS tcond_error, + shf."TCondUnit" AS tcond_unit, + CASE WHEN shf."TCondUnit" = 'TCU' + THEN 0.4184 * shf."ThermlCond" + ELSE shf."ThermlCond" + END AS tc_si, + shf."SampleType" AS sample_type, + shf."NumSamples" AS num_samples, + shf."ThermlGrad" AS therml_grad, + shf."TGradRange" AS tgrad_range, + shf."TGError" AS tg_error, + shf."GradUnit" AS grad_unit, + shf."HeatFlow" AS heat_flow, + shf."HtFlowUnit" AS ht_flow_unit, + CASE WHEN shf."HtFlowUnit" = 'HFU' + THEN 41.84 * shf."HeatFlow" + ELSE shf."HeatFlow" + END AS heat_flow_si, + shf."Quality" AS quality, + src."FirstAuth" AS first_auth, + src."PubYear" AS pub_year, + src."Title" AS title, + src."Journal" AS journal, + src."Volume" AS volume, + src."PageNo" AS page_no, + shf."HtFlowEst" AS ht_flow_est, + r."EntryDate" AS entry_date, + CASE WHEN shf."HtFlowUnit" = 'HFU' + THEN 41.84 * shf."HtFlowEst" + ELSE shf."HtFlowEst" + END AS ht_flow_est_si, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_GtSumHeatFlow" AS shf + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = shf."RecrdSetID" + JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + JOIN "NMW_WellLocations" AS loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellZDatum" AS z ON z."RecrdsetID" = r."RecrdSetID" + JOIN "NMW_Sources" AS src ON src."SourceID" = r."SourceID" + WHERE loc."Exclude" = 0 + AND loc."Lat_dd83" IS NOT NULL + AND loc."Long_dd83" IS NOT NULL + """)) + + # ogc_dst + op.execute(text(f'DROP VIEW IF EXISTS "{_DST_VIEW}"')) + op.execute(text(f""" + CREATE VIEW "{_DST_VIEW}" AS + WITH flow_history AS ( + SELECT + "DSTInterval", + string_agg("Operation", '; ' ORDER BY "OBJECTID") AS flow_history + FROM "NMW_WsDstFlowHistory" + GROUP BY "DSTInterval" + ) + SELECT DISTINCT + i."OBJECTID" AS id, + hdr."CurWellNam" AS well_name, + hdr."CurWellNum" AS well_num, + hdr."API" AS api, + i."DSTName" AS dst_name, + dh."DSTOprator" AS dst_operator, + i."DSTNumber" AS dst_number, + i."DSTDate" AS dst_date, + loc."County" AS county, + loc."State" AS state, + loc."Lat_dd83" AS lat_dd83, + loc."Long_dd83" AS long_dd83, + s."From_Depth" AS from_depth, + s."To_Depth" AS to_depth, + i."TargetFm" AS target_fm, + i."PackrFrom" AS packer_from, + i."PackerTo" AS packer_to, + i."SrfChokeSz" AS srf_choke_sz, + i."BotChokeSz" AS bot_choke_sz, + s."SmpDpUnt" AS depth_unit, + z."Elev_GL" AS elev_gl, + z."Elev_unspc" AS elev_unspc, + p."PrsGageDpt" AS prs_gage_dpt, + i."PipeDia" AS pipe_dia, + i."PipeLength" AS pipe_length, + fh.flow_history AS flow_history, + p."PrsInShtIn" AS init_flow, + p."FlwPrsInMin" AS flw_prs_in_min, + p."PrsFnShtIn" AS fin_flow, + p."FlwPrsFinMin" AS flw_prs_fin_min, + p."PrsInitClsdIn" AS prs_init_clsd_in, + p."InShtInMin" AS in_sht_in_min, + p."EquilPress" AS fin_shut_in, + p."FnShtInMin" AS fn_sht_in_min, + p."HydrostPrsIn" AS hydrost_prs_in, + p."HydStPrsFl" AS hyd_st_prs_fl, + dh."PressUnits" AS press_units, + p."BlankedOff" AS blanked_off, + p."FmTemp" AS fm_temp, + ST_SetSRID( + ST_MakePoint(loc."Long_dd83", loc."Lat_dd83"), 4326 + ) AS geom + FROM "NMW_WsDstIntervals" AS i + JOIN "NMW_WsDstHeaders" AS dh ON dh."DSTGUID" = i."DSTGUID" + JOIN "NMW_WellSamples" AS s ON s."SamplSetID" = dh."SamplSetID" + JOIN "NMW_WellRecords" AS r ON r."RecrdSetID" = s."RecrdsetID" + JOIN "NMW_WellHeaders" AS hdr ON hdr."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellLocations" AS loc ON loc."WellDataID" = r."WellDataID" + LEFT JOIN "NMW_WellZDatum" AS z ON z."RecrdsetID" = r."RecrdSetID" + LEFT JOIN "NMW_WsDstPressure" AS p ON p."DSTInterval" = i."DSTInterval" + LEFT JOIN flow_history AS fh ON fh."DSTInterval" = i."DSTInterval" + WHERE loc."Lat_dd83" IS NOT NULL + AND loc."Long_dd83" IS NOT NULL + """)) + + +def downgrade() -> None: + op.execute(text(f'DROP VIEW IF EXISTS "{_DST_VIEW}"')) + op.execute(text(f'DROP VIEW IF EXISTS "{_HEAT_FLOW_VIEW}"')) + op.execute(text(f'DROP VIEW IF EXISTS "{_TEMP_DEPTH_VIEW}"')) + op.execute(text(f'DROP VIEW IF EXISTS "{_BHT_MEAS_VIEW}"')) diff --git a/core/pygeoapi-config.yml b/core/pygeoapi-config.yml index 1bae81d9..45f3bac1 100644 --- a/core/pygeoapi-config.yml +++ b/core/pygeoapi-config.yml @@ -288,3 +288,141 @@ resources: geom_field: project_area {thing_collections_block} + + geothermal_wells_bht: + type: collection + title: Geothermal Wells — Bottom-Hole Temperature + description: Geothermal wells with bottom-hole temperature (BHT) measurements from the NM_Wells database. + keywords: [geothermal, wells, bottom-hole-temperature, bht] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_geothermal_wells_bht + geom_field: geom + + geothermal_wells_temperature_profile: + type: collection + title: Geothermal Wells — Temperature-Depth Profile + description: Geothermal wells with downhole temperature-vs-depth series from the NM_Wells database. + keywords: [geothermal, wells, temperature, depth, profile] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_geothermal_wells_temperature_profile + geom_field: geom + + bht_measurements: + type: collection + title: BHT Measurements + description: Individual bottom-hole temperature measurements with well header and location data from the NM_Wells database. + keywords: [geothermal, bht, bottom-hole-temperature, measurements] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_bht_measurements + geom_field: geom + + temp_depth_measurements: + type: collection + title: Temperature-Depth Measurements + description: Individual downhole temperature readings with well header, location, and elevation data from the NM_Wells database. + keywords: [geothermal, temperature, depth, measurements] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_temp_depth_measurements + geom_field: geom + + heat_flow: + type: collection + title: Heat Flow + description: Summary heat-flow records with thermal conductivity, gradient, and publication attribution from the NM_Wells database. + keywords: [geothermal, heat-flow, thermal-conductivity, gradient] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_heat_flow + geom_field: geom + + dst: + type: collection + title: Drill Stem Tests + description: Drill stem test intervals with pressure, flow history, and well header data from the NM_Wells database. + keywords: [geothermal, dst, drill-stem-test, pressure, formation] + extents: + spatial: + bbox: [-109.05, 31.33, -103.00, 37.00] + crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 + providers: + - type: feature + name: PostgreSQL + data: + host: {postgres_host} + port: {postgres_port} + dbname: {postgres_db} + user: {postgres_user} + password: {postgres_password_env} + search_path: [public] + id_field: id + table: ogc_dst + geom_field: geom diff --git a/db/__init__.py b/db/__init__.py index a376381b..4e2e7fb3 100644 --- a/db/__init__.py +++ b/db/__init__.py @@ -59,6 +59,7 @@ from db.thing_geologic_formation_association import * from db.aquifer_type import * from db.nma_legacy import * +from db.nmw_legacy import * from db.transducer import * from sqlalchemy import ( diff --git a/db/nmw_legacy.py b/db/nmw_legacy.py new file mode 100644 index 00000000..c20d0cb7 --- /dev/null +++ b/db/nmw_legacy.py @@ -0,0 +1,763 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""1:1 staging mirror of the legacy NM_Wells SQL Server database. + +PURPOSE +------- +These models are a FAITHFUL, column-for-column copy of the NM_Wells source +tables. They are a *staging layer*: data lands here unchanged from the SQL +dump, then a later transform phase maps it into the Ocotillo data model +(Location / Thing / FieldEvent / FieldActivity / Sample / Observation, plus +status_history, measuring_point_history, contact, publication, etc.). + +This file mirrors the convention of ``db/nma_legacy.py`` (the NM_Aquifer +mirror): ``NMW_`` table prefix, original source column names preserved via the +first positional arg to ``mapped_column``, snake_case Python attributes. + +SOURCE +------ +NM_Wells is delivered as a SQL dump. Physical source table names are +``tbl_well_*`` (snake_case). To feed the existing CSV->Pandas->ORM transfer +pipeline, export each source table to CSV (same flow as ``nma_csv_cache``). + +SCOPE (this commit) +------------------- +Mirrors the five "Migrate First / Main" tables that have an authoritative +field-level mapping in the planning workbook +("NM_Wells + Subsurface library.xlsx", sheet 3): + + tbl_well_locations -> NMW_WellLocations + tbl_well_headers -> NMW_WellHeaders + tbl_well_records -> NMW_WellRecords + tbl_well_z_datum -> NMW_WellZDatum + tbl_well_samples -> NMW_WellSamples + +Also mirrors the Geothermal and Drill Stem Test "Migrate First" tables +(columns + lengths taken directly from the NM_Wells SQL dump DDL, so these are +more precise than the five Main tables above whose lengths the sheet omitted): + + Geothermal: + tbl_gt_bht_headers -> NMW_GtBhtHeaders tbl_gt_bht_data -> NMW_GtBhtData + tbl_gt_conductivity -> NMW_GtConductivity tbl_gt_heat_flow -> NMW_GtHeatFlow + tbl_gt_sum_heat_flow-> NMW_GtSumHeatFlow tbl_gt_temp_depths -> NMW_GtTempDepths + tbl_ws_intervals -> NMW_WsIntervals + Drill Stem Tests: + tbl_ws_dst_headers -> NMW_WsDstHeaders tbl_ws_dst_intervals -> NMW_WsDstIntervals + tbl_ws_dst_flow_history -> NMW_WsDstFlowHistory + tbl_ws_dst_fluid_properties -> NMW_WsDstFluidProperties + tbl_ws_dst_pressure -> NMW_WsDstPressure + +Geothermal/DST relationship chains (kept as plain indexed GUID columns, NOT +enforced FKs, since this is staging): + well_samples.SamplSetID <- gt_bht_headers / gt_temp_depths / gt_sum_heat_flow + / ws_intervals / ws_dst_headers (SamplSetID) + gt_bht_headers.BHTGUID <- gt_bht_data.BHTGUID + ws_intervals.IntrvlGUID <- gt_conductivity / gt_heat_flow (IntrvlGUID) + ws_dst_headers.DSTGUID <- ws_dst_intervals.DSTGUID + ws_dst_intervals.DSTInterval <- ws_dst_flow_history / ws_dst_fluid_properties + / ws_dst_pressure (DSTInterval) + well_records.RecrdSetID <- gt_sum_heat_flow.RecrdSetID + +The transform of geothermal/DST into the Ocotillo model is not yet designed +(no field-level mapping in the workbook); see docs/nm_wells-migration.md. + +TRANSFORM NOTES +--------------- +Each column carries an inline note describing its eventual Ocotillo target +(from the mapping sheet). "Drop" = not carried into the Ocotillo model (kept +here only for staging fidelity / audit). See docs/nm_wells-migration.md for +the full plan and the cross-table relationship re-routing +(legacy RecrdSetID -> field_event). + +TYPE MAPPING (SQL Server -> SQLAlchemy) +--------------------------------------- + uniqueidentifier -> postgresql UUID(as_uuid=True) + int -> Integer + smallint -> SmallInteger + real / float -> Float + nvarchar -> String (source lengths not in the sheet; widened) + datetime2 -> DateTime + timestamp -> dropped (SQL Server rowversion; no value as staging data) + +PRIMARY KEYS (verified against the NM_Wells SQL dump DDL) +-------------------------------------------------------- +- NMW_WellHeaders -> WellDataID, NMW_WellRecords -> RecrdSetID, + NMW_WellSamples -> SamplSetID: declared PRIMARY KEY constraints in source. +- NMW_WellLocations, NMW_WellZDatum: source declares no PK, only unique indexes + on OBJECTID and GlobalID; OBJECTID (identity, never NULL) is used. +- Geothermal/DST: declared PKs where present (BHTGUID, IntrvlGUID, DSTGUID, + DSTInterval); the rest are heaps keyed on the OBJECTID identity column. + +LEXICON FLAGGING (Phase 2) +-------------------------- +Every ``ref_*`` table is loaded as a ``LexiconCategory`` whose rows become +``LexiconTerm``s (see transfers/reference_lexicon_transfer.py). The mirror +columns that hold those coded values will, in the Phase-2 Ocotillo model, +become ``lexicon_term`` foreign keys / enums. + +``LEXICON_REF_BY_COLUMN`` below flags every such attribute, mapping +``{tablename: {source_column: ref_source_table}}``. The lexicon *category* for +each ref table is assigned by ``transfers/reference_lexicon_transfer.py`` (one +category per ref table), so this map records the stable ref-table name rather +than the derived category string. ``LEXICON_CANDIDATES_NO_REF`` lists coded +columns that have no ``ref_*`` table and will need a NEW category / enum. +""" + +from sqlalchemy import ( + DateTime, + Float, + Integer, + SmallInteger, + String, +) +from sqlalchemy.dialects.postgresql import UUID +from sqlalchemy.orm import mapped_column + +from db.base import Base + +# Attributes that will become lexicon_term FKs / enums in the Phase-2 transform. +# {tablename: {source_column: ref_source_table}}. The lexicon category per ref +# table is assigned by transfers/reference_lexicon_transfer.py. +LEXICON_REF_BY_COLUMN: dict[str, dict[str, str]] = { + "NMW_WellLocations": { + "UnitLetter": "ref_unit_letters", + "State": "ref_states", + "County": "ref_county", + "Basin": "ref_basins", + "SourceDatum": "ref_coordinate_datum", + "SourceUnits": "ref_xy_units", + "LocAccType": "ref_coordinate_accuracy", + "LocAccMeas": "ref_coordinate_method", + }, + "NMW_WellHeaders": { + "WellClass": "ref_well_class", + "WellType": "ref_well_types", + "WellOrient": "ref_well_orientations", + "CurStatus": "ref_well_status", + }, + "NMW_WellRecords": { + "RecrdClass": "ref_well_record_class", + }, + "NMW_WellZDatum": { + "DepthDatum": "ref_ground_levels", + "DepthUnits": "ref_unit_depths", + "Z_datum": "ref_altitude_datums", + "Z_units": "ref_unit_depths", + "ElevSource": "ref_altitude_methods", + }, + "NMW_WellSamples": { + "SamplClass": "ref_sample_class", + "SampleType": "ref_sample_types", + "SmpDpUnt": "ref_unit_depths", + }, + "NMW_GtBhtHeaders": { + "BoreUnits": "ref_length_units", + "TempUnit": "ref_unit_temps", + }, + "NMW_GtBhtData": { + "TempUnit": "ref_unit_temps", + }, + "NMW_GtConductivity": { + "CnductUnit": "ref_unit_conductivity", + }, + "NMW_GtHeatFlow": { + "Kpr_unit": "ref_unit_conductivity", + "Q_unit": "ref_unit_heat_flow", + }, + "NMW_GtSumHeatFlow": { + "LithClass": "ref_lith_class", + "UnitBasis": "ref_unit_basis", + "DepthUnit": "ref_unit_depths", + "GradUnit": "ref_unit_gradients", + "SampleType": "ref_sample_types", + "TCondUnit": "ref_unit_conductivity", + "HtFlowUnit": "ref_unit_heat_flow", + }, + "NMW_GtTempDepths": { + "TempUnit": "ref_unit_temps", + }, + "NMW_WsDstHeaders": { + "PressUnits": "ref_pres_units", + "TempUnit": "ref_unit_temps", + "PipeDiaUnt": "ref_length_units", + "PipeLenUnt": "ref_length_units", + "ChokeSizUn": "ref_length_units", + }, +} + +# Coded/categorical columns with NO existing ref_* table; Phase 2 must create a +# new lexicon category or enum for these. {tablename: [source_column, ...]}. +LEXICON_CANDIDATES_NO_REF: dict[str, list[str]] = { + "NMW_GtBhtHeaders": ["DrillFluid"], + "NMW_GtHeatFlow": ["Ka_unit"], + "NMW_GtSumHeatFlow": ["Quality"], + "NMW_WsDstHeaders": ["TestType"], + "NMW_WsDstIntervals": ["Status"], + "NMW_WsDstFlowHistory": ["Operation", "RecovType"], + "NMW_WsDstPressure": ["DSTFluid"], +} + + +class NMW_WellLocations(Base): + """1:1 mirror of NM_Wells ``tbl_well_locations`` (Main / Migrate First). + + Transform target: ``location`` (point from Lat/Long_dd83, state, county) + plus a new ``NMW_Location`` table for the legacy PLSS/UTM attributes. + """ + + __tablename__ = "NMW_WellLocations" + + # No declared PK in source; OBJECTID (identity, unique index, always + # non-null) chosen over the also-unique GlobalID, which permits a NULL. + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # Drop + well_data_id = mapped_column( + "WellDataID", UUID(as_uuid=True), index=True + ) # -> NMW_Location.well_id (relates header/location/records) + well_id_legacy = mapped_column("Well_ID", String) # Drop + import_id = mapped_column("Import_ID", Integer) # Drop + township = mapped_column("Township", Float) # -> NMW_Location.township + nors_tdir = mapped_column("NorS_TDir", String) # -> NMW_Location.township_n_s + range_ = mapped_column("Range", Float) # -> NMW_Location.range + eorw_rdir = mapped_column("EorW_RDir", String) # -> NMW_Location.range_e_w + sectn = mapped_column("Sectn", SmallInteger) # -> NMW_Location.section + sectn_part = mapped_column("SectnPart", String) # -> NMW_Location.section_portion + unit_letter = mapped_column("UnitLetter", String) # -> NMW_Location.unit_letter + utm_zone = mapped_column("UTM_zone", String) # -> NMW_Location.utm_zone + state = mapped_column("State", String) # -> location.state + county = mapped_column("County", String) # -> location.county + basin = mapped_column("Basin", String) # -> NMW_Location.basin + footage_ns = mapped_column("Footage_NS", Float) # -> NMW_Location.footage_n_s + nors_fdir = mapped_column("NorS_FDir", String) # -> NMW_Location.direction_n_s + footage_ew = mapped_column("Footage_EW", Float) # -> NMW_Location.footage_e_w + eorw_fdir = mapped_column("EorW_FDir", String) # -> NMW_Location.direction_e_w + lat_min = mapped_column("Lat_min", SmallInteger) # Drop (mostly empty) + lat_sec = mapped_column("Lat_sec", Float) # Drop (mostly empty) + long_deg = mapped_column("Long_deg", SmallInteger) # Drop (mostly empty) + long_min = mapped_column("Long_min", SmallInteger) # Drop (mostly empty) + long_sec = mapped_column("Long_sec", Float) # Drop (mostly empty) + lat_dd27 = mapped_column("Lat_dd27", Float) # -> NMW_Location.latitude_dd27 + long_dd27 = mapped_column("Long_dd27", Float) # -> NMW_Location.longitude_dd27 + lat_dd83 = mapped_column("Lat_dd83", Float) # -> location.point + long_dd83 = mapped_column("Long_dd83", Float) # -> location.point + source_id = mapped_column("SourceID", String) # -> publication.id + source_datum = mapped_column("SourceDatum", String) # -> NMW_Location.source_datum + source_units = mapped_column("SourceUnits", String) # -> NMW_Location.source_units + loc_acc_type = mapped_column("LocAccType", String) # Drop + loc_acc_meas = mapped_column("LocAccMeas", String) # Drop + loc_acc_val = mapped_column("LocAccVal", Float) # Drop + duplicated = mapped_column("Duplicated", SmallInteger) # Drop + exclude = mapped_column("Exclude", SmallInteger) # Drop + comments = mapped_column("Comments", String) # (unmapped) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + api = mapped_column("API", String) # Drop + + +class NMW_WellHeaders(Base): + """1:1 mirror of NM_Wells ``tbl_well_headers`` (Main / Migrate First). + + Transform target: ``thing`` (name/type/well_depth/completion_date), + ``status_history``, ``contact`` (operator + owner), ``publication``, + ``thing_geologic_formation_association``, ``thing_id_link.alternate_id``, + plus new ``well_detail`` and ``well_purpose`` tables. + """ + + __tablename__ = "NMW_WellHeaders" + + object_id = mapped_column("OBJECTID", Integer) # Drop + # WellDataID is the key relating header <-> location <-> records. + well_data_id = mapped_column( + "WellDataID", UUID(as_uuid=True), primary_key=True + ) # Keep + well_spot_id = mapped_column( + "WellSpotID", UUID(as_uuid=True) + ) # Drop (purpose unclear) + api = mapped_column("API", String) # -> thing_id_link.alternate_id + well_class = mapped_column("WellClass", String) # -> thing.type + well_type = mapped_column("WellType", String) # -> well_purpose.purpose + well_orient = mapped_column("WellOrient", String) # -> well_detail.well_orient + cur_well_nam = mapped_column("CurWellNam", String) # -> thing.name + cur_well_num = mapped_column("CurWellNum", String) # -> well_detail.well_number + cur_status = mapped_column("CurStatus", String) # -> status_history.status + prd_pool_cnt = mapped_column("PrdPoolCnt", SmallInteger) # Drop + cur_operatr = mapped_column("CurOperatr", String) # -> contact.name (type=operator) + cur_owner = mapped_column("CurOwner", String) # -> contact.name (type=owner) + total_depth = mapped_column("TotalDepth", Float) # -> thing.well_depth + well_tvd = mapped_column("Well_TVD", Float) # Drop + fm_td = mapped_column("Fm_TD", String) # -> thing_geologic_formation_association.id + age_td = mapped_column("Age_TD", String) # Drop + spud_date = mapped_column("SpudDate", DateTime) # Drop + compl_date = mapped_column("ComplDate", DateTime) # -> thing.well_completion_date + plug_date = mapped_column("PlugDate", DateTime) # Drop + plug_back = mapped_column("PlugBack", Float) # Drop + bridge_plug = mapped_column("BridgePlug", String) # Drop + scout_tickt = mapped_column("ScoutTickt", SmallInteger) # Drop + dwn_hole_sur = mapped_column("DwnHoleSur", SmallInteger) # Drop + geol_log = mapped_column("GeolLog", SmallInteger) # Drop + geophys_log = mapped_column("Geophyslog", SmallInteger) # Drop + gthrm_exist = mapped_column("GthrmExist", SmallInteger) # Drop + petro_data = mapped_column("PetroData", SmallInteger) # Drop + core_exists = mapped_column("CoreExists", SmallInteger) # Drop + cuttings = mapped_column("Cuttings", SmallInteger) # Drop + sample_data = mapped_column("SampleData", SmallInteger) # Drop + comments = mapped_column("Comments", String) # -> well_detail.comments + import_id = mapped_column("Import_ID", String) # Drop + import_db = mapped_column("Import_DB", String) # Drop + + +class NMW_WellRecords(Base): + """1:1 mirror of NM_Wells ``tbl_well_records`` (Main / Migrate First). + + Transform target: ``field_event`` (+ ``field_activity``). The legacy + wells -> records relationship (RecrdSetID) is re-routed to + wells -> field_event during transform. RecrdClass tags which records are + geothermal. + """ + + __tablename__ = "NMW_WellRecords" + + object_id = mapped_column("OBJECTID", Integer) # Drop + recrd_set_id = mapped_column( + "RecrdSetID", UUID(as_uuid=True), primary_key=True + ) # -> field_event.id + well_data_id = mapped_column( + "WellDataID", UUID(as_uuid=True), index=True + ) # FK -> header/location WellDataID + recrd_class = mapped_column("RecrdClass", String) # -> field_activity.activity_type + source_id = mapped_column( + "SourceID", String + ) # -> publication.id (text in source, not a real FK) + action_date = mapped_column("ActionDate", DateTime) # -> field_event.event_date + well_name = mapped_column("WellName", String) # Drop + well_number = mapped_column("WellNumber", String) # Drop + api_suffix = mapped_column("API_suffix", String) # Drop + entered_by = mapped_column("EnteredBy", String) # Drop + entry_date = mapped_column("EntryDate", DateTime) # Drop + comments = mapped_column("Comments", String) # -> field_event.notes + + +class NMW_WellZDatum(Base): + """1:1 mirror of NM_Wells ``tbl_well_z_datum`` (Main / Migrate First). + + Transform target: ``measuring_point_history`` (elevation -> height, + datum -> description, units/source -> new fields). + """ + + __tablename__ = "NMW_WellZDatum" + + # No declared PK in source; OBJECTID (identity, unique index, always + # non-null) chosen over the also-unique GlobalID, which permits a NULL. + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # Drop + recrdset_id = mapped_column( + "RecrdsetID", UUID(as_uuid=True), index=True + ) # FK -> records + elev_gl = mapped_column( + "Elev_GL", Float + ) # -> measuring_point_history.measuring_point_height + elev_df = mapped_column( + "Elev_DF", Float + ) # -> measuring_point_history.measuring_point_height + elev_kb = mapped_column( + "Elev_KB", Float + ) # -> measuring_point_history.measuring_point_height + elev_unspc = mapped_column( + "Elev_unspc", Float + ) # -> measuring_point_history.measuring_point_height + datum_elev = mapped_column("DatumElev", Float) # Drop (redundant) + depth_datum = mapped_column( + "DepthDatum", String + ) # -> measuring_point_history.measuring_point_description + depth_units = mapped_column( + "DepthUnits", String + ) # -> measuring_point_history.measuring_point_units [new field] + z_datum = mapped_column("Z_datum", String) # Drop (only 7 records) + z_units = mapped_column("Z_units", String) # Drop + elev_source = mapped_column( + "ElevSource", String + ) # -> measuring_point_history.source [new field] + elv_acc_type = mapped_column("ElvAccType", String) # Drop + elv_acc_meas = mapped_column("ElvAccMeas", String) # Drop + elv_acc_val = mapped_column("ElvAccVal", Float) # Drop + comments = mapped_column("Comments", String) # Drop + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_WellSamples(Base): + """1:1 mirror of NM_Wells ``tbl_well_samples`` (Main / Migrate First). + + Transform target: ``sample`` (date/notes/created_by) + ``observation`` + (depth units). The many boolean attribute flags (Porosity, Geothermal, + etc.) are dropped (mostly empty in source). + """ + + __tablename__ = "NMW_WellSamples" + + object_id = mapped_column("OBJECTID", Integer) # Drop + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), primary_key=True + ) # -> sample.id + recrdset_id = mapped_column( + "RecrdsetID", UUID(as_uuid=True), index=True + ) # -> field_activity.id + smp_set_name = mapped_column("SmpSetName", String) # Drop + sampl_class = mapped_column("SamplClass", String) # Drop (mostly 'data') + sample_type = mapped_column("SampleType", String) # Drop (mostly empty) + sample_fm = mapped_column("SampleFm", String) # Drop (mostly empty) + sample_loc = mapped_column("SampleLoc", String) # Drop (no entries) + sample_date = mapped_column("SampleDate", DateTime) # -> sample.sample_date + from_depth = mapped_column("From_Depth", Float) # -> observation (depth) + to_depth = mapped_column("To_Depth", Float) # -> observation (depth) + smp_dp_unt = mapped_column("SmpDpUnt", String) # -> observation.unit + from_tvd = mapped_column("From_TVD", Float) # Drop + to_tvd = mapped_column("To_TVD", Float) # Drop + from_elev = mapped_column("From_Elev", Float) # Drop (empty) + to_elev = mapped_column("To_Elev", Float) # Drop (empty) + porosity = mapped_column("Porosity", SmallInteger) # Drop + permeablty = mapped_column("Permeablty", SmallInteger) # Drop + density = mapped_column("Density", SmallInteger) # Drop + dst_tests = mapped_column("DST_Tests", SmallInteger) # Drop + thin_sect = mapped_column("ThinSect", SmallInteger) # Drop + geochron = mapped_column("Geochron", SmallInteger) # Drop + geochem = mapped_column("Geochem", SmallInteger) # Drop + geothermal = mapped_column("Geothermal", SmallInteger) # Drop + whole_rock = mapped_column("WholeRock", SmallInteger) # Drop + paleontlgy = mapped_column("Paleontlgy", SmallInteger) # Drop + entered_by = mapped_column("EnteredBy", String) # -> sample.created_by_name + entry_date = mapped_column("EntryDate", DateTime) # -> sample.created_at + notes = mapped_column("Notes", String) # -> sample.notes + + +# ============================================================================= +# GEOTHERMAL (Area=Geothermal, "Migrate First") +# ============================================================================= + + +class NMW_GtBhtHeaders(Base): + """1:1 mirror of NM_Wells ``tbl_gt_bht_headers`` (bottom-hole-temp header).""" + + __tablename__ = "NMW_GtBhtHeaders" + + object_id = mapped_column("OBJECTID", Integer) # Drop (identity) + bht_guid = mapped_column("BHTGUID", UUID(as_uuid=True), primary_key=True) + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), index=True + ) # FK -> well_samples.SamplSetID + bore_dia = mapped_column("BoreDia", Float) + bore_units = mapped_column("BoreUnits", String(16)) + drill_fluid = mapped_column("DrillFluid", String(16)) + temp_unit = mapped_column("TempUnit", String(1)) + fld_salinity = mapped_column("FldSalinity", Float) + fld_rstvity = mapped_column("FldRstvity", Float) + fluid_ph = mapped_column("Fluid_pH", Float) + fld_density = mapped_column("FldDensity", Float) + fld_level = mapped_column("FldLevel", Float) + fld_viscsty = mapped_column("FldViscsty", Float) + fluid_loss = mapped_column("FluidLoss", String(50)) + notes = mapped_column("Notes", String(255)) + + +class NMW_GtBhtData(Base): + """1:1 mirror of NM_Wells ``tbl_gt_bht_data`` (BHT readings).""" + + __tablename__ = "NMW_GtBhtData" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + bht_guid = mapped_column( + "BHTGUID", UUID(as_uuid=True), index=True + ) # FK -> gt_bht_headers.BHTGUID + depth = mapped_column("Depth", Float) + bht = mapped_column("BHT", Float) + temp_unit = mapped_column("TempUnit", String(5)) + hrs_snce_cir = mapped_column("HrsSnceCir", Float) + date_measrd = mapped_column("DateMeasrd", DateTime) + comments = mapped_column("Comments", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_WsIntervals(Base): + """1:1 mirror of NM_Wells ``tbl_ws_intervals`` (sample depth intervals).""" + + __tablename__ = "NMW_WsIntervals" + + object_id = mapped_column("OBJECTID", Integer) # Drop (identity) + intrvl_guid = mapped_column("IntrvlGUID", UUID(as_uuid=True), primary_key=True) + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), index=True + ) # FK -> well_samples.SamplSetID + sample_id = mapped_column("SampleID", String(128)) + from_depth = mapped_column("From_Depth", Float) + to_depth = mapped_column("To_Depth", Float) + from_tvd = mapped_column("From_TVD", Float) + to_tvd = mapped_column("To_TVD", Float) + from_elev = mapped_column("From_Elev", Float) + to_elev = mapped_column("To_Elev", Float) + intv_notes = mapped_column("Intv_Notes", String(255)) + + +class NMW_GtConductivity(Base): + """1:1 mirror of NM_Wells ``tbl_gt_conductivity`` (thermal conductivity).""" + + __tablename__ = "NMW_GtConductivity" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + intrvl_guid = mapped_column( + "IntrvlGUID", UUID(as_uuid=True), index=True + ) # FK -> ws_intervals.IntrvlGUID + cnductvity = mapped_column("Cnductvity", Float) + cnduct_unit = mapped_column("CnductUnit", String(3)) + comments = mapped_column("Comments", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_GtHeatFlow(Base): + """1:1 mirror of NM_Wells ``tbl_gt_heat_flow`` (per-interval heat flow).""" + + __tablename__ = "NMW_GtHeatFlow" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + intrvl_guid = mapped_column( + "IntrvlGUID", UUID(as_uuid=True), index=True + ) # FK -> ws_intervals.IntrvlGUID + gradient = mapped_column("Gradient", Float) + ka = mapped_column("Ka", Float) + ka_unit = mapped_column("Ka_unit", String(3)) + pm = mapped_column("Pm", Float) + kpr = mapped_column("Kpr", Float) + kpr_unit = mapped_column("Kpr_unit", String(3)) + q = mapped_column("Q", Float) + q_unit = mapped_column("Q_unit", String(3)) + comments = mapped_column("Comments", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_GtSumHeatFlow(Base): + """1:1 mirror of NM_Wells ``tbl_gt_sum_heat_flow`` (summary heat flow).""" + + __tablename__ = "NMW_GtSumHeatFlow" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + recrd_set_id = mapped_column( + "RecrdSetID", UUID(as_uuid=True), index=True + ) # FK -> well_records.RecrdSetID + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), index=True + ) # FK -> well_samples.SamplSetID + lith_class = mapped_column("LithClass", String(50)) + unit_basis = mapped_column("UnitBasis", String(16)) + unit_name = mapped_column("UnitName", String(128)) + geo_id = mapped_column("GeoID", String(16)) + from_depth = mapped_column("FromDepth", Float) + to_depth = mapped_column("ToDepth", Float) + depth_unit = mapped_column("DepthUnit", String(8)) + from_elev = mapped_column("From_Elev", Float) + to_elev = mapped_column("To_Elev", Float) + therml_grad = mapped_column("ThermlGrad", Float) + tg_error = mapped_column("TGError", Float) + grad_unit = mapped_column("GradUnit", String(3)) + tgrad_range = mapped_column("TGradRange", String(15)) + sample_type = mapped_column("SampleType", String(50)) + num_samples = mapped_column("NumSamples", SmallInteger) + therml_cond = mapped_column("ThermlCond", Float) + tcond_error = mapped_column("TCondError", Float) + tcond_unit = mapped_column("TCondUnit", String(3)) + tcond_range = mapped_column("TCondRange", String(15)) + heat_flow = mapped_column("HeatFlow", Float) + ht_flow_err = mapped_column("HtFlowErr", Float) + ht_flow_unit = mapped_column("HtFlowUnit", String(3)) + ht_flow_est = mapped_column("HtFlowEst", Float) + quality = mapped_column("Quality", String(50)) + comments = mapped_column("Comments", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_GtTempDepths(Base): + """1:1 mirror of NM_Wells ``tbl_gt_temp_depths`` (temp-vs-depth profile).""" + + __tablename__ = "NMW_GtTempDepths" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), index=True + ) # FK -> well_samples.SamplSetID + depth = mapped_column("Depth", Float) + temp = mapped_column("Temp", Float) + temp_unit = mapped_column("TempUnit", String(1)) + intrvl_grad = mapped_column("IntrvlGrad", Float) + comments = mapped_column("Comments", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +# ============================================================================= +# DRILL STEM TESTS (Area=Drill Stem Tests, "Migrate First") +# ============================================================================= + + +class NMW_WsDstHeaders(Base): + """1:1 mirror of NM_Wells ``tbl_ws_dst_headers`` (DST header).""" + + __tablename__ = "NMW_WsDstHeaders" + + object_id = mapped_column("OBJECTID", Integer) # Drop (identity) + dst_guid = mapped_column("DSTGUID", UUID(as_uuid=True), primary_key=True) + sampl_set_id = mapped_column( + "SamplSetID", UUID(as_uuid=True), index=True + ) # FK -> well_samples.SamplSetID + test_type = mapped_column("TestType", String(50)) + dst_operator = mapped_column("DSTOprator", String(50)) + press_units = mapped_column("PressUnits", String(8)) + temp_unit = mapped_column("TempUnit", String(1)) + pipe_dia_unt = mapped_column("PipeDiaUnt", String(8)) + pipe_len_unt = mapped_column("PipeLenUnt", String(8)) + choke_siz_un = mapped_column("ChokeSizUn", String(8)) + notes = mapped_column("Notes", String(255)) + + +class NMW_WsDstIntervals(Base): + """1:1 mirror of NM_Wells ``tbl_ws_dst_intervals`` (DST interval).""" + + __tablename__ = "NMW_WsDstIntervals" + + object_id = mapped_column("OBJECTID", Integer) # Drop (identity) + dst_interval = mapped_column("DSTInterval", UUID(as_uuid=True), primary_key=True) + dst_guid = mapped_column( + "DSTGUID", UUID(as_uuid=True), index=True + ) # FK -> ws_dst_headers.DSTGUID + dst_name = mapped_column("DSTName", String(128)) + target_fm = mapped_column("TargetFm", String(16)) + dst_date = mapped_column("DSTDate", DateTime) + dst_number = mapped_column("DSTNumber", SmallInteger) + status = mapped_column("Status", String(255)) + status_date = mapped_column("StatusDate", DateTime) + packr_from = mapped_column("PackrFrom", Float) + packer_to = mapped_column("PackerTo", Float) + srf_choke_sz = mapped_column("SrfChokeSz", Float) + bot_choke_sz = mapped_column("BotChokeSz", Float) + pipe_dia = mapped_column("PipeDia", Float) + pipe_length = mapped_column("PipeLength", Float) + notes = mapped_column("Notes", String(255)) + + +class NMW_WsDstFlowHistory(Base): + """1:1 mirror of NM_Wells ``tbl_ws_dst_flow_history`` (DST flow events).""" + + __tablename__ = "NMW_WsDstFlowHistory" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + dst_interval = mapped_column( + "DSTInterval", UUID(as_uuid=True), index=True + ) # FK -> ws_dst_intervals.DSTInterval + operation = mapped_column("Operation", String(255)) + start_time = mapped_column("StartTime", DateTime) + end_time = mapped_column("EndTime", DateTime) + duration = mapped_column("Duration", Float) + pressure = mapped_column("Pressure", Float) + temp = mapped_column("Temp", Float) + recov_column = mapped_column("RecovColmn", Float) + recov_type = mapped_column("RecovType", String(255)) + notes = mapped_column("Notes", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_WsDstFluidProperties(Base): + """1:1 mirror of NM_Wells ``tbl_ws_dst_fluid_properties`` (recovered fluid).""" + + __tablename__ = "NMW_WsDstFluidProperties" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + dst_interval = mapped_column( + "DSTInterval", UUID(as_uuid=True), index=True + ) # FK -> ws_dst_intervals.DSTInterval + source_loc = mapped_column("SourceLoc", String(255)) + resistivity = mapped_column("Resistivty", Float) + temp = mapped_column("Temp", Float) + chlorides = mapped_column("Chlorides", Float) + notes = mapped_column("Notes", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +class NMW_WsDstPressure(Base): + """1:1 mirror of NM_Wells ``tbl_ws_dst_pressure`` (DST pressure readings).""" + + __tablename__ = "NMW_WsDstPressure" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + dst_interval = mapped_column( + "DSTInterval", UUID(as_uuid=True), index=True + ) # FK -> ws_dst_intervals.DSTInterval + prs_gage_dpt = mapped_column("PrsGageDpt", Float) + blanked_off = mapped_column("BlankedOff", SmallInteger) + in_sht_in_min = mapped_column("InShtInMin", Float) + flw_prs_in_min = mapped_column("FlwPrsInMin", Float) + prs_in_sht_in = mapped_column("PrsInShtIn", Float) + prs_init_clsd_in = mapped_column("PrsInitClsdIn", Float) + fn_sht_in_min = mapped_column("FnShtInMin", Float) + flw_prs_fin_min = mapped_column("FlwPrsFinMin", Float) + prs_fn_sht_in = mapped_column("PrsFnShtIn", Float) + sht_in_pr_mth = mapped_column("ShtInPrMth", String(255)) + hydrost_prs_in = mapped_column("HydrostPrsIn", Float) + hyd_st_prs_fl = mapped_column("HydStPrsFl", Float) + hydst_pr_mth = mapped_column("HydstPrMth", String(255)) + equil_press = mapped_column("EquilPress", Float) + eql_prs_mth = mapped_column("EqlPrsMth", String(255)) + flow_prs_min = mapped_column("FlowPrsMin", Float) + flow_prs_max = mapped_column("FlowPrsMax", Float) + flow_prs_mth = mapped_column("FlowPrsMth", String(255)) + dst_fluid = mapped_column("DSTFluid", String(128)) + fm_temp = mapped_column("FmTemp", Float) + temp_corrtn = mapped_column("TempCorrtn", Float) + temp_flowng = mapped_column("TempFlowng", Float) + temp_unit = mapped_column("TempUnit", String(5)) + notes = mapped_column("Notes", String(255)) + global_id = mapped_column("GlobalID", UUID(as_uuid=True)) # Drop + + +# ============================================================================= +# PUBLICATIONS +# ============================================================================= + + +class NMW_Sources(Base): + """1:1 mirror of NM_Wells ``tbl_sources`` (publication / data source registry). + + Transform target: ``publication``. Each row is a citable source keyed by + the free-text ``SourceID`` string that appears in ``NMW_WellRecords.SourceID``. + """ + + __tablename__ = "NMW_Sources" + + object_id = mapped_column("OBJECTID", Integer, primary_key=True) # identity PK + source_id = mapped_column("SourceID", String, index=True) # join key (text FK) + first_auth = mapped_column("FirstAuth", String) + pub_year = mapped_column("PubYear", String) + title = mapped_column("Title", String) + journal = mapped_column("Journal", String) + volume = mapped_column("Volume", String) + page_no = mapped_column("PageNo", String) + report_no = mapped_column("ReportNo", String) + publisher = mapped_column("Publisher", String) + city = mapped_column("City", String) + url = mapped_column("URL", String) + comments = mapped_column("Comments", String) + + +# ============================================================================= +# TODO(remaining "Migrate First" tables, no DDL/mapping yet) +# ----------------------------------------------------------------------------- +# Subsurface Library: dst_scan, log_scanned, Well_Header, well_operators +# See docs/nm_wells-migration.md for the full inventory + recommendations. +# ============================================================================= + + +# ============= EOF ============================================= diff --git a/docker-compose.yml b/docker-compose.yml index 78120d76..94991fb9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -23,9 +23,11 @@ services: retries: 20 app: - build: + build: context: . dockerfile: ./docker/app/Dockerfile + args: + INSTALL_DEV: "true" environment: - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} diff --git a/docs/nm_wells-migration.md b/docs/nm_wells-migration.md new file mode 100644 index 00000000..35f335b5 --- /dev/null +++ b/docs/nm_wells-migration.md @@ -0,0 +1,190 @@ +# NM_Wells → Ocotillo migration + +Migration of the legacy **NM_Wells** SQL Server database (and the related +Subsurface Library) into OcotilloAPI. Source of truth for table inventory and +field-level recommendations: `NM_Wells + Subsurface library.xlsx` (planning +workbook, not in repo). + +## Two-phase approach + +1. **Phase 1 — 1:1 staging mirror** *(current)*. Land source tables unchanged + into `NMW_*` mirror tables (`db/nmw_legacy.py`), column-for-column. No + transform. Mirrors the `db/nma_legacy.py` (NM_Aquifer) convention. This lets + us load the SQL dump first and transform later without re-reading the source. +2. **Phase 2 — transform** *(later)*. Map mirror rows into the Ocotillo data + model (`Location → Thing → FieldEvent → FieldActivity → Sample → + Observation`, plus `status_history`, `measuring_point_history`, `contact`, + `publication`, `thing_geologic_formation_association`, `thing_id_link`) using + the existing CSV→Pandas→ORM transfer pattern in `transfers/`. Per-column + targets are recorded inline in `db/nmw_legacy.py` and summarized below. + +## Source access + +NM_Wells is delivered as a SQL dump. Physical source table names are +`tbl_well_*` / `tbl_gt_*` / `tbl_ws_*` (snake_case). To use the existing +transfer pipeline, export each source table to CSV (same flow as the +NM_Aquifer `nma_csv_cache`). + +## Phase 1 scope (mirrored now) + +Five "Migrate First / Main" tables — the only ones with an authoritative +field-level mapping in the workbook (sheet 3): + +| Source table | Mirror model | Cols | +|----------------------|----------------------|------| +| `tbl_well_locations` | `NMW_WellLocations` | 40 | +| `tbl_well_headers` | `NMW_WellHeaders` | 35 | +| `tbl_well_records` | `NMW_WellRecords` | 12 | +| `tbl_well_z_datum` | `NMW_WellZDatum` | 18 | +| `tbl_well_samples` | `NMW_WellSamples` | 30 | + +Migration: `alembic/versions/u7v8w9x0y1z2_nmw_legacy_staging_mirror_tables.py`. + +Geothermal + Drill Stem Test "Migrate First" tables are also mirrored (columns ++ lengths taken straight from the SQL-dump DDL): + +| Source table | Mirror model | Cols | PK | +|-------------------------------|----------------------------|------|----| +| `tbl_gt_bht_headers` | `NMW_GtBhtHeaders` | 16 | BHTGUID | +| `tbl_gt_bht_data` | `NMW_GtBhtData` | 10 | OBJECTID | +| `tbl_ws_intervals` | `NMW_WsIntervals` | 12 | IntrvlGUID | +| `tbl_gt_conductivity` | `NMW_GtConductivity` | 7 | OBJECTID | +| `tbl_gt_heat_flow` | `NMW_GtHeatFlow` | 13 | OBJECTID | +| `tbl_gt_sum_heat_flow` | `NMW_GtSumHeatFlow` | 30 | OBJECTID | +| `tbl_gt_temp_depths` | `NMW_GtTempDepths` | 9 | OBJECTID | +| `tbl_ws_dst_headers` | `NMW_WsDstHeaders` | 11 | DSTGUID | +| `tbl_ws_dst_intervals` | `NMW_WsDstIntervals` | 17 | DSTInterval | +| `tbl_ws_dst_flow_history` | `NMW_WsDstFlowHistory` | 13 | OBJECTID | +| `tbl_ws_dst_fluid_properties` | `NMW_WsDstFluidProperties`| 9 | OBJECTID | +| `tbl_ws_dst_pressure` | `NMW_WsDstPressure` | 28 | OBJECTID | + +Migration: `alembic/versions/v8w9x0y1z2a3_nmw_geothermal_dst_mirror_tables.py`. + +Link columns (`SamplSetID`, `BHTGUID`, `IntrvlGUID`, `DSTGUID`, `DSTInterval`, +`RecrdSetID`) are kept as plain indexed GUID columns — NOT enforced FKs — since +this is staging. Relationship chains documented in `db/nmw_legacy.py`. + +### PK / type notes +- SQL Server `uniqueidentifier` → postgres `UUID`; `real/float` → `Float`; + `nvarchar` → `String` (source lengths absent from the sheet, so widened); + `datetime2` → `DateTime`; `timestamp` (rowversion) → **dropped**. +- PKs **verified against the dump DDL**: Headers→`WellDataID`, + Records→`RecrdSetID`, Samples→`SamplSetID` (declared PKs); + Locations→`OBJECTID`, ZDatum→`OBJECTID` (no declared PK in source — unique + indexes on both OBJECTID and GlobalID; OBJECTID identity is never NULL). + +## Loading the mirror (Phase 1 transfer) + +`transfers/nmw_mirror_transfer.py` (`transfer_nmw_mirror(session, limit)`) loads +each source table into its `NMW_*` table 1:1. It is data-driven over +`NMW_MIRROR_SPECS` (one `(model, source_table)` per mirror), derives column +handling from each model's `__table__` metadata, coerces types +(uuid/int/float/datetime/string; NULL/NaN/NaT → None; rowversion dropped), and +chunk-upserts via `INSERT ... ON CONFLICT () DO NOTHING`. + +**Row source** (selected at runtime): +- **SQL Server data dump** — set `NMW_SQL_DUMP` to a `.sql` file of + `INSERT [dbo].[tbl_*] (...) VALUES (...)` statements. `transfers/nmw_sql_dump.py` + splits statements with **sqlparse** and `write_table_csv` writes one CSV per + table (handles `N'...'`/escaped `''`, embedded commas/parens, `CAST(... AS ...)`, + multi-row `VALUES`, `0x` binary → NULL, UTF-16/UTF-8 BOM). The mirror then + bulk-loads each CSV with Postgres **`COPY ... FROM STDIN`** (truncate + COPY; + Postgres casts text → column types). CSV dir = `NMW_CSV_DIR` (default temp). +- **CSV exports** — fallback when `NMW_SQL_DUMP` is unset; per-table CSVs in + `nma_csv_cache` / GCS `nma_csv/`, inserted row-by-row with type coercion. + +> Note: `NMWells.sql` as provided is **schema-only** (DDL, no `INSERT`s) — it +> seeded the models/migrations. A separate **data** dump (with `INSERT`s) is +> what `NMW_SQL_DUMP` should point at. + +Run via the standalone orchestrator `transfers/transfer_geothermal.py` +(`python -m transfers.transfer_geothermal`) — **separate** from the deprecated +`transfers/transfer.py` (NM_Aquifer driver), which must not gain new migrations. +The orchestrator runs the reference→lexicon load (`TRANSFER_GEOTHERMAL_REFERENCE`) +then the mirror load (`TRANSFER_NMW_MIRROR`); both default on. After the mirror +load it calls `refresh_materialized_views` to `REFRESH` the materialized OGC views +(currently `ogc_geothermal_wells_temperature_profile`; missing views are skipped). +Assumes the schema already exists (`alembic upgrade head`). + +## OGC views (pygeoapi) + +`alembic/versions/w9x0y1z2a3b4_add_geothermal_ogc_views.py` adds two point +layers over the `NMW_*` mirror (geometry from `NMW_WellLocations` Lat/Long_dd83): + +- `ogc_geothermal_wells_bht` — one feature per geothermal well with + bottom-hole-temperature data (`NMW_GtBhtData`); aggregate BHT stats. +- `ogc_geothermal_wells_temperature_profile` — **materialized** view, one feature + per geothermal well with a downhole temperature-vs-depth series + (`NMW_GtTempDepths`, ~370k source rows), series as an ordered JSON array. + Indexed (unique `well_data_id`, GiST `geom`); `REFRESH MATERIALIZED VIEW` after + a data reload. + +- `ogc_geothermal_wells_summary_heat_flow` — one feature per geothermal well with + summary heat-flow determinations (`NMW_GtSumHeatFlow`): aggregate heat flow, + thermal gradient, thermal conductivity, quality, plus a `measurements` JSON + series (one element per determination, ordered by depth). Linked directly via + `NMW_GtSumHeatFlow.RecrdSetID → NMW_WellRecords.RecrdSetID`. +- `ogc_geothermal_wells_interval_heat_flow` — one feature per geothermal well + with **per-interval** heat-flow values (`NMW_GtHeatFlow`): aggregate heat flow + (Q), gradient, conductivity (Kpr), diffusivity (Ka), plus a `measurements` JSON + series (one element per interval, ordered by depth). Linked via + `NMW_GtHeatFlow.IntrvlGUID → NMW_WsIntervals.IntrvlGUID → NMW_WellSamples → + NMW_WellRecords`. + +Well linkage: `gt_*.SamplSetID → NMW_WellSamples.SamplSetID → +NMW_WellRecords.RecrdSetID → NMW_WellLocations.WellDataID`. + +## Phase 2 transform map (summary) + +Key relationship re-routing: legacy `WellDataID` ties header/location/records; +`RecrdSetID` ties records→children. In Ocotillo, **wells → records** becomes +**wells (Thing) → field_event**, and `RecrdClass` → `field_activity.activity_type`. + +| Source | Ocotillo target | +|-------------------------------------|-----------------| +| `tbl_well_locations` Lat/Long_dd83 | `location.point` | +| `tbl_well_locations` State/County | `location.state` / `location.county` | +| `tbl_well_locations` PLSS/UTM/dd27 | **new** `NMW_Location` (township, range, section, unit_letter, utm_zone, basin, footages, dd27, source_datum, source_units) | +| `tbl_well_headers` CurWellNam/WellClass/TotalDepth/ComplDate | `thing.name` / `thing.type` / `thing.well_depth` / `thing.well_completion_date` | +| `tbl_well_headers` CurStatus | `status_history.status` | +| `tbl_well_headers` CurOperatr/CurOwner | `contact.name` (type = operator / owner) | +| `tbl_well_headers` API | `thing_id_link.alternate_id` | +| `tbl_well_headers` Fm_TD | `thing_geologic_formation_association` | +| `tbl_well_headers` WellType/WellOrient/CurWellNum/Comments | **new** `well_purpose.purpose` / **new** `well_detail` (well_orient, well_number, comments) | +| `tbl_well_records` RecrdSetID/ActionDate/Comments | `field_event` (id/event_date/notes) | +| `tbl_well_records` RecrdClass | `field_activity.activity_type` | +| `tbl_well_z_datum` Elev_GL/DF/KB/unspc | `measuring_point_history.measuring_point_height` | +| `tbl_well_z_datum` DepthDatum/DepthUnits/ElevSource | `measuring_point_history` description / units / source | +| `tbl_well_samples` SamplSetID/SampleDate/Notes/EnteredBy | `sample` (id/sample_date/notes/created_by_name) | +| `tbl_well_samples` From_Depth/To_Depth/SmpDpUnt | `observation` (depths / unit) | + +**New tables Phase 2 needs:** `NMW_Location`, `well_detail`, `well_purpose`. + +## Reference tables → lexicon + +The legacy `ref_*` lookup tables ("Add to lexicon" in workbook sheet 1) are +loaded into the lexicon by `transfers/reference_lexicon_transfer.py` +(`transfer_reference_tables`), registered as a **foundational** transfer in +`transfers/transfer.py` (runs before wells). Each `ref_*` table becomes a +`LexiconCategory` (name = table minus `ref_`) and its rows become +`LexiconTerm`s linked to that category. Idempotent (`ON CONFLICT DO NOTHING`), +matching `core.initializers.init_lexicon`. + +Term/definition columns are **auto-detected** from each CSV header (the +workbook has no column lists for `ref_*`); override `term_col`/`definition_col` +on a `RefTableSpec` if detection is wrong. The Subsurface Library `LU_*` +lookups are also "Add to lexicon" — add them to `REFERENCE_TABLE_SPECS` once +their CSVs are available. + +## Not yet mapped (no field list / DDL in workbook) + +Remaining "Migrate First" tables not yet mirrored: + +- **Publications:** `tbl_sources` (DDL available in dump; not yet requested) +- **Subsurface Library:** `dst_scan`, `log_scanned`, `Well_Header`, + `well_operators` + +`tbl_well_bores` (Geothermal area) is "Review", not "Migrate First". + +"Don't migrate" (per workbook): OCD injection/wells imports, `*_ZOLD`, +`*_DataAsOf-*` snapshots, `geometry_columns`, `spatial_ref_sys`, `sysdiagrams`. diff --git a/docs/nm_wells-transfer-runbook.md b/docs/nm_wells-transfer-runbook.md new file mode 100644 index 00000000..60731a8b --- /dev/null +++ b/docs/nm_wells-transfer-runbook.md @@ -0,0 +1,254 @@ +# NM_Wells 1:1 Mirror Transfer — Runbook + +Operational steps to run the NM_Wells (geothermal) Phase-1 mirror transfer and verify it +worked. Phase 1 is a faithful, column-for-column copy of the legacy NM_Wells SQL Server +tables into the Postgres `NMW_*` staging mirror — no transform to the Ocotillo model. + +- Code: `transfers/transfer_geothermal.py` (orchestrator), `transfers/nmw_mirror_transfer.py` + (loader), `transfers/export_nmw_csvs.py` (CSV export), `transfers/nmw_sql_dump.py` (dump parser). +- Models: `db/nmw_legacy.py` (18 `NMW_*` tables). +- Design notes: [docs/nm_wells-migration.md](nm_wells-migration.md). +- Jira: [BDMS-945](https://nmbgmr.atlassian.net/browse/BDMS-945) (story), + [BDMS-969](https://nmbgmr.atlassian.net/browse/BDMS-969) (this e2e run), + [BDMS-970](https://nmbgmr.atlassian.net/browse/BDMS-970) (SQL Server access — blocker). + +--- + +## 0. Prerequisites + +- [ ] **SQL Server access** ([BDMS-970](https://nmbgmr.atlassian.net/browse/BDMS-970)): + password reset done, can reach the NM_Wells host (Argon / Agustin / Sediment / + SQL dev / SQLServer2019 as applicable). +- [ ] Python env ready: `uv venv && source .venv/bin/activate && uv sync --locked`. +- [ ] Target Postgres + PostGIS reachable; `.env` has `POSTGRES_*` (or Cloud SQL) creds. +- [ ] `.env` has the SQL Server source creds (only needed for the live export, step 1): + +```bash +NMW_HOST= +NMW_PORT=1433 +NMW_USER= +NMW_PASSWORD= +NMW_DATABASE=NM_Wells +``` + +Pick **one** row source for the load: + +| Source | When | Set | +|--------|------|-----| +| Per-table CSVs | live export via pymssql (default path) | nothing (`NMW_SQL_DUMP` unset) | +| SQL dump `.sql` | you have an SSMS data dump | `NMW_SQL_DUMP=/path/to/dump.sql` | + +--- + +## 1. Apply schema (migrations) + +The transfer assumes the schema already exists — it does not create/drop tables. + +```bash +alembic upgrade head +``` + +Creates the 18 `NMW_*` tables + FKs and the 8 OGC backing views. Migration chain: +`c0d1e2f3a4b5` (tables+FK) → `d1e2f3a4b5c6` (per-well views) → `e2f3a4b5c6d7` (measurement views). + +Verify the tables and views exist: + +```bash +psql "$DATABASE_URL" -c '\dt "NMW_*"' # expect 18 tables +psql "$DATABASE_URL" -c '\dv ogc_*' # ogc_* views +psql "$DATABASE_URL" -c '\dm ogc_*' # matview: ogc_geothermal_wells_temperature_profile +``` + +--- + +## 2. Export source tables to CSV (live source) + +Skip if you're loading from a `.sql` dump (`NMW_SQL_DUMP` set). + +```bash +uv run python -m transfers.export_nmw_csvs +``` + +- Writes `transfers/data/nma_csv_cache/
.csv`, one per mirrored table. +- Prints per-table row counts — **record these**; they are the source-of-truth counts for + the post-load comparison in step 4. +- Any `FAILED: ...` line means that table didn't export — investigate before loading. + +--- + +## 3. Run the transfer + +Smoke-test with a row cap first, then run the full load. + +```bash +# Smoke test: 1000 rows/table +TRANSFER_LIMIT=1000 uv run python -m transfers.transfer_geothermal + +# Full load (all rows) +uv run python -m transfers.transfer_geothermal +``` + +Relevant env (all optional, sane defaults): + +| Var | Default | Effect | +|-----|---------|--------| +| `TRANSFER_LIMIT` | 0 (all) | rows per table | +| `NMW_SQL_DUMP` | unset | load from `.sql` dump instead of CSVs | +| `NMW_CSV_DIR` | temp dir | where dump-derived CSVs are written (dump path) | +| `TRANSFER_GEOTHERMAL_REFERENCE` | 1 | load `ref_*` → lexicon | +| `TRANSFER_NMW_MIRROR` | 1 | load `NMW_*` mirror + refresh matviews | + +The orchestrator: loads reference→lexicon, loads the mirror parent→child in FK order, then +refreshes the materialized OGC view. It prints a summary dict — confirm +`mirror.errors == 0` and `reference.errors == 0`. + +Re-running is safe: dump path is truncate+COPY (CASCADE), CSV path is +`INSERT ... ON CONFLICT DO NOTHING`. No duplicate rows. + +--- + +## 4. Verify — row counts + +Compare each mirror table's row count against the source counts captured in step 2 +(or against SQL Server directly). + +```bash +psql "$DATABASE_URL" <<'SQL' +SELECT 'NMW_WellHeaders' AS t, count(*) FROM "NMW_WellHeaders" +UNION ALL SELECT 'NMW_WellLocations', count(*) FROM "NMW_WellLocations" +UNION ALL SELECT 'NMW_WellRecords', count(*) FROM "NMW_WellRecords" +UNION ALL SELECT 'NMW_WellSamples', count(*) FROM "NMW_WellSamples" +UNION ALL SELECT 'NMW_WellZDatum', count(*) FROM "NMW_WellZDatum" +UNION ALL SELECT 'NMW_Sources', count(*) FROM "NMW_Sources" +UNION ALL SELECT 'NMW_GtBhtHeaders', count(*) FROM "NMW_GtBhtHeaders" +UNION ALL SELECT 'NMW_GtBhtData', count(*) FROM "NMW_GtBhtData" +UNION ALL SELECT 'NMW_GtTempDepths', count(*) FROM "NMW_GtTempDepths" +UNION ALL SELECT 'NMW_GtConductivity', count(*) FROM "NMW_GtConductivity" +UNION ALL SELECT 'NMW_GtHeatFlow', count(*) FROM "NMW_GtHeatFlow" +UNION ALL SELECT 'NMW_GtSumHeatFlow', count(*) FROM "NMW_GtSumHeatFlow" +UNION ALL SELECT 'NMW_WsDstHeaders', count(*) FROM "NMW_WsDstHeaders" +UNION ALL SELECT 'NMW_WsDstIntervals', count(*) FROM "NMW_WsDstIntervals" +UNION ALL SELECT 'NMW_WsDstFlowHistory', count(*) FROM "NMW_WsDstFlowHistory" +UNION ALL SELECT 'NMW_WsDstFluidProperties', count(*) FROM "NMW_WsDstFluidProperties" +UNION ALL SELECT 'NMW_WsDstPressure', count(*) FROM "NMW_WsDstPressure" +UNION ALL SELECT 'NMW_WsIntervals', count(*) FROM "NMW_WsIntervals" +ORDER BY t; +SQL +``` + +**Pass:** every count matches source (or matches `TRANSFER_LIMIT` if capped). Note any table +where the count is 0 or short — likely a failed export or an FK-skipped child row. + +--- + +## 5. Verify — FK integrity + +No child row should reference a missing parent. Spot-check the main hierarchy +(`NMW_WellHeaders` is the root parent): + +```bash +psql "$DATABASE_URL" <<'SQL' +-- locations / records with no matching well header (expect 0) +SELECT 'orphan_locations' AS check, count(*) +FROM "NMW_WellLocations" l +LEFT JOIN "NMW_WellHeaders" h ON l."WellDataID" = h."WellDataID" +WHERE h."WellDataID" IS NULL +UNION ALL +SELECT 'orphan_records', count(*) +FROM "NMW_WellRecords" r +LEFT JOIN "NMW_WellHeaders" h ON r."WellDataID" = h."WellDataID" +WHERE h."WellDataID" IS NULL; +SQL +``` + +**Pass:** both counts are 0. (FK constraints are enforced at load, so a non-zero here means +data was loaded out of order or a constraint is missing — investigate.) + +**Known exception:** `orphan_locations` reports **51** rows. These are `NMW_WellLocations` +rows whose `WellDataID` is blank in the source (`tbl_well_locations.csv`): empty values load +as NULL, NULL FK columns are exempt from FK enforcement, and the `LEFT JOIN ... IS NULL` +check counts them as orphans. This is a source data-quality issue, not a load-order or +constraint problem — accepted as-is. `orphan_records` must still be 0. + +--- + +## 6. Verify — OGC views + matview + +Refresh happens automatically in step 3. To refresh manually: + +```bash +psql "$DATABASE_URL" -c 'REFRESH MATERIALIZED VIEW ogc_geothermal_wells_temperature_profile;' +``` + +Confirm each backing view returns rows and the per-well views emit **one feature per well** +(no count multiplication from duplicate location rows): + +```bash +psql "$DATABASE_URL" <<'SQL' +SELECT 'ogc_geothermal_wells_bht' AS v, count(*) FROM ogc_geothermal_wells_bht +UNION ALL SELECT 'ogc_geothermal_wells_temperature_profile', count(*) FROM ogc_geothermal_wells_temperature_profile +UNION ALL SELECT 'ogc_geothermal_wells_summary_heat_flow', count(*) FROM ogc_geothermal_wells_summary_heat_flow +UNION ALL SELECT 'ogc_geothermal_wells_interval_heat_flow', count(*) FROM ogc_geothermal_wells_interval_heat_flow +UNION ALL SELECT 'ogc_bht_measurements', count(*) FROM ogc_bht_measurements +UNION ALL SELECT 'ogc_temp_depth_measurements',count(*) FROM ogc_temp_depth_measurements +UNION ALL SELECT 'ogc_heat_flow', count(*) FROM ogc_heat_flow +UNION ALL SELECT 'ogc_dst', count(*) FROM ogc_dst; +SQL +``` + +Then hit the OGC API (with the app running) — all 6 collections should resolve and return +GeoJSON features: + +```bash +for c in geothermal_wells_bht geothermal_wells_temperature_profile \ + bht_measurements temp_depth_measurements heat_flow dst; do + echo "== $c ==" + curl -s "http://localhost:8000/ogcapi/collections/$c/items?limit=1" | head -c 400 + echo +done +``` + +**Pass:** each returns HTTP 200 with a `FeatureCollection`; per-well collections show +distinct wells (no duplicate `WellDataID`). + +--- + +## 7. Verify — migrations reversible (non-prod only) + +On a scratch/test DB, confirm a clean down/up cycle drops and recreates all 18 tables + 8 +views with no orphans: + +```bash +alembic downgrade base +psql "$DATABASE_URL" -c '\dt "NMW_*"' # expect 0 +psql "$DATABASE_URL" -c '\dv ogc_*' # expect 0 +alembic upgrade head # recreate +``` + +Automated coverage for this lives in `tests/test_nmw_mirror.py` (19 tests): +`uv run pytest tests/test_nmw_mirror.py`. + +--- + +## Sign-off checklist (closes BDMS-969 → unblocks BDMS-951 / BDMS-954) + +- [ ] Schema applied; 18 tables + 8 views present (step 1). +- [ ] Source CSVs exported; per-table source counts recorded (step 2). +- [ ] Transfer ran with `errors == 0` (step 3). +- [ ] Row counts match source (step 4). +- [ ] No orphan FK rows (step 5). +- [ ] All 6 OGC collections resolve; one feature per well (step 6). +- [ ] Migrations down/up clean on scratch DB (step 7). + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| `export_nmw_csvs` connection refused | SQL Server access not granted | [BDMS-970](https://nmbgmr.atlassian.net/browse/BDMS-970); check `NMW_HOST/PORT`, VPN | +| `TRUNCATE ... cannot truncate a table referenced in a foreign key` | parent truncated before child | loader uses `TRUNCATE ... CASCADE` (B2); confirm you're on current branch | +| Mirror column holds literal `CAST(...)` string | dump parser missed a parameterised type | fixed in `nmw_sql_dump.py` (B1); confirm branch is current | +| Per-well OGC view count > # wells | duplicate `NMW_WellLocations` rows | views dedup via `DISTINCT ON (WellDataID)` (B3); confirm branch is current | +| matview empty / stale | refresh skipped | `REFRESH MATERIALIZED VIEW ogc_geothermal_wells_temperature_profile;` | +| child table row count short | FK-skipped rows (`ON CONFLICT`/missing parent) | check parent loaded first; re-run full load | diff --git a/pyproject.toml b/pyproject.toml index 2e575142..2253d7ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -90,6 +90,7 @@ dependencies = [ "sqlalchemy-continuum==1.6.0", "sqlalchemy-searchable==2.1.0", "sqlalchemy-utils==0.42.1", + "sqlparse>=0.5.5", "starlette==1.3.1", "starlette-admin[i18n]==0.16.1", "typer==0.26.7", @@ -100,6 +101,7 @@ dependencies = [ "utm==0.8.1", "uvicorn==0.49.0", "yarl==1.24.2", + "pymssql>=2.3.13", ] [tool.uv] diff --git a/requirements.txt b/requirements.txt index 125e0bff..36342504 100644 --- a/requirements.txt +++ b/requirements.txt @@ -172,9 +172,9 @@ authlib==1.7.2 \ --hash=sha256:2cea25fefcd4e7173bdf1372c0afc265c8034b23a8cd5dcb6a9164b826c64231 \ --hash=sha256:3e1faedc9d87e7d56a164eca3ccb6ace0d61b94abe83e92242f8dc8bba9b4a9f # via ocotilloapi -babel==2.18.0 \ - --hash=sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d \ - --hash=sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35 +babel==2.17.0 \ + --hash=sha256:0c54cffb19f690cdcc52a3b50bcbf71e07a808d1c80d549f2459b9d2cf0afb9d \ + --hash=sha256:4d0b53093fdfb4b21c92b5213dba5a1b23885afa8383709427046b21c366e5f2 # via # pygeoapi # starlette-admin @@ -365,7 +365,6 @@ colorama==0.4.6 ; sys_platform == 'win32' \ # click # typer cryptography==48.0.1 \ - --hash=sha256:08a597acce1ff37f347400087776599e2348a3a8bc53b44120e463cd274efe4a \ --hash=sha256:09f73a725d582cef64b91281a322cd798d14a33b2b6f2b7ad9531dc336d84c02 \ --hash=sha256:0df56b056bc17c1b7d6821dfa65216e62bd232d8ab05eb3db44e71d235651471 \ --hash=sha256:0ee6ea481db1ab889cba043ec1eda17bb9c1ea79db6722f779c3667f9f70322f \ @@ -373,7 +372,6 @@ cryptography==48.0.1 \ --hash=sha256:266f4ee051abb2f725b74ef8072b521ce1feacf685a3364fa6a6b45548db791a \ --hash=sha256:2c37f2461406063b417837f5f3daab668652acd82423efcd7f0a9f04be972de1 \ --hash=sha256:32143b24adb918f078134e1e230f1eb8cc04886b92c28b5f0041aaf3e5699225 \ - --hash=sha256:33842cf0888951cef5bc7ac724ab844a42044c1727b967b7f8997289a0464f92 \ --hash=sha256:3752f2dbc8f07a30aad2932c986cea495b03bb554887828225da104f732852b6 \ --hash=sha256:39489bfca54c7a1f6b297efcd8bc608ab92d16c4ca631b0cad4da46724588b24 \ --hash=sha256:3e4a1a3232eef2e6c732827d5722db29a0cc8b27af2a4d865b094cf954be9ca1 \ @@ -383,17 +381,13 @@ cryptography==48.0.1 \ --hash=sha256:48fe40804d4caa2288f24e70ca8c64c42dd826da0ad7e4f1b41b2128d679e6c8 \ --hash=sha256:4ab0a343c807bbcd90c971cd1ecf072937cd01847a9e002bef88fb47ac6be577 \ --hash=sha256:4fdc69f8e4316bcf0c8c8ec1f26f285d12e8142d88d96c876a59a03be3f6ae67 \ - --hash=sha256:6184ca7b174f28d7c703f1290d4b297217c45355f77a98f67e9b7f14549ac54a \ --hash=sha256:66fd0771e7b9c6dcd44cf1120690d2338d16d72795cf40cae2786a39eba65429 \ --hash=sha256:6b2c0c3e6ccf3ade7750f836ef3ee36eea250cc467d45c256895573ac08cc6f1 \ - --hash=sha256:735824ec41b7f74a7c45fb1591349333e4c696cb6c044e5f46356e560143e4cd \ - --hash=sha256:7e234ac052af99f2700826a5c29ea99d9c1b1f80341cde62d11c8154dc8e0bd9 \ --hash=sha256:869c3b8a53bfe27147832df48b32adadf558249d50e76cb3769d40e986b13265 \ --hash=sha256:86be3b1b0b6bf09482fb50a979c508d2950ed95f5621ec77f4e385962006b83a \ --hash=sha256:86fe77abb1bd87afb251d4d02ada7ecf53a32cee9b67d976abb2e45a13297475 \ --hash=sha256:88c852a0ae366e262e5a1744b685e6a433dc8788dd2a277e418bf4904203609d \ --hash=sha256:8ace4507d1e6533c125f4fac754f8bb8b6a74c08e92179dabd7e16571a3efbf3 \ - --hash=sha256:92a46e1d638daa264ba2971c0b0489c9409787943efae4d60ffda3d091ef832c \ --hash=sha256:9621de99d2da096006b629979efd8ae7eb2d8b822488d0c89ee4000c306c59b1 \ --hash=sha256:9a49ca6c81417f6a5edb50375a60cccdd70fa0a91a5211829dbea74eba94d2ac \ --hash=sha256:9bd3f92d76217892b15df84ca256c2c113d386fdda7a7d8691aeeced976507c6 \ @@ -420,9 +414,9 @@ cryptography==48.0.1 \ # google-auth # joserfc # ocotilloapi -dateparser==1.4.1 \ - --hash=sha256:f25d4e051a84be27a35bd297e3e1dc59ff78373701b89be352ba80372d22d0d0 \ - --hash=sha256:f265df13c0380e2e07543ba74b67c0681aaa1096981ffcd35227e1aa0cb81c7c +dateparser==1.3.0 \ + --hash=sha256:5bccf5d1ec6785e5be71cc7ec80f014575a09b4923e762f850e57443bddbf1a5 \ + --hash=sha256:8dc678b0a526e103379f02ae44337d424bd366aac727d3c6cf52ce1b01efbb5a # via pygeofilter dnspython==2.8.0 \ --hash=sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af \ @@ -434,9 +428,9 @@ dnspython==2.8.0 \ dotenv==0.9.9 \ --hash=sha256:29cf74a087b31dafdb5a446b6d7e11cbce8ed2741540e2339c69fbef92c94ce9 # via ocotilloapi -ecdsa==0.19.2 \ - --hash=sha256:62635b0ac1ca2e027f82122b5b81cb706edc38cd91c63dda28e4f3455a2bf930 \ - --hash=sha256:840f5dc5e375c68f36c1a7a5b9caad28f95daa65185c9253c0c08dd952bb7399 +ecdsa==0.19.1 \ + --hash=sha256:30638e27cf77b7e15c4c4cc1973720149e1033827cfd00661ca5c8cc0cdb24c3 \ + --hash=sha256:478cba7b62555866fcb3bb3fe985e06decbdb68ef55713c4e5ab98c57d508e61 # via python-jose email-validator==2.3.0 \ --hash=sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4 \ @@ -454,9 +448,9 @@ fastapi-pagination==0.15.15 \ --hash=sha256:d6e9e4bc4d6e20709dcabc11b16056cd5cd184c995ee214b0190f6b81426fa0c \ --hash=sha256:dc828d7cd15614c650c284bd2c3a98a8a2d9ce340508be3970dc8986908a02aa # via ocotilloapi -filelock==3.29.4 \ - --hash=sha256:10cdb3656fc44541cdf30652a93fb10ec6b05325620eb316bd26893e4201538a \ - --hash=sha256:dac1648087d5115554850d113e7dd8c83ab2d38e3435dde2d4f163847e57b767 +filelock==3.18.0 \ + --hash=sha256:adbc88eabb99d2fec8c9c1b229b171f18afa655400173ddc653d5d01501fb9f2 \ + --hash=sha256:c401f4f8377c4464e6db25fff06205fd89bdd83b65eb0488ed1b160f780e21de # via pygeoapi flask==3.1.3 \ --hash=sha256:0ef0e52b8a9cd932855379197dd8f94047b359ca0a78695144304cb45f87c9eb \ @@ -592,82 +586,53 @@ googleapis-common-protos==1.75.0 \ # google-api-core # ocotilloapi greenlet==3.5.2 \ - --hash=sha256:01e32e9d2b1714a2b06184cb3071ff2a2fd9bc7d065e39198ab21f7253dad421 \ - --hash=sha256:0488ca77c94da5e09d1d9958f98b58cebba1b8fd9664c24898499133de927574 \ - --hash=sha256:049827baab63dda8ab8ec5a6d07fc6eb0f418319cfc757fc8737a605e99ca1ad \ --hash=sha256:0629377725977252159de1ebd3c6e49c170a63856e585446797bb3d66d4d9c34 \ --hash=sha256:09201fa698768db245920b00fdc86ee3e73540f01ca6db162be9632642e1a473 \ --hash=sha256:0977af2df83136f81c1f76e76d4e2fe7d0dc56ea9c101a86af26a95190b9ca32 \ - --hash=sha256:120b77c2a18ebf629c3a7886f68c6d01e065654844ad468f15bb93ace66f2094 \ --hash=sha256:1587ff8b58fdf806993ed1490a06ac19c22d47b219c68b30954380029045d8d4 \ - --hash=sha256:1724499fc08388208408681c53c5062e9803c334e5a0bdaeb616228ba882aac8 \ - --hash=sha256:1adc23c50f22b0f5979521909a8360ab4a3d3bef8b641ce633a04cf1b1c967ea \ --hash=sha256:1c31219badba285858ba8ed117f403dea7fafee6bade9a1991875aae530c3ceb \ --hash=sha256:1d554cd96841a68d464d75a3736f8e87408a7b02b1930a75fa32feb408ad62f8 \ - --hash=sha256:1f052fff492c52fdfa99bd3b3c1389a53de37dae76a0562741417f0d018f02b3 \ --hash=sha256:24c59cb7db9d5c694cb8fd0c76eef8e456b2123afdfa7e4b8f2a67a0860d7682 \ --hash=sha256:26aed8d9503ca78889141a9739d71b383efea5f472a7c522b5410f7eb2a1b163 \ --hash=sha256:2c3b3311af72b3d3b03cc0f1ffd11f072e834be5d0444105cf715fc44434e39c \ - --hash=sha256:2c6d6bfa4fdd7c39a0dbf112cdf28edbd19c517c810eefb6e4e71b0d55933a4c \ --hash=sha256:2debcd0ef9455b7d4879589903efc8e497d4b8fb8c0ae772309e44d1ca5e957f \ --hash=sha256:2ee6288f1933d698b4f098127ed17bda2910a75d2807915bd16294a972055d6c \ - --hash=sha256:30252d191d6959df1d040b559a38fc017139606c5ecc2ad00416557c0355d742 \ --hash=sha256:36cfea2aa075d544617176b2e84450480f0797070ad8799a8c41ada2fe449d32 \ --hash=sha256:3be00501fb4a8c37f6b4b3c4773808ceb26ea65c7ea64fd5735d0f330b3786de \ --hash=sha256:3c2315045f9983e2e50d7e89d95405c21bddb8745f2da4487bc080ab3525f904 \ --hash=sha256:3c417cd6c593bbbef6f7aa31a79f37d3db7d18832fc56b694a2150130bde784e \ --hash=sha256:3dff6cd3aac35f6cd3fc23460105acf576f5faf6c378de0bc088bf37c913864a \ --hash=sha256:423167363c510a75b649f5cd58d873c29498ea03598b9e4b1c3b73e0f899f3d5 \ - --hash=sha256:4e554809538bd4867f24421b43abde170f9c9b8192149b30df5e164bcac6124f \ --hash=sha256:537c5c4f30395020bb9f48f53146070e3b997c3c75da14011ab732aaa19ce3ef \ - --hash=sha256:561dd919c02236a613fbf226791cbd77ee5002cbd5cb7e838869aa3ac7a71e16 \ --hash=sha256:5795e883e915333c0d5648faaa691857fbc7180136883edc377f50f0d509c2a8 \ --hash=sha256:5930d3946ecae99fa7fc0e3f3ae515426ad85058ebd9bfc6c00cca8016e6206b \ - --hash=sha256:5eba55076d79e8a5176e6925295cfb901ebc95dae493342ede22230f75d8bee2 \ --hash=sha256:6d78b5c1c178dad90447f1b8452262709d3eef4c98f825569e74c9d0b2260ac9 \ - --hash=sha256:6d9e19257794e28821c9ebd5e23f86d7c267cd9d390089374f068d2049f949e3 \ --hash=sha256:6e9e49d732ee92a189bb7035e293029244aeba648297a9b856dc733d17ca7f0d \ --hash=sha256:6f1e473c06ae8be00c9034c2bb10fa277b08a93287e3111c395b839f01d27e1f \ --hash=sha256:6f96ed6f4adc1066954ae95f45717657cb67468ef3b89e9a3632e14a625a8f39 \ - --hash=sha256:711028c953cd6ce5dc01bbb5a1747e3ad6bd8b2f7ded73778bb936e8dab9e3b6 \ - --hash=sha256:76dae33e97b52743a19210931ee3e78a88fe1438bc2fc4ee5e7512d289bfad4f \ --hash=sha256:7a7bfc200be40d04961d7e80e8337d726c0c1a50777e588123c3ed8ba731dcb9 \ --hash=sha256:7bb811753703739ad318112f16eccfaabdac050037b6d092debaa8b23566b4ce \ --hash=sha256:7fe6062b1f35534e1e8fb28dfed406cf4eeff3e0bca3a0d9f8ff69f20a4abb00 \ - --hash=sha256:87359c23eb4e8f1b16da68faad29bf5aeb80e3628d7d8e4aa2e41c36879ddedd \ - --hash=sha256:89da99ee8345b458ea2f16831dad31c88ddcdec454b48704d569a0b8fb28f146 \ --hash=sha256:9558cae989faeab6fbb425cd98a0cfa4190a47fba6443973fbee0a1eb0b0b6c3 \ --hash=sha256:98a52d6a50d4deaba304331d83ee3e10ebbdc1517fcca40b2715d1de4534065c \ --hash=sha256:9dc23f0e5ad76415457212a4b947d22ebe4dc80baf02adf7dd5647a90f38bb4e \ - --hash=sha256:9df9daae96848508450011d0d86ed7c95f8829a354ce438284a77b24896fd1f8 \ - --hash=sha256:9e194b996aa1b89d933cfe136e5eb39b22a8b72ba59d376ef39a55bca4dbf47f \ --hash=sha256:a0314aa832c94633355dc6f3ee54f195159533355a323f26926fc63b98b2ccbb \ --hash=sha256:a1759fa4f14c398508cf20dc8037de55cc23ae8bd14c185c2718257837195ca5 \ --hash=sha256:a1789a6244ea1ba61fd4386c9a6a31873e9b0234762103364be98ef87dcb19f3 \ --hash=sha256:a207023f1cf8695fd82580b8099c09c5809be18bc2282362cdfb965dd884a317 \ --hash=sha256:a2ddf9eddc617681108dd071b3feabf3f4a4cd64846254aec4d4ceda098b639a \ - --hash=sha256:a3f76a94e2d6e1fee8f302265679d8cc47d71a203936dd03c6e2ace0f9cfd46d \ - --hash=sha256:a850f6224088ef7dcc70f1a545cb6b3d119c35d6dca63b925b9f35da0635cdad \ --hash=sha256:a9476cbead736dc48ce89e3cd97acff95ecc48cbf21273603a438f9870c4a014 \ --hash=sha256:a96457a30384de52d9c5d2fd33abf6c1daae3db392cd556738f408b1a79a1cf0 \ --hash=sha256:b4ac902af825cbac8e9b2fccab8122236fd2ba6c8b71a080116d2c2ec72671b1 \ --hash=sha256:b4cad42662c796334c2d24607c411e3ed82481c1fb4e1e8ec3a5a8416060092e \ --hash=sha256:b9318cdeb9abdbfdd8bc8464ee4a06dffde2c7846e1def138365a6240ab2c9a5 \ - --hash=sha256:bc18b8d33e6976804b9b792fe11cb3b1fee8b646e8a9e20bf521a429ddf73520 \ - --hash=sha256:bf493b3c1c0a2324c49b0472e2280ba4665f3510d8115f6f807759a6163b15f7 \ --hash=sha256:c0ea4eb3de23f0bac1d75205e10ccfa9b418b17b01a2d7bf19e3b69dda08900a \ --hash=sha256:c1b906220d83c140361cdd12eef970fb5881a168b98ee58a43786426173da14c \ - --hash=sha256:c1c1e5ad80f1f38ea479b83b39dccb20874cfe9ad5e52f87225fa294ba4d39a1 \ --hash=sha256:c674a1dd4fe41f6a93febe7ab366ceabf15080ea31a9307811c56dac5f435f73 \ - --hash=sha256:ca92411942154023c65851e6077d8ca0d00f19de5fa80bb2c6f196ff6c920ba9 \ - --hash=sha256:d7792398872f89466c6671d5d193537eff163ecf7fac78d82e6ddc25017fb4f5 \ --hash=sha256:db548d5ab6c2a8ead82c013f875090d79b5d7d2b67fc513934ce6cf66492ad7f \ --hash=sha256:dbebc038fcdda8f8f21cce985fd04e34e0f42007e7fc7ab7ad285caf77974b95 \ - --hash=sha256:e063263ce9047878480d7e536012fc8b7c8e1922989eb5f03b9ab998a2ee7b7e \ --hash=sha256:e4af5d4961818ab651d09c1448a03b1ba2a1726a076266ebb62330bab9f3238c \ - --hash=sha256:e976f9f6941f57d87a194c91868622c8b22a142a741d2fde31655c319133ade6 \ --hash=sha256:f41feb9f2b59e2e61ac9bea4e344ddd9396bf3cacb2583f73a3595ed7df6f8e7 \ - --hash=sha256:f4d67c1684db3f9782c37ee4bade3f86f5a23a8fcf3f8359224106018ca40728 \ --hash=sha256:f9bbd6216c45a563c2a61e478e038b439d9f248bde44f775ea37d339da643af4 \ --hash=sha256:f9ed777c6891d8253e54468576f55e27f8fc1a662a664f946a191003574c0a74 \ --hash=sha256:feb721811d2754bfd16b48de151dd6b1f222c048e625151f2ca44cfdfd69f59c @@ -751,9 +716,9 @@ mako==1.3.12 \ # via # alembic # ocotilloapi -markdown-it-py==4.2.0 \ - --hash=sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49 \ - --hash=sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a +markdown-it-py==4.0.0 \ + --hash=sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147 \ + --hash=sha256:cb0a2b4aa34f932c007117b194e945bd74e0ec24133ceb5bac59009cda1cb9f3 # via rich markupsafe==3.0.3 \ --hash=sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf \ @@ -904,34 +869,23 @@ numpy==2.5.0 \ --hash=sha256:39a0433bd4086ebd462960cf375e19195bb07b53dc1d87dd5fcf47ad78576f03 \ --hash=sha256:3b94d0d0deceebfad3e67ae5c0e5eb87371e8f7a0581cd04a779928c2450cf1e \ --hash=sha256:44353e2878930039db472b99dc353d749826e4010bd4d2a7f835e94a97a5c748 \ - --hash=sha256:489780423903667933b4ed6197b6ec3b75ea5dd17d1d8f0f38d798feb6921561 \ --hash=sha256:48f54b00711f83a5f796b70c518e8c2b3c5848dda03a54911f23eb68519b9b60 \ --hash=sha256:520e6b8be0a4b65840ac8090d4f51cef4bed66e2b0894d5a520f099adc24a9b2 \ --hash=sha256:5a129578019311b6e56bdd714250f19b518f7dceeeb8d1af5490f4942d3f891c \ --hash=sha256:5dc71423499fab3f46f7a7201155ade1669ea101f2f429d332df9e72f8161731 \ - --hash=sha256:6206db0af545d73d068add6d992279145f158428d1da6cc49adc4b630c5d6ee5 \ --hash=sha256:694d8f74e156f7fd01179f1aa8faa2f648ab6ae0f70b6c3fe57a03249aea2303 \ - --hash=sha256:6f2d6873e2940c860a309d21e25b1e69af6aaffdd80aa056b04c16380db1c4f2 \ --hash=sha256:6f9836778081a0a3c02a6a21493f3e9f5b311f8d2541934f31f05583dc999ea4 \ - --hash=sha256:7174ce8265fc7f7417d171c9ea8fe905220748893ea67a2a7abe726ec331c4b0 \ --hash=sha256:750fb097caf26fa878746d9d119f6f9da12dedcbff1eea966c3e3447647c4a9e \ --hash=sha256:835e454dd99b238cdc5a3f63bce2371296f5ebc53ca1e0f8e6ddbb6d92a29aab \ --hash=sha256:84881d825ca75249b189bbee875fcfe3238aa5c479e6100893cda566e8e86826 \ --hash=sha256:929f0c79ac38bcbd7154fe631dc907abfeddbcc5027a896bd1f7767323271e7a \ --hash=sha256:9990713e9c38154c6861e7547f1e3fc7a87e75ff09bab24ef1cc81d81c2835e9 \ - --hash=sha256:a1a4874217b36d5ac8fc876f52e39df56f8182c88463e9e2dceabf7ca8b7efb8 \ - --hash=sha256:a55e1eb2bca2cfd17a16b213c99dfc8502d47b0d494224d2122277d0400935ca \ - --hash=sha256:aaa760137137e8d3c920d27927748215b56014f92667dc9b6c27dfc61249255a \ - --hash=sha256:b8c3daaf99de52415d20b42f8e8155c78642cb04207d02f9d317a0dcf1b3fb54 \ - --hash=sha256:bf80333980bf37f523341ddd72c783f39d6829ec7736b9eb99086388a2d52cc2 \ - --hash=sha256:c83b664b0e6eee9594fa920cf0639d8af796606d3fad6cc70180c87e4b97c7be \ --hash=sha256:cc4f247a47bbf070bfd70be53ccdcf47b800af563535e7bbe172322197c30e21 \ --hash=sha256:cda12aa4779d42b8771180aba759c96f527d43446d8f380ab59e2b35e8489efd \ --hash=sha256:d371c92cfa09da00022f501ab67fafaea813d752eb30ac44336d45b1e5b0268a \ --hash=sha256:d4313cef1594c5ce46c31b6e54e918338f63f16ee9322304e8c9114d6d81c8bd \ --hash=sha256:e1da54b53e75cd9fcfc23efcc7edab2c6aecf97b6037566d8a0fe804af8ec57c \ --hash=sha256:ebb81d9d5443e0309d6c54894c3fbed74ad7da0714352a67b6d773cd189eae73 \ - --hash=sha256:ece55976ced6bca95a03ae2839e2e5ccffe8eb6a3e7022415645eb154a81e4e6 \ --hash=sha256:edadfbd4794b1086c0d822f81863e8a68fc129d132fd0bb9e31e955d7fbbbdb7 \ --hash=sha256:f27582c55ba4c750b7c58c8faf021d2cd9324a662b466229db8a417b41368af9 \ --hash=sha256:f7e5fa4382967ae6548bd2f174219afb908e294b0d5f625af01166edd5f7d9aa @@ -941,19 +895,19 @@ numpy==2.5.0 \ # pandas-stubs # rasterio # shapely -opentelemetry-api==1.42.1 \ - --hash=sha256:51a69edacadbc03a8950ace1c4c21099cacc538820ac2c9e36277e78cebba714 \ - --hash=sha256:56c63bea9f77b62856be8c47600474acad853b2924b99b1687c4cb6297166716 +opentelemetry-api==1.39.1 \ + --hash=sha256:2edd8463432a7f8443edce90972169b195e7d6a05500cd29e6d13898187c9950 \ + --hash=sha256:fbde8c80e1b937a2c61f20347e91c0c18a1940cecf012d62e65a7caf08967c9c # via # opentelemetry-sdk # opentelemetry-semantic-conventions -opentelemetry-sdk==1.42.1 \ - --hash=sha256:083cd4bbfaa5aa7b5a9e552430d9951219967cfb27aa61feb13a77aba1fc839d \ - --hash=sha256:8c834e8f8c9ba4171d4ec843d0cb8a67e4c7394d3f9e9297e582cbd9456ddbf7 +opentelemetry-sdk==1.39.1 \ + --hash=sha256:4d5482c478513ecb0a5d938dcc61394e647066e0cc2676bee9f3af3f3f45f01c \ + --hash=sha256:cf4d4563caf7bff906c9f7967e2be22d0d6b349b908be0d90fb21c8e9c995cc6 # via apitally -opentelemetry-semantic-conventions==0.63b1 \ - --hash=sha256:3daf963611334b365e98a57438183eb012d3bfb40b2d931a9af613476b8701a9 \ - --hash=sha256:dfe5ef4dee82586b746f522b818ceb298d00b3d59f660042bd79404bff8d0682 +opentelemetry-semantic-conventions==0.60b1 \ + --hash=sha256:87c228b5a0669b748c76d76df6c364c369c28f1c465e50f661e39737e84bc953 \ + --hash=sha256:9fa8c8b0c110da289809292b0591220d3a7b53c1526a23021e977d68597893fb # via opentelemetry-sdk packaging==26.2 \ --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e \ @@ -1269,9 +1223,9 @@ pygeoapi==0.23.4 \ --hash=sha256:7f0fd854575a0da049b64907b56fc0f77ab97768414c1397897e60a0e563438d \ --hash=sha256:935a22761eb0d8736f7b0f2c8384672f5341577509803e35f33f6e78299221ae # via ocotilloapi -pygeofilter==0.4.0 \ - --hash=sha256:cbb4a5f14af0b87e4f0c0c81c659ff64e44351c98e9f61d36af515d896fa8a05 \ - --hash=sha256:ddb74c8233f4fd1b62b80a0ecf4e4f9aff178b8c61334754288d1622c8db71ec +pygeofilter==0.3.3 \ + --hash=sha256:8b9fec05ba144943a1e415b6ac3752ad6011f44aad7d1bb27e7ef48b073460bd \ + --hash=sha256:e719fcb929c6b60bca99de0cfde5f95bc3245cab50516c103dae1d4f12c4c7b6 # via pygeoapi pygeoif==1.6.0 \ --hash=sha256:02f84807dadbaf1941c4bb2a9ef1ebac99b1b0404597d2602efdbb58910c69c9 \ @@ -1289,6 +1243,23 @@ pyjwt==2.13.0 \ --hash=sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423 \ --hash=sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728 # via ocotilloapi +pymssql==2.3.13 \ + --hash=sha256:0fddd24efe9d18bbf174fab7c6745b0927773718387f5517cf8082241f721a68 \ + --hash=sha256:123c55ee41bc7a82c76db12e2eb189b50d0d7a11222b4f8789206d1cda3b33b9 \ + --hash=sha256:16c5957a3c9e51a03276bfd76a22431e2bc4c565e2e95f2cbb3559312edda230 \ + --hash=sha256:1c6d0b2d7961f159a07e4f0d8cc81f70ceab83f5e7fd1e832a2d069e1d67ee4e \ + --hash=sha256:2137e904b1a65546be4ccb96730a391fcd5a85aab8a0632721feb5d7e39cfbce \ + --hash=sha256:30918bb044242865c01838909777ef5e0f1b9ecd7f5882346aefa57f4414b29c \ + --hash=sha256:5c045c0f1977a679cc30d5acd9da3f8aeb2dc6e744895b26444b4a2f20dad9a0 \ + --hash=sha256:7d7037d2b5b907acc7906d0479924db2935a70c720450c41339146a4ada2b93d \ + --hash=sha256:8d66ce0a249d2e3b57369048d71e1f00d08dfb90a758d134da0250ae7bc739c1 \ + --hash=sha256:aa5e07eff7e6e8bd4ba22c30e4cb8dd073e138cd272090603609a15cc5dbc75b \ + --hash=sha256:d663c908414a6a032f04d17628138b1782af916afc0df9fefac4751fa394c3ac \ + --hash=sha256:db77da1a3fc9b5b5c5400639d79d7658ba7ad620957100c5b025be608b562193 \ + --hash=sha256:e053b443e842f9e1698fcb2b23a4bff1ff3d410894d880064e754ad823d541e5 \ + --hash=sha256:fc5482969c813b0a45ce51c41844ae5bfa8044ad5ef8b4820ef6de7d4545b7f2 \ + --hash=sha256:ff5be7ab1d643dbce2ee3424d2ef9ae8e4146cf75bd20946bc7a6108e3ad1e47 + # via ocotilloapi pyparsing==3.3.2 \ --hash=sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d \ --hash=sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc @@ -1369,80 +1340,17 @@ pytz==2026.2 \ # ocotilloapi # pandas # pygeoapi -pyyaml==6.0.3 \ - --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \ - --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \ - --hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \ - --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ - --hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \ - --hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \ - --hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \ - --hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \ - --hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \ - --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ - --hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \ - --hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \ - --hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \ - --hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \ - --hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \ - --hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \ - --hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \ - --hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \ - --hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \ - --hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \ - --hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \ - --hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \ - --hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \ - --hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \ - --hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \ - --hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \ - --hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \ - --hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \ - --hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \ - --hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \ - --hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \ - --hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \ - --hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \ - --hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \ - --hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \ - --hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \ - --hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \ - --hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \ - --hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \ - --hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \ - --hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \ - --hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \ - --hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \ - --hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \ - --hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \ - --hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \ - --hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \ - --hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \ - --hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \ - --hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \ - --hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \ - --hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \ - --hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \ - --hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \ - --hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \ - --hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \ - --hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \ - --hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \ - --hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \ - --hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \ - --hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \ - --hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \ - --hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \ - --hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \ - --hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \ - --hash=sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f \ - --hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \ - --hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \ - --hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \ - --hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \ - --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ - --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ - --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 +pyyaml==6.0.2 \ + --hash=sha256:0ffe8360bab4910ef1b9e87fb812d8bc0a308b0d0eef8c8f44e0254ab3b07133 \ + --hash=sha256:17e311b6c678207928d649faa7cb0d7b4c26a0ba73d41e99c4fff6b6c3276484 \ + --hash=sha256:41e4e3953a79407c794916fa277a82531dd93aad34e29c2a514c2c0c5fe971cc \ + --hash=sha256:50187695423ffe49e2deacb8cd10510bc361faac997de9efef88badc3bb9e2d1 \ + --hash=sha256:68ccc6023a3400877818152ad9a1033e3db8625d899c72eacb5a668902e4d652 \ + --hash=sha256:70b189594dbe54f75ab3a1acec5f1e3faa7e8cf2f1e08d9b561cb41b845f69d5 \ + --hash=sha256:8388ee1976c416731879ac16da0aff3f63b286ffdd57cdeb95f3f2e085687563 \ + --hash=sha256:bc2fa7c6b47d6bc618dd7fb02ef6fdedb1090ec036abab80d4681424b84c1183 \ + --hash=sha256:d584d9ec91ad65861cc08d42e834324ef890a082e591037abe114850ff7bbc3e \ + --hash=sha256:efdca5630322a10774e8e98e1af481aad470dd62c3170801852d752aa7a783ba # via pygeoapi rasterio==1.5.0 \ --hash=sha256:015c1ab6e5453312c5e29692752e7ad73568fe4d13567cbd448d7893128cbd2d \ @@ -1477,121 +1385,72 @@ referencing==0.37.0 \ # via # jsonschema # jsonschema-specifications -regex==2026.5.9 \ - --hash=sha256:002205cafd2a9e78c6290c7d1df277bf3277b3b7a30e0b4bb0dac2e2e3f7cb2d \ - --hash=sha256:01f0f5f55f4b64dacec85dc116d3c05fd23ad3ff037bbc73a2085775953c2611 \ - --hash=sha256:01f28d868834624c934b8d2e0aa1c8341337e37831f4a012f18a5afcba4cbaf3 \ - --hash=sha256:075160bf16658e16d35233300b8453aac25de4cbea808d22348b6979668e924d \ - --hash=sha256:0de5cf193997384ed2ca6f1cd4f78055b255d93d82d5a8cd6ba0d11c10b167e4 \ - --hash=sha256:0e1b1b4e496afbb24f4a62aba855ee4f88f25578927697b340702e48c9ee6bc2 \ - --hash=sha256:0f03aa6898aaaac4592479821df16e68e8d0e29e903e65d8f2dfb2f19028a989 \ - --hash=sha256:0f9eede6a5cbdc02d4978090186390936e1776a7d1359b21e41014c609880bcf \ - --hash=sha256:1268eddd8486dc561d08eee1156e40aa3a8fe10f4bdec8fa653b455fcbffd12c \ - --hash=sha256:15ee42209947f4ca045412eae98416317238163618ace2a8e54f99586a466733 \ - --hash=sha256:164eba9b755ea6f244b0d881196fbc1fac09714e9782c9e2732b813142033c8e \ - --hash=sha256:19c16ceb4a267a8789e25733e583983eeab9f0f8664e66b0bd1c5d21f14c2d4b \ - --hash=sha256:1bd7587a2948b4085195d5a3374eaf4a425dc3e55784c038175355ecf3bbbf8a \ - --hash=sha256:1e6da47d679b7010ef27556b6e0f99771b744936db1792a10ceac6547ae1503e \ - --hash=sha256:205109e96b3cf5adf8f4cd62bedde9487feb282b9497a3535451e5a24cd706a0 \ - --hash=sha256:2099f7e7ff7b6aa3192312650a56e91cc091e49d50b04e4f6f8b6e28b3b27f1c \ - --hash=sha256:246de9d60aa3f8538b519834dd95cbf276ea263d6a7bd5a3666dc3fa0230505b \ - --hash=sha256:24b2355ef5cc9aa5b8f07d17704face1c166fdcc2290fa7bd6e6c925655a8346 \ - --hash=sha256:2a661a7d270a61f7cf460caee8b9fa2d5ef9e5c681234bcb9e0fe14f488e7dfc \ - --hash=sha256:2acfb48634f64996b57f90f39afa692ff362162722581921fe92239a59960f3c \ - --hash=sha256:2efa205e6d98b24d1f3ab395c11aa15cdf10935bca283d0285e0499c284fba21 \ - --hash=sha256:31037c82eccb44b7ea2e9e221d7c01429430e989a1f4b91ea5a855f6017b509a \ - --hash=sha256:3527bb4942d2c14552155406cdedd906567456821848aed1cb4933a391bf5eca \ - --hash=sha256:39617fb0cde9c0e6306dc70e3bfc096f3da793219879f7ae7aa341a69fbdcf6d \ - --hash=sha256:398c521292f4c7fb807001dcd54694d3a1fcafc179a36ad9cc56f98df85930b6 \ - --hash=sha256:3b1e39888c5e0c7d92cea4fc777396c4a90363b05de75d02eb459a4752200808 \ - --hash=sha256:3dd4a3ff360dfb836fecdb93a4598f9d6e2ac81e3e397125145c6221bf58cf4c \ - --hash=sha256:3ddd90103f9e5c471c49c7852ecc1fe27c7e45eb99e977aefe7caa4e779f4f58 \ - --hash=sha256:446ddd671e43ab535810c4b21cff7104945c701d4a14d1e6d1cd6f4e445a8bea \ - --hash=sha256:45375819235558a4ff1c4971dc32881f022613abdb180128f5cb4768c1765a1c \ - --hash=sha256:46f1326ca6e65b0879d23ca302c0f2415aad42ff0309b9c818e7949fe19a41d8 \ - --hash=sha256:48036f6374aaa79eb3b754ec29c61d1c6b1606749d705a13f8854fa2539671f6 \ - --hash=sha256:4ebe8f0b5ec5a5024dc4a4c59f444c4e9afc5f2abdbb8962065b75d27fb971f9 \ - --hash=sha256:4eeb011098fcb77af513dcef521a3dbecbf8849b1e38940759d293b7a93f5026 \ - --hash=sha256:508f56a89ba9cb26e4168cbc37dbd60a28d82430a9e18ad1d25fe0883c314ca2 \ - --hash=sha256:5604dfd046dc37eca90250fc3be938b076c8059fa772ac0ed6f499b0f0fb0415 \ - --hash=sha256:56a33f191f17d8c417f99945ebdc1e691d3af9605d86ec68c7e54a57e3e17af6 \ - --hash=sha256:57e8915c7986aa33d25e4d3629cef711cd2863f2961b10409f0c04cb8b7d9020 \ - --hash=sha256:57eeeb05db7979413dec5438f2db21d7ecbba787cde7a711df1a6f6df672aa06 \ - --hash=sha256:5b73ab8afcf66c622db143d1c6fda4e58e4d537ee4f125229ad47b1ab80f34c0 \ - --hash=sha256:5e41809d2683fcde7d5a8c87a6567ba1fb1ce0de9f31bff578de00a4b2d76daa \ - --hash=sha256:6351571c8a42b505eb555c0dc47d740d0fb66977dc142919eea6f4325b7c56a0 \ - --hash=sha256:6441cc660d76107934a09c22167200839a0e89604a6297f78a974e66e931d2c0 \ - --hash=sha256:65c8c8c37377794bd5b2f3ebe51919042bf17aec802e23c833d89782ed0c78af \ - --hash=sha256:6ba42b2e7e7f46cf68cc6a5ca36fa07959f9bbd9c6bdcc47b6ee76549a590248 \ - --hash=sha256:71b61c5bfe1c806332defc42ad6c780b3c55f661986d7f40283a3a88274b4c00 \ - --hash=sha256:728d8bfd28a8845c8b6bc5dc7ce010453d206396786c0765c2740cb65f37791e \ - --hash=sha256:7b92817338591505f282cf3864c145244b1edcf5381d237038df955001091538 \ - --hash=sha256:7e30b874d341fac767d7df5a0870540541c2c054b80cfaac116e8d367a8a7ff2 \ - --hash=sha256:7e87577720152d2caae19fe2baaf1f8d5ca12091e9e229f03915c37d1e4b9178 \ - --hash=sha256:83d0ee4a57d1c87cb549e195ec300b8f0ec3a82eba66d835e4e2ed8634fe4499 \ - --hash=sha256:8676474c07469d6f33dd1085ca2cd45f65785f32518f2b20e36d9953ca07f994 \ - --hash=sha256:86f40a5d6444db30a125c9c9177e6b25dad981cbc37451fd838f145e6edac92e \ - --hash=sha256:872acc074bd29ffc9913ecdfedf6ea77502312ca44a4aa0d3779089c6069d8de \ - --hash=sha256:8abd33fef90b2a9efac5557d6033ca82d1195ed3a15fea5af15ba7b463c6a63b \ - --hash=sha256:8c6e4218fbdfbcd4f6c19efca40930d24a621bf4b48cb76bc6640543bd28ef20 \ - --hash=sha256:8e76e8161ad00694cfce6767d5dea860c6391ac5b83e5c3a39661e696f11fc7e \ - --hash=sha256:8f3af7a4903c5c04a11a196a5aa75cdd7dd3f8508132f9fb3259d9f5908e3b88 \ - --hash=sha256:91328f1c23d47595ca3ef0a7557fa129c5a23404b775c770697d2f35b33e0107 \ - --hash=sha256:916714069da19329ef7de197dcbc77bb3104145c7c2c864dbfbe318f46b88b14 \ - --hash=sha256:93a7860539414dddaefba2b40f8771765ae17949d4c7182b876ce429e11a8309 \ - --hash=sha256:954cc214c04663ee6d266fc61739cad83054683048de65c5bd1d640ad28098ac \ - --hash=sha256:96f5f58b54a063d7ea9dca08e1cf57bfe10499c4d579ee672da284f57f5f0070 \ - --hash=sha256:97cf3bc1b7d7d2306772ec07366c80d9df00ff79e79cea32898883a646d2fae2 \ - --hash=sha256:98bd73080e8756255137e1bd3f3f00295bbc5aa383c0e0f973920e9134d7c4ad \ - --hash=sha256:992604d02e6d9c6d786c24a706a71ecffe1020fc1ef264044474cd81fa2c3919 \ - --hash=sha256:a24852d3c29ad9e47593593d8a247c44ccc3d0548ef12c822d6ed0810affe676 \ - --hash=sha256:a6a563446a41adc451393dc6b8e6ad87979efaee3c8738690a8d1b08ebead1b4 \ - --hash=sha256:a8234aa23ec39894bfe4a3f1b85616a7032481964a13ac6fc9f10de4f6fca270 \ - --hash=sha256:a8820737949116ffff55fe18f9fc644530063ba6ebfcb8314239416e78f1347c \ - --hash=sha256:a9e1328e17c84c1a5d22ec9f785ecef4a967fab9a42b6a8dc3bcbebd0a0c9e44 \ - --hash=sha256:aa0fbdbac82cb3e4450d0ccde7d7a35607f4cb2dd9fba4b8b69bfaf8c9fa6aed \ - --hash=sha256:b310768746dd314ea6e2ff4cc89ef215426813396ff4e94ee8e6f7096c8b6e03 \ - --hash=sha256:b46b0f094dc1d3b90356c85a0bd2c9bafc4a6a190b9d6f8ddd5a033b6e088ed4 \ - --hash=sha256:b4bb445ff3f725f59df8f6014edb547ee928ec7023a774f6a39a3f953038cbb2 \ - --hash=sha256:b6d189041f15691cfa2b6c4290448ec221244d225b3f5fe9e7771b34ffcdf6e2 \ - --hash=sha256:b96350aa424e79d4fd6b567b344dcbe2b2d6bfc48dfe7717587e1fa6d43da6ff \ - --hash=sha256:be3372b9df6ddecff6486d37e19095a7b4973137caf5512407a89f4455361f41 \ - --hash=sha256:bfe1ce50cbfb569d74e1e4337da6468961f31dbea55fd85aa5de59c0947a805a \ - --hash=sha256:c010eb8caca74bdb40c07498d7ece26b4428fd3f04aa8a72c9ac6f79e8faaac6 \ - --hash=sha256:c8b9b9d294cfea3cd19c718ade7cc93492b2c4991abd9a68d0b3477ae6d8e100 \ - --hash=sha256:c9411dd64ca95477225734a93dfc8583b51916b8d5942f99d6cac21e09965451 \ - --hash=sha256:ca518ed29c46eecba6010b15f1b9a479314d2de409536e71b6a13aa04e3b8a77 \ - --hash=sha256:ccf5249114cc3e772ecdd88a98a86eca0fd74c61ce32a94743758c083fc05d48 \ - --hash=sha256:cd2846168eb9ee3c513902bc8225409cb1caab31d04728b145171fa1625d9621 \ - --hash=sha256:d29eebfc9525db68cad3c97eedd7f754fa265aa5cd0cf4f863b2421e1b48fc9f \ - --hash=sha256:d3d7eb5c9a7f6df82ed3cfac9beb93882a5cbcb5b8b157b56cb2b3b276574ac1 \ - --hash=sha256:d626b84406444b165fc0ba981604edea39f0588ff1f92baa23fe50799ea9afdb \ - --hash=sha256:d641a8c9a61618047796d572a39a79b26167b0411d2c3031937b2fe2d081e2cf \ - --hash=sha256:d659eee77986549c9ea45b861c7567e44d6287c3dc9a4565478853f7b9fe2ff6 \ - --hash=sha256:d6b8a143aca6c39b446ea8092cde25cc8fe9304d4f5fecfbc1a9dbb0282703c2 \ - --hash=sha256:d726ca3f0d76969bf1e8e477d160d3d666bbf999f6860bd314889e5345782046 \ - --hash=sha256:d7bdc0ab8f3dd7e1b4f9ab88634e13374669db86bb3c72e8292f07ae313f539f \ - --hash=sha256:daff2bdbaf1d23e52fdff7c0b7bc2048b68f978df6a4d107ac981f94caef2e66 \ - --hash=sha256:dd2810d22146b6d838acc5ec15602cb6b47920aa4e33015df3868eedfd20bab8 \ - --hash=sha256:ddda5340e6c01a293027dd46232fa79eaff1b48058ce7a98f572b6445b088041 \ - --hash=sha256:dea2e88e1cce4522496cce630e11e67b98b7076620bc4336c3f674bc21a375f4 \ - --hash=sha256:debb893095e944091c16e641a6e33c1b0f4cb61ab945ec5afbf53ce7068834d8 \ - --hash=sha256:dfbe4579b9f08036aa7d101d1835437a20783574ac66327e6b29b4018a138081 \ - --hash=sha256:e1d93bf647916292e8edcec150c07ddf3dc50179ccaf770c04a7f9e452155372 \ - --hash=sha256:e82db382b44d0111b22601c509c89f64434816c9e0eef9d1989cda8cc6ff1c04 \ - --hash=sha256:ea9c8ecfa1b73c73b626534d6626e5340d429630943672b8480724f44e84b962 \ - --hash=sha256:ead4b163ac30a29574510cd4b3e2e985ac5290c05fc7095557d6a5f403fc31b5 \ - --hash=sha256:ecd353045824e4477562a2ac718c25799cdaaa41f7aa925a806a8a3e6848a5b9 \ - --hash=sha256:ed2c9e8068b614c574d8d30e543d617cf5379b0535d46f97ef00e904745a08b5 \ - --hash=sha256:ed457d8e98ae812ed7732bef7bf78de78e834eae0372a74e23ca90ef21d910f9 \ - --hash=sha256:ef31cbfe458e21c6122ba8150ff060e0c7789ed0d26eb423f25472584920b555 \ - --hash=sha256:f079e50a0d3cc3cd5091fa9ff45869a2e6b2cd35895731edafb0327901a8d86d \ - --hash=sha256:f3844f134e834076677dd369976e9f5068679fcb8e50102fdf6b7ac96a3ec127 \ - --hash=sha256:f7a7c26137296beba7784de6eba69c6a93a63ccebc385e4962fe67e267a91225 \ - --hash=sha256:fa411799ca8da32a8d38d020a88faa5b6f91657d284761352940ecf9f7c3bbdd \ - --hash=sha256:fd03c4f0e33280d15cae17159b899245d6b7c53d21def19b263b39655061f5ce \ - --hash=sha256:fd190e88a895a8901325fad284a3f74ea52b1da8525b76cc811fa9b1edf0ce2b \ - --hash=sha256:ff8d372ac2acdc048d1c19916f27ee61bc5722728458ba6ca5052f2c72d51763 +regex==2026.2.19 \ + --hash=sha256:015088b8558502f1f0bccd58754835aa154a7a5b0bd9d4c9b7b96ff4ae9ba876 \ + --hash=sha256:02b9e1b8a7ebe2807cd7bbdf662510c8e43053a23262b9f46ad4fc2dfc9d204e \ + --hash=sha256:03d191a9bcf94d31af56d2575210cb0d0c6a054dbcad2ea9e00aa4c42903b919 \ + --hash=sha256:0d0e72703c60d68b18b27cde7cdb65ed2570ae29fb37231aa3076bfb6b1d1c13 \ + --hash=sha256:11c138febb40546ff9e026dbbc41dc9fb8b29e61013fa5848ccfe045f5b23b83 \ + --hash=sha256:127ea69273485348a126ebbf3d6052604d3c7da284f797bba781f364c0947d47 \ + --hash=sha256:17648e1a88e72d88641b12635e70e6c71c5136ba14edba29bf8fc6834005a265 \ + --hash=sha256:1e7a08622f7d51d7a068f7e4052a38739c412a3e74f55817073d2e2418149619 \ + --hash=sha256:2905ff4a97fad42f2d0834d8b1ea3c2f856ec209837e458d71a061a7d05f9f01 \ + --hash=sha256:294c0fb2e87c6bcc5f577c8f609210f5700b993151913352ed6c6af42f30f95f \ + --hash=sha256:2c1693ca6f444d554aa246b592355b5cec030ace5a2729eae1b04ab6e853e768 \ + --hash=sha256:2f914ae8c804c8a8a562fe216100bc156bfb51338c1f8d55fe32cf407774359a \ + --hash=sha256:2fedd459c791da24914ecc474feecd94cf7845efb262ac3134fe27cbd7eda799 \ + --hash=sha256:311fcccb76af31be4c588d5a17f8f1a059ae8f4b097192896ebffc95612f223a \ + --hash=sha256:3aa0944f1dc6e92f91f3b306ba7f851e1009398c84bfd370633182ee4fc26a64 \ + --hash=sha256:4071209fd4376ab5ceec72ad3507e9d3517c59e38a889079b98916477a871868 \ + --hash=sha256:43cdde87006271be6963896ed816733b10967baaf0e271d529c82e93da66675b \ + --hash=sha256:46e69a4bf552e30e74a8aa73f473c87efcb7f6e8c8ece60d9fd7bf13d5c86f02 \ + --hash=sha256:4a02faea614e7fdd6ba8b3bec6c8e79529d356b100381cec76e638f45d12ca04 \ + --hash=sha256:50f1ee9488dd7a9fda850ec7c68cad7a32fa49fd19733f5403a3f92b451dcf73 \ + --hash=sha256:516ee067c6c721d0d0bfb80a2004edbd060fffd07e456d4e1669e38fe82f922e \ + --hash=sha256:5390b130cce14a7d1db226a3896273b7b35be10af35e69f1cca843b6e5d2bb2d \ + --hash=sha256:5a8f28dd32a4ce9c41758d43b5b9115c1c497b4b1f50c457602c1d571fa98ce1 \ + --hash=sha256:5e3a31e94d10e52a896adaa3adf3621bd526ad2b45b8c2d23d1bbe74c7423007 \ + --hash=sha256:5e56c669535ac59cbf96ca1ece0ef26cb66809990cda4fa45e1e32c3b146599e \ + --hash=sha256:5ec1d7c080832fdd4e150c6f5621fe674c70c63b3ae5a4454cebd7796263b175 \ + --hash=sha256:6380f29ff212ec922b6efb56100c089251940e0526a0d05aa7c2d9b571ddf2fe \ + --hash=sha256:64128549b600987e0f335c2365879895f860a9161f283b14207c800a6ed623d3 \ + --hash=sha256:654dc41a5ba9b8cc8432b3f1aa8906d8b45f3e9502442a07c2f27f6c63f85db5 \ + --hash=sha256:655f553a1fa3ab8a7fd570eca793408b8d26a80bfd89ed24d116baaf13a38969 \ + --hash=sha256:6c8fb3b19652e425ff24169dad3ee07f99afa7996caa9dfbb3a9106cd726f49a \ + --hash=sha256:6fb8cb09b10e38f3ae17cc6dc04a1df77762bd0351b6ba9041438e7cc85ec310 \ + --hash=sha256:7187fdee1be0896c1499a991e9bf7c78e4b56b7863e7405d7bb687888ac10c4b \ + --hash=sha256:74ff212aa61532246bb3036b3dfea62233414b0154b8bc3676975da78383cac3 \ + --hash=sha256:77cfd6b5e7c4e8bf7a39d243ea05882acf5e3c7002b0ef4756de6606893b0ecd \ + --hash=sha256:790dbf87b0361606cb0d79b393c3e8f4436a14ee56568a7463014565d97da02a \ + --hash=sha256:80caaa1ddcc942ec7be18427354f9d58a79cee82dea2a6b3d4fd83302e1240d7 \ + --hash=sha256:8457c1bc10ee9b29cdfd897ccda41dce6bde0e9abd514bcfef7bcd05e254d411 \ + --hash=sha256:8497421099b981f67c99eba4154cf0dfd8e47159431427a11cfb6487f7791d9e \ + --hash=sha256:8abe671cf0f15c26b1ad389bf4043b068ce7d3b1c5d9313e12895f57d6738555 \ + --hash=sha256:8df08decd339e8b3f6a2eb5c05c687fe9d963ae91f352bc57beb05f5b2ac6879 \ + --hash=sha256:8e6e77cd92216eb489e21e5652a11b186afe9bdefca8a2db739fd6b205a9e0a4 \ + --hash=sha256:8edda06079bd770f7f0cf7f3bba1a0b447b96b4a543c91fe0c142d034c166161 \ + --hash=sha256:93d881cab5afdc41a005dba1524a40947d6f7a525057aa64aaf16065cf62faa9 \ + --hash=sha256:997862c619994c4a356cb7c3592502cbd50c2ab98da5f61c5c871f10f22de7e5 \ + --hash=sha256:9cbc69eae834afbf634f7c902fc72ff3e993f1c699156dd1af1adab5d06b7fe7 \ + --hash=sha256:9e6693b8567a59459b5dda19104c4a4dbbd4a1c78833eacc758796f2cfef1854 \ + --hash=sha256:9fff45852160960f29e184ec8a5be5ab4063cfd0b168d439d1fc4ac3744bf29e \ + --hash=sha256:a09ae430e94c049dc6957f6baa35ee3418a3a77f3c12b6e02883bd80a2b679b0 \ + --hash=sha256:a178df8ec03011153fbcd2c70cb961bc98cbbd9694b28f706c318bee8927c3db \ + --hash=sha256:ab780092b1424d13200aa5a62996e95f65ee3db8509be366437439cdc0af1a9f \ + --hash=sha256:b5100acb20648d9efd3f4e7e91f51187f95f22a741dcd719548a6cf4e1b34b3f \ + --hash=sha256:b9ab8dec42afefa6314ea9b31b188259ffdd93f433d77cad454cd0b8d235ce1c \ + --hash=sha256:bcf57d30659996ee5c7937999874504c11b5a068edc9515e6a59221cc2744dd1 \ + --hash=sha256:c0761d7ae8d65773e01515ebb0b304df1bf37a0a79546caad9cbe79a42c12af7 \ + --hash=sha256:c0924c64b082d4512b923ac016d6e1dcf647a3560b8a4c7e55cbbd13656cb4ed \ + --hash=sha256:c13228fbecb03eadbfd8f521732c5fda09ef761af02e920a3148e18ad0e09968 \ + --hash=sha256:c227f2922153ee42bbeb355fd6d009f8c81d9d7bdd666e2276ce41f53ed9a743 \ + --hash=sha256:c7e121a918bbee3f12ac300ce0a0d2f2c979cf208fb071ed8df5a6323281915c \ + --hash=sha256:cce8027010d1ffa3eb89a0b19621cdc78ae548ea2b49fea1f7bfb3ea77064c2b \ + --hash=sha256:d00c95a2b6bfeb3ea1cb68d1751b1dfce2b05adc2a72c488d77a780db06ab867 \ + --hash=sha256:d793c5b4d2b4c668524cd1651404cfc798d40694c759aec997e196fe9729ec60 \ + --hash=sha256:d96162140bb819814428800934c7b71b7bffe81fb6da2d6abc1dcca31741eca3 \ + --hash=sha256:e581f75d5c0b15669139ca1c2d3e23a65bb90e3c06ba9d9ea194c377c726a904 \ + --hash=sha256:ea8dfc99689240e61fb21b5fc2828f68b90abf7777d057b62d3166b7c1543c4c # via dateparser requests==2.34.2 \ --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ @@ -1739,63 +1598,28 @@ sniffio==1.3.1 \ # via ocotilloapi sqlalchemy==2.0.51 \ --hash=sha256:0378d055e9e8cd6ce4d8dff683bdd3d7d413533c4ee51d67a2b1e0f9eacc0f23 \ - --hash=sha256:0592bdadf86ddcabfd72d9ab66ea8a5d8d2cc6be1cc51fa7e66c03868ac5eac1 \ --hash=sha256:08a204d8b5638717c26a24df18fcf40af45a6b22e35b70b1d62f0113c2e278e8 \ --hash=sha256:0c2c62877097e1a0db401fba5cb4debee33265e5b2a55c4ccb489c02c53b4f72 \ - --hash=sha256:0e8203d2fbd5c6254692ef0a72c740d75b2f3c7ca345404f4c1a4604813c77c0 \ --hash=sha256:0f053118c30e53161857a953e4de667d90e274980dccbe5dd3829bbbeece72a5 \ --hash=sha256:0f6bcad487aee1c638d707235682fc96f741de00663619881ab235400d03289e \ - --hash=sha256:111604e637da87031255ddc26c7d7bc22bc6af6f5d459ccff3af1b4660233a85 \ - --hash=sha256:1181256e0f16479691b5616d36375dc2620ad8332b25978763c3d206ad3f3f1d \ --hash=sha256:159bb6ba32059f57ad7375a8f50d844dd2f19d14954ecf820cd33e20debd46b2 \ - --hash=sha256:1aa10c0daee6705294d181daadaa793221e1a59ed55000a3fab1d42b088ce4ba \ - --hash=sha256:1af05726b3d0cdba1c55284bf408fd3b792e690fe2399bfb8304565551cda652 \ --hash=sha256:1bed1ee8b01da6088210aa9412023326fb98a599ba502e6118308601dcbef77f \ - --hash=sha256:1d21ce524ab86c23046e992a5b81cb54c21079c6df6e78b8fc77d77cac70a6b9 \ - --hash=sha256:1e47b1199c2e832e325eacabc8d32d2487f58c9358f97e9a00f5eb93c5680d84 \ - --hash=sha256:247acaa29ccef6250dfd6a3eedf8f94ddf23564180a39fe362e32ae9dbdbde46 \ - --hash=sha256:2a97eaad21c84b4ef8010b11eeba9fe6153eb0b3df3ff8b6abc309df1b978ef7 \ - --hash=sha256:2cf39aabdf48e87c1c2c2ed6d20d33ffa0733b3071ce9c5f66357947dd009080 \ - --hash=sha256:2e54ff2dd657f2e3e0fbf2b097db1182f7bfea263eca4353f00065bae2a67c3d \ --hash=sha256:39a76529db6305693d8d4affa58ad5b5e2e18edd62daea628b29b97930b3513d \ --hash=sha256:4004ada0aafe8ae1991b2cd1d99c6d9146126e123bd6f883c260d974aa012e54 \ - --hash=sha256:436728ce18a80f6951a1e11cc6112c2ede9faf20766f1a26195a7c441ca12dbd \ --hash=sha256:483b11bd46bf35fc14c52faf338b04300c9e6ce554bce9b11be85bfec3bc3195 \ - --hash=sha256:4a011ea4510683319ce4ed274b56ee05194b39b6da9d09ca7a39388f0fa84dcc \ - --hash=sha256:581921d849d6e6f994d560389192955e80e2950e18fcdfe2ccea863e01158e6e \ - --hash=sha256:59cab3686b1bc039dd9cded2f8d0c08a246e84e76bd4ab5b4f18c7cdae293825 \ - --hash=sha256:6b588fd681ddf0c196b8df1ea49a8913514894b2b8f945a9511b4b48871f99c8 \ --hash=sha256:6e46fc36029eff666391e0531e5387b62ce6c4f1d8e50b3fb3099eaca1b42522 \ --hash=sha256:6ea306caaae6bd5afd0a46050003c88f6bf33227377a49298c498c3cb88ff491 \ --hash=sha256:72ca54c952107ba5cd58854b67a5a6268631289d21651a1235396f3b98b47400 \ - --hash=sha256:740cf6f35351b1ac3d82369152acf1d51d37e3dcf85d4dc0a22ca01410eabe2a \ --hash=sha256:7c2056838b6685b72fdb36c99996cf862753461a62f2e84f4196371d3b2d6a07 \ --hash=sha256:7c6b36ed71f41942bdcd2ad2522be46bfce09d5705be5640ecf19bbc7660e4b7 \ - --hash=sha256:7d78702b26ba1c18b2d0fb2ea940ba7f17a9581b42e8361ff93920ebbee1235a \ --hash=sha256:804dccd8a4a6242c4e30ad961e540e18a588f6527202f2d6791b01845d59fdc9 \ --hash=sha256:9161cfc9efce70d1715f47d6ff40f79c6778c00d53be4fbc09d70301e4b83ba7 \ --hash=sha256:96747bfbadb055466e5b46d572618170046b45ce5a4879167f50d70a5319a499 \ - --hash=sha256:9f380393be5abeb6815f68fd39271b95127173511b6706b0a630a9995d53f8f5 \ - --hash=sha256:a42ad6afcbaaa777241e347aa2e29155993045a0d6b7db74da61053ffe875fe0 \ - --hash=sha256:a5b2ed6d828f1f09bd812861f4f59ca3bc3803f9df871f4555187f0faf018604 \ - --hash=sha256:a6d26094615306d116dd5e4a51b0304c99dd2356fc569eed6922a80a6bd3b265 \ - --hash=sha256:aa18ae738b5170e253ad0bb6c4b0f07585081e8a6e50893e4d911d47b39a0904 \ - --hash=sha256:ad30ae663711786303fbcd46a47516302d201ee49a877cb3fac61f672895110a \ - --hash=sha256:b21f0e7efc7a5c509e953784e9d1575ebb8b4318960e7e7d7a93bb803626cf64 \ --hash=sha256:b3e693d15533a45cd5906f0589f9c35090bef6ef45bf1e8195c424aa0ae06a8d \ - --hash=sha256:b7f08588854bbb724041d9ae9d980d40040c922382e1d9a2ecb390edc4fd5032 \ --hash=sha256:b93ab07b5292dbe7e6b8da89475275e7042744283921344b56105f3eeb0f828b \ --hash=sha256:bb024d8b621d0be75f4f44ecc7c950450026e76d66dc8f791bb5331d7fed59d5 \ - --hash=sha256:bb1f5062f98b0b3290e72b707747fdd7e0f22d6956b236ba7ca7f5c9971d2da2 \ --hash=sha256:c45a496d6bc05dec41dcd4c3a2b183723f47473255c159cd80b503c8f246424d \ - --hash=sha256:c5d98a2709840027f5a347c3af0a7c3d5f6c1ff93af2ca1c54494e23cba8f389 \ - --hash=sha256:c68568f3facf8f66fa76c60e0ced69b67666ffa9941d1d0a3756fda196049080 \ - --hash=sha256:c95ef01f53233a305a874a44a63fbfb1d81cd79b49de0f8529b3548cde437e37 \ - --hash=sha256:ca216e8af5c05e326efc7e28716ac2381a7cf9791749f5ee1849dccdc99c9b00 \ - --hash=sha256:ca8435d13829b92f4a97362d91975154a4015db3a2634154e1754e9a915e6b86 \ - --hash=sha256:dc261707bf5739aea8a541593f3cc1d463c2701fb05fbcbba0ce031b69a21260 \ - --hash=sha256:e5ea1a213be1fcd5e49d9904c3b9939211ded90bc2a64e93f4c01963474285de \ - --hash=sha256:fa268106c8987639a17a18514cfe0cd9bf17420ab887e1e1bf486da8836135b1 + --hash=sha256:e5ea1a213be1fcd5e49d9904c3b9939211ded90bc2a64e93f4c01963474285de # via # alembic # geoalchemy2 @@ -1818,6 +1642,10 @@ sqlalchemy-utils==0.42.1 \ # via # ocotilloapi # sqlalchemy-searchable +sqlparse==0.5.5 \ + --hash=sha256:12a08b3bf3eec877c519589833aed092e2444e68240a3577e8e26148acc7b1ba \ + --hash=sha256:e20d4a9b0b8585fdf63b10d30066c7c94c5d7a7ec47c889a2d83a3caa93ff28e + # via ocotilloapi starlette==1.3.1 \ --hash=sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0 \ --hash=sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6 @@ -1872,9 +1700,9 @@ tzdata==2025.3 \ # ocotilloapi # pandas # tzlocal -tzlocal==5.4.3 \ - --hash=sha256:24ce97bb58e2a973f7640ec2553ab4e6f6d5a0d0d1aa9dc43bca21d89e1feb82 \ - --hash=sha256:3a8c9bc18cf47e1dcde252ea0e6a72a6cde320a400b6ac6db1f1f8cccd553c00 +tzlocal==5.3.1 \ + --hash=sha256:cceffc7edecefea1f595541dbd6e990cb1ea3d19bf01b2809f362a03dd7921fd \ + --hash=sha256:eb1a66c3ef5847adf7a834f1be0800581b683b5608e74f86ecbcef8ab91bb85d # via dateparser urllib3==2.7.0 \ --hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \ @@ -1891,9 +1719,9 @@ uvicorn==0.49.0 \ --hash=sha256:ba3d14c3ee7e41c6c654c46c9eb489d33213cdd30aa1696eab1374337c13f68f \ --hash=sha256:ebf4271aa580d9de97f93192d4595176df6e91f9aae919ca73e4fc07df1e66a3 # via ocotilloapi -werkzeug==3.1.8 \ - --hash=sha256:63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50 \ - --hash=sha256:9bad61a4268dac112f1c5cd4630a56ede601b6ed420300677a869083d70a4c44 +werkzeug==3.1.6 \ + --hash=sha256:210c6bede5a420a913956b4791a7f4d6843a43b6fcee4dfa08a65e93007d0d25 \ + --hash=sha256:7ddf3357bb9564e407607f988f683d72038551200c704012bb9a4c523d42f131 # via flask yarl==1.24.2 \ --hash=sha256:0063adad533e57171b79db3943b229d40dfafeeee579767f96541f106bac5f1b \ diff --git a/tests/test_nmw_mirror.py b/tests/test_nmw_mirror.py new file mode 100644 index 00000000..a6a78586 --- /dev/null +++ b/tests/test_nmw_mirror.py @@ -0,0 +1,329 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""Structural + unit tests for the NM_Wells Phase-1 staging mirror. + +Covers SPEC §V invariants for the mirror schema, migrations and OGC views: + + V1 - all 18 NMW_* mirror tables exist with a primary key + V2 - mirror load order respects parent->child (FK parent precedes child) + V3 - migrations build all 8 OGC views + V5 - the temperature-profile OGC view is MATERIALIZED + V6 - each geothermal pygeoapi collection maps to an existing DB relation + V10 - FK enforcement lives in the migration (DB-level FK constraints exist) + +Full data round-trip against a real SQL dump is out of scope here (SPEC §T.T14); +the dump parser is unit-tested directly instead. +""" + +import os + +import pytest +import yaml +from sqlalchemy import inspect as sa_inspect, text + +from db.engine import engine, session_ctx +from transfers.nmw_mirror_transfer import NMW_MIRROR_SPECS +from transfers.nmw_sql_dump import _parse_value, iter_table_rows + +ROOT = os.path.dirname(os.path.dirname(__file__)) + +# DB relations created by the OGC-view migrations (d1e2f3a4b5c6, e2f3a4b5c6d7). +OGC_VIEWS = [ + "ogc_geothermal_wells_bht", + "ogc_geothermal_wells_temperature_profile", # MATERIALIZED + "ogc_geothermal_wells_summary_heat_flow", + "ogc_geothermal_wells_interval_heat_flow", + "ogc_bht_measurements", + "ogc_temp_depth_measurements", + "ogc_heat_flow", + "ogc_dst", +] +MATERIALIZED_VIEW = "ogc_geothermal_wells_temperature_profile" + +# pygeoapi collections added by this PR and the DB relation each is backed by. +GEOTHERMAL_COLLECTIONS = { + "geothermal_wells_bht": "ogc_geothermal_wells_bht", + "geothermal_wells_temperature_profile": "ogc_geothermal_wells_temperature_profile", + "bht_measurements": "ogc_bht_measurements", + "temp_depth_measurements": "ogc_temp_depth_measurements", + "heat_flow": "ogc_heat_flow", + "dst": "ogc_dst", +} + + +def _mirror_tablenames() -> list[str]: + return [spec.model.__tablename__ for spec in NMW_MIRROR_SPECS] + + +# --------------------------------------------------------------------------- V1 +def test_all_mirror_tables_present_with_pk(): + """All 18 NMW_* mirror tables exist in the schema, each with a PK (V1).""" + names = _mirror_tablenames() + assert len(names) == 18, f"expected 18 mirror specs, got {len(names)}" + + insp = sa_inspect(engine) + existing = set(insp.get_table_names()) + for table in names: + assert table in existing, f"mirror table {table} missing from schema" + pk = insp.get_pk_constraint(table)["constrained_columns"] + assert pk, f"mirror table {table} has no primary key" + + +def test_well_headers_pk_is_well_data_id(): + """Spot-check that original SQL Server column names are preserved (V1).""" + insp = sa_inspect(engine) + pk = insp.get_pk_constraint("NMW_WellHeaders")["constrained_columns"] + assert pk == ["WellDataID"] + + +# ----------------------------------------------------------------------- V2/V10 +def test_mirror_tables_have_fk_constraints(): + """The migration creates DB-level FK constraints (V10) - at least one + child table must carry a foreign key.""" + insp = sa_inspect(engine) + total_fks = sum(len(insp.get_foreign_keys(t)) for t in _mirror_tablenames()) + assert total_fks > 0, "no FK constraints found on NMW_* mirror tables" + + +def test_fk_parent_loads_before_child(): + """Every FK parent table is loaded before its child in NMW_MIRROR_SPECS so + the parent row exists when the child is inserted (V2).""" + order = {name: i for i, name in enumerate(_mirror_tablenames())} + insp = sa_inspect(engine) + checked = 0 + for child in order: + for fk in insp.get_foreign_keys(child): + parent = fk["referred_table"] + if parent not in order or parent == child: + continue # self-ref or FK to a non-mirror table + checked += 1 + assert order[parent] <= order[child], ( + f"{parent} (parent) must load before {child} (child) " + f"in NMW_MIRROR_SPECS" + ) + assert checked > 0, "expected at least one intra-mirror FK to validate" + + +# --------------------------------------------------------------------------- V3 +def test_ogc_views_exist(): + """All 8 OGC views built by the migrations exist as relations (V3).""" + with session_ctx() as session: + rows = session.execute( + text( + "SELECT table_name FROM information_schema.tables " + "WHERE table_schema = 'public' " + "UNION SELECT matviewname FROM pg_matviews WHERE schemaname = 'public'" + ) + ).all() + relations = {r[0] for r in rows} + for view in OGC_VIEWS: + assert view in relations, f"OGC view {view} missing" + + +# --------------------------------------------------------------------------- V5 +def test_temperature_profile_is_materialized(): + """The temperature-profile view is MATERIALIZED so it can be refreshed + after a mirror load (V5).""" + with session_ctx() as session: + names = { + r[0] + for r in session.execute( + text("SELECT matviewname FROM pg_matviews WHERE schemaname = 'public'") + ).all() + } + assert MATERIALIZED_VIEW in names, f"{MATERIALIZED_VIEW} is not a materialized view" + + +# --------------------------------------------------------------------------- V6 +class _Default(dict): + """format_map() helper: unknown placeholders render empty.""" + + def __missing__(self, key): # noqa: D401 + return "" + + +def _load_pygeoapi_config() -> dict: + """pygeoapi-config.yml is a ``{placeholder}`` template (see core/pygeoapi.py + _write_config); substitute dummy values before parsing as YAML.""" + raw = open(os.path.join(ROOT, "core", "pygeoapi-config.yml")).read() + rendered = raw.format_map( + _Default( + server_url="http://test", + postgres_host="h", + postgres_port="5432", + postgres_db="d", + postgres_user="u", + postgres_password_env="p", + thing_collections_block="", + ) + ) + return yaml.safe_load(rendered) + + +def test_geothermal_collections_back_existing_relations(): + """Each new geothermal pygeoapi collection points at a DB relation that + actually exists (V6).""" + cfg = _load_pygeoapi_config() + resources = cfg["resources"] + + with session_ctx() as session: + rows = session.execute( + text( + "SELECT table_name FROM information_schema.tables " + "WHERE table_schema = 'public' " + "UNION SELECT matviewname FROM pg_matviews WHERE schemaname = 'public'" + ) + ).all() + relations = {r[0] for r in rows} + + for coll, expected_table in GEOTHERMAL_COLLECTIONS.items(): + assert coll in resources, f"collection {coll} missing from pygeoapi config" + tables = { + p.get("table") + for p in resources[coll].get("providers", []) + if p.get("table") + } + assert ( + expected_table in tables + ), f"collection {coll} should be backed by {expected_table}, got {tables}" + assert ( + expected_table in relations + ), f"backing relation {expected_table} for {coll} does not exist in DB" + + +# ----------------------------------------------------------------- dump parser +@pytest.mark.parametrize( + "raw,expected", + [ + ("NULL", None), + ("null", None), + ("123", 123), + ("-5", -5), + ("-1.5", -1.5), + ("'abc'", "abc"), + ("N'abc'", "abc"), + ("'O''Brien'", "O'Brien"), # doubled '' unescaped + ("CAST(42 AS int)", 42), + ("CAST(N'x' AS nvarchar(10))", "x"), + ("0xDEADBEEF", None), # binary / rowversion not mirrored + ], +) +def test_parse_value_coercion(raw, expected): + assert _parse_value(raw) == expected + + +def test_iter_table_rows_parses_inserts(tmp_path): + """iter_table_rows decodes column/value pairs from SSMS INSERT statements.""" + dump = tmp_path / "dump.sql" + dump.write_text( + "INSERT [dbo].[tbl_demo] ([OBJECTID], [Name], [Note]) " + "VALUES (1, N'alpha', NULL), (2, 'beta', CAST(N'c' AS nvarchar(1)));\n", + encoding="utf-8", + ) + rows = list(iter_table_rows(str(dump), "tbl_demo")) + assert rows == [ + {"OBJECTID": 1, "Name": "alpha", "Note": None}, + {"OBJECTID": 2, "Name": "beta", "Note": "c"}, + ] + + +# ------------------------------------------------------ regression: FK truncate +def test_truncate_referenced_parent_needs_cascade(): + """A bare TRUNCATE of an FK-referenced mirror parent is rejected; the loader + must use CASCADE (V13 / B2). Guards the dump-load reload path.""" + import sqlalchemy.exc + + # Bare truncate of the FK-referenced parent should error... + with session_ctx() as session: + with pytest.raises(sqlalchemy.exc.DBAPIError): + session.execute(text('TRUNCATE TABLE "NMW_WellHeaders"')) + # ...while CASCADE (what the loader does) succeeds. + with session_ctx() as session: + session.execute(text('TRUNCATE TABLE "NMW_WellHeaders" CASCADE')) + session.commit() + + +# --------------------------------------------- regression: one feature per well +def test_bht_view_one_feature_per_well_with_duplicate_locations(): + """Two location rows for one WellDataID must not multiply BHT features or + inflate bht_count; the view dedups locations via DISTINCT ON (V14 / B3).""" + wid = "11111111-1111-1111-1111-111111111111" + rsid = "22222222-2222-2222-2222-222222222222" + ssid = "33333333-3333-3333-3333-333333333333" + bht = "44444444-4444-4444-4444-444444444444" + with session_ctx() as session: + session.execute( + text('INSERT INTO "NMW_WellHeaders" ("WellDataID") VALUES (:w)'), + {"w": wid}, + ) + # TWO location rows, same WellDataID, valid coords (the bug trigger). + session.execute( + text( + 'INSERT INTO "NMW_WellLocations" ' + '("OBJECTID","WellDataID","Lat_dd83","Long_dd83") VALUES ' + "(901,:w,33.0,-107.0),(902,:w,33.0,-107.0)" + ), + {"w": wid}, + ) + session.execute( + text( + 'INSERT INTO "NMW_WellRecords" ("RecrdSetID","WellDataID") ' + "VALUES (:r,:w)" + ), + {"r": rsid, "w": wid}, + ) + session.execute( + text( + 'INSERT INTO "NMW_WellSamples" ("SamplSetID","RecrdsetID") ' + "VALUES (:s,:r)" + ), + {"s": ssid, "r": rsid}, + ) + session.execute( + text( + 'INSERT INTO "NMW_GtBhtHeaders" ("BHTGUID","SamplSetID") ' + "VALUES (:b,:s)" + ), + {"b": bht, "s": ssid}, + ) + session.execute( + text( + 'INSERT INTO "NMW_GtBhtData" ("OBJECTID","BHTGUID","BHT","Depth") ' + "VALUES (911,:b,150.0,1000.0)" + ), + {"b": bht}, + ) + session.commit() + try: + with session_ctx() as session: + rows = session.execute( + text( + 'SELECT bht_count FROM "ogc_geothermal_wells_bht" ' + "WHERE well_data_id = :w" + ), + {"w": wid}, + ).all() + assert len(rows) == 1, f"expected one feature per well, got {len(rows)}" + assert ( + rows[0][0] == 1 + ), f"bht_count inflated by duplicate locations: {rows[0][0]}" + finally: + with session_ctx() as session: + session.execute(text('TRUNCATE TABLE "NMW_WellHeaders" CASCADE')) + session.commit() + + +# ============= EOF ============================================= diff --git a/transfers/export_nmw_csvs.py b/transfers/export_nmw_csvs.py new file mode 100644 index 00000000..51daf403 --- /dev/null +++ b/transfers/export_nmw_csvs.py @@ -0,0 +1,89 @@ +"""Export NM_Wells SQL Server tables to CSV files for the transfer pipeline. + +Connects to the NM_Wells SQL Server database and exports each source table to +transfers/data/nma_csv_cache/
.csv, which is where nmw_mirror_transfer.py +looks for them when NMW_SQL_DUMP is not set. + +Usage: + uv run python -m transfers.export_nmw_csvs + +Required environment variables (add to .env): + NMW_HOST SQL Server hostname or IP + NMW_PORT SQL Server port (default: 1433) + NMW_USER SQL Server username + NMW_PASSWORD SQL Server password + NMW_DATABASE Database name (default: NM_Wells) +""" + +import os +from pathlib import Path + +import pymssql +from dotenv import load_dotenv + +from transfers.nmw_mirror_transfer import NMW_MIRROR_SPECS + +load_dotenv(override=False) + +TABLES = [spec.source_table for spec in NMW_MIRROR_SPECS] + +_data_root = os.environ.get("TRANSFERS_DATA_DIR") +OUT_DIR = ( + Path(_data_root) if _data_root else Path(__file__).parent / "data" / "nma_csv_cache" +) + + +def _get_connection(): + host = os.environ["NMW_HOST"] + port = int(os.environ.get("NMW_PORT", 1433)) + user = os.environ["NMW_USER"] + password = os.environ["NMW_PASSWORD"] + database = os.environ.get("NMW_DATABASE", "NM_Wells") + return pymssql.connect( + server=host, + port=port, + user=user, + password=password, + database=database, + ) + + +def export_table(cursor, table: str, out_path: Path) -> int: + cursor.execute(f"SELECT * FROM dbo.{table}") + columns = [desc[0] for desc in cursor.description] + rows = cursor.fetchall() + + import csv + + with out_path.open("w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(columns) + writer.writerows(rows) + + return len(rows) + + +def main(): + OUT_DIR.mkdir(parents=True, exist_ok=True) + print( + f"Connecting to {os.environ.get('NMW_HOST')} / {os.environ.get('NMW_DATABASE', 'NM_Wells')}" + ) + conn = _get_connection() + cursor = conn.cursor() + + for table in TABLES: + out_path = OUT_DIR / f"{table}.csv" + print(f" Exporting {table}...", end=" ", flush=True) + try: + n = export_table(cursor, table, out_path) + print(f"{n} rows -> {out_path.name}") + except Exception as e: + print(f"FAILED: {e}") + + cursor.close() + conn.close() + print("Done.") + + +if __name__ == "__main__": + main() diff --git a/transfers/nmw_mirror_transfer.py b/transfers/nmw_mirror_transfer.py new file mode 100644 index 00000000..d59ef4ea --- /dev/null +++ b/transfers/nmw_mirror_transfer.py @@ -0,0 +1,385 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""Load the NM_Wells SQL dump into the ``NMW_*`` 1:1 staging mirror tables. + +Phase 1 of the NM_Wells migration (see db/nmw_legacy.py and +docs/nm_wells-migration.md). This is a faithful copy: each source table's CSV +export is read and its rows are inserted into the matching ``NMW_*`` mirror +model with NO transformation beyond type coercion. The Phase 2 transform into +the Ocotillo model is separate. + +Generic + data-driven: one ``MirrorSpec`` per (model, source table). Column +handling is derived from each model's ``__table__`` metadata, so adding a new +mirror table requires only a model + a spec entry (no per-table code). + +Two row sources, selected at runtime: + +1. **SQL Server data dump** (preferred): set ``NMW_SQL_DUMP`` to a ``.sql`` file + of ``INSERT [dbo].[tbl_*] (...) VALUES (...)`` statements. Each table is + written to a CSV by ``transfers.nmw_sql_dump.write_table_csv`` (sqlparse) and + bulk-loaded with Postgres ``COPY ... FROM STDIN`` (truncate + COPY; Postgres + casts text -> column types). CSV output dir defaults to a temp dir, override + with ``NMW_CSV_DIR``. +2. **CSV exports** (fallback when ``NMW_SQL_DUMP`` is unset): per-table CSVs read + with ``transfers.util.read_csv`` (``transfers/data/nma_csv_cache/
.csv`` + then GCS ``nma_csv/
.csv``), inserted row-by-row with type coercion. + +In both cases the source column names are the original SQL Server names +(OBJECTID, WellDataID, ...), which match the mirror columns' DB names exactly. + +Idempotent: rows upsert via ``INSERT ... ON CONFLICT () DO NOTHING``. +""" + +import itertools +import os +import tempfile +import uuid +from dataclasses import dataclass + +import pandas as pd +from sqlalchemy import DateTime, Float, Integer, LargeBinary, SmallInteger, String, text +from sqlalchemy.dialects.postgresql import UUID +from sqlalchemy.dialects.postgresql import insert as pg_insert +from sqlalchemy.orm import Session + +from db.nmw_legacy import ( + NMW_GtBhtData, + NMW_GtBhtHeaders, + NMW_GtConductivity, + NMW_GtHeatFlow, + NMW_GtSumHeatFlow, + NMW_GtTempDepths, + NMW_Sources, + NMW_WellHeaders, + NMW_WellLocations, + NMW_WellRecords, + NMW_WellSamples, + NMW_WellZDatum, + NMW_WsDstFlowHistory, + NMW_WsDstFluidProperties, + NMW_WsDstHeaders, + NMW_WsDstIntervals, + NMW_WsDstPressure, + NMW_WsIntervals, +) +from transfers.logger import logger +from transfers.nmw_sql_dump import iter_table_rows, write_table_csv +from transfers.util import read_csv + +# Path to a SQL Server data-dump .sql file. When set, rows are parsed from it; +# otherwise the loader falls back to per-table CSV exports. +_SQL_DUMP_ENV = "NMW_SQL_DUMP" +# Optional output dir for the per-table CSVs written from the dump (COPY path). +# Defaults to a fresh temp dir. +_CSV_DIR_ENV = "NMW_CSV_DIR" +# pg8000 encodes the parameter count as an unsigned short (max 65535). +# Keep chunk size below that ceiling based on actual column count per table. +_MAX_PG8000_PARAMS = 65535 +_CHUNK_SIZE = 2000 + +# Materialized OGC views over the geothermal mirror that need a REFRESH after a +# (re)load. Regular views reflect the tables live and need no refresh. +_MATERIALIZED_VIEWS = ("ogc_geothermal_wells_temperature_profile",) + + +@dataclass +class MirrorSpec: + """Maps a mirror model to its NM_Wells source CSV/table name.""" + + model: type + source_table: str + + +# All NMW_* mirror tables. Order is irrelevant (no enforced cross-table FKs in +# the staging layer), but parents are listed before children for readability. +NMW_MIRROR_SPECS: list[MirrorSpec] = [ + # Parents before children (FK constraints enforced on insert) + # Root + MirrorSpec(NMW_WellHeaders, "tbl_well_headers"), + # -> WellHeaders + MirrorSpec(NMW_WellLocations, "tbl_well_locations"), + MirrorSpec(NMW_WellRecords, "tbl_well_records"), + # -> WellRecords + MirrorSpec(NMW_WellZDatum, "tbl_well_z_datum"), + MirrorSpec(NMW_WellSamples, "tbl_well_samples"), + # Publications (standalone, no FK) + MirrorSpec(NMW_Sources, "tbl_sources"), + # Geothermal -> WellSamples + MirrorSpec(NMW_GtBhtHeaders, "tbl_gt_bht_headers"), + MirrorSpec(NMW_GtBhtData, "tbl_gt_bht_data"), # -> GtBhtHeaders + MirrorSpec(NMW_WsIntervals, "tbl_ws_intervals"), + MirrorSpec(NMW_GtConductivity, "tbl_gt_conductivity"), # -> WsIntervals + MirrorSpec(NMW_GtHeatFlow, "tbl_gt_heat_flow"), # -> WsIntervals + MirrorSpec( + NMW_GtSumHeatFlow, "tbl_gt_sum_heat_flow" + ), # -> WellRecords + WellSamples + MirrorSpec(NMW_GtTempDepths, "tbl_gt_temp_depths"), # -> WellSamples + # Drill Stem Tests -> WellSamples + MirrorSpec(NMW_WsDstHeaders, "tbl_ws_dst_headers"), + MirrorSpec(NMW_WsDstIntervals, "tbl_ws_dst_intervals"), # -> WsDstHeaders + MirrorSpec(NMW_WsDstFlowHistory, "tbl_ws_dst_flow_history"), # -> WsDstIntervals + MirrorSpec( + NMW_WsDstFluidProperties, "tbl_ws_dst_fluid_properties" + ), # -> WsDstIntervals + MirrorSpec(NMW_WsDstPressure, "tbl_ws_dst_pressure"), # -> WsDstIntervals +] + + +def _coerce(value, col_type): + """Coerce a single cell to the Python value for ``col_type`` (or None). + + Treats NaN/NaT as None. (pandas keeps NaN/NaT in typed columns even after a + ``.where(notnull, None)``, so the missing-value check must happen here.) + """ + if value is None: + return None + try: + if pd.isna(value): + return None + except (TypeError, ValueError): + pass # non-scalar / unhashable: fall through and coerce normally + if isinstance(col_type, UUID): + if isinstance(value, uuid.UUID): + return value + try: + return uuid.UUID(str(value).strip()) + except (ValueError, AttributeError, TypeError): + return None + if isinstance(col_type, (Integer, SmallInteger)): + try: + return int(value) + except (ValueError, TypeError): + return None + if isinstance(col_type, Float): + try: + return float(value) + except (ValueError, TypeError): + return None + if isinstance(col_type, DateTime): + # read_csv does not parse_dates, so values are typically raw strings. + # Parse explicitly to avoid driver-dependent insert failures. + if hasattr(value, "to_pydatetime"): + return value.to_pydatetime() + ts = pd.to_datetime(value, errors="coerce") + return None if pd.isna(ts) else ts.to_pydatetime() + if isinstance(col_type, String): + s = str(value) + return s[: col_type.length] if col_type.length else s + # Fallback (should not hit for our mirror types). + return value + + +def _row_source(spec: MirrorSpec): + """Return ``(iterator_of_raw_dicts, source_label)`` for a spec. + + SQL dump if ``NMW_SQL_DUMP`` is set, otherwise CSV. Raises on a hard read + error so the caller can record/skip the table. + """ + dump = os.getenv(_SQL_DUMP_ENV) + if dump: + return iter_table_rows(dump, spec.source_table), f"sql:{os.path.basename(dump)}" + df = read_csv(spec.source_table) + return (rec for rec in df.to_dict("records")), "csv" + + +def _flush(session: Session, model, rows: list[dict], pk_cols: list[str]) -> int: + """Upsert a batch; return inserted row count.""" + if not rows: + return 0 + stmt = pg_insert(model).values(rows).on_conflict_do_nothing(index_elements=pk_cols) + result = session.execute(stmt) + session.commit() + return result.rowcount if result.rowcount and result.rowcount > 0 else 0 + + +def _copy_csv_into_table( + session: Session, table_name: str, header: list[str], csv_path: str +) -> None: + """Bulk-load a CSV into ``table_name`` via Postgres COPY (pg8000 stream).""" + collist = ", ".join(f'"{c}"' for c in header) + sql = ( + f'COPY "{table_name}" ({collist}) FROM STDIN ' + "WITH (FORMAT CSV, HEADER true, NULL '')" + ) + raw = session.connection().connection # underlying pg8000 DBAPI connection + cursor = raw.cursor() + with open(csv_path, "rb") as f: + cursor.execute(sql, stream=f) + + +def _copy_load_table( + session: Session, spec: MirrorSpec, dump: str, out_dir: str, limit: int = 0 +) -> dict: + """Dump -> per-table CSV (sqlparse) -> COPY into the mirror table.""" + table = spec.model.__table__ + name = spec.source_table + # Load only model columns (rowversion/LargeBinary excluded). COPY relies on + # Postgres to cast text -> column types, so no Python coercion is needed. + columns = [c.name for c in table.columns if not isinstance(c.type, LargeBinary)] + out_csv = os.path.join(out_dir, f"{name}.csv") + + n, header = write_table_csv(dump, name, out_csv, columns=columns, limit=limit) + if n == 0: + logger.warning("Skipping %s (no rows in dump)", name) + return {"table": name, "skipped": True, "reason": "no rows", "source": "sql"} + + # Staging reload: truncate then COPY (no upsert; tables are a 1:1 snapshot). + # CASCADE because mirror tables carry FK constraints (e.g. NMW_WellLocations + # / NMW_WellRecords -> NMW_WellHeaders); a bare TRUNCATE of a referenced + # parent is rejected. Specs load parents before children (see V2), so a + # cascaded truncate only clears child tables that are reloaded afterwards. + session.execute(text(f'TRUNCATE TABLE "{table.name}" CASCADE')) + _copy_csv_into_table(session, table.name, header, out_csv) + session.commit() + logger.info("COPY %s -> %s: %d rows (%s)", name, table.name, n, out_csv) + return {"table": name, "skipped": False, "rows": n, "inserted": n, "source": "sql"} + + +def _load_table(session: Session, spec: MirrorSpec, limit: int = 0) -> dict: + """Load one source table (SQL dump or CSV) into its mirror. Stats dict.""" + table = spec.model.__table__ + name = spec.source_table + # Loadable columns from the model (rowversion/LargeBinary excluded defensively). + cols = {c.name: c for c in table.columns if not isinstance(c.type, LargeBinary)} + pk_cols = [c.name for c in table.primary_key] + + try: + rows_iter, src = _row_source(spec) + except Exception as e: # noqa: BLE001 - missing source must not abort the run + logger.warning("Skipping %s (could not read source): %s", name, e) + return {"table": name, "skipped": True, "reason": str(e)} + + if limit and limit > 0: + rows_iter = itertools.islice(rows_iter, limit) + + chunk_size = min(_CHUNK_SIZE, _MAX_PG8000_PARAMS // max(len(cols), 1)) + total = 0 + inserted = 0 + batch: list[dict] = [] + warned_cols = False + for rec in rows_iter: + total += 1 + if not warned_cols: + missing = [n for n in cols if n not in rec] + if missing: + logger.warning( + "%s: mirror columns absent from source: %s", name, missing + ) + warned_cols = True + # NaN/NaT (CSV) and NULL (SQL) normalize to None inside _coerce. + row = {n: _coerce(rec.get(n), cols[n].type) for n in cols if n in rec} + if any(row.get(pk) is None for pk in pk_cols): + continue # cannot upsert without a PK value + batch.append(row) + if len(batch) >= chunk_size: + inserted += _flush(session, spec.model, batch, pk_cols) + batch = [] + inserted += _flush(session, spec.model, batch, pk_cols) + + if total == 0: + logger.warning("Skipping %s (no source rows from %s)", name, src) + return {"table": name, "skipped": True, "reason": "no rows", "source": src} + + logger.info( + "Mirror %s -> %s [%s]: %d source rows, %d inserted", + name, + table.name, + src, + total, + inserted, + ) + return { + "table": name, + "skipped": False, + "rows": total, + "inserted": inserted, + "source": src, + } + + +def transfer_nmw_mirror(session: Session, limit: int = None) -> tuple: + """Load all NM_Wells source tables into the ``NMW_*`` staging mirror. + + Source is a SQL dump (``NMW_SQL_DUMP``) when set, else per-table CSVs. Same + ``(session, limit)`` signature as the other session-based transfers. Returns + ``(num_tables_loaded, total_rows_inserted, errors)``. + """ + limit = int(limit or 0) + dump = os.getenv(_SQL_DUMP_ENV) + out_dir = None + if dump: + if not os.path.exists(dump): + raise FileNotFoundError(f"{_SQL_DUMP_ENV} set but file not found: {dump}") + out_dir = os.getenv(_CSV_DIR_ENV) or tempfile.mkdtemp(prefix="nmw_csv_") + os.makedirs(out_dir, exist_ok=True) + logger.info("NMW mirror source: SQL dump %s -> CSV %s -> COPY", dump, out_dir) + else: + logger.info("NMW mirror source: CSV exports (set %s for a dump)", _SQL_DUMP_ENV) + + results = [] + errors = [] + for spec in NMW_MIRROR_SPECS: + try: + if dump: + results.append(_copy_load_table(session, spec, dump, out_dir, limit)) + else: + results.append(_load_table(session, spec, limit)) + except Exception as e: # noqa: BLE001 - isolate per-table failures + logger.critical("NMW mirror load failed for %s: %s", spec.source_table, e) + session.rollback() + errors.append({"table": spec.source_table, "error": str(e)}) + + loaded = [r for r in results if not r.get("skipped")] + skipped = [r for r in results if r.get("skipped")] + inserted = sum(r.get("inserted", 0) for r in loaded) + logger.info( + "NMW mirror load complete: %d tables loaded, %d skipped, %d rows inserted, " + "%d errors", + len(loaded), + len(skipped), + inserted, + len(errors), + ) + return len(loaded), inserted, errors + + +def refresh_materialized_views(session: Session) -> list[str]: + """REFRESH the geothermal materialized OGC views (skip any not present). + + Call after a mirror (re)load so the materialized views reflect new data. + Plain (non-concurrent) REFRESH — runs inside the session transaction. + """ + refreshed = [] + for view in _MATERIALIZED_VIEWS: + exists = session.execute( + text( + "SELECT EXISTS(" + "SELECT 1 FROM pg_matviews WHERE schemaname='public' AND matviewname=:n" + ")" + ), + {"n": view}, + ).scalar() + if not exists: + logger.warning("Skip refresh; materialized view missing: %s", view) + continue + logger.info("REFRESH MATERIALIZED VIEW %s", view) + session.execute(text(f'REFRESH MATERIALIZED VIEW "{view}"')) + session.commit() + refreshed.append(view) + return refreshed + + +# ============= EOF ============================================= diff --git a/transfers/nmw_sql_dump.py b/transfers/nmw_sql_dump.py new file mode 100644 index 00000000..f7010b84 --- /dev/null +++ b/transfers/nmw_sql_dump.py @@ -0,0 +1,228 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""Parse a SQL Server data-dump ``.sql`` file into per-table CSVs. + +``INSERT [dbo].[
] () VALUES ()[, () ...]`` statements +(SSMS "Generate Scripts -> data" / bcp INSERT mode) are split with ``sqlparse`` +and decoded to plain Python values: + + NULL -> None + N'...' / '...' -> str (doubled '' unescaped) + 123 / -1.5 -> int / float + CAST(expr AS type) -> the inner expr, recursively + 0x.... -> None (binary / rowversion; not mirrored) + +``iter_table_rows`` yields ``{column: value}`` dicts; ``write_table_csv`` writes +one table to a CSV suitable for a Postgres ``COPY ... FROM`` bulk load (NULL -> +empty field, so load with ``NULL ''``). + +Encoding is auto-detected from the BOM (SSMS writes UTF-16 LE); falls back to +utf-8. +""" + +import csv +import itertools +import re +from typing import Iterator, Optional + +import sqlparse + + +def _detect_encoding(path: str) -> str: + with open(path, "rb") as f: + head = f.read(4) + if head[:2] in (b"\xff\xfe", b"\xfe\xff"): + return "utf-16" + if head[:3] == b"\xef\xbb\xbf": + return "utf-8-sig" + return "utf-8" + + +def _split_top_level(s: str) -> list[str]: + """Split a comma list at paren-depth 0, respecting single-quoted strings.""" + parts: list[str] = [] + buf: list[str] = [] + depth = 0 + in_quote = False + i = 0 + n = len(s) + while i < n: + c = s[i] + if in_quote: + buf.append(c) + if c == "'": + if i + 1 < n and s[i + 1] == "'": # escaped '' + buf.append("'") + i += 2 + continue + in_quote = False + i += 1 + continue + if c == "'": + in_quote = True + buf.append(c) + elif c == "(": + depth += 1 + buf.append(c) + elif c == ")": + depth -= 1 + buf.append(c) + elif c == "," and depth == 0: + parts.append("".join(buf).strip()) + buf = [] + else: + buf.append(c) + i += 1 + if buf: + parts.append("".join(buf).strip()) + return parts + + +def _iter_value_groups(s: str) -> Iterator[str]: + """Yield the inside of each top-level ``( ... )`` group in a VALUES list.""" + depth = 0 + in_quote = False + start = -1 + i = 0 + n = len(s) + while i < n: + c = s[i] + if in_quote: + if c == "'": + if i + 1 < n and s[i + 1] == "'": + i += 2 + continue + in_quote = False + i += 1 + continue + if c == "'": + in_quote = True + elif c == "(": + if depth == 0: + start = i + 1 + depth += 1 + elif c == ")": + depth -= 1 + if depth == 0 and start >= 0: + yield s[start:i] + start = -1 + i += 1 + + +# The AS-target type may itself be parameterised, e.g. CAST(1.50 AS Decimal(18, 2)) +# or CAST(N'x' AS nvarchar(10)); allow one level of parens in the type name. +_CAST_RE = re.compile(r"(?is)^CAST\s*\((.*)\s+AS\s+[^()]+(?:\([^)]*\))?\s*\)$") + + +def _parse_value(tok: str): + t = tok.strip() + if not t or t.upper() == "NULL": + return None + m = _CAST_RE.match(t) + if m: + return _parse_value(m.group(1).strip()) + # N'...' or '...' + if t[:1] == "'" or t[:2].upper() == "N'": + q = t.find("'") + inner = t[q + 1 :] + if inner.endswith("'"): + inner = inner[:-1] + return inner.replace("''", "'") + if t[:2].lower() == "0x": # binary / rowversion + return None + if re.fullmatch(r"[-+]?\d+", t): + return int(t) + try: + return float(t) + except ValueError: + return t + + +_INSERT_RE = re.compile( + r"(?is)INSERT\s+(?:\[dbo\]\.)?\[?(?P
\w+)\]?\s*" + r"\((?P.*?)\)\s*VALUES\s*(?P.*)$" +) + + +def _iter_insert_statements(path: str, table: str) -> Iterator[str]: + """Yield raw INSERT statement strings for ``table`` using sqlparse.""" + enc = _detect_encoding(path) + target = table.lower() + with open(path, encoding=enc, errors="ignore") as f: + # parsestream splits the dump into statements lazily. + for statement in sqlparse.parsestream(f): + s = str(statement).strip() + if not s: + continue + low = s.lower() + if "insert" not in low or target not in low: + continue + yield s + + +def iter_table_rows(path: str, table: str) -> Iterator[dict]: + """Yield ``{column: value}`` dicts for every INSERT into ``table``.""" + for stmt in _iter_insert_statements(path, table): + m = _INSERT_RE.search(stmt) + if not m or m.group("table").lower() != table.lower(): + continue + cols = [c.strip().strip("[]") for c in _split_top_level(m.group("cols"))] + vals_part = m.group("vals").strip().rstrip(";") + for group in _iter_value_groups(vals_part): + vals = [_parse_value(v) for v in _split_top_level(group)] + if len(vals) != len(cols): + continue # malformed row; skip + yield dict(zip(cols, vals)) + + +def _csv_cell(value) -> str: + """Render a parsed value for a COPY-friendly CSV (None -> empty field).""" + return "" if value is None else str(value) + + +def write_table_csv( + path: str, + table: str, + out_csv: str, + columns: Optional[list[str]] = None, + limit: int = 0, +) -> tuple[int, list[str]]: + """Write one source table's rows to ``out_csv``. Returns (n_rows, header). + + ``columns`` restricts/orders the output columns (e.g. the target model's + columns); missing source values become empty fields. If omitted, the first + row's keys define the header. None -> empty so Postgres COPY ``NULL ''`` + treats it as NULL. + """ + rows = iter_table_rows(path, table) + if limit and limit > 0: + rows = itertools.islice(rows, limit) + + header: Optional[list[str]] = None + writer = None + n = 0 + with open(out_csv, "w", newline="", encoding="utf-8") as fo: + for rec in rows: + if header is None: + header = list(columns) if columns else list(rec.keys()) + writer = csv.writer(fo) + writer.writerow(header) + writer.writerow([_csv_cell(rec.get(c)) for c in header]) + n += 1 + return n, (header or list(columns or [])) + + +# ============= EOF ============================================= diff --git a/transfers/reference_lexicon_transfer.py b/transfers/reference_lexicon_transfer.py new file mode 100644 index 00000000..bfd391e2 --- /dev/null +++ b/transfers/reference_lexicon_transfer.py @@ -0,0 +1,402 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""Load the legacy NM_Wells ``ref_*`` reference tables into the lexicon. + +The planning workbook ("NM_Wells + Subsurface library.xlsx", sheet 1) flags the +``ref_*`` tables as "Add to lexicon". Each ref table is a small code/description +lookup; this transfer loads its rows as ``LexiconTerm`` rows and links them to a +``LexiconCategory`` named after the table (``ref_well_class`` -> ``well_class``). + +Idempotent: mirrors ``core.initializers.init_lexicon`` — categories and terms +upsert via ``ON CONFLICT DO NOTHING`` (both ``name``/``term`` are unique); +term<->category associations are inserted only when missing (no unique +constraint exists on that table). + +Row source is the same as the mirror loader: a SQL Server data dump when +``NMW_SQL_DUMP`` is set (parsed by ``transfers.nmw_sql_dump.iter_table_rows``), +otherwise per-table CSV exports via ``transfers.util.read_csv``. + +NOTE(columns): the ref tables' actual column names are not in the workbook, so +term/definition columns are AUTO-DETECTED per table (see ``_pick_columns``). If +auto-detection is wrong for a table, set ``term_col`` / ``definition_col`` +explicitly on its ``RefTableSpec`` below. The chosen columns are logged. + +NOTE(LU_*): the Subsurface Library ``LU_*`` lookups (LU_EnteredBy, LU_LogType, +LU_Status, LU_Type_Wellheader, LU_WorkType) are also "Add to lexicon"; add them +to ``REFERENCE_TABLE_SPECS`` once their CSVs are available. +""" + +import itertools +import os +from dataclasses import dataclass +from typing import Optional + +import pandas as pd +from sqlalchemy import select +from sqlalchemy.dialects.postgresql import insert as pg_insert +from sqlalchemy.orm import Session + +from db import ( + LexiconCategory, + LexiconTerm, + LexiconTermCategoryAssociation, +) +from transfers.logger import logger +from transfers.nmw_sql_dump import iter_table_rows +from transfers.util import read_csv + +# Same source selector as the mirror loader: a SQL Server data dump when +# NMW_SQL_DUMP is set, otherwise per-table CSV exports. +_SQL_DUMP_ENV = "NMW_SQL_DUMP" +# lexicon_term.term (and its FK targets) is String(100). +_TERM_MAX_LEN = 100 + +# Column-name hints for auto-detecting the code/term vs definition columns. +_META_COLS = {"objectid", "ssma_timestamp", "globalid", "id", "import_id"} +_TERM_HINTS = ("code", "abbr", "symbol", "letter", "key", "short") +_DEF_HINTS = ( + "description", + "desc", + "definition", + "meaning", + "label", + "name", + "long", + "title", + "value", + "text", +) + + +@dataclass +class RefTableSpec: + """One legacy ref table -> one lexicon category. + + term_col / definition_col are optional overrides; when None the columns are + auto-detected from the CSV header. + """ + + source_table: str + category: str + term_col: Optional[str] = None + definition_col: Optional[str] = None + description: Optional[str] = None + + +def _spec(table: str) -> RefTableSpec: + """Build a spec with category ``nmw_
`` (e.g. ``nmw_ref_states``).""" + return RefTableSpec( + source_table=table, + category=f"nmw_{table}", + description=f"Imported from NM_Wells {table}", + ) + + +# All ref_* tables marked "Add to lexicon" in the workbook (sheet 1). +# ref_nm_quads (Review, ~2k rows) is intentionally excluded; add/remove specs +# here as the mapping is refined. +REFERENCE_TABLE_SPECS: list[RefTableSpec] = [ + _spec(t) + for t in ( + "ref_altitude_datums", + "ref_altitude_methods", + "ref_basins", + "ref_coordinate_accuracy", + "ref_coordinate_datum", + "ref_coordinate_method", + "ref_county", + "ref_data_reliability", + "ref_date_drilled", + "ref_depth_types", + "ref_display_scales", + "ref_ground_levels", + "ref_gt_data_sources", + "ref_gt_well_types", + "ref_ign_comps", + "ref_indurations", + "ref_initials", + "ref_length_units", + "ref_lith_class", + "ref_lith_types", + "ref_ll_sources", + "ref_mm_facies", + "ref_perforation_types", + "ref_porosity_methods", + "ref_pres_units", + "ref_prod_meth_quality", + "ref_prod_methods", + "ref_prod_units", + "ref_sample_class", + "ref_sample_types", + "ref_states", + "ref_textures", + "ref_unit_basis", + "ref_unit_conductivity", + "ref_unit_depths", + "ref_unit_gradients", + "ref_unit_heat_flow", + "ref_unit_letters", + "ref_unit_temps", + "ref_well_action_class", + "ref_well_class", + "ref_well_commodity", + "ref_well_log_class", + "ref_well_orientations", + "ref_well_record_class", + "ref_well_status", + "ref_well_types", + "ref_work_types", + "ref_xy_units", + ) +] + + +def _pick_columns(columns: list[str], spec: RefTableSpec) -> tuple[str, str]: + """Resolve (term_col, definition_col) for a ref table. + + Honors explicit overrides on the spec, else auto-detects from the column + names using name hints, ignoring meta columns (OBJECTID, GlobalID, ...). + """ + cols = [c for c in columns if str(c).strip().lower() not in _META_COLS] + if not cols: + cols = list(columns) + low = {c: str(c).strip().lower() for c in cols} + + term_col = spec.term_col + if term_col is None: + term_col = next( + (c for c in cols if any(h in low[c] for h in _TERM_HINTS)), cols[0] + ) + + def_col = spec.definition_col + if def_col is None: + def_col = next( + (c for c in cols if c != term_col and any(h in low[c] for h in _DEF_HINTS)), + None, + ) + if def_col is None: + def_col = cols[1] if len(cols) > 1 else term_col + + return term_col, def_col + + +def _clean(value) -> Optional[str]: + if value is None or pd.isna(value): + return None + s = str(value).strip() + return s or None + + +def _get_or_create_category(session: Session, spec: RefTableSpec) -> int: + """Return the lexicon_category.id for the spec, creating it if needed.""" + cat_id = session.execute( + select(LexiconCategory.id).where(LexiconCategory.name == spec.category) + ).scalar_one_or_none() + if cat_id is not None: + return cat_id + + session.execute( + pg_insert(LexiconCategory) + .values(name=spec.category, description=spec.description) + .on_conflict_do_nothing(index_elements=["name"]) + ) + session.commit() + return session.execute( + select(LexiconCategory.id).where(LexiconCategory.name == spec.category) + ).scalar_one() + + +def _iter_source_rows(table: str, limit: int = 0): + """Yield raw ``{column: value}`` dicts for a ref table. + + SQL dump when NMW_SQL_DUMP is set (same source as the mirror loader), + otherwise per-table CSV. Mirrors transfers.nmw_mirror_transfer._row_source. + """ + dump = os.getenv(_SQL_DUMP_ENV) + if dump: + it = iter_table_rows(dump, table) + else: + df = read_csv(table) + it = (rec for rec in df.to_dict("records")) + if limit and limit > 0: + it = itertools.islice(it, limit) + return it + + +def _transfer_one(session: Session, spec: RefTableSpec, limit: int = 0) -> dict: + """Load a single ref table into the lexicon. Returns a stats dict.""" + try: + rows = list(_iter_source_rows(spec.source_table, limit)) + except Exception as e: # noqa: BLE001 - missing source should not abort the run + logger.warning("Skipping %s (could not read source): %s", spec.source_table, e) + return {"table": spec.source_table, "skipped": True, "reason": str(e)} + + if not rows: + logger.warning("Skipping %s (empty)", spec.source_table) + return {"table": spec.source_table, "skipped": True, "reason": "empty"} + + # Column names from the union of row keys (CSV rows and SSMS INSERTs are + # column-consistent, but be defensive). + columns: list[str] = [] + seen = set() + for rec in rows: + for k in rec: + if k not in seen: + seen.add(k) + columns.append(k) + + term_col, def_col = _pick_columns(columns, spec) + logger.info( + "%s -> category=%s term_col=%s definition_col=%s (%d rows)", + spec.source_table, + spec.category, + term_col, + def_col, + len(rows), + ) + + category_id = _get_or_create_category(session, spec) + + # Build unique (term -> definition) map, dropping empties and overlong terms. + term_defs: dict[str, str] = {} + truncated = 0 + for rec in rows: + term = _clean(rec.get(term_col)) + if term is None: + continue + if len(term) > _TERM_MAX_LEN: + term = term[:_TERM_MAX_LEN] + truncated += 1 + definition = _clean(rec.get(def_col)) or term + term_defs.setdefault(term, definition) + + if not term_defs: + logger.warning("Skipping %s (no usable terms)", spec.source_table) + return {"table": spec.source_table, "skipped": True, "reason": "no terms"} + if truncated: + logger.warning( + "%s: truncated %d term(s) to %d chars", + spec.source_table, + truncated, + _TERM_MAX_LEN, + ) + + term_names = list(term_defs) + existing_terms = dict( + session.execute( + select(LexiconTerm.term, LexiconTerm.id).where( + LexiconTerm.term.in_(term_names) + ) + ).all() + ) + new_rows = [ + {"term": t, "definition": d} + for t, d in term_defs.items() + if t not in existing_terms + ] + if new_rows: + session.execute( + pg_insert(LexiconTerm) + .values(new_rows) + .on_conflict_do_nothing(index_elements=["term"]) + ) + session.commit() + existing_terms = dict( + session.execute( + select(LexiconTerm.term, LexiconTerm.id).where( + LexiconTerm.term.in_(term_names) + ) + ).all() + ) + + term_ids = [tid for tid in existing_terms.values() if tid is not None] + existing_links = set() + if term_ids: + existing_links = set( + session.execute( + select(LexiconTermCategoryAssociation.term_id).where( + LexiconTermCategoryAssociation.category_id == category_id, + LexiconTermCategoryAssociation.term_id.in_(term_ids), + ) + ).scalars() + ) + + assoc_rows = [ + {"term_id": tid, "category_id": category_id} + for tid in term_ids + if tid not in existing_links + ] + if assoc_rows: + session.execute(pg_insert(LexiconTermCategoryAssociation).values(assoc_rows)) + session.commit() + + return { + "table": spec.source_table, + "skipped": False, + "rows": len(rows), + "terms": len(term_defs), + "created_terms": len(new_rows), + "linked": len(assoc_rows), + } + + +def transfer_reference_tables(session: Session, limit: int = None) -> tuple: + """Foundational transfer: load all ``ref_*`` tables into the lexicon. + + Same ``(session, limit)`` signature as the other foundational transfers + (aquifer systems, geologic formations). Returns + ``(num_tables, total_created_terms, errors)``. + """ + limit = int(limit or 0) + dump = os.getenv(_SQL_DUMP_ENV) + if dump: + if not os.path.exists(dump): + raise FileNotFoundError(f"{_SQL_DUMP_ENV} set but file not found: {dump}") + logger.info("Reference lexicon source: SQL dump %s", dump) + else: + logger.info( + "Reference lexicon source: CSV exports (set %s for a dump)", _SQL_DUMP_ENV + ) + + results = [] + errors = [] + for spec in REFERENCE_TABLE_SPECS: + try: + results.append(_transfer_one(session, spec, limit)) + except Exception as e: # noqa: BLE001 - isolate per-table failures + logger.critical( + "Reference lexicon transfer failed for %s: %s", spec.source_table, e + ) + session.rollback() + errors.append({"table": spec.source_table, "error": str(e)}) + + loaded = [r for r in results if not r.get("skipped")] + skipped = [r for r in results if r.get("skipped")] + created = sum(r.get("created_terms", 0) for r in loaded) + linked = sum(r.get("linked", 0) for r in loaded) + logger.info( + "Reference lexicon transfer complete: %d tables loaded, %d skipped, " + "%d terms created, %d associations, %d errors", + len(loaded), + len(skipped), + created, + linked, + len(errors), + ) + return len(loaded), created, errors + + +# ============= EOF ============================================= diff --git a/transfers/transfer.py b/transfers/transfer.py index 419d4870..b1cae6ba 100644 --- a/transfers/transfer.py +++ b/transfers/transfer.py @@ -13,8 +13,17 @@ # See the License for the specific language governing permissions and # limitations under the License. # =============================================================================== +"""DEPRECATED: legacy NM_Aquifer -> Ocotillo transfer orchestrator. + +This module (the original AMPAPI / NM_Aquifer migration driver) is deprecated. +Do not add new migrations here. New migrations get their own standalone +orchestrator script; e.g. the NM_Wells geothermal migration lives in +``transfers/transfer_geothermal.py``. +""" + import os import time +import warnings from concurrent.futures import ThreadPoolExecutor, as_completed from contextlib import contextmanager from dataclasses import dataclass @@ -324,6 +333,12 @@ def _drop_and_rebuild_db() -> None: @timeit def transfer_all(metrics: Metrics) -> list[ProfileArtifact]: + warnings.warn( + "transfers.transfer is deprecated; new migrations get their own " + "orchestrator (e.g. transfers/transfer_geothermal.py).", + DeprecationWarning, + stacklevel=2, + ) message("STARTING TRANSFER", new_line_at_top=False) if get_bool_env("DROP_AND_REBUILD_DB", False): logger.info("Dropping schema and rebuilding database from migrations") diff --git a/transfers/transfer_geothermal.py b/transfers/transfer_geothermal.py new file mode 100644 index 00000000..6945ea01 --- /dev/null +++ b/transfers/transfer_geothermal.py @@ -0,0 +1,111 @@ +# =============================================================================== +# Copyright 2026 ross +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# =============================================================================== +"""Standalone orchestrator for the NM_Wells (geothermal) migration. + +Separate from the deprecated ``transfers/transfer.py`` (NM_Aquifer driver). This +script runs the NM_Wells Phase-1 staging migration: + +1. Reference -> lexicon load (``ref_*`` lookups), gated by + ``TRANSFER_GEOTHERMAL_REFERENCE`` (default True). +2. NM_Wells 1:1 staging mirror load into the ``NMW_*`` tables, gated by + ``TRANSFER_NMW_MIRROR`` (default True). Row source is a SQL Server data dump + when ``NMW_SQL_DUMP`` is set, otherwise per-table CSV exports. + +Assumes the schema already exists (run ``alembic upgrade head`` first). Does not +drop/rebuild the database. + +Run: + python -m transfers.transfer_geothermal +Env: + TRANSFER_LIMIT=1000 # rows per table (0/unset = all) + NMW_SQL_DUMP=/path/to/data.sql # optional; else CSV + TRANSFER_GEOTHERMAL_REFERENCE=1 + TRANSFER_NMW_MIRROR=1 +""" + +import os + +from dotenv import load_dotenv + +# Load .env FIRST, before any database imports. Do not override env vars already +# set by the runtime (e.g. Cloud Run jobs). +load_dotenv(override=False) + +# In managed runtimes DB_DRIVER is sometimes omitted while CLOUD_SQL_* are set. +if ( + not (os.getenv("DB_DRIVER") or "").strip() + and (os.getenv("CLOUD_SQL_INSTANCE_NAME") or "").strip() +): + os.environ["DB_DRIVER"] = "cloudsql" + +from db.engine import session_ctx # noqa: E402 +from services.env import get_bool_env # noqa: E402 +from transfers.logger import logger # noqa: E402 +from transfers.nmw_mirror_transfer import ( # noqa: E402 + refresh_materialized_views, + transfer_nmw_mirror, +) +from transfers.reference_lexicon_transfer import transfer_reference_tables # noqa: E402 + + +def run_geothermal_transfer(limit: int = None) -> dict: + """Run the NM_Wells geothermal staging migration. Returns a summary dict.""" + limit = int(limit if limit is not None else os.getenv("TRANSFER_LIMIT", 0) or 0) + summary: dict = {} + + logger.info("========== NM_WELLS (GEOTHERMAL) MIGRATION ==========") + logger.info("limit=%s", limit or "all") + + if get_bool_env("TRANSFER_GEOTHERMAL_REFERENCE", True): + logger.info("---- Reference tables -> lexicon ----") + with session_ctx() as session: + tables, created, errors = transfer_reference_tables(session, limit=limit) + summary["reference"] = { + "tables": tables, + "terms_created": created, + "errors": len(errors), + } + else: + logger.info("Skipping reference->lexicon (TRANSFER_GEOTHERMAL_REFERENCE=0)") + + if get_bool_env("TRANSFER_NMW_MIRROR", True): + logger.info("---- NM_Wells 1:1 staging mirror ----") + with session_ctx() as session: + tables, inserted, errors = transfer_nmw_mirror(session, limit=limit) + summary["mirror"] = { + "tables": tables, + "rows_inserted": inserted, + "errors": len(errors), + } + logger.info("---- Refresh materialized OGC views ----") + with session_ctx() as session: + summary["refreshed_views"] = refresh_materialized_views(session) + else: + logger.info("Skipping NM_Wells mirror (TRANSFER_NMW_MIRROR=0)") + + logger.info("NM_Wells migration complete: %s", summary) + return summary + + +def main() -> None: + run_geothermal_transfer() + + +if __name__ == "__main__": + main() + + +# ============= EOF ============================================= diff --git a/transfers/util.py b/transfers/util.py index 5fd1a471..ff5c4f4e 100644 --- a/transfers/util.py +++ b/transfers/util.py @@ -406,6 +406,11 @@ def extract_organization(alternate_id: str) -> str: def get_transfers_data_path(name: str) -> Path: + # Explicit override wins: CSVs live flat in this dir (no nma_csv_cache subdir). + env_root = os.environ.get("TRANSFERS_DATA_DIR") + if env_root: + return Path(env_root) / Path(name).name + def data_path(r): return Path(r) / "transfers" / "data" diff --git a/uv.lock b/uv.lock index a792ff34..5704e2b6 100644 --- a/uv.lock +++ b/uv.lock @@ -1550,6 +1550,7 @@ dependencies = [ { name = "pygeoapi" }, { name = "pygments" }, { name = "pyjwt" }, + { name = "pymssql" }, { name = "pyproj" }, { name = "pyshp" }, { name = "python-dateutil" }, @@ -1567,6 +1568,7 @@ dependencies = [ { name = "sqlalchemy-continuum" }, { name = "sqlalchemy-searchable" }, { name = "sqlalchemy-utils" }, + { name = "sqlparse" }, { name = "starlette" }, { name = "starlette-admin", extra = ["i18n"] }, { name = "typer" }, @@ -1663,6 +1665,7 @@ requires-dist = [ { name = "pygeoapi", specifier = "==0.23.4" }, { name = "pygments", specifier = "==2.20.0" }, { name = "pyjwt", specifier = "==2.13.0" }, + { name = "pymssql", specifier = ">=2.3.13" }, { name = "pyproj", specifier = "==3.7.2" }, { name = "pyshp", specifier = "==2.3.1" }, { name = "python-dateutil", specifier = "==2.9.0.post0" }, @@ -1680,6 +1683,7 @@ requires-dist = [ { name = "sqlalchemy-continuum", specifier = "==1.6.0" }, { name = "sqlalchemy-searchable", specifier = "==2.1.0" }, { name = "sqlalchemy-utils", specifier = "==0.42.1" }, + { name = "sqlparse", specifier = ">=0.5.5" }, { name = "starlette", specifier = "==1.3.1" }, { name = "starlette-admin", extras = ["i18n"], specifier = "==0.16.1" }, { name = "typer", specifier = "==0.26.7" }, @@ -2301,6 +2305,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a3/5e/ecf12fdb62546d64385c158514e9b2b671f7832108ef2ecd2020ce0af2d1/pyjwt-2.13.0-py3-none-any.whl", hash = "sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728", size = 31274, upload-time = "2026-05-21T19:54:35.362Z" }, ] +[[package]] +name = "pymssql" +version = "2.3.13" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/cc/843c044b7f71ee329436b7327c578383e2f2499313899f88ad267cdf1f33/pymssql-2.3.13.tar.gz", hash = "sha256:2137e904b1a65546be4ccb96730a391fcd5a85aab8a0632721feb5d7e39cfbce", size = 203153, upload-time = "2026-02-14T05:00:36.865Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d4/4f/ee15b1f6b11e7c3accdc7da7840a019b63f12ba09eaa008acc601182f516/pymssql-2.3.13-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:30918bb044242865c01838909777ef5e0f1b9ecd7f5882346aefa57f4414b29c", size = 3156333, upload-time = "2026-02-14T05:00:01.21Z" }, + { url = "https://files.pythonhosted.org/packages/79/03/aea5c77bad4a52649a1d9f786a1d9ce1c83d50f1a75df288e292737b6d80/pymssql-2.3.13-cp313-cp313-macosx_15_0_x86_64.whl", hash = "sha256:1c6d0b2d7961f159a07e4f0d8cc81f70ceab83f5e7fd1e832a2d069e1d67ee4e", size = 2957990, upload-time = "2026-02-14T05:00:03.11Z" }, + { url = "https://files.pythonhosted.org/packages/5f/f8/30ac16fba32ff066b05f12c392d7b812fe11f06cb62d1d86ca5177c50a8b/pymssql-2.3.13-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16c5957a3c9e51a03276bfd76a22431e2bc4c565e2e95f2cbb3559312edda230", size = 3065264, upload-time = "2026-02-14T05:00:05.377Z" }, + { url = "https://files.pythonhosted.org/packages/a9/98/7568447bf85921d21453fd56e19b6c9591d595fde0546c5a569f3ae937a8/pymssql-2.3.13-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0fddd24efe9d18bbf174fab7c6745b0927773718387f5517cf8082241f721a68", size = 3190039, upload-time = "2026-02-14T05:00:06.925Z" }, + { url = "https://files.pythonhosted.org/packages/35/f1/4d9d275ebaac42cdd49d40d504ccb648f27710660c8b60cc427752438c09/pymssql-2.3.13-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:123c55ee41bc7a82c76db12e2eb189b50d0d7a11222b4f8789206d1cda3b33b9", size = 3710151, upload-time = "2026-02-14T05:00:08.424Z" }, + { url = "https://files.pythonhosted.org/packages/6f/bd/a5cc6244fd27d3ea0cc82f12a7d38a24d7fd90b0022afd250014e8bfba15/pymssql-2.3.13-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e053b443e842f9e1698fcb2b23a4bff1ff3d410894d880064e754ad823d541e5", size = 3453156, upload-time = "2026-02-14T05:00:09.978Z" }, + { url = "https://files.pythonhosted.org/packages/26/d0/c20ff0bbffd18db528bcc7b0c68b25c12ad563ed67c56ceca87c58f7399e/pymssql-2.3.13-cp313-cp313-win_amd64.whl", hash = "sha256:5c045c0f1977a679cc30d5acd9da3f8aeb2dc6e744895b26444b4a2f20dad9a0", size = 1995236, upload-time = "2026-02-14T05:00:11.495Z" }, + { url = "https://files.pythonhosted.org/packages/ec/5f/6b64f78181d680f655ab40ba7b34cb68c045a2f4e04a10a70d768cd383b7/pymssql-2.3.13-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:fc5482969c813b0a45ce51c41844ae5bfa8044ad5ef8b4820ef6de7d4545b7f2", size = 3158377, upload-time = "2026-02-14T05:00:13.581Z" }, + { url = "https://files.pythonhosted.org/packages/ff/24/155dbb0992c431496d440f47fb9d587cd0059ee20baf65e3d891794d862a/pymssql-2.3.13-cp314-cp314-macosx_15_0_x86_64.whl", hash = "sha256:ff5be7ab1d643dbce2ee3424d2ef9ae8e4146cf75bd20946bc7a6108e3ad1e47", size = 2959039, upload-time = "2026-02-14T05:00:15.883Z" }, + { url = "https://files.pythonhosted.org/packages/c9/89/b453dd1b1188779621fb974ac715ab2e738f4a0b69f7291ab014298bd80d/pymssql-2.3.13-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8d66ce0a249d2e3b57369048d71e1f00d08dfb90a758d134da0250ae7bc739c1", size = 3063862, upload-time = "2026-02-14T05:00:17.537Z" }, + { url = "https://files.pythonhosted.org/packages/02/e5/96f57c78162013678ecc3f3f7e5fb52c83ee07beef26906d0870770c3ef6/pymssql-2.3.13-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d663c908414a6a032f04d17628138b1782af916afc0df9fefac4751fa394c3ac", size = 3188155, upload-time = "2026-02-14T05:00:19.011Z" }, + { url = "https://files.pythonhosted.org/packages/cd/a2/4bee9484734ae0c55d10a2f6ff82dd4e416f52420755161b8760c817ad64/pymssql-2.3.13-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:aa5e07eff7e6e8bd4ba22c30e4cb8dd073e138cd272090603609a15cc5dbc75b", size = 3709344, upload-time = "2026-02-14T05:00:21.139Z" }, + { url = "https://files.pythonhosted.org/packages/37/cf/3520d96afa213c88db4f4a1988199db476d869a62afdd5d9c4635c184631/pymssql-2.3.13-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:db77da1a3fc9b5b5c5400639d79d7658ba7ad620957100c5b025be608b562193", size = 3451799, upload-time = "2026-02-14T05:00:22.504Z" }, + { url = "https://files.pythonhosted.org/packages/25/50/4be9bd9cf4b43208a7175117a533ece200cfe4131a39f9909bdc7560ddeb/pymssql-2.3.13-cp314-cp314-win_amd64.whl", hash = "sha256:7d7037d2b5b907acc7906d0479924db2935a70c720450c41339146a4ada2b93d", size = 2049139, upload-time = "2026-02-14T05:00:23.951Z" }, +] + [[package]] name = "pyparsing" version = "3.3.2" @@ -2893,6 +2919,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7c/25/7400c18c3ee97914cc99c90007795c00a4ec5b60c853b49db7ba24d11179/sqlalchemy_utils-0.42.1-py3-none-any.whl", hash = "sha256:243cfe1b3a1dae3c74118ae633f1d1e0ed8c787387bc33e556e37c990594ac80", size = 91761, upload-time = "2025-12-13T03:14:15.014Z" }, ] +[[package]] +name = "sqlparse" +version = "0.5.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/90/76/437d71068094df0726366574cf3432a4ed754217b436eb7429415cf2d480/sqlparse-0.5.5.tar.gz", hash = "sha256:e20d4a9b0b8585fdf63b10d30066c7c94c5d7a7ec47c889a2d83a3caa93ff28e", size = 120815, upload-time = "2025-12-19T07:17:45.073Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/4b/359f28a903c13438ef59ebeee215fb25da53066db67b305c125f1c6d2a25/sqlparse-0.5.5-py3-none-any.whl", hash = "sha256:12a08b3bf3eec877c519589833aed092e2444e68240a3577e8e26148acc7b1ba", size = 46138, upload-time = "2025-12-19T07:17:46.573Z" }, +] + [[package]] name = "starlette" version = "1.3.1"