BDMS 159: well transfers updates, water level transfer updates, & other revisions

[jacob-a-brown]

Why

This PR addresses the following problem / context:

• well and water level transfer scripts need to be updated for the new and revised models
• well depth, hole depth, and well casing depth need to be validated against each other if information is present
• MeasuredBy from the WaterLevels table needs to be mapped to contacts in Ocotillo

How

Implementation summary - the following was changed / added / removed:

• update well transfer script to incorporate new and revised fields.
  • to enable effective_start to be added to the location association model I needed to add the well manually and not through add_thing
• updated MeasuredBy mappings with Ethan's and Stacy's input. If a record cannot be mapped the new name is what is from NM_Aquifer and both organization and role are set to Unknown. We'll know if that record comes from NM_Aquifer since the associated nma_pk_waterlevels field will be populated.

Notes

Any special considerations, workarounds, or follow-up work to note?

• this incorporates the changes made in PR NO TICKET: thing update and fix #179 (eager loading). because of eager loading .unique() needed to be added to select statements in the transfer script
• this incorporates changes that have been made to the feature branch and are not yet present in staging
• because the same session is used throughout the transfer process, when data is added to the DB sometimes SQLAlchemy stores that data in memory (cached), so when I retrieve a record it doesn't always reflect the data types in the database. This is why I have to expunge and refresh locations from the session. Otherwise the point field is sometimes a string and the latlon property fails (it's not retrieved as a WKBElement and therefore has no x or y attributes)

WORK TO BE DONE

This PR is to catch the transfer scripts up to the current state of the models (for the most part). There are additions and amendments we want to make, such as well metadata fields. To keep this PR focused and to preclude complications that will arise from model changes, revisions will occur in future PRs. As of now these are (as I recall, there is likely more we want to do):

• updating the StatusHistory table in the well transfer script if a well is marked as destroyed
• adding nma_phone to the Phone table for access to invalid legacy data (mostly phone numbers that are 7 and not 10 digits)
• adding organizations from OwnersData to enable owners' organization field to be valid when transferring data from OwnersData (NM_Aquifer) to Contact (Ocotillo)
• adding well metadata fields when warranted and after careful consideration (e.g. sample_ok, formation, open_well_logger_ok)
• adding tables for WellPurpose and WellCasing since a well may serve multiple purposes and well casing may be made of many materials
• adding the polymorphic notes table

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.44444% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
services/util.py	37.50%	5 Missing ⚠️
schemas/location.py	87.50%	1 Missing ⚠️
services/thing_helper.py	85.71%	1 Missing ⚠️

Files with missing lines	Coverage Δ
api/search.py	97.77% <100.00%> (ø)
api/thing.py	98.46% <ø> (ø)
db/location.py	95.00% <100.00%> (ø)
db/thing.py	100.00% <100.00%> (ø)
schemas/contact.py	100.00% <100.00%> (ø)
schemas/thing.py	99.15% <100.00%> (ø)
services/geospatial_helper.py	74.46% <100.00%> (ø)
tests/conftest.py	98.59% <100.00%> (ø)
tests/test_location.py	100.00% <ø> (ø)
tests/test_observation.py	95.65% <ø> (ø)
... and 5 more