Skip to content

BDMS-115 Sensor model updates#189

Merged
jirhiker merged 13 commits into
stagingfrom
kas-sensor-model-update-20251008
Oct 17, 2025
Merged

BDMS-115 Sensor model updates#189
jirhiker merged 13 commits into
stagingfrom
kas-sensor-model-update-20251008

Conversation

@ksmuczynski

Copy link
Copy Markdown
Contributor

Why

This PR addresses the following problem / context:
This PR introduces enhancements and fixes across multiple modules related to sensor management and data validation. These changes enhance the system's ability to manage sensor deployments and validate data in support of groundwater monitoring workflows.

  • Updates the Sensor model to serve as the central asset inventory for all physical hardware used for data collection.
  • Refactors the Deployment model to better track sensor installations and their relationships with Thing entities.
  • Improves schema definitions for creating, updating, and responding to sensor-related data.
  • Improves the sensor transfer logic and aligns it with the updated sensor model.

How

Implementation summary - the following was changed / added / removed:

  • Added new fields to track equipment details:
    • model - Equipment model/make
    • serial_no - Serial number
    • pcn_number - Property control number
    • owner_agency - Owning organization
    • sensor_status - Current status (In Service, In Repair, etc.)
    • notes - General notes field
  • Added controlled vocabulary terms for:
    • Sensor types (Pressure Transducer, Data Logger, etc.)
    • Sensor status values (In Service, In Repair, Retired, Lost)
  • Added CreateSensor, UpdateSensor, and SensorResponse schemas in schemas/sensor.py.
  • Updated the Deployment model in db/deployment.py to include detailed attributes like hanging_cable_length and hanging_point_description.
  • Introduced new fixtures in tests/conftest.py for testing sensor and deployment relationships.
  • Removed outdated validation logic for datetime_installed and datetime_removed in the Sensor model.
  • Added and updated unit tests to cover the new transfer logic.
  • Updated documentation strings for better clarity.

Notes

Any special considerations, workarounds, or follow-up work to note?

  • Transfer related notes:
    • Sensors are present multiple times in the Equipment table, once for each deployment. To avoid having duplicates the sensor is only created once when the script first comes across it in NM_Aquifer. The nma_pk_equipment field is populated from the record from which the sensor was created. This assumes that the data is the same between the different records; the data is not checked from one record to another.
    • Sensors for wells are first grouped by DateInstalled and then added. If the DateRemoved field is NULL for the last deployment the sensor_status field is set to In Service. Otherwise if DateRemoved is not NULL for the latest deployment the sensor_status field is set to Retired. Is this the correct way to proceed?
    • The equipment types that are kept, for now, are Pressure transducer, Acoustic sounder, and Barometer

ksmuczynski and others added 13 commits October 8, 2025 14:15
Documentation about the Sensor model was lacking.

This commit updates the `Sensor` model documentation and provides clarity on the purpose and function of the table.
Refinements to the schema design have necessitated the addition and removal of fields in the Sensor model.

- Added `comment` parameter to capture field descriptions.
- Added new 'nma_pk_equipment`,'`sensor_type`, `sensor_status`, `pcn_number`, `owner_agency`, and `notes` fields.
- Removed `datetime_installed`, `datetime_removed`, and `recording_interval` fields. These fields are now stored in the Deployment model.
A new `sensor_type` category was necessary to store valid `sensor_type` values.
The lexicon needs a new `sensor_status` category, but category named `status` already exists.It is unclear what kind of status it refers to.

This commit changes the category `status` to `well_status` and adds a new category `sensor_status` and related valid values.
The test files needed to be updated to reflect the field changes made in the `Sensor` model.

This commit updates the `test_sensor.py` file.
The test files needed to be updated to reflect the field changes made in the `Sensor` model.

This commit updates the `conftest.py` file.
The GET `get_sensors` endpoint was using outdated logic with a non-existent `observed_property` filter.

This commit updates the `get_sensors` endpoint by replacing `observed_property` with `parameter_id`.
The Sensor pydantic schemas needed to be updated to match the modifications made to the Schema database model.

This commit adds the `sensor_type`, `pcn_number`, `owner_agency`, and `sensor_status` fields. This commit removes the `datetime_installed`, `datetime_removed` and `recording_interval` fields.
There were specific tests focused on validating installation and removal dates, but the `datetime_installed` and `datetime_removed` fields no longer exist in the Sensor model.

This commit comments out the tests attempting to validate the `datetime_installed` and `datetime_removed` fields.
Tests were failing because the PATCH endpoint was commented out of the ``/sensor` router.

This commit re-enables the PATCH endpoint, but comments out the validation on the `datetime_removed` adn `datetime_installed` fields, since they are no longer associated with the Sensor model.
@codecov-commenter

codecov-commenter commented Oct 17, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.00000% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
api/sensor.py 83.33% 1 Missing ⚠️
Files with missing lines Coverage Δ
db/deployment.py 100.00% <100.00%> (ø)
db/sensor.py 100.00% <100.00%> (ø)
schemas/sensor.py 100.00% <100.00%> (ø)
tests/conftest.py 98.59% <ø> (ø)
tests/test_sensor.py 100.00% <100.00%> (ø)
api/sensor.py 96.87% <83.33%> (ø)

@jirhiker jirhiker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. excited to merge this with the transducer pr

Comment thread db/sensor.py
Comment on lines +75 to +77
sensor_status: Mapped[str] = lexicon_term(
nullable=True,
comment="A controlled vocabulary field to indicate the current status of the equipment (e.g., 'In Service', 'In Repair', 'Retired', 'Lost', etc).",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something that goes in the StatusHistory table?

Comment thread api/sensor.py
Comment on lines +134 to +135
thing_id: int = None, # Optional filter for thing_id. Filter by the Thing where equipment is deployed
parameter_id: int = None, # Filter by the parameter the sensor/equipment measures

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these query parameters are now present in this endpoint we should make tests to ensure they work as expected and to safeguard future development.

@jirhiker jirhiker merged commit debc490 into staging Oct 17, 2025
3 checks passed
@jirhiker jirhiker deleted the kas-sensor-model-update-20251008 branch October 17, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants