Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: extend DBFlexContextSchema validation to only allow valid units #1364

Merged
merged 27 commits into from
Mar 18, 2025
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
01e46e3
feat: extend flexContext schema validation to only allow valid units
joshuaunity Mar 11, 2025
f7b33f8
Merge branch 'main' into feat/flexcontext-extended-validation
joshuaunity Mar 11, 2025
38c69ef
refactor: moved newly added unit validaitons to DBFlexContextSchema f…
joshuaunity Mar 11, 2025
57f77b5
chore: added to changelog
joshuaunity Mar 11, 2025
a66c98d
chore: added test for DBFlexContextSchema unit validation feat
joshuaunity Mar 12, 2025
6b09e53
Merge branch 'main' of https://github.com/FlexMeasures/flexmeasures i…
joshuaunity Mar 12, 2025
0e7faa3
chore: modified changelog
joshuaunity Mar 12, 2025
a48646b
chore: typo changes and modified changelog
joshuaunity Mar 13, 2025
81c54c4
chore: added accurate example units for flexContext fields in modal UI
joshuaunity Mar 13, 2025
accc0bc
chore: added new unit validation util funciton
joshuaunity Mar 14, 2025
267f264
refactor: refactored schema and tests to use the new validation util …
joshuaunity Mar 14, 2025
7aac5e6
refactor: mapped schema keys to expected user facing keys
joshuaunity Mar 17, 2025
3914e1e
chore: set proper test DB address
joshuaunity Mar 17, 2025
b7a9534
chore: modified validation response contents for capacity price fields
joshuaunity Mar 17, 2025
85e6998
chore: little modifications to schema validaiton
joshuaunity Mar 17, 2025
2502b7b
chore: more concise code
joshuaunity Mar 17, 2025
bf78d37
tests: added extra test case
joshuaunity Mar 17, 2025
44aaa59
feat: add test case for unsupported time series specs in DB flex-context
Flix6x Mar 18, 2025
ca60216
fix: move comment to the correct class method
Flix6x Mar 18, 2025
e48a763
fix: assign ValidationError to field name rather than to the general …
Flix6x Mar 18, 2025
1caef47
feat: add another test case for unsupported time series specs in DB f…
Flix6x Mar 18, 2025
d306374
fix: remove redundant validation case
Flix6x Mar 18, 2025
fbaf78f
revert: more explicit check in relation to the error message
Flix6x Mar 18, 2025
faeae26
fix: consumption-price and production-price are energy prices rather …
Flix6x Mar 18, 2025
6065a5a
fix: more user-focused error message
Flix6x Mar 18, 2025
1f110c7
fix: more common units for Japanese prices
Flix6x Mar 18, 2025
1e9655c
fix: correct price examples for energy prices and capacity prices
Flix6x Mar 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ New features
* Added form modal to edit an asset's ``flex_context`` [see `PR #1320 <https://github.com/FlexMeasures/flexmeasures/pull/1320>`_]
* Better y-axis titles for charts that show multiple sensors with a shared unit [see `PR #1346 <https://github.com/FlexMeasures/flexmeasures/pull/1346>`_]
* Add CLI command ``flexmeasures jobs save-last-failed`` for saving the last failed jobs [see `PR #1342 <https://www.github.com/FlexMeasures/flexmeasures/pull/1342>`_ and `PR #1359 <https://github.com/FlexMeasures/flexmeasures/pull/1359>`_]
* Extended ``DBFlexContextSchema`` validation to include validation for units [see `PR #1364 <https://github.com/FlexMeasures/flexmeasures/pull/1364>`_]

Infrastructure / Support
----------------------
Expand Down
56 changes: 54 additions & 2 deletions flexmeasures/data/schemas/scheduling/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,13 @@
)
from flexmeasures.data.schemas.utils import FMValidationError
from flexmeasures.data.schemas.times import AwareDateTimeField, PlanningDurationField
from flexmeasures.utils.unit_utils import ur, units_are_convertible
from flexmeasures.utils.unit_utils import (
ur,
units_are_convertible,
is_energy_price_unit,
is_power_unit,
is_energy_unit,
)


class FlexContextSchema(Schema):
Expand Down Expand Up @@ -153,6 +159,8 @@ def check_prices(self, data: dict, **kwargs):
f"""Please switch to using `production-price: {{"sensor": {data[field_map["production-price-sensor"]].id}}}`."""
)

# make sure that the prices fields are valid price units

# All prices must share the same unit
data = self._try_to_convert_price_units(data)

Expand Down Expand Up @@ -224,7 +232,6 @@ def _get_variable_quantity_unit(


class DBFlexContextSchema(FlexContextSchema):

@validates_schema
def forbid_time_series_specs(self, data: dict, **kwargs):
"""Do not allow time series specs for the flex-context fields saved in the db."""
Expand Down Expand Up @@ -260,6 +267,51 @@ def forbid_fixed_prices(self, data: dict, **kwargs):
"Fixed prices are not currently supported for production_price in flex-context fields in the DB."
)

@validates_schema
def validate_fields_unit(self, data: dict, **kwargs):
"""Check that each field value has a valid unit."""

self._validate_price_fields(data)
self._validate_power_fields(data)
self._validate_inflexible_device_sensors(data)

def _validate_price_fields(self, data: dict):
"""Validate price fields."""
self._validate_fields(data, "price", is_energy_price_unit)

def _validate_power_fields(self, data: dict):
"""Validate power fields."""
self._validate_fields(data, "power", is_power_unit)

def _validate_fields(self, data: dict, field_type: str, unit_validator):
"""Validate fields based on type and unit validator."""
fields = [field for field in data if field_type in field]
for field in fields:
if field in data:
if isinstance(data[field], str):
if not unit_validator(data[field]):
raise ValidationError(
f"{field_type.capitalize()} field '{field}' must be a {field_type} unit."
)
elif isinstance(data[field], Sensor):
if not unit_validator(data[field].unit):
raise ValidationError(
f"{field_type.capitalize()} field '{field}' must be a {field_type} unit."
)
else:
raise ValidationError(
f"{field_type.capitalize()} field '{field}' must be a fixed value or sensor."
)

def _validate_inflexible_device_sensors(self, data: dict):
"""Validate inflexible device sensors."""
if "inflexible_device_sensors" in data:
for sensor in data["inflexible_device_sensors"]:
if not is_power_unit(sensor.unit) and not is_energy_unit(sensor.unit):
raise ValidationError(
f"Inflexible device sensor '{sensor.id}' must have a power or energy unit."
)


class MultiSensorFlexModelSchema(Schema):
"""
Expand Down