Skip to content

1111 return stac11 items #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from
+329 −14
Draft

1111 return stac11 items #411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
488e95c
add field for items to metadata
jdries Jun 5, 2025
26c99fd
basics to return new-style item links if they exist
jdries Jun 5, 2025
24a6a29
basics to return new-style item links if they exist
jdries Jun 5, 2025
9398d35
add test skeletons for results Collection and items
bossie Sep 15, 2025
67301fc
Merge branch 'master' into 1111_return_stac11_items
bossie Sep 15, 2025
9374765
issue 1111 get_result_metadata also returns items
ElienVandermaesenVITO Oct 2, 2025
3df0a5e
issue 1111 get_result_metadata also returns items
ElienVandermaesenVITO Oct 2, 2025
7aeca37
issue 1111 improve exception
ElienVandermaesenVITO Oct 2, 2025
eea9cb1
issue 1111 add signing to assets in items
ElienVandermaesenVITO Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ and start a new "In Progress" section above it.
- `DiskWorkspace`: support unified asset keys ([Open-EO/openeo-geopyspark-driver#1111](https://github.com/Open-EO/openeo-geopyspark-driver/issues/1111))
- Support persisting results metadata URI in job registry ([Open-EO/openeo-geopyspark-driver#1255](https://github.com/Open-EO/openeo-geopyspark-driver/issues/1255))
- More fine-grained `convert_node` cache control ([Open-EO/openeo-geopyspark-driver#1331](https://github.com/Open-EO/openeo-geopyspark-driver/issues/1331)/[#422](https://github.com/Open-EO/openeo-python-driver/pull/422))
- `get_result_metadata` can return items ([Open-EO/openeo-geopyspark-driver#1111](https://github.com/Open-EO/openeo-geopyspark-driver/issues/1111))


## 0.134.0
Expand Down
68 changes: 68 additions & 0 deletions openeo_driver/dummy/dummy_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -900,6 +900,74 @@ def get_result_assets(self, job_id: str, user_id: str) -> Dict[str, dict]:
},
},
}
elif job_id == "07024ee9-7847-4b8a-b260-6c879a2b3cdd":
return {
"5d2db643-5cc3-4b27-8ef3-11f7d203b221_2023-12-31T21:41:00Z": {
"geometry": {
"coordinates": [[
[3.359808992021044,51.08284561357965],
[3.359808992021044,51.88641704215104],
[4.690166134878123,51.88641704215104],
[4.690166134878123,51.08284561357965],
[3.359808992021044,51.08284561357965]
]],
"type": "Polygon",
},
"href": f"{self._output_root()}/{job_id}/5d2db643-5cc3-4b27-8ef3-11f7d203b221_2023-12-31T21:41:00Z",
"assets": {
"openEO": {
"datetime": "2023-12-31T21:41:00Z",
"roles": ["data"],
"bbox": [
3.359808992021044,
51.08284561357965,
4.690166134878123,
51.88641704215104
],
"geometry": {
"coordinates": [[
[3.359808992021044,51.08284561357965],
[3.359808992021044,51.88641704215104],
[4.690166134878123,51.88641704215104],
[4.690166134878123,51.08284561357965],
[3.359808992021044,51.08284561357965]
]],
"type": "Polygon"
},
"href": "s3://openeo-data-staging-waw4-1/batch_jobs/j-250605095828442799fdde3c29b5b047/openEO_20231231T214100Z.tif",
"nodata": "nan",
"type": "image/tiff; application=geotiff",
"bands": [{
"name": "LST",
"common_name": "surface_temperature",
"aliases": [
"LST_in:LST"
]
}],
"raster:bands": [{
"name": "LST",
"statistics": {
"valid_percent": 66.88,
"maximum": 281.04800415039,
"stddev": 19.598456945276,
"minimum": 224.46798706055,
"mean": 259.57087672984
}
}]
}
},
"id": "5d2db643-5cc3-4b27-8ef3-11f7d203b221_2023-12-31T21:41:00Z",
"properties": {
"datetime": "2023-12-31T21:41:00Z"
},
"bbox": [
3.359808992021044,
51.08284561357965,
4.690166134878123,
51.88641704215104
]
}
}
# default return:
return {
"output.tiff": {
Expand Down
77 changes: 63 additions & 14 deletions openeo_driver/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1160,16 +1160,20 @@ def job_results_canonical_url() -> str:
if TREAT_JOB_RESULTS_V100_LIKE_V110 or requested_api_version().at_least("1.1.0"):
ml_model_metadata = None

def job_result_item_url(item_id) -> str:
def job_result_item_url(item_id, is11 = False) -> str:
signer = get_backend_config().url_signer

method_start = ".get_job_result_item"
if is11:
method_start = method_start + "11"
if not signer:
return url_for(".get_job_result_item", job_id=job_id, item_id=item_id, _external=True)
return url_for(method_start, job_id=job_id, item_id=item_id, _external=True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to avoid a new endpoint (/jobs/<job_id>/results/items11/<item_id>)? I suppose the original _get_job_result_item method can also check whether the item ID corresponds to either:

  • an item derived from an asset (old case) or;
  • an actual item generated in the batch job (new case aka stac11).


expires = signer.get_expires()
secure_key = signer.sign_job_item(job_id=job_id, user_id=user_id, item_id=item_id, expires=expires)
user_base64 = user_id_b64_encode(user_id)
return url_for(
".get_job_result_item_signed",
method_start + "_signed",
job_id=job_id,
user_base64=user_base64,
secure_key=secure_key,
Expand All @@ -1178,19 +1182,32 @@ def job_result_item_url(item_id) -> str:
_external=True,
)

for filename, metadata in result_assets.items():
if ("data" in metadata.get("roles", []) and
any(media_type in metadata.get("type", "") for media_type in
["geotiff", "netcdf", "text/csv", "application/parquet"])):
links.append(
{"rel": "item", "href": job_result_item_url(item_id=filename), "type": stac_item_media_type}
)
elif metadata.get("ml_model_metadata", False):
# TODO: Currently we only support one ml_model per batch job.
ml_model_metadata = metadata

if len(result_metadata.items) > 0 :
for item_id, metadata in result_metadata.items.items():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for item_id, metadata in result_metadata.items.items():
for item_id in result_metadata.items.keys():

links.append(
{"rel": "item", "href": job_result_item_url(item_id=filename), "type": "application/json"}
{"rel": "item", "href": job_result_item_url(item_id=item_id, is11=True), "type": stac_item_media_type}
)
for asset_key, asset in metadata.get("assets").items():
links.append(
{"rel": "item", "href": job_result_item_url(item_id=asset.get("href")),
"type": stac_item_media_type}
)
else:

for filename, metadata in result_assets.items():
if ("data" in metadata.get("roles", []) and
any(media_type in metadata.get("type", "") for media_type in
["geotiff", "netcdf", "text/csv", "application/parquet"])):
links.append(
{"rel": "item", "href": job_result_item_url(item_id=filename), "type": stac_item_media_type}
)
elif metadata.get("ml_model_metadata", False):
# TODO: Currently we only support one ml_model per batch job.
ml_model_metadata = metadata
links.append(
{"rel": "item", "href": job_result_item_url(item_id=filename), "type": "application/json"}
)

result = dict_no_none(
{
Expand Down Expand Up @@ -1357,11 +1374,43 @@ def get_job_result_item_signed(job_id, user_base64, secure_key, item_id):
signer.verify_job_item(signature=secure_key, job_id=job_id, user_id=user_id, item_id=item_id, expires=expires)
return _get_job_result_item(job_id, item_id, user_id)

@api_endpoint
@blueprint.route('/jobs/<job_id>/results/items11/<user_base64>/<secure_key>/<item_id>', methods=['GET'])
def get_job_result_item11_signed(job_id, user_base64, secure_key, item_id):
expires = request.args.get('expires')
signer = get_backend_config().url_signer
user_id = user_id_b64_decode(user_base64)
signer.verify_job_item(signature=secure_key, job_id=job_id, user_id=user_id, item_id=item_id, expires=expires)
return _get_job_result_item11(job_id, item_id, user_id)

@blueprint.route('/jobs/<job_id>/results/items/<item_id>', methods=['GET'])
@auth_handler.requires_bearer_auth
def get_job_result_item(job_id: str, item_id: str, user: User) -> flask.Response:
return _get_job_result_item(job_id, item_id, user.user_id)

@api_endpoint(version=ComparableVersion("1.1.0").or_higher)
@blueprint.route('/jobs/<job_id>/results/items11/<item_id>', methods=['GET'])
@auth_handler.requires_bearer_auth
def get_job_result_item11(job_id: str, item_id: str, user: User) -> flask.Response:
return _get_job_result_item11(job_id, item_id, user.user_id)

def _get_job_result_item11(job_id, item_id, user_id):
if item_id == DriverMlModel.METADATA_FILE_NAME:
return _download_ml_model_metadata(job_id, item_id, user_id)
Comment on lines +1398 to +1399
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this ever be the case?


metadata = backend_implementation.batch_jobs.get_result_metadata(
job_id=job_id, user_id=user_id
)

if item_id not in metadata.items:
raise OpenEOApiException("Item with id {item_id!r} not found in job {job_id!r}".format(item_id=item_id, job_id=job_id), status_code=404)
item_metadata = metadata.items.get(item_id,None)

resp = jsonify(item_metadata)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basic, will probably require more work e.g. sign its asset URLs?

resp.mimetype = stac_item_media_type
return resp


def _get_job_result_item(job_id, item_id, user_id):
if item_id == DriverMlModel.METADATA_FILE_NAME:
return _download_ml_model_metadata(job_id, item_id, user_id)
Expand Down
Loading