`ucd_json.ucd_json` map page errors `AttributeError: 'list' object has no attribute 'get'` - UCD needs to expose thumbnails

[christinklez]

5 collections with map page errors.

28059:

• registry: https://registry.cdlib.org/admin/library_collection/collection/28059
• rikolti: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/grid?dag_run_id=manual__2026-02-24T17%3A19%3A05%2B00%3A00&tab=mapped_tasks&task_id=mapping.map_page&base_date=2026-02-24T17%3A19%3A05%2B00%3A00

27548:

• registry: https://registry.cdlib.org/admin/library_collection/collection/27548
• rikolti: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/grid?dag_run_id=manual__2026-02-24T18%3A57%3A11%2B00%3A00&tab=mapped_tasks&task_id=mapping.map_page&map_index=0&base_date=2026-02-24T18%3A57%3A11%2B00%3A00

27275:

• registry: https://registry.cdlib.org/admin/library_collection/collection/27275
• rikolti: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/grid?dag_run_id=manual__2026-02-24T19%3A40%3A37%2B00%3A00&tab=mapped_tasks&task_id=mapping.map_page&base_date=2026-02-24T19%3A40%3A37%2B00%3A00

27307:

• registry: https://registry.cdlib.org/admin/library_collection/collection/27307
• rikolti: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/grid?dag_run_id=manual__2026-02-24T19%3A54%3A27%2B00%3A00&tab=details&task_id=mapping.map_page&base_date=2026-02-24T19%3A54%3A27%2B00%3A00

28457:

• registry: https://registry.cdlib.org/admin/library_collection/collection/28457
• rikolti: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/grid?dag_run_id=manual__2026-02-24T21%3A14%3A38%2B00%3A00&tab=logs&task_id=mapping.map_page&base_date=2026-02-24T21%3A14%3A38%2B00%3A00

Example error message from 28059:

[2026-02-24, 17:21:17 UTC] {taskinstance.py:2905} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 401, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py", line 265, in execute
    return_value = super().execute(context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 401, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 235, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 252, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/rikolti/dags/shared_tasks/mapping_tasks.py", line 80, in map_page_task
    mapped_page_status = map_page(
                         ^^^^^^^^^
  File "/usr/local/airflow/dags/rikolti/metadata_mapper/lambda_function.py", line 125, in map_page
    source_metadata_records = source_vernacular.parse(api_resp)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/rikolti/metadata_mapper/mappers/ucd_json/ucd_json_mapper.py", line 116, in parse
    return self.get_records(records)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/rikolti/metadata_mapper/mappers/mapper.py", line 28, in get_records
    return [
           ^
  File "/usr/local/airflow/dags/rikolti/metadata_mapper/mappers/mapper.py", line 30, in <listcomp>
    for record in records if not self.skip(record)
                                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/rikolti/metadata_mapper/mappers/ucd_json/ucd_json_mapper.py", line 102, in skip
    thumbnail_url = record.get("associatedMedia", {}).get('@id')
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'get'

[barbarahui]

OK, so I think there are a few problems with the metadata that UCD is providing for these PDF objects.

We get the thumbnail url by first looking for image/@id. If that doesn't exist, then we look for associatedMedia/@id.

For collection 28059, there is no image/@id. There is an associatedMedia/@id element, but it looks like this:

        "associatedMedia": [
            {
                "@id": "/item/ark:/87293/d3xm01/media/text"
            },
            {
                "@id": "/item/ark:/87293/d3xm01/media/images"
            },
            {
                "@id": "/item/ark:/87293/d3xm01/media/d3xm01.pdf"
            }
        ],

How do we get the thumbnail url from this? Are there thumbnail urls for PDF objects?

Also, the data that UCD is giving to us is not of a consistent structure. The records for the collections in this ticket contain a list of associatedMedia elements, whereas those with a single associatedMedia/@id element contain a dictionary, e.g. this record from collection 28456:

        "associatedMedia": {
            "@id": "/item/ark:/87287/d7x66t/media/Summer_1999.pdf"
        },

This kind of data inconsistency is a bit of a pain to code around. What's our policy on this? I don't suppose we can ask the contributor to regenerate the metadata to make the structure consistent?

The other issue with the above record from collection 28456 is that the url points to a pdf, which isn't a valid thumbnail file.

[christinklez]

I looked into these and my notes are here: https://docs.google.com/document/d/1ZI510k3R3Pg_SpWTOekuLZY-WxPdf81Q2somytAwsrM/edit?tab=t.ryo7zdxgeoht#heading=h.axnkpcdwu4bt

It looks like UCD is serving image thumbnails (displays on their public site) - I've asked them to expose those thumbnail filenames in their sitemaps! Closing this issue.