Exporting non exsisting fields results in an error #20
Description
When exporting fields which do not exist in the database a user friendly warning is shown.
But later on it fails anyway.
sh ./product_template_export.sh
[17:36:28] INFO Starting export for model 'product.template'... exporter.py:48
INFO Connecting to Odoo server at odoo.-international.com... conf_lib.py:40
WARNING Field 'document_file' (base: 'document_file') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
WARNING Field 'exclude_for' (base: 'exclude_for') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
WARNING Field 'file_name' (base: 'file_name') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
WARNING Field 'html_color' (base: 'html_color') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
WARNING Field 'is_custom' (base: 'is_custom') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
WARNING Field 'price_extra' (base: 'price_extra') not found on model 'product.template'. An empty column will be created. export_threaded.py:321
INFO Hybrid export mode activated. Using 'read' with XML ID enrichment. export_threaded.py:601
INFO Starting new export session: 42a72d080acdd384 export_threaded.py:661
INFO Searching for records to export in model 'product.template'... export_threaded.py:662
[17:36:29] INFO Processing 21458 records in batches of 1000. export_threaded.py:723
[17:36:34] WARNING Batch 0 failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:36:39] WARNING Batch 0-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:36:44] WARNING Batch 0-a-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:37:06] WARNING Batch 0-a-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:37:26] WARNING Batch 0-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:37:31] WARNING Batch 0-b-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:37:56] ERROR A task in a worker thread failed: cannot cast List type (inner: 'Null', to: 'String') export_threaded.py:504
╭───────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/bosd/git/odoo-data-flow/src/odoo_data_flow/export_threaded.py:473 in _process_export_batches │
│ │
│ 470 │ │ │ │ │ if df.is_empty(): │
│ 471 │ │ │ │ │ │ continue │
│ 472 │ │ │ │ │ │
│ ❱ 473 │ │ │ │ │ final_batch_df = _clean_and_transform_batch( │
│ 474 │ │ │ │ │ │ df, field_types, polars_schema │
│ 475 │ │ │ │ │ ) │
│ 476 │
│ │
│ /home/bosd/git/odoo-data-flow/src/odoo_data_flow/export_threaded.py:137 in _format_batch_results │
│ │
│ 134 │ │ │ │ if field in record: │
│ 135 │ │ │ │ │ value = record[field] │
│ 136 │ │ │ │ │ if isinstance(value, (list, tuple)) and value: │
│ ❱ 137 │ │ │ │ │ │ new_record[field] = value[1] │
│ 138 │ │ │ │ │ else: │
│ 139 │ │ │ │ │ │ new_record[field] = value │
│ 140 │ │ │ │ else: │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range
[17:40:22] ERROR Export for batch 2-a-b-b failed permanently: list index out of range export_threaded.py:275
╭───────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/bosd/_12_18/.venv/lib64/python3.13/site-packages/httpx/_transports/default.py:101 in map_httpcore_exceptions │
│ │
│ 98 │ if len(HTTPCORE_EXC_MAP) == 0: │
│ 99 │ │ HTTPCORE_EXC_MAP = _load_httpcore_exceptions() │
│ 100 │ try: │
│ ❱ 101 │ │ yield │
│ 102 │ except Exception as exc: │
│ 103 │ │ mapped_exc = None │
│ 104 │
│ │
│ /home/bosd/_12_18/.venv/lib64/python3.13/site-packages/httpx/_transports/default.py:250 in handle_request │
│ │
│ 247 │ │ │ extensions=request.extensions, │
│ 248 │ │ ) │
│ 249 │ │ with map_httpcore_exceptions(): │
│ ❱ 250 │ │ │ resp = self._pool.handle_request(req) │
│ 251 │ │ │
│ 252 │ │ assert isinstance(resp.stream, typing.Iterable) │
│ 253
│ │
│ /usr/lib64/python3.13/contextlib.py:162 in __exit__ │
│ │
│ 159 │ │ │ │ # tell if we get the same exception back │
│ 160 │ │ │ │ value = typ() │
│ 161 │ │ │ try: │
│ ❱ 162 │ │ │ │ self.gen.throw(value) │
│ 163 │ │ │ except StopIteration as exc: │
│ 164 │ │ │ │ # Suppress StopIteration *unless* it's the same exception that │
│ 165 │ │ │ │ # was passed to throw(). This prevents a StopIteration │
│ ] │
│ 333 │ │ return wrapper │
│ │
│ /home/bosd/_12_18/.venv/lib64/python3.13/site-packages/polars/lazyframe/frame.py:2407 in collect │
│ │
│ 2404 │ │ │
│ 2405 │ │ # Only for testing purposes │
│ 2406 │ │ callback = _kwargs.get("post_opt_callback", callback) │
│ ❱ 2407 │ │ return wrap_df(ldf.collect(engine, callback)) │
│ 2408 │ │
│ 2409 │ @overload │
│ 2410 │ def collect_async( │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
InvalidOperationError: cannot cast List type (inner: 'Null', to: 'String')
[17:41:03] WARNING Batch 3 failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:41:08] WARNING Batch 3-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:41:14] WARNING Batch 3-a-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:41:34] WARNING Batch 3-a-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:41:53] WARNING Batch 3-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:41:58] WARNING Batch 3-b-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:42:20] WARNING Batch 3-b-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:42:35] ERROR A task in a worker thread failed: cannot cast List type (inner: 'Null', to: 'String') export_threaded.py:504
╭───────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/bosd/git/odoo-data-flow/src/odoo_data_flow/export_threaded.py:473 in _process_export_batches │
│ │
│
│ 2405 │ │ # Only for testing purposes │
│ 2406 │ │ callback = _kwargs.get("post_opt_callback", callback) │
│ ❱ 2407 │ │ return wrap_df(ldf.collect(engine, callback)) │
│ 2408 │ │
│ 2409 │ @overload │
│ 2410 │ def collect_async( │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
InvalidOperationError: cannot cast List type (inner: 'Null', to: 'String')
[17:42:40] WARNING Batch 4 failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:42:46] WARNING Batch 4-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:43:01] WARNING Batch 4-a-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:43:21] WARNING Batch 4-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:43:26] WARNING Batch 4-b-a failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:43:47] WARNING Batch 4-b-b failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying. export_threaded.py:161
[17:44:03] ERROR A task in a worker thread failed: cannot cast List type (inner: 'Null', to: 'String') export_threaded.py:504
InvalidOperationError: cannot cast List type (inner: 'Null', to: 'String')
[17:44:08] WARNING Batch 5 failed with a network error (The read operation timed out). This is often a server timeout on large batches. Automatically splitting the batch and retrying.
Fix this error and add a test to prevent future regression