Repository navigation
Make TCX/GPX parsing schema-aware - #2
Merged
Merged
Conversation
Replace the namespace-blind XML flatten + name-heuristic numeric coercion with declarative field tables derived from the TCX/GPX/extension XSDs (xml_fields.py). parse_tcx/parse_gpx now emit canonical column names directly, unrecognized elements are kept as namespace-qualified string columns instead of colliding or being dropped, and scripts/inspect_schemas.py audits a file or directory for schema coverage and coercion failures. This is Claude's implementation against the agreed plan, not yet reviewed by a human. Expect follow-up commits addressing review feedback before this branch is merged via PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XmlField.convert now holds a callable (Series -> Series) instead of an enum tag that build_dataframe branched on. NUMERIC/NUMERIC_M_TO_KM/ NUMERIC_MS_TO_KMH/to_datetime are the converters themselves, collapsing build_dataframe's dispatch to a single call per column. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trim walk_fields' docstring to its contract instead of restating FieldPath's own type documentation and _walk's internals. Add short comments/docstrings to _walk, _row_from_fields, and _index_by_time covering the xsi:type skip, the leaf-vs-container assumption, and each private helper's job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The audit script adds review surface area beyond the core schema-aware parsing change. Split it out to inspect-schemas-script (stacked on this branch) as a follow-up PR; drop its now-unused NUMERIC_CONVERTERS export and the pyright extraPaths entry it needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that inspect_schemas.py (its other consumer) lives on its own branch, column_converts had exactly one caller within this branch; fold its collapsing loop directly into build_dataframe instead of keeping it as a separately named/tested function. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hdop/vdop/pdop/satellites/fix_type/dgps_*/magnetic_variation/ geoid_height and the static text labels (name/cmt/desc/src/sym/type/ url/urlname) are schema-known but aren't fitness data, so ActivityParser no longer surfaces them by default. They're still available via the low-level parse_gpx for anyone who needs them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… type attrs Reviewing test coverage surfaced three ways parse_tcx/parse_gpx could silently lose data despite the "unrecognized extensions aren't dropped" design goal: unrelated elements sharing a leaf name collided into one column, elements with both text and children lost the children, and any attribute merely named "type" was skipped regardless of namespace. Colliding paths now fall back to a full-path column name instead of overwriting, mixed content keeps both text and children, and only genuine xsi:type is skipped.
…path extract_xml_fields (used for the extra metadata dict) had the identical namespace-blind "type" attribute check as _walk, just in a separate code path.
TrainerRoad's TCX export resets AvgSpeed to no namespace even though its LX wrapper is correctly namespaced, so it fell through as an unrecognized column. When a leaf has no namespace and its exact path doesn't match, retry once using the immediate parent's namespace instead of guessing more broadly; a leaf that still doesn't match a real field stays unrecognized.
Moves schema-version/extension support and unrecognized-column handling out of scattered prose into a single "TCX & GPX files" table, and drops per-fixture behavior descriptions from tests/files/README.md that duplicated each fixture's own comment and its referencing test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites TCX/GPX parsing (
parse_tcx_gpx.py) to be schema-derived. Elements/attributes are matched against declarative field tables (xml_fields.py) transcribed from the TCX v2, GPX 1.0/1.1, and known Garmin/Cluetrust extension schemas, rather than matched by bare localname substrings.{*}-wildcard/localname-substring approach, eliminating cross-namespace collision risk.xsi:typeskip is now namespace-aware, rather than matching any attribute merely namedtype.gpx_records_selectortrimmed to drop GPS-fix-quality diagnostics and static descriptive labels that aren't fitness data (still available via low-levelparse_gpx).Test plan
uv run pytest— 66 passinguv run ruff check/ruff format --checkcleanuv run pyrightclean