Skip to content

TRY round 2: arbitrate the ecology indices against Baseflor, and bring shade tolerance into the contract #373

Description

@lambda2

#328 promotes TRY facts into species columns under one rule: fill "only when the column is empty — never overwriting an existing value, whatever its source". That rule was written against the first TRY export, whose traits landed in columns nothing else fills.

The second export changes the situation, and for one group of columns the rule silently does nothing.

The ecology columns are not empty

config/traits.yml says so in its own comment:

# --- Ecology (light/humidity/soil indices are 0-10 Ellenberg-like scales
#     from Baseflor/Catminat: 0 is a meaningful value, never "empty") ---
light:                { category: ecology, unit: index_0_10, plausible_range: [0, 10] }
atmospheric_humidity: { category: ecology, unit: index_0_10, plausible_range: [0, 10] }
ground_humidity:      { category: ecology, unit: index_0_10, plausible_range: [0, 10] }
soil_salinity:        { category: ecology, unit: index_0_10, plausible_range: [0, 10] }
soil_nutriments:      { category: ecology, unit: index_0_10, plausible_range: [0, 10] }

light, ground_humidity, soil_nutriments and soil_salinity hold Ellenberg-like values from Baseflor/Catminat. The incoming TRY facts are the actual Ellenberg indicators (1131-1136), for 3,900-4,600 species each.

Apply #328's rule unchanged and the outcome is: wherever Baseflor has a value, TRY is discarded; wherever it does not, TRY lands. That may well be the right default — Baseflor is a curated regional flora and TRY is an aggregation of contributed datasets — but right now it is an accident of a rule written for a different situation, and it throws away every disagreement without recording one.

Two sources offering the same kind of value for the same species is an arbitration problem. lib/ingester/ already exists to arbitrate sources; this is its job, not a migrator's.

Columns outside #328's list

#328 enumerates what it promotes: "colors, sexuality, inflorescence, fruit shape, ligneous/growth form, growth rate, lifespan/duration, nitrogen fixation, average height, C/N ratio". Export 52176 adds destinations that are not in it:

Column Source trait Note
light, ground_humidity, soil_nutriments, soil_salinity Ellenberg 1131-1135 the arbitration case above
minimum_root_depth_cm 6 root depth empty today; answers #31
shade_tolerance 603 absent from traits.yml entirely
bloom_months, fruit_months 2956, 4333 bitmasks, zero_means_empty: true

shade_tolerance is outside the data contract

db/schema.rb:226 declares shade_tolerance as string(255) on species. config/traits.yml has no entry for it. A column outside the contract has no declared vocabulary, no plausible range, and nothing validating what gets written to it — which is part of why it has stayed empty.

#31 ("No shade_tolerance / root_depth_minimum") has been open since 2020 and this export is the first data we have had for either half of it. Before anything fills the column it needs a traits.yml entry with an allowed_values list. nitrogen_fixation: { category: ecology, allowed_values: [None, Low, Medium, High] } is the precedent for the shape; check what the column holds today before choosing the vocabulary.

c_n_ratio is a string column receiving a number

c_n_ratio is string(255), category: usda_legacy. TRY trait 146 gives a numeric ratio in g g⁻¹ for 6,516 species. #328 already lists C/N among what it promotes, so today's answer is "format the number into the string column" — which leaves a numeric trait unfilterable and unrangeable, and quietly changes what a legacy column contains.

Worth deciding here rather than in the crawlers repo: keep the string, or add a numeric column beside it and deprecate the string per the "no silent breaking changes" rule #350 sets out.

Scope

  • An arbitration rule for the ecology indices, living where source arbitration lives (lib/ingester/), not as a special case inside a migrator. State the precedence and the reason.
  • A divergence report where TRY and Baseflor disagree on the same column, routed to the corrections queue — Promote TRY trait facts into empty species columns, with attribution #328 already asks for this shape of check; extend it rather than duplicating.
  • traits.yml entries for shade_tolerance (with allowed_values) and for minimum_root_depth_cm promotion, plus whatever 1134/1136 need if the crawlers side decides to promote them.
  • A decision on c_n_ratio's column shape, with a migration if it changes.
  • Extend Promote TRY trait facts into empty species columns, with attribution #328's covered-column list, or supersede that part of it — do not leave two lists that can drift.
  • Nothing here writes 0 to a zero_means_empty column.

Acceptance criteria

Touches: lib/ingester/, config/traits.yml, app/workers/migrators/, lib/checks/, db/migrate/, spec/
Blocked by: treflehq/trefle-crawlers#14, #15 and #16 (the facts must exist first — cross-repo, so not a native edge). Do not label ready-for-agent until those imports have run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    data-integrityRelative to the data qualityenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions