You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#328 promotes TRY facts into species columns under one rule: fill "only when the column is empty — never overwriting an existing value, whatever its source". That rule was written against the first TRY export, whose traits landed in columns nothing else fills.
The second export changes the situation, and for one group of columns the rule silently does nothing.
light, ground_humidity, soil_nutriments and soil_salinity hold Ellenberg-like values from Baseflor/Catminat. The incoming TRY facts are the actual Ellenberg indicators (1131-1136), for 3,900-4,600 species each.
Apply #328's rule unchanged and the outcome is: wherever Baseflor has a value, TRY is discarded; wherever it does not, TRY lands. That may well be the right default — Baseflor is a curated regional flora and TRY is an aggregation of contributed datasets — but right now it is an accident of a rule written for a different situation, and it throws away every disagreement without recording one.
Two sources offering the same kind of value for the same species is an arbitration problem. lib/ingester/ already exists to arbitrate sources; this is its job, not a migrator's.
#328 enumerates what it promotes: "colors, sexuality, inflorescence, fruit shape, ligneous/growth form, growth rate, lifespan/duration, nitrogen fixation, average height, C/N ratio". Export 52176 adds destinations that are not in it:
db/schema.rb:226 declares shade_tolerance as string(255) on species. config/traits.yml has no entry for it. A column outside the contract has no declared vocabulary, no plausible range, and nothing validating what gets written to it — which is part of why it has stayed empty.
#31 ("No shade_tolerance / root_depth_minimum") has been open since 2020 and this export is the first data we have had for either half of it. Before anything fills the column it needs a traits.yml entry with an allowed_values list. nitrogen_fixation: { category: ecology, allowed_values: [None, Low, Medium, High] } is the precedent for the shape; check what the column holds today before choosing the vocabulary.
c_n_ratio is a string column receiving a number
c_n_ratio is string(255), category: usda_legacy. TRY trait 146 gives a numeric ratio in g g⁻¹ for 6,516 species. #328 already lists C/N among what it promotes, so today's answer is "format the number into the string column" — which leaves a numeric trait unfilterable and unrangeable, and quietly changes what a legacy column contains.
Worth deciding here rather than in the crawlers repo: keep the string, or add a numeric column beside it and deprecate the string per the "no silent breaking changes" rule #350 sets out.
Scope
An arbitration rule for the ecology indices, living where source arbitration lives (lib/ingester/), not as a special case inside a migrator. State the precedence and the reason.
traits.yml entries for shade_tolerance (with allowed_values) and for minimum_root_depth_cm promotion, plus whatever 1134/1136 need if the crawlers side decides to promote them.
A decision on c_n_ratio's column shape, with a migration if it changes.
Touches: lib/ingester/, config/traits.yml, app/workers/migrators/, lib/checks/, db/migrate/, spec/
Blocked by: treflehq/trefle-crawlers#14, #15 and #16 (the facts must exist first — cross-repo, so not a native edge). Do not label ready-for-agent until those imports have run.
#328 promotes TRY facts into species columns under one rule: fill "only when the column is empty — never overwriting an existing value, whatever its source". That rule was written against the first TRY export, whose traits landed in columns nothing else fills.
The second export changes the situation, and for one group of columns the rule silently does nothing.
The ecology columns are not empty
config/traits.ymlsays so in its own comment:light,ground_humidity,soil_nutrimentsandsoil_salinityhold Ellenberg-like values from Baseflor/Catminat. The incoming TRY facts are the actual Ellenberg indicators (1131-1136), for 3,900-4,600 species each.Apply #328's rule unchanged and the outcome is: wherever Baseflor has a value, TRY is discarded; wherever it does not, TRY lands. That may well be the right default — Baseflor is a curated regional flora and TRY is an aggregation of contributed datasets — but right now it is an accident of a rule written for a different situation, and it throws away every disagreement without recording one.
Two sources offering the same kind of value for the same species is an arbitration problem.
lib/ingester/already exists to arbitrate sources; this is its job, not a migrator's.Columns outside #328's list
#328 enumerates what it promotes: "colors, sexuality, inflorescence, fruit shape, ligneous/growth form, growth rate, lifespan/duration, nitrogen fixation, average height, C/N ratio". Export 52176 adds destinations that are not in it:
light,ground_humidity,soil_nutriments,soil_salinityminimum_root_depth_cmshade_tolerancetraits.ymlentirelybloom_months,fruit_monthszero_means_empty: trueshade_toleranceis outside the data contractdb/schema.rb:226declaresshade_toleranceasstring(255)onspecies.config/traits.ymlhas no entry for it. A column outside the contract has no declared vocabulary, no plausible range, and nothing validating what gets written to it — which is part of why it has stayed empty.#31 ("No shade_tolerance / root_depth_minimum") has been open since 2020 and this export is the first data we have had for either half of it. Before anything fills the column it needs a
traits.ymlentry with anallowed_valueslist.nitrogen_fixation: { category: ecology, allowed_values: [None, Low, Medium, High] }is the precedent for the shape; check what the column holds today before choosing the vocabulary.c_n_ratiois a string column receiving a numberc_n_ratioisstring(255),category: usda_legacy. TRY trait 146 gives a numeric ratio in g g⁻¹ for 6,516 species. #328 already lists C/N among what it promotes, so today's answer is "format the number into the string column" — which leaves a numeric trait unfilterable and unrangeable, and quietly changes what a legacy column contains.Worth deciding here rather than in the crawlers repo: keep the string, or add a numeric column beside it and deprecate the string per the "no silent breaking changes" rule #350 sets out.
Scope
lib/ingester/), not as a special case inside a migrator. State the precedence and the reason.traits.ymlentries forshade_tolerance(withallowed_values) and forminimum_root_depth_cmpromotion, plus whatever 1134/1136 need if the crawlers side decides to promote them.c_n_ratio's column shape, with a migration if it changes.0to azero_means_emptycolumn.Acceptance criteria
shade_tolerancehas atraits.ymlentry, and a write outside its vocabulary fails a spec.Touches:
lib/ingester/,config/traits.yml,app/workers/migrators/,lib/checks/,db/migrate/,spec/Blocked by: treflehq/trefle-crawlers#14, #15 and #16 (the facts must exist first — cross-repo, so not a native edge). Do not label ready-for-agent until those imports have run.