TextDB can walk up directory trees

[schwarzmario]

Currently, TextDB cannot walk up directory trees, i.e. fails when a validity.yaml contains things like

  apply:
    - ../raw/cal/p16/r006/l200-p16-r006-cal-T%-par_raw-overwrite.yaml

which exist in legend-dataprod-overrides (e.g. in https://github.com/legend-exp/legend-dataflow-overrides/blob/main/hit/validity.yaml).

I added a init argument allow_up_tree (default to False), which allows to traverse directory trees upwards, out of the current directory.

I also made the hidden parameter be passed to sub-directories.

[schwarzmario]

@gipert: What do you think?

[gipert]

well trying to access something that lies outside the database doesn't make a lot of sense to me, can't you just fix by using as root path a folder that contains all the data you are interested in?

[schwarzmario]

That also does not make a lot of sense to me either, but that's how legend-dataprod-overrides currently looks like.
Using an upper root folder (e.g. .../dataprod/overrides) does not work, since there is no validity file in there. And once I go to where the validity file is (e.g. in the hit/ subdirectory) I get the error that I cannot go up the tree.

So I only see three options:

• Rewriting the overrides (a mess; apparently DAQ-energy is required on raw and hit level)
• Messing a bit with TextDB
• I miss a very obsious solution

[ggmarshall]

On the overrides, maybe could symlink things? not the most pleasant but might be easiest

[ggmarshall]

potentially I could also modify things on the dataflow side to handle this better which might be the nicest solution

[gipert]

@ggmarshall I think this would be good. I strongly prefer this to implementing hacks in TextDB.

In general the overrides repository is a mess and we should refactor it as we already discussed...

[gipert]

I also made the hidden parameter be passed to sub-directories.

@schwarzmario I would still like to have this in since it's a bug fix...

[schwarzmario]

Sure I'll cherrypick the fix.

I prefer a solution, where one can still use TextDB (or similar) stand-alone on the overrides. I currently use this to verify if SiPM calibrations still apply for later runs (otherwise, I would have to manually add all overwrite files...).

[schwarzmario]

here you are: #40