Fix validation checks for duplicate keys

[puneeter]

Description

Resolves #4088
Also linked #4077

Development notes

• The OmegaConfigloader now validates the keys to the most granular level (creates . separated keys to validate duplication)

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[merelcht]

Thanks for the PR @puneeter and also for giving more context on Slack (https://kedro-org.slack.com/archives/C03RKP2LW64/p1723193517960259). I've left some suggestions to improve the PR. Most importantly, I think we should change the validation to only check parameters and not all other config type files.

[merelcht]

Thanks @puneeter ! This looks great now ⭐

[noklam]

isn't config already a DictConfig? why do you need to convert it again?

[puneeter]

So, DictConfig .items() method resolves the values and that's not what I want. So, I convert the DictConfig to an unresolved dictionary. Let me know if you have some alternative/better way to handle this.

[noklam]

Can you give me an example with just DictConfig? I am not sure if I am following, if config is resolved here already how would converting it back to DictConfig helps?

[puneeter]

See the line 361 in this module. We are doing an iteration over the dictionary keys and values. Copying the function here:

    def _get_all_keys(self, cfg: Any, parent_key: str = "") -> set[str]:
        keys: set[str] = set()

        for key, value in cfg.items():
            full_key = f"{parent_key}.{key}" if parent_key else key
            if isinstance(value, dict):
                keys.update(self._get_all_keys(value, full_key))
            else:
                keys.add(full_key)
        return keys

With cfg being DictConfig, the iteration would resolve the interpolations, e.g, ${test_env} in one of our tests. But if I do OmegaConf.to_container(cfg, resolve=False) it will return a python dict and not DictConfig and since I use resolve=False it won't resolve the interpolations.

[noklam]

Thanks, this make sense!

[noklam]

Looks good in general, left some comment.

[noklam]

Thanks for the push and making all the necessary changes, this looks good to me now!

Thank you! ✨