[ENH] Test framework for ptf-v2

[phoeenniixx]

Description

This PR solves #1838
Implements Test framework for v2

[phoeenniixx]

Hi @fkiraly, @PranavBhatP, I tried adding _get_test_datamodule_from to the metadata class, please review it and tell me if I am on a right path

[phoeenniixx]

• I added the whole initialisation logic of the datamodule to _get_test_datamodule_from so that any kind of data module can be added to tests without making any changes to _integration
• changed the make_dataloaders_v2 to make_datasets_v2 as now it is returning TimeSeries dataset in place of whole dataloaders and datamodule - now _get_test_datamodule_from returns the datamodules and dataloaders.
• now the _integration test only checks if metadata is a dict.

The open issue is: how to do metadata testing for different datamodules?
By metadata testing I mean testing if the metadata is calcualted correctly and passed correctly to dataloaders and model for initialisation.

[fkiraly]

The open issue is: how to do metadata testing for different datamodules?
By metadata testing I mean testing if the metadata is calcualted correctly and passed correctly to dataloaders and model for initialisation.

Hmmm - here is an idea: each model (intermediate base class) could have a _check_metadata that is called from the test.
It passes if the metadata is of expected type, and fails otherwise with a descriptive exception.

For performance reasons, I would not carry out the check in usage, just in the test.

For the metadata generated by D1, this is universal and can simply be tested by D1 tests - all D1 should produce the same metadata.

[phoeenniixx]

each model (intermediate base class) could have a _check_metadata that is called from the test.
It passes if the metadata is of expected type, and fails otherwise with a descriptive exception.

Sorry I didn't quite understand what you mean here.

• If you mean we add _check_metadata to the base class _BasePtForecasterV2 (or its child class) how to add different tests for different metadata - Imagine one model is from EncoderDecoderDataModule and other from TSLibDataModule then, how would _check_metadata would know which metadata to test? We'll need to add the testing logic of both the data modules to the same method, but what if there is one more data module in future? I think that will make the method little bit messy?

class _BasePtForecasterV2(_BasePtForecaster):
    def _check_metadata():
          if EncoderDecoderDataModule:
                    return test_enc_dec_dm()
          elif TSLibDataModule:
                     return test_tslib_dm()
          elif:
               ... # so on?

• If you mean each MODEL has _check_metadata then there will be duplicate code for all models having same datamodule?

These are my initial thoughts, but I still need your help to understand what exactly is the idea, I'm a bit lost here😅

[fkiraly]

If you mean each MODEL has _check_metadata

That is what I mean

then there will be duplicate code for all models having same datamodule?

... unless you add the check_metadata to a common intermediate base class that all models using that data module inherit from.
(which I did not say but thought would be implied by this design)

[phoeenniixx]

Hi @fkiraly, do we need to add something else to this PR? We can add metadata testing in some other PR?

[fkiraly]

I think this is ready - the conflicts with #1886 need to be resolved though.

[phoeenniixx]

done

[fkiraly]

I made some small changes:

• deduplicate code copy-pasted twice in _conftest - why did you put the same code in two files?
• revert change to _BasePtForecaster name, the name remains the same until we move to v2
• changed "metadata" filename to pkg