Ability to run tests only for files that changed

[HaddadJoe]

Hi, we're running green in a large repo with 1000s of tests available. This sadly takes some time to run. When opening a small PR, we often don't want to run all the tests. Ideally within the PR we just want to run the tests for the file that changed. It would be nice if green can support such feature where it relies on git file changes to be able to detect what tests to run.

This is a rough pseudo-code to do that using unittest. Would green be open to support such feature?

import subprocess
import os
import unittest


def get_changed_directories():
    # Use git to get the list of changed directories
    changed_files = subprocess.check_output(["git", "diff", "--name-only", "master..HEAD"]).decode().split("\n")
    changed_directories = {os.path.dirname(file) for file in changed_files if os.path.isdir(os.path.dirname(file))}
    return changed_directories


def get_test_modules_in_directories(directories):
    test_modules = []
    for directory in directories:
        # Assuming test modules have a certain naming convention like test_*.py
        test_files = [f for f in os.listdir(directory) if f.startswith("test_") and f.endswith(".py")]
        test_modules.extend(
            [directory.replace("/", ".") + "." + os.path.splitext(test_file)[0] for test_file in test_files]
        )
    return test_modules


def run_tests(test_modules):
    # Load and run unittest test cases from specified test modules
    loader = unittest.TestLoader()
    suites = [loader.loadTestsFromName(module) for module in test_modules]
    test_suite = unittest.TestSuite(suites)
    runner = unittest.TextTestRunner()
    runner.run(test_suite)


if __name__ == "__main__":
    changed_directories = get_changed_directories()
    test_modules = get_test_modules_in_directories(changed_directories)
    run_tests(test_modules)

[sodul]

One problem I see with this is that green would need to know which source control system you currently use (git is currently the most popular lately but not the only one). The more difficult thing is that there is no direct mapping between the tests and the source code.

Consider this project:

src/alice/a.py
src/alice/test/test_a.py
src/bob/b.py
src/bob/test/test_b.py
functional/test_things.py

So now if you change a.py I can see how we could assume that we need to run test_a.py but if it turns out that b.py imports a.py we would then probably need to run test_b.py as well. There is a similar problem with the tests under the functional folder in that they might depend on any of the code under src but we would need to be aware of that.

Tools such as Bazel have support to resolve dependency trees and that's actually a complicated thing. Bazel does NOT rely on git to know if something has changed and relies on explicitly declared dependencies in BUILD.bazel files as well as advanced caching mechanisms. See https://bazel.build for more details. Of course Bazel is just one such tools and there are other re-implementatios of Blaze (original name of Bazel in use at Google) such as Bucks (Facebook version).

Overall there are 2 issues:

• detect what changed, we could use a cache and md5 checksums like Bazel does, or only work with git.
• have a full understanding of the import tree of tests. This could be done by importing every single test and check if any of the imports is in the list of things that have changed. This could get tricky especially with code that uses conditional imports.

Overall I feel this is a very complicated thing to get to work reliably and universally, which would be much beyond the scope of green to be a simple and reliable tool to run unittests.

Of course this is ultimately @CleanCut's decision as the owner of the project.