Skip to content

Commit 8f11e99

Browse files
committed
TUR-21619: Update docstring+typing (#4)
* update docstring * idk
1 parent 1d26240 commit 8f11e99

File tree

1 file changed

+71
-37
lines changed

1 file changed

+71
-37
lines changed

src/xdist/scheduler/customgroup.py

+71-37
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,82 @@
1-
"""Run tests across a variable number of nodes based on custom groups.
2-
3-
# TODO: This is more of a spec/description, update docs/remove this section document within the class
4-
Example:
5-
- 10 test cases exist
6-
- 4 test cases are marked with @pytest.mark.low
7-
- 4 test cases are marked with @pytest.mark.medium
8-
- 2 test cases are marked with @pytest.mark.high
9-
- A pytest.ini file contains the following lines:
10-
[pytest]
11-
12-
markers=
13-
low: 4
14-
medium: 2
15-
high: 1
16-
17-
Then the 4 low test cases will be ran on 4 workers (distributed evenly amongst the 4, as the load.py scheduler functions)
18-
Then the 4 medium test cases will be ran on 2 workers (again, distributed evenly), only after the low test cases are complete (or before they start).
19-
Then the 2 high test cases will be ran on 1 worker (distributed evenly), only after the low and medium test cases are complete (or before they start).
20-
21-
This allows a pytest user more custom control over processing tests.
22-
One potential application would be measuring the resource utilization of all test cases. Test cases that are not
23-
resource intensive can be ran on many workers, and more resource intensive test cases can be ran once the low
24-
resource consuming tests are done on fewer workers, such that resource consumption does not exceed available resources.
25-
"""
261
from __future__ import annotations
272

283
from itertools import cycle
29-
from typing import Sequence
4+
from typing import Sequence, Any
305

316
import pytest
327

33-
from xdist.remote import Producer, WorkerInteractor
8+
from xdist.remote import Producer
349
from xdist.report import report_collection_diff
3510
from xdist.workermanage import parse_spec_config
3611
from xdist.workermanage import WorkerController
3712

3813
class CustomGroup:
39-
"""
40-
# TODO: update docs here
14+
"""Implement grouped load scheduling across a variable number of nodes.
15+
16+
This distributes tests into groups based on the presence of xdist_custom pytest marks.
17+
Groups are ran sequentially with tests within each group running in parallel.
18+
The number of workers assigned to each group is based on the xdist_custom pytest mark.
19+
Tests without the xdist_custom pytest mark are assigned to a "default" group and run
20+
using all available workers.
21+
22+
Example:
23+
Consider 12 pytest test cases.
24+
- 4 test cases are marked with @pytest.mark.xdist_custom(name="low_4")
25+
- 2 test cases are marked with @pytest.mark.xdist_custom(name="med_2")
26+
- 2 test cases are marked with @pytest.mark.xdist_custom(name="high_1")
27+
- 4 test cases are not marked with a xdist_custom mark.
28+
Consider the pytest run was initiated with 4 workers (-n 4)
29+
- The 4 test cases marked with "low_4" would run in a group using 4 workers
30+
- The 2 test cases marked with "med_2" would run in a group using 2 workers
31+
- The 2 test cases marked with "high_1" would run in a group with 1 worker
32+
- The 4 unmarked test cases would run in a group using 4 workers.
33+
Only one group would run at any given time. For example, while the "high_1" tests are executing,
34+
the other pending test groups would not be scheduled or excuting. The order in which groups
35+
are executed is variable. For example, "high_1" may execute first, or it may execute second, etc.
36+
If a group pytest mark specifies more workers than the pytest run is initialized with the
37+
number of workers the run was initialized with will be used instead (-n argument is a maximum).
38+
39+
Attributes::
40+
41+
:terminal: Terminal reporter for writing terminal output
42+
43+
:numnodes: The expected number of nodes taking part. The actual
44+
number of nodes will vary during the scheduler's lifetime as
45+
nodes are added by the DSession as they are brought up and
46+
removed either because of a dead node or normal shutdown. This
47+
number is primarily used to know when the initial collection is
48+
completed.
49+
50+
:node2collection: Map of nodes and their test collection. All
51+
collections should always be identical.
52+
53+
:node2pending: Map of nodes and the indices of their pending
54+
tests. The indices are an index into ``.pending`` (which is
55+
identical to their own collection stored in
56+
``.node2collection``).
57+
58+
:pending: List of indices of globally pending tests. These are
59+
tests which have not yet been allocated to a chunk for a node
60+
to process.
61+
62+
:collection: The one collection once it is validated to be
63+
identical between all the nodes. It is initialised to None
64+
until ``.schedule()`` is called.
65+
66+
:log: A py.log.Producer instance.
67+
68+
:config: Config object, used for handling hooks.
69+
70+
:dist_groups: Execution groups. Updated based on xdist_custom pytest marks.
71+
Maps group names to tests, test indices, pending indices, and stores the number of workers to use
72+
for that test execution group.
73+
74+
:pending_groups: List of dist_group keys that are pending
75+
76+
:is_first_time: Boolean to track whether we have called schedule() before or not
77+
78+
:do_resched: Boolean to track whether we should schedule another distribution group.
79+
Accessed in dsession.py
4180
"""
4281

4382
def __init__(self, config: pytest.Config, log: Producer | None = None) -> None:
@@ -52,12 +91,10 @@ def __init__(self, config: pytest.Config, log: Producer | None = None) -> None:
5291
else:
5392
self.log = log.loadsched
5493
self.config = config
55-
self.maxschedchunk = self.config.getoption("maxschedchunk")
56-
# TODO: Type annotation incorrect
57-
self.dist_groups: dict[str, str] = {}
94+
self.dist_groups: dict[str, Any] = {}
5895
self.pending_groups: list[str] = []
59-
self.is_first_time = True
60-
self.do_resched = False
96+
self.is_first_time: bool = True
97+
self.do_resched: bool = False
6198

6299
@property
63100
def nodes(self) -> list[WorkerController]:
@@ -264,9 +301,6 @@ def schedule(self) -> None:
264301
if not self.collection:
265302
return
266303

267-
if self.maxschedchunk is None:
268-
self.maxschedchunk = len(self.collection)
269-
270304
dist_groups = {}
271305

272306
if self.is_first_time:

0 commit comments

Comments
 (0)