Skip to content

Commit 76bc7b2

Browse files
author
Sylvain MARIE
committed
Added an example in the documentation on how to use with pytest-xdist. Fixes #32
1 parent 929f68e commit 76bc7b2

File tree

1 file changed

+48
-0
lines changed

1 file changed

+48
-0
lines changed

docs/index.md

+48
Original file line numberDiff line numberDiff line change
@@ -432,6 +432,54 @@ All the behaviours described above are pre-wired to help most users getting star
432432

433433
This plugin mostly relies on the fixtures mechanism and the `pytest_runtest_makereport` hook. It should therefore be quite portable across pytest versions (at least it is tested against pytest 2 and 3, for both python 2 and 3).
434434

435+
### pytest x-dist
436+
437+
You may wish to rely on `pytest-xdist` to parallelize/distribute your tests. In that case, you can not rely on the `[module/session]_results_[dct/df]` fixtures described previously to collect your synthesis. To cover this case, `pytest-harvest` provides equivalent methods `get_[module/session]_results_[dct/df](session, [module_name])`. You can use these methods in any pytest hook, for example in the `pytest_sessionfinish` hook.
438+
439+
Below is a complete example of `conftest.py` that works both *with* and *without* `pytest-xdist` enabled.
440+
441+
```python
442+
from collections import OrderedDict
443+
import logging
444+
import pandas as pd
445+
from pathlib import Path
446+
from pytest_harvest import get_session_results_df
447+
448+
# Define the folder in which temporary worker's results will be stored when xdist is enabled
449+
RESULTS_PATH = Path('./.xdist_results/')
450+
RESULTS_PATH.mkdir(exist_ok=True)
451+
452+
def pytest_sessionfinish(session):
453+
""" Gather all results, in a way that works both with and without pytest-x-dist """
454+
try:
455+
# x-dist mode: retrieve x-dist worker id and save its partial results
456+
pytest_worker_id = session.config.slaveinput['slaveid']
457+
logging.warning('PARTIAL pytest session finish for x-dist worker %s' % pytest_worker_id)
458+
session_results_df = get_session_results_df(session)
459+
session_results_df.to_csv(RESULTS_PATH / ('%s_session_results.csv' % pytest_worker_id))
460+
461+
except AttributeError:
462+
# x-dist master OR no x-dist at all: put everything available together
463+
logging.warning('MAIN pytest session finish')
464+
session_results_dfs = OrderedDict()
465+
try:
466+
# try to collect master results in case x-dist was not active
467+
session_results_dfs['master'] = get_session_results_df(session)
468+
except:
469+
# x-dist master process - read all partial x-dist results files if any
470+
for results_file in RESULTS_PATH.glob('*_session_results.csv'):
471+
wid = results_file.name[:-len('_session_results.csv')]
472+
try:
473+
session_results_dfs[wid] = pd.read_csv(results_file)
474+
except Exception as e:
475+
logging.warning('Unable to read results from worker %s: [%s] %s' % (wid, e.__class__, e))
476+
477+
# concatenate all pandas tables and for example write them in a final csv
478+
df = pd.concat([results_df for results_df in session_results_dfs.values()]).reset_index()
479+
df.to_csv('all_results.csv')
480+
481+
```
482+
435483
## Main features / benefits
436484

437485
* **Collect test execution information easily**: with the default `[module/session]_results_[dct/df]` fixtures, and with `get_session_synthesis_dct(session)` (advanced users), you can collect all the information you need, without the hassle of writing hooks.

0 commit comments

Comments
 (0)