Create a decision task for taskcluster #19401

jgraham · 2019-09-30T13:57:44Z

This replaces the current scheme in which all tasks are specified in the .taskcluster.yml file with a setup in which only an initial "decision" task is specified in that file. The job of the decision task is to examine the event that caused it to be scheduled and the corresponding changes (if any) and select an appropriate set of jobs to run.

Compared to the previous setup of having a single set of tasks this provides some new features:

Ability to only schedule those tasks that should actually be run. Previously for PRs we were using a scheme inherited from travis in which all the jobs would be scheduled but a subset of them would bail out without doing anything. This made for inefficient use of resources and poor UI since users would have to look at the results of jobs that didn't run, alongside those of jobs that actually did run.
Ability to specify dependencies between tasks. This allows tasks that don't start until another task completes. An example of this partially implemented in this PR is downloading a single browser binary used for all subsequent runs of that browser, ensuring constant versions both for the initial task and also for any retries. So far the dependency is implemented, but not the logic to actually use the downloaded Firefox for subsequent tasks.

There are also various things that now become possible that previously were hard to implement. For example we could invent a tc-try:<jobs> syntax to allow trying specific master runs on a PR without having to push to the triggers/ branches.

The current state of the feature is that tasks are described in a custom YAML format file in tools/ci/tasks/test.yml (there is some intent to allow using multiple tasks in the future). The format of this file and processing model is described in tools/ci/README.md. The setup has been verified on the jgraham/web-platform-tests fork of this repo.

One problem with landing this is that taskcluster always reads the .taskcluster.yml from the branch head, whereas PR tests are run on the merge of the base branch with the PR head. This means that some care is required not to break anything in such merges, or alternatively to rebase all PRs once the change lands.

ghost · 2019-09-30T13:57:46Z

Submitting the task to Taskcluster failed. Details

InterpreterError at template.tasks[0].payload.command[3]: object has no property after

foolip · 2019-10-22T09:11:17Z

One problem with landing this is that taskcluster always reads the .taskcluster.yml from the branch head, whereas PR tests are run on the merge of the base branch with the PR head. This means that some care is required not to break anything in such merges, or alternatively to rebase all PRs once the change lands.

What's your preference here? Rebasing all PRs would cause a huge spike in CI resource usage and noise that'll probably cause people to mark all their notifications as read for a few days.

What does the "care is required" approach look like in practice? What are the sorts of changes we can't make in the setup after this is landed that would break existing PRs?

See also web-platform-tests/wpt-pr-bot#81 for an idea I proposed yesterday.

foolip

Reviewed up to and including tools/ci/tc/decision.py, need a break :)

tools/ci/README.md

.taskcluster.yml

foolip · 2019-10-22T09:16:35Z

.taskcluster.yml

+          owner: "${event.sender.login}@users.noreply.github.com"
+          source: ${event.repository.clone_url}
+        payload:
+          image: harjgam/web-platform-tests:0.33


Needs to be updated if #19754 is merge before this. It'd be a conflict, but the conflicts in this file will be huge.

.taskcluster.yml

tools/ci/tc/decision.py

tools/ci/tc/taskgraph.py

tools/ci/tc/tasks/test.yml

tools/ci/tc/tests/test_valid.py

tools/wpt/browser.py

tools/wptrunner/wptrunner/wptrunner.py

jgraham · 2019-11-04T10:31:02Z

I updated this, and replied to some comments, so I think it's ready for a second round of review.

foolip · 2019-11-04T11:12:32Z

@jgraham there are still conflicts, can you resolve those first?

jgraham · 2019-11-04T12:02:03Z

s/still/again/ since all the conflicts were resolved in the last push and then more changes happened on master.

foolip · 2019-11-06T13:17:27Z

I didn't notice the intermediate state of no conflicts :)

community-tc-integration · 2019-11-11T16:51:41Z

Submitting the task to Taskcluster failed. Details

InterpreterError at template.tasks[0].payload.command[3]: object has no property after

foolip · 2019-11-15T11:10:33Z

@jgraham more conflicts and CI failures :)

Maybe we should try to work on this sync on IRC at some point to get it green and then immediately reviewed and merged?

.taskcluster.yml

jgraham · 2019-11-18T19:33:21Z

@foolip I found the parts of this that got lost, so AFAIK it's now up to date with master, and the resolved review issues are correct. There are still some open questions, so I anticipate needing at least one more round of fixes.

community-tc-integration · 2019-11-18T19:39:44Z

Submitting the task to Taskcluster failed. Details

InterpreterError at template.tasks[0].payload.command[3]: object has no property after

A decision task is the first task run, and ensures that we only schedule subsequent tasks which are relevant to the current push or pull request. This dynamic scheduling helps reduce load since we avoid spinning up workers for tasks that ultimately don't run, and unlocks new possibilities since we are able to schedule tasks that are dependent on other tasks. The tasks, their scheduling criteria and their dependnecies are specified in a a YAML format configuration file in tools/ci/tc/tasks/test.yml. This has a bespoke format, adopting some ideas from Azure and Gecko's taskcluster integration. The format is documented in `tools/ci/tc/README.md`. The data in that file undergoes trandformations to produce a set of tasks, which are then filtered according to the event that caused the decision task to run. To initially prove out the implementation of dependent tasks we make the Firefox tasks depend on a download task. But this does not yet include the work to actually make the dependent tasks download Firefox from the parent task.

PR #19401 dropped the --checkout flag to run_tc.py for jobs triggered by master pushes. Luckily, there is a new assertion in place that has caught the bug. Examples: Before: https://community-tc.services.mozilla.com/tasks/BaIMAbGZR_SsgHYyUAWA0g/runs/0/logs/https%3A%2F%2Fcommunity-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FBaIMAbGZR_SsgHYyUAWA0g%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L37 After: https://community-tc.services.mozilla.com/tasks/L77FVIkURm6XiP96FK3JMg/runs/0/logs/https%3A%2F%2Fcommunity-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FL77FVIkURm6XiP96FK3JMg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L24 Note: this is NOT covered by unit tests.

wpt-pr-bot added ci infra tox.ini wpt wptrunner The automated test runner, commonly called through ./wpt run labels Sep 30, 2019

wpt-pr-bot assigned gsnedders Sep 30, 2019

wpt-pr-bot requested review from gsnedders and jugglinmike September 30, 2019 13:57

github-actions bot added the pull-request-has-preview label Sep 30, 2019

wpt-pr-bot added the Taskcluster label Sep 30, 2019

jgraham force-pushed the decision_task branch 2 times, most recently from 9ae1398 to 16e0829 Compare October 1, 2019 12:38

moz-wptsync-bot added the mozilla:gecko-blocked label Oct 1, 2019

foolip removed the pull-request-has-preview label Oct 20, 2019

foolip reviewed Oct 22, 2019

View reviewed changes

jgraham force-pushed the decision_task branch from ffc986d to ac6475f Compare October 22, 2019 19:34

moz-wptsync-bot removed the mozilla:gecko-blocked label Oct 22, 2019

jgraham force-pushed the decision_task branch 6 times, most recently from 620562b to e6e2412 Compare November 11, 2019 15:04

jgraham closed this Nov 11, 2019

jgraham reopened this Nov 11, 2019

jgraham force-pushed the decision_task branch from e6e2412 to 962143c Compare November 11, 2019 16:53

jgraham force-pushed the decision_task branch from 962143c to c98757f Compare November 15, 2019 11:48

wpt-pr-bot added webdriver wg-testing_browser labels Nov 15, 2019

jgraham force-pushed the decision_task branch from 6c89690 to 636f37d Compare November 15, 2019 14:48

foolip reviewed Nov 15, 2019

View reviewed changes

.taskcluster.yml Show resolved Hide resolved

jgraham closed this Nov 18, 2019

jgraham reopened this Nov 18, 2019

jgraham force-pushed the decision_task branch from f9383ee to ec265e7 Compare November 21, 2019 23:05

foolip approved these changes Nov 22, 2019

View reviewed changes

jgraham merged commit f57ead1 into master Nov 22, 2019

jgraham deleted the decision_task branch November 22, 2019 16:47

jgraham restored the decision_task branch November 22, 2019 16:47

foolip mentioned this pull request Nov 22, 2019

TaskCluster has a big queue and its slowing down processing the checks on pull requests #20256

Closed

Hexcles mentioned this pull request Nov 22, 2019

[taskcluster] Fix the start flags for master push #20406

Closed

stephenmcgruer mentioned this pull request Nov 29, 2019

Postmortem: Taskcluster Decision Task PR broke master commit checks (multiple times) #20523

Closed

2 tasks

gsnedders deleted the decision_task branch January 24, 2020 18:05

gsnedders restored the decision_task branch January 24, 2020 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a decision task for taskcluster #19401

Create a decision task for taskcluster #19401

jgraham commented Sep 30, 2019

ghost commented Sep 30, 2019

foolip commented Oct 22, 2019

foolip left a comment

foolip Oct 22, 2019

jgraham commented Nov 4, 2019

foolip commented Nov 4, 2019

jgraham commented Nov 4, 2019

foolip commented Nov 6, 2019

community-tc-integration bot commented Nov 11, 2019

foolip commented Nov 15, 2019

jgraham commented Nov 18, 2019

community-tc-integration bot commented Nov 18, 2019

Create a decision task for taskcluster #19401

Create a decision task for taskcluster #19401

Conversation

jgraham commented Sep 30, 2019

ghost commented Sep 30, 2019

foolip commented Oct 22, 2019

foolip left a comment

Choose a reason for hiding this comment

foolip Oct 22, 2019

Choose a reason for hiding this comment

jgraham commented Nov 4, 2019

foolip commented Nov 4, 2019

jgraham commented Nov 4, 2019

foolip commented Nov 6, 2019

community-tc-integration bot commented Nov 11, 2019

foolip commented Nov 15, 2019

jgraham commented Nov 18, 2019

community-tc-integration bot commented Nov 18, 2019