Problems getting parallel_tests to work in GitHub Action workflows

[bob-ashforth-sp]

We're having problems getting parallel_tests to work in GitHub Action workflows. Is this a known issue, or does anyone have any examples of correct commands to use in the workflow?

[grosser]

Did not head that before.
Best follow https://github.com/grosser/parallel_tests/wiki/Distributed-Parallel-Tests-on-CI-systems or post what exactly you are seeing.

[bob-ashforth-sp]

Thanks for the link, Michael. Since we use GitHub Actions, and the linux runners have only two cores, so I turned to implementing parallelism using GitHub Action matrix strategy, with straight rspec and my own grouping of tests per machines. I’m getting great speed out of that… but am getting pushback from our engineering team at not using parallel_tests, they are that fond of it. 😊 One thing which is pertinent to our internal discussion is the status of the parallel_tests gem: I could have sworn that somewhere on your site I read that it is not being actively maintained, because your company is moving away from Ruby/Rails… am I hallucinating? I had not looked at your Wiki until today, and find it intriguing. Several things are not clear to me in the wiki: * Will parallel_tests collate the tests run on the two GitHub runner machine’s two cores into a single SimpleCov report? * For using matrix strategy on GitHub Actions, my code needs to specify an array of group names (for my use simply Group1..Groupn) and the corresponding array of indexes into the group array. I’m hoping that the cpu-parallelization of parallel_tests would not clash with that, do you see any reason that it would? * The grouping is what intrigues me. Is the ‘test’ argument shown in the wiki the top-level folder of the rspec tree? Also, I would hope that there is no limit on the number of machines, and that you are just using three as an example in the wiki, can you confirm? I’m at the stage now where I am collecting the multiple SimpleCov result sets (distinguished by the index noted above) and about to make my first pass at collating them into a single report. After that, my intent was to revisit the per-machine load assignment, which I’m currently hardwiring to specified folders per machine, and make that assignment programmatically. This is where, if your grouping function can provide that load balancing, I would turn back to getting parallel_test to work in our GitHub Actions environment. (I would love to move this discussion to email rather than the parallel_tests issues section, if you’re open to that. The result of our discussion could then be added to the issues section, sort of a ‘squash and merge’ effect. 😊) From: Michael Grosser ***@***.***> Reply-To: grosser/parallel_tests ***@***.***> Date: Friday, February 18, 2022 at 9:44 AM To: grosser/parallel_tests ***@***.***> Cc: Robert Ashforth ***@***.***>, Author ***@***.***> Subject: Re: [grosser/parallel_tests] Problems getting parallel_tests to work in GitHub Action workflows (Issue #841) Did not head that before. Best follow https://github.com/grosser/parallel_tests/wiki/Distributed-Parallel-Tests-on-CI-systems or post what exactly you are seeing. — Reply to this email directly, view it on GitHub<#841 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASKCCF2EUOS6JLYV25C4YDDU3ZSNPANCNFSM5OYI6VAA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[grosser]

for singlecov: you'll have to combine singlecov results manually afaik, there might be something in simplecov docs ... alternatively make it generate multiple reports and the combine them something like https://github.com/grosser/cc-amend
or google around https://jes.al/2020/11/merging-simplecov-results-with-parallel-rails-specs

... for the other questions: that's just way too much text, sorry but I don't have enough time to dig into all this, please ask small precise questions or try googling around for support

[Amnesthesia]

@grosser One issue we've run into here is that whilst we get the specs to run, each subprocess now lacks environment variables that the parent had and a bunch of specs are now failing

[grosser]

there is nothing in parallel_tests that removes env vars though ... maybe the issue is that 1 test used to set an env var and then the other used it ?
to verify: does the failing test also fail when run on it's own ?

[Amnesthesia]

@grosser Turns out this happened when I was trying to pass -n 4 to it to force 4 processes. We run Github Actions runners in Kubernetes, and each pod is allocated 4000 millicpu, which would be 4 cores. I'm pretty new to this and not entirely sure of how to deal with this, so I tried to restrict it to using 4 processes to correspond to the 4000 millicpu, but inside the container it still thinks it has 12 CPU, which means it probably has the same amount of cores but restricted usage of those cores

Not entirely sure how to deal with this — perhaps we still need to make it run with 12 cores even if it's restricted to using the equivalent of a third of that.

To pass that flag (without passing any other options) I had passed parallel:spec[,,,-n4] because when I tried to pass it any other way it would throw with "Cannot convert Integer to String". This happened because line 271 in lib/parallel_tests/tasks.rb:

abort unless system(*command) # allow to chain tasks e.g. rake parallel:spec parallel:features

Here, command became [..., '-n', 4] and it failed when that 4 was parsed as an Integer rather than a string. But passing it through as parallel:spec[,,,-n4] seems to have somehow made my environment variables nil

[grosser]

parallel:spec[4]

should work on latest version

[grosser]

run a single test in paralell_tests locally and verify that there is an env issue
afaik it works just fine:

>> ENV['X'] = '1'
=> "1"
>> system 'ruby', '-e', 'puts ENV["X"]'
1
=> true

[Amnesthesia]

$ bundle exec rake parallel:spec[4]
rake aborted!
TypeError: no implicit conversion of Integer into String
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/gems/parallel_tests-3.10.0/lib/parallel_tests/tasks.rb:261:in `system'
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/gems/parallel_tests-3.10.0/lib/parallel_tests/tasks.rb:261:in `block (3 levels) in <main>'
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/gems/sentry-ruby-core-5.3.1/lib/sentry/rake.rb:24:in `execute'
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/bin/ruby_executable_hooks:22:in `eval'
/Users/amnesthesia/.rvm/gems/ruby-2.7.4/bin/ruby_executable_hooks:22:in `<main>'
Tasks: TOP => parallel:spec

I solved it by setting PARALLEL_TEST_PROCESSORS=4

[grosser]

parallel_tests-3.10.0 -> bundle update parallel_tests

[krtschmr]

@rashforth https://github.blog/changelog/2022-09-01-github-actions-larger-runners-are-now-in-public-beta/
contact support to skip beta waitlist que. we run 8 and 16 core via GH Actions