ci(containerised-test): add coverage

[caspiano]

Adds coverage job to the containerised test flow.

This change requires specs in each repository to mount a coverage folder into the test container.

[chillfox]

I tested it on staff-api and it runs, but there is this error:

Can't set personality: Operation not permitted
kcov: error: Can't start/attach to /app/app

And it uploads a file saying zero coverage by the looks of it.

[chillfox]

SimonKagstrom/kcov#151

docker disables the personality syscall, so need to use --security-opt seccomp=unconfined

docker run -it --rm --security-opt seccomp=unconfined -v $(pwd):/source kcov

[caspiano]

@chillfox adding the following to the test container in the docker-compose should solve it

    security_opt:
      - seccomp:unconfined

[chillfox]

Timed out after 25 minutes
Error: The action has timed out.

[caspiano]

@chillfox I've got an arbitrary timeout of 25 minutes in the job, you could try removing it or bumping it to be higher ~ say 45 minutes.

I'm surprised it's such a long running job

[chillfox]

@caspiano I would assume that it is broken, but yeah, might be worth giving it some more time to be sure.

[chillfox]

Still times out at 45

[caspiano]

It doesn't look like the specs are being run, so it might be worth trying your new images from #55, however we'll need to merge them first

[chillfox]

yeah,
will give that a go and if it doesn't work then I guess it will be time to fork https://github.com/flant/kcov-alpine and fix that.

[caspiano]

I added some verbose logging, looks like the crkcov runner is trying to read a file that doesn't exist.

https://github.com/PlaceOS/staff-api/runs/5566730681?check_suite_focus=true#step:3:96

[chillfox]

so it works locally in Elementary OS (Ubuntu based) but not in Alpine.
In Alpien the covearge.json file is not created

[chillfox]

so kcov runs in Alpine and does produce the report. So it is just crkcov that does not...

[chillfox]

Status:

• kcov works in Alpine (generates the report json).
• crkcov does not work in Alpine (the report json is missing).
• crkcov does work in Elementary OS (so it will most likely work in Debian/Ubuntu).

Tried:

• Messing around with GH actions.
• Created a small test app + docker image and script to test locally.
• Light modifications to crkcov (disabling everything that deletes files).

Ways forward:

• Run crkcov in a Debian/Ubuntu image (this means all our tests would need to be capable of running in Debian/Ubuntu).
• Fork crkcov, make its specs run in Alpine (maybe write additional tests to pin down the issue), see how deep the rabbit hole goes.

[caspiano]

I reckon trying to get crkcov to work in alpine may be the best way forward. There's could be a small platform based assumption.
Working on an ubuntu image is a fine way to move forward, but it may be a bit of work, albeit straightforward work

[chillfox]

finally got it working in alpine

[caspiano]

Epic!!!!!! Great work @chillfox

Can't wait to use it

[chillfox]

container needs to be run with docker run --security-opt seccomp=unconfined

or in compose

security_opt:
      - seccomp:unconfined

[chillfox]

While this works on a small project it fails on a larger project like the portal.

➜ 5:00PM portal-api (test/coverage) ✔ ./test --coverage
...
Unhandled exception: Error writing file: Broken pipe (IO::Error)
  from /usr/share/crystal/src/gc/boehm.cr:129:5 in 'unbuffered_write'
  from /usr/share/crystal/src/slice.cr:1169:12 in 'write'
  from /usr/share/crystal/src/string.cr:4829:5 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/ld-musl-x86_64.so.1 in '??'

➜ 5:04PM portal-api (test/coverage) ✗ ./test --shell
...
/app # crystal-coverage 
Unhandled exception: Error executing process: '/bin/sh': Argument list too long (IO::Error)
  from /usr/share/crystal/src/crystal/system/unix/process.cr:225:7 in 'raise_exception_from_errno'
  from /usr/share/crystal/src/io.cr:1183:7 in 'run:shell:input:output:error'
  from /usr/share/crystal/src/process.cr:419:3 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/ld-musl-x86_64.so.1 in '??'

Likely due to the shell trying to expand some recursive argument like a list of files or something like that.

See https://github.com/place-technology/portal-api/pull/396 for testing.

[caspiano]

@chillfox could be how crystal-coverage concatenates the spec files together?

[chillfox]

well, the size of data in the pipe should not be a problem, but if it shells out/calls another program (like kcov) with too long of an argument list then that is the issue (I think the argument limit is a kernel limitation)

[chillfox]

maybe

if print_only
  puts final_output
else
  system("crystal", ["eval", final_output])
end

I guess I could compile crystal-coverage with some pp in there

[chillfox]

➜ 5:36PM code_coverage (master) ✗ grep -rni 'system' crystal-coverage/src   
crystal-coverage/src/coverage/runtime/outputters/html_report.cr:75:    system("rm -rf coverage/")
crystal-coverage/src/coverage/runtime/outputters/coveralls.cr:43:      system("curl -X POST https://coveralls.io/api/v1/jobs -H 'content-type: multipart/form-data'  -F [email protected]")
crystal-coverage/src/coverage/inject/cli.cr:50:        system("crystal", ["eval", final_output])

[chillfox]

max argument in alpine container

/app # getconf ARG_MAX
131072

same thing on my desktop (same problem)

➜ 9:34AM code_coverage (master) ✗ getconf ARG_MAX       
2097152

[chillfox]

A small test project where it works.

➜ 10:02AM code_coverage (master) ✗ cloc src spec    
       3 text files.
       3 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.00 s (922.0 files/s, 8605.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          3              5              1             22
-------------------------------------------------------------------------------
SUM:                             3              5              1             22
-------------------------------------------------------------------------------
➜ 10:02AM code_coverage (master) ✗ /usr/local/bin/crystal-coverage
.
Lines 80.00% covered

Finished in 1.34 milliseconds
1 examples, 0 failures, 0 errors, 0 pending

A slightly larger project where it fails

➜ 10:01AM boundless-scuttlecoil cloc src spec                                                                     fossil:(  next_state_perf )  ✔ 
      26 text files.
      26 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.02 s (1245.5 files/s, 158939.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                         26            418            368           2532
-------------------------------------------------------------------------------
SUM:                            26            418            368           2532
-------------------------------------------------------------------------------
➜ 10:01AM boundless-scuttlecoil /usr/local/bin/crystal-coverage                                                   fossil:(  next_state_perf )  ✔ 
Unhandled exception: Error executing process: '/bin/sh': Argument list too long (IO::Error)
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/crystal/system/unix/process.cr:225:7 in 'raise_exception_from_errno'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/io.cr:1183:7 in 'run:shell:input:output:error'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/process.cr:419:3 in '__crystal_main'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /usr/local/bin/crystal-coverage in '_start'
  from ???
➜ 10:01AM boundless-scuttlecoil 

Smaller project with another error

➜ 10:11AM boundary-scuttlecoil cloc src spec                                                                                fossil:(  trunk )  ✔ 
       5 text files.
       5 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.00 s (1065.2 files/s, 231786.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          5            146             68            874
-------------------------------------------------------------------------------
SUM:                             5            146             68            874
-------------------------------------------------------------------------------
➜ 10:11AM boundary-scuttlecoil /usr/local/bin/crystal-coverage                                                              fossil:(  trunk )  ✔ 
Showing last frame. Use --error-trace for full trace.

error in line 1
Error: can't find file 'coverage/runtime'

If you're trying to require a shard:
- Did you remember to run `shards install`?
- Did you make sure you're running the compiler in the same directory as your shard.yml?

small project with an error

➜ 10:16AM echo_ip cloc src spec                                                                                             fossil:(  trunk )  ✔ 
       1 text file.
       1 unique file.                              
       0 files ignored.

1 error:
Unable to read:  spec

github.com/AlDanial/cloc v 1.82  T=0.00 s (271.3 files/s, 34726.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          1             25             13             90
-------------------------------------------------------------------------------
➜ 10:16AM echo_ip /usr/local/bin/crystal-coverage                                                                           fossil:(  trunk )  ✔ 
Showing last frame. Use --error-trace for full trace.

error in line 1
Error: can't find file 'coverage/runtime'

If you're trying to require a shard:
- Did you remember to run `shards install`?
- Did you make sure you're running the compiler in the same directory as your shard.yml?

[chillfox]

This now works

➜ 11:09AM portal-api (test/coverage) ✗ /usr/local/bin/crystal-coverage -p | less

And a new error

➜ 11:11AM portal-api (test/coverage) ✗ /usr/local/bin/crystal-coverage -p | crystal eval 
Showing last frame. Use --error-trace for full trace.

In lib/action-controller/spec/curl_context.cr:35:1

 35 | ::CURL_CONTEXT__ = [] of ActionController::Server
      ^---------------
Error: already initialized constant CURL_CONTEXT__