[feature] progress in testing log

[jiangying000]

currently nextest supports progress bar, but I did not find a way to enable a log like below, it is an example made by hand:

Finished test [unoptimized + debuginfo] target(s) in 6m 01s
Starting 500 tests across 161 binaries (17 skipped)
[1/500]    PASS [   0.009s]                                        bcs-ext tests::test_bcs_enum_compact
[2/500]     PASS [   0.009s]                                        bcs-ext tests::test_serialize
  ....many lines
[400/500]    PASS [   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown
 [401/500]     PASS [   0.234s]                starcoin-service-registry::basc test_basic

so that when a open a part of CI log ( like the one on github, github give us tail of log when testing is running) we can see current testing progress

[sunshowers]

I agree, this would be valuable. Let me see what I can do here.

[sunshowers]

I'm going to call this a progress counter, as distinct from the progress bar.

When should the counter be displayed?

I think it would make sense to display it whenever the progress bar isn't displayed. Specifically, we'd want to add a CLI flag called --show-progress, with values bar, counter, auto (default) or none. If --show-progress is auto, then:

1. If this is a CI environment, set it to counter.
2. If standard error is a terminal, set it to bar.
3. Otherwise, set it to counter.

If --show-progress bar is specified and standard error is not a terminal, then this implies none.

We'll also want to deprecate --hide-progress-bar, make it mean --show-progress none, and make it conflict with --show-progress so both of those can't be specified at the same time.

Where should the progress counter be shown?

I think the best place to show the progress would be to the left of the timer: e.g.

        PASS [400/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

One thing to be careful of here is padding. This is what it should look like:

        PASS [ 99/500   0.009s]                                        bcs-ext tests::test_serialize
        PASS [100/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

What should be displayed?

I think the progress counter should be kept really simple for now. Only the number of completed tests and the total number of tests to run should be displayed. We could potentially show failed tests here but that'll be too noisy I think.

Would you like to implement this, @jiangying000? I think it should maybe be 1.5 days of work or so.

[jiangying000]

I added this to my schedule, would like to finish this in Mid Oct.

[sunshowers]

Perfect, thanks!

[jiangying000]

Rain, sorry for being late, I have been deeling with some complex bugs in my full time work for some time and it might take longer, but I will definitely finish the feature

and please feel free to unassign me if you want to make it step forward faster

[sunshowers]

No worries! I don't plan to work on this any time soon so I'm happy to wait for your contribution.

[jiangying000]

I'm going to call this a progress counter, as distinct from the progress bar.

When should the counter be displayed?

I think it would make sense to display it whenever the progress bar isn't displayed. Specifically, we'd want to add a CLI flag called --show-progress, with values bar, counter, auto (default) or none. If --show-progress is auto, then:

1. If this is a CI environment, set it to counter.
2. If standard error is a terminal, set it to bar.
3. Otherwise, set it to counter.

If --show-progress bar is specified and standard error is not a terminal, then this implies none.

We'll also want to deprecate --hide-progress-bar, make it mean --show-progress none, and make it conflict with --show-progress so both of those can't be specified at the same time.

Where should the progress counter be shown?

I think the best place to show the progress would be to the left of the timer: e.g.

        PASS [400/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

One thing to be careful of here is padding. This is what it should look like:

        PASS [ 99/500   0.009s]                                        bcs-ext tests::test_serialize
        PASS [100/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

What should be displayed?

I think the progress counter should be kept really simple for now. Only the number of completed tests and the total number of tests to run should be displayed. We could potentially show failed tests here but that'll be too noisy I think.

Would you like to implement this, @jiangying000? I think it should maybe be 1.5 days of work or so.

@sunshowers hello, rain, could you take a look, is this spec still valid a year later. I am going to impl this.

[sunshowers]

That's awesome @jiangying000! Yes, it looks right.

[bobrik]

I took an initial stab at it: #2618.