Fix flaky specs #2522

solnic · 2025-01-24T10:08:33Z

Issue Description

This issue gathers random failures we get from some flaky tests.

Failure 01

Ruby 3.3

Failures:

  1) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for transaction event with span counts
     Failure/Error: expect(client.transport).to have_recorded_lost_event(:network_error, 'transaction')
       expected #<Sentry::HTTPTransport:0x00007fad0197b5e8 @logger=#<Logger:0x00007fad0197b868 @level=0, @progname=ni...nt_reports=true, @discarded_events={}, @last_client_report_sent=2025-01-24 10:01:40.480909663 +0000> to have recorded lost event :network_error and "transaction"
     # ./spec/sentry/client/event_sending_spec.rb:492:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

  2) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for error event
     Failure/Error: expect(client.transport).to have_recorded_lost_event(:network_error, 'error')
       expected #<Sentry::HTTPTransport:0x00007fad00bd46b0 @logger=#<Logger:0x00007fad00bd48e0 @level=0, @progname=ni...nt_reports=true, @discarded_events={}, @last_client_report_sent=2025-01-24 10:01:41.297596915 +0000> to have recorded lost event :network_error and "error"
     # ./spec/sentry/client/event_sending_spec.rb:486:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

  3) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)
     Failure/Error: expect(string_io.string).to match(/Event sending failed: Failed to open TCP connection/)

       expected "D, [2025-01-24T10:01:41.915649 #2632] DEBUG -- sentry: Initializing the Sentry background worker wit...entry: [Transport] Sending envelope with items [event] 14d88737df1a4ab7b8581c7189f963e2 to Sentry\n" to match /Event sending failed: Failed to open TCP connection/
       Diff:
       @@ -1,3 +1,5 @@
       -/Event sending failed: Failed to open TCP connection/
       +D, [2025-01-24T10:01:41.915649 #2632] DEBUG -- sentry: Initializing the Sentry background worker with 2 threads
       +D, [2025-01-24T10:01:41.915735 #2632] DEBUG -- sentry: Sentry HTTP Transport will connect to http://sentry.localdomain/
       +D, [2025-01-24T10:01:41.919898 #2632] DEBUG -- sentry: [Transport] Sending envelope with items [event] 14d88737df1a4ab7b8581c7189f963e2 to Sentry
     # ./spec/sentry/client/event_sending_spec.rb:471:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

Finished in 12.55 seconds (files took 0.73758 seconds to load)
1076 examples, 3 failures, 1 pending

Failed examples:

rspec ./spec/sentry/client/event_sending_spec.rb:489 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for transaction event with span counts
rspec ./spec/sentry/client/event_sending_spec.rb:483 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for error event
rspec ./spec/sentry/client/event_sending_spec.rb:467 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)

Failure 02

jruby / rack3

  1) Sentry::Client#event_from_exception sets threads interface without stacktrace
     Failure/Error: expect(thread[:name]).to eq("Thread 1")

       expected: "Thread 1"
            got: nil

       (compared using ==)
     # ./spec/sentry/client_spec.rb:370:in `block in <main>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block in setup'

Finished in 31.92 seconds (files took 4.12 seconds to load)

Failures 03

Ruby 2.6

Randomized with seed 4066

/home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:69:in `rescue in send_data': end of file reached (Sentry::ExternalError)
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:32:in `send_data'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:65:in `send_envelope'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:173:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/client.rb:101:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:271:in `close'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:245:in `block in init'
/opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:225:in `rbuf_fill': end of file reached (EOFError)
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1509:in `block in transport_request'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1479:in `request'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/net/http.rb:35:in `request'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:52:in `block in send_data'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:920:in `start'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:49:in `send_data'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:65:in `send_envelope'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:173:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/client.rb:101:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:271:in `close'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:245:in `block in init'
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
Coverage report generated for RSpec to /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/coverage/coverage.xml. 10539 / 10898 LOC (96.71%) covered.
/opt/hostedtoolcache/Ruby/2.6.10/x64/bin/ruby -I/home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-core-3.13.2/lib:/home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-support-3.13.2/lib /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-core-3.13.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb  --exclude-pattern spec/isolated/\*\*/\*_spec.rb --order rand failed

Failure 03

Ruby 2.5

Failures:

  1) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events inline causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)
     Failure/Error: expect(string_io.string).to match(/Event sending failed: Failed to open TCP connection/)

       expected "D, [2025-02-04T10:36:10.837790 #2503] DEBUG -- sentry: config.background_worker_threads is set to 0,...nE, [2025-02-04T10:36:15.957837 #2503] ERROR -- sentry: Event capturing failed: execution expired\n" to match /Event sending failed: Failed to open TCP connection/
       Diff:
       @@ -1,5 +1,9 @@
       -/Event sending failed: Failed to open TCP connection/
       +D, [[2025](https://github.com/getsentry/sentry-ruby/actions/runs/13133532880/job/36643752347?pr=2537#step:6:2026)-02-04T10:36:10.837790 #2503] DEBUG -- sentry: config.background_worker_threads is set to 0, all events will be sent synchronously
       +D, [2025-02-04T10:36:10.837828 #2503] DEBUG -- sentry: Sentry HTTP Transport will connect to http://sentry.localdomain
       +D, [2025-02-04T10:36:10.840989 #2503] DEBUG -- sentry: [Transport] Sending envelope with items [event] 25d5c36b24e243d0a4ec19359f739bcd to Sentry
       +E, [2025-02-04T10:36:15.957715 #2503] ERROR -- sentry: Event sending failed: execution expired
       +E, [2025-02-04T10:36:15.957837 #2503] ERROR -- sentry: Event capturing failed: execution expired
     # ./spec/sentry/client/event_sending_spec.rb:466:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:387:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.5.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.5.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `loop'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.5.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.5.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.5.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

Finished in 12.47 seconds (files took 0.51572 seconds to load)
1079 examples, 1 failure, 25 pending

Failed examples:

rspec ./spec/sentry/client/event_sending_spec.rb:463 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events inline causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)

Randomized with seed 59399

Failure 04

jruby
rack3

Failures:

  1) Sentry::Client#event_from_exception sets threads interface without stacktrace
     Failure/Error: expect(thread[:name]).to eq("Thread 1")

       expected: "Thread 1"
            got: nil

       (compared using ==)
     # ./spec/sentry/client_spec.rb:370:in `block in <main>'

Finished in 29.07 seconds (files took 4.18 seconds to load)
1099 examples, 1 failure, 51 pending

Failed examples:

rspec ./spec/sentry/client_spec.rb:355 # Sentry::Client#event_from_exception sets threads interface without stacktrace

Failure 05

jruby

Failures:

  1) Sentry::Sidekiq::SentryContextClientMiddleware with user sets user of the current scope to the job
     Failure/Error: expect(queue.size).to be(1)

       expected #<Integer:3> => 1
            got #<Integer:7> => 3

       Compared using equal?, which compares object identity,
       but expected and actual are not the same object. Use
       `expect(actual).to eq(expected)` if you don't care about
       object identity in this example.
     # ./spec/sentry/sidekiq/sentry_context_middleware_spec.rb:173:in `block in <main>'

Finished in 5.96 seconds (files took 3.66 seconds to load)
70 examples, 1 failure, 1 pending

Failed examples:

rspec ./spec/sentry/sidekiq/sentry_context_middleware_spec.rb:170 # Sentry::Sidekiq::SentryContextClientMiddleware with user sets user of the current scope to the job

Randomized with seed 59324

Failure 06

ruby 3.4

  1) Sentry::Configuration#profiler_class= sets the profiler class to StackProf when Vernier is not available
     # Skipping because one or more guards `[[:ruby_version?, :<, "3.2"]]` returned false
     # ./spec/sentry/configuration_spec.rb:718

Failures:

  1) Sentry::Client#event_from_transaction correct dynamic_sampling_context when head SDK
     Failure/Error:
       expect(event.dynamic_sampling_context).to eq({
         "environment" => "development",
         "public_key" => "12345",
         "sample_rate" => "1.0",
         "sampled" => "true",
         "transaction" => "test transaction",
         "trace_id" => transaction.trace_id
       })

       expected: {"environment" => "development", "public_key" => "12345", "sample_rate" => "1.0", "sampled" => "true", "trace_id" => "2173a1acc82d48478d2b4d33675c9e24", "transaction" => "test transaction"}
            got: {"environment" => "development", "public_key" => "12345", "sample_rate" => "0.5", "sampled" => "false", "trace_id" => "2173a1acc82d48478d2b4d33675c9e24", "transaction" => "test transaction"}

       (compared using ==)

       Diff:
       @@ -1,6 +1,6 @@
        "environment" => "development",
        "public_key" => "12345",
       -"sample_rate" => "1.0",
       -"sampled" => "true",
       +"sample_rate" => "0.5",
       +"sampled" => "false",
        "trace_id" => "2173a1acc82d48478d2b4d33675c9e24",
        "transaction" => "test transaction",
     # ./spec/sentry/client_spec.rb:264:in 'block (3 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.4.0/gems/webmock-3.25.0/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.4.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in 'block in RSpec::Retry#run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.4.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in 'RSpec::Retry#run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.4.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in 'RSpec::Core::Example::Procsy#run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.4.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in 'block (2 levels) in RSpec::Retry.setup'

Finished in 11.46 seconds (files took 0.76603 seconds to load)
1100 examples, 1 failure, 1 pending

Failed examples:

rspec ./spec/sentry/client_spec.rb:261 # Sentry::Client#event_from_transaction correct dynamic_sampling_context when head SDK

Randomized with seed 60504

Failure 07

jruby

1) Sentry::Sidekiq::SentryContextClientMiddleware does not add user to the job if they're absent in the current scope
     Failure/Error: expect(queue.size).to be(1)

       expected #<Integer:3> => 1
            got #<Integer:9> => 4

       Compared using equal?, which compares object identity,
       but expected and actual are not the same object. Use
       `expect(actual).to eq(expected)` if you don't care about
       object identity in this example.
     # ./spec/sentry/sidekiq/sentry_context_middleware_spec.rb:161:in `block in <main>'

Finished in 6.1 seconds (files took 3.72 seconds to load)
70 examples, 1 failure

Failed examples:

rspec ./spec/sentry/sidekiq/sentry_context_middleware_spec.rb:[158](https://github.com/getsentry/sentry-ruby/actions/runs/13653741004/job/38168055672#step:7:159) # Sentry::Sidekiq::SentryContextClientMiddleware does not add user to the job if they're absent in the current scope

Randomized with seed 21411

Failure 08

Ruby 3.1
Rails 7.1

Failures:

  1) Sentry::Rails::Tracing with traces_sample_rate set records transaction alone
     Failure/Error: expect(second_span[:timestamp] - second_span[:start_timestamp]).to be_between(10.0 / 1_000_000, 10.0 / 1000)
       expected 0.018867015838623047 to be between 1.0e-05 and 0.01 (inclusive)
     # ./spec/sentry/rails/tracing_spec.rb:92:in `block (3 levels) in <top (required)>'

Finished in 30.82 seconds (files took 1.39 seconds to load)
152 examples, 1 failure, 7 pending

Failed examples:

rspec ./spec/sentry/rails/tracing_spec.rb:59 # Sentry::Rails::Tracing with traces_sample_rate set records transaction alone

Randomized with seed 41705

The text was updated successfully, but these errors were encountered:

sl0thentr0py · 2025-03-17T14:33:39Z

@solnic can we close this now?

solnic · 2025-03-18T09:17:51Z

@sl0thentr0py not yet :( I fixed a bunch of those (and possibly reduced the risk of introducing new ones in the future) but under jruby there are at least two more flaky tests to fix.

solnic self-assigned this Jan 24, 2025

solnic linked a pull request Feb 14, 2025 that will close this issue

Fix flaky specs #2561

Merged

solnic mentioned this issue Feb 14, 2025

Fix flaky specs #2561

Merged

solnic closed this as completed in #2561 Feb 17, 2025

solnic reopened this Feb 18, 2025

solnic mentioned this issue Feb 25, 2025

Flaky spec fixes #2565

Merged

sl0thentr0py added the ci label Mar 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky specs #2522

Fix flaky specs #2522

solnic commented Jan 24, 2025 •

edited

Loading

sl0thentr0py commented Mar 17, 2025

solnic commented Mar 18, 2025

Fix flaky specs #2522

Fix flaky specs #2522

Comments

solnic commented Jan 24, 2025 • edited Loading

Issue Description

Failure 01

Failure 02

Failures 03

Failure 03

Failure 04

Failure 05

Failure 06

Failure 07

Failure 08

sl0thentr0py commented Mar 17, 2025

solnic commented Mar 18, 2025

solnic commented Jan 24, 2025 •

edited

Loading