WIP: Introduce have_reported_error matcher #2849

skatkov · 2025-06-04T22:53:33Z

rspec-rails is missing support for Rails ErrorReporter, this was introduced to rails in v7 and has been evolving ever since. With my client, we have moved to using ErrorReporter as a unified error reporting interface, so we can easily move from one error tracking software to another with minimal code changes. And we had a need to test this interface with rspec, so we implemented our own matcher to handle this.

I'm suggesting our internal implementation as is. This is probably not suitable as is for this gem, but I'd like to open discussion with this starting point.

Example usage

@example Checking for any error
  expect { Rails.error.report(StandardError.new) }.to have_reported_error

@example Checking for specific error class
  expect { Rails.error.report(MyError.new) }.to have_reported_error(MyError)

@example Checking for specific error instance with message
  expect { Rails.error.report(MyError.new("message")) }.to have_reported_error(MyError.new("message"))

@example Checking error attributes
  expect { Rails.error.report(StandardError.new, context: "test") }.to have_reported_error.with_context(section: "test")

@example Checking error message patterns
  expect { Rails.error.report(StandardError.new("test message")) }.to have_reported_error(/test/)

@example Negation
  expect { "safe code" }.not_to have_reported_error

TODO

Outline of things that we want to do before marking this as completed.

Use does_not_match? for negated matcher.
Use a more strict values_match?(@attributes, actual) for matcher
Let's do matching_reports method.
WIP: Introduce have_reported_error matcher #2849 (comment)

pirj

Thank you!

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

lib/rspec/rails/matchers/have_reported_error.rb

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

features/matchers/have_reported_error_matcher.feature

JonRowe · 2025-06-25T17:14:51Z

features/matchers/have_reported_error_matcher.feature

+  Scenario: Using in controller specs
+    Given a file named "spec/controllers/users_controller_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe UsersController, type: :controller do
+        describe "POST #create" do
+          it "reports validation errors" do
+            expect {
+              post :create, params: { user: { email: "invalid" } }
+            }.to have_reported_error(ValidationError)
+          end
+        end
+      end
+      """
+    When I run `rspec spec/controllers/users_controller_spec.rb`
+    Then the examples should all pass
+
+  Scenario: Using in request specs
+    Given a file named "spec/requests/users_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe "Users", type: :request do
+        describe "POST /users" do
+          it "reports processing errors" do
+            expect {
+              post "/users", params: { user: { name: "Test" } }
+            }.to have_reported_error.with(context: "user_creation")
+          end
+        end
+      end
+      """
+    When I run `rspec spec/requests/users_spec.rb`
+    Then the examples should all pass
+


This isn't a controller / request specific matcher so these feel out of place, why these, why not every aspect of Rails... I'd just cut them.

spec/rspec/rails/matchers/have_reported_error_spec.rb

JonRowe · 2025-06-25T17:25:27Z

👋 I think this would be a great addition, sorry for the size of the review its been on my todo list for a while, its mostly just grammar / wording tweaks plus a few "fit" things.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

skatkov · 2025-06-27T11:45:47Z

@JonRowe thanks for review.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Which of these you would rather go with?
.have_reported_error(StandardError, /wrong attributes/).with(param1: 'test')

or

.have_reported_error(StandardError).with(message:/wrong attributes/, params1: 'test1')

??

I assume, that first option is prefered to keep similarity with raised_error matcher.
https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/raise-error/

JonRowe · 2025-06-27T11:59:45Z

Given that you already have with, I think:

.have_reported_error(StandardError).with(/wrong attributes/, params1: 'test1')

But if you insisted on message: in with I could be persuaded

pirj · 2025-06-27T14:31:21Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.
raise_error accepts have_attributes as an argument matcher. It supports with_message. Wdyt if we reserve with for additional args passed to report, but keep it separate from with_message?

skatkov · 2025-06-29T18:44:41Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.

The only argument that the exception class accepts is "message". We can of course subclass it and support way more than that, but I'm not sure if we should support this in a matcher?

I have a feeling, that having a matcher interface similar to raise_error would make it simpler for people to memorize it. I don't mind changing .with to .have_attributes as well.

JonRowe · 2025-06-30T08:02:50Z

I hadn't clocked that the extra attributes were in addition to the error, I think matching the raise_error interface in terms of report_error(ErrorClass, message_or_regexp) makes sense with the existing with becoming something clearer, with_metadata? What does Rails call the extra data?

skatkov · 2025-06-30T09:58:19Z

Here is a source code for Rails.error.report.

https://github.com/rails/rails/blob/36601bbb02bc3570f5609db2127c77afca575d6c/activesupport/lib/active_support/error_reporter.rb#L233C23-L233C112

These attributes in rails case have a name "context". so with_context might be more appropriate.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-11T18:55:38Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def failure_message
+          if !@error_subscriber.events.empty? && !@attributes.empty?


Cosmetic: any? or present??

pirj · 2025-07-11T18:57:19Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def failure_message
+          if !@error_subscriber.events.empty? && !@attributes.empty?
+            event_context = @error_subscriber.events.last.attributes[:context]


Won’t it be confusing to use last when several errors were reported?

pirj · 2025-07-11T19:07:30Z

lib/rspec/rails/matchers/have_reported_error.rb

+              return "Expected error message to be '#{@expected_message}', but got: #{reported_errors}"
+            end
+          else
+            if @expected_error && !actual_error.is_a?(@expected_error)


actual_error should be is_a? expected_class, no? Just by looking at the implementation. Or what is the case when they won’t match?

pirj · 2025-07-11T19:22:25Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+          case @expected_message
+          when Regexp
+            error.message&.match(@expected_message)


Cosmetic: is safe nav necessary?

yeah, sometimes error object could be nil.

How? Please add a spec for this case.

spec/rspec/rails/matchers/have_reported_error_spec.rb

pirj · 2025-07-12T09:56:44Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        private
+
+        def error_matches_expectation?


Is the last condition necessary? We already check if it’s empty? before calling this method.

This particular method checks that the error matches one that we expect, not just that the error occurred.

Here we check if @reports.empty? and return.

Right after the check, we call error_matches_expectation?.

The last condition check @reports.count.positive? here is redundant.

pirj

A fee minor things and simplifications and it looks good to go.

Let’s leave out matching multiple with chains, severity qualifiers, and all extras for later.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-12T13:08:34Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def matches?(block)
+          if block.nil?
+            raise ArgumentError, "this matcher doesn't work with value expectations"


Does this check have the same effect as defining a method supports_value_expectations? that returns false? It should be.

RSpec is actually allowing the value expectation to reach the matches? method, even with supports_value_expectations? defined as false. This mechanism is NOT preventing the value expectation from reaching matches?. The explicit check was necessary in my testing! But maybe I missed something?

Ha! Nice catch. The RSpec.deprecate call somehow didn't fail the build. I'll have a closer look.

The block is not nil though for expect(:foo).to reports_error, so this check is useless in its current implementation.
I suggest removing it, and relying on the supports_value_expectations?.

I recall now that we kept in 3.x the possibility to pass a lambda to expect, or define subject as a lambda and use is_expected.to with block matchers. We just print a deprecation message now.

Swallowed deprecation messages are due to #2857

After the aftermentioned fix, with

def supports_value_expectations? false end

The following passes:

it "warns when used as a value expectation" do expect { expect(Rails.error.report(StandardError.new("test error"))).to have_reported_error }.to raise_error(/implicit block expectation.+deprecated/) end

We can change the code when that swallowed deprecation fix is merged.

This can be removed now and any test marked as pending with the reason instead.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-12T13:18:59Z

lib/rspec/rails/matchers/have_reported_error.rb

+      #
+      # @param expected_error_or_message [Class, String, Regexp, nil] the expected error class, message string, or message pattern
+      # @param expected_message [String, Regexp, nil] the expected error message to match
+      def have_reported_error(expected_error_or_message = UndefinedValue, expected_message = nil)


We have default attributes both here and in the initializer. Worth leaving just here?

There are still defaults both here and there:

def initialize(expected_error_or_message = UndefinedValue, expected_message = nil)

Let's remove there.

pirj · 2025-07-12T13:27:05Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), context: { context: "user_processing", severity: :error })


Nested context here, too. Rename inner to topic: or :section?

features/matchers/have_reported_error_matcher.feature

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

Co-authored-by: Phil Pirozhkov <[email protected]>

skatkov force-pushed the have-reported-error branch from 8b837a8 to d2d0e51 Compare June 7, 2025 20:44

pirj reviewed Jun 7, 2025

View reviewed changes

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

skatkov force-pushed the have-reported-error branch 3 times, most recently from 71685ad to 6135d5d Compare June 22, 2025 20:59

pirj reviewed Jun 23, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

JonRowe requested changes Jun 25, 2025

View reviewed changes

skatkov force-pushed the have-reported-error branch 4 times, most recently from f90fc3b to 2488a71 Compare July 1, 2025 23:28

pirj reviewed Jul 2, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jul 2, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jul 11, 2025

View reviewed changes

pirj reviewed Jul 12, 2025

View reviewed changes

features/matchers/have_reported_error_matcher.feature Outdated Show resolved Hide resolved

skatkov and others added 28 commits September 24, 2025 13:13

should require a block matcher

d32437f

Create ErrorEvent struct to store errors

9e938ce

simplify matcher

bc9acfb

disable matcher chaining

f41ba1f

improved argument error message in case block is not passed to matcher

7de1c60

further clean-up

32d35ef

Apply suggestions from code review

492c728

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

clean-up features

c3e9343

remove a require from test

9907d1b

rename .with to .with_context

f58e4a7

have_reported_error to accept class name and a message

834b3bc

switch to error_report matcher type of api

dad179b

refactor

1b81eac

Using UndefinedValue as a default attribute

a30c7ae

Look through all errors, don't just go with last error

07a63d8

warn about nil being passed

9d87a40

Use ErrorCollector from Rails instead of self-written version'

d620e9d

Use defined UNDEFINED value instead of defining our own

07a8868

Update features/matchers/README.md

ef38475

Define supports_value_expectations? to false

248e743

remove log/development.log

cf1def7

Don't use default values in HaveReportedError initializer

569fa0e

Don't allow to redefine attributes with chaining in with_context method

c09b42c

adjust .feature file to properly illustrate report usage

47840d1

handle hegated matchers safely, not allowing any attributes

b6d0b0d

Remove from features/matchers/README.md

091d8b8

Co-authored-by: Phil Pirozhkov <[email protected]>

cleaning up a class

ad78776

remove context attribute in specs

aa4c84d

skatkov force-pushed the have-reported-error branch from b5568a0 to aa4c84d Compare September 24, 2025 11:13

block chaining for .or method as well

b20a699

Uh oh!

WIP: Introduce have_reported_error matcher #2849

Are you sure you want to change the base?

WIP: Introduce have_reported_error matcher #2849

Uh oh!

Conversation

skatkov commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example usage

TODO

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JonRowe commented Jun 25, 2025

Uh oh!

skatkov commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonRowe commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pirj commented Jun 27, 2025

Uh oh!

skatkov commented Jun 29, 2025

Uh oh!

JonRowe commented Jun 30, 2025

Uh oh!

skatkov commented Jun 30, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

skatkov commented Jun 4, 2025 •

edited

Loading

skatkov commented Jun 27, 2025 •

edited

Loading

JonRowe commented Jun 27, 2025 •

edited

Loading