Attachments from hooks are dumped at the end of Scenarios and lose context

[dpsi]

👓 What did you see?

I recently upgraded to Cucumber 10.1 and found that the reports from the external html-report formatter do not work the same way the native html formatter did in previous versions of cucumber.
Attachments attached in hooks (such as AfterStep) are dumped at the end of the scenario.
I have included the JSON and message output as well if it might be helpful. I did have to redact them so some messages might be missing, but it should get the point across.

Generated HTML report

JSON report

Message format

✅ What did you expect to see?

I was expecting something like this:

where each attachment is shown preceding the step it was hooked on which is how it behaved before I upgraded Cucumber to 10.1:

Old Cucumber HTML report

Old JSON report

The report also does not really seem to look like what react-components' demo shows: https://cucumber-react-preview.netlify.app/?story=app--report--hooks-attachment although I will mention I don't really like how they collapse the hooks, but at least the attachments are shown where they were made.

📦 Which tool/library version are you using?

cucumber (10.1.1)
cucumber-ci-environment (10.0.1)
cucumber-core (15.3.0)
cucumber-cucumber-expressions (18.0.1)
cucumber-gherkin (34.0.0)
cucumber-html-formatter (21.15.1)
cucumber-messages (27.2.0)
cucumber-tag-expressions (8.0.0)
Ruby MRI 3.4.2

Worked as expected on:
cucumber (3.2.0)
cucumber-core (3.2.1)
cucumber-expressions (6.0.1)
cucumber-tag_expressions (1.1.1)
cucumber-wire (0.0.1)
gherkin (5.1.0)
Ruby MRI 3.4.2

🔬 How could we reproduce it?

test.feature:

Feature: Attachments in Cucumber Reports
Scenario: Attaching attachments in multiple steps in a Scenario
  Given I open a web browser
  When I go to the "about:license" page
  And I go to the "about:logo" page
  Then I go to the "about:robots" page

step_definitions/test.rb:

Given(/^I open a web browser$/) do
  # code to open a browser
end

When(/^I go to the "(.+)" page$/) do |page|
  # code to go to a page
end

support/env.rb:

AfterStep do |scenario|
  screenshot = @browser.save_screenshot
  attach(screenshot, 'image/png')
end
After do |scenario|
  screenshot = @browser.save_screenshot
  attach(screenshot, 'image/png')
end

📚 Any additional context?

Seems to be similar to the problem described in #1645
Similar to the reporter above, the attachments are not helpful if dumped at the end. In my usage, there may be several attachments per-hook or step.
The problem is even worse if Scenario Outline is used, all of the attachments for all of the examples are lumped together which makes it hard to find the attachments for a specific example:

Which differs from react-components' demo: https://cucumber-react-preview.netlify.app/?story=app--report--examples-tables-attachment

[mpkorstanje]

I see that you are using cucumber-html-formatter v21.15.1, could you try with the latest v22.2.0?

[mpkorstanje]

@luke-hill I see that provided message format renders just fine with v22.2.0. So I reckon the dependencies just need a bump.

https://github.com/cucumber/cucumber-ruby/blob/0758783a649e4f796e8391bf75d2afe87626fefc/cucumber.gemspec#L31C1-L31C63

[dpsi]

Yes, the dependency probably needs to be loosened as gem 'cucumber-html-formatter', '= 22.2.0' resulted bundler Installing cucumber 1.3.20 (was 10.1.1)

@mpkorstanje the report does improve with 22.2.0 although IMHO it is worse than the cucumber 3.2 report style 🥲

Seems like the hooks were "improved" in cucumber/react-components#415, maybe some further refinement is needed as apart from the duration there is no benefit in showing the hooks without attachments. (For example these hooks don't even tell me where they come from despite message having a source reference)

[luke-hill]

The original linked item you mentioned was never actioned. I'm currently knee deep in one item so I might not get to this for a while.

Bumping the requirements I can do as a small minor bump though - so I'll get on that (Hopefully that fixes the issue)

EDIT: It looks as though you bumped from cucumber 3.2, in which case the differences will be inevitably huge. We changed the entire message protocol around cucumber v5-6 or so. So it's likely you'll see big changes as the formatters were standardised then

[luke-hill]

Dependencies were bumped in cucumber 10.2.0.

If further issues are wanted to be improved via a feature, the new issue has been made