Support Cucumber reporters #9

vitalets · 2023-04-12T16:25:08Z

Cucumber has several built-in reporters (also called formatters). It would be useful to output results of Playwright test runner to these reporters (especially html-reporter or messages reporter).
There are different approaches:

Write custom Playwright reporter that will create Cucumber report
Use Playwright json report and convert it into Cucumber messages, and then convert to html

Ideally, final usage should allow to use any Cucumber report:

import { defineConfig, devices } from '@playwright/test';
import { cucumberReporter } from 'playwright-bdd';

export default defineConfig({
  // ...
  reporter: cucumberReporter('json:cucumber-report.json'),
});

Known problem - reporting Scenario Outline.
In Cucumber it is possible to have the same test title with different data. In Playwright each test must have unique title (it was also discussed in #3 (comment)). Example of Cucumber html report for Scenario Outline:

I would appreciate more ideas on that from your reporting usage.

The text was updated successfully, but these errors were encountered:

evix1101 · 2023-04-12T17:55:00Z

Not elegant, and my needs are currently minimal. I'm supporting tests against multiple environments at the project level via extending PlaywrightTestProject

import { defineConfig, devices, PlaywrightTestProject } from '@playwright/test';

export interface ExtendedPlaywrightTestProject extends PlaywrightTestProject {
  testEnv?: string;
}

const config = defineConfig({ 
  testDir: './.features-gen',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: process.env.CI ? [ ['html'], ['junit', { outputFile: 'test-results/results.xml' }] ] : [ ['html'],['list']],
  use: {
    trace: 'retain-on-failure',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'prod-chromium',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: 'https://www.google.com/',
      },
      testEnv: 'prod',
    } as ExtendedPlaywrightTestProject, 
    {
      name: 'dev-chromium',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: 'https://dev.google.com/',
      },
      testEnv: 'dev',
    } as ExtendedPlaywrightTestProject,
  ],
});

export default config;

My thought was there might be a slick way to make this more dynamic to support the scenario outlines.

Jag46 · 2023-08-10T05:44:59Z

import { defineConfig, devices } from '@playwright/test';
import { cucumberReporter } from 'playwright-bdd';

export default defineConfig({
// ...
reporter: cucumberReporter('json:cucumber-report.json'),
});

Hi @vitalets , when can we expect cucumberReporter function? It would be great if get this up running soon..Thank you..!

vitalets · 2023-08-10T06:42:33Z

@Jag46 can't give concrete date but will increase the priority for that.
Btw are you interested in particular json reporter or html?

Jag46 · 2023-08-10T08:24:16Z

Thanks @vitalets , we are looking for a cucumber html report. That would great addition if you can help us with this blocker. Thanks

harikishanl · 2023-08-24T22:16:42Z

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

vitalets · 2023-08-26T09:31:40Z

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

Ok, got it. Could you clarify what parts of cucumber html reporter are the most important for you? E.g. formatting Scenario Outlines or something else? As there are a lot of events in both playwright and cucumber test run flows.

harikishanl · 2023-08-29T22:29:28Z

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

Ok, got it. Could you clarify what parts of cucumber html reporter are the most important for you? E.g. formatting Scenario Outlines or something else? As there are a lot of events in both playwright and cucumber test run flows.

Hi @vitalets - It would be helpful if we can cover: 'Feature, Scenario Outline, Scenario, Given, When, Then & Examples' in the report

vitalets · 2023-11-12T07:51:16Z

Cucumber messages can be validated with https://github.com/cucumber/compatibility-kit

dlitmano · 2024-01-07T12:43:21Z

@vitalets please inform me if i can help you. I love this project and want to support it!

vitalets · 2024-01-08T10:14:10Z

Thank you @dlitmano ! Answered in #82

magarwal19 · 2024-01-19T09:58:18Z

Looking forward for this feature!!

vitalets · 2024-01-19T12:08:45Z

Looking forward for this feature!!

It's already 80% complete. Planning to include it in the nearest release.

magarwal19 · 2024-02-01T10:32:18Z

Hi @vitalets
do we have any planned ETA for this?
My team is working on integrating test management tool and it needs the results to be pushed using cucumber jsons

vitalets · 2024-02-01T16:16:39Z

Hi @vitalets do we have any planned ETA for this? My team is working on integrating test management tool and it needs the results to be pushed using cucumber jsons

Hi @magarwal19 , I would say it's within 1-2 weeks. Cucumber json will be supported.

magarwal19 · 2024-02-01T18:51:56Z

This is great news!!
thank you :)

magarwal19 · 2024-02-14T08:10:49Z

Hi @vitalets
can you please let us know if we are planning to deploy this soon?

vitalets · 2024-02-14T15:20:53Z

Hi @magarwal19
Base part is ready, I need some time to write documentation.
Would it be helpful for you, if I push it in a separate branch and you will pull playwright-bdd directly from github and check Cucumber reporters?

magarwal19 · 2024-02-14T15:28:40Z

Hi @magarwal19 Base part is ready, I need some time to write documentation. Would it be helpful for you, if I push it in a separate branch and you will pull playwright-bdd directly from github and check Cucumber reporters?

Hi @vitalets
i have integrated it by installing the version of playwrigh-bdd npm package..
if we can get a beta release, then i can surely use that and check the cucumber reports

vitalets · 2024-02-15T09:20:01Z

I've published playwright-bdd beta with Cucumber reporters support.
Please follow the instructions below to check it with your projects.
I will appreciate any feedback on that!
@magarwal19 and everybody :)

1.Install playwright-bdd@next:

npm i -D playwright-bdd@next

2.Use cucumberReporter helper in the playwright config:

import { defineConfig } from '@playwright/test';
import { defineBddConfig, cucumberReporter } from 'playwright-bdd';

const testDir = defineBddConfig({
  importTestFrom: 'steps/fixtures.ts',
  paths: ['features'],
  require: ['steps/*.ts'],
});

export default defineConfig({
  testDir,
  reporter: [
    cucumberReporter('html', { outputFile: 'cucumber-report/report.html' }),
    cucumberReporter('json', { outputFile: 'cucumber-report/report.json' }),
    cucumberReporter('junit', { outputFile: 'cucumber-report/report.xml' }),
    cucumberReporter('message', { outputFile: 'cucumber-report/report.ndjson' }),
  ],
});

Here is the diff example.

Example of html report:

Please note that minimal Playwright version is 1.34.

magarwal19 · 2024-02-15T15:46:40Z

Hi @vitalets
thank you for sharing this!
two things i observed:

It is adding keyword as After, even when i do not have after. Also After is not BDD keyword so it throws error when i import in test management tool
There is huge data added as part of key "embeddings", not sure for what part do we need this..

HTML reports look good :)

vitalets · 2024-02-16T05:50:58Z

@magarwal19 could you provide more details on both two points?

magarwal19 · 2024-02-19T07:49:03Z

Hi @vitalets
few updates here..

When we are creating junit reports, we need to add <testsuites> </testsuites>
and then add the current report under this.. Due to this, whenever we use any utility to parse our junit xml it is failing
also not sure if we can add encoding at first step like
<?xml version="1.0" encoding="UTF-8"?>
currently we are getting it like <?xml version="1.0"?>

magarwal19 · 2024-02-19T10:18:25Z

@magarwal19 could you provide more details on both two points?

Hi @vitalets
i analysed this and this seems to be dependent on Playwright execution. Since i have Screenshot and videos to be on in my playwright.config.js file, there gets an entry added for After in results.json

If we can remove this, it will make this result json more light and easy to read

after.json

vitalets · 2024-02-21T10:29:31Z

If we can remove this, it will make this result json more light and easy to read

Yes, Playwright's automatic screenshots are attached as an after step, b/c we need some entity to attach them.
So if you don't need screenshots and videos in the report, why you can't just disable them in playwright.config.js?

magarwal19 · 2024-02-21T12:13:32Z

Hi @vitalets few updates here..

When we are creating junit reports, we need to add <testsuites> </testsuites>
and then add the current report under this.. Due to this, whenever we use any utility to parse our junit xml it is failing
also not sure if we can add encoding at first step like
<?xml version="1.0" encoding="UTF-8"?>
currently we are getting it like <?xml version="1.0"?>

Hi @vitalets
have updated this comment.. some how it was not showing complete information earlier

magarwal19 · 2024-02-21T12:14:48Z

If we can remove this, it will make this result json more light and easy to read

Yes, Playwright's automatic screenshots are attached as an after step, b/c we need some entity to attach them. So if you don't need screenshots and videos in the report, why you can't just disable them in playwright.config.js?

We would definitely need them in report, but in my case we did not need in json file but in html file..
will figure out using a script at my end

vitalets · 2024-02-22T09:47:57Z

@magarwal19

Will check on <testsuites> </testsuites> and <?xml version="1.0" encoding="UTF-8"?>. Btw, does your parser correctly handles Cucumber's original junit report? b/c I've ported this code from Cucumber without changes.
Got it. So the problem is large files due to embedded screenshots and videos, that are actually not needed? I will think about it. Also related to Support for having large attachments saved externally to the report cucumber/html-formatter#281

magarwal19 · 2024-02-22T11:22:21Z

@magarwal19

Will check on <testsuites> </testsuites> and <?xml version="1.0" encoding="UTF-8"?>. Btw, does your parser correctly handles Cucumber's original junit report? b/c I've ported this code from Cucumber without changes.

Got it. So the problem is large files due to embedded screenshots and videos, that are actually not needed? I will think about it. Also related to Support for having large attachments saved externally to the report cucumber/html-formatter#281

@vitalets
I was using junit xml...in junit it generate data with testsuites... this might be a change that in junit it shows but in cucumber it might not...

thank you for thinking about point 2 :)

vitalets · 2024-02-25T08:53:17Z

@magarwal19 I've checked original Cucumber junit report, it returns the same structure as playwright-bdd:

starts with <?xml version="1.0"?>, not <?xml version="1.0" encoding="UTF-8"?>
does not contain <testsuites> </testsuites>

This is the full Cucumber report. Could you try to load it with your tool?

<?xml version="1.0"?>
<testsuite failures="4" skipped="0" name="cucumber-js" time="0.004758075" tests="4">
  <testcase classname="Playwright site" name="Check title" time="0.000913249">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="Check title 2" time="0.000599414">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="A Get started step is Playwright" time="0.002910996">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="A Get started step is Play" time="0.00033441600000000003">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Play"...............................................skipped]]></system-out>
  </testcase>
</testsuite>

magarwal19 · 2024-02-26T17:46:00Z

Hi @vitalets
sure will check this.. thank you for sharing this!
i think teams integration plugin expects testsuites to be there..

NikkTod · 2024-03-07T16:32:51Z

Hi @vitalets, the html report looks great.
One thing I wanted to ask if you can make it possible the traces of a failed test to be opened directly within the browser with a click. Currently I need to download the trace file and open it with command 'npx playwright show-trace ....'.

Within TeamCity when I am using the playwright html report I also have the option to just click it and it will open in a frame, so no need to download it.

this is sample from the html how it is opened in TeamCity iframe

Maybe passing the URL of the traces could help

vitalets · 2024-03-09T07:59:58Z

One thing I wanted to ask if you can make it possible the traces of a failed test to be opened directly within the browser with a click. Currently I need to download the trace file and open it with command 'npx playwright show-trace ....'.

@NikkTod
Created a separate issue for that, pls check out questions there: #103

vitalets · 2024-03-11T08:47:39Z

Finally, Cucumber reporters are released in playwright-bdd v6 🥳
It was not-easy journey, but result is worth the efforts.
There were two main challenges during development:

Attachments. In Cucumber attachments are bound to the steps, but in Playwright attachments are bound to the whole test. Although there is no 100% way to map attachments to steps in Playwright, for BDD steps it's possible, b/c they are always serial. I've changed the approach several times, because I wanted to support Playwright's merge-reports mode, that provides a bit different data to the Reporter API. Now you can fully benefit from Playwright's sharding: run BDD tests on the multiple machines and get combined Cucumber report.
Projects. Cucumber reports do not support concept of projects that exists in Playwright. In Cucumber you get separate reports for Chrome, Firefox, etc, but in Playwright it's a single run and output. Playwright-bdd adopts Cucumber messaging protocol to handle several projects: basically it prepends project name to the feature uri making it unique. You can observe the result in the html reporter:

Having all browsers in a single run & report is also a good performance boost!

Thank you all for your help to achieve this goal. Feel free to share your feedback in separate issues ❤️

vitalets mentioned this issue Apr 15, 2023

Duplicate tests getting generated to handle Outline Scenarios #3

Closed

vitalets added the collecting feedback label Apr 21, 2023

vitalets mentioned this issue May 23, 2023

Wrap each cucumber step in test.step #22

Closed

vitalets mentioned this issue Sep 9, 2023

Feature: Scenarios coverage report #53

Open

vitalets mentioned this issue Nov 10, 2023

Feature: Add keywords (Given, When, Then) to html report #69

Closed

This was referenced Dec 15, 2023

Feature: Scenario outline with examples in the title #67

Closed

Question: how to move "background" step out of "beforeHooks" #78

Closed

This was referenced Jan 7, 2024

Feature: Cucumber.js json Report generation #82

Closed

Project roadmap for 2024 #83

Open

This was referenced Feb 2, 2024

Question: How can I integrate the results of BDD tests with reportportal.io? #88

Closed

[Feature]: Provide more reliable way to map attachments with steps microsoft/playwright#29323

Closed

vitalets mentioned this issue Feb 22, 2024

Question: In playwright report, the scenario is displayed twice #92

Closed

vitalets added the reporter label Mar 11, 2024

vitalets mentioned this issue Mar 11, 2024

Open Playwright trace from HTML report without downloading #103

Closed

vitalets closed this as completed Mar 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Cucumber reporters #9

Support Cucumber reporters #9

vitalets commented Apr 12, 2023 •

edited

Loading

evix1101 commented Apr 12, 2023 •

edited

Loading

Jag46 commented Aug 10, 2023

vitalets commented Aug 10, 2023

Jag46 commented Aug 10, 2023

harikishanl commented Aug 24, 2023

vitalets commented Aug 26, 2023

harikishanl commented Aug 29, 2023 •

edited

Loading

vitalets commented Nov 12, 2023

dlitmano commented Jan 7, 2024

vitalets commented Jan 8, 2024

magarwal19 commented Jan 19, 2024

vitalets commented Jan 19, 2024

magarwal19 commented Feb 1, 2024

vitalets commented Feb 1, 2024

magarwal19 commented Feb 1, 2024

magarwal19 commented Feb 14, 2024 •

edited

Loading

vitalets commented Feb 14, 2024 •

edited

Loading

magarwal19 commented Feb 14, 2024

vitalets commented Feb 15, 2024

magarwal19 commented Feb 15, 2024

vitalets commented Feb 16, 2024

magarwal19 commented Feb 19, 2024 •

edited

Loading

magarwal19 commented Feb 19, 2024 •

edited

Loading

vitalets commented Feb 21, 2024

magarwal19 commented Feb 21, 2024

magarwal19 commented Feb 21, 2024

vitalets commented Feb 22, 2024

magarwal19 commented Feb 22, 2024

vitalets commented Feb 25, 2024

magarwal19 commented Feb 26, 2024

NikkTod commented Mar 7, 2024

vitalets commented Mar 9, 2024 •

edited

Loading

vitalets commented Mar 11, 2024

Support Cucumber reporters #9

Support Cucumber reporters #9

Comments

vitalets commented Apr 12, 2023 • edited Loading

evix1101 commented Apr 12, 2023 • edited Loading

Jag46 commented Aug 10, 2023

vitalets commented Aug 10, 2023

Jag46 commented Aug 10, 2023

harikishanl commented Aug 24, 2023

vitalets commented Aug 26, 2023

harikishanl commented Aug 29, 2023 • edited Loading

vitalets commented Nov 12, 2023

dlitmano commented Jan 7, 2024

vitalets commented Jan 8, 2024

magarwal19 commented Jan 19, 2024

vitalets commented Jan 19, 2024

magarwal19 commented Feb 1, 2024

vitalets commented Feb 1, 2024

magarwal19 commented Feb 1, 2024

magarwal19 commented Feb 14, 2024 • edited Loading

vitalets commented Feb 14, 2024 • edited Loading

magarwal19 commented Feb 14, 2024

vitalets commented Feb 15, 2024

magarwal19 commented Feb 15, 2024

vitalets commented Feb 16, 2024

magarwal19 commented Feb 19, 2024 • edited Loading

magarwal19 commented Feb 19, 2024 • edited Loading

vitalets commented Feb 21, 2024

magarwal19 commented Feb 21, 2024

magarwal19 commented Feb 21, 2024

vitalets commented Feb 22, 2024

magarwal19 commented Feb 22, 2024

vitalets commented Feb 25, 2024

magarwal19 commented Feb 26, 2024

NikkTod commented Mar 7, 2024

vitalets commented Mar 9, 2024 • edited Loading

vitalets commented Mar 11, 2024

vitalets commented Apr 12, 2023 •

edited

Loading

evix1101 commented Apr 12, 2023 •

edited

Loading

harikishanl commented Aug 29, 2023 •

edited

Loading

magarwal19 commented Feb 14, 2024 •

edited

Loading

vitalets commented Feb 14, 2024 •

edited

Loading

magarwal19 commented Feb 19, 2024 •

edited

Loading

magarwal19 commented Feb 19, 2024 •

edited

Loading

vitalets commented Mar 9, 2024 •

edited

Loading