Skip to content

Add support for execution sharding #2600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Please see [CONTRIBUTING.md](./CONTRIBUTING.md) on how to contribute to Cucumber
## [Unreleased]
### Added
- Export configuration types ([#2598](https://github.com/cucumber/cucumber-js/pull/2598))
- Add support for execution sharding ([#2303](https://github.com/cucumber/cucumber-js/pull/2303))

## [12.1.0] - 2025-07-19
### Added
Expand Down
1 change: 1 addition & 0 deletions compatibility/cck_spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ describe('Cucumber Compatibility Kit', () => {
names: [],
tagExpression: '',
order: 'defined',
shard: '',
},
support: {
requireModules: ['ts-node/register'],
Expand Down
2 changes: 2 additions & 0 deletions exports/api/report.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ export interface IConfiguration {
requireModule: string[];
retry: number;
retryTagFilter: string;
shard: string;
strict: boolean;
tags: string;
worldParameters: JsonObject;
Expand Down Expand Up @@ -146,6 +147,7 @@ export interface ISourcesCoordinates {
names: string[];
order: IPickleOrder;
paths: string[];
shard: string;
tagExpression: string;
}

Expand Down
1 change: 1 addition & 0 deletions exports/root/report.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ export interface IConfiguration {
requireModule: string[];
retry: number;
retryTagFilter: string;
shard: string;
strict: boolean;
tags: string;
worldParameters: JsonObject;
Expand Down
68 changes: 68 additions & 0 deletions features/sharding.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
Feature: Running scenarios using sharding
As a developer running features
I want an easy way to run specific scenarios by tag
So that I don't waste time running my whole test suite when I don't need to

Background:
Given a file named "features/a.feature" with:
"""
Feature: some feature
@a
Scenario: first scenario
Given a step
@b
Scenario Outline: second scenario - <ID>
Given a step
@c
Examples:
| ID |
| X |
| Y |
@d
Examples:
| ID |
| Z |
"""
And a file named "features/step_definitions/cucumber_steps.js" with:
"""
const {Given} = require('@cucumber/cucumber')
Given('a step', function() {})
"""

Scenario: run a single scenario
When I run cucumber-js with `--shard 1/5`
Then it passes
And it runs the scenario "first scenario"

Scenario: run every other scenario starting at 1
When I run cucumber-js with `--shard 1/2`
Then it passes
And it runs the scenarios:
| NAME |
| first scenario |
| second scenario - Y |

Scenario: run every 3rd scenario starting at 1
When I run cucumber-js with `--shard 1/3`
Then it passes
And it runs the scenarios:
| NAME |
| first scenario |
| second scenario - Z |

Scenario: run even scenarios
When I run cucumber-js with `--shard 2/2`
Then it passes
And it runs the scenarios:
| NAME |
| second scenario - X |
| second scenario - Z |

Scenario: no scenarios in shard
When I run cucumber-js with `--shard 5/5`
Then it passes
And it runs 0 scenarios
1 change: 1 addition & 0 deletions src/api/convert_configuration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export async function convertConfiguration(
names: flatConfiguration.name,
tagExpression: flatConfiguration.tags,
order: flatConfiguration.order,
shard: flatConfiguration.shard,
},
support: {
requireModules: flatConfiguration.requireModule,
Expand Down
1 change: 1 addition & 0 deletions src/api/convert_configuration_spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ describe('convertConfiguration', () => {
order: 'defined',
paths: [],
tagExpression: '',
shard: '',
},
support: {
requireModules: [],
Expand Down
5 changes: 5 additions & 0 deletions src/api/load_sources_spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ describe('loadSources', () => {
paths: [],
names: [],
tagExpression: '',
shard: '',
},
environment
)
Expand Down Expand Up @@ -112,6 +113,7 @@ describe('loadSources', () => {
paths: ['features/test.feature:8'],
names: [],
tagExpression: '',
shard: '',
},
environment
)
Expand All @@ -127,6 +129,7 @@ describe('loadSources', () => {
paths: [],
names: ['two'],
tagExpression: '',
shard: '',
},
environment
)
Expand All @@ -142,6 +145,7 @@ describe('loadSources', () => {
paths: [],
names: [],
tagExpression: '@tag2',
shard: '',
},
environment
)
Expand All @@ -157,6 +161,7 @@ describe('loadSources', () => {
paths: ['@rerun.txt'],
names: [],
tagExpression: '',
shard: '',
},
environment
)
Expand Down
6 changes: 6 additions & 0 deletions src/api/plugins.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { PluginManager } from '../plugin'
import publishPlugin from '../publish'
import filterPlugin from '../filter'
import shardingPlugin from '../sharding'
import { UsableEnvironment } from '../environment'
import { IRunConfiguration, ISourcesCoordinates } from './types'

Expand Down Expand Up @@ -37,5 +38,10 @@ export async function initializeForRunCucumber(
filterPlugin,
configuration.sources
)
await pluginManager.initCoordinator(
'runCucumber',
shardingPlugin,
configuration.sources
)
return pluginManager
}
7 changes: 7 additions & 0 deletions src/api/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,13 @@ export interface ISourcesCoordinates {
* Run in the order defined, or in a random order
*/
order: IPickleOrder
/**
* Shard tests and execute only the selected shard, format `<index>/<total>`
* @example 1/4
* @remarks
* Shards use 1-based numbering
*/
shard: string
}

/**
Expand Down
4 changes: 4 additions & 0 deletions src/configuration/argv_parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,10 @@ const ArgvParser = {
'--order <TYPE[:SEED]>',
'run scenarios in the specified order. Type should be `defined` or `random`'
)
.option(
'--shard <INDEX/TOTAL>',
'run shard INDEX of TOTAL shards. The index starts at 1'
)
.option(
'-p, --profile <NAME>',
'specify the profile to use (repeatable)',
Expand Down
1 change: 1 addition & 0 deletions src/configuration/default_configuration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export const DEFAULT_CONFIGURATION: IConfiguration = {
requireModule: [],
retry: 0,
retryTagFilter: '',
shard: '',
strict: true,
tags: '',
worldParameters: {},
Expand Down
5 changes: 5 additions & 0 deletions src/configuration/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,11 @@ export interface IConfiguration {
* @see {@link https://github.com/cucumber/cucumber-js/blob/main/docs/parallel.md}
*/
parallel: number
/**
* Shard tests and execute only the selected shard, format `<index>/<total>`
* @default ""
*/
shard: string
/**
* Publish a report of your test run to https://reports.cucumber.io/
* @default false
Expand Down
3 changes: 3 additions & 0 deletions src/sharding/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { shardingPlugin } from './sharding_plugin'

export default shardingPlugin
19 changes: 19 additions & 0 deletions src/sharding/sharding_plugin.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import { InternalPlugin } from '../plugin'
import { ISourcesCoordinates } from '../api'

export const shardingPlugin: InternalPlugin<ISourcesCoordinates> = {
type: 'plugin',
coordinator: async ({ on, options }) => {
on('pickles:filter', async (allPickles) => {
if (!options.shard) {
return allPickles
}

const [shardIndexStr, shardTotalStr] = options.shard.split('/')
const shardIndex = parseInt(shardIndexStr, 10) - 1
const shardTotal = parseInt(shardTotalStr, 10)

return allPickles.filter((_, i) => i % shardTotal === shardIndex)
})
},
}