Skip to content

results-api.mdx

Latest commit

 

History

History
420 lines (330 loc) · 12.6 KB

results-api.mdx

File metadata and controls

420 lines (330 loc) · 12.6 KB
title API for test results | Cypress UI Coverage
description Programmatically fetch UI Coverage results in a CI environment and fail the build if the test coverage is not acceptable.
sidebar_label Results API
sidebar_position 100

Results API

The @cypress/extract-cloud-results module provides the getUICoverageResults utility to programmatically fetch UI Coverage results for a run in a CI environment. This allows you to determine if test coverage meets your requirements before merging code changes.

Supported CI Providers

The utility supports the following CI providers. Refer to the linked guides for setup details:

For other CI providers, contact Cypress Support to request support.

Installation

Install the @cypress/extract-cloud-results module in your install step in CI.

npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz

:::caution

Do not check this module in as a dependency. We recommend you install it separately outside of your normal module installation. Use --force to get the latest version.

If you check this in as a dependency, your installation will fail when we update the package.

:::

Usage

1. Get the UI Coverage Results

Write a script to fetch UI Coverage results and assert test coverage criteria. This script will be executed in CI.

Basic example

This snippet uses the getUICoverageResults() helper to log out the results. It assumes your Project ID and Record Key variable are set. The following should work in any of the supported CI Providers out of the box:

// Assuming these environment variables are set:
// CYPRESS_PROJECT_ID=your-id
// CYPRESS_RECORD_KEY=your-record-key

const { getUICoverageResults } = require('@cypress/extract-cloud-results')

getUICoverageResults().then((results) => {
  // use `console.dir` instead of `console.log` because the data is nested
  console.dir(results, { depth: Infinity })
})

How to assert test coverage meets your requirements

The following example demonstrates how to verify that test coverage meets minimum thresholds:

const { getUICoverageResults } = require('@cypress/extract-cloud-results')

getUICoverageResults({
  projectId: process.env.CYPRESS_PROJECT_ID, // Optional if set from env
  recordKey: process.env.CYPRESS_RECORD_KEY, // Optional if set from env
  runTags: [process.env.RUN_TAGS], // Required if recording multiple runs
}).then((results) => {
  const { runNumber, uiCoverageReportUrl, summary, views } = results

  console.log(
    `Received ${summary.isPartialReport ? 'partial' : ''} results for run #${runNumber}.`
  )
  console.log(`See full report at ${uiCoverageReportUrl}.`)

  // Verify overall coverage
  if (summary.coverage < 80) {
    throw new Error(
      `Project coverage is ${summary.coverage}, below the minimum threshold of 80%.`
    )
  }

  const criticalViews = [/login/, /checkout/]

  // Verify critical view coverage
  views.forEach((view) => {
    const { displayName, coverage, uiCoverageReportUrl } = view

    if (
      criticalViews.some((pattern) => pattern.test(displayName)) &&
      coverage < 95
    ) {
      throw new Error(
        `Critical view "${displayName}" coverage is ${coverage}%, below the required 95%. See: ${uiCoverageReportUrl}`
      )
    }
  })

  console.log('UI Coverage is above minimum thresholds.')
})

getUICoverageResults arguments

getUICoverageResults accpets the following arguments:

getUICoverageResults({
  // The Cypress project ID.
  // Optional if the CYPRESS_PROJECT_ID env is set
  projectId: string
  // The project's record key.
  // Optional if the CYPRESS_RECORD_KEY env is set
  recordKey: string
  // The run tags associated with the run.
  // Required IF you are recording multiple Cypress runs from a single CI build.
  // Pass the run tags you used when recording in each run
  runTags: string[]
})

Result Details

The getUICoverageResults utility returns the following data:

{
  // The run number of the identified build.
  runNumber: number
  // The run url for the identified build.
  runUrl: 'https://cloud.cypress.io/projects/:project_id/runs/:run_number'
  // The status of the identified build.
  runStatus: 'passed' | 'failed' | 'errored' | 'timedOut' | 'cancelled' | 'noTests'
   // The url that links to UI Coverage report for the identified build.
  uiCoverageReportUrl: 'https://cloud.cypress.io/[...]'
  summary: {
    // Indicates whether a complete UI Coverage report was generated.
    // For example, if a run was cancelled and the report expected to run
    // for 20 specs, but only 10 ran, this would result in a partial report.
    isPartialReport: boolean
    // The report coverage from 0-100 with 2 decimal precision (e.g 92.45).
    coverage: number
    // The number of views tested and analyzed.
    viewCount: number
    // The number of interactive elements that were tested.
    testedElementsCount:number
    // The number of interactive elements that were not tested.
    untestedElementsCount: number
  }
  // The list of tested views and the coverage of each page.
  views: [{
    // The sanatized URL pattern shown in the report.
    displayName: string
    // The view coverage from 0-100 with 2 decimal precision (e.g 92.45).
    coverage: number
    // The number of interactive elements that were tested on this view.
    testedElementsCount:number
    // The number of interactive elements that were not tested on this view.
    untestedElementsCount: number
    // The url that links the report for this view.
    uiCoverageReportUrl: 'https://cloud.cypress.io/[...]'
  }]
}

Comparing against a baseline

For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the Block pull requests and set policies guide.

2. Add to CI Workflow

In your CI workflow that runs your Cypress tests,

  1. Update your install job to install the @cypress/extract-cloud-results module.
  2. Pass in the necessary arguments to getUICoverageResults.
  3. Add a new step to the job that runs your Cypress tests to verify the UI Coverage results.

:::info

If you record multiple runs in a single CI build, you must record these runs using the --tag parameter and then call getUICoverageResults with the runTags argument for each run.

This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. Tags can also be used to activate Profiles that apply different configuration settings to your runs.

Example

  • Let's imagine that within a single CI build you call cypress run --record multiple times because you're running one set of tests against a staging environment, followed by a production environment.
  • In this scenario, you pass a different --tag to each cypress run
    • cypress run --record --tag staging
    • cypress run --record --tag production
  • When calling getUICoverageResults you would then pass these same tags to get the unique set of results for each run
    • getUICoverageResults({ runTags: ['staging']})
    • getUICoverageResults({ runTags: ['production']})

:::

Example Job Workflow Update:

name: My Workflow
on: push

env:
  CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

jobs:
  run-cypress:
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      - name: install
        run: npm install
      - name: Run
        run: npx cypress run --record
+     - name: Verify UI Coverage Results
+       run: |
+          npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz
+          node ./scripts/verifyUICoverageResults.js
name: Run Cypress Tests

image: node:latest

stages:
  - test

run-cypress:
  stage: test
  secrets:
    CYPRESS_RECORD_KEY:
      vault: vault/cypressRecordKey
  script:
    - npm install
    - npx cypress run --record
+   - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz
+   - node ./scripts/verifyUICoverageResults.js
pipeline {
  agent {
    docker {
      image 'cypress/base:22.15.0'
    }
  }

  environment {
    CYPRESS_PROJECT_ID: 'xxxx'
    CYPRESS_RECORD_KEY = credentials('cypress-record-key')
  }

  stages {
    stage('build and test') {
      steps {
        sh 'npm ci'
        sh 'npx cypress run --record'
      }
    }

+   stage('Verify UI Coverage Results') {
+     steps {
+       sh 'npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz'
+       sh 'node ./scripts/verifyUICoverageResults.js'
+     }
+   }
  }
}
jobs:
  - job: run_tests
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '20.x'
          displayName: 'Install Node.js'

      - script: npm i
        displayName: 'Install npm dependencies'

      - script: npx cypress run --record
        displayName: 'Run Cypress tests'
        env:
          # avoid warnings about terminal
          TERM: xterm
          CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY)

+     - script: |
+           npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz
+           node ./scripts/verifyUICoverageResults.js
+       displayName: 'Verify UI Coverage Results'
+       env:
+         CYPRESS_PROJECT_ID: $(CYPRESS_PROJECT_ID)
+         CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY)
version: 2.1

jobs:
  linux-test:
    docker:
      - image: cypress/base:22.15.0

    working_directory: ~/repo
    steps:
      - checkout
      - run: npm install
      - run: npx run cypress:run --record
+     - run: npm install --force https://cdn.cypress.io/extract-cloud-results/beta/v1/extract-cloud-results.tgz
+     - run: node ./scripts/verifyUICoverageResults.js

workflows:
  version: 2
  tests:
    jobs:
      - run-cypress
phases:
  install:
    runtime-versions:
      nodejs: latest
    commands:
      # Set COMMIT_INFO variables to send Git specifics to Cypress Cloud when recording
      # https://docs.cypress.io/app/continuous-integration/overview#Git-information
      - export COMMIT_INFO_BRANCH="$(git rev-parse HEAD | xargs git name-rev |
        cut -d' ' -f2 | sed 's/remotes\/origin\///g')"
      - export COMMIT_INFO_MESSAGE="$(git log -1 --pretty=%B)"
      - export COMMIT_INFO_EMAIL="$(git log -1 --pretty=%ae)"
      - export COMMIT_INFO_AUTHOR="$(git log -1 --pretty=%an)"
      - export COMMIT_INFO_SHA="$(git log -1 --pretty=%H)"
      - export COMMIT_INFO_REMOTE="$(git config --get remote.origin.url)"
      - npm ci
  pre_build:
    commands:
      - npm run cypress:verify
      - npm run cypress:info
  build:
    commands:
      - CYPRESS_INTERNAL_ENV=staging CYPRESS_PROJECT_ID=[slug] npx cypress run --record --key [KEY]
+  post_build:
+    commands:
+      - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz
+      - CYPRESS_INTERNAL_ENV=staging CYPRESS_PROJECT_ID=[slug] CYPRESS_RECORD_KEY=[KEY] node ./scripts/verifyUICoverageResults.js
```diff title=".drone.yaml" kind: pipeline name: default

environment: CYPRESS_PROJECT_ID: example_project_slug CYPRESS_RECORD_KEY: from_secret: example_record_key_secret

steps:

  • name: test image: node:latest commands:
    • npm install
    • npx cypress run --record
</TabItem>
</Tabs>

## Required CI environment variables

<ResultsApiEnvVars />