Random errors in the CI runs

[Sethuraman]

I have been hunting around for a solution for this problem and didn't find any, hence the post. We are using MSW to mock out responses. It is a react app, with vite and testing with vitest. We are using axios to make api calls.

We find that tests at random fail in CI but we are unable to reproduce it locally. Even in CI it fails once in a while.

I have tried a few things but this my setup right now -

Inside setupTests.ts, I have this -

export function makeServer(
  defaultHandlers: HttpHandler[] = fakeResponseHandlers
): SetupServer {
  const server = setupServer(...defaultHandlers)

  beforeAll(() =>
    server.listen({
      onUnhandledRequest: (req) => {
        console.error(`[MSW] ❗ Unhandled request: ${req.method} ${req.url}`)
        console.trace(`[MSW Trace]`)
      },
    })
  )

  afterEach(() => server.resetHandlers())
  afterAll(() => server.close())

  return server
}

My test file looks like this -

import { makeServer } from '@/testHelpers/setupTests'
import { beforeEach, describe, expect, it } from 'vitest'
import { http, HttpResponse } from 'msw'
import { userEvent } from '@testing-library/user-event/index'
import { screen, waitFor } from '@testing-library/react'
import {
  renderWithProviders,
} from '@/testHelpers/setupRenderer'
import { ControlRoutes } from '@/Routes'

const server = makeServer()
describe('RoleDetails', () => {

  beforeEach(() => {
    server.use(
      http.post(
        '/squareup.bitplat.control.api.v1beta1.UserService/CreateOrUpdateRole',
        () => {
          return HttpResponse.json({
            role: {
              role_name: 'treasury',
              description: 'New Role Description',
              user_identity_keys: ['user1', 'user2'],
            },
          })
        }
      )
    )
    server.use(
      http.post(
        '/squareup.bitplat.control.api.v1beta1.UserService/GetRoles',
        () => {
          return HttpResponse.json({
            roles: [
              {
                role_name: 'treasury',
                description: 'New Role Description',
                user_identity_keys: ['user1', 'user2'],
              },
            ],
          })
        }
      )
    )
  })

  it('shows server error when saving role details', async () => {
    renderRouteDetails('/role-details')
    server.use(
      http.post(
        '/squareup.bitplat.control.api.v1beta1.UserService/CreateOrUpdateRole',
        () => {
          return HttpResponse.text('an error occurred', {
            status: 500,
          })
        }
      )
    )
    await userEvent.type(screen.getByLabelText(/Role Name/i), 'treasury')
    await userEvent.type(
      screen.getByLabelText(/Description/i),
      'New Role Description'
    )
    await userEvent.type(
      screen.getByPlaceholderText(/LDAP\/Slack Handle/i),
      'user1'
    )
    await userEvent.click(screen.getByText(/Add User/i))
    await userEvent.click(screen.getByText(/Save/i))

    await waitFor(() => {
      expect(screen.getByText(/an error occurred/i)).toBeInTheDocument()
    })
  })

  function renderRouteDetails(path: string) {
    return renderWithProviders(<ControlRoutes />, { route: path })
  }
})

In my CI I get an error as below

stderr \| src/features/user/RoleDetails.test.tsx > RoleDetails > shows server error when saving role details
--
  | [MSW] ❗ Unhandled request: POST http://localhost:3000/squareup.bitplat.control.api.v1beta1.UserService/GetRoles
  | Trace: [MSW Trace]
  | at onUnhandledRequest (/builds/bitplat-control/bitplat-control/web/src/testHelpers/setupTests.ts:24:17)
  | at onUnhandledRequest (file:///builds/bitplat-control/bitplat-control/web/node_modules/msw/src/core/utils/request/onUnhandledRequest.ts:67:5)
  | at handleRequest (file:///builds/bitplat-control/bitplat-control/web/node_modules/msw/src/core/utils/handleRequest.ts:79:5)
  | at _Emitter.<anonymous> (file:///builds/bitplat-control/bitplat-control/web/node_modules/msw/src/node/SetupServerCommonApi.ts:62:26)
  | at emitAsync (file:///builds/bitplat-control/bitplat-control/web/node_modules/@mswjs/interceptors/src/utils/emitAsync.ts:23:5)
  | at file:///builds/bitplat-control/bitplat-control/web/node_modules/@mswjs/interceptors/src/utils/handleRequest.ts:121:5
  | at until (file:///builds/bitplat-control/bitplat-control/web/node_modules/@open-draft/until/src/until.ts:23:18)
  | at handleRequest (file:///builds/bitplat-control/bitplat-control/web/node_modules/@mswjs/interceptors/src/utils/handleRequest.ts:110:18)
  | at XMLHttpRequestController.xhrRequestController.onRequest (file:///builds/bitplat-control/bitplat-control/web/node_modules/@mswjs/interceptors/src/interceptors/XMLHttpRequest/XMLHttpRequestProxy.ts:64:34)

I also have tried making my vitest config to say only one worker at a time like this -

const vitestConfig =
  process.env.CI == 'true'
    ? {
        globals: true,
        environment: 'jsdom',
        setupFiles: './src/testHelpers/setupTests.ts',
        maxWorkers: 1,
        minWorkers: 1,
        isolate: true,
      }
    : {
        globals: true,
        environment: 'jsdom',
        setupFiles: './src/testHelpers/setupTests.ts',
      }

We still get these random errors. If I retrigger the build run it works fine. Does anything obvious standout?

My hunch is another test is resetting the handlers before this test can complete. If so, how do I about debugging this.

[kettanaito]

Hi, @Sethuraman. Afaik, it's a bad idea to nest before* and after* hooks in utility functions. You're making it much harder for your testing framework to apply then. That's likely the root cause for this intermittence. Move them out on the root level setupTest.ts and instead reuse the same server instance if you want a helper like makeServer(). We are showcasing that setup in the Node.js integration in the docs.

The second mistake I've spotted in your code is that you're adding server.use() overrides after render(). By doing that, you are welcoming a race condition between your component rendering and MSW learning about your request handlers. Please add those overrides before your render as a rule of thumb.