Vitest for unit testing

[benmccann]

@sveltejs/kit/vite adds improved support for Vitest and I think we should probably add it to create-svelte though I'd be interested to hear more opinions about this.

If we do add it to create-svelte there are some questions about the ideal test setup. E.g. do we write component tests with Vitest (possibly with @testing-library/svelte) or Playwright (@playwright/experimental-ct-svelte)? And do we keep both test runners (e.g.: vitest scans *.test.(js|ts), playwright scans *.e2e.(js|ts)) or run everything with a single test runner? Are there other libraries we should include like svelte-inline-compile.

Should we just kick the can on these questions and not include component testing for the time being? Or document the options and let users decide?

[vekunz]

I think that both, Vitest and Playwright are necessary for comprehensive testing. To differentiate between the two: Vitest for component tests, where every dependency is mocked (and "normal" unit tests for business code of course). Playwright for end-to-end tests, where nothing is mocked (or at most outgoing HTTP calls).

[benmccann]

Thanks for sharing your feedback guys. @davipon I wonder if your recommendation might change if we fixed sveltejs/svelte#2588? Although that perhaps only addresses the slots issue. I'm not quite as familiar with what you're referring to when you talk about there being issues with directives and two-way bindings as I haven't done much testing of individual components myself. Could you expand on those issues? Are there issues in the Svelte repo for those items we might be able to help get prioritized?

[wd-David]

@benmccann, sorry for the late reply. The fix of sveltejs/svelte#2588 is great, and yes, it only addresses the slots issue.

I'm working on test recipes of how to test Svelte components using Vitest, Testing Library, and svelte-htm (svelte-inline-component was removed from npm) - svelte-component-test-recipes, which includes two-way bindings, use directives, slots, ...etc. (highly inspired by the Svelte Society Recipes)

The only issue I had was sveltejs/svelte#6584, which sveltejs/svelte#6646 can fix, but I'm not sure if it's the right way because most of the users, including me, don't have a holistic view of the compiler, may need the core team to triage.

[benmccann]

Thanks. It looks like an issue in svelte-htm to me rather than in Svelte: kenoxa/svelte-htm#234

[benmccann]

Thanks for sharing your testing recipe on Discord! I'm posting a link here so that I don't lose track of it: https://github.com/davipon/svelte-component-test-recipes

[wd-David]

Thanks for sharing your testing recipe on Discord! I'm posting a link here so that I don't lose track of it: https://github.com/davipon/svelte-component-test-recipes

@vekunz @bertybot It'd be great if you could take a look at those recipes.
By leveraging your experience, I believe we can make it more useful for the community.

[wd-David]

I just updated my svelte-add-vitest adder and used @sveltejs/kit/experimental/vite under the hood.
Maybe it's a good starting point to add vitest to create-svelte.

[bertybot]

Created a PR to add Vitest with svelte-create here #5708

My PR adds @testing-library/svelte

Sorry, totally missed this discussion. Didn't even know @playwright/experimental-ct-svelte was an option. Just going to throw my hat in the ring by saying that at my company we have been writing all of our tests using @testing-library/svelte and it has been amazing. We have around like 200 unit tests on Svelte components and we haven't had any problems, and our code quality has went up a ton thanks to them. The only real problem we have run into is the testing of slots really as @vekunz mentioned.

I'll look into @playwright/experimental-ct-svelte but, to me e2e tests are completely different than unit tests. So I think, it makes sense to have two different test runners. Playwright being for testing the application as a whole (routes folder) , and Vitest for testing the individual parts (lib folder).

Has anyone got an example of a working vitest config utilizing. @playwright/experimental-ct-svelte ?

[bertybot]

I had to remove @testing-library/svelte from my PR but, I published the branch with testing-library in my fork if people want to check it out. https://github.com/bertybot/kit/tree/vitest-testing-library-svelte

[wd-David]

Has anyone got an example of a working vitest config utilizing. @playwright/experimental-ct-svelte ?

Currently @playwright/experimental-ct-svelte doesn't interpret your vite.config.js as mentioned here: microsoft/playwright#14298, so we need to explicitly configure vite in playwright-ct.config.ts:

// playwright-ct.config.ts
import type { PlaywrightTestConfig } from '@playwright/experimental-ct-svelte';
import { resolve } from 'node:path';

const config: PlaywrightTestConfig = {
  testDir: 'tests/component',
  use: {
    ctViteConfig: {
      resolve: {
        alias: {
          $lib: resolve('src/lib')
        }
      }
    }
  }
};

export default config;

I'm working on a blog post about the component test using both Vitest & Playwright, you can check the demo repo here: svelte-component-test-demo.

[bertybot]

@davipon the example repo is awesome! Exactly what I was looking for. Nice work dude!

[cheapCoder]

@davipon I am trying to use playwright to do e2e testing and unit testing, similar to your demo, is there any way to get test coverage?

[wd-David]

@cheapCoder I didn't do test coverage in playwright personally, but if you're not looking for the experimental component test coverage, maybe you can check this out: playwright-test-coverage.

[wd-David]

It might be off-topic here, but we also need to figure out how to handle SvelteKit runtime modules like $app/stores & $app/navigation (refs: #1485) when it comes to the component test no matter you're using Vitest, Playwright, or Storybook.

Here are the workarounds I saw so far:

• Use Vitest mock capability (ref: vitest-svelte-kit, Unable to test files with $app/stores import after migrating to new vite.config.js format #5525)
• TestHarness.svelte (Playwright compatible) (ref: Shim SvelteKit runtime import aliases / Importing $app/* fails #1485 (comment))
• Use storybookjs/builder-vite and add aliases to runtime modules (ref: Shim SvelteKit runtime import aliases / Importing $app/* fails #1485 (comment))
• Use Node experimental ESMoudle Loaders API (ref: Template tests #19 (comment))

IMO they're hacky for anyone who wants to start testing their SvelteKit app. It'd be better to have a unifying approach.

[bertybot]

Use Vitest mock capability (ref: vitest-svelte-kit, #5525)

Shooting from the hip right now, but to me this seems like the best approach. Acknowledging a lot of my bias, I have way more experience testing in C# than JS but, in C# you are supposed to mock every single external dependency, that way you are only testing your actual code. I would argue that actually running Sveltekit code in your tests would be non standard since you wouldn't be testing your code in isolation.

[bertybot]

Alright so here's my 2 cents on the problem. The problem is currently we have the option for 2 test runners for individual component testing, vitest with testing-library/svelte and Playwright. Using Playwright would be nice since we already use it for e2e tests and it would be nice to not have to install and learn another thing. My problem with this is that their Component testing is currently considered experimental, and seems to have a few issues. Whereas, I would argue that I think Vitest with testing-library/svelte seems to be more stable and currently my company uses it to test production code. I would not feel comfortable right now telling people to start using something experimental like playwright component testing. My point is I don't think either of these approaches are definitive, there are pluses and minuses to each, and I don't think we will ever have a definitive approach. I know a lot of people really like Vitest and would be willing to use it instead of playwright even with playwright installed, and there will definitely always be people that just want to use playwright especially once their component testing gets more stable. Only time will tell.

My suggestion would be maybe taking an approach similar to the create-vue where we allow users to chose the test runner they want based on preferences. Basically saying for now we should just embrace the controversy lol.

[bjon]

Anyone here with a work-around for this?

[wd-David]

@bjon , you need to mock SvelteKit's runtime modules like $app/*:

vi.mock('$app/enviornment', () => ({
  browser: false,
  dev: true,
  prerender: false
)

I'm also working on a repo: Svelte Component Test Recipes. Still WIP, it might be helpful.

[bjon]

@davipon thanks alot!

[ledfordjs]

@davipon How would you update the stores value for each test like $page.params?

[Shinypaper]

@davipon How would you update the stores value for each test like $page.params?

I also have the same question here as @ledfordjs.
I am able to set up one value in the setup file but having the ability to change these values specifically is important.

For a bit more context if want to have different options available for different user roles I can do something like this on $page.data

// mocks.ts
export const nonAdminRolePage = readable({
  url,
  data: {
    user: {
      id: '1',
      email: '[email protected]',
      admin: false 
    }
  },
})

export const adminRolePage = readable({
  url,
  data: {
    user: {
      id: '1',
      email: '[email protected]',
      admin: true
    }
  },
})

// testfile.js
beforeEach(() => {
   vi.mock('$app/stores', () => {
      return { page: adminRolePage }
    })
})
test('admin user sees admin stuff', () => {
   expect(...admin to see admin stuff)
  }
})

The above test works well but in order to test both the admin path and the non admin path i would expect to be able to do something like this:

const { nonAdminRolePage, adminRolePage } = await vi.hoisted(() => import('path-to/mocks'))

 afterEach(() => {
    cleanup()
    // RESET MOCK HERE SOMEHOW?
  })

describe('non admin', () => {

  beforeEach(() => {
    vi.mock('$app/stores', async () => {
      return { page: nonAdminRolePage }
    })
  })

  test('non admin user does not see admin stuff', () => {
   expect(...non  admin to see not see admin stuff)
  }
})

describe('admin', () => {

  beforeEach(() => {
    vi.mock('$app/stores', async () => {
      return { page: adminRolePage }
    })
  })

  test('should see admin stuff', () => {
    expect(...admin to see see admin stuff)
  });
})

[jasonbhuang]

Thanks @davipon for the great example project its really helped me understand the testing set up and flow a lot better. However I am still struggling to understand how to test the Load data object in sveltekit. I understand you can set up api mocks but what about server side rendered logic and properties as shown here

https://kit.svelte.dev/docs/load

I am using server side to securely call some api's and only returning part of the data that I want exposed. Vitest simple starts the render by saying data is not rendered. I can't even return a static test string via the page data object.

[opack]

Today I struggled on the same problem with +page.server.ts and its load function not being executed. I found this simple solution: just call the load function and feed its result to the data prop of the page :-)

import { load } from './+page.server'

describe("Page Synonyms", () => {
        it("displays a list of aliases", () => {
        // ...
        // We must call `load` ourselves because Vitest does not do it for the moment, due to page.server not being able to be called outside of SvelteKit context (https://github.com/sveltejs/kit/issues/1485#issuecomment-920923252)
        const data = load()
        render(page, {
            props: {
                data
            }
        })
        // Asserts
    })
})

And voilà! Hope that helps you too 😉

[DanielRios549]

Does not work if load function uses a parameter. You need to mock the parameters too