Attach custom data

[pmerwin]

Expected Behavior

I want to add any file found in cypress/logs to be uploaded at end of the test

Actual Behavior

@saucelabs/cypress-plugin
config = sauceLabsPlugin(on, config, {
region: 'us-west-1',
build: 'cypress-learn-webapp',
tags: ['learn-webapp-e2e'],
webAssetsDir: 'cypress/logs'
});

this breaks and videos are not uploaded ^

Steps to Reproduce the Problem

Try and send custom data with the plugin

Specifications

• Version:
• Platform:
• Subsystem:

[alexplischke]

I want to add any file found in {x} to be uploaded at end of the test

The reporter does not have that kind of an ability. I'm happy to take it as a feature request though!

webAssetsDir

I'll remove this from our options interface. This field has a very specific use case on our Sauce Labs VMs and is configured by our test runners. It's not meant to configured by end users. Sorry for the confusion this may have caused!

[pmerwin]

Awesome, I am trying to add some axe json reports I have landed in cypress/logs:

would be cool if could do something like:

config = sauceLabsPlugin(on, config, {
region: 'us-west-1',
build: 'cypress-learn-webapp',
tags: ['learn-webapp-e2e'],
addArtifacts: 'cypress/logs'
});

and the plugin will upload all supported file types found in logs (json, txt), in addition to the video, etc

[alexplischke]

@pmerwin I have usability question. The plugin currently creates a report (i.e. one Sauce Labs job) per spec file. For the kind of feature you're requesting, where exactly would you expect those artifacts to be uploaded to?
To all of them? That'd feel a little excessive and redundant.
Just one of them? Feels weird or arbitrary.
It'd be a different story if all of the specs were reported as a single Sauce Labs job.

[pmerwin]

I was thinking we could have a supported file types like .log,. txt, .json, and anything found of those types in the 'artifact dir' will be uploaded.

config = sauceLabsPlugin(on, config, {
region: 'us-west-1',
build: 'cypress-learn-webapp',
tags: ['learn-webapp-e2e'],
addArtifacts: 'cypress/logs'
});

Uploaded to the sauce job where the video is sent and available there via dropdown for logs.

https://share.zight.com/mXuQ5Nn0

• sorry I missed your comment I would have replied much sooner :)

[pmerwin]

Something like this:
extend the type:

// Configuration options for the Reporter.
export interface Options {
  region?: Region;
  build?: string;
  tags?: string[];
  webAssetsDir?: string;
  addArtifacts?: string; // Add this line
}

Key Changes:
addArtifacts Directory: The custom directory is created if it doesn't exist, and files from this directory are scanned for upload.
File Type Check: Both in collectAssets and syncAssets, only files with extensions matching webAssetsTypes are uploaded.

// 
  /**
   * Gathers test assets, including custom artifacts, for upload to Sauce Labs.
   * Only uploads files that match the defined `webAssetsTypes`.
   */
  collectAssets(results: RunResult[], report: TestRun): Asset[] {
    const assets: Asset[] = [];

    // Add Cypress assets like videos, screenshots
    for (const result of results) {
      const specName = result.spec.name;
      if (result.video) {
        assets.push({
          filename: this.resolveVideoName(path.basename(result.video)),
          path: result.video,
          data: fs.createReadStream(result.video),
        });
      }
      result.screenshots?.forEach((s) => {
        assets.push({
          filename: this.resolveAssetName(specName, path.basename(s.path)),
          path: s.path,
          data: fs.createReadStream(s.path),
        });
      });
    }

    // Add custom artifacts from the specified directory, checking their type
    if (this.opts.addArtifacts) {
      const artifactFiles = fs.readdirSync(this.opts.addArtifacts);
      for (const file of artifactFiles) {
        const ext = path.extname(file);
        if (webAssetsTypes.includes(ext)) {  // Ensure only allowed file types are uploaded
          const filePath = path.join(this.opts.addArtifacts!, file); // Use non-null assertion
          assets.push({
            filename: file,
            path: filePath,
            data: fs.createReadStream(filePath),
          });
        }
      }
    }

    // Add console log and Sauce test report
    assets.push(
      {
        data: this.ReadableStream(this.getConsoleLog(results[0])), // Adjust as needed
        filename: 'console.log',
      },
      {
        data: this.ReadableStream(report.stringify()),
        filename: 'sauce-test-report.json',
      },
    );

    return assets;
  }

  // Sync custom artifacts from the addArtifacts folder
  if (this.opts.addArtifacts) {
    const artifactFiles = fs.readdirSync(this.opts.addArtifacts);
    artifactFiles.forEach((file) => {
      const ext = path.extname(file);
      if (webAssetsTypes.includes(ext)) { // Ensure only allowed file types are synced
        fs.copyFileSync(path.join(this.opts.addArtifacts!, file), path.join(this.webAssetsDir || '', file));
      }
    });
  }
// 

[pmerwin]

I got this working locally :)

https://app.saucelabs.com/tests/9181172207dc43ffa730c2b506a5b2f0

https://share.zight.com/BluPQA6o

[alexplischke]

Thanks for the demo! 📺

We do have a usability concern though 🤔 I'll try my best explaining it! 🤞

The plugin creates a report per spec file. In your example/PR, a selection of files from the artifact folder are uploaded as a test result to Sauce. What happens when it's the next spec's turn to have a report created? Those files will be uploaded yet again. And again for the next one and so on. Should we skip the files we've already encountered between specs?

I'd love to find out more about your use case. Perhaps we can still design something that satisfies your needs!

I want to add any file found in cypress/logs to be uploaded at end of the test

What creates those files—or where from? If it's spec specific, we could potentially follow a specific pattern to be able to associate those files back to the spec. Let's say you are creating logs from within your spec files and writing them to cypress/logs/{spec-name}/your_file.log. In this case, the reporter would know which test report these files belong to. That could work.

[pmerwin]

I have it working here, we cleanse the dir after each spec run if it exists.

#81

https://share.zight.com/Kou8d0bx

[alexplischke]

That could indeed work! I shall bring it up with the team 🎉

[alexplischke]

@pmerwin Alright! I'm back with two proposals that I'd love to hear your thoughts on (and anyone else following this thread). Mind you, they're not mutually exclusive.

Proposal 1: Task for File Attachments

The idea here is to introduce a new task that you call from within your spec files, allowing you to explicitly "attach" files. The input could be a filepath or a stream along with the desired filename. The attached file will be uploaded to the job that matches the specified spec file.

Disclaimer: The task name in the screenshot merely serves to illustrate the example. Naming TBD.

Proposal 2: Folder Uploads

Similar to how Cypress itself stores videos and screenshots in user specified folders that underneath are spec specific (e.g. cypress/screenshots/{spec_name}/...), but unlike Cypress, we’d be consuming artifacts this way:

Upload all files from a user specified folder {user_specified}/{spec_name}/.... Keep contents after upload, but clean up at startup.

[pmerwin]

Awesome, we are currently using the fork I made, but I would be willing to test your strategy and adopt it instead.

[alexplischke]

@pmerwin Any personal preference from you and your team with regards to which of these two approaches (proposal 1 vs 2)? Either works?

[pmerwin]

Why not allow both? :)