[UI/UX support for attestations] User Testing

[kborchers]

• Validate key UX patterns through rapid feedback with target users or stakeholders.
• Develop user testing protocol and script.
• Conduct outreach and schedule user testing.
• Openly publish synthesis & summary of testing findings.
• Implement feedback and findings from user testing to improve style guide.

[Erioldoesdesign]

@oilsinwater here's the issue that describes the user testing re. rapid feedback for folks lookin at UIs etc.

[Erioldoesdesign]

Usertesting script for developed versions of UI below

---

Interview Guide

As interviewer, remind yourself of the alias/code name of the user. We always give all participants a code name to refer to them for safety, security and privacy of information so that as little of the subsequent information given in the interview is attachable to a real documented identity as possible.

Remind yourself of some of the demographic information from the screener survey such as:

• Workplace/Open Source org/project affiliation
  • {Notes here}
• Gender Identification/pronouns
  • {Notes here}
• Region/location aka morning, daytime, nighttime of interview
  • {Notes here}
• Their perceived/self-reported technological and security proficiency and/or the persona type (as found in this public issue: [UI/UX support for attestations] Attestations Personas v0.1 #70 (comment)) in order to avoid/clarify certain jargon/terminology in the interview including which registry they have most experience with e.g. npm, RubyGems or PyPI
  • {Notes here}

---

Pre-amble

Hello, your code name is NAME, we may refer to you as this throughout the interview for privacy reasons.

Thank you for joining this conversation. I'm NAME from ORG NAME where I do research. We are conducting research across multiple programming language ecosystems on behalf of the Open SSF, looking at understanding how developers, software engineers and open source software users understand the security, safety and integrity of the packages they install and their 'open source software supply chains'.

Over the next 45-60 minutes, we’re going to show you some mock-up UI examples on registry pages (npm, RubyGems and/or PyPI) and ask you a series of questions that help us to understand what the UI tells you and how you understand it.

Your insights, opinions and experiences will help to inform the next stages of this UI.

I want to hear your honest thoughts, so please think aloud. Feel free to speak your mind. We are not testing your speed or knowledge, we are exploring your the processes and information to see if it meets your needs.

• Are you ok if we record so we can ensure notes we take are accurate? We do not use video.
• We will also be taking handwritten notes, without identifying you. So if there are pauses, that’s because I am writing. Is that ok with you?
• Remember, you have the right to stop the test at any time, you can ask to pass on any questions and you can ask for breaks if there's anything you need to manage during this time.
• You’re welcome to skip any questions you don’t want to answer or stop the interview at any time for any reason.

Be sure to give the user co-host and/or screensharing permissions and also press record to cloud on zoom.

Question: Any questions before we start?

[Put this info in a chat]

To be transparent and open, you can track this project's work on our github repository here: https://github.com/ossf/wg-securing-software-repos/issues

You can contact us on EMAIL HERE

If you have feedback on me or any comments you'd like to ensure are not seen by myself then you can email my project manager directly EMAIL HERE.

You can check out our privacy policies here: LINK HERE

---

General questions

Text in {these brackets} is guidance for the Superbloom researcher asking questions; they include things to notice, pay attention to, or probe deeper into if mentioned or not mentioned.

Please open up this link: {LINK TO THE PROTOTYPES IN PENPOT]

And share your screen with me.

Question # 1: Imagine you have been told to investigate this paackage to use at work. Can you look and tell me what the 5 most important pieces of information are to you to decide if this package is safe and secure to use for your job? (start with Light UI options)

Notes for question # 1:

Question # 2: Can you describe to me what you understand to review a packages ‘build security’ or ‘software supply chain security’ means?

{Here we are trying to assess that the user tester understands the difference between the packages build security aka what some attestations attest to when they attest the build vs. other types of package security aka source code is not malicious etc. If the user is unsure}

{Can help the user by sending this to read: https://docs.xygeni.io/xygeni-products/build-security/build-security-concepts#software-attestation and “A SLSA provenance is an attestation that a particular build platform produced a set of software artifacts through execution of a build definition:” https://docs.xygeni.io/xygeni-products/build-security/build-security-concepts#provenance}

Notes for question # 2:

Question # 3: Imagine you are very concerned with the packages ‘build security’ and are looking for evidence to support your choice to use this package in your workplace. Where is information that helps you to feel confident that this package was built from what it is supposed to be built from?

Notes for question # 3:

Question # 4: Can you take a closer look at the icons/images in this section {Show an Attestation UI element up close} Can you tell me whether you think the icons, headings and information match what you’d expect to see with each other?

Notes for question # 4:

Question # 5: How useful do you think this information is as it is displayed here?

Notes for question # 5:

Question # 5a: Do you understand what it is communicating?

Notes for question # 5a:

Question # 6: Are there any changes and/or additions you would make to the elements here to help you have more confidence in this information?

Notes for question # 6:

---

Question # 7: (Move onto Medium UI options and then heavy UI options)

Notes for question # 7:

Question # 8 : Take a look at this section. What do you think this is telling you?

{Pay attention to if the user says that this UI tells them ‘There are no vulnerabilities in this package - because this is inaccurate and untrue. 4 known advisories

Note for question # 8:

---

Wrap up

{General question for each persona but doesn't feel right to put in the first general questions section}

Question # 9: Is there anything else you’d like to add regarding what we’ve spoken about?

Note for question # 9:

Thank you so much for your time

Q. Would you like any responses, details or specific words removed from the notes from this conversation? We publish the results openly so please indicate what you’d like removed before the 19th of August 2025.

Q. Reminder of the stipend process - $25 USD via a service called [Tremendous.com](http://Tremendous.com) we need to be able to email this to you. We are also investigating direct links we can give in Signal (or other services) alternatively you can donate this $25 USD to an org or charity of your choosing using Tremendous’s inbuilt service to do so.

[Erioldoesdesign]

Usability/User Test UI research anonymised notes

All answers to each question have been merged under the heading of the question for all participants to keep folks as anonymous as possible.

---

General questions

---

Text in {these brackets} is guidance for the Superbloom researcher asking questions; they include things to notice, pay attention to, or probe deeper into if mentioned or not mentioned.

Please open up this link: {Link to appropriate UI version}

And share your screen with me.

Question # 1: Imagine you have been told to investigate this paackage to use at work. Can you look and tell me what the 5 most important pieces of information are to you to decide if this package is safe and secure to use for your job? (start with Light UI options)

Notes for question # 1:

Taking in information… scrolls once. if there are badges that are well known - mainstream - just being mainstream. Mainstream means if there are too many/a lot of downloads - a lot of people are using it. it goes through a lot of eyes. people are rewviewing it and thought about security - lots of people does not guarentee safety - sponsors might be well known but does not guarentee security.

Published 16 days ago - could be an older package - public for a long time - vulnerabilities can be introduced. Does not ensure safety. Maybe a lot of downloads. So, meaning a lot of people are using it. So, I think it would go through a lot of eyes, and many people are already. Reviewing it might have thought about security.

Should i check dependancies - they might introduce security vulnerabilities to a package.

I'm noticing that there's a lot of information, I don't think this tells me whether it's secure or not.

I don’t know what is dependants. Versions means it has been there a long time.

Tab for code - as a developer i might take a look at the code. I’m interested in seeing the code. If they have expertise in security then they can look and decide how safe it is - best practice.

If i’m thinking abotu using this i think about every information that could be useful.

Package is hosted on github - github runs some security checks on repos periodically - this will be helpful. Go directly in the security info on repo - add directly in registry page - i think that if not cluttered that’d be helpful.

report malware - gives confidence as well. people might use if they find it. i dunno what run kit is.

Contents is a lot of information! also many of the topics might be a lot and need to click/read.

Unsure if they’re helpful.

Features might be helpful but not clearly about security. brief description of features.

Browser support is good to know - browser have security features installed. Check marks on latest is good - compatible with the latest versions. there might be some browsers might flag.

Digitally signed - this means it’s signed by the author. push code, that commit is also digitally signed. when we use this package then the code releated to this package is digitall signed. I’m not sure.I might look it up. Wants to click /more info

SBOM + SLSA 3 - i do not know

i don’t know what source commit is - the initial commit? the file they are using in production.

Public ledger? Blockchains? whenever you add something to a codebase is has your own secure hash codes. This helps. some confidence with just the info - want to explore more and go through the page.

Checksum! oh! i have read. that these can be used if your download is original or not.

The attestation section - more confidence. Sigstore rings a bell. Gives confidence that it has been attested by somebody - i can reach out to somebody in addition to the author

Should say who the publisher to be better sentence

Good focus tab on security. Lock icon is good.

Advisories are useful. Severity is good. Source is good.

“Built by github, signed by npm’ Audit signatures is really good, means that somebody is auditing it periodically. Package is hosted on GitHub.I know that GitHub runs security checks periodically. I think this will be a helpful information.

More info is better (than less). Security section is good. If you see HTTPS, it gives me more confidence.

Report malware. This gives me confidence as well, I think people might use it if they If they find something's not secure.

Many of the topics I need to click on this and read more about it. I don't even know a lot of them, Maybe they are helpful, but. I would just Skip them.

it's very technical. I mean, it's… I think if I know more about, like, how to make it secure, it might be helpful to read through the features. But… I think I'd… do not know enough.

The browsers also have some security features installed, I'm thinking that these are comparable with these browsers latest versions, and if these are compatible. Gives some confidence?

(About axis) I think this one we use for APIs, making queries.

(Looking at light UI)
This is say's digitally signed. I do not know about SBOM or SLSA3. But digitally signed what I understand is it's digitally signed by the author of this package. when we Commit, our the code The Snippet that we are pushing is also digitally signed. So then…What this means is that when we use this package, and the Package would be digitally signed when we use it. So, I'm actually not very sure.

But I'm curious If this is going to help, and I might look it up, what does this mean? Is it going to help, this might be the first time I'm looking at it. I would like to explore this 'digitally sign' more. I would expect some more information.

I don't know what that is Source commit...I don't know what source committed, but I think it might be the initial commit. Build file would be maybe the file they are using in production. I'm not sure. I think I have heard about public ledger. In the context of blockchains, But it might be different. I think… public ledger, it's Whenever you add
Something to a code base. It has your… Uh, it has your own Secure hash codes or something like that. maybe it's related to that. But I think this, this helps (give confidence).

I would like to click on transparency log entry. so let me just Scroll, just to make sure how big this is. It does give me some confidence, even just, u Clicking on it And getting this information (in the rekor log). But I would like to explore more. And go through the page.There's a signature, and there's a public key certificate. There's lots of information here.

The first thing I would probably do is go to the GitHub page and see how often it's been updated. In terms of security, I don't even look at the package page. I probably would go and see who else is using it (maintainers, contributors, users etc).

On this (registry) page, I don't know if I would really look at anything else besides the GitHub (repo link).

Maybe after the GitHub page, the development documentation or any of the other pages to make sure there's no other, red flags. (For example I would look at) the installation or the usage just to see if there's any red flag information.

Okay, and what would you be expecting to see on the GitHub page? Is there anything specific that you would be looking for in that repository? I'm just trying to see who else the contributors are. I'm looking to see how often it's been updated and is there a product structure thought out, in terms of if it's just one script, and if that's the case, I'm not gonna look at that, because it's likely not fleshed out. But it does depend on what the project is for. But I'm looking to see how much do they have documented, how often has it been updated Do they have an up-to-date README

I didn't even realize that (there is info on the registry page) I just look at this in the GitHub repository, but it has the maintainers here (on the registry page), so I guess that could be another indicator. I'm just so used to going to the GitHub page directly to see what's going on with this repo versus (on the registry page) but I'm seeing that they have the maintainers, They have the changelog, they have the homepage. I'm just looking to see if everything there is fleshed out, and all of those links (are legit).

I don't know. personally, this doesn't really mean anything to me, when it says, 'a trusted publisher', I haven't looked into that, I don't know what a 'PyPI Trusted publisher' is. Maybe if I did look into it, I could (feel confident about) signed by PyPI, but what does that mean? I have no idea.

Is it really easy to become a trusted publisher? Is it really difficult? What are the requirements to gain that status? That's something I would like to know more about.

I don't know what these two things are - the SBOM and the SLSA3. But some of the information underneath the build confirmed might be useful. This is stuff that I would typically look at (in the repository)so it's nice that it's front and center, because I'm not gonna lie, sometimes, pages don't have the project (info) so nicely 'there' and I have to look at a random link sometimes. It's nice that it's right there and I don't even have to think about it.

(When looking at different UI's and expanded information)
In terms of the useful information, it's the same. I guess the thing that I would say is why is the drop-down for this, like, over here? (For a UI mock up positioning error) that just kind of goes against my understanding of UI, but maybe that's wrong. Personally, this (medium and expanded UI) is a lot more confusing to me than the other one (more simple). Because when I look at this, what does that mean?

I feel like that one (simple with example links) was just very clear what Each thing meant, whereas this I have to go through and click. It was nice that (information) was right there, all the build was confirmed, here's the link etc. Directly in face - these are things that i’m most likely to care about. First version is best.

Drop down better in column. - I just wanna see project description first. Meta info should be in sidebar.

Integrity checksum - i don’t know what that means - i can go and learn what i need.

Attestation. Some kind of verification. It’s real and valid. I don’t know what it means. No safaety or security. Anyone can put that there. if that was my first time.

(Looking at Light UI for npm versions)
I understand the size and the font and everything. Right now I'm trying to imagine it as official UI, because I can see this is some official UI and a few components are out of place which are added by team.
User tester prefers UI that doesn't push down the existing content too much. The components don’t tell me a whole lot, because I have to click on each one to figure out. Going through each one, SLSA3 is good. And SBOM was called, digital sign, build confirmed I have to look into What they actually mean?
In all 7 of the UI's, I felt The security tab was missing in the navbar. I would really love to see the security tab in the nav bar. Because having a standalone page to expand everything would be very nice.
Considering some of these pages. Especially Axios, is huge (long content) because this documentation is completely written on one page. The entire readme is just published here, same for PyPi. So having the security information in the bottom really reduces their value and visibility. Visibility equals value.
So definitely keeping them in the visible spots. Top also didn't look very good, because the maintainers has sponsor commitments for a lot of these packages. It would take the community time to get used to (new UI), because they have seen these pages just like this for years.
But the sidebar UI looked great. But even with the sidebar, it doesn't reduce its importance I was expecting the location to be lower, above collaborators, below collaborators. Definitely above the buttons (report malware and try on runkit) I feel like the attestation part could be right here just like how GitHub does with its sidebar. These are the super important information that this npm package has chosen. But having a full security page I can dive into as much as possible is very helpful.
Provance info could also come up higher. Just 4 lines. Can match npm.
Clickable sections - don’t need drop downs if the registry page don’t want that ui style.

(Light UI)

Mini version is very good. These gives me the signals quickly + security page for details.

Quickest and fastest to give me what I need to know.

Page 2 - better than the med UI - it’s ok. More details and place to click. This is not from the project it’spypi giving extra details.

Page 3 & 4 - like the first version better. This is not part of the proejct - shouldn’t be under the projects.

Why should i install something with advisories. Not in blue - red/orange

Sidebar - Took time to find info in sidebar - looked because i was trying to look other wise would have missed complete. Good but missable.

Card version - looks fine but not inside of project description. most of the time I come to read the package - could work lower in the page under secyrity info etc. heading.

(Light UI)

Large package - if there’s an issue on this page and i scrolled down and saw it it would be useful.

Kind of useful - i have gone through the project together - title saying security. Nice if there was quickinfo to read up on what each of them mean’t. If I saw this without the context I’m not sure. Can’t see security heading.

Extra info is a lot better. Still needs to be a title. Esepcially for peopel not security concious.

SBOM + SLSA3 - not heard of before.

All links make sense

Where did the attestation go? Also didn’t notice no transparency log. An the security advisories.

(Expandables)

Security isn’tmy first look when installing a package - having at top is disturbing. I like this because it’ssummarised. Personally if it was 1 liner rather than a button - doesn’t take up too much space. Forces you to be security concious.

(Sidebar)

Prefer in the main page than the sidebar. Less insterested in the main page info than the sidebar. Sidebar is pushing down the essential info. But swapping is good.

I don’t want to skip the security information from main section.

(Boxes)

The summary is good - this is ok. Same vibe i get. Where you can see the security from get go. this is fine.

Question # 2: Can you describe to me what you understand to review a packages ‘build security’ or ‘software supply chain security’ means?

{Here we are trying to assess that the user tester understands the difference between the packages build security aka what some attestations attest to when they attest the build vs. other types of package security aka source code is not malicious etc. If the user is unsure}

{Can help the user by sending this to read: https://docs.xygeni.io/xygeni-products/build-security/build-security-concepts#software-attestation and “A SLSA provenance is an attestation that a particular build platform produced a set of software artifacts through execution of a build definition:” https://docs.xygeni.io/xygeni-products/build-security/build-security-concepts#provenance}

Notes for question # 2:

Does build mean, like, what kind of, Libraries are used, something like the Stack used to build it? I'm thinking that build security would be, maybe you chose some programming language. Maybe you chose one over the other. And that, inherently, might mean that you might be carrying all those, uh, things related to that software. To your package.That's what it means to me that. Sharing, this is how we built it, that's what we used. Whatever Software language or library that's gonna come.

Steps that the package is being run through - certain tools and being scanned, going through testing before getting to the end user.

Haven’t thought about the package that i expected it to be. Did the people i expect make this tested this.

We’re not solving the build security really at all here. It’s outside the registry page ability - why should someone care about build security - can the project talk about.

You need to take people through the learning journey - links and news reports - because build security was not taken seriously this is what happens etc.

Consequences need to be seen in real world impact. We never thougfht this is that big - news story.

My opinionis more this package was developed- was any vulnerability introduced through the building process - of the package development any issues.

Question # 3: Imagine you are very concerned with the packages ‘build security’ and are looking for evidence to support your choice to use this package in your workplace. Where is information that helps you to feel confident that this package was built from what it is supposed to be built from?

Notes for question # 3:

Build… Is that the same thing as the code used in production? Or is it something different? Payout? That actually is unclear what… build means

Severity levels are useful to quickly assess

Question # 4: Can you take a closer look at the icons/images in this section {Show an Attestation UI element up close} Can you tell me whether you think the icons, headings and information match what you’d expect to see with each other?

Notes for question # 4:

(Re. Icons) I think, for the most part, everything looks good.

Icons and heading mostly made sense but noted that the build section was most confusing depending on clear labelling of build file/ build logs links - expanded in text in other questions

When you look at the headings of the boxes. And the icons with them, do they make sense? Like, the icons with the heading

at the first glance, I did not pay attention, but now that I look at it the magnifier with the key. It looks really good. I think it showcases that we are trying to have a deeper look. The signature (icon), again, goes well with the attestation. I think package Doesn't make much sense,I was thinking, like, more like code. Like arrow brackets with the slash.

Build icon looks transfer from one to another. Crane.

Maybe bricks are better. Developing and dependancies and build.

Like the full colour icon - not outline icon.

Integrity icon is ok - mag + key.

Everything is fine. For the advisorys tab - what is it about.

(!) icon and the advisors bit odd with the explainor. Triangle shape.

Build icon - crane, perfectly fine with both.

Question # 5: How useful do you think this information is as it is displayed here?

Notes for question # 5:

(About the read me on the npm page) I think this is a lot of information.

The easy thing to show to people right now. Changing teh ecosystems mindset - good start - will read these.

Question #5a: Do you understand what it is communicating?

Notes for question # 5a:

(Re. Checksums)
I would want to know what those (verification and validation) steps are and maybe some visual Of the steps e.g. this went through the first iteration etc. Obviously, not every single detail, but about build security I'm thinking iterative number of steps. I would like to know what those steps are and maybe a quick visual. If we're thinking the same thingI'm thinking of a CI/CD pipeline, and it's first do this, then do this, then do that.

This might just be my lack of knowledge about packages, but I didn't know that, checksum was gonna be there to verify that this is the valid package. I don't even know how to even implement that, or maybe those things are automatically implemented. I don't know anything about that. There's a clue here about something that you need to do, but it's still not clear, laid out 'this is how to'. I'm assuming that when you download it there's a hash associated with it, and I just compare it with it to, see I'm assuming that. some package management tools, have this e.g. if I type pip install. Whatever the name of this package is, That package is checking - does this hash Match the one online? I don't know anything about that.

That's something I would look into more - Let's make sure we're doing checksums, every single time we install, or if we're doing the install of the package in the CI/CD, we need to make sure we do it there, so then we're not compromising our code. I personally would Research more to make this information useful. But I don't think that there needs to necessarily be more here. maybe just links or something like that.

Question # 6: Are there any changes and/or additions you would make to the elements here to help you have more confidence in this information?

Notes for question # 6:

Software supply chain security - i wanna known what those steps are to ascertain that. How to follow those steps. I’m thinking of a CI/CD pipeline. I dunnon how to even implment that checksum. I have an idea on what to do but i would know what to do re. matching hashes.

I would need to investigate to check on and want to understand more. Links etc.

having light UI sidebar version + the security page version for longer info.

---

Question # 7: (Move onto Medium UI options and then heavy UI options)

Notes for question # 7:

(Looking at Medium UI)
Package build, there's a checksum, if I remember correctly, I have read that Checksum can be used to verify if your download is original. so I think this is useful.

Attestation, Trusted identity, yeah, this also gives me more confidence about safety, that this has been attested.

Can you tell me what that means to you, that this has been attested? -> sigstore does ring a bell for me, but I cannot remember what. The release manager, trusted publisher, and a trusted app...it does give me confidence that it has been attested by somebody, meaning, if there is some issue, I can reach out to somebody, in addition to the author. About a problem or an issue, as well as the package creator and maintainers there's also another authority. Somebody else is saying, I've attested to this being all expected and what it should be. That's helpful. In addition to saying, a trusted publisher. I wanted to know which one. So which trusted publisher, the confusion with the sentence structure is, so maybe it needs to say the release file was uploaded By trusted publisher NPM. If it's NPM that's the trusted publisher Or it needs to be via NPM by trusted publisher name.

(Looking at Heavy UI version)

Provenance, I'm not sure what that means.

Built by GitHub and signed by NPM. I think this is more clear, but who is signing it?

So, I think now that there is a section for attestation I'm kind of thinking it's the same thing here, and I know that it's signed by NPM, so I think they're putting their name on it, and that would be kind of similar to attestation.

Checksum, I like that there's a copy. I think that that would be a button. I think this is helpful, I can easily click on it, copy it, check it. Oh, I like the build Icon too (construction crane version), even though I still don't know what that means.

when you said code icon, with the source Do you see, like, it's like a little browser with the little arrows? Oh, yeah, I think this is better than the packaged box (icon).

I see, the audit signatures, That's where I focus, our data's really good, so I think It means to me that maybe somebody's auditing it periodically, That's good to know.

Integrity, then I'll just read through the text here, you can use this command to verify that...so it says you can use this command But I don't know where is the command. Oh. So… that one, the NPM audit signatures, is the command. Okay? I felt somehow, like, these are two separate subsections. (When a section has a hyperlink the user thinks the section is finished after the hyper link)

I like the third one more. (Heavy UI) I think the security really stands out in this, specific tab. And, the advisories are up front. There's a check mark, verified checkmark.
even though there's advisories, they're showing them, and that's a good indicator of trust, is that things are transparent.

(Security tab heavy UI version)

install with caution - good summaries.

more detail so gives something to go to. How does this info get here and get updated by whom? is the package distributor doing this? no i don’t care for this? is it kept up to date? updated by who when

When I go to the package page I just want to see the project description first, and I think that makes. You're putting the information that's all meta info about it, putting that on the sidebar, whereas, this is the actual project that you care about.

Again, these are the things that I'm most likely to care about. I'm a little bit torn between this one (cards UI) and the previous one with the dropdown on the side, because, the things that I'm most likely to click on are the GitHub, and that's all right there. then if I knew what Trusted publisher was and it had some meaning to me.

(Medium UI)

the main difference here is this integrity checksum. I don't know what that means, learn how to check integrity of a package. So, I like that the links included. I can go and learn what does that actually mean?

that's the main difference that I kind of see here I don't know enough about it to know that this is really useful. If I knew what that meant, then I could be glad that they have that. But I don't know.

Does attestation mean anything to you? -> that word? No, not in particular. Off the top of my head, I don't know what that means. I'm assuming it means some verification, 'attesting to something'. This is something real and valid, but in terms of A package, I'm just not aware of it.

Does it give you any sense of, um… safety or security? -> no, again, because, I don't know what that really means. this whole thing doesn't mean anything to me, because I know that, Anyone can just put that info on there (the registry page). Maybe if this was, my first time doing anything with Python, I'd be like, oh, cool, but I've been using Python long enough to - I don't know what that means. I've never seen that before. And for all I know, they just put that in there, You can just put anything that you would want put there, somebody has to review it, but also it's not infallible. So, good to know that people are going through Checking process.

This is, like, a different tab. I think that this is a little more useful for me, because it has this install with caution, Known security advisories, this has it really summarised. I think it goes into a little bit more detail, so it gives me something to actually go to, whereas I felt like the other ones were just, This is verified! (Believe us!)

Again, this would depend how does this information get here? PyPI is not the people distributing them. How is this being updated? Because, if the package distributors are doing this. I don't care for this, because we can just put whatever. even if it's not even malicious, they might not keep up to date with it. at the top, it says it's a distributor-free distributing packages, not a service for auditing them, so I'm torn. I like that (all the info is) all in one place, all the security-related stuff, it's not hindering the other information that I need. But maybe even, At the top of the page if there was a blogs or articles, Written by, published by the maintainers, or published by the package owners that would give you more information as to who wrote this and when it was written, and
When it was updated, just knowing how recent this this information is useful.

If this was real life, I'd have to go through and click and see, what do each of these things mean, and then once I could form my opinion, I'd be okay. I do appreciate that, that it's all in one place.
Signposts where you need to go to do, and what you need to do, as opposed to not.

Software supply chain security So you mean, the steps that it goes through once a build is released?
Build security or software, supply chain security, means the steps that the Build goes through, from my understanding, that would be stuff like is it being run through Certain tools, I'm not a security person, so I don't know what it needs to run through, but is it being run through, Certain tools and being scanned by certain tools, is it being manually tested, or what kind of testing is it going through before it makes it to the end? (The package registry page)

depending on what the package is, some don't need, full-on Penetration testing or anything like that. Maybe some do, If it's something that is really important. if I am understanding what you're describing correctly, some packages it's useful to know this (package) has come from.

I didn't really think about that angle in terms of Is this actually the package you think it is? I haven't thought about that angle. I don't want to say that that was my words.I didn't think about that. I was thinking more Did the people that make this test this? Not so much as when I download this. Is this what I think it is? I've always assume that's the case, if the GitHub page looks okay.

the last one that you sent (heavy UI) To me, because it says security, and I know that's what this page is about, whereas The other ones are kind of just, Floating (UI) on the project description. The security tab/page gives me something that if the director or manager asks if I'm looking for something re. Security, I can point them to this directly and say - hey, you probably shouldn't use this because of this because of severity. I would probably dig a little bit more, but, if this page is saying severity high, that's enough for me to say, we probably shouldn't use it, but If it's saying it's in the green, then I would do maybe a little bit more digging on other sources and just see how other people are using this and whatnot.

(Heavy UI version)
I'll start from the top. The icon has got it was something I clearly saw as something new. So, that worked out well.
I would say this thing probably goes down, because it feels like a disclaimer - having it right on the top doesn't help me much, because it's a false sense of security, or an ill sense of security, as I like to call it, because all security information is just that, advisories. Just like investment advice. You just can't legally give investment advice to anyone without liability.
Having this at the bottom does help quite a bit. It does provide a disclaimer and having it in the bottom might be more helpful.
Starting from security, the top part, I was expecting something like this with the four known advisories, latest, Severity high. like an overview. THis works.
Any package maintainer who comes into here, or any user who comes into here looking for security updates can actually find out very quickly what's going on with this package.
Similarly, to the OpenSSF is a scorecard comes into play quite well, put it right on top so people can easily. I really like that there is a source which you can click to look into it much more.
A lot of people do the phishing situation, where instead of the Axios, they'll just remove, like, one letter or use the Italian S with the small cursive word top, and you could have, like, a phishing situation where you're not seeing the right information for the package. I'm hoping NPM does the validation.
This situation where you have the exact links, you can just click here, and you'll go exactly the right page.
Coming down to provenance, it's really helpful, especially when I didn't know what attestations was. Having a definition for this is really helpful. Even for security advisories.
Whenever I am writing or reading documentation, having these 'learn more' elements means I do not know the full expected URL/destination. What I'll do right now on my keyboard. I press the Ctrl key And then just open another tab, because I really want to read the entire page. This isn’t accessible to visually challenged people looking at the docs.
Learn more also feels like a legal document somewhere or terms and Conditions. Boring stuff. But if this is, like, something really important that you can see. This is very nice. Uh, source, which commit.
Looking at 'build from' this would have the logs, but only GitHub Actions. Maybe GitLab CI/CD or something like that might do differently, but there's a separate link for Build Form, and the logs are in a separate page. Jenkins definitely does that. So I can see the reasoning. This is good. Jenkins has, like, a dashboard page, and then a logs page differently, so it's slash logs and slash full logs.
Those two links go to two different places? I'm not sure but I would expect them to but because they look the same from the shortened version I cannot be sure. So my confusion is why same links.
Checksum is good but having the checksums for humans has never worked out, In my limited time doing documentation and code. Something I really like is having this like a validity checkmark? a genuine sign. Small thing, but it's something that checks themselves for computers, not humans.
We as humans only use checks as to give our insecure selves some security. A package may have got corrupted in the build, And NPM could be the ones actually verifying, This thing is actually verified. That checkmark makes sure our package inside, the one we are hosting right now. Verifies exactly to the one that got built.
The copy (button on checksum) is okay. I'm just double checking with the checksum. As well as having the checksum hash, they're able to be copied or seen. People that know how to check a checksum with the automatic process would be helped by checks and verified, and they know the process and get confidence from seeing checksum verified. The message would not be checks and verify, It would be integrity verified, because that's what checksum asks you to do. So we would probably tell them what 'action' checksum does, and then people can look into what checks and doesn't.
An 'I' symbol here for information (next to checksum) then 'checksum' and the integrity matches.
With the copy button, it's asking the user to do something, inviting the users to, like do the process of checking the checksum. There being the checksum you can copy and run the commands. I'm trying to figure out a process. This is a process that you're describing, there's this one small step that is here (re. Checking the checksum). Would people like to do that step as well? and if you do that step, then it's a complete process, they don't have to do one extra, is what I'm thinking?
Having this (checksum) here, gives that sort of certainty, that checksum is present and matches. I don't need to look into the checksums. That way it gives you that attestation but, checksum is its own thing. Matching is a computer activity that NPM needs to do.
Even with NPM for non-popular packages, according to the NPM team how they'll define a package that is not popular, maybe less than 100,000 downloads. Then they can just put up checks are not verified. So that way, we are giving both types of information to users that the package did come to us, it's stored in the registry, but we have not verified it.
The checksum is available for you to verify. And if you do put it as a recommendation, put it in the back. They probably need to also put the API output. Say someone's trying to fetch this package, this checksum needs to show the API response, so that people can automatically do it and know it's done the right thing. Because very few people will actually hit this copy button. Hopefully, they would want to verify all packages with their package.json. I would love to see that part.
Are people clicking the copy button? or maybe they're getting through scraping, or querying these packages in the package.json and getting that output that way.
Attestation is good."npm confirm that this package was uploaded to NPM Market Actions using the source and build instructions detailed above."is a big claim, I understand it, but It feels like a lot. I would love to see the transparency log of How it (the attestation) figures that out. But it sounds good and I trust npm, and this looks very nice in terms of the UI version.
(Provenance UI from existing npm pages) This is at the bottom of the NPM pages. This actually gives me that same safety/assurance that it was built and signed on GitHub Actions, then published on NCAM.
build file and build from... are they different? like, terms, or could they be the same term? Or is it just better to say build file, you think? Built from this location but if you are giving them the logs, build logs.
Particular build environment. Then doesn't really matter if Jenkins built it, or built Kite, or GitHub GitLab, CI/CD, whoever built it, because the build logs tell me the location of that thing. But that's why they seemed both the same, and having the build file as a direct line. Means I can exactly what the recipe used to build this was. And then the logs for that recipe when it was actually being cooked. Then publishing is also there. So, that gives me the attestation.
The public ledger entry proves that. The source commit shows me that, and the build file also. Then I feel like the attestation part might be a bit duplicate and doesn't provide us a lot more extra. Instead of this being its own UI component.
(Looking at the attestation UI) We can have attestation to be a nice sticker. You don't need that statement, 'NPM confirms that this package… that statement is not particularly useful'. The situation for me would be, once someone is reading these pages again and again it's not as useful, having a sentence versus having a symbol. Have a big check mark green and then a red one, where it's not signed. 'Audited by NPM', or 'not NPM audited'.
The sentence basically says that It was audited using NPM audit, then uploaded to NPM by GitHub Actions, but this (the 'Github Actions statement plus links to the build files, transparency logs etc.) already shows me that. There won't be any other service other than GitHub Actions to be deploying Or uploading the package using the source and the builder interruptions detailed about.
it could be NPM audit tick.
(Looking at integrity in the UI)
This is good. I read the first line, and to do integrity checking is checksums. Didn't we just checksum The integrity, but right below when you say you can use this command (npm audit), we probably will move this up and it will work out just fine. So people will immediately see that command right there. And also move the 'Learn More' link because it looks like a common re. It comes right after referencing the command. I would probably put this as a docs page that gives you all the commands and helpful flags. Because the version number will change for NPM. Wherever it's latest, let's say I don't have this version, 9.5.0. then npm will just not run, because it doesn't know about audit. Npm audit signatures command will only run for NPM CLI version, version 9.5.0, and above.
If this is in the docs then they can give, like, a small warning or an error message, 'you need to install your dependencies, then check for signatures'. That way, we are teaching them through the process, rather than just going to the docs. As a documentarian, it does hurt me that no one reads talks.
(When asked if there's any duplicated info on the Heavy UI page) Duplicated information is never bad in that regard. It just creates is a situation where there might be information overload or possibly some people might think it's poorly maintained due to duplication but generally it's not bad. The duplication might be helpful. As a user who saw this page for the first time, I am more confused, and NPM is, like, very reputable, I would be more confused. By both links point to the same page.
(Looking at Medium UI for PyPI)
Something that I see is the links are very visible with the small open in new tab icon. I would not expect it to be on the top of the page, because the project description would start here.
Verified by PyPy could open up here in the nav/top header bar of the PyPI website.
A lot of people actually work very hard on their readme pages, because they show up like this.
(Similar comments re. Build files, logs etc. the links need to be more clearly defined)
GitHub actions the order you want (re. The build/attestation message)
The link under checksum is good - It will tell me what checksums are and then how to check the integrity of a package. I immediately know this is all about telling me the integrity of the package and I can check that.
I would expect a sign. Attestation for here… Yeah, it looks nice. This thing, uh, this might be looking nicer than that, because.
Attestation messaging is good when there is not the specific provenance info like on the npm pages. The messaging is good as PiPy is taking. Some sort of responsibility for the file, in terms of the trusted identity and the trusted publication. We can shorten these lines into 'By trusted publisher GitHub' take. And then 'pypi Trusted Identity' with the sig-store.
This information, having it on the README may be not the most helpful. Having a security icon here (in the side nav on pypi) it can have all the security validations.
Two things that are missing would be the build file, so that I don't need to look around, who really does the publishing of this package? See if they're signing it correctly and everything. If someone wants to take a look, Because PyPi doesn't have an audit, I think. Just publishes a package. With certificates.
Use the same icons that NPM has, and that way it becomes a more compact section. We can actually show people what attestations really are, and it's all these sites that make up the attestation, which would be very helpful.

(Medium UI version)

I don’t want this above the project description - needs to be below the description. Too much things on the top. AT first I thought there was no description.

(Heavy UI pypi)

https://design.penpot.app/#/view?file-id=3caea71c-ff55-8052-8006-93b07359b5cf&page-id=11fdd9b4-7e92-8040-8006-a1aeaebf9f0a&section=interactions&index=0&share-id=e21d721b-b52f-80fd-8006-a1b49cd395a9

Big security on the top - i don’t know why?

Blue bar - install with caution. I know what pip is. If new usersthey’ll stop

Either skip all things - who cares or they’ll stop and not install.

With security advisories - is it for this version or older versions? Has to say for this current version? older versions tend to have problems. Needs to clarify between older versions and newers versions.

Attestation - this is all sigstore. Reads the text. This is nice. Nice logo. I know what it means and it makes sense. Is there a better way to explain. It’s a lot of text at this moment - expandable - it makes sense - also give the log and everything. If you want to make it clearer - add more links re. learn more - document that explains trusted publishing on pypi. Blog post on trusted publishing for pypi.

Everything below Attestation is good.

Looking at Integrity - we do hash matching - anything with sigsum - we verify with the file .sigsum? there is another way to check. Signed phrase. Sigstore.json files used to verify.

See more details on hashes.

Needs to be a way for user to verify that’s nice.

Manual checksums - I do this - we still verify the opengpg signatures - we make sure they match with those hashes.

Build tor for rpm. Download for linux signature - using both these files to match/check these.

Comands to verify. Tor used to be easier before.

Which file shoudl be first checksum or gpg/pgp - has to be run in the correct order or doesn’t work. Sigstore is way ahead - 1 single comman - the user doesn’t have to remember and run the signing key etc. etc.

Is this the signature or someone elses signature.

Looked at teh hashes on tor/download signature etc. Tor docs verification amazing

Don’t verify the gpg part but the verify the hash part. neovin - install. rustup.rs

Install only packages from the distribution.

Build tools have the way to ping these hashes. Many people don’t know about it.

Hash checking is one command - many people do not know or publish those hashes In the release. Most packages don’t care about security.

Release pages should have that information. Have commanad next to hashes to verify with sigstore.

Manual process

download the source - and the signature (it’s extra open pgp)

First command to fidn the sigsum

I downloaded the file i was expecting - match the hash with the other hash in CLI

run a verify command with the signature file.

Build part is super - which yaml file it is coming from.

What is the guarentee that this is the same file? Should be the exact build file. Instead of head it should be the one used to build. needs to be the exact commit

Axen job output for logs.

Source commit is good

Check sum being higher is good it’s useful. repeated is fine.

Want UI Near top.

New security page - confusing at first but have to help peopel discover - I only go to the read me.

Should i read rest of page? do I know this maintainer - web of trust. Always look at left side column first.

(Medium UI) https://design.penpot.app/#/view?file-id=3caea71c-ff55-8052-8006-93b07359b5cf&page-id=13c39313-5ca6-805a-8006-9df7ede5ebc3&section=interactions&index=0&share-id=e21d721b-b52f-80fd-8006-a1a8f4b42432

Check the dependancies, what tool do I use to build it? if it’s built with something not secure.

Figure out is there any vulnerabilities. Want to go through the source code yourself and look for security vulnerabilities myself

Tools to help for security ratings. Security ratings - Suna i use to check. Link it with github repo and goes through and lets you know - A, B C rating. Lower you go the less confience about security. Generally usedby who owns the package - goes through your source code generally to determin the. Fork the package into own github and then run suna agent.

Build source, Build logs, transparency log - how it’s built? what are the steps? and the vulnerabilities. Are the keys secure and generally understanding if anything was added and anything was removed - what is the final build.

Attestations - an npm sign off. Give you validation. Graviate towards projects more people used - if there are 3/4 you use the most downloads - if bulk ofpeople use it your can trust the society.

Lower levelpackages - not a lot of people nice if there was npm validate the package - we’recool with this - there’s a package that came out today and you put it in and have issues, security issues. rather than going to the source code - ‘we can validate that this is secure from npm side’

Two things - check to be sure that the package itself is not malicious. Package itself is not vulnerable. This is the attestation understanding. The first one is what the attestation is assuring. ‘We beileve this package not malicious.

I value the information - i don’t like where it’s placed - i’d like its own tab.

It’s blocking you.

(Heavy UI)

So much better!

I can still see all the source and abuild. Can I goon and use this package.

Not super likely to click on the security tab. If I’m looking for a package i’ve used before. These are popular package - i just do npm install.

If i’m after a specific package, hashing or cryptothen the security tab is like research and I’m comparing between packages. This is alot more relevant to check it.

On Axios i just install straight up go on the social idea it should work.

Integrity section is nice to check. That’s not something I would do normally, this is interesting - I can test it to be sure. No untowards. Copy button next to the CLI command.

Checksum - i’m not sure - i haven’t worked in security, i don’t know it’s not something I would use. I think it’s that the data is correct.

Used to confirm…it didn’t loose some amount of data.

Default stackoverflow replaced by chat gpt and ask ai for now ‘Hey what is this, what is it for?’

Depends - clicking a link is tricky - not nice to click and you go there isn’t what i’m looking for. ‘Wants a little more info’

Note at bottom should be near the top of the page. Above the audit command.

Question # 8 : Take a look at this section. What do you think this is telling you?

{Pay attention to if the user says that this UI tells them ‘There are no vulnerabilities in this package - because this is inaccurate and untrue. 4 known advisories

Note for question # 8:

(Looking at Heavy UI version)
This is helpful. Tabs focused on security. And I like the lock icon. A lot of feeling of security.
And Security Advisories, I think this is really useful. Right up front, I know there are some advisories. I'm gonna read through this database in npmSEC Database. I really like that they are showing the severity of this, which is high. I think latest would mean when this advisory was noticed for the first time. So that is helpful. The warning icon for non-advisories on that. I like that they've given the source (learn more external link) as well. I would expect this to be a hyperlink.

This part is very important - i need to know what packages this package is using and what the worries are there. How to travel backwards. Reducing the blog reading, looking around for the security info and getting right to the advisories and also the SBOM a little. I don’t have to ask github co-pilot. Also it’s an active community/package. Being aware of active conversations about security directly in npm e.g. from synk or otherswise. Link an issue etc. etc.
Having these two advisories here is great. I can look into them. So getting NPM to build a page for security means everyone will come to NPM for getting their security advisories so they don't need to look into CVEs first for a particular package, or read it from a blog somewhere.
I understand that the package maintainers, and also especially open source companies, building packages have to go through a reputation loss. But the long-term advantages of having documented advisories and seeing what you did (to resolve the security risks) helps. Or even having a summary that you can publish per advisory. Something that the GitHub security section does these days and helps quite a bit, because everyone understands that security flaws will be there. They just want to make sure that their supply chain and their softwares remain protected from the advisory. Everyone wants to know, like, you know, what, where, why.
That's something I recently did for my own talk, since an attack also happened for us at our company and I feel like it's the same risk. That we or NPM needs to de-risk for other people as well.

With security advisories - is it for this version or older versions? Has to say for this current version? older versions tend to have problems. Needs to clarify between older versions and newers versions.

I can see severity here and this information. Immediately, a breakdown and summary of what the security of this package is like.

---

Wrap up

{General question for each persona but doesn't feel right to put in the first general questions section}

Question # 9: Is there anything else you’d like to add regarding what we’ve spoken about?

Note for question # 9:

User also noticed the fonts were different than the regular npm font.

That issues related to security could also be surfaced in the security page + Someone from Redhat is proposing something like a CBOM. This is the cryptographic bill of materials. Considering how advanced security is going to be, and then with quantum-level security also coming, which can crack crypto graphic signatures. So, definitely having that security page early on puts that info in where someone in the security background, someone who is in the assessment, who will do the operations bit will just head to the security page.
Make sure the UI looks like the npm ui, doesn't have to look clickable in a different way but needs to look like it belongs.

Security of packages - what the heck am i installing on my computer. never used to thin that much - only packages from fedora so not worried. The more I used other packages - whos going to look at this - new procedure go through the source code of everything we install/update. Talked to people who break into orgs. Go through the entire OSS source code.

Big companies are nervous about some OSS packages and maintain their own internal verified registries.

Adding the rating/advisories. Severity is really good - as part of the summary.

Thereneeds to be no advisories. Nice to use to compare.