Scorecard showing old results

[sammachin]

node-red-contrib-midi was updated to 1.1.2 on 13/05/2022 with fixes for the license and versions,

However the scorecard ran but still showed the incorrect results

Validation locally with node-red-dev shows that it should have passes for license and versions.

[pbrunier]

I have the same issue. Yesterday I published a new version with correct license and today the scorecard still reflects the old version. https://flows.nodered.org/node/node-red-contrib-dyson-purelink

Running node-red-dev locally shows it is ok:
node-red-contrib-dyson-purelink@0.6.1
✅ Package is MIT licensed

[marcus-j-davies]

Hello,

I apologize if this seems pushy, its not intended to be, only many many Node devs including my myself, have made improvements to our packages, as we get scolded by the scorecard.

Only to find out the scorecard is not updating and our packages still being scored down (aggressively) despite meeting certain standards.

Many many people on the forums also raise this same issue, and I have also added comments, onto various discussions that are on the forums, without much feedback on the issue.

This issue is now more than a year old, and I cant see much in the way of any focus into getting it sorted.

I was looking through the codebase of the flow library, to try and get it fixed myself (as I could imagine your all busy), but seems the codebase is more intricate then I can understand.

The scorecard does, I feel, make quite a difference into the level of quality seen by users.

Once again, sorry if this seems flippant - it really isn't meant to , but maybe an update on the issue will be welcome, given the impact a scorecard can have.

[knolleary]

You are quite right @marcus-j-davies - we need to get to the bottom of this. I'll revisit it this week.

[marcus-j-davies]

Thanks @knolleary,

I'm all for prioritisation, but this one has been going on for sometime, and seeing it crop up more often lately, plus I released a 2nd Node, only to receive 3rd degree burns by the scorecard, and subsequently corrected of course, but not earning the "well done" badge 😅

[knolleary]

I've just pushed some extra debug into production to see if I can make any sense of what is going on.

What is truly confusing is that the way this works is:

1. flow library downloads the tar file of the module, unpacks it, extracts all its information and updates database.
2. it then points the scorecard tool at the same directory that contains the just-downloaded version - but for some reason the scorecard 'sees' a different version.

Hoping the extra debug I've added sheds some light.

[marcus-j-davies]

Hi @knolleary

This is of interest to me...

The validate method in the dev-cli command employees getFromNPM
but it checks to see if the package has already been saved to temp (say in a previous run first)

https://github.com/node-red/node-red-dev-cli/blob/2595fbb2bfc4b96f3de0963ea660831c8f79122f/src/libs/npmget.js#L9

The call to getFromNPM
https://github.com/node-red/node-red-dev-cli/blob/2595fbb2bfc4b96f3de0963ea660831c8f79122f/src/commands/validate.js#L40

EDIT
On second review - I think it just creates the directory

[knolleary]

12 hours later, we've had a number of nodes refresh in the library and none have exhibited the mismatch behaviour... so we continue to wait.

[sammachin]

@knolleary I just published some updates to my package @sammachin/node-red-matter-bridge the first update to 0.0.3 refreshed the scorecard correctly but then a subsequent update to fix the package.json for the node-red version in 0.0.4 and 0.0.5 doesn't seem to have been picked up.

[knolleary]

Thanks @sammachin

From the debug I added last night I can see the following:

Scorecard package.json: @sammachin/node-red-matter-bridge 0.0.5
---Validating Package---
@sammachin/node-red-matter-bridge@0.0.3

The first line is a new line I added - it loads the package.json file on disk and confirms the version number inside it - 0.0.5.
The last line is from the scorecard tool - reporting it has evaluated 0.0.3.

I have logged into the VM hosting the flow library and checked the files on disk - sure enough only 0.0.5 is present. And when I manually run the scorecard with the same args as the flow library used, I get the 0.0.5 scorecard.

I'm going to have to patch the scorecard tool with some extra debug to figure this out.

[sammachin]

thats very odd, if I remember correctly it downloads the package to a tmp folder with a unique name too? (some random chars added to the end)
I wonder if the scorecard tool isn't closing things properly after the first run?

[marcus-j-davies]

Another avenue that maybe of interest.

The checks make a lot of require calls on the package

On a historical project of mine I needed a way to invalidate the nodejs cache via.
delete require.cache[require.resolve(...)]

leaning towards the comment by @sammachin - is the process a singleton?, as surely the cache should be cleared after the process exits?

to a tmp folder with a unique name too?

I don't think it does, but then I'm not that familiar with the codebase
https://github.com/node-red/node-red-dev-cli/blob/2595fbb2bfc4b96f3de0963ea660831c8f79122f/src/libs/npmget.js#L7

[knolleary]

@marcus-j-davies 🎉 that'll explain it.