[Bug] Test data should not be part of Scoop installation

[xxthunder]

Bug Report

Current Behavior

Initial installation with Scoop installer 'https://get.scoop.sh' does not work anymore behind our company's proxy.

Expected Behavior

Initial installation shall successfully install scoop.

Additional context/output

Scoop worked great and it still works. But on new machines without any existing Scoop installation the installer fails:

PS C:\D\scoop> Remove-Item * -Force -Recurse
PS C:\D\scoop> (New-Object System.Net.WebClient).DownloadFile('https://get.scoop.sh', 'c:\D\scoop\install.ps1')
PS C:\D\scoop> $VerbosePreference = "Continue"
PS C:\D\scoop> .\install.ps1 -ScoopDir 'C:\D\scoop'
VERBOSE: -------- PSBoundParameters --------
VERBOSE: [ScoopDir, C:\D\scoop]
VERBOSE: -------- Environment Variables --------
VERBOSE: $env:USERPROFILE: C:\Users\xxxxxxxx
VERBOSE: $env:ProgramData: C:\ProgramData
VERBOSE: $env:SCOOP:
VERBOSE: $env:SCOOP_CACHE:
VERBOSE: $env:SCOOP_GLOBAL:
VERBOSE: -------- Selected Variables --------
VERBOSE: SCOOP_DIR: C:\D\scoop
VERBOSE: SCOOP_CACHE_DIR: C:\D\scoop\cache
VERBOSE: SCOOP_GLOBAL_DIR: C:\ProgramData\scoop
VERBOSE: SCOOP_CONFIG_HOME: C:\Users\xxxxxxxx\.config
Initializing...
Downloading...
VERBOSE: Downloading https://github.com/ScoopInstaller/Scoop/archive/master.zip to C:\D\scoop\apps\scoop\current\scoop.zip
ParentContainsErrorRecordException: C:\D\scoop\install.ps1:493
Line |
 493 |      $downloader.downloadFile($SCOOP_PACKAGE_REPO, $scoopZipfile)
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (403) Forbidden."

PS C:\D\scoop>

If I try to download that master.zip from GitHub in my browser I get this:

The root cause is that the malware scanner of our company fails to read the file. The scanner does not block the file, the reading of the file in order to check it fails. It used to work some weeks ago.

We could find out the problem: The problem is the file test/fixtures/decompress/TestCases.zip which is content of above master.zip.

The problem was introduced in that file with commit https://github.com/ScoopInstaller/Scoop/tree/b130e606cf983590ca172514d57f7daecb263558 .

The previous commit https://github.com/ScoopInstaller/Scoop/tree/c6fc2de30636ed9d022dd78ae8f8d22a16fa1a78 used to work.

[xxthunder]

I had a look into this TestCases.zip file. I guess our malware checker is extracting it recursively. But this zip contains a lot of archive files of different type but with the same file 'empty' in it. So if they are all extracted in the same directory, the unpacker (in my case 7zip) complains about already existing files. So one might argue that the malware checker is buggy but I guess changing it might take some time (if it is even possible at all). On the other hand for Scoop getting installed no one needs this zip file as it is only test data?

[rashil2000]

I have updated the title and labelled it as 'enhancement'

/cc @niheaven @issaclin32

[issaclin32]

The only "simple" solution I can think of, is to add another release branch, which is identical to master but without the CI and test files.

Let's see if @niheaven has a better solution.

[niheaven]

The build or develop system should be extracted from scoop, and may be added back if the user turn it on. I'm working on it but should wait for a while...

But the problem may introduced by recent added test files since it work before. @xxthunder, Could you help us testing which files are blocked? Check the release of scoop (https://github.com/ScoopInstaller/Scoop/releases) and you could try 0.2.0 and 0.2.1. The empty dummy files are in all test archives and are not just added in recent commits, so maybe some post scripts that remove unused temp files are missing.

[xxthunder]

@niheaven Sorry, just saw your replies. Sure, I will check the different releases this week. Stay tuned ...

[rasa]

If the issue is the .exe files found in the .7z files, can't we just obfuscate them somehow? Perhaps by encrypting the .7z or base64'ing them first?

[xxthunder]

@niheaven I tried to download the zip and tar.gz of each release of scoop with my company notebook:

• 0.2.3: not working
• 0.2.2: not working
• 0.2.1: not working
• 0.2.0: working
• 0.1.0: working

This fits perfectly into my previous analysis that commit b130e60 is the culprit as it introduced this TestCases.zip with a lot of packed empty files.

[niheaven]

Thanks, I'll review b130e60 again.

[xxthunder]

@niheaven You are absolutely right, those empty files were there before and therefore cannot be the root cause:

[xxthunder]

@niheaven But an interesting fact is that my diff tool complains about one of those added 7z files. No idea, why. Latest 7z has no trouble unpacking it.

[xxthunder]

My workaround for this issue: ScoopInstaller/Install#40