Serious Performance Problems with Antimalware Service Executable

[tresf]

Server 2016 (and likely others) ship with an anti-malware executable called MsMpEng.exe. This executable nearly deadlocks (picture) the server when scoop packages are being installed. The anti-malware service greatly lengthens install times and seems to cause file-locking on occasion (picture), potentially breaking the package install state.

To reproduce:

1. Download an install the Server 2016 Core Evaluation ISO from Microsoft
2. Install scoop from front-page install instructions
3. scoop install git

Edited: Whitelisting the ~/scoop directory is a valid workaround.

Set-MpPreference -ExclusionPath "$(Resolve-Path '~/scoop')"

[lukesampson]

The download and extraction path for Scoop install should always be under ~/scoop, assuming you install Scoop recently. Older Scoop installs might be installing under ~/appdata/local/scoop. You can check with scoop which scoop — it should start with ~/scoop if you have started with a recent version.

I wonder if the -ExclusionPath parameter needs to be expanded to the full path to work, e.g. C:\Users\username\scoop instead of just ~/scoop.

[tresf]

it should start with ~/scoop if you have started with a recent version.

This was a fresh install, yes it's ~/scoop. Is there a 100% guarantee that all IO happens in this location? Scoop seems to make use of 7zip for many functions... is there a chance it uses a temporary directory for extraction which is outside of the scoop install directory?

I wonder if the -ExclusionPath parameter needs to be expanded to the full path to work, e.g. C:\Users\username\scoop instead of just ~/scoop.

I thought of this and tested using legacy variable expansion %USERPROFILE%/scoop to no avail. I will try to hard-code the absolute path as well.

[tresf]

I wonder if the -ExclusionPath parameter needs to be expanded to the full path to work, e.g. C:\Users\username\scoop instead of just ~/scoop.

Confirmed, this fixes the problem. Updated:

Set-MpPreference -ExclusionPath "$(Resolve-Path '~/scoop')"

Can we add this to scoop itself? If so, where's a good place to incorporate it?

[lukesampson]

I don't think we can make Scoop automatically update this setting since it requires admin permissions.

We could try to detect this situation and show a warning with instructions about how to resolve the problem.

I've also started a wiki page on Overcoming Problems with Antivirus and Anti-Malware—please feel free to add info there.

[lukesampson]

The best place to check for Windows Defender might be just be before a download.

We could check with

$defender = Get-Command Get-MpPreference -ErrorAction SilentlyContinue
if($defender) {
    warn "It looks like Windows Defender is running etc..."
}

[tresf]

Both the wiki and the warning proposal look like a good start. Feedback:

• In my opinion, the Defender item seems like a good addition to a FAQ/preflight rather than it's own wiki article.
• In regards to the logic you have recommended, I don't think it will work properly because many systems have Defender installed which will throw a false-positive when Get-MpPrefence succeeds. Here's what I would propose instead:

$defender = Get-Service -Name WinDefend -ErrorAction SilentlyContinue
if ($defender -And $defender.Status) {
   # Parse "Stopped" | "Running" ... 
}

I really wish a boolean was available instead of parsing console. I always find parsing console to be risky if the terminal is non-English. (since I'm not sure if these object properties are translated on localized copies of Windows or not -- I assume they are not)

Anyway... here's the expanded Service object for convenience.

Name                : WinDefend
RequiredServices    : {RpcSs}
CanPauseAndContinue : False
CanShutdown         : False
CanStop             : False
DisplayName         : Windows Defender Service
DependentServices   : {}
MachineName         : .
ServiceName         : WinDefend
ServicesDependedOn  : {RpcSs}
ServiceHandle       :
Status              : Stopped
ServiceType         : Win32OwnProcess
StartType           : Manual
Site                :
Container           :

Defender seems to ship with 3 services in total: WinDefend | WdNisSvc | Sense. I am not aware of the impact (or relevance) of the remaining two services. For my 2016 system, WinDefend (aka MsMpEng.exe) was the only problem child.

[lukesampson]

Thanks—that does look like a better way to check for Windows Defender.

I think it would be a good idea to collect some more information about this before showing a warning that might not apply. I have Windows Defender running on Windows 10 and it has never locked up files, at least not noticeably.

Could it be that these realtime scanning locks are specific to Win 2016 Server, or low-spec machines? I was thinking we could keep the Wiki page as a place to collect this information, but I have added it to the FAQ as well.

Also, the status is an enum so it looks like we can check it precisely, e.g.

$defender.status -eq [system.serviceprocess.servicecontrollerstatus]::running

[tresf]

Could it be that these realtime scanning locks are specific to Win 2016 Server, or low-spec machines?

Could be. This VM is a single-core machine, so that would help explain the bottleneck. I'll install a second and fourth processor and benchmark accordingly.

[tresf]

Unit test:

scoop install git

CPU Count	MsMpEng CPU Used	Duration Increase
1	47% - 99%	+30s (+050%) 60s-->90s
2	15% - 48%	+00s (+000%) 90s --> 90s
4	4% - 22%	+45s (+150%) 90s --> 135s

Summary: Additional CPUs mask the problem. VirtualBox intentionally defaults (and often recommends) to a single core, so this problem will continue to happen for hobbyists working from a sandbox, justifying it's position in the FAQ, I think.

I recommend testing on physical hardware and benchmarking as well. Here's what I used for testing:

# With whitelist
Set-MpPreference -ExclusionPath "$(Resolve-Path '~\scoop\')"; scoop uninstall git; date; scoop install git; date;

# Without whitelist
Set-MpPreference -ExclusionPath "C:\Bogus"; scoop uninstall git; date; scoop install git; date;

With this type of VM (VirtualBox) a reboot is necessary between adding/removing CPU cores, so I ran the command once prior to any unit tests as a baseline for performance (accomodate for starting services, powershell startup slowness, et al.)

Whether or not this justifies a warning or not will greatly depend on others' results. I find the above results to be inconclusive.

[lukesampson]

Here's my testing — significantly faster when the directory is excluded!

OS	CPUs	Memory	No Exclusion	Exclusion	Difference
Win 10	2	4096	42.3s	16.6s	-25.7s (2.5x faster)

A couple of handy commands I found...

To remove the exclusion path:

sudo Remove-MpPreference -ExclusionPath (resolve-path ~/scoop)

To time the install:

Measure-Command { scoop install git }

[lukesampson]

I've added a scoop checkup command which helps users diagnose this Windows Defender problem (and eventually others). See 222bac5.

Here's some example output:

$ scoop checkup
Windows Defender may slow down or disrupt installs with realtime scanning.
  Consider running:
    sudo Add-MpPreference -ExclusionPath 'C:\Users\luke\scoop'
  (Requires 'sudo' command. Run 'scoop install sudo' if you don't have it.)
Windows Defender may slow down or disrupt installs with realtime scanning.
  Consider running:
    sudo Add-MpPreference -ExclusionPath 'c:\programdata\scoop'
  (Requires 'sudo' command. Run 'scoop install sudo' if you don't have it.)

Found 2 potential problems.

As you can see, it doesn't change any security settings, it just provides advice about how to do so.

It could be argued that this is less visible than automatically checking during every install. The reason I don't want to check automatically—for all users, all the time—is that I want to be conservative when it comes to advising users to disable realtime Windows Defender scanning, which could potentially reduce the security of their systems.

Putting this in scoop checkup means we can provide a tailored recommendation when people explicitly ask for it.

See also Anti-Virus and Anti-Malware problems on the wiki.

I'm closing this issue now.

[tresf]

The reason I don't want to check automatically—for all users, all the time—is that I want to be conservative when it comes to advising users to disable realtime Windows Defender scanning

Agreed.

Perhaps a compromise is to offer two versions of checkup. One that runs at install and issues a very soft warning. e.g.

Scoop has found 2 potential problems. Run scoop checkup --full for more information. (full being the default option)

And by triggered by first install via something like scoop checkup --simple.

I would also recommend the items like wget alias are bundled into here because they can really confuse someone coming from a POSIX environment. The checkup can really be used as a way to prevent people hitting tripwires.