Squid 4.14 and helpers use 100% of CPU

[cezaryjk]

Dear Team,

We used to run, Squid 2.5.4 on Windows Server 2008 R2. It was working like a charm!
Thanks a lot!
Since we have migrated to Win 2019 on Hyper-V node, we had started using newer version of Squid, too.
The configuration was copied from old one, with some minor updates.
We use AD Basic Authentication and LDAP lookup for group membership as well.

It works very good... up to couple hours, and then it starts to use all CPU
Any thoughts?

Best regards
Cezary
PS. I would like to add a screenshot, but... I don't know how :(

[cezaryjk]

[cezaryjk]

Hi again,
And actually, it's working: pages are being served. Maybe a bit slower. And, of course, server becomes not so responsive after "saturation".
Squid service restart (from Squid console: "Stop Squid Service" and "Start...") restores stable conditions for several hours.

Our configuration is a bit complex, but I inherited it from my predecessor. And we have some policies in here as well.
I've been trying to tune it: disabled file cache, increased and changed helpers startup parameter and other. Still no luck...
Now, it looks like that:
VM server:
Hyper-V on Windows Server 2019
Memory: 16GB (dynamic from 8GB to 16GB)
Processor: 4 virtual processors
HDD: Separate for OS and for Squid files (dynamically expanding)

squid.config:
`http_port 3128

dns_nameservers 10.147.x.y 10.147.x.z

#cache_dir aufs /cygdrive/d/squidcachedir/cache 3000 16 256

cache_mem 1 GB
memory_cache_mode always

coredump_dir /var/cache/squid

hierarchy_stoplist cgi-bin ?

acl QUERY urlpath_regex cgi-bin ?
no_cache deny QUERY

auth_param basic program D:/Squid/lib/squid/basic_ldap_auth.exe -b "DC=xxx,DC=yyy,DC=zz" -R -D "CN=ssss,OU=pppp,OU=qqqq,OU=rrrr,DC=uuu,DC=www,DC=xxx" -f (sAMAccountName=%s) -w "password" -h 10.147.d.e -p 389

auth_param basic children 10 startup=5 idle=5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320

acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines

acl INSIDE_IP dst 10.147.x.0/23
acl INSIDE_IP dst 10.147.y.0/23
always_direct allow INSIDE_IP

acl SSL_ports port 443 563 7770 8000

acl Safe_ports port 80 8080 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 563 # snews
acl Safe_ports port 554 # IVMS, RSTP
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

acl blacklist url_regex "D:/Squid/etc/squid/black_new.list"
acl blacklist_proxyuser url_regex "D:/Squid/etc/squid/blackproxy_new.list"
acl whitelist url_regex "D:/Squid/etc/squid/white_new.list"
acl goldlist url_regex "D:/Squid/etc/squid/gold_new.list"
acl windowsupdate dstdomain "D:/Squid/etc/squid/windowsupdate.list"

acl BannedExe url_regex -i .(exe|m3u|mp2|mp3|ra|ram|rm|viv|vivo|vob|vqf|wav|wma|vbs|shs|pif|tar|ace|com)($|?)
acl BannedFiles url_regex -i .(afx|asf|au|avi|divx|exe|m3u|mov|mp2|mp3|mpeg|mpg|qt|ra|ram|rm|viv|vivo|vob|vqf|wav|wma|wmv|vbs|shs|pif|tar|ace|com)($|?)

external_acl_type ldap_group children-startup=10 children-max=15 children-idle=10 %LOGIN D:/Squid/lib/squid/ext_ldap_group_acl.exe -D "CN=ssss,OU=pppp,OU=qqqq,OU=rrrr,DC=uuu,DC=www,DC=xx" -w "password" -b "CN=Users,DC=uuu,DC=www,DC=xx" -f "(&(cn=%g)(member=%u)(objectClass=group))" -F "(sAMAccountName=%s)" -B "DC=uuu,DC=www,DC=xx" -h 10.147.161.209 -p 3268

acl everyone proxy_auth REQUIRED

acl Comp_ProxyUsers external ldap_group SW_Proxy
acl Comp_ProxyUsersYouTube external ldap_group SW_Proxy_YT
acl Comp_ProxyPowerUsers external ldap_group SW_Proxy_PowerUser

http_access allow localhost manager
http_access deny manager

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
acl wuCONNECT dstdomain outlookdiagnostics.azureedge.net

http_access allow CONNECT wuCONNECT localnet

http_access allow windowsupdate localnet

http_access allow goldlist
http_access allow Comp_ProxyPowerUsers !blacklist
http_access allow Comp_ProxyUsersYouTube !blacklist !BannedExe
http_access allow Comp_ProxyUsers !blacklist !BannedFiles !blacklist_proxyuser
http_access allow everyone whitelist
http_access deny blacklist
http_access deny all`

Can you see anything bad in it?
BR
Cezary

[ra-at-diladele-com]

Hello Cezary,

I am very sorry we only package the application into MSI - so can only support issues related to the installation.
I would advise to consult the Squid developers. As a quick test you can deploy our other app (which we do support fully) from https://webproxy.diladele.com/download.html - but it is not squid :(

[cezaryjk]

Hi Rafael,

Sorry to hear that...
All links related to Windows version seems to lead to you...

So, I will try on squid-cache org site.

Maybe somebody had similar issue and will share the solution?
So don't close my case yet, please :)

BR
Cezary

[coffeecodeconverter]

i know this is old, but, one thing it could be seeing as you went from server 2008 to 2019 might be:

A.) windows defender on-access scanner aggressively checking the Squid app files, and logs

B.) 2019 is a more demanding OS, if the underlying hardware spec is the same, or similar, a slower response is to be expected.

C.) as a test, you could install win server 2008 in a VM, run same version of Squid, see if it still hogs the CPU with your config that would be the fairest test and accurately tell you whether the squid version differences really are the only culprit.

D.) the ordering / sequence of your rules might be playing a role, go over with a fine tooth-comb and enure non are repeating rules, as well as seeing if any rules can be combined for effiency, for example "www.microsoft.com" and "microsoft.com" as 2 rules is the same as saying ".microsoft.com" in a single rule (you might find many can be reduced - if unsure, or for a super quick glance without you having to check yourself, throw your config into chatgpt.com or at least give it your list of block and allow rules and ask if they can be optimized)