FreeBSD: Large sysctl -a output takes >1h to process

[ltning]

Suggestion: On FreeBSD, rather than 'sysctl -a', use 'sysctl -a -W' to list only writable sysctls. This excludes statistical and other informational sysctls which on a 12.2-system easily adds up to several megabytes (hundred-and-something thousand lines) of, for our intents and purposes, garbage.

Since this won't work for all platforms, I'm not going to propose a patch that simply adds '-W' to the sysctl() call in the code, although this does work for me.

[olevole]

+1 for this improvement. Perhaps it might look like this?

augeasproviders_sysctl/lib/puppet/provider/sysctl/augeas.rb

@@ -78,11 +78,14 @@
     resources ||= []
 
     sep = '='
+    sysctl_all_args = '-a'
+
     if Facter.value(:kernel) == 'FreeBSD'
       sep = ':'
+      sysctl_all_args = '-aW'
     end
 
-    sysctl('-a').each_line do |line|
+    sysctl(sysctl_all_args).each_line do |line|
       line = line.force_encoding("US-ASCII").scrub("")
       value = line.split(sep)
 

Tested with FreeBSD 12.2, FreeBSD 13-ALPHA1

[olevole]

By the way, if we can customize the argument per platform, we can get rid of sep = '=' , thanks to the '-e' argument: https://man.freebsd.org/sysctl/8

-e Separate the name and the value of the variable(s) with `='.
This is useful for producing output which can be fed back to the
sysctl utility. This option is ignored if either -N or -n is
specified, or a variable is being set

E.g:

    sysctl_all_args = '-a'

    if Facter.value(:kernel) == 'FreeBSD'
      sysctl_all_args = '-aeW'
    end

    sysctl(sysctl_all_args).each_line do |line|
...

[raphink]

I'm fine with that. Can you create a PR please?

[olevole]

I'm fine with that. Can you create a PR please?

No problems! maybe there is a better variant of the variable name instead of 'sysctl_all_args' ?

@ltning BTW, >1h this is a rather strange time. 'sysctl -a' works fast on my systems, but your suggestion is reasonable:

# uname -smr
FreeBSD 13.0-ALPHA1 amd64

# sysctl -a | wc -l
   14566

# sysctl -aW | wc -l
    1532

# time sysctl -a 2>&1 | tail -n1

real    0m0.088s
user    0m0.009s
sys     0m0.094s

[ltning]

Thanks @ALL for picking this up.

@ltning BTW, >1h this is a rather strange time. 'sysctl -a' works fast on my systems, but your suggestion is reasonable:

It depends. Also note that this refers to the time spent in this sysctl module, not by sysctl itself.
If there's heavy use of ZFS, there are situations where stats coming out of sysctl -a can be pretty large. And prior to somewhat recent commits in openzfs/freebsd, it could literally take hours for this module to process its output. Even after the upstream fixes (but without the patch discussed here) it regularly takes many minutes on affected systems.

[olevole]

I'm fine with that. Can you create a PR please?

done, tested with: FreeBSD 13.0-ALPHA1 amd64

[trevor-vaughan]

@ltning Would you mind trying #52 to see if it speeds things up even more?