SO 2.3.40: Proxy coverage

[TheRealPancakes]

I experimented with installing SO 2.3.40 this week with proxy settings enabled. The goal was to see if enabling a proxy would negate the need for a standalone sensor to connect to hosts on the internet directly. My findings indicate that even with a proxy enabled, there is still a significant amount of traffic that bypasses the proxy. Is this intentional / expected?

[TheRealPancakes]

I should mention, the proxy configuration was done via so-setup, not via the instructions at [https://docs.securityonion.net/en/2.3/proxy.html?highlight=proxy].

The manual proxy instructions don't match what the set_proxy() function actually does.

[gebhard73]

The documentation (https://docs.securityonion.net/en/2.3/proxy.html) has been updated yesterday. Perhas this helps, too: #3791

[rwwiv]

What traffic are you expecting to move through the proxy? There is no way to set a global proxy on Linux, so apps/components typically need to be configured separately to route their traffic through a proxy. We try to streamline this process during setup by doing the following:

• Configuring several tools that setup and soup use to run through a proxy. (git, docker-cli, yum/apt)
• Setting environment variables that are used by several common CLI tools that are also used by setup and soup. (curl, wget, others)
• Setting pillar values that are used by salt to configure a couple of components of Security Onion.

If there's traffic you expect should be going through the proxy and isn't currently, let us know and we'll look into it.

[TheRealPancakes]

Well, I was hoping to get as much traffic through the proxy as possible so that the the firewall policy for non-proxied egress traffic from the SO sensor would be fairly small, static, and succinct. I'm trying to strike a balance between an airgap install and unfettered outbound access. I just pulled the IP addresses that the SO 2.3.40 standalone sensor has connected to through the firewall today and converted those to hostnames via zeek dns logs. The results look like :

   1247 grafana.com
    590 mirror.grid.uchicago.edu
    540 mirror.den01.meanservers.net
    517 mirror.rackspace.com
    442 mirror.cogentco.com
    368 mirror.ash.fastserv.com
    348 raw.githubusercontent.com
    328 notary.kolide.co
    306 mirror.phx1.us.spryservers.net
    289 mirror.san.fastserv.com
    273 yum.tamu.edu
    272 spout.ussg.indiana.edu
    258 bay.uchicago.edu
    255 mirror.nodesdirect.com
    246 ftp.osuosl.org
    245 lv-o-mirrors1.its.maine.edu
    244 mirrors.advancedhosters.com
    240 repo.ialab.dsu.edu
    218 mirrors.tummy.com
    217 mirrors.rit.edu
    195 repo1.sea.innoscale.net
    186 mirror.hackingand.coffee
    176 mirrors.maine.edu
    171 mirror.chpc.utah.edu
    165 centos.host-engine.com
    164 mirrors.gigenet.com
    162 centos.mirrors.hoobly.com
    154 centos.den.host-engine.com
    150 ftp.ussg.iu.edu
    149 mirror.ette.biz
    139 mirror.fileplanet.com
    132 repo.miserver.it.umich.edu
    128 centos.mirror.ndchost.com
    124 centos.vwtonline.net
    122 yum.geos.tamu.edu
    107 mirror.pit.teraswitch.com
    101 mirror.mia11.us.leaseweb.net
     99 mirror.phx1.us.securenet.host
     89 mirrors.radwebhosting.com
     66 mirror.cybersecurity.nmt.edu
     64 mirror.mobap.edu
     61 repo1.ash.innoscale.net
     57 repos.forethought.net
     50 mirror.clarkson.edu
     43 linux.mirrors.es.net
     42 atl.mirrors.clouvider.net
     39 mirror.umd.edu
     30 la.mirrors.clouvider.net
     28 anontelemetry.phusionpassenger.com
     25 mirror.nodespace.net
     25 mirror.metrocast.net
     15 whiteoak.umd.edu
     15 mirror.teklinks.com
     13 mirrors.cmich.edu
     12 mirror.us.oneandone.net
     12 mirror.ancl.hawaii.edu
     12 github.com
     11 securitycheck.phusionpassenger.com
     11 ftpmirror.your.org
      9 mirrors.usinternet.com
      9 ftp.usf.edu
      8 mirrors.raystedman.org
      8 mail01.raystedman.org
      7 smoke.rc.rit.edu
      7 mirror.cs.uwp.edu
      5 mirror.sjc02.svwh.net
      5 mirror1.sjc02.svwh.net

I also pulled the access logs from the proxy for the same time period and they look like :

     82 http://mirrorlist.centos.org/?
     26 repo.saltstack.com:443
     26 packages.wazuh.com:443
     26 mirrors.fedoraproject.org:443
     26 download.docker.com:443
     12 http://mirror.fileplanet.com/centos/7.9.2009/os/x86_64/repodata/repomd.xml
      6 http://mirror.sjc02.svwh.net/centos/7.9.2009/updates/x86_64/repodata/repomd.xml
      6 http://mirror.sfo12.us.leaseweb.net/centos/7.9.2009/os/x86_64/repodata/repomd.xml
      6 http://mirror.centos.lax1.serverforge.org/7.9.2009/updates/x86_64/repodata/repomd.xml
      6 http://mirror.centos.lax1.serverforge.org/7.9.2009/extras/x86_64/repodata/repomd.xml
      4 rules.emergingthreatspro.com:443
      4 http://repos-lax.psychz.net/centos/7.9.2009/os/x86_64/repodata/repomd.xml
      4 http://repos.lax.layerhost.com/centos/7.9.2009/updates/x86_64/repodata/repomd.xml
      4 http://mirrors.oit.uci.edu/centos/7.9.2009/updates/x86_64/repodata/repomd.xml
      4 http://mirror.sjc02.svwh.net/centos/7.9.2009/extras/x86_64/repodata/repomd.xml
      4 http://centos.mirror.ndchost.com/7.9.2009/updates/x86_64/repodata/repomd.xml
      4 http://centos.mirror.ndchost.com/7.9.2009/extras/x86_64/repodata/repomd.xml
      4 http://centos-distro.1gservers.com/7.9.2009/extras/x86_64/repodata/repomd.xml
      4 github.com:443
      4 d2lzkl7pfhq30w.cloudfront.net:443
      2 http://sjc.edge.kernel.org/centos/7.9.2009/os/x86_64/repodata/repomd.xml
      2 http://mirrors.xtom.com/centos/7.9.2009/extras/x86_64/repodata/repomd.xml
      2 http://mirrors.tummy.com/mirrors/CentOS/7.9.2009/extras/x86_64/repodata/repomd.xml
      2 http://mirrors.cat.pdx.edu/centos/7.9.2009/updates/x86_64/repodata/repomd.xml
      2 http://mirror.san.fastserv.com/pub/linux/centos/7.9.2009/extras/x86_64/repodata/repomd.xml
      2 http://mirror.hostduplex.com/centos/7.9.2009/os/x86_64/repodata/repomd.xml
      2 http://centosn7.centos.org/centos/7.9.2009/extras/x86_64/repodata/repomd.xml
      2 http://centosn7.centos.org/centos/7.9.2009/extras/x86_64/repodata/7a834a0ed5bb846f96ca406c28148120c86a53f69b8c8ff6abd6f3c12698a18e-primary.sqlite.bz2

So it appears like at least one issue here is that yum isn't consistently obeying the proxy directive in /etc/yum.conf

Also, I can "printenv | grep proxy" and see the correct variables set, run yum update, and observe the traffic going out the firewall instead of via the proxy...

[TheRealPancakes]

Ok, one mystery down, my astute colleague pointed out that fastestmirror is not so spectacular at obeying proxy settings. So, I've set 'enabled=0' in '/etc/yum/pluginconf.d/fastestmirror.conf' and now yum appears to be doing the right thing. I would recommend adding that to set_proxy().

I'll monitor for a few days and see what other quick wins there might be for getting more traffic through the proxy.

[TheRealPancakes]

Ok, here's the new list of things still going through the firewall :

   1294 grafana.com
    364 raw.githubusercontent.com
    345 notary.kolide.co
     30 anontelemetry.phusionpassenger.com
     28 github.com
     12 securitycheck.phusionpassenger.com
      7 stats.grafana.org
      3 usage.influxdata.com
      3 d3nj60yw3f0wsq.cloudfront.net

I'm not sure yet what causes all the connections to grafana.com.

Some entries look to be related to the kolide updater.

One looks to be related to : Passenger anonymous telemetry which does support a proxy but it would be preferable to disable it

One looks to be related to : Grafana telemetry which can be disabled by setting reporting_enabled to false.

The last two look to be related to : InfluxDB telemetry which can be disabled by setting reporting-disabled to true

I know I would appreciate an install or config-time option to disable all known telemetry for installed packages.

[TheRealPancakes]

I'm not 100% certain, but I'm guessing the hits to grafana.com are from having "check_for_updates" enabled.

[TOoSmOotH]

#3838

We are looking into the telemetry and options to disable it.