Trust internal proxy cert for ET Pro rules

[andrejas]

Hi all,

I need to trust an internal certificate used by the proxy server to enable so-rule-update to pull the ET Pro ruleset.

I followed several instructions found online to trust our internal CA, but so far none of them worked. Can anyone share some steps that worked for them on SOII?

Thanks in advance.

[weslambert]

What happens if you run the following?

docker exec so-idstools /bin/bash -c "CURL_CA_BUNDLE=''; cd /opt/so/idstools/etc && idstools-rulecat --force"

[andrejas]

I get the following:

# docker exec so-idstools /bin/bash -c "CURL_CA_BUNDLE=''; cd /opt/so/idstools/etc && idstools-rulecat --force"
2021-12-09 10:15:47,489 - <INFO> - Loading ./rulecat.conf.
2021-12-09 10:15:47,492 - <INFO> - Forcing Suricata version to 6.0.
2021-12-09 10:15:47,493 - <INFO> - Fetching https://rules.emergingthreatspro.com/0000000000000000/suricata-6.0.0/etpro.rules.tar.gz.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 1424, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/idstools-rulecat", line 12, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/idstools/scripts/rulecat.py", line 842, in main
    files = Fetch(args).run()
  File "/usr/local/lib/python3.9/site-packages/idstools/scripts/rulecat.py", line 326, in run
    files.update(self.fetch(url))
  File "/usr/local/lib/python3.9/site-packages/idstools/scripts/rulecat.py", line 314, in fetch
    idstools.net.get(
  File "/usr/local/lib/python3.9/site-packages/idstools/net.py", line 55, in get
    remote = urlopen(url)
  File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)>

In the meantime, on the SO II host, I can now connect to the emergingthreatspro.com server without wget or curl showing any SSL errors. So it looks like the CA is now trusted on the SO host itself, but not yet on the idstools container. What can I do to change this?

[andrejas]

It's working now.

The main difference between earlier attempts and my last attempts is that today I trusted the root CA pem files, while earlier I attempted to work with crt files downloaded using openssl by connecting to the web server I was targeting, via the proxy server.

[Jackson-Pollock]

@andrejas How do you solved this?

[defensivedepth]

@andrejas Can you post a bit more detail of how you solved this?

[andrejas]

@defensivedepth and @Jackson-Pollock I just tried to reproduce the steps I followed, but it turns out that between the time I was working on this and the time I write this, the team responsible for the proxy created a rule that allows requests to the ET Pro server to pass through without being inspected. This means there is no more internal certificate involved, so I have no easy way to verify if what I did actually worked.

So the best pointers I can give you are to:

• Use PEM format certificates provided directly by whoever manages your internal CA.
• Make sure they end up in your idstools container (I believe this is the container doing the rule pulling).

These might be the right steps to do that:

• Get the docker-idstools container ID by running

	docker ps | grep ids
	foo77777ebar   soii2:5000/security-onion-solutions/so-idstools:2.3.52             "./entrypoint.sh"        2 days ago     Up 2 days                                                                                                                        so-idstools

• Copy the certificate(s) into the container.

	docker  cp -v /tmp/cert/$[CERT1].pem foo77777ebar:/etc/ssl/certs
	docker  cp -v /tmp/cert/$[CERT2].pem foo77777ebar:/etc/ssl/certs 

• Download the current ruleset:
  docker exec so-idstools /bin/bash -c "CURL_CA_BUNDLE=''; cd /opt/so/idstools/etc && idstools-rulecat --force"

But again, I have no way to verify if this works.

Hope it helps. Let me know if it did :)

[andrejas]

BTW, if the above doesn't work, the next thing I'd recommend is to follow the typical recommended steps to trust a CA in CentOS, such as covered on https://stackoverflow.com/questions/37043442/how-to-add-certificate-authority-file-in-centos-7

[petiepooo]

Assuming you've added your self-signed CA to the OS's trusted root store, eg via 'sudo dpkg-reconfigure ca-certificates' on Ubuntu, copying the salt/idstools/init.sls from default to local and adding the following patch works for me, and persists between reboots:

--- ./init.sls.orig	2023-02-03 19:48:37.464341622 +0000
+++ ./init.sls	2023-02-03 19:49:27.465582173 +0000
@@ -54,6 +54,7 @@
     - binds:
       - /opt/so/conf/idstools/etc:/opt/so/idstools/etc:ro
       - /opt/so/rules/nids:/opt/so/rules/nids:rw
+      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro
     - watch:
       - file: idstoolsetcsync

(I know this is an older thread, but the marked answer isn't actually an answer, and it didn't seem to be answered anywhere else. Devs, please consider adding the above bind-mount, at least when proxy is selected during setup.)

[TOoSmOotH]

In 2.4 you can specify custom bind mounts for your certificate in the config section. This should allow you to specify the cert file for your OS.

[corycandia]

Can we talk about this more?
"you can specify custom bind mounts for your certificate in the config section"

Where specifically?

I created the local root CA PEM files in /usr/share/pki/ca-trust-source/ and ran sudo update-ca-trust extract

This allowed installation to run, but I am having same issue as OP, with same error.

Hunting through 'Configuration'. not finding anything about container binds yet. Probably missing it.

[corycandia]

This issue seems similar to this post: #7681

Is OPs issue a needed feature request? If a proxy is specified, shouldn't subcomponents also pull in that variable/setting? Also, since custom certificates may be needed for inspection, shouldn't that be a part of the environment variables passed to sub components?

Seems like a bad idea for support and consistency to have customers modifying the state files for subcomponents?

My example is: since this post, the product has already changed enough that a local customization would perhaps have broken after soup.

salt/idstools/init.sls
has already been pulled into two state files, with the bulk of the work being in:
salt/idstools/enabled.sls
(which follows recent trends of breaking actions out from an init.sls in each folder)

[TOoSmOotH]

@corycandia You need to check the "Advanced" slider. Then look under docker -> containers -> so-idstools -> custom_bind_mounts where you can add the appropriate bind mounts. There is no need to modify states in 2.4.

On your comment on the issue, the specific problem is idstools is a container and it has to have its own proxy settings/certs. We pass the proxy host and password into the container so it will try those. That is the extent of the current settings. Anything beyond that requires more work such as adding the custom bind.