CSRF and Webhooks

[akieferatos]

I have seen a lot of different discussions regarding this but nothing seems to solve the issue. Recently upgraded to 3.4 and now getting CSRF verification failed when trying to use webhooks. We also had the issue when trying to login, but used debug to determine we needed to update CSRF_TRUSTED_ORIGINS, and now that works.

The original intent was to create a webhook that calls to an ansible server to run a job. However, I am getting a 403 CSRF Verification Failed within netbox itself. I see a tcp connection to our ansible server, but the application never gets to authentication. The issue is, I am not sure which process is throwing the exception in order to debug. Here are my steps:

1. Created test webhook
   Event - Create
   Assigned Models - circuits | circuit
   POST - https://ansible/#/templates/job_template/xxx/launch/ or https://ansible/api/v2/job_templates/xxx/launch/
   Additional Headers - csrftoken:
   Body Template -
   { "extra_vars": {
   "provider": "{{ data['name'] }}"
   }
   }
2. Create new circuit

This is what we see...

Status:failed
Callable:extras.webhooks_worker.process_webhook
Meta:{}
Args:
Kwargs:
webhook: TEST-WEBHOOK
model_name: circuit
event: create
data: {'id': 123, 'url': '/api/circuits/circuits/123/', 'display': 'AK', 'cid': 'AK', 'provider': OrderedDict([('id', 4), ('url', '/api/circuits/providers/4/'), ('display', 'Altice'), ('name', 'Altice'), ('slug', 'altice')]), 'type': OrderedDict([('id', 4), ('url', '/api/circuits/circuit-types/4/'), ('display', 'ALT-10G-FIBER'), ('name', 'ALT-10G-FIBER'), ('slug', 'alt-10g-fiber')]), 'status': {'value': 'active', 'label': 'Active'}, 'tenant': None, 'install_date': None, 'termination_date': None, 'commit_rate': None, 'description': '', 'termination_a': None, 'termination_z': None, 'comments': '', 'tags': [], 'custom_fields': {}, 'created': '2023-02-23T17:09:30.007028Z', 'last_updated': '2023-02-23T17:09:30.007054Z'}
snapshots: {'prechange': None, 'postchange': {'created': '2023-02-23T17:09:30.007Z', 'last_updated': '2023-02-23T17:09:30.007Z', 'description': '', 'comments': '', 'cid': 'AK', 'provider': 4, 'type': 4, 'status': 'active', 'tenant': None, 'install_date': None, 'termination_date': None, 'commit_rate': None, 'termination_a': None, 'termination_z': None, 'custom_fields': {}, 'tags': []}}
timestamp: 2023-02-23 17:09:30.075530+00:00
username: admin
request_id: 2d4f984a-5efe-4031-9e0c-6c7cf3f9d439
Depends On:
Exception:
Traceback (most recent call last):
File "/opt/netbox/venv/lib64/python3.8/site-packages/rq/worker.py", line 1111, in perform_job
rv = job.perform()
File "/opt/netbox/venv/lib64/python3.8/site-packages/rq/job.py", line 923, in perform
self._result = self._execute()
File "/opt/netbox/venv/lib64/python3.8/site-packages/rq/job.py", line 946, in _execute
result = self.func(*self.args, **self.kwargs)
File "/opt/netbox/netbox/extras/webhooks_worker.py", line 100, in process_webhook
raise requests.exceptions.RequestException(
requests.exceptions.RequestException: Status 403 returned with content 'b'\n

Status 403 returned with content
403 Forbidden
CSRF verification failed. Request aborted. You are seeing this message because this HTTPS site requires a Referer header to be sent by your Web browser, but none was sent. This header is required for security reasons, to ensure that your browser is not being hijacked by third parties.If you have configured your browser to disable Referer headers, please re-enable them, at least for this site, or for HTTPS connections, or for same-origin requests.If you are using the <meta name="referrer" content="no-referrer"> tag or including the Referrer-Policy: no-referrer header, please remove them. The CSRF protection requires the Referer header to do strict referer checking. If you're concerned about privacy, use alternatives like noreferrer for links to third-party sites. More information is available with DEBUG=True. webhook FAILED to process.
Result:None

I cannot tell if this is an issue with django, nginx, or something else. We have tried to create exemtions for csrf based on other posts but without success. Do you have any recommendations on where to start to debug this?

[kkthxbye-code]

The exception is telling you that the request the webhook sent returned the error. So the URL host (ansible) either resolves to netbox or another django application, or maybe you have some proxy set that's pointing the wrong place.

Not sure what the purpose of this is either, in any rate it wont solve anything:

Additional Headers - csrftoken:

[akieferatos]

Thank you for responding. :)

Yes, we agree we are seeing the exception after the request is sent. The URL does point and resolve to our ansible host. We do not have any external proxy either. In reading various documentation on django, there seems to be modifications to the default behavior in recent updates, which changes how cross-site requests are governed? If this is a configuration issue, then we are having trouble knowing where to look first. I think we are trying to understand the application flow here so we know how to troubleshoot.

webhook triggered -- > middleware/CSRF check --- > Forwarded request (url)

I know this is not exact, but I suspect the transaction fails in middleware before the transaction ever egresses the server. If that is the case, what are the requirements needed to satisfy CSRF? In previous articles, we had read where a CSRF token is required, and if missing, could result in a 403; hence why we added to 'additional headers' (as a test).

Is there a way to debug this to know exactly where it is failing? Do you have any recommendations on what is needed?

https://docs.netbox.dev/en/stable/configuration/security/#csrf_cookie_name

This is what we followed for our configs regarding CSRF.

[jeremystretch]

CSRF is processed by Django for incoming requests. It has no bearing on or relevance to outbound requests, such as webhooks. The 403 you're seeing is the response sent by the remote server, indicating that the webhook being sent is invalid. It has nothing to do with NetBox's own CSRF mechanism.

My guess is the remote URL you've specified for the webhook is a UI path (rather than an API) and not intended to be accessed remotely.

[akieferatos]

You are exactly right. I actually just figured this out as well. I noticed that the original url was https://hostname/#/path/ which does not work. After switching this and the authorization token, everything worked fine. Thanks for the explanation on CSRF it is very helpful.

[jeremystretch]

Happy to help! It's one of those concepts you rarely encounter until it's a problem.