ENH: http collector: log request info for errors

also log the request's headers and body

Author: Sebastian Wagner

CHANGELOG.md

@@ -28,7 +28,7 @@ CHANGELOG
 - `intelmq.bots.collectors.eset.collector`: Added (PR#1554 by Mikk Margus Möll).
 - `intelmq.bots.collectors.http.collector_http`:
   - Added PGP signature check functionality (PR#1602 by sinus-x).
-  - If status code is not 2xx, the response header and body are logged in debug logging level (#1615).
+  - If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615).
 
 #### Parsers
 - `intelmq.bots.parsers.eset.parser`: Added (PR#1554 by Mikk Margus Möll).

docs/Bots.md

@@ -273,7 +273,7 @@ For extracted files, every extracted file is sent in its own report. Every repor
 
 If the HTTP response' status code is not 2xx, this is treated as error.
 
-In Debug logging level, the reponse's header and body are logged for further inspection.
+In Debug logging level, the request's and response's headers and body are logged for further inspection.
 
 * * *
 

intelmq/bots/collectors/http/collector_http.py

@@ -78,6 +78,8 @@ def process(self):
         resp = self.session.get(url=http_url)
 
         if resp.status_code // 100 != 2:
+            self.logger.debug('Request headers: %r.', resp.request.headers)
+            self.logger.debug('Request body: %r.', resp.request.body)
             self.logger.debug('Response headers: %r.', resp.headers)
             self.logger.debug('Response body: %r.', resp.text)
             raise ValueError('HTTP response status code was %i.' % resp.status_code)

intelmq/tests/bots/collectors/http/test_collector.py

@@ -156,15 +156,17 @@ def test_pgp(self, mocker):
         )
         self.assertMessageEqual(0, OUTPUT[2])
 
-    def test_debug_response_log(self, mocker):
+    def test_debug_request_response_log(self, mocker):
         """
-        Test if the reponse is logged in case of errors as DEBUG
+        Test if the request and response is logged in case of errors as DEBUG
         """
         mocker.get(self.sysconfig['http_url'], status_code=400,
                    headers={'some': 'header'},
                    text='Should be in logs')
         self.run_bot(allowed_error_count=1,
                      parameters={'logging_level': 'DEBUG'})
+        self.assertLogMatches(r"Request headers: .*\.", 'DEBUG')
+        self.assertLogMatches("Request body: None.", 'DEBUG')
         self.assertLogMatches("Response headers: {'some': 'header'}.", 'DEBUG')
         self.assertLogMatches("Response body: 'Should be in logs'.", 'DEBUG')