README.md update

Updated Splunk python SDK to 1.6.20

Author: gjanders

README.md

@@ -49,9 +49,9 @@ The restoration script then validates that the username entered in the lookup fi
 
 If you are using the dynamic version of the restore dashboard (custom command `postversioncontrolrestore`, an alternative report named "Splunk Version Control Audit Query POST" runs to check the audit logs, this report determines if the restoration request was made by the user in question. The report returns 0 or more results and if it returns results for the particular user, the restore proceeds.
 
-Due to the above there is the possiblity that multiple users may trigger a restore while a restore is in progress, a kvstore is used to prevent this from occurring and an additional restore attempt when the restore process is in progress results in an error message to try again.
+Due to the above there is the possibility that multiple users may trigger a restore while a restore is in progress, a kvstore is used to prevent this from occurring and an additional restore attempt when the restore process is in progress results in an error message to try again.
 
-If a user attempts to restore the objects of another user, or attempts to restore the objects as a different user, this is allowed if the user has the admin role (which is determined by the saved search "SplunkVersionControl CheckAdmin").
+If a user attempts to restore the objects of another user, or attempts to restore the objects as a different user, this is allowed if the user has the admin role (which is determined by the saved search "SplunkVersionControl CheckAdmin"). You can change this behaviour if you wish by changing this report...
 
 ## Why use a lookup file and not trigger a remote command execution?
 A custom command named postversioncontrolrestore and the accompanying dashboard `splunkversioncontrolrestore_dynamic` were created for this purpose in version 1.0.7
@@ -177,7 +177,7 @@ There are also many online resources to help with learning git
 
 ### Splunk Version Control Restore
 - destURL - URL of the remote or local Splunk instance that should be queried for restores, this needs to point to the REST port of the instance (port 8089)
-- destUsername - the username to use on the instance to login
+- destUsername - the username to use on the instance to login. Note that the user will run reports from this app and will require access to the `_audit` index along with access to the REST endpoint for checking if users are admins. Finally this is the user used to restore a knowledge object
 - destPassword - the password to use on the instance to login, use `password:<name in passwords.conf>` and the app will attempt to find the password in your passwords.conf file
 - gitTempDir - a directory that the git clone will create, and potentially be deleted. For example /tmp/git_restore or e:\temp\git_restore
 - gitRepoURL - an SSH based git repo URL which will be used to checkout the required tag to restore from 
@@ -232,6 +232,9 @@ Note that you can run this from the command line if the logs are not getting pop
 
 `splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://<your_input_name_goes_here>`
 
+If the issue relates to restoration, ensure that the user configured for the restore section has the required access to run the reports that access the `_audit` index, along with the REST endpoint for users. Finally the user to restore reports must be able to write the knowledge objects.
+For further information also refer to the Security Concerns section of this document. 
+
 Finally the log files are mentioned under the "Where are the logs?" section of this document
 
 ### Problems with the Splunk Version Control Restore or Splunk Version Control Backup modular input
@@ -290,7 +293,7 @@ To do this you will need to install Version Control For SplunkCloud on your Splu
 
 ## Release Notes
 ### 1.2.8
-
+Updated README.md
 
 ### 1.2.7
 Updated Splunk python SDK to 1.6.18

lib/splunklib/__init__.py

@@ -16,5 +16,20 @@
 
 from __future__ import absolute_import
 from splunklib.six.moves import map
-__version_info__ = (1, 6, 18)
+import logging
+
+DEFAULT_LOG_FORMAT = '%(asctime)s, Level=%(levelname)s, Pid=%(process)s, Logger=%(name)s, File=%(filename)s, ' \
+                 'Line=%(lineno)s, %(message)s'
+DEFAULT_DATE_FORMAT = '%Y-%m-%d %H:%M:%S %Z'
+
+
+# To set the logging level of splunklib
+# ex. To enable debug logs, call this method with parameter 'logging.DEBUG'
+# default logging level is set to 'WARNING'
+def setup_logging(level, log_format=DEFAULT_LOG_FORMAT, date_format=DEFAULT_DATE_FORMAT):
+    logging.basicConfig(level=level,
+                        format=log_format,
+                        datefmt=date_format)
+
+__version_info__ = (1, 6, 20)
 __version__ = ".".join(map(str, __version_info__))

lib/splunklib/binding.py

@@ -30,6 +30,8 @@
 import logging
 import socket
 import ssl
+import sys
+import time
 from base64 import b64encode
 from contextlib import contextmanager
 from datetime import datetime
@@ -47,6 +49,7 @@
 except ImportError as e:
     from xml.parsers.expat import ExpatError as ParseError
 
+logger = logging.getLogger(__name__)
 
 __all__ = [
     "AuthenticationError",
@@ -68,7 +71,7 @@ def new_f(*args, **kwargs):
         start_time = datetime.now()
         val = f(*args, **kwargs)
         end_time = datetime.now()
-        logging.debug("Operation took %s", end_time-start_time)
+        logger.debug("Operation took %s", end_time-start_time)
         return val
     return new_f
 
@@ -294,8 +297,7 @@ def wrapper(self, *args, **kwargs):
                 with _handle_auth_error("Autologin failed."):
                     self.login()
                 with _handle_auth_error(
-                        "Autologin succeeded, but there was an auth error on "
-                        "next request. Something is very wrong."):
+                        "Authentication Failed! If session token is used, it seems to have been expired."):
                     return request_fun(self, *args, **kwargs)
             elif he.status == 401 and not self.autologin:
                 raise AuthenticationError(
@@ -452,6 +454,12 @@ class Context(object):
     :type splunkToken: ``string``
     :param headers: List of extra HTTP headers to send (optional).
     :type headers: ``list`` of 2-tuples.
+    :param retires: Number of retries for each HTTP connection (optional, the default is 0).
+                    NOTE THAT THIS MAY INCREASE THE NUMBER OF ROUND TRIP CONNECTIONS TO THE SPLUNK SERVER AND BLOCK THE
+                    CURRENT THREAD WHILE RETRYING.
+    :type retries: ``int``
+    :param retryDelay: How long to wait between connection attempts if `retries` > 0 (optional, defaults to 10s).
+    :type retryDelay: ``int`` (in seconds)
     :param handler: The HTTP request handler (optional).
     :returns: A ``Context`` instance.
 
@@ -469,7 +477,8 @@ class Context(object):
     """
     def __init__(self, handler=None, **kwargs):
         self.http = HttpLib(handler, kwargs.get("verify", False), key_file=kwargs.get("key_file"),
-                            cert_file=kwargs.get("cert_file"), context=kwargs.get("context"))  # Default to False for backward compat
+                            cert_file=kwargs.get("cert_file"),  context=kwargs.get("context"), # Default to False for backward compat
+                            retries=kwargs.get("retries", 0), retryDelay=kwargs.get("retryDelay", 10))
         self.token = kwargs.get("token", _NoAuthenticationToken)
         if self.token is None: # In case someone explicitly passes token=None
             self.token = _NoAuthenticationToken
@@ -498,13 +507,13 @@ def get_cookies(self):
         return self.http._cookies
 
     def has_cookies(self):
-        """Returns true if the ``HttpLib`` member of this instance has at least
-        one cookie stored.
+        """Returns true if the ``HttpLib`` member of this instance has auth token stored.
 
-        :return: ``True`` if there is at least one cookie, else ``False``
+        :return: ``True`` if there is auth token present, else ``False``
         :rtype: ``bool``
         """
-        return len(self.get_cookies()) > 0
+        auth_token_key = "splunkd_"
+        return any(auth_token_key in key for key in self.get_cookies().keys())
 
     # Shared per-context request headers
     @property
@@ -616,7 +625,7 @@ def delete(self, path_segment, owner=None, app=None, sharing=None, **query):
         """
         path = self.authority + self._abspath(path_segment, owner=owner,
                                               app=app, sharing=sharing)
-        logging.debug("DELETE request to %s (body: %s)", path, repr(query))
+        logger.debug("DELETE request to %s (body: %s)", path, repr(query))
         response = self.http.delete(path, self._auth_headers, **query)
         return response
 
@@ -679,7 +688,7 @@ def get(self, path_segment, owner=None, app=None, headers=None, sharing=None, **
 
         path = self.authority + self._abspath(path_segment, owner=owner,
                                               app=app, sharing=sharing)
-        logging.debug("GET request to %s (body: %s)", path, repr(query))
+        logger.debug("GET request to %s (body: %s)", path, repr(query))
         all_headers = headers + self.additional_headers + self._auth_headers
         response = self.http.get(path, all_headers, **query)
         return response
@@ -757,14 +766,20 @@ def post(self, path_segment, owner=None, app=None, sharing=None, headers=None, *
             headers = []
 
         path = self.authority + self._abspath(path_segment, owner=owner, app=app, sharing=sharing)
-        logging.debug("POST request to %s (body: %s)", path, repr(query))
+
+        # To avoid writing sensitive data in debug logs
+        endpoint_having_sensitive_data = ["/storage/passwords"]
+        if any(endpoint in path for endpoint in endpoint_having_sensitive_data):
+            logger.debug("POST request to %s ", path)
+        else:
+            logger.debug("POST request to %s (body: %s)", path, repr(query))
         all_headers = headers + self.additional_headers + self._auth_headers
         response = self.http.post(path, all_headers, **query)
         return response
 
     @_authentication
     @_log_duration
-    def request(self, path_segment, method="GET", headers=None, body="",
+    def request(self, path_segment, method="GET", headers=None, body={},
                 owner=None, app=None, sharing=None):
         """Issues an arbitrary HTTP request to the REST path segment.
 
@@ -793,9 +808,6 @@ def request(self, path_segment, method="GET", headers=None, body="",
         :type app: ``string``
         :param sharing: The sharing mode of the namespace (optional).
         :type sharing: ``string``
-        :param query: All other keyword arguments, which are used as query
-            parameters.
-        :type query: ``string``
         :return: The response from the server.
         :rtype: ``dict`` with keys ``body``, ``headers``, ``reason``,
                 and ``status``
@@ -824,13 +836,28 @@ def request(self, path_segment, method="GET", headers=None, body="",
         path = self.authority \
             + self._abspath(path_segment, owner=owner,
                             app=app, sharing=sharing)
+
         all_headers = headers + self.additional_headers + self._auth_headers
-        logging.debug("%s request to %s (headers: %s, body: %s)",
+        logger.debug("%s request to %s (headers: %s, body: %s)",
                       method, path, str(all_headers), repr(body))
-        response = self.http.request(path,
-                                     {'method': method,
-                                     'headers': all_headers,
-                                     'body': body})
+
+        if body:
+            body = _encode(**body)
+
+            if method == "GET":
+                path = path + UrlEncoded('?' + body, skip_encode=True)
+                message = {'method': method,
+                           'headers': all_headers}
+            else:
+                message = {'method': method,
+                           'headers': all_headers,
+                           'body': body}
+        else:
+            message = {'method': method,
+                       'headers': all_headers}
+
+        response = self.http.request(path, message)
+
         return response
 
     def login(self):
@@ -1135,12 +1162,14 @@ class HttpLib(object):
 
     If using the default handler, SSL verification can be disabled by passing verify=False.
     """
-    def __init__(self, custom_handler=None, verify=False, key_file=None, cert_file=None, context=None):
+    def __init__(self, custom_handler=None, verify=False, key_file=None, cert_file=None, context=None, retries=0, retryDelay=10):
         if custom_handler is None:
             self.handler = handler(verify=verify, key_file=key_file, cert_file=cert_file, context=context)
         else:
             self.handler = custom_handler
         self._cookies = {}
+        self.retries = retries
+        self.retryDelay = retryDelay
 
     def delete(self, url, headers=None, **kwargs):
         """Sends a DELETE request to a URL.
@@ -1254,7 +1283,16 @@ def request(self, url, message, **kwargs):
             its structure).
         :rtype: ``dict``
         """
-        response = self.handler(url, message, **kwargs)
+        while True:
+            try:
+                response = self.handler(url, message, **kwargs)
+                break
+            except Exception:
+                if self.retries <= 0:
+                    raise
+                else:
+                    time.sleep(self.retryDelay)
+                    self.retries -= 1
         response = record(response)
         if 400 <= response.status:
             raise HTTPError(response)
@@ -1290,7 +1328,10 @@ def __init__(self, response, connection=None):
         self._buffer = b''
 
     def __str__(self):
-        return self.read()
+        if six.PY2:
+            return self.read()
+        else:
+            return str(self.read(), 'UTF-8')
 
     @property
     def empty(self):
@@ -1389,7 +1430,7 @@ def request(url, message, **kwargs):
         head = {
             "Content-Length": str(len(body)),
             "Host": host,
-            "User-Agent": "splunk-sdk-python/1.6.18",
+            "User-Agent": "splunk-sdk-python/1.6.20",
             "Accept": "*/*",
             "Connection": "Close",
         } # defaults