Now tested on Windows and Splunk Cloud (note this version of the app is not installed on SplunkCloud, the VersionControl for SplunkCloud is the app to install on the SplunkCloud instance, this variation of the app includes only what is required to remotely backup/restore a SplunkCloud instance

This app is still used for SplunkCloud instances, but this app is installed on-prem

Updates include:
- Updated python SDK to 1.6.13
- New options in both backup & restore so that you can specify the location of the git / SSH command
- The ability to only backup particular apps by default rather than to backup all and rely on an exclusion list (appsList)
- Support for passwords.conf instead of plain text passwords
- Proxy support
- Re-wrote the runOSProcess function so that it works on Windows as expected

The README.md has had various updates including more details around setup and how this was tested on Windows

Author: gjanders

README.md

@@ -5,17 +5,17 @@ python.version = python3
 srcURL = <value>
 * This the URL to be used for the REST API access of the Splunk instance, https://localhost:8089/ for example (does not have to be localhost)
 srcUsername = <value>
-* username to use for REST API of srcURL argument (only required if not using useLocalAuth)
+* username to use for REST API of srcURL argument (required if not using useLocalAuth)
 srcPassword = <value>
-* password to use for REST API of srcURL argument (only required if not using useLocalAuth)
+* password to use for REST API of srcURL argument (required if not using useLocalAuth), use 'password:<name in passwords.conf>' and the app will attempt to find the password in your passwords.conf file
 gitTempDir = <value>
-* location where to store the output of the script on the filesystem
+* location where to store the output of the script on the filesystem (note this directory will be deleted/re-created but the parent dir must exist)
 gitRepoURL = <value>
 * git repository URL to store the objects (SSH URL only)
 noPrivate = <boolean>
-* disable the backup of user level / private objects (true/false)
+* disable the backup of user level / private objects (true/false), default false
 noDisabled = <boolean>
-* disable the backup of objects with a disabled status in Splunk (true/false)
+* disable the backup of objects with a disabled status in Splunk (true/false), default false
 includeEntities = <value>
 * comma separated list of object values to include
 excludeEntities = <value>
@@ -25,32 +25,44 @@ includeOwner = <value>
 excludeOwner = <value>
 * comma separated list of owners objects that should be transferred
 debugMode = <boolean>
-* turn on DEBUG level logging (defaults to INFO) (true/false)
+* turn on DEBUG level logging (defaults to INFO) (true/false), default false
 useLocalAuth = <boolean>
-* do not use the srcUsername/srcPassword, use the session_key of the user running the modular input instead (works on localhost only) (true/false)
+* do not use the srcUsername/srcPassword, use the session_key of the user running the modular input instead (works on localhost only) (true/false), default false
 remoteAppName = <value>
-* defaults to SplunkVersionControl, this app needs to contain the savedsearches and potentially the splunkversioncontrol_globalexclusionlist
+* defaults to SplunkVersionControl, this app needs to contain the savedsearches and potentially the splunkversioncontrol_globalexclusionlist, use SplunkVersionControlCloud on a cloud-based instance
 appsList = <value>
-* Comma separated list of apps, this changes Splunk Version Control to not list all applications and instead only runs a backup on the specified apps. Useful for Splunk Cloud where you cannot access the apps REST endpoint
+* Comma separated list of apps, this changes Splunk Version Control to not list all applications and instead only runs a backup on the specified apps
+git_command = <value>
+* defaults to 'git', can be overriden (for example on a Windows server) to use a full path to the git command
+ssh_command = <value> 
+* defaults to 'ssh', can be overriden (for example on a Windows server) to use a full path to the ssh command
+proxy = <value>
+* If supplied provides a proxy setting to use to access the srcURL (https proxy). Use https://user:password:passwordinpasswordsconf@10.10.1.0:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:password@10.10.1.0:3128
 
 [splunkversioncontrol_restore://<name>]
 destURL = <value>
 * This the URL to be used for the REST API access of the Splunk instance, https://localhost:8089/ for example (does not have to be localhost)
 destUsername = <value>
 * username to use for REST API of srcURL argument (only required if not using useLocalAuth)
 destPassword = <value>
-* password to use for REST API of srcURL argument (only required if not using useLocalAuth)
+* password to use for REST API of srcURL argument (only required if not using useLocalAuth), use 'password:<name in passwords.conf>' and the app will attempt to find the password in your passwords.conf file
 gitTempDir = <value>
-* location where to store the output of the script on the filesystem
+* location where to store the output of the script on the filesystem (note this directory will be deleted/re-created but the parent dir must exist)
 gitRepoURL = <value>
 * git repository URL to store the objects (SSH URL only)
 auditLogsLookupBackTime = <value>
 * This is how far back the audit logs will be checked to ensure that a restore entry is valid, this should be set to your interval time or slightly more, defaults to -1h (use Splunk format)
 debugMode = <boolean>
-* turn on DEBUG level logging (defaults to INFO) (true/false)
+* turn on DEBUG level logging (defaults to INFO) (true/false), default false
 useLocalAuth = <boolean>
-* do not use the srcUsername/srcPassword, use the session_key of the user running the modular input instead (works on localhost only) (true/false)
+* do not use the srcUsername/srcPassword, use the session_key of the user running the modular input instead (works on localhost only) (true/false), default false
 remoteAppName = <value>
-* defaults to SplunkVersionControl, this app needs to contain the savedsearches and potentially the splunkversioncontrol_globalexclusionlist
+* defaults to SplunkVersionControl, this app needs to contain the savedsearches and potentially the splunkversioncontrol_globalexclusionlist, use SplunkVersionControlCloud on a cloud-based instance
 timewait = <value>
 * defaults to 600, if the kvstore contains an entry advising there is a restore running, how many seconds should pass before the entry is deleted and the restore happens anyway?
+git_command = <value>
+* defaults to 'git', can be overriden (for example on a Windows server) to use a full path to the git command
+ssh_command = <value>
+* defaults to 'ssh', can be overriden (for example on a Windows server) to use a full path to the ssh command
+proxy = <value>
+* If supplied provides a proxy setting to use to access the destURL (https proxy). Use https://user:password:passwordinpasswordsconf@10.10.1.0:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:password@10.10.1.0:3128

README/inputs.conf.spec

@@ -12,7 +12,7 @@ label = SplunkVersionControl
 [launcher]
 author = Gareth Anderson
 description = Version Control software for Splunk instances (backup/restore from git)
-version = 1.0.12
+version = 1.1.0
 
 [package]
 id = SplunkVersionControl

default/app.conf

@@ -1,13 +1,13 @@
-[source::.../splunkversioncontrol_backup.log]
+[source::...(/|\\)var(/|\\)log(/|\\)splunk(/|\\)splunkversioncontrol_backup.log]
 sourcetype = splunkversioncontrol
 
-[source::.../splunkversioncontrol_restore.log]
+[source::...(/|\\)var(/|\\)log(/|\\)splunk(/|\\)splunkversioncontrol_restore.log]
 sourcetype = splunkversioncontrol
 
-[source::.../splunkversioncontrol_rest_restore.log]
+[source::...(/|\\)var(/|\\)log(/|\\)splunk(/|\\)splunkversioncontrol_rest_restore.log]
 sourcetype = splunkversioncontrol
 
-[source::.../splunkversioncontrol_postversioncontrolrestore.log]
+[source::...(/|\\)var(/|\\)log(/|\\)splunk(/|\\)splunkversioncontrol_postversioncontrolrestore.log]
 sourcetype = splunkversioncontrol
 
 [splunkversioncontrol]

default/props.conf

@@ -16,5 +16,5 @@
 
 from __future__ import absolute_import
 from splunklib.six.moves import map
-__version_info__ = (1, 6, 12)
+__version_info__ = (1, 6, 13)
 __version__ = ".".join(map(str, __version_info__))

lib/splunklib/__init__.py

@@ -1378,7 +1378,7 @@ def request(url, message, **kwargs):
         head = {
             "Content-Length": str(len(body)),
             "Host": host,
-            "User-Agent": "splunk-sdk-python/1.6.12",
+            "User-Agent": "splunk-sdk-python/1.6.13",
             "Accept": "*/*",
             "Connection": "Close",
         } # defaults

lib/splunklib/binding.py

@@ -16,7 +16,7 @@
 import sys
 
 from io import TextIOWrapper, TextIOBase
-from splunklib.six import ensure_text
+from splunklib.six import ensure_str
 from .event import ET
 
 try:
@@ -43,15 +43,8 @@ def __init__(self, output = sys.stdout, error = sys.stderr):
         :param output: Where to write the output; defaults to sys.stdout.
         :param error: Where to write any errors; defaults to sys.stderr.
         """
-        if isinstance(output, TextIOBase):
-            self._out = output
-        else:
-            self._out = TextIOWrapper(output)
-
-        if isinstance(error, TextIOBase):
-            self._err = error
-        else:
-            self._err = TextIOWrapper(error)
+        self._out = output
+        self._err = error
 
         # has the opening <stream> tag been written yet?
         self.header_written = False
@@ -63,18 +56,20 @@ def write_event(self, event):
         """
 
         if not self.header_written:
-            self._out.write(ensure_text("<stream>"))
+            self._out.write("<stream>")
             self.header_written = True
 
         event.write_to(self._out)
 
     def log(self, severity, message):
         """Logs messages about the state of this modular input to Splunk.
         These messages will show up in Splunk's internal logs.
+
         :param severity: ``string``, severity of message, see severities defined as class constants.
         :param message: ``string``, message to log.
         """
-        self._err.write(ensure_text("%s %s\n" % (severity, message)))
+
+        self._err.write("%s %s\n" % (severity, message))
         self._err.flush()
 
     def write_xml_document(self, document):
@@ -83,11 +78,10 @@ def write_xml_document(self, document):
 
         :param document: An ``ElementTree`` object.
         """
-        data = ET.tostring(document)
-        self._out.write(ensure_text(data))
+        self._out.write(ensure_str(ET.tostring(document)))
         self._out.flush()
 
     def close(self):
         """Write the closing </stream> tag to make this XML well formed."""
-        self._out.write(ensure_text("</stream>"))
+        self._out.write("</stream>")
         self._out.flush()

lib/splunklib/modularinput/event_writer.py

@@ -105,8 +105,7 @@ def run_script(self, args, event_writer, input_stream):
                 return 1
 
         except Exception as e:
-            err_string = EventWriter.ERROR + str(e)
-            event_writer._err.write(err_string)
+            event_writer.log(EventWriter.ERROR, str(e))
             return 1
 
     @property

lib/splunklib/modularinput/script.py

@@ -16,6 +16,7 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+from splunklib import six
 from splunklib.six.moves import map as imap
 
 from .decorators import ConfigurationSetting
@@ -135,8 +136,14 @@ def fix_up(cls, command):
                 raise AttributeError('No EventingCommand.transform override')
             SearchCommand.ConfigurationSettings.fix_up(command)
 
+        # TODO: Stop looking like a dictionary because we don't obey the semantics
+        # N.B.: Does not use Python 2 dict copy semantics
         def iteritems(self):
             iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
             return imap(lambda name_value: (name_value[0], 'events' if name_value[0] == 'type' else name_value[1]), iteritems)
 
+        # N.B.: Does not use Python 3 dict view semantics
+        if not six.PY2:
+            items = iteritems
+
         # endregion

lib/splunklib/searchcommands/eventing_command.py

@@ -19,6 +19,7 @@
 from .decorators import ConfigurationSetting
 from .search_command import SearchCommand
 
+from splunklib import six
 from splunklib.six.moves import map as imap, filter as ifilter
 
 # P1 [O] TODO: Discuss generates_timeorder in the class-level documentation for GeneratingCommand
@@ -324,6 +325,8 @@ def fix_up(cls, command):
             if command.generate == GeneratingCommand.generate:
                 raise AttributeError('No GeneratingCommand.generate override')
 
+        # TODO: Stop looking like a dictionary because we don't obey the semantics
+        # N.B.: Does not use Python 2 dict copy semantics
         def iteritems(self):
             iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
             version = self.command.protocol_version
@@ -334,6 +337,10 @@ def iteritems(self):
                         lambda name_value: (name_value[0], 'stateful') if name_value[0] == 'type' else (name_value[0], name_value[1]), iteritems)
             return iteritems
 
+        # N.B.: Does not use Python 3 dict view semantics
+        if not six.PY2:
+            items = iteritems
+
         pass
         # endregion