Version 1.2.0

This version includes a few major changes:
- `file_per_ko` mode, disabled by default, if enabled outputs 1 file per knowledge object instead of including all knowledge objects of a type within 1 file
- `next_scheduled_time` attribute removed from savedsearches (this results in less unnnessary git commits)
- code updated so that newlines are used in the json files, this makes the files stored in git more human readable and easier to see what changed between backups
- support added for http/https based git repositories in addition to ssh-based repo's

If you would like to use `file_per_ko` this will result in a lot more files in the git repository but this will make it easier to see the history of changes in each file

Note that you must set `file_per_ko` to true in both the backup & restore for this to work as expected, also if you change the setting you will need to re-create or wipe the repo as the files are stored differently

Updated all dashboards to include version="1.1" tag as required by new Splunk versions

Updated to Splunk python SDK 1.1.16

This version fixes a bug introduced by 1.1.13, version 1.1.13 was removed from SplunkBase due to an error in the code

Author: gjanders

README.md

@@ -165,6 +165,11 @@ There are also many online resources to help with learning git
 - proxy - optional, if supplied provides a proxy setting to use to access the srcURL (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry "passwordinpasswordsconf". If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128
 - git_name - optional, if provided runs git config user.name to override the name used on this particular repository for git commits
 - git_email - optional, if provided runs git config user.email to override the email used on this particular repository for git commits
+"More settings"
+- git_branch - optional, sets the git branch to use, defaults to master
+- git_proxy - optional, if supplied provides a proxy setting to use to access the git repository (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128
+- file_per_ko - optional, do you want one file per knowledge object? Or a combined file? Defaults to false (i.e. 1 large file for global dashboards in an app). Note that if you change this setting you will need to re-create or wipe the repository as the files are stored differently...Note this setting should match in both backup and restore modular inputs for a particular repo
+
 "More settings"
 - interval - how often the backup should run, if not set the backup will only run on restart of the Splunk instance or when you save this configuration...
 
@@ -182,6 +187,9 @@ There are also many online resources to help with learning git
 - git_command - optional, the location of the git command, this is mainly used on Windows where the git command may not be in the PATH of the user running Splunk
 - ssh_command - optional, the location of the ssh command, this is mainly used on Windows where the git command may not be in the PATH of the user running Splunk 
 - proxy - optional, if supplied provides a proxy setting to use to access the destURL (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry "passwordinpasswordsconf". If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128
+- git_branch - optional, sets the git branch to use, defaults to master
+- git_proxy - optional, if supplied provides a proxy setting to use to access the git repository (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128
+- file_per_ko - optional, do you want one file per knowledge object? Or a combined file? Defaults to false (i.e. 1 large file for global dashboards in an app). Note that if you change this setting you will need to re-create or wipe the repository as the files are stored differently...Note this setting should match in both backup and restore modular inputs for a particular repo
 
 "More settings"
 - interval - how often should the remote server be checked to see if a restore is required. If you are on-prem and using the dynamic restore dashboard you do not need to set an interval, if this is a cloud based system or using the non-dynamic dashboard this is the interval to check the remote server for if a restore needs to be run (i.e. how long it is between a user requesting a restore and this script checking/polling the remote system to run the restoration job)
@@ -266,6 +274,23 @@ To do this you will need to install Version Control For SplunkCloud on your Splu
 [SplunkVersionControlCloud github](https://github.com/gjanders/SplunkVersionControlCloud)
 
 ## Release Notes 
+### 1.2.0
+This version includes a few major changes:
+- `file_per_ko` mode, disabled by default, if enabled outputs 1 file per knowledge object instead of including all knowledge objects of a type within 1 file
+- `next_scheduled_time` attribute removed from savedsearches (this results in less unnnessary git commits)
+- code updated so that newlines are used in the json files, this makes the files stored in git more human readable and easier to see what changed between backups
+- support added for http/https based git repositories in addition to ssh-based repo's
+
+If you would like to use `file_per_ko` this will result in a lot more files in the git repository but this will make it easier to see the history of changes in each file
+
+Note that you must set `file_per_ko` to true in both the backup & restore for this to work as expected, also if you change the setting you will need to re-create or wipe the repo as the files are stored differently
+
+Updated all dashboards to include version="1.1" tag as required by new Splunk versions
+
+Updated to Splunk python SDK 1.1.16
+
+This version fixes a bug introduced by 1.1.13, version 1.1.13 was removed from SplunkBase due to an error in the code
+
 ### 1.1.13
 Updated saved search `Splunk Version Control Audit Query POST` with new regex
 

README/inputs.conf.spec

@@ -8,7 +8,7 @@ srcPassword = <value>
 gitTempDir = <value>
 * location where to store the output of the script on the filesystem (note this directory will be deleted/re-created but the parent dir must exist)
 gitRepoURL = <value>
-* git repository URL to store the objects (SSH URL only)
+* git repository URL to store the objects
 sslVerify = <boolean>
 * Set to 'true' or 'false' to enable/disable SSL verification for REST requests to `srcUrl`. Set to a path to specify a file with valid CA. (https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification)
 noPrivate = <boolean>
@@ -43,6 +43,10 @@ git_email = <value>
 * If set this runs git config user.email '<value>' once the backup git repo is cloned
 git_branch = <value>
 * Sets the git branch to use, defaults to master
+git_proxy  = <value>
+* If supplied provides a proxy setting to use to access the git repository (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128</description>
+file_per_ko = <boolean>
+* Do you want one file per knowledge object? Or a combined file? Defaults to false (i.e. 1 large file for global dashboards in an app). Note that if you change this you will need to re-create or wipe the repository as the files are stored differently...Note this setting should match in both backup and restore modular inputs for a particular repo
 
 [splunkversioncontrol_restore://<name>]
 destURL = <value>
@@ -54,10 +58,9 @@ destPassword = <value>
 gitTempDir = <value>
 * location where to store the output of the script on the filesystem (note this directory will be deleted/re-created but the parent dir must exist)
 gitRepoURL = <value>
-* git repository URL to store the objects (SSH URL only)
+* git repository URL to restore the objects from
 sslVerify = <boolean>
 * Set to 'true' or 'false' to enable/disable SSL verification for REST requests to `srcUrl`. Set to a path to specify a file with valid CA. (https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification)
-
 auditLogsLookupBackTime = <value>
 * This is how far back the audit logs will be checked to ensure that a restore entry is valid, this should be set to your interval time or slightly more, defaults to -1h (use Splunk format)
 debugMode = <boolean>
@@ -76,3 +79,7 @@ proxy = <value>
 * If supplied provides a proxy setting to use to access the destURL (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128
 git_branch = <value>
 * Sets the git branch to use, defaults to master
+git_proxy  = <value>
+* If supplied provides a proxy setting to use to access the git repository (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128</description>
+file_per_ko = <boolean>
+* Do you want one file per knowledge object? Or a combined file? Defaults to false (i.e. 1 large file for global dashboards in an app). Note that if you change this you will need to re-create or wipe the repository as the files are stored differently...Note this setting should match in both backup and restore modular inputs for a particular repo

bin/splunkversioncontrol_backup.py

@@ -6,6 +6,7 @@
 import os
 import sys
 import xml.dom.minidom, xml.sax.saxutils
+import platform
 from splunkversioncontrol_backup_class import SplunkVersionControlBackup
 from splunkversioncontrol_utility import runOSProcess, get_password
 
@@ -45,7 +46,7 @@
             </arg>
             <arg name="gitRepoURL">
                 <title>gitRepoURL</title>
-                <description>git repository URL to store the objects (SSH URL only)</description>
+                <description>git repository URL to store the objects</description>
             </arg>
             <arg name="sslVerify">
                 <title>sslVerify</title>
@@ -136,6 +137,16 @@
                 <description>Sets the git branch to use, defaults to master</description>
                 <required_on_create>false</required_on_create>
             </arg>
+            <arg name="git_proxy">
+                <title>git_proxy</title>
+                <description>If supplied provides a proxy setting to use to access the git repository (https proxy). Use https://user:password:[email protected]:3128 and the application will obtain the password for the entry 'passwordinpasswordsconf'. If password: is not used the password is used as per a normal proxy setting, for example https://user:[email protected]:3128</description>
+                <required_on_create>false</required_on_create>
+            </arg>
+            <arg name="file_per_ko">
+                <title>file_per_ko</title>
+                <description>Do you want one file per knowledge object? Or a combined file? Defaults to false (i.e. 1 large file for global dashboards in an app)</description>
+                <required_on_create>false</required_on_create>
+            </arg>
         </args>
     </endpoint>
 </scheme>
@@ -266,12 +277,36 @@ def validate_arguments():
             sys.exit(5)
 
     gitRepoURL = val_data['gitRepoURL']
-    (stdout, stderr, res) = runOSProcess("%s ls-remote %s" % (git_command, gitRepoURL), logger)
+    proxy_command = ""
+    if gitRepoURL.find("http") == 0:
+        gitRepoHTTP = True
+    else:
+        gitRepoHTTP = False
+
+    git_proxies = {}
+    if 'git_proxy' in val_data:
+        git_proxies["https"] = val_data['git_proxy']
+        if git_proxies['https'].find("password:") != -1:
+            start = git_proxies['https'].find("password:") + 9
+            end = git_proxies['https'].find("@")
+            logger.debug("Attempting to replace git_proxy=%s by subsituting=%s with a password" % (git_proxies['https'], git_proxies['https'][start:end]))
+            temp_password = get_password(git_proxies['https'][start:end], session_key, logger)
+            git_proxies['https'] = git_proxies['https'][0:start-9] + temp_password + git_proxies['https'][end:]
+
+    if gitRepoHTTP and len(git_proxies) > 0:
+        logger.debug("Adding environment variable HTTPS_PROXY before git commands")
+        proxy_command = "HTTPS_PROXY=" + git_proxies["https"]
+        if platform.system() == "Windows":
+            proxy_command = "set " + proxy_command + " & "
+        else:
+            proxy_command = "export " + proxy_command + " ; "
+
+    (stdout, stderr, res) = runOSProcess("%s %s ls-remote %s" % (proxy_command, git_command, gitRepoURL), logger)
     #If we didn't manage to ls-remote perhaps we just need to trust the fingerprint / this is the first run?
-    if res == False:
+    if res == False and not gitRepoHTTP:
         (stdout, stderrout, res) = runOSProcess(ssh_command + " -n -o \"BatchMode yes\" -o StrictHostKeyChecking=no " + gitRepoURL[:gitRepoURL.find(":")], logger)
         (stdout, stderr, res) = runOSProcess("%s ls-remote %s" % (git_command, gitRepoURL), logger)
-    
+
     if res == False:
         print_error("Failed to validate the git repo URL, stdout of '%s', stderr of '%s'" % (stdout, stderr))
         sys.exit(6)