Merge branch 'feature/caching' into develop

Author: blovering

CHANGELOG.md

@@ -4,34 +4,106 @@
 
 ### Features
 
-* Added support for IPv6, can now connect to a splunkd listening on an IPv6 
-  address.
+* Support for IPv6, can now connect to a splunkd listening on an IPv6 address.
+* Improvements to entity state management
+* Improvements to usability of entity collections 
 * Improvements to unit tests
 
 ### Breaking changes
 
-* Renamed the core module from `splunk` to `splunklib`. The Splunk product 
-  already ships with an internal Python module named `splunk` and the name 
-  conflict with the SDK prevented installing the SDK into Splunk Python sandbox
-  for use by Splunk extensions.
+#### Module name
 
-* Update all classes in the core library modules to use new-style classes
-* Rename Job.setpriority to Job.set_priority
-* Rename Job.setttl to Job.set_ttl
+The core module was renamed from `splunk` to `splunklib`. The Splunk product 
+ships with an internal Python module named `splunk` and the name conflict 
+with the SDK prevented installing the SDK into Splunk Python sandbox for use 
+by Splunk extensions. This module name change enables the Python SDK to be 
+installed on the Splunk server.
+
+#### State caching
+
+The client module was modified to enable Entity state caching which required
+changes to the `Entity` interface and changes to the typical usage pattern. 
+  
+Previously, entity state values where retrieved with a call to `Entity.read`
+which would issue a round-trip to the server and return a dictionary of values
+corresponding to the entity `content` field and, in a similar way, a call to
+`Entity.readmeta` would issue in a round-trip and return a dictionary
+contianing entity metadata values. 
+  
+With the change to enable state caching, the entity is instantiated with a
+copy of its entire state record, which can be accessed using a variety of
+properties:
+
+* `Entity.state` returns the entire state record
+* `Entity.content` returns the content field of the state record
+* `Entity.metadata` returns the metadata field of the state record
+
+`Entity.refresh` is a new method that issues a round-trip to the server
+and updates the local, cached state record.
+
+`Entity.read` still exists but has been changed slightly to return the
+entire state record and not just the content field. Note that `read` does
+not update the cached state record. The `read` method is basically a thin
+wrapper over the corresponding HTTP GET that returns a parsed entity state
+record instaed of the raw HTTP response.
+
+The entity _callable_ returns the `content` field as before, but now returns
+the value from the local state cache instead of issuing a round-trip as it
+did before.
+
+It is important to note that refreshing the local state cache is always 
+explicit and always requires a call to `Entity.refresh`. So, for example
+if you call `Entity.update` and then attempt to retrieve local values, you 
+will not see the newly updated values, you will see the previously cached
+values. The interface is designed to give the caller complete control of
+when round-trips are issued and enable multiple updates to be made before
+refreshing the entity.
+  
+The `update` and action methods are all designed to support a _fluent_ style
+of programming, so for example you can write:
+
+    entity.update(attr=value).refresh()
 
-* Naming context - previously the binding context (`binding.Context`) and all
-  tests & samples took a single (optional) `namespace` argument that specified
-  both the app and owner names to use for the binding context. However, the
-  underlying Splunk REST API takes these as separate `app` and `owner` arguments
-  and it turned out to be more convenient to reflect these arguments directly
-  in the SDK, so the binding context (and all samples & test) now take separate
-  (and optional) `app` and `owner` arguments instead of the prior `namespace` 
-  argument.
+And
 
-  You can find a detailed description of Splunk namespaces in the Splunk REST
-  API reference under the section on accessing Splunk resources at:
+    entity.disable().refresh()
+  
+An important benefit and one of the primary motivations for this change is
+that iterating a collection of entities now results in a single round-trip
+to the server, because every entity collection member is initialized with
+the result of the initial GET on the collection resource instead of requiring
+N+1 round-trips (one for each entity + one for the collection), which was the
+case in the previous model. This is a significant improvement for many
+common scenarios.
 
-  * http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTresources
+#### Collections
+
+The `Collection` interface was changed so that `Collection.list` and the 
+corresponding collection callable return a list of member `Entity` objects
+instead of a list of member entity names. This change was a result of user
+feedback indicating that people expected to see eg: `service.apps()` return 
+a list of apps and not a list of app names.
+
+#### Naming context
+
+Previously the binding context (`binding.Context`) and all tests & samples took
+a single (optional) `namespace` argument that specified both the app and owner
+names to use for the binding context. However, the underlying Splunk REST API
+takes these as separate `app` and `owner` arguments and it turned out to be more
+convenient to reflect these arguments directly in the SDK, so the binding 
+context (and all samples & test) now take separate (and optional) `app` and
+`owner` arguments instead of the prior `namespace` argument.
+
+You can find a detailed description of Splunk namespaces in the Splunk REST
+API reference under the section on accessing Splunk resources at:
+
+* http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTresources
+
+#### Misc. API
+
+* Update all classes in the core library modules to use new-style classes
+* Rename Job.setpriority to Job.set_priority
+* Rename Job.setttl to Job.set_ttl
 
 ### Bug fixes
 

README.md

@@ -140,15 +140,13 @@ The second layer is referred to as the _client_ layer and builds on the
 _binding_ layer to provide a friendlier interface to Splunk that abstracts 
 away some of the lower level details of the _binding_ layer.
 
-    from pprint import pprint
-
     import splunk.client as client
 
     # host defaults to localhost and port defaults to 8089
     service = client.connect(username="admin", password="changeme")
 
     for user in service.users:
-        pprint(user())
+        print user.name
 
 ### Unit tests
 

examples/analytics/input.py

@@ -37,17 +37,17 @@ def __init__(self, application_name, splunk_info, index = ANALYTICS_INDEX_NAME):
         self.splunk = client.connect(**splunk_info)
         self.index = index
 
-        if self.index not in self.splunk.indexes.list():
+        if not self.splunk.indexes.contains(self.index):
             self.splunk.indexes.create(self.index)
+        assert(self.splunk.indexes.contains(self.index))
 
-        assert(self.index in self.splunk.indexes.list())
-
-        if ANALYTICS_SOURCETYPE not in self.splunk.confs["props"].list():
+        if not self.splunk.confs['props'].contains(ANALYTICS_SOURCETYPE):
             self.splunk.confs["props"].create(ANALYTICS_SOURCETYPE)
             stanza = self.splunk.confs["props"][ANALYTICS_SOURCETYPE]
             stanza.submit("LINE_BREAKER = (%s)" % EVENT_TERMINATOR)
             stanza.submit("CHARSET = UTF-8")
             stanza.submit("SHOULD_LINEMERGE = false")
+        assert(self.splunk.confs['props'].contains(ANALYTICS_SOURCETYPE))
 
     @staticmethod
     def encode(props):

examples/conf.py

@@ -111,8 +111,7 @@ def list(self, opts):
             for stanza in conf:
                 if (spres and argv[1] == stanza.name) or not spres:
                     print "[%s]" % stanza.name
-                    entity = stanza.read()
-                    for key, value in entity.iteritems():
+                    for key, value in stanza.content.iteritems():
                         if (kpres and argv[2] == key) or not kpres:
                             print "%s = %s" % (key, value)
                 print

examples/dashboard/feed.py

@@ -161,9 +161,6 @@ def iterate(job):
     return (created_job, lambda job: iterate(job))
 
 def main(argv):
-    global urllib2
-    usage = "async.py <sync | async>"
-
     # Parse the command line args.
     opts = parse(argv, {}, ".splunkrc")
 

examples/follow.py

@@ -33,6 +33,7 @@ def follow(job, count, items):
         total = count()
         if total <= offset:
             time.sleep(1) # Wait for something to show up
+            job.refresh()
             continue
         stream = items(offset+1)
         reader = results.ResultsReader(stream)
@@ -63,13 +64,12 @@ def main():
     # Wait for the job to transition out of QUEUED and PARSING so that
     # we can if its a transforming search, or not.
     while True:
-        entity = job.read()
-        state = entity['dispatchState']
-        if state not in ['QUEUED', 'PARSING']:
+        job.refresh()
+        if job['dispatchState'] not in ['QUEUED', 'PARSING']:
             break
         time.sleep(2) # Wait
 
-    if entity['reportSearch'] is not None: # Is it a transforming search?
+    if job['reportSearch'] is not None: # Is it a transforming search?
         count = lambda: int(job['numPreviews'])
         items = lambda _: job.preview()
     else:

examples/handlers/handler_certs.py

@@ -110,5 +110,5 @@ def request(url, message, **kwargs):
 opts = utils.parse(sys.argv[1:], RULES, ".splunkrc")
 ca_file = opts.kwargs['ca_file']
 service = client.connect(handler=handler(ca_file), **opts.kwargs)
-pprint(service.apps.list())
+pprint([app.name for app in service.apps])
 

examples/handlers/handler_debug.py

@@ -38,5 +38,4 @@ def request(url, message, **kwargs):
 
 opts = utils.parse(sys.argv[1:], {}, ".splunkrc")
 service = client.connect(handler=handler(), **opts.kwargs)
-pprint(service.apps.list())
-
+pprint([app.name for app in service.apps])

examples/handlers/handler_proxy.py

@@ -69,7 +69,7 @@ def handler(proxy):
 proxy = opts.kwargs['proxy']
 try:
     service = client.connect(handler=handler(proxy), **opts.kwargs)
-    pprint(service.apps.list())
+    pprint([app.name for app in service.apps])
 except urllib2.URLError as e:
     if e.reason.errno == 1 and sys.version_info < (2, 6, 3):
         # There is a bug in Python < 2.6.3 that does not allow proxies with

examples/handlers/handler_urllib2.py

@@ -43,5 +43,5 @@ def request(url, message, **kwargs):
 
 opts = utils.parse(sys.argv[1:], {}, ".splunkrc")
 service = client.connect(handler=request, **opts.kwargs)
-pprint(service.apps.list())
+pprint([app.name for app in service.apps])