Merge pull request #31 from splunk/fross-feature/naming

Lots of new functionality

Author: Fred Ross

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Splunk Python SDK Changelog
 
+## 0.8.5
+
+### Features
+
+* Expanded endpoint coverage. Now at parity with the Java SDK.
+* Replaced ResultsReader with something shorter. Iteration now
+  results either Message objects or dicts, and moved preview from
+  iteration to a field.
+* Entities can be fetched from collections by name plus namespace
+  combinations (which are unique, unlike names alone). Fetching
+  entries by name alone properly throws errors on name conflicts.
+* Added a distinct AuthenticationError and optional autologin/autorelogin.
+* Reduced roundtrips and listings with specific lookups in __getitem__ 
+  and similar methods.
+* Put in types and operations to make URL encoding of strings consistent.
+* Pagination is implemented to stream search results a hunk at a time.
+* Lots of docstrings expanded.
+* Lots of small bugs fixed.
+
 ## 0.8.0 (beta)
 
 ### Features

docs/tutorial.rst

@@ -0,0 +1,113 @@
+Introduction
+------------
+
+Here's a simple program using the Python SDK. Obviously you'll have to
+change the host, username, password, and any other data that you may
+have customized. And don't experiment on your production Splunk
+server! Install the free version of Splunk on your own machine to
+experiment.::
+
+    import splunklib.client as client
+    c = client.connect(host="localhost",
+                        port=8089,
+                        scheme="https",
+                        username="admin",
+                        password="changeme")
+    saved_searches = c.saved_searches
+    mss = saved_searches.create("my_saved_search", "search * | head 10")
+    assert "my_saved_search" in saved_searches
+    saved_searches.delete("my_saved_search")
+
+It's worth spending a few minute in ``ipython`` examining the objects
+produced in this example. ``c`` is a ``Service``[[TODO: link to
+reference docs]], which has [fields](link to list of fields in
+Service's docs) that provide access to most of Splunk's contents.
+``saved_searches`` is a ``Collection``, and each entity in it is
+identified by a unique name (``"my_saved_search"`` in the example).
+All the names should be alphanumeric plus ``_`` and ``-``; no spaces
+are allowed[#]_.
+
+.. [#] Splunk has two names for each entity: the pretty one meant to
+be displayed to users in the web browser, and the alphanumeric one
+that shows up in the URL of the REST call. It is the latter that is
+used in the SDK. Thus the Search app in Splunk is called "Search"
+in the web interface, but to fetch it via the SDK, you would write
+``c.apps['search']``, not ``c.apps['Search']``. The "Getting
+Started" app is ``c.apps['gettingstarted']``, not ``c.apps['Getting Started']``.
+
+A ``Collection`` acts like a dictionary. You can call ``keys``,
+``iteritems``, and ``itervalues`` just like on a dictionary. However,
+you cannot assign to keys. ``saved_searches['some_name'] = ...`` is
+nonsense. Use the ``create`` method instead. Also, 
+``del saved_searches['some_name']`` does not currently work. Use the
+``delete`` method instead.
+
+Note that in the example code we did not assert::
+
+    mss == saved_searches["my_saved_search"]
+
+The Python objects you are manipulating represent snapshots of the
+server's state at some point in the past. There is no good way of
+defining equality on these that isn't misleading in many cases, so we
+have made ``==`` and ``!=`` raise exceptions for entities.
+
+Another side effect of using snapshots: after we delete the saved
+search in the example, ``mss`` is still bound to the same local object
+representing that search, even though it no longer exists on the
+server. If you need to update your snapshot, call the ``refresh``
+method[#]_. For more on caching and snapshots, see [[TODO: link to
+section on roundtrips and caching]]
+
+.. [#] Calling ``refresh`` on an entity that has already been deleted
+raises an ``HTTPError``.
+
+You can access the fields of an entity either as if they were keys in
+a dictionary, or fields of an object::
+
+    mss['search'] == "search * | head 10"
+    mss.search    == "search * | head 10"
+
+    mss['action.email'] == '0'
+    mss.action.email    == '0'
+
+A ``.`` isn't a valid character in identifiers in Python. The second
+form is actually a series of field lookups. As as side effect, you can
+get groups of fields that share prefixes.::
+
+    mss['action'] == {'email': '0',
+                      'populate_lookup': '0',
+                      'rss': '0',
+                      'script': '0',
+                      'summary_index': '0'}
+    mss.action == {'email': '0',
+                   'populate_lookup': '0',
+                   'rss': '0',
+                   'script': '0',
+                   'summary_index': '0'}
+
+Those look like dictionaries, but they're actually a subclass called
+``Record`` [[TODO: link to reference documentation]] that allows keys
+to be looked up as fields. [[TODO: Implement keys() on entities, and
+document it here]] In addition to fields, each kind of entity has a
+range of methods.::
+
+    mss.dispatch()    # Runs the saved search.
+    mss.suppress(30)  # Suppress all alerts from this saved search for 30 seconds
+
+This should be enough information to understand the reference
+documentation and start using the SDK productively.
+
+Roundtrips and caching
+----------------------
+
+The rate limiting step in most programs that call REST APIs is calls
+to the server. The SDK is designed to minimize and postpone these as
+much as possible. When you fetch an object from the SDK, you get a
+snapshot. If there are updates on the server after that snapshot, you
+won't know about them until you call ``refresh`` on your object. The
+object might even have been deleted.
+
+
+
+
+

examples/analytics/input.py

@@ -37,17 +37,17 @@ def __init__(self, application_name, splunk_info, index = ANALYTICS_INDEX_NAME):
         self.splunk = client.connect(**splunk_info)
         self.index = index
 
-        if not self.splunk.indexes.contains(self.index):
+        if not self.index in self.splunk.indexes:
             self.splunk.indexes.create(self.index)
-        assert(self.splunk.indexes.contains(self.index))
+        assert(self.index in self.splunk.indexes)
 
-        if not self.splunk.confs['props'].contains(ANALYTICS_SOURCETYPE):
+        if ANALYTICS_SOURCETYPE not in self.splunk.confs['props']:
             self.splunk.confs["props"].create(ANALYTICS_SOURCETYPE)
             stanza = self.splunk.confs["props"][ANALYTICS_SOURCETYPE]
             stanza.submit("LINE_BREAKER = (%s)" % EVENT_TERMINATOR)
             stanza.submit("CHARSET = UTF-8")
             stanza.submit("SHOULD_LINEMERGE = false")
-        assert(self.splunk.confs['props'].contains(ANALYTICS_SOURCETYPE))
+        assert(ANALYTICS_SOURCETYPE in self.splunk.confs['props'])
 
     @staticmethod
     def encode(props):

examples/analytics/output.py

@@ -37,6 +37,18 @@ class TimeRange:
     WEEK="1w"
     MONTH="1mon"    
 
+def counts(job, result_key):
+    applications = []
+    reader = results.ResultsReader(job.results())
+    for result in reader:
+        if isinstance(result, dict):
+            applications.append({
+                    "name": result[result_key],
+                    "count": int(result["count"] or 0)
+                    })
+    return applications
+    
+
 class AnalyticsRetriever:
     def __init__(self, application_name, splunk_info, index = ANALYTICS_INDEX_NAME):
         self.application_name = application_name
@@ -46,32 +58,12 @@ def __init__(self, application_name, splunk_info, index = ANALYTICS_INDEX_NAME):
     def applications(self):
         query = "search index=%s | stats count by application" % (self.index)
         job = self.splunk.jobs.create(query, exec_mode="blocking")
-
-        applications = []
-        reader = results.ResultsReader(job.results())
-        for kind,result in reader:
-            if kind == results.RESULT:
-                applications.append({
-                    "name": result["application"],
-                    "count": int(result["count"] or 0)
-                })
-
-        return applications
+        return counts(job, "application")
 
     def events(self):
         query = "search index=%s application=%s | stats count by event" % (self.index, self.application_name)
         job = self.splunk.jobs.create(query, exec_mode="blocking")
-
-        events = []
-        reader = results.ResultsReader(job.results())
-        for kind,result in reader:
-            if kind == results.RESULT:
-                events.append({
-                    "name": result["event"],
-                    "count": int(result["count"] or 0)
-                })
-
-        return events
+        return counts(job, "event")
 
     def properties(self, event_name):
         query = 'search index=%s application=%s event="%s" | stats dc(%s*) as *' % (
@@ -81,17 +73,18 @@ def properties(self, event_name):
 
         properties = []
         reader = results.ResultsReader(job.results())
-        for kind,result in reader:
-            if kind == results.RESULT:
-                for field, count in result.iteritems():
-                    # Ignore internal ResultsReader properties
-                    if field.startswith("$"):
-                        continue
-
-                    properties.append({
+        for result in reader:
+            if not isinstance(result, dict):
+                continue
+            for field, count in result.iteritems():
+                # Ignore internal ResultsReader properties
+                if field.startswith("$"):
+                    continue
+
+                properties.append({
                         "name": field,
                         "count": int(count or 0)
-                    })
+                        })
 
         return properties
 
@@ -105,8 +98,8 @@ def property_values(self, event_name, property):
 
         values = []
         reader = results.ResultsReader(job.results())
-        for kind,result in reader:
-            if kind == results.RESULT:
+        for result in reader:
+            if isinstance(result, dict):
                 if result[property]:
                     values.append({
                         "name": result[property],
@@ -125,8 +118,8 @@ def events_over_time(self, event_name = "", time_range = TimeRange.MONTH, proper
 
         over_time = {}
         reader = results.ResultsReader(job.results())
-        for kind,result in reader:
-            if kind == results.RESULT:
+        for result in reader:
+            if isinstance(result, dict):
                 # Get the time for this entry
                 time = result["_time"]
                 del result["_time"]

examples/job.py

@@ -183,7 +183,7 @@ def sid(self, spec):
         """Convert the given search specifier into a search-id (sid)."""
         if spec.startswith('@'):
             index = int(spec[1:])
-            jobs = self.service.jobs()
+            jobs = self.service.jobs.list()
             if index < len(jobs):
                 return jobs[index].sid
         return spec # Assume it was already a valid sid

examples/oneshot.py

@@ -28,12 +28,9 @@
 
 def pretty(response):
     reader = results.ResultsReader(response)
-    while True:
-        kind = reader.read()
-        if kind == None: break
-        if kind == results.RESULT:
-            event = reader.value
-            pprint(event)
+    for result in reader:
+        if isinstance(result, dict):
+            pprint(result)
 
 def main():
     usage = "usage: oneshot.py <search>"
@@ -44,7 +41,7 @@ def main():
     search = opts.args[0]
     service = connect(**opts.kwargs)
     socket.setdefaulttimeout(None)
-    response = service.jobs.create(search, exec_mode="oneshot")
+    response = service.jobs.oneshot(search)
 
     pretty(response)
 

examples/search.py

@@ -77,12 +77,12 @@ def main(argv):
 
     job = service.jobs.create(search, **kwargs_create)
     while True:
-        stats = job.refresh()(
-            'isDone', 
-            'doneProgress', 
-            'scanCount', 
-            'eventCount', 
-            'resultCount')
+        job.refresh()
+        stats = {'isDone': job['isDone'],
+                 'doneProgress': job['doneProgress'],
+                 'scanCount': job['scanCount'],
+                 'eventCount': job['eventCount'],
+                 'resultCount': job['resultCount']}
         progress = float(stats['doneProgress'])*100
         scanned = int(stats['scanCount'])
         matched = int(stats['eventCount'])

examples/spurl.py

@@ -26,8 +26,8 @@
 
 # Invoke the url using the given opts parameters
 def invoke(path, **kwargs):
-    message = { "method": kwargs.get("method", "GET"), }
-    return binding.connect(**kwargs).request(path, message)
+    method = kwargs.get("method", "GET")
+    return binding.connect(**kwargs).request(path, method=method)
 
 def print_response(response):
     if response.status != 200:

setup.py

@@ -25,6 +25,8 @@
 
     description="The Splunk Software Development Kit for Python.",
 
+    install_requires=['ordereddict'],
+
     license="http://www.apache.org/licenses/LICENSE-2.0",
 
     name="splunk-sdk",