Secure beta changes (#41)

* Split sdc into monitor and common

First step towards adding support for sysdig secure in the python sdc
client--splitting existing api into common and monitor-only
halves.

The new terminology is that "sdc" refers to the common code for both
monitor and secure. _SdcCommon is the (private) base class that has
endpoints/methods common to both monitor and secure. SdMonClient is the
class that provides all sysdig monitor capabilities, inheriting from
_SdcCommon. SdSecureClient will implement sysdig secure.

Split all the methods into the two classes as follows based on endpoint:

Common endpoints:
- /api/user/me
- /api/token (and /api/token/<user>/<id>)
- /api/agents/connected
- /api/notificationChannels
- /api/history/timelines
- /api/data
- /api/events
- /api/sysdig
- /api/users
- /api/teams
- /api/user
- /api/agents/config

Monitor-only endpoints:
- /api/data/metrics
- /api/alerts
- /api/notifications
- /data/drilldownDashboardDescriptors.json
- /data/drilldownDashboards
- /ui/dashboards
- /api/groupConfigurations
- /api/agents/falco_rules

Add __init__.py changes so all 3 objects are imported.

Change __checkResponse so it's simply hidden instead of hidden + mangled
with the class name. Allows sharing in inherited classes.

* Add secure methods and tests.

Add methods
SdSecureClient.{get_set}_{user,system}_rules(new-rules-content). Set
actually only works for on-premise deployments. New examples
{get,set}_secure_{user,sytem}_falco_rules.py corespond to the
SdSecureClient methods.

Add new methods SdSecureClient.get_policy_events_{range,duration} to
fetch policy events. The API allows for an initial query based either on
duration or from/to. It returns a context dict containing the current
offset for pagination. That context can be provided to
get_more_policy_events() which increments the offset. The caller should
stop when the number of returned events is 0.

Also add a test script test/test_secure_apis.sh that can be used for
travis, and run that test script in travis. This script depends on a
staging account [email protected]. The API key and customer
id are passed to travis via environment variables.

The test script tests the following:

 - Trying to set the system falco rules file. This will fail with a
   405 (method not allowed) error, as this account is saas.
 - Trying to get the system falco rules file. This is compared to a
   known system rules file in test/sample-falco-rules.yaml.
 - Trying to set and get the user falco rules file. This uses the date
   as part of the rule name to ensure a fresh value is set/read.
 - Starting an agent with the python-sdc-testing account and triggering
   policy events, while checking for recent policy events. It should
   eventually find some policy events.

Also, for consistency's sake switch to external env variables for all
keys--instead of using a mix of embedded and externally provided
environment variables for api keys/access keys, use externally provided
keys everywhere.

* Add methods to manage security policies

Add methods/example programs for limited management of security
policies:
 - create default policies from corresponding system falco rules file
 - list all policies
 - delete all policies

Add tests to use these example programs in conjunction with the system
rules file for the python-sdc-testing staging account.

* Add sphinx docs

Add docs for all SdSecureClient methods. Also update the docs
specification to properly handle both the SdSecureClient and SdMonClient
objects (thanks @philrz!).

I was able to run "make docs" and browse the docs for both
objects. Shared methods for both objects are actually listed twice,
under each object. This seems ok for now as long as they're pulling docs
from a single location.

* Add ability to summarize policy events

Add the ability to summarize policy events by output string. This
removes any likely container information (xxx (id=yyy)) from output
strings, and then sorts by frequency descending.

New option --summarize controls whether to print summary or raw outputs,
and --limit allows you to print only the first <limit> entries.

* Make sure ssl_verify is honored for all methods.

Make sure the instance variable self.ssl_verify, which is set from the
environment variable SDC_SSL_VERIFY, is actually used for any call to
requests.

This is necessary to handle communicating with servers using self-signed
certificates.

* Add programs/methods to manipulate secure policies

Add methods and example programs to manipulate policies, specifically:
  - Get a policy
  - Add a policy
  - Modify a policy
  - Delete a policy

The methods all work on json objects so they are dependent on the data
format. But they allow for some scriptability.

* Add missing newline.

Summary output was all being printed on one line.

* Move to monitor/common

Was put under secure as a part of merge.

* call hidden unmangled checkResponse

Part of changes for secure/monitor split.

Author: Mark Stemm

.travis.yml

@@ -1,64 +1,68 @@
 language: python
-env:
-  global:
-    secure: h6qCuQnKi7fIzR53XqePRg2Hh0NrypqLyvYlJ+GEegEIP2PsPEwXMXZbswD7r+P4zsx7z2mBbR6VVYcjm1s0Zlf8RM2CNJ6IluAzDsg5Er7X7MAPaDEsfGJohZiTOPJvJNOV1BU7Qc4Y8Sqmd5ujVIJ4pL67oMegtmD36QBUAr7ERNrzbDXnPupIkcU/vf3z3vvCnH7/DIyP5hs1P+OAsgHY49bBMXxMEeMYQc0rtCN3F9yCmqh2f1mfEPouQnegKwcVFf3H08WQhYulAyNtrN0kcipMwF9+7cpLWtVyVMKepXvxXC0ltbaUTUqjvg5Op8WLOTG2xwXRdj6Fi985JPCvmsnX9teWT8WyxMex8ig7BVpontPajrhZJPfPWqj+oypqu8GMo/G0xD0uBGIVkadIQI4DQC1xFjKBBwntLdukeOApiAMhBEQtjQ5tO8vZw3kXifBdRHVuVHUjW6Lxd2R0xCKI0ZUmNggDaidsDRSU0C0HmHPHJELfGVm9Ohbsc01dYGiPczplV7Nxc4OKiPBZ5cvoKyNZFmM7ABtMKfGZR9PFv3A+6zzjnqIbHVdszzKN6ODKeotdVRt/VV7cMwZknwUqf7+fE/pjaHGpQ5ubX8wxhPzyxfcJHzKzYklFGCT8Rv/+VooYWJaKyjZ/e63fsjRTfTX84KRpSdWe4U8=
+services:
+  - docker
 python:
 - '2.7'
+install:
+  - sudo apt-get install linux-headers-$(uname -r) dkms gcc-multilib g++-multilib
+  - pip install pyyaml requests
 script:
 - echo "Testing source version"
-- examples/create_alert.py XXX
-- examples/update_alert.py XXX
-- examples/delete_alert.py XXX
-- examples/dashboard.py XXX
-- examples/create_dashboard.py XXX
-- examples/delete_dashboard.py XXX
-- examples/get_data_advanced.py XXX ip-10-0-1-140.ec2.internal
-- examples/get_data_datasource.py XXX
-- examples/get_data_simple.py XXX
-- examples/list_alerts.py XXX
-- examples/list_alert_notifications.py XXX
-- examples/resolve_alert_notifications.py XXX 1
-- examples/list_dashboards.py XXX
-- examples/list_hosts.py XXX
-- examples/list_metrics.py XXX
-- examples/post_event.py XXX "test_event_name" -d "test event description"
-- examples/post_event_simple.py XXX "test_event_name" "test event description"
-- examples/list_events.py XXX
-- examples/delete_event.py XXX
-- examples/print_data_retention_info.py XXX
-- examples/print_explore_grouping.py XXX
-- examples/print_user_info.py XXX
-- examples/list_users.py XXX
-- examples/list_sysdig_captures.py XXX
-- examples/create_sysdig_capture.py XXX ip-10-0-1-115.ec2.internal apicapture 10
-- examples/notification_channels.py XXX
-- examples/user_team_mgmt.py XXX example-team [email protected]
+- examples/create_alert.py $DEMO_KUBE_API_TOKEN
+- examples/update_alert.py $DEMO_KUBE_API_TOKEN
+- examples/delete_alert.py $DEMO_KUBE_API_TOKEN
+- examples/dashboard.py $DEMO_KUBE_API_TOKEN
+- examples/create_dashboard.py $DEMO_KUBE_API_TOKEN
+- examples/delete_dashboard.py $DEMO_KUBE_API_TOKEN
+- examples/get_data_advanced.py $DEMO_KUBE_API_TOKEN ip-10-0-1-140.ec2.internal
+- examples/get_data_datasource.py $DEMO_KUBE_API_TOKEN
+- examples/get_data_simple.py $DEMO_KUBE_API_TOKEN
+- examples/list_alerts.py $DEMO_KUBE_API_TOKEN
+- examples/list_alert_notifications.py $DEMO_KUBE_API_TOKEN
+- examples/resolve_alert_notifications.py $DEMO_KUBE_API_TOKEN 1
+- examples/list_dashboards.py $DEMO_KUBE_API_TOKEN
+- examples/list_hosts.py $DEMO_KUBE_API_TOKEN
+- examples/list_metrics.py $DEMO_KUBE_API_TOKEN
+- examples/post_event.py $DEMO_KUBE_API_TOKEN "test_event_name" -d "test event description"
+- examples/post_event_simple.py $DEMO_KUBE_API_TOKEN "test_event_name" "test event description"
+- examples/list_events.py $DEMO_KUBE_API_TOKEN
+- examples/delete_event.py $DEMO_KUBE_API_TOKEN
+- examples/print_data_retention_info.py $DEMO_KUBE_API_TOKEN
+- examples/print_explore_grouping.py $DEMO_KUBE_API_TOKEN
+- examples/print_user_info.py $DEMO_KUBE_API_TOKEN
+- examples/list_users.py $DEMO_KUBE_API_TOKEN
+- examples/list_sysdig_captures.py $DEMO_KUBE_API_TOKEN
+- examples/create_sysdig_capture.py $DEMO_KUBE_API_TOKEN ip-10-0-1-115.ec2.internal apicapture 10
+- examples/notification_channels.py $DEMO_KUBE_API_TOKEN
+- examples/user_team_mgmt.py $DEMO_KUBE_API_TOKEN example-team [email protected]
+- unset SDC_TOKEN
+- bash test/test_secure_apis.sh
 - echo "Testing pip version"
 - rm -rf sdcclient
 - pip install sdcclient
-# - examples/create_alert.py XXX
-# - examples/delete_alert.py XXX
-# - examples/dashboard.py XXX
-# - examples/create_dashboard.py XXX
-# - examples/delete_dashboard.py XXX
-# - examples/get_data_advanced.py XXX ip-10-0-2-180.ec2.internal
-# - examples/get_data_datasource.py XXX
-# - examples/get_data_simple.py XXX
-# - examples/list_alerts.py XXX
-# - examples/list_alert_notifications.py XXX
-# - examples/resolve_alert_notifications.py XXX 1
-# - examples/list_dashboards.py XXX
-# - examples/list_hosts.py XXX
-# - examples/list_metrics.py XXX
-# - examples/post_event.py XXX "test event name" -d "test event description"
-# - examples/post_event_simple.py XXX "test event name" "test event description"
-# - examples/list_events.py XXX
-# - examples/delete_event.py XXX
-# - examples/print_data_retention_info.py XXX
-# - examples/print_explore_grouping.py XXX
-# - examples/print_user_info.py XXX
-# - examples/list_sysdig_captures.py XXX
-# - examples/create_sysdig_capture.py XXX ip-10-0-2-202.ec2.internal apicapture 10
+# - examples/create_alert.py $DEMO_KUBE_API_TOKEN
+# - examples/delete_alert.py $DEMO_KUBE_API_TOKEN
+# - examples/dashboard.py $DEMO_KUBE_API_TOKEN
+# - examples/create_dashboard.py $DEMO_KUBE_API_TOKEN
+# - examples/delete_dashboard.py $DEMO_KUBE_API_TOKEN
+# - examples/get_data_advanced.py $DEMO_KUBE_API_TOKEN ip-10-0-2-180.ec2.internal
+# - examples/get_data_datasource.py $DEMO_KUBE_API_TOKEN
+# - examples/get_data_simple.py $DEMO_KUBE_API_TOKEN
+# - examples/list_alerts.py $DEMO_KUBE_API_TOKEN
+# - examples/list_alert_notifications.py $DEMO_KUBE_API_TOKEN
+# - examples/resolve_alert_notifications.py $DEMO_KUBE_API_TOKEN 1
+# - examples/list_dashboards.py $DEMO_KUBE_API_TOKEN
+# - examples/list_hosts.py $DEMO_KUBE_API_TOKEN
+# - examples/list_metrics.py $DEMO_KUBE_API_TOKEN
+# - examples/post_event.py $DEMO_KUBE_API_TOKEN "test event name" -d "test event description"
+# - examples/post_event_simple.py $DEMO_KUBE_API_TOKEN "test event name" "test event description"
+# - examples/list_events.py $DEMO_KUBE_API_TOKEN
+# - examples/delete_event.py $DEMO_KUBE_API_TOKEN
+# - examples/print_data_retention_info.py $DEMO_KUBE_API_TOKEN
+# - examples/print_explore_grouping.py $DEMO_KUBE_API_TOKEN
+# - examples/print_user_info.py $DEMO_KUBE_API_TOKEN
+# - examples/list_sysdig_captures.py $DEMO_KUBE_API_TOKEN
+# - examples/create_sysdig_capture.py $DEMO_KUBE_API_TOKEN ip-10-0-2-202.ec2.internal apicapture 10
 notifications:
   slack:
     secure: GJ0H2wAaW67t3+x+cCjOVLw/b+YAZjH9rebAfluCFJklJS9u+V6/qqIyiNDAkMPIcgilFyhiyOQCZYXapgthQ1ieFbZZo//mrRtuGo2wuxCAwcOx2mXAZpZ4I5b3+Q/znVqSuqkwFRid1d138z4TW7sYSIburouDX3QUNoUOy+g7VJxCFQCcqlN8LYxGJHQYdOZa9zIGCtKrOtZ/B8C1TLgXmDMwAAVNO2WzL4GiBTLCGuMsQWMTLw2Qmv1ayZPztmeDWo1C9oa7HIH8Bg2YVjssR87el28X+EqEO533mgYjPmW2/hii30WVFOUE5hMdvKeQLBvy5N3/OCch1np0RQBd8eYEtaPv38rc5L2wAnUq9G5Zzr252z7vnwSLi6lap9jWU8tOerSTEPU+jG05PnuCnufVDXVNPyiPsH6BDP4qxHmLjooNpxfe63Df7NNyUi2I3QoroLj/UzI7zZVQjJEqsTrr5BbsH4z6NTGY91+ZqobBn62+hV3ESAam0ivQgC7s2AKko0qkKyIUGjj7ozU8ebo1UpagNvKC/J9szMqtdXJgKtG8BeonyLMeN6MEEyEvcMJbB4dCcfet+1Sb9AZWnGvYVdajhVLb1HE6OrbzZyhC3KqCe06J9O5BCY9ncy+l16i7MyIfcKTibHQlxPU+Id/VijD97JSRXxnd2i4=

README.md

@@ -1,12 +1,12 @@
-Sysdig Cloud Python client library
+Sysdig Monitor/Secure Python client library
 ===
 
 [![Build Status](https://travis-ci.org/draios/python-sdc-client.png?branch=master)](https://travis-ci.org/draios/python-sdc-client)
 [![Current version on PyPI](http://img.shields.io/pypi/v/sdcclient.svg)](https://pypi.python.org/pypi/sdcclient)
 
-A Python client API for Sysdig Cloud.
+A Python client API for Sysdig Monitor/Sysdig Secure.
 
-This module is a wrapper around the Sysdig Cloud API, which is documented [here](http://support.sysdigcloud.com/hc/en-us/articles/205233166-The-Sysdig-Cloud-API-Specification). It exposes most of the sysdig REST API functionality as an easy to use and easy to install Python interface. The repository includes a rich set of examples (in the [examples](examples/) subdir) that quickly address several use cases.
+This module is a wrapper around the Sysdig Monitor/Sysdig Secure APIs, which are documented [here](http://support.sysdigcloud.com/hc/en-us/articles/205233166-The-Sysdig-Cloud-API-Specification). It exposes most of the sysdig REST API functionality as an easy to use and easy to install Python interface. The repository includes a rich set of examples (in the [examples](examples/) subdir) that quickly address several use cases.
 
 Installation
 ------------
@@ -20,36 +20,38 @@ Installation
 
 Quick start
 -----------
-- If you are interested in exporting metrics data from Sysdig Cloud, take a look at [examples/get_data_simple.py](examples/get_data_simple.py) and [examples/get_data_advanced.py](examples/get_data_advanced.py).
+- If you are interested in exporting metrics data from Sysdig Monitor, take a look at [examples/get_data_simple.py](examples/get_data_simple.py) and [examples/get_data_advanced.py](examples/get_data_advanced.py).
 - If you want to programmatically create an alert, refer to [examples/create_alert.py](examples/create_alert.py)
 - If you want to programmatically create a dashboard, refer to [examples/create_dashboard.py](examples/create_dashboard.py)
 
 Usage
 -----
 
-_Note:_ in order to use this API you must obtain a Sysdig Cloud token. You can get your user's token in the _Sysdig Cloud API_ section of the [settings](https://app.sysdigcloud.com/#/settings/user) page.
+_Note:_ in order to use this API you must obtain a Sysdig Monitor/Secure API token. You can get your user's token in the _Sysdig Monitor API_ section of the settings page for [monitor](https://app.sysdigcloud.com/#/settings/user) or [secure](https://secure.sysdig.com/#/settings/user).
 
-The library exports a single class, `SdcClient`, that is used to connect to Sysdig Cloud and execute actions. It can be instantiated like this:
+The library exports two classes, `SdMonClient` and `SdSecureClient` that are used to connect to Sysdig Monitor/Secure and execute actions. They can be instantiated like this:
 
 ``` python
-from sdcclient import SdcClient
+from sdcclient import SdMonClient
 
-sdc_token = "MY_API_TOKEN"
+api_token = "MY_API_TOKEN"
 
 #
-# Instantiate the SDC client
+# Instantiate the Sysdig Monitor client
 #
-sdclient = SdcClient(sdc_token)
+client = SdMonClient(api_token)
 ```
 
+For backwards compatibility purposes, a third class `SdcClient` is exported which is an alias of `SdMonClient`.
+
 Once instantiated, all the methods documented below can be called on the object.
 
-####Return Values
-Every method in the SdcClient class returns **a list with two entries**. The first one is a boolean value indicating if the call was successful. The second entry depends on the result:
+#### Return Values
+Every method in the SdMonClient/SdSecureClient classes returns **a list with two entries**. The first one is a boolean value indicating if the call was successful. The second entry depends on the result:
 - If the call was successful, it's a dictionary reflecting the json returned by the underlying REST call
 - If the call failed, it's a string describing the error
 
-For an example on how to parse this output, take a look at a simple example like [get_data_simple.py](examples/get_data_simple.py) 
+For an example on how to parse this output, take a look at a simple example like [get_data_simple.py](examples/get_data_simple.py)
 
 Function List
 -------------
@@ -58,7 +60,7 @@ Please Refer to the [Python Script Library documentation page](http://python-sdc
 
 On-Premises Installs
 --------------------
-For [On-Premises Sysdig Cloud installs](https://support.sysdigcloud.com/hc/en-us/articles/206519903-On-Premises-Installation-Guide), additional configuration is necessary to point to your API server rather than the default SaaS-based one, and also to easily connect when using a self-signed certificate for SSL. One way to handle this is by setting environment variables before running your Python scripts:
+For [On-Premises Sysdig Monitor installs](https://support.sysdigcloud.com/hc/en-us/articles/206519903-On-Premises-Installation-Guide), additional configuration is necessary to point to your API server rather than the default SaaS-based one, and also to easily connect when using a self-signed certificate for SSL. One way to handle this is by setting environment variables before running your Python scripts:
 
 ```
 export SDC_URL='https://<YOUR-API-SERVER-HOSTNAME-OR-IP>'
@@ -68,7 +70,7 @@ export SDC_SSL_VERIFY='false'
 Alternatively, you can specify the additional arguments in your Python scripts as you instantiate the SDC client:
 
 ```
-sdclient = SdcClient(sdc_token, sdc_url='https://<YOUR-API-SERVER-HOSTNAME-OR-IP>', ssl_verify=False)
+client = SdMonClient(api_token, sdc_url='https://<YOUR-API-SERVER-HOSTNAME-OR-IP>', ssl_verify=False)
 ```
 
 

doc/index.rst

@@ -16,6 +16,11 @@ This page documents the functions available in the `Python Script Library <https
 Function List
 =============
 .. py:module:: sdcclient
-.. autoclass:: SdcClient
+.. autoclass:: SdMonClient
    :members:
+   :inherited-members:
+   :undoc-members:
+.. autoclass:: SdSecureClient
+   :members:
+   :inherited-members:
    :undoc-members:

examples/add_policy.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+#
+# Add a new policy
+#
+
+import os
+import sys
+import json
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
+from sdcclient import SdSecureClient
+
+def usage():
+    print 'usage: %s <sysdig-token>' % sys.argv[0]
+    print 'Reads policy json from standard input'
+    print 'You can find your token at https://secure.sysdig.com/#/settings/user'
+    sys.exit(1)
+
+#
+# Parse arguments
+#
+if len(sys.argv) != 2:
+    usage()
+
+sdc_token = sys.argv[1]
+
+policy_json = sys.stdin.read()
+
+#
+# Instantiate the SDC client
+#
+sdclient = SdSecureClient(sdc_token, 'https://secure.sysdig.com')
+
+res = sdclient.add_policy(policy_json)
+
+#
+# Return the result
+#
+if res[0]:
+    print json.dumps(res[1], indent=2)
+else:
+    print res[1]
+    sys.exit(1)
+
+

examples/create_default_policies.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+#
+# Create the default set of policies given the falco rules file.
+# Existing policies with the same name are unchanged. New policies
+# as needed will be added. Returns JSON representing the new
+# policies created.
+#
+
+import os
+import sys
+import json
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
+from sdcclient import SdSecureClient
+
+def usage():
+    print 'usage: %s <sysdig-token>' % sys.argv[0]
+    print 'You can find your token at https://secure.sysdig.com/#/settings/user'
+    sys.exit(1)
+
+#
+# Parse arguments
+#
+if len(sys.argv) != 2:
+    usage()
+
+sdc_token = sys.argv[1]
+
+#
+# Instantiate the SDC client
+#
+sdclient = SdSecureClient(sdc_token, 'https://secure.sysdig.com')
+
+res = sdclient.create_default_policies()
+
+#
+# Return the result
+#
+if res[0]:
+    print json.dumps(res[1], indent=2)
+else:
+    print res[1]
+    sys.exit(1)
+
+

examples/delete_all_policies.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+#
+# Delete all secure policies.
+#
+
+import os
+import sys
+import json
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
+from sdcclient import SdSecureClient
+
+def usage():
+    print 'usage: %s <sysdig-token>' % sys.argv[0]
+    print 'You can find your token at https://secure.sysdig.com/#/settings/user'
+    sys.exit(1)
+
+#
+# Parse arguments
+#
+if len(sys.argv) != 2:
+    usage()
+
+sdc_token = sys.argv[1]
+
+#
+# Instantiate the SDC client
+#
+sdclient = SdSecureClient(sdc_token, 'https://secure.sysdig.com')
+
+res = sdclient.delete_all_policies()
+
+#
+# Return the result
+#
+if res[0]:
+    print res[1]
+else:
+    print res[1]
+    sys.exit(1)
+
+