Add policy priorities (#45)

* Add API wrappers to set/get policy priorities.

New methods get_policy_priorities/set_policy_priorites map to GET
/api/policies/priorities, PUT /api/policies/priorities, respsectively.

They work on raw json objects. Some minimal massaging of the objects
will be handled in the example programs.

* Update usage string.

It doesn't read the policy name from the cmdline, and reads the policy
description from stdin.

* Add priorities order to list_policies

Add the ability to return the order of policies only instead of the full
list of priorities. With -o/--order-only, only the list of ids is
returned.

Also ensure that the list of priorities is always sorted in priorities
order.

* Wrapper program to set policy order.

Example program to set the order of secure policies. Takes a simple list
of policy ids as returned by list_policies --order-only, and wraps it in
the necessary version/date object before using set_policy_priorities.

* Add tests for new policy priorities apis

Use list_policies -o to get the current policies order, reverse the
order using "jq reverse", and set the order.

Use grep -b to find where in the policies list the policy "Write Below
Binary Dir" is, before and after setting the order. You should find that
the position of the policy moves to farther in the output e.g. the end.

Also use tee when fetching the falco rules file so if there's an error
we will see it.

* Update monitor tests to use python-sdc-client acct

Move all of the monitor unit tests from .travis.yml into a script
test_monitor_apis.sh, and get rid of the (commented-out) tests for the
pip version of sdcclient.

In test_monitor_apis.sh, instead of relying on a hard-coded hostname,
start an agent on the fly using docker and use the current hostname for
get_data_advanced/create_sysdig_capture.

Also make changes to let multiple copies of test script run at once:

Many of the test programs relied on hard coded names of alerts,
dashboards, notification channels, etc. This caused problems when
running two copies of the test script at once, for example for the
travisci push and pr tests.

Fix this by making the various alert/dashboard/channel names
configurable in each example script, and using unique names in the
wrapper script that calls the individual scripts.

Author: mstemm

.travis.yml

@@ -7,62 +7,8 @@ install:
   - sudo apt-get install linux-headers-$(uname -r) dkms gcc-multilib g++-multilib
   - pip install pyyaml requests
 script:
-- echo "Testing source version"
-- examples/create_alert.py $DEMO_KUBE_API_TOKEN
-- examples/update_alert.py $DEMO_KUBE_API_TOKEN
-- examples/delete_alert.py $DEMO_KUBE_API_TOKEN
-- examples/dashboard.py $DEMO_KUBE_API_TOKEN
-- examples/create_dashboard.py $DEMO_KUBE_API_TOKEN
-- examples/delete_dashboard.py $DEMO_KUBE_API_TOKEN
-- examples/get_data_advanced.py $DEMO_KUBE_API_TOKEN ip-10-0-1-140.ec2.internal
-- examples/get_data_datasource.py $DEMO_KUBE_API_TOKEN
-- examples/get_data_simple.py $DEMO_KUBE_API_TOKEN
-- examples/list_alerts.py $DEMO_KUBE_API_TOKEN
-- examples/list_alert_notifications.py $DEMO_KUBE_API_TOKEN
-- examples/resolve_alert_notifications.py $DEMO_KUBE_API_TOKEN 1
-- examples/list_dashboards.py $DEMO_KUBE_API_TOKEN
-- examples/list_hosts.py $DEMO_KUBE_API_TOKEN
-- examples/list_metrics.py $DEMO_KUBE_API_TOKEN
-- examples/post_event.py $DEMO_KUBE_API_TOKEN "test_event_name" -d "test event description"
-- examples/post_event_simple.py $DEMO_KUBE_API_TOKEN "test_event_name" "test event description"
-- examples/list_events.py $DEMO_KUBE_API_TOKEN
-- examples/delete_event.py $DEMO_KUBE_API_TOKEN
-- examples/print_data_retention_info.py $DEMO_KUBE_API_TOKEN
-- examples/print_explore_grouping.py $DEMO_KUBE_API_TOKEN
-- examples/print_user_info.py $DEMO_KUBE_API_TOKEN
-- examples/list_users.py $DEMO_KUBE_API_TOKEN
-- examples/list_sysdig_captures.py $DEMO_KUBE_API_TOKEN
-- examples/create_sysdig_capture.py $DEMO_KUBE_API_TOKEN ip-10-0-1-115.ec2.internal apicapture 10
-- examples/notification_channels.py $DEMO_KUBE_API_TOKEN
-- examples/user_team_mgmt.py $DEMO_KUBE_API_TOKEN example-team [email protected]
-- unset SDC_TOKEN
+- bash test/test_monitor_apis.sh
 - bash test/test_secure_apis.sh
-- echo "Testing pip version"
-- rm -rf sdcclient
-- pip install sdcclient
-# - examples/create_alert.py $DEMO_KUBE_API_TOKEN
-# - examples/delete_alert.py $DEMO_KUBE_API_TOKEN
-# - examples/dashboard.py $DEMO_KUBE_API_TOKEN
-# - examples/create_dashboard.py $DEMO_KUBE_API_TOKEN
-# - examples/delete_dashboard.py $DEMO_KUBE_API_TOKEN
-# - examples/get_data_advanced.py $DEMO_KUBE_API_TOKEN ip-10-0-2-180.ec2.internal
-# - examples/get_data_datasource.py $DEMO_KUBE_API_TOKEN
-# - examples/get_data_simple.py $DEMO_KUBE_API_TOKEN
-# - examples/list_alerts.py $DEMO_KUBE_API_TOKEN
-# - examples/list_alert_notifications.py $DEMO_KUBE_API_TOKEN
-# - examples/resolve_alert_notifications.py $DEMO_KUBE_API_TOKEN 1
-# - examples/list_dashboards.py $DEMO_KUBE_API_TOKEN
-# - examples/list_hosts.py $DEMO_KUBE_API_TOKEN
-# - examples/list_metrics.py $DEMO_KUBE_API_TOKEN
-# - examples/post_event.py $DEMO_KUBE_API_TOKEN "test event name" -d "test event description"
-# - examples/post_event_simple.py $DEMO_KUBE_API_TOKEN "test event name" "test event description"
-# - examples/list_events.py $DEMO_KUBE_API_TOKEN
-# - examples/delete_event.py $DEMO_KUBE_API_TOKEN
-# - examples/print_data_retention_info.py $DEMO_KUBE_API_TOKEN
-# - examples/print_explore_grouping.py $DEMO_KUBE_API_TOKEN
-# - examples/print_user_info.py $DEMO_KUBE_API_TOKEN
-# - examples/list_sysdig_captures.py $DEMO_KUBE_API_TOKEN
-# - examples/create_sysdig_capture.py $DEMO_KUBE_API_TOKEN ip-10-0-2-202.ec2.internal apicapture 10
 notifications:
   slack:
     secure: GJ0H2wAaW67t3+x+cCjOVLw/b+YAZjH9rebAfluCFJklJS9u+V6/qqIyiNDAkMPIcgilFyhiyOQCZYXapgthQ1ieFbZZo//mrRtuGo2wuxCAwcOx2mXAZpZ4I5b3+Q/znVqSuqkwFRid1d138z4TW7sYSIburouDX3QUNoUOy+g7VJxCFQCcqlN8LYxGJHQYdOZa9zIGCtKrOtZ/B8C1TLgXmDMwAAVNO2WzL4GiBTLCGuMsQWMTLw2Qmv1ayZPztmeDWo1C9oa7HIH8Bg2YVjssR87el28X+EqEO533mgYjPmW2/hii30WVFOUE5hMdvKeQLBvy5N3/OCch1np0RQBd8eYEtaPv38rc5L2wAnUq9G5Zzr252z7vnwSLi6lap9jWU8tOerSTEPU+jG05PnuCnufVDXVNPyiPsH6BDP4qxHmLjooNpxfe63Df7NNyUi2I3QoroLj/UzI7zZVQjJEqsTrr5BbsH4z6NTGY91+ZqobBn62+hV3ESAam0ivQgC7s2AKko0qkKyIUGjj7ozU8ebo1UpagNvKC/J9szMqtdXJgKtG8BeonyLMeN6MEEyEvcMJbB4dCcfet+1Sb9AZWnGvYVdajhVLb1HE6OrbzZyhC3KqCe06J9O5BCY9ncy+l16i7MyIfcKTibHQlxPU+Id/VijD97JSRXxnd2i4=

examples/create_alert.py

@@ -6,6 +6,7 @@
 #
 #
 
+import getopt
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
@@ -14,12 +15,26 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-a|--alert <name>] <sysdig-token>' % sys.argv[0]
+    print '-a|--alert: Set name of alert to create'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"a:",["alert="])
+except getopt.GetoptError:
+    usage()
+
+alert_name = "tomcat cpu > 80% on any host"
+for opt, arg in opts:
+    if opt in ("-a", "--alert"):
+        alert_name = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -29,8 +44,8 @@
 #
 # Find notification channels (you need IDs to create an alert).
 #
-notify_channels = [ {'type': 'SLACK', 'channel': 'sysdig-demo2-alerts'},
-                    {'type': 'EMAIL', 'emailRecipients': ['demo-kube@draios.com', '[email protected]']},
+notify_channels = [ {'type': 'SLACK', 'channel': '#python-sdc-test-alert'},
+                    {'type': 'EMAIL', 'emailRecipients': ['python-sdc-testing@draios.com', '[email protected]']},
                     {'type': 'SNS', 'snsTopicARNs': ['arn:aws:sns:us-east-1:273107874544:alarms-stg']}
                     ]
 
@@ -44,7 +59,7 @@
 #
 # Create the alert.
 #
-res = sdclient.create_alert('tomcat cpu > 80% on any host',  # Alert name.
+res = sdclient.create_alert(alert_name,  # Alert name.
                             'this alert was automatically created using the python Sysdig Cloud library', # Alert description.
                             6, # Syslog-encoded severity. 6 means 'info'.
                             60, # The alert will fire if the condition is met for at least 60 seconds.

examples/create_dashboard.py

@@ -6,6 +6,7 @@
 # will monitor.
 #
 
+import getopt
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
@@ -14,12 +15,27 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-d|--dashboard <name>] <sysdig-token>' % sys.argv[0]
+    print '-d|--dashboard: Set name of dashboard to create'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"d:",["dashboard="])
+except getopt.GetoptError:
+    usage()
+
+# Name for the dashboard to create
+dashboardName = "API test - cassandra in prod"
+for opt, arg in opts:
+    if opt in ("-d", "--dashboard"):
+        dashboardName = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -30,8 +46,6 @@
 # Create the new dashboard, applying to cassandra in production
 #
 
-# Name for the dashboard to create
-dashboardName = "API test - cassandra in prod"
 # Name of the view to copy
 viewName = "Overview by Process"
 # Filter to apply to the new dashboard.
@@ -56,14 +70,12 @@
 # the dev namespace
 #
 
-# Name for the dashboard to create
-dashboardName = "API test - cassandra in dev"
 # Name of the dashboard to copy
-dashboardToCopy = "API test - cassandra in prod"
+dashboardCopy = "Copy Of {}".format(dashboardName)
 # Filter to apply to the new dashboard. Same as above.
 dashboardFilter = "kubernetes.namespace.name = dev and proc.name = cassandra"
 
-res = sdclient.create_dashboard_from_dashboard(dashboardName, dashboardToCopy, dashboardFilter)
+res = sdclient.create_dashboard_from_dashboard(dashboardCopy, dashboardName, dashboardFilter)
 
 #
 # Check the result

examples/dashboard.py

@@ -4,6 +4,7 @@
 # edit the content, and then delete it.
 #
 
+import getopt
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
@@ -12,12 +13,26 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-d|--dashboard <name>] <sysdig-token>' % sys.argv[0]
+    print '-d|--dashboard: Set name of dashboard to create'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"d:",["dashboard="])
+except getopt.GetoptError:
+    usage()
+
+dashboard_name = "My Dashboard"
+for opt, arg in opts:
+    if opt in ("-d", "--dashboard"):
+        dashboard_name = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -28,7 +43,6 @@
 #
 # Create an empty dashboard
 #
-dashboard_name = 'My Dashboard'
 dashboard_configuration = None
 res = sdclient.create_dashboard(dashboard_name)
 
@@ -142,4 +156,4 @@
     print 'Dashboard deleted successfully'
 else:
     print res[1]
-    sys.exit(1)
+    sys.exit(1)

examples/delete_alert.py

@@ -3,6 +3,7 @@
 # This example shows how to delete an alert
 #
 
+import getopt
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
@@ -11,12 +12,26 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-a|--alert <name>] <sysdig-token>' % sys.argv[0]
+    print '-a|--alert: Set name of alert to delete'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"a:",["alert="])
+except getopt.GetoptError:
+    usage()
+
+alert_name = "tomcat cpu > 80% on any host"
+for opt, arg in opts:
+    if opt in ("-a", "--alert"):
+        alert_name = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -29,7 +44,7 @@
     sys.exit(1)
 
 for alert in res[1]['alerts']:
-    if alert['name'] == "tomcat cpu > 80% on any host":
+    if alert['name'] == alert_name:
         print "Deleting alert"
         res = sdclient.delete_alert(alert)
         if not res[0]:

examples/delete_dashboard.py

@@ -3,6 +3,7 @@
 # This example shows how to delete a dashboard
 #
 
+import getopt
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
@@ -11,12 +12,26 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-p|--pattern <name>] <sysdig-token>' % sys.argv[0]
+    print '-p|--pattern: Delete all dashboards containing the provided pattern'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"p:",["pattern="])
+except getopt.GetoptError:
+    usage()
+
+pattern = "API Test"
+for opt, arg in opts:
+    if opt in ("-p", "--pattern"):
+        pattern = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -32,10 +47,10 @@
     sys.exit(1)
 
 #
-# Delete all the dashboards containing "API test"
+# Delete all the dashboards containing pattern
 #
 for dashboard in res[1]['dashboards']:
-    if 'API test' in dashboard['name']:
+    if pattern in dashboard['name']:
         print "Deleting " + dashboard['name']
         res = sdclient.delete_dashboard(dashboard)
         if not res[0]:

examples/delete_event.py

@@ -3,6 +3,7 @@
 # Delete user events from Sysdig Cloud
 #
 
+import getopt
 import json
 import os
 import sys
@@ -12,12 +13,26 @@
 #
 # Parse arguments
 #
-if len(sys.argv) != 2:
-    print 'usage: %s <sysdig-token>' % sys.argv[0]
+def usage():
+    print 'usage: %s [-e|--event <name>] <sysdig-token>' % sys.argv[0]
+    print '-e|--event: Name of event to delete'
     print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
     sys.exit(1)
 
-sdc_token = sys.argv[1]
+try:
+    opts, args = getopt.getopt(sys.argv[1:],"e:",["event="])
+except getopt.GetoptError:
+    usage()
+
+event_name = "test_event_name"
+for opt, arg in opts:
+    if opt in ("-e", "--event"):
+        event_name = arg
+
+if len(args) != 1:
+    usage()
+
+sdc_token = args[0]
 
 #
 # Instantiate the SDC client
@@ -27,7 +42,7 @@
 #
 # Get the events that match a name
 #
-res = sdclient.get_events(name='test_event_name')
+res = sdclient.get_events(name=event_name)
 
 if not res[0]:
 	print res[1]