Merge Zuul changes from Openstack

Merge branch 'upstream' which tracks the Openstack git repo.

Author: Alice Chen

README.rst

@@ -9,7 +9,7 @@ Contributing
 To browse the latest code, see: https://git.openstack.org/cgit/openstack-infra/zuul/tree/
 To clone the latest code, use `git clone git://git.openstack.org/openstack-infra/zuul`
 
-Bugs are handled at: https://launchpad.net/zuul
+Bugs are handled at: https://storyboard.openstack.org/#!/project/679
 
 Code reviews are, as you might expect, handled by gerrit. The gerrit they
 use is http://review.openstack.org

doc/source/cloner.rst

@@ -75,3 +75,15 @@ cloner processes the clones in the order supplied, so you should swap the
 projects::
 
   zuul-cloner project project/plugins/plugin1
+
+Cached repositories
+-------------------
+
+The ``--cache-dir`` option can be used to reduce network traffic by
+cloning from a local repository which may not be up to date.
+
+If the ``--cache-dir`` option is supplied, zuul-cloner will start by
+cloning any projects it processes from those found in that directory.
+The URL of origin remote of the resulting clone will be reset to use
+the ``git_base_url`` and then the remote will be updated so that the
+repository has all the information in the upstream repository.

doc/source/triggers.rst

@@ -34,6 +34,10 @@ want Zuul to gate.  For instance, you may want to grant ``Verified
 be added to Gerrit.  Zuul is very flexible and can take advantage of
 those.
 
+If using Gerrit 2.7 or later, make sure the user is a member of a group
+that is granted the ``Stream Events`` permission, otherwise it will not
+be able to invoke the ``gerrit stream-events`` command over SSH.
+
 Timer
 -----
 

doc/source/zuul.rst

@@ -36,7 +36,7 @@ server) are specified in a third section.
 The three sections of this config and their options are documented below.
 You can also find an example zuul.conf file in the git
 `repository
-<https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_
+<https://git.openstack.org/cgit/openstack-infra/zuul/tree/etc/zuul.conf-sample>`_
 
 gearman
 """""""
@@ -86,6 +86,8 @@ gerrit
 zuul
 """"
 
+.. _layout_config:
+
 **layout_config**
   Path to layout config file.  Used by zuul-server only.
   ``layout_config=/etc/zuul/layout.yaml``
@@ -272,10 +274,12 @@ include, and currently supports one type of inclusion, a python file::
     - python-file: local_functions.py
 
 **python-file**
-  The path to a python file.  The file will be loaded and objects that
-  it defines will be placed in a special environment which can be
-  referenced in the Zuul configuration.  Currently only the
-  parameter-function attribute of a Job uses this feature.
+  The path to a python file (either an absolute path or relative to the
+  directory name of :ref:`layout_config <layout_config>`).  The
+  file will be loaded and objects that it defines will be placed in a
+  special environment which can be referenced in the Zuul configuration.
+  Currently only the parameter-function attribute of a Job uses this
+  feature.
 
 Pipelines
 """""""""
@@ -462,7 +466,8 @@ explanation of each of the parameters::
     This may be used for any event.  It requires that a certain kind
     of approval be present for the current patchset of the change (the
     approval could be added by the event in question).  It follows the
-    same syntax as the "approval" pipeline requirement below.
+    same syntax as the :ref:`"approval" pipeline requirement below
+    <pipeline-require-approval>`.
 
   **timer**
     This trigger will run based on a cron-style time specification.
@@ -497,7 +502,8 @@ explanation of each of the parameters::
     This may be used for any event.  It requires that a certain kind
     of approval be present for the current patchset of the change (the
     approval could be added by the event in question).  It follows the
-    same syntax as the "approval" pipeline requirement below.
+    same syntax as the :ref:`"approval" pipeline requirement below
+    <pipeline-require-approval>`.
 
 
 **require**
@@ -507,6 +513,8 @@ explanation of each of the parameters::
   the conditions specified here must be met or the item will not be
   enqueued.
 
+.. _pipeline-require-approval:
+
   **approval**
   This requires that a certain kind of approval be present for the
   current patchset of the change (the approval could be added by the
@@ -561,6 +569,15 @@ explanation of each of the parameters::
   well.  To suppress this behavior (and allow jobs to continue
   running), set this to ``false``.  Default: ``true``.
 
+**ignore-dependencies**
+  In any kind of pipeline (dependent or independent), Zuul will
+  attempt to enqueue all dependencies ahead of the current change so
+  that they are tested together (independent pipelines report the
+  results of each change regardless of the results of changes ahead).
+  To ignore dependencies completely in an independent pipeline, set
+  this to ``true``.  This option is ignored by dependent pipelines.
+  The default is: ``false``.
+
 **success**
   Describes where Zuul should report to if all the jobs complete
   successfully.
@@ -924,7 +941,7 @@ return ``SUCCESS`` immediately.  This can be useful if you require
 that all changes be processed by a pipeline but a project has no jobs
 that can be run on it.
 
-.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml
+.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml
 
 Project Templates
 """""""""""""""""
@@ -978,6 +995,10 @@ also be added::
      check:
       - foobar-extra-special-job
 
+Individual jobs may optionally be added to pipelines (e.g. check,
+gate, et cetera) for a project, in addtion to those provided by
+templates.
+
 The order of the jobs listed in the project (which only affects the
 order of jobs listed on the report) will be the jobs from each
 template in the order listed, followed by any jobs individually listed
@@ -1034,10 +1055,19 @@ example, this would give you a list of Gerrit commands to reverify or
 recheck changes for the gate and check pipelines respectively::
 
   ./tools/zuul-changes.py --review-host=review.openstack.org \
-      http://zuul.openstack.org/ gate 'reverify no bug'
+      http://zuul.openstack.org/ gate 'reverify'
   ./tools/zuul-changes.py --review-host=review.openstack.org \
-      http://zuul.openstack.org/ check 'recheck no bug'
-
-If you send a SIGUSR2 to the zuul-server process, Zuul will dump a stack
-trace for each running thread into its debug log. This is useful for
-tracking down deadlock or otherwise slow threads.
+      http://zuul.openstack.org/ check 'recheck'
+
+If you send a SIGUSR2 to the zuul-server process, or the forked process
+that runs the Gearman daemon, Zuul will dump a stack trace for each
+running thread into its debug log. It is written under the log bucket
+``zuul.stack_dump``.  This is useful for tracking down deadlock or
+otherwise slow threads.
+
+When `yappi <https://code.google.com/p/yappi/>`_ (Yet Another Python
+Profiler) is available, additional functions' and threads' stats are
+emitted as well. The first SIGUSR2 will enable yappi, on the second
+SIGUSR2 it dumps the information collected, resets all yappi state and
+stops profiling. This is to minimize the impact of yappi on a running
+system.

etc/status/.gitignore

@@ -1,4 +1,4 @@
 public_html/jquery.min.js
-public_html/jquery-visibility.min.js
+public_html/jquery-visibility.js
 public_html/bootstrap
 public_html/jquery.graphite.js

etc/status/fetch-dependencies.sh

@@ -3,10 +3,10 @@ BASE_DIR=$(cd $(dirname $0); pwd)
 echo "Destination: $BASE_DIR/public_html"
 
 echo "Fetching jquery.min.js..."
-curl --silent http://code.jquery.com/jquery.min.js > $BASE_DIR/public_html/jquery.min.js
+curl -L --silent http://code.jquery.com/jquery.min.js > $BASE_DIR/public_html/jquery.min.js
 
 echo "Fetching jquery-visibility.min.js..."
-curl --silent https://raw.githubusercontent.com/mathiasbynens/jquery-visibility/master/jquery-visibility.js > $BASE_DIR/public_html/jquery-visibility.min.js
+curl -L --silent https://raw.githubusercontent.com/mathiasbynens/jquery-visibility/master/jquery-visibility.js > $BASE_DIR/public_html/jquery-visibility.js
 
 echo "Fetching jquery.graphite.js..."
 curl -L --silent https://github.com/prestontimmons/graphitejs/archive/master.zip > jquery-graphite.zip

etc/status/public_html/index.html

@@ -20,15 +20,14 @@
 <head>
     <title>Zuul Status</title>
     <link rel="stylesheet" href="bootstrap/css/bootstrap.min.css">
-    <link rel="stylesheet" href="bootstrap/css/bootstrap-responsive.min.css">
     <link rel="stylesheet" href="styles/zuul.css" />
 </head>
 <body>
 
     <div id="zuul_container"></div>
 
     <script src="jquery.min.js"></script>
-    <script src="jquery-visibility.min.js"></script>
+    <script src="jquery-visibility.js"></script>
     <script src="jquery.graphite.js"></script>
     <script src="jquery.zuul.js"></script>
     <script src="zuul.app.js"></script>

etc/status/public_html/jquery.zuul.js

@@ -50,7 +50,7 @@
         }, options);
 
         var collapsed_exceptions = [];
-        var current_filter = read_cookie('zuul_filter_string', current_filter);
+        var current_filter = read_cookie('zuul_filter_string', '');
         var $jq;
 
         var xhr,
@@ -111,6 +111,7 @@
                     );
                 }
 
+                $job_line.append($('<div style="clear: both"></div>'));
                 return $job_line;
             },
 
@@ -266,9 +267,21 @@
 
                 var $change_link = $('<small />');
                 if (change.url !== null) {
-                    $change_link.append(
-                        $('<a />').attr('href', change.url).text(change.id)
-                    );
+                    if (/^[0-9a-f]{40}$/.test(change.id)) {
+                        var change_id_short = change.id.slice(0, 7);
+                        $change_link.append(
+                            $('<a />').attr('href', change.url).append(
+                                $('<abbr />')
+                                    .attr('title', change.id)
+                                    .text(change_id_short)
+                            )
+                        );
+                    }
+                    else {
+                        $change_link.append(
+                            $('<a />').attr('href', change.url).text(change.id)
+                        );
+                    }
                 }
                 else {
                     $change_link.text(change_id);
@@ -357,6 +370,11 @@
                     icon_title = 'Waiting until closer to head of queue to' +
                         ' start jobs';
                 }
+                else if (change.live !== true) {
+                    // Grey icon
+                    icon_name = 'grey.png';
+                    icon_title = 'Dependent change independently tested';
+                }
                 else if (change.failing_reasons &&
                          change.failing_reasons.length > 0) {
                     var reason = change.failing_reasons.join(', ');

requirements.txt

@@ -5,17 +5,15 @@ PyYAML>=3.1.0
 Paste
 WebOb>=1.2.3,<1.3
 paramiko>=1.8.0
-GitPython>=0.3.2.RC1
+GitPython>=0.3.2.1
 lockfile>=0.8
 ordereddict
-python-daemon
+python-daemon>=2.0.4
 extras
 statsd>=1.0.0,<3.0
 voluptuous>=0.7
 gear>=0.5.4,<1.0.0
 apscheduler>=2.1.1,<3.0
-python-swiftclient>=1.6
-python-keystoneclient>=0.4.2
 PrettyTable>=0.6,<0.8
 babel>=1.0
 six>=1.6.0

test-requirements.txt

@@ -1,12 +1,13 @@
 hacking>=0.9.2,<0.10
 
 coverage>=3.6
-sphinx>=1.1.2,<1.2
+sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
 sphinxcontrib-blockdiag>=0.5.5
-docutils==0.9.1
 discover
 fixtures>=0.3.14
+python-keystoneclient>=0.4.2
 python-subunit
+python-swiftclient>=1.6
 testrepository>=0.0.17
 testtools>=0.9.32
 sphinxcontrib-programoutput