Merge pull request #1128 from aws/develop

Merge Release 2.4.0

Author: lukeseawalker

.github/no-response.yml

@@ -1,7 +1,7 @@
 # Configuration for probot-no-response - https://github.com/probot/no-response
 
 # Number of days of inactivity before an Issue is closed for lack of response
-daysUntilClose: 14
+daysUntilClose: 7
 # Label requiring a response
 responseRequiredLabel: closing-soon-if-no-response 
 # Comment to post when closing an Issue for lack of response. Set to `false` to disable

.gitignore

@@ -13,3 +13,4 @@ build/
 .coverage
 assets/
 report.html
+tests_outputs/

CHANGELOG.rst

@@ -2,6 +2,72 @@
 CHANGELOG
 =========
 
+2.4.0
+=====
+
+**ENHANCEMENTS**
+
+* Add support for EFA on Centos 7, Amazon Linux and Ubuntu 1604
+* Add support for Ubuntu in China region ``cn-northwest-1``
+
+* SGE:
+
+  * process nodes added to or removed from the cluster in batches in order to speed up cluster scaling.
+  * scale up only if required slots/nodes can be satisfied
+  * scale down if pending jobs have unsatisfiable CPU/nodes requirements
+  * add support for jobs in hold/suspended state (this includes job dependencies)
+  * automatically terminate and replace faulty or unresponsive compute nodes
+  * add retries in case of failures when adding or removing nodes
+  * configure scheduler to handle rescheduling and cancellation of jobs running on failing or terminated nodes
+
+* Slurm:
+
+  * scale up only if required slots/nodes can be satisfied
+  * scale down if pending jobs have unsatisfiable CPU/nodes requirements
+  * automatically terminate and replace faulty or unresponsive compute nodes
+  * decrease SlurmdTimeout to 120 seconds to speed up replacement of faulty nodes
+
+* Automatically replace compute instances that fail initialization and dump logs to shared home directory.
+* Dynamically fetch compute instance type and cluster size in order to support updates in scaling daemons
+* Always use full master FQDN when mounting NFS on compute nodes. This solves some issues occurring with some networking
+  setups and custom DNS configurations
+* List the version and status during ``pcluster list``
+* Remove double quoting of the post_install args
+* ``awsbsub``: use override option to set the number of nodes rather than creating multiple JobDefinitions
+* Add support for AWS_PCLUSTER_CONFIG_FILE env variable to specify pcluster config file
+
+**CHANGES**
+
+* Update openmpi library to version 3.1.4 on Centos 7, Amazon Linux and Ubuntu 1604. This also changes the default
+  openmpi path to ``/opt/amazon/efa/bin/`` and the openmpi module name to ``openmpi/3.1.4``
+* Set soft and hard ulimit on open files to 10000 for all supported OSs
+* For a better security posture, we're removing AWS credentials from the ``parallelcluster`` config file
+  Credentials can be now setup following the canonical procedure used for the aws cli
+* When using FSx or EFS do not enforce in sanity check that the compute security group is open to 0.0.0.0/0
+* When updating an existing cluster, the same template version is now used, no matter the pcluster cli version
+* SQS messages that fail to be processed in ``sqswatcher`` are now re-queued only 3 times and not forever
+* Reset ``nodewatcher`` idletime to 0 when the host becomes essential for the cluster (because of min size of ASG or
+  because there are pending jobs in the scheduler queue)
+* SGE: a node is considered as busy when in one of the following states "u", "C", "s", "d", "D", "E", "P", "o".
+  This allows a quick replacement of the node without waiting for the ``nodewatcher`` to terminate it.
+* Do not update DynamoDB table on cluster updates in order to avoid hitting strict API limits (1 update per day).
+
+**BUG FIXES**
+
+* Fix issue that was preventing Torque from being used on Centos 7
+* Start node daemons at the end of instance initialization. The time spent for post-install script and node
+  initialization is not counted as part of node idletime anymore.
+* Fix issue which was causing an additional and invalid EBS mount point to be added in case of multiple EBS
+* Install Slurm libpmpi/libpmpi2 that is distributed in a separate package since Slurm 17
+* ``pcluster ssh`` command now works for clusters with ``use_public_ips = false``
+* Slurm: add "BeginTime", "NodeDown", "Priority" and "ReqNodeNotAvail" to the pending reasons that trigger
+  a cluster scaling
+* Add a timeout on remote commands execution so that the daemons are not stuck if the compute node is unresponsive
+* Fix an edge case that was causing the ``nodewatcher`` to hang forever in case the node had become essential to the
+  cluster during a call to ``self_terminate``.
+* Fix ``pcluster start/stop`` commands when used with an ``awsbatch`` cluster
+
+
 2.3.1
 =====
 

CONTRIBUTING.md

@@ -56,6 +56,6 @@ If you discover a potential security issue in this project we ask that you notif
 
 ## Licensing
 
-See the [LICENSE](https://github.com/aws/aws-parallelcluster/blob/develop/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+See the [LICENSE](https://github.com/aws/aws-parallelcluster/blob/develop/LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
 
 We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

README.rst

@@ -41,8 +41,6 @@ Then, run pcluster configure:
 
   $ pcluster configure
   Cluster Template [default]:
-  AWS Access Key ID []:
-  AWS Secret Access Key ID []:
   Acceptable Values for AWS Region ID:
       ap-south-1
       ...
@@ -105,23 +103,8 @@ HPC forum which may be helpful:https://forums.aws.amazon.com/forum.jspa?forumID=
 Changes
 -------
 
-CfnCluster 1.6 IAM Change
-=========================
-Between CfnCluster 1.5.4 and 1.6.0 we made a change to the CfnClusterInstancePolicy that adds “s3:GetObject” permissions
-on objects in <REGION>-cfncluster bucket, "autoscaling:SetDesiredCapacity", "autoscaling:DescribeTags" permissions and
-"cloudformation:DescribeStacks" permissions on <REGION>:<ACCOUNT_ID>:stack/cfncluster-*.
+CfnCluster to AWS ParallelCluster
+=================================
+In Version `2.0.0`, we changed the name of CfnCluster to AWS ParallelCluster. With that name change we released several new features, which you can read about in the `Change Log`_.
 
-If you’re using a custom policy (e.g. you specify "ec2_iam_role" in your config) be sure it includes this new permission. See https://aws-parallelcluster.readthedocs.io/en/latest/iam.html
-
-CfnCluster 1.5 IAM Change
-=========================
-Between CfnCluster 1.4.2 and 1.5.0 we made a change to the CfnClusterInstancePolicy that adds “ec2:DescribeVolumes” permissions. If you’re using a custom policy (e.g. you specify "ec2_iam_role" in your config) be sure it includes this new permission. See https://aws-parallelcluster.readthedocs.io/en/latest/iam.html
-
-CfnCluster 1.2 and Earlier
-==========================
-
-For various security (on our side) and maintenance reasons, CfnCluster
-1.2 and earlier have been deprecated.  AWS-side resources necessary to
-create a cluster with CfnCluster 1.2 or earlier are no longer
-available.  Existing clusters will continue to operate, but new
-clusters can not be created.
+.. _`Change Log`: https://github.com/aws/aws-parallelcluster/blob/develop/CHANGELOG.rst#200

amis.txt

@@ -1,100 +1,102 @@
 # alinux
-ap-northeast-1: ami-0af8c1a29f58c3b91
-ap-northeast-2: ami-036c289fda8701f9d
-ap-northeast-3: ami-000902aa3082732ce
-ap-south-1: ami-00ff6216daa4b0a69
-ap-southeast-1: ami-03b015a13daa9ff8d
-ap-southeast-2: ami-0c2528255cc7c4cec
-ca-central-1: ami-05bad5df22b9502e5
-cn-north-1: ami-0227bedfc6798cba1
-cn-northwest-1: ami-08143603c5f390f20
-eu-central-1: ami-003262ea853b26050
-eu-north-1: ami-06cac8aed0729f14c
-eu-west-1: ami-0691d6d6d4d209e09
-eu-west-2: ami-0d241a5c57ee3421d
-eu-west-3: ami-0e59dd1d2794a857c
-sa-east-1: ami-07b044055a13cf93e
-us-east-1: ami-0f8b01b1377483305
-us-east-2: ami-049afa5b53a7880d8
-us-gov-east-1: ami-02ee5c66a10526bd1
-us-gov-west-1: ami-7da7d01c
-us-west-1: ami-02c87842ea944292e
-us-west-2: ami-09b457d5cba24514a
+ap-northeast-1: ami-0dcc18768374b4441
+ap-northeast-2: ami-022e7c66ccb807c9f
+ap-northeast-3: ami-04402be7b85999df8
+ap-south-1: ami-0a14b1f0e7427a4bb
+ap-southeast-1: ami-02079735c20c1ac4e
+ap-southeast-2: ami-0c65952cdec26ae39
+ca-central-1: ami-01f28f8381746746f
+cn-north-1: ami-0da67c26ce2e8d111
+cn-northwest-1: ami-03dc8f759de9de690
+eu-central-1: ami-0ff6d2a86b9199e82
+eu-north-1: ami-0cb08caa10d113ed7
+eu-west-1: ami-0b5c32b12b9c340d0
+eu-west-2: ami-0c218c2aaa7185f03
+eu-west-3: ami-011e0eee21d52f23e
+sa-east-1: ami-0d154ae55458941fd
+us-east-1: ami-0d130bdfab2037f8a
+us-east-2: ami-00d2a10466c577ac7
+us-gov-east-1: ami-0f5003922daf22962
+us-gov-west-1: ami-ba83fbdb
+us-west-1: ami-0b6f7961ee845966e
+us-west-2: ami-0d611d90619419e93
 # centos6
-ap-northeast-1: ami-0476984f547d1f4f2
-ap-northeast-2: ami-06ecb1e81881cd450
-ap-northeast-3: ami-04d195b55ddf56228
-ap-south-1: ami-0b1abd2bf8810487c
-ap-southeast-1: ami-0576b4b2db8272abf
-ap-southeast-2: ami-09a18baab0a142123
-ca-central-1: ami-0aa03a3f1b737c651
-eu-central-1: ami-092bd9c46746d940b
-eu-north-1: ami-07b83433077d8345b
-eu-west-1: ami-09880c7e25df69af8
-eu-west-2: ami-0eba961d9f30431b2
-eu-west-3: ami-0d0b243ac76765544
-sa-east-1: ami-0dfdc6ab8bf7935ea
-us-east-1: ami-00f71e3be938f3077
-us-east-2: ami-0b29637d31cf774aa
-us-west-1: ami-08dc392067bcf9807
-us-west-2: ami-0fa309858f6ce66ee
+ap-northeast-1: ami-086781b933db101a5
+ap-northeast-2: ami-07d646c87d889d816
+ap-northeast-3: ami-082ece6e5fe8f6fd1
+ap-south-1: ami-02389426198baf430
+ap-southeast-1: ami-02105387481bd0ad0
+ap-southeast-2: ami-0050fad9761b3957c
+ca-central-1: ami-0e70755a47200df23
+eu-central-1: ami-03979ebb9cfee2ccc
+eu-north-1: ami-085a9ecbf9f64f65b
+eu-west-1: ami-070ba56e38a744df5
+eu-west-2: ami-08553013e6e986028
+eu-west-3: ami-0afff5bc147c847e0
+sa-east-1: ami-0635a9bdc378fe67f
+us-east-1: ami-091f37e900368fe1a
+us-east-2: ami-055404b3df678da86
+us-west-1: ami-0e438402399c457d7
+us-west-2: ami-0651b7e7cfde4b3a0
 # centos7
-ap-northeast-1: ami-0f13f45e966236e46
-ap-northeast-2: ami-016c726d8902d133c
-ap-northeast-3: ami-037c3a13cd142c8f8
-ap-south-1: ami-06b7212503b9d9637
-ap-southeast-1: ami-0c39937e9ae643ecd
-ap-southeast-2: ami-0164dbfb6b7b938f5
-ca-central-1: ami-0ee7cb4d2673e78de
-eu-central-1: ami-0bcced571d9cc0142
-eu-north-1: ami-00255a59ce6bd8147
-eu-west-1: ami-00c07933e0ea22f7d
-eu-west-2: ami-09aa34259643c50eb
-eu-west-3: ami-04ce6f74e1070a795
-sa-east-1: ami-0a625e9dcf563db57
-us-east-1: ami-0658a809b3e89b0c9
-us-east-2: ami-07cef254f8886ea4e
-us-west-1: ami-0454b933360a077e4
-us-west-2: ami-03b7e311ae2f4aacb
+ap-northeast-1: ami-09bae677f8f58842d
+ap-northeast-2: ami-0eeb6c96d0e6c2d90
+ap-northeast-3: ami-084c0dbc04f722758
+ap-south-1: ami-031f8f67a53de53fe
+ap-southeast-1: ami-041ca5c2f5b748966
+ap-southeast-2: ami-06c7f5584ecfcac3a
+ca-central-1: ami-0afc2ea67b3963398
+eu-central-1: ami-0205eaef48a9fc97a
+eu-north-1: ami-0420576e18a5fcb7c
+eu-west-1: ami-0f67868de5be7b0b3
+eu-west-2: ami-057fa1a5314e3c414
+eu-west-3: ami-05b2808c2dc4fb82c
+sa-east-1: ami-0da1262e3c5d9af72
+us-east-1: ami-031eb9c5390c0f8f6
+us-east-2: ami-0050bd80a1cecfe37
+us-west-1: ami-09bd008b253048b80
+us-west-2: ami-003da28849bc413f5
 # ubuntu1404
-ap-northeast-1: ami-0ce1c5516c087ef8d
-ap-northeast-2: ami-0744c53e9582abcd4
-ap-northeast-3: ami-0d0faa548bcca5fac
-ap-south-1: ami-00721e9f7f8235dba
-ap-southeast-1: ami-03df9d0a89a448c63
-ap-southeast-2: ami-06116e2159f6ba6bf
-ca-central-1: ami-0d180013cf3d07fc9
-cn-north-1: ami-0ef85bbc4ba66c301
-eu-central-1: ami-04b116ae9a44c861f
-eu-north-1: ami-0de1c666987bbdb1f
-eu-west-1: ami-01b114f6a268d6a42
-eu-west-2: ami-0f9ad3c001b80325a
-eu-west-3: ami-0f921986737ab8306
-sa-east-1: ami-0d1d30ad051235185
-us-east-1: ami-0422aa8ec2e452870
-us-east-2: ami-02447e477105886bd
-us-gov-east-1: ami-03538e53996b83762
-us-gov-west-1: ami-90a2d5f1
-us-west-1: ami-0f4a99f972b9b4882
-us-west-2: ami-04caeb57df33aba89
+ap-northeast-1: ami-0939e3e1030d4f7d2
+ap-northeast-2: ami-0481c6b023e2328b4
+ap-northeast-3: ami-0a535e1d0bb7bc502
+ap-south-1: ami-000e99acc047832ae
+ap-southeast-1: ami-09ca9a6a8fee71ba5
+ap-southeast-2: ami-09646cc49a932a37e
+ca-central-1: ami-06ac5db73837bc364
+cn-north-1: ami-07e16a5709c99f963
+cn-northwest-1: ami-05348579489ba3673
+eu-central-1: ami-0032889c720d364dc
+eu-north-1: ami-0976908358f0bfa01
+eu-west-1: ami-0f5c65a609ad3afb4
+eu-west-2: ami-08c2d96c2805037e7
+eu-west-3: ami-0f6cd6ac9be8f2b32
+sa-east-1: ami-0d0da341da4802af9
+us-east-1: ami-017bfe181606779d8
+us-east-2: ami-043eb896e1bb2b948
+us-gov-east-1: ami-060ced48ab370aadf
+us-gov-west-1: ami-32f98153
+us-west-1: ami-0d48f8a9d5735efde
+us-west-2: ami-0169da6ccb6347f50
 # ubuntu1604
-ap-northeast-1: ami-041f6050eff86f024
-ap-northeast-2: ami-0df4c1dafbfee5031
-ap-northeast-3: ami-08d3ef362e1d06e56
-ap-south-1: ami-0ef148f6ae69767d7
-ap-southeast-1: ami-0b63a13236ce5b8d9
-ap-southeast-2: ami-0f5a3072f23556b07
-ca-central-1: ami-0c88262f6fd2738fc
-cn-north-1: ami-017ea2a40c48f9af4
-eu-central-1: ami-06a21b6e0815065a4
-eu-north-1: ami-0418320f06192d788
-eu-west-1: ami-0809bc00666e41cfa
-eu-west-2: ami-04d8578267aaa2ac4
-eu-west-3: ami-02de781189ccb9f92
-sa-east-1: ami-088d6a838e8dc6b11
-us-east-1: ami-0a8c4ea1bd1ff7651
-us-east-2: ami-04d5c390495e0509f
-us-gov-east-1: ami-0bfb76fbbbb68030d
-us-gov-west-1: ami-eeaed98f
-us-west-1: ami-0a33d79d5f920cc2c
-us-west-2: ami-00050b3048393bc12
+ap-northeast-1: ami-06b328a6ee03ccdf4
+ap-northeast-2: ami-0179e2707f709f813
+ap-northeast-3: ami-0c9b72bae5efc9f61
+ap-south-1: ami-0f21d1eb3339ebd6a
+ap-southeast-1: ami-01899e9a659eb2267
+ap-southeast-2: ami-049c81a79d55b2c8a
+ca-central-1: ami-0b8928a1f643684eb
+cn-north-1: ami-0ae967dc97d5eb57a
+cn-northwest-1: ami-0ba0b1ed49ce7b1b1
+eu-central-1: ami-002422c65a5bb1af8
+eu-north-1: ami-0d3c7ce730c73ab00
+eu-west-1: ami-00328873639859269
+eu-west-2: ami-0c1de72c6acf4b187
+eu-west-3: ami-090d577bb6d08e95b
+sa-east-1: ami-08df8912b098a3f42
+us-east-1: ami-08e1d33a6a64499de
+us-east-2: ami-0219fdb6f47395d88
+us-gov-east-1: ami-0af2c8e5bf3c334b0
+us-gov-west-1: ami-7b85fd1a
+us-west-1: ami-066818f6a6be06fb5
+us-west-2: ami-07122cb5a96b7fee9