Skip to content

Commit 9e76b10

Browse files
lukeseawalkerlukeseawalker
authored
Merge Release 2.2.1
Merge Release 2.2.1
2 parents 1e0fbc6 + 77a2abb commit 9e76b10

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+5868
-1611
lines changed

‎cli/.isort.cfg ‎.isort.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[settings]
22
line_length=120
33
known_future_library=future
4-
known_third_party=boto3,botocore,awscli,tabulate,argparse,configparser,pytest
4+
known_third_party=boto3,botocore,awscli,tabulate,argparse,configparser,pytest,pytest,pytest-datadir,pytest-html,pytest-rerunfailures,pytest-xdist,argparse,retrying,junitparser,Jinja2
55
# 3 - Vertical Hanging Indent
66
# from third_party import (
77
# lib1,

‎CHANGELOG.rst

+67-17
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,65 @@
22
CHANGELOG
33
=========
44

5+
2.2.1
6+
=====
7+
8+
**ENHANCEMENTS**
9+
10+
* Add support for FSx Lustre in Centos 7. In case of custom AMI, FSx Lustre is
11+
only supported with Centos 7.5 and Centos 7.6.
12+
* Check AWS EC2 instance account limits before starting cluster creation
13+
* Allow users to force job deletion with ``SGE`` scheduler
14+
15+
**CHANGES**
16+
17+
* Set default value to ``compute`` for ``placement_group`` option
18+
* ``pcluster ssh``: use private IP when the public one is not available
19+
* ``pcluster ssh``: now works also when stack is not completed as long as the master IP is available
20+
* Remove unused dependency on ``awscli`` from ParallelCluster package
21+
22+
**BUG FIXES**
23+
24+
* ``awsbsub``: fix file upload with absolute path
25+
* ``pcluster ssh``: fix issue that was preventing the command from working correctly when stack status is
26+
``UPDATE_ROLLBACK_COMPLETE``
27+
* Fix block device conversion to correctly attach EBS nvme volumes
28+
* Wait for Torque scheduler initialization before completing master node setup
29+
* ``pcluster version``: now works also when no ParallelCluster config is present
30+
* Improve ``nodewatcher`` daemon logic to detect if a SGE compute node has running jobs
31+
32+
**DOCS**
33+
34+
* Add documentation on how to use FSx Lustre
35+
* Add tutorial for encrypted EBS with a Custom KMS Key
36+
* Add ``ebs_kms_key_id`` to Configuration section
37+
38+
**TESTING**
39+
40+
* Define a new framework to write and run ParallelCluster integration tests
41+
* Improve scaling integration tests to detect over-scaling
42+
* Implement integration tests for awsbatch scheduler
43+
* Implement integration tests for FSx Lustre file system
44+
545
2.1.1
646
=====
7-
* China regions `cn-north-1` and `cn-northwest-1`
47+
* Add China regions `cn-north-1` and `cn-northwest-1`
848

949
2.1.0
1050
=====
11-
* RAID support
12-
* EFS support
13-
* AWS Batch Multinode Parallel Support
51+
* Add configuration for RAID 0 and 1 volumes
52+
* Add Elastic File System (EFS) support
53+
* Add AWS Batch Multinode Parallel jobs support
54+
* Add support for Stockholm region (`eu-north-1`)
55+
* Add `--env` and `--env-blacklist` options to the `awsbsub` command to export environment variables
56+
in the job environment
57+
* Add `--input-file` option to the `awsbsub` command to stage-in input files from the client
58+
* Add new `PCLUSTER_JOB_S3_URL` variable to the job execution environment pointing to the S3 URL used
59+
for job data stage-in/out
60+
* Add S3 URL for job data staging to the `awsbstat -d` output
61+
* Add `--working-dir` and `--parent-working-dir` options to the `awsbsub` command to specify
62+
the working-directory or the parent working directory for the job
63+
* Add CPUs and Memory information to the `awsbhosts -d` command
1464

1565
2.0.2
1666
=====
@@ -56,41 +106,41 @@ CHANGELOG
56106
1.5.2
57107
=====
58108
* feature:``cfncluster``: Added ClusterUser as a stack output. This makes it easier to get the username of the head node.
59-
* feature:``cfncluster``: Added `cfncluster ssh cluster_name`, this allows you to easily ssh into your clusters. It allows arbitrary command execution and extra ssh flags to be provided after the command. See https://aws-parallelcluster.readthedocs.io/en/latest/commands.html#ssh
60-
* change:``cfncluster``: Moved global cli flags to the command specific flags. For example `cfncluster --region us-east-1 create` now becomes `cfncluster create --region us-east-1`
109+
* feature:``cfncluster``: Added `cfncluster ssh cluster_name`, this allows you to easily ssh into your clusters.
110+
It allows arbitrary command execution and extra ssh flags to be provided after the command.
111+
See https://aws-parallelcluster.readthedocs.io/en/latest/commands.html#ssh
112+
* change:``cfncluster``: Moved global cli flags to the command specific flags.
113+
For example `cfncluster --region us-east-1 create` now becomes `cfncluster create --region us-east-1`
61114
* bugfix:``cfncluster-cookbook``: Fix bug that prevented c5d/m5d instances from working
62115
* bugfix:``cfncluster-cookbook``: Set CPU as a consumable resource in slurm
63116
* bugfix:``cfncluster-node``: Fixed Slurm behavior to add CPU slots so multiple jobs can be scheduled on a single node
64117

65118
1.5.1
66119
=====
67120
* change:``cfncluster``: Added "ec2:DescribeVolumes" permissions to
68-
CfnClusterInstancePolicy
121+
CfnClusterInstancePolicy
69122
* change:``cfncluster``: Removed YAML CloudFormation template, it can be
70-
generated by the https://github.com/awslabs/aws-cfn-template-flip tool
123+
generated by the https://github.com/awslabs/aws-cfn-template-flip tool
71124

72125
* updates:``cfncluster``: Add support for eu-west-3 region
73126

74127
* feature:``cfncluster-cookbook``: Added parameter to specify custom
75-
cfncluster-node package
128+
cfncluster-node package
76129

77130
* bugfix:``cfncluster``: Fix --template-url command line parameter
78131
* bugfix:``cfncluster-cookbook``: Poll on EBS Volume attachment status
79-
* bugfix:``cfncluster-cookbook``: Fixed SLURM cron job to publish pending
80-
metric
81-
* bugfix:``cfncluster-node``: Fixed Torque behaviour when scaling up from an
82-
empty cluster
132+
* bugfix:``cfncluster-cookbook``: Fixed SLURM cron job to publish pending metric
133+
* bugfix:``cfncluster-node``: Fixed Torque behaviour when scaling up from an empty cluster
83134

84135

85136
1.4.2
86137
=====
87138
* bugfix:``cfncluster``: Fix crash when base directory for config file
88139
does not exist
89140
* bugfix:``cfncluster``: Removed extraneous logging message at
90-
cfncluster invocation, re-enabled logging in
91-
~/.cfncluster/cfncluster-cli.log
141+
cfncluster invocation, re-enabled logging in ~/.cfncluster/cfncluster-cli.log
92142
* bugfix: ``cfncluster-node``: Fix scaling issues with CentOS 6 clusters caused
93-
by incompatible dependencies.
143+
by incompatible dependencies.
94144
* updates:``ami``: Update all base AMIs to latest patch levels
95145
* updates:``cfncluster-cookbook``: Updated to cfncluster-cookbook-1.4.1
96146

@@ -161,7 +211,7 @@ by incompatible dependencies.
161211

162212
1.0.0
163213
=====
164-
Offiical release of the CfnCluster 1.x CLI, templates and AMIs. Available in all regions except BJS, with
214+
Official release of the CfnCluster 1.x CLI, templates and AMIs. Available in all regions except BJS, with
165215
support for Amazon Linux, CentOS 6 & 7 and Ubuntu 14.04 LTS. All AMIs are built via packer from the CfnCluster
166216
Cookbook project (https://github.com/aws/aws-parallelcluster-cookbook).
167217

‎amis.txt

+95-95
Original file line numberDiff line numberDiff line change
@@ -1,100 +1,100 @@
11
# alinux
2-
ap-northeast-1: ami-0ac1d0c35dc3b2a97
3-
ap-northeast-2: ami-0dddb89f66485f828
4-
ap-northeast-3: ami-0dbb09bfada65298f
5-
ap-south-1: ami-0ed1bb8f2ab0edffb
6-
ap-southeast-1: ami-058bcd8377aba9bef
7-
ap-southeast-2: ami-012557ce9426ef1a0
8-
ca-central-1: ami-059dda8ee9af7c20c
9-
cn-north-1: ami-00237da8e056b0936
10-
cn-northwest-1: ami-0974fc483e449f5ee
11-
eu-central-1: ami-09cff6787920e967c
12-
eu-north-1: ami-086f4f382fb1119f0
13-
eu-west-1: ami-080a7d5c75253bd1b
14-
eu-west-2: ami-0b284f1028a743865
15-
eu-west-3: ami-099197e40d0c5de6e
16-
sa-east-1: ami-02e542e4935ff9647
17-
us-east-1: ami-0cd2dd3198972a68c
18-
us-east-2: ami-057e70f0fbb007ab6
19-
us-gov-east-1: ami-022f96b137a63c9ff
20-
us-gov-west-1: ami-fefb989f
21-
us-west-1: ami-09686d4090e35a702
22-
us-west-2: ami-0c588cdc9e91b0db3
2+
ap-northeast-1: ami-01e7e307e734daa9b
3+
ap-northeast-2: ami-096da9252851971b7
4+
ap-northeast-3: ami-0fde1cce0915721ec
5+
ap-south-1: ami-0856113449f34cfde
6+
ap-southeast-1: ami-019762c344b80574a
7+
ap-southeast-2: ami-09ddff457fb3815ea
8+
ca-central-1: ami-0ef3d34adf231688a
9+
cn-north-1: ami-053a5a11a4ac83842
10+
cn-northwest-1: ami-0c5a99564e44467e7
11+
eu-central-1: ami-01d7252afc45b0d8b
12+
eu-north-1: ami-02224ba7786413561
13+
eu-west-1: ami-0a32ae196621ce1cd
14+
eu-west-2: ami-0e8d810df84d61f8b
15+
eu-west-3: ami-09a1847f7683c6351
16+
sa-east-1: ami-03c4f6e67dcea925d
17+
us-east-1: ami-096b5898281e68ea3
18+
us-east-2: ami-0c8b41c511db3c17c
19+
us-gov-east-1: ami-08cff3f9ef830bdfb
20+
us-gov-west-1: ami-afa0c8ce
21+
us-west-1: ami-08e9806b160f9aa44
22+
us-west-2: ami-04eba5b9de0d94ec6
2323
# centos6
24-
ap-northeast-1: ami-003cfe6266cadd576
25-
ap-northeast-2: ami-032b3f2a4f1ac91a0
26-
ap-northeast-3: ami-0c4de7aece7b2db33
27-
ap-south-1: ami-0581583c3d7507d9e
28-
ap-southeast-1: ami-0d00309b80b772532
29-
ap-southeast-2: ami-00ecf7e455945e8bc
30-
ca-central-1: ami-0416d41ed6dbc0bd7
31-
eu-central-1: ami-0b1f52047bb2b7f83
32-
eu-north-1: ami-04fc976e8108996e6
33-
eu-west-1: ami-073f1f5db6cfdd3d1
34-
eu-west-2: ami-0ace56d1d9c1aa466
35-
eu-west-3: ami-0d377bccde07c887d
36-
sa-east-1: ami-0b1c8f6aad337d5b6
37-
us-east-1: ami-0919d912e0e33d247
38-
us-east-2: ami-0bbd43b2b8991cdae
39-
us-west-1: ami-0ac1cf1e68288fa36
40-
us-west-2: ami-08b28682da5721f5b
24+
ap-northeast-1: ami-09c0d7b7eba653962
25+
ap-northeast-2: ami-04bb0577f1425f61f
26+
ap-northeast-3: ami-092c70a47ecb4f730
27+
ap-south-1: ami-0b6b4a5c5952ba214
28+
ap-southeast-1: ami-0d8fee73833faed87
29+
ap-southeast-2: ami-03e0e45d59cf130b3
30+
ca-central-1: ami-0f5cabbce86422cc0
31+
eu-central-1: ami-0c47523135cb69662
32+
eu-north-1: ami-064af23ad3c8bac88
33+
eu-west-1: ami-048edb0ed06f91ef8
34+
eu-west-2: ami-098b95acdef72fd6c
35+
eu-west-3: ami-0e25f1a2ef5f8d237
36+
sa-east-1: ami-07bcda88df87d3db2
37+
us-east-1: ami-0fd0ae64eb509cc23
38+
us-east-2: ami-09c42f6a076482eeb
39+
us-west-1: ami-060bcf74fb6278986
40+
us-west-2: ami-009c08111f57a2d4c
4141
# centos7
42-
ap-northeast-1: ami-0c0049e4eeb0ef1ac
43-
ap-northeast-2: ami-00f8f46a043a04530
44-
ap-northeast-3: ami-0043e5e12872a00d4
45-
ap-south-1: ami-03dd063b05c3082f1
46-
ap-southeast-1: ami-00fcdc55bd29f691e
47-
ap-southeast-2: ami-0baa1787fd7a71950
48-
ca-central-1: ami-0c2a8b54dfd0c405f
49-
eu-central-1: ami-00b3f34240b6021dd
50-
eu-north-1: ami-07ac74edc9e96b343
51-
eu-west-1: ami-0879d97613ba8075a
52-
eu-west-2: ami-06aa0ca6f28c29fdb
53-
eu-west-3: ami-0b28dcaf2b4d00d87
54-
sa-east-1: ami-00a570d9536621660
55-
us-east-1: ami-09d092c1b054832df
56-
us-east-2: ami-0e0cda1cdec7fcecf
57-
us-west-1: ami-084c065e503e74449
58-
us-west-2: ami-070345cb145f2d2f0
42+
ap-northeast-1: ami-0307ad022fdd9bd50
43+
ap-northeast-2: ami-01610b2d147974908
44+
ap-northeast-3: ami-04dd95194c8990d40
45+
ap-south-1: ami-0d848603d9bf30b76
46+
ap-southeast-1: ami-084b71a47f7c857b7
47+
ap-southeast-2: ami-0b06de8c266942eff
48+
ca-central-1: ami-0a2b442fa56fe5db0
49+
eu-central-1: ami-089c27327ed56f33c
50+
eu-north-1: ami-016fe6e29bff94e38
51+
eu-west-1: ami-00e9df6764697605f
52+
eu-west-2: ami-00884b56887c9585a
53+
eu-west-3: ami-0252465d6dff43cfb
54+
sa-east-1: ami-02f4e6da1e0de2347
55+
us-east-1: ami-08b43bfe7a0d16eeb
56+
us-east-2: ami-0c71e12e95cc937ef
57+
us-west-1: ami-0f925f166f349eaa4
58+
us-west-2: ami-05f6a646767bf0c87
5959
# ubuntu1404
60-
ap-northeast-1: ami-053a84278e34b7a59
61-
ap-northeast-2: ami-03d314c5ac10beb53
62-
ap-northeast-3: ami-070f9069b20a797ec
63-
ap-south-1: ami-0641d7ca885530589
64-
ap-southeast-1: ami-0e3de99412375e882
65-
ap-southeast-2: ami-09eae4580e8fc835a
66-
ca-central-1: ami-08aeb7a57f73b58ab
67-
cn-north-1: ami-00f2cae5406fb3fce
68-
eu-central-1: ami-0b24a435216670b4a
69-
eu-north-1: ami-0921b515f8ed512c3
70-
eu-west-1: ami-076fbdec21cd5c940
71-
eu-west-2: ami-0fb01b5b56bc27509
72-
eu-west-3: ami-0ea8b79c622d0a4a2
73-
sa-east-1: ami-00cf3910c959e9fd5
74-
us-east-1: ami-095016c5ff0ab7ae6
75-
us-east-2: ami-02a21f90af8a453f0
76-
us-gov-east-1: ami-09274e2587c4154f2
77-
us-gov-west-1: ami-6cf0930d
78-
us-west-1: ami-099ebeb456dbf0646
79-
us-west-2: ami-05ddc7ec0edb9069f
60+
ap-northeast-1: ami-0141f9239dd88f7eb
61+
ap-northeast-2: ami-069d8e1e9b74ee48d
62+
ap-northeast-3: ami-07853be3b108fb14c
63+
ap-south-1: ami-02744b942f4094210
64+
ap-southeast-1: ami-070fef5acc9fd3de2
65+
ap-southeast-2: ami-097c066ff342e12fa
66+
ca-central-1: ami-075be3409c2c2738d
67+
cn-north-1: ami-0e90250aa31ac4a12
68+
eu-central-1: ami-093d2529752e5e2f2
69+
eu-north-1: ami-01ea61e8b26299de1
70+
eu-west-1: ami-0d1cb938a6d6b693c
71+
eu-west-2: ami-0487ce94890b5869b
72+
eu-west-3: ami-01a321e226dbf081a
73+
sa-east-1: ami-02366044e8d7372fa
74+
us-east-1: ami-00f8fb8ed06007c1f
75+
us-east-2: ami-0e23f662335e169a6
76+
us-gov-east-1: ami-0a34c5de0403642d2
77+
us-gov-west-1: ami-e4a3cb85
78+
us-west-1: ami-021201edbaab31f29
79+
us-west-2: ami-085bb9cfafd2e3b3a
8080
# ubuntu1604
81-
ap-northeast-1: ami-026669cfcef23b3de
82-
ap-northeast-2: ami-048214d1413ed8462
83-
ap-northeast-3: ami-038900edb73cb9496
84-
ap-south-1: ami-0cc424ec58256ea88
85-
ap-southeast-1: ami-059ba95190db36590
86-
ap-southeast-2: ami-04df2433ab61d3f37
87-
ca-central-1: ami-039a1b0ada060b5ce
88-
cn-north-1: ami-072046713a0458796
89-
eu-central-1: ami-0d816068d1164f4d2
90-
eu-north-1: ami-046c32486a9abf742
91-
eu-west-1: ami-0f641e63ebaf647b1
92-
eu-west-2: ami-067c1c0157477c166
93-
eu-west-3: ami-0102caf8c6ec0768a
94-
sa-east-1: ami-0ef5c70aec338bcfb
95-
us-east-1: ami-098c8e582ca818cff
96-
us-east-2: ami-055279b0b09d12a71
97-
us-gov-east-1: ami-0b664ce8c427b77eb
98-
us-gov-west-1: ami-5ef6953f
99-
us-west-1: ami-0db5e85c0b1ce2c20
100-
us-west-2: ami-02393fa61ac61547a
81+
ap-northeast-1: ami-00bacec2848062b6b
82+
ap-northeast-2: ami-0a6edfea96ed6c9db
83+
ap-northeast-3: ami-09eeb02948489e793
84+
ap-south-1: ami-0dcc47340ec3a0e45
85+
ap-southeast-1: ami-0771cf73e73259040
86+
ap-southeast-2: ami-0252c6d83fa35183c
87+
ca-central-1: ami-090b3ee62a0b5412b
88+
cn-north-1: ami-0b89a3058ea57b76f
89+
eu-central-1: ami-0c7ccc7ec89bd0d75
90+
eu-north-1: ami-0675d81a659cb530e
91+
eu-west-1: ami-010af7d7788dac778
92+
eu-west-2: ami-01290e9a5298ae134
93+
eu-west-3: ami-0996ed0efb2b8cc4a
94+
sa-east-1: ami-0a3ef670136cc81ee
95+
us-east-1: ami-05f79ab77fc13e20a
96+
us-east-2: ami-08a675767416c627a
97+
us-gov-east-1: ami-08b15aaa28124fac1
98+
us-gov-west-1: ami-eba3cb8a
99+
us-west-1: ami-059b8cd1b0041071a
100+
us-west-2: ami-034bffb8da06d6951

‎cli/.flake8

+9-4
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,18 @@ ignore =
99
# E402 module level import not at top of file
1010
# D101 Missing docstring in public class
1111
# D102 Missing docstring in public method
12+
# D205 1 blank line required between summary line and description
13+
# D400 First line should end with a period
14+
# D401 First line should be in imperative mood
1215
per-file-ignores =
1316
pcluster/*.py: D103
14-
pcluster/config_sanity.py: E402
15-
pcluster/easyconfig.py: E402
16-
pcluster/cfnconfig.py: E402
17-
tests/pcluster/pcluster-unittest.py: D101, D102
17+
pcluster/config_sanity.py: E402, D103
18+
pcluster/easyconfig.py: E402, D103
19+
pcluster/cfnconfig.py: E402, D103
20+
tests/pcluster/pcluster-unittest.py: D101, D102, D103
1821
tests/awsbatch/test_*.py: D101, D102
22+
../tests/integration-tests/tests/*: D103
23+
../tests/integration-tests/*: D205, D400, D401
1924
exclude =
2025
.tox,
2126
.git,

‎cli/awsbatch/awsbsub.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ def _upload_and_get_command(boto3_factory, args, job_s3_folder, job_name, config
214214
# upload input files, if there
215215
if args.input_file:
216216
for file in args.input_file:
217-
s3_uploader.put_file(file, file)
217+
s3_uploader.put_file(file, os.path.basename(file))
218218

219219
# upload command, if needed
220220
if args.command_file or not sys.stdin.isatty() or args.env:

0 commit comments

Comments
 (0)