DCP2.0 (#104)

* Update run.py

* Delete fabfile.py

* Move previous fab functions into run.py

* Debugging from lambda

* tabs to spaces, issues/112

* issues/105

* issues/110, fix passing USE_PLUGINS

* fix tabs to spaces in run.py

* tab/space error in run_batch_general

* issues/107

* issues/107

* add 'cheapest' mode to monitor

* get credentials working

* move machine type/price/date out of fleet file

* move machine type/price/date out of fleet file

* support running of monitor in cheapest mode

* fix indentation in CHECK_IF_DONE

* move cpDone

* don't move done file, you still need to check it

* don't move done file, you still need to check it

* fix cmdstm for batch files

* fixes to make log export work

* python 3 support in run.py

* move Python2 files to their own folder

* pin watchtower to last python2 version

* support for CellProfiler 4

* fix alarm time designation

* Update README.md

* Update requirements.txt

* Update LICENSE

* Add a bit about run_batch_general

Author: bethac07

LICENSE

@@ -1,6 +1,6 @@
 Distributed-CellProfiler is distributed under the following BSD-style license:
 
-Copyright © 2017 Broad Institute, Inc.  All rights reserved.
+Copyright © 2020 Broad Institute, Inc.  All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are

README.md

@@ -22,23 +22,27 @@ job. When the job is completed, the code is also able to stop resources and clea
 Edit the config.py file with all the relevant information for your job. Then, start creating 
 the basic AWS resources by running the following script:
 
- $ fab setup
+ $ python run.py setup
 
 This script intializes the resources in AWS. Notice that the docker registry is built separately,
 and you can modify the worker code to build your own. Anytime you modify the worker code, you need
 to update the docker registry using the Makefile script inside the worker directory.
 
 ### Step 2
-After the first script runs successfully, the job can now be submitted to AWS using the 
-following command:
+After the first script runs successfully, the job can now be submitted to AWS using EITHER of the 
+following commands:
 
  $ python run.py submitJob files/exampleJob.json
+ 
+ OR
+ 
+ $ python run_batch_general.py
 
-This uploads the tasks that are configured in the json file. This assumes that your data is stored
-in S3, and the json file has the paths to find input and output directories. You have to customize
-the exampleJob.json file with paths that make sense for your project. Also, the tasks that compose
-your job are CP groups, and each one will be run in parallel. You need to define each task in this
-json file to guide the parallelization.
+Running either script uploads the tasks that are configured in the json file. This assumes that your 
+data is stored in S3, and the json file has the paths to find input and output directories. You have to 
+customizethe exampleJob.json file or the run_batch_general file with paths that make sense for your project. 
+The tasks that composeyour job are CP groups, and each one will be run in parallel. You need to define each 
+task in your input file to guide the parallelization.
 
 ### Step 3
 After submitting the job to the queue, we can add computing power to process all tasks in AWS. This
@@ -48,10 +52,8 @@ with the following command:
 
  $ python run.py startCluster files/exampleFleet.json
 
-The exampleFleet.json file has to be updated to determine the type of EC2 instances that you want
-and how much you are willing to pay for each machine-hour. Make sure to adjust the values in this
-json file according to your application. After the cluster is ready, the code informs you that
-everything is setup, and saves the spot fleet identifier in a file for further references.
+After the cluster is ready, the code informs you that everything is setup, and saves the spot fleet identifier 
+in a file for further reference.
 
 ### Step 4
 When the cluster is up and running, you can monitor progress using the following command:

config.py

@@ -15,6 +15,8 @@
 ECS_CLUSTER = 'default'
 CLUSTER_MACHINES = 3
 TASKS_PER_MACHINE = 1
+MACHINE_TYPE = ['m4.xlarge']
+MACHINE_PRICE = 0.10
 EBS_VOL_SIZE = 30                       # In GB.  Minimum allowed is 22.
 
 # DOCKER INSTANCE RUNNING ENVIRONMENT:
@@ -37,6 +39,7 @@
 CHECK_IF_DONE_BOOL = 'True'  #True or False- should it check if there are a certain number of non-empty files and delete the job if yes?
 EXPECTED_NUMBER_FILES = 7    #What is the number of files that trigger skipping a job?
 MIN_FILE_SIZE_BYTES = 1      #What is the minimal number of bytes an object should be to "count"?
+NECESSARY_STRING = ''        #Is there any string that should be in the file name to "count"?
 
 # PLUGINS
 USE_PLUGINS = 'True'

fabfile.py

@@ -1,15 +1,10 @@
 {
   "IamFleetRole": "arn:aws:iam::XXXXXXXXXXXXX:role/aws-ec2-spot-fleet-role",
   "AllocationStrategy": "lowestPrice",
-  "TargetCapacity": 3,
-  "SpotPrice": "0.10",
-  "ValidFrom": "2016-07-20T20:28:54Z",
-  "ValidUntil": "2017-07-20T20:28:54Z",
   "TerminateInstancesWithExpiration": true,
   "LaunchSpecifications": [
     {
       "ImageId": "ami-fad25980",
-      "InstanceType": "m4.xlarge",
       "KeyName": "your_key_file_name",
       "IamInstanceProfile": {
         "Arn": "arn:aws:iam::XXXXXXXXXXXX:instance-profile/ecsInstanceRole"

files/exampleFleet_us-east-1.json

@@ -1,15 +1,10 @@
 {
   "IamFleetRole": "arn:aws:iam::XXXXXXXXXXXXX:role/aws-ec2-spot-fleet-role",
   "AllocationStrategy": "lowestPrice",
-  "TargetCapacity": 3,
-  "SpotPrice": "0.10",
-  "ValidFrom": "2016-07-20T20:28:54Z",
-  "ValidUntil": "2017-07-20T20:28:54Z",
   "TerminateInstancesWithExpiration": true,
   "LaunchSpecifications": [
     {
       "ImageId": "ami-c9c87cb1",
-      "InstanceType": "m4.xlarge",
       "KeyName": "your_key_file_name",
       "IamInstanceProfile": {
         "Arn": "arn:aws:iam::XXXXXXXXXXXX:instance-profile/ecsInstanceRole"

files/exampleFleet_us-west-2.json

@@ -1,3 +1 @@
-fabric==1.14.0
-boto>=2.38.0
 boto3>=1.0.0