Simplify local deployment

.travis.yml

@@ -15,15 +15,15 @@ before_install:
     https://s3.amazonaws.com/amazon-ecs-cli/ecs-cli-linux-amd64-latest &&
     chmod +x $HOME/.local/bin/ecs-cli
 
-install:
-  - docker build -t $IMAGE_REPO_NAME .
+# install:
+#   - docker build -t $IMAGE_REPO_NAME .
 
-after_success:
-  - docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD";
-  - docker build -t $IMAGE_REPO_NAME:latest .
-  - docker tag $IMAGE_REPO_NAME:latest $IMAGE_REPO_NAME:$IMAGE_TAG
-  - docker push $IMAGE_REPO_NAME:latest
-  - docker push $IMAGE_REPO_NAME:$IMAGE_TAG
+# after_success:
+#   - docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD";
+#   - docker build -t $IMAGE_REPO_NAME:latest .
+#   - docker tag $IMAGE_REPO_NAME:latest $IMAGE_REPO_NAME:$IMAGE_TAG
+#   - docker push $IMAGE_REPO_NAME:latest
+#   - docker push $IMAGE_REPO_NAME:$IMAGE_TAG
 
 matrix:
   include:

Dockerfile

@@ -1,5 +1,4 @@
 FROM ubuntu
-MAINTAINER Kai Hendry <[email protected]>
 
 # BEGIN STUFF THAT SHOULD BE IN A BASE IMAGE
 RUN apt-get update

Makefile

@@ -1,14 +1,22 @@
+# Create the local `.env` file
 .env:
-	./setup-env.sh
+	./local-env-setup.bash
 
+# Create a MINIMAL local installation of Unee-T
+# This ONLY includes the following components:
+#	- BZFE (bugzilla-customization repo)
+#	- MEFE (frontend)
+#	- APIENROLL (apienroll repo)
+#	- UNIT (unit repo)
+#	- INVITE (invite repo)
 up: .env
 	docker-compose up
 
 validate:
 	docker-compose -f docker-compose.yml config
 
 build:
-	docker build -t uneet/bugzilla-customisation .
+	docker build -t uneet/bugzilla-customisation:latest .
 
 down:
 	docker-compose down -v
@@ -17,7 +25,7 @@ pull:
 	docker-compose pull
 
 mysqlogin:
-	mysql -h 127.0.0.1 -P 3306 -u root --password=uniti bugzilla
+	mysql -h 127.0.0.1 -P ${MYSQL_PORT} -u ${MYSQL_BZ_USER} --password=${MYSQL_PASSWORD_BZ_USER} ${MYSQL_DATABASE}
 
 clean:
 	sudo chown -R $$USER skin custom/ extensions/

README.md

@@ -1,82 +1,122 @@
-Requires [docker](https://www.docker.com/) &
-[docker-compose](https://docs.docker.com/compose/). Linux is definitely a plus, else run on a VPS.
+# Overview:
 
-# Development servers
+## WARNINGS:
 
-* [AWS_PROFILE](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html) `uneet-dev` AWS account # 812644853088
-* [Bugzilla](https://dashboard.dev.unee-t.com)
-* [Meteor](https://case.dev.unee-t.com)
-* auroradb.dev.unee-t.com
+There are a few legacy things that we need to cleanup, we know...
 
-# Production servers
+## What this does:
 
-* [AWS_PROFILE](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html) `uneet-prod` AWS account # 192458993663
-* [Bugzilla](https://dashboard.unee-t.com)
-* [Meteor](https://case.unee-t.com)
-* auroradb.unee-t.com
+This repo has been built to do several things:
+- Create the docker image for the BZFE
+- Deploy the latest version of the Unee-T Dashboard (BZFE)
+- Facilitate the local installation of a **semi-functional Unee-T environment** locally (See the *Developing locally* section below).
+
+## Important information and GOTCHAs:
+
+- Our code has been built on and for Ubuntu Linux.
+- Our code has NOT been tested on other platforms.
+- Our code will most likely NOT work on any platform other that Ubuntu at the moment.
+- We use [docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/) to containerise each Unee-T component/service
+- We use AWS for **A LOT** of things. 
+- The current version of the Unee-T code will NOT work as expected if not fully (i.e with all the necessary components and dependencies) deployed on AWS.
+- We use Meteor/Mongo for the case interface.
+- We use Bugzilla/AuroraDb-MySQL for case management.
+- AWS AuroraDb is a MUST since we are using lambdas inside Db events, calls and procedures. The current version of the Unee-T code will NOT work if not on AWS AuroraDb.
+- We use Travis CI for Test.
+- We use Travis CI for automated deployments.
 
 # Developing locally
 
-We used to start from a [prime sql](https://github.com/unee-t/bz-database), but
-now we being from existing development snapshots.
+To develop locally, you need to to start from a [primed sql file](https://github.com/unee-t/bz-database).
 
-The idea now is to start from a **snapshot** of the remote development (dev)
-environment. Our remote dev environment is hosted on AWS and so are all the
-secrets, so you really need to get access or a copy of the credentials from one
-of the existing Unee-T developers.
+Make sure that the MongoDb and the BzDB are in sync!
+We try to handle "orphan" ressources graciously but there might be some edge cases that we've missed.
 
-Finally your Frontend's Mongo state must be in sync! Use scripts in
-https://github.com/unee-t/frontend/blob/master/backup/ to backup and restore
-Mongo.
+## Initial setup (local):
 
-To initialise / reset the database for development:
+Make sure your local .env is correctly setup.
+You can run `make .env` to do that
 
-	make clean
-	export MYSQL_ROOT_PASSWORD=$(aws --profile uneet-dev ssm get-parameters --names MYSQL_ROOT_PASSWORD --with-decryption
---query Parameters[0].Value --output text)
-	# Get a snapshot of dev
-	mysqldump -R -h auroradb.dev.unee-t.com -P 3306 -u root --password=$MYSQL_ROOT_PASSWORD bugzilla > dev-backup.sql
+Make sure to have a look at the file `./local-env-setup.bash` first!
 
-You want to Aurora's [mock mysql.lambda_async](https://github.com/unee-t/bz-database/issues/137#issuecomment-523731990).
+Once the environment variables are OK you can run
+`make up`
 
-Make sure your local .env is correctly setup with `./env-setup.bash`
+This will create several services locally:
+- case (MEFE) accessible with your browser at: http://localhost:3000/
+- dasboard (BZFE) accessible with your browser at: http://localhost:8081/
 
 	docker-compose up -d db # Just start the database at first, should be empty
 	# Restore dev snapshot
 	mysql -h db -P 3306 -u root --password=$MYSQL_ROOT_PASSWORD bugzilla < dev-backup.sql
 	make up
 
-The dashboard administrator username / password is:
+## Initialise / reset the database for development:
 
-	aws --profile uneet-dev ssm get-parameters --names BZFE_ADMIN_USER --with-decryption --query Parameters[0].Value --output text
-	aws --profile uneet-dev ssm get-parameters --names BZFE_ADMIN_PASS --with-decryption --query Parameters[0].Value --output text
+- `make down`
+- `make clean`
+- `make up`
 
-# Bugzilla configuration notes
+WIP - explain how to dump the latest version on the seed database in the local environment.
 
-Bugzilla is setup by a variety of sources:
+## Lambdas:
 
-* the initial [vanilla stable bugzilla base image](https://github.com/unee-t/bugzilla)
-* \*-params.json - seemingly just for URL and mailfrom address set via public URLs
-* localconfig - created with the start script to set database connection parameters
-* bugzilla_admin - for initial administrator user/pass (only used when starting from a blank slate)
-* custom skin and templates - set via the Dockerfile
+You want to Aurora's [mock mysql.lambda_async](https://github.com/unee-t/bz-database/issues/137#issuecomment-523731990).
 
-Largely co-ordinated by environment varibles in:
+## Debug your Docker image by entering it
 
-* .env for local
-* aws-env.dev for development / testing /staging
-* aws-env.prod for production
+	docker exec -it bugzilla-customisation_bugzilla_1 /bin/bash
 
-Note that the **BUGZILLA_ADMIN_KEY** needs to be in place on the table **user_api_keys**. Please study how https://github.com/unee-t/reset-demo works.
+## Debug mysql queries locally
 
-# Debug your Docker image by entering it
+	innotop -h 127.0.0.1 -P 3306 -u root --password=$MYSQL_ROOT_PASSWORD
 
-	docker exec -it bugzilla-customisation_bugzilla_1 /bin/bash
+## How to check for mail when in test mode
 
-# Release process for production
+	docker exec -it bugzilla_bugzilla_1 /bin/bash
+	cat data/mailer.testfile
+
+# Architecture:
+
+- When you install Unee-T, you are creating a Unee-T *Installation*.
+- Each Unee-T installation is designed to have 3 different *Environments*:
+  - DEV: for test and staging in "Real Life" condition (NOT local)
+  - PROD: What you will use in production.
+  - DEMO: A sandboxed environment that is running the same version of the code as the PROD environment.
+
+  We are rellying HEAVILY on AWS services like SES, SQS, Lambdas, ECS, etc...
+  This makes local development a bit more difficult.
+
+## Main environment on Unee-T.com (Public)
+
+### DEV/STAGING
+
+* [BZFE - Unee-T Dashboard](https://dashboard.dev.unee-t.com)
+* [MEFE - Unee-T Case](https://case.dev.unee-t.com)
+
+### PRODUCTION
+
+* [BZFE - Unee-T Dashboard](https://dashboard.unee-t.com)
+* [MEFE - Unee-T Case](https://case.unee-t.com)
 
-Consider doing this at a quiet time though not on a Friday afternoon as
-developers would like to relax typically then like everyone else.
+### DEMO
+
+* [BZFE - Unee-T Dashboard](https://dashboard.demo.unee-t.com)
+* [MEFE - Unee-T Case](https://case.demo.unee-t.com)
+
+## AWS accounts:
+
+Each *Environment* is deployed on a dedicated AWS account.
+An *Installation* is linked to 3 different AWS accounts (DEV/STAGING, PROD and DEMO).
+
+## Environment variables:
+
+Secrets and environment variables are managed in [AWS's parameter
+store](https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home?region=ap-southeast-1#Parameters:sort=Name).
+
+# Release process:
+
+Consider doing this at a quiet time though not on a Friday afternoon as developers would like to relax typically then like everyone else.
 
 Release manager needs to ensure a seamless UX for the end user by:
 
@@ -90,39 +130,52 @@ Release manager needs to ensure a seamless UX for the end user by:
 8. Write release notes aka communicate with users about new features or fixes that make their lives easier
 9. Solicit feedback from users
 
-# JSON API
+# Deployment
 
-<https://bugzilla.readthedocs.io/en/latest/api/>
+Deployments are automated with Travis CI.
 
-	curl http://localhost:8081/rest/bug/1?api_key=$(aws --profile uneet-dev ssm get-parameters --names BUGZILLA_ADMIN_KEY --with-decryption --query Parameters[0].Value --output text) | jq
+- The DEV/STAGING environment is re-deployed/updated each time a commit is pushed to the `master` branch.
+- The PROD and DEMO environments are re-deployed/updated each time we do a *tag release* of the `master` branch.
 
-There are more examples in Postman.
+Each components are updated separately.
+Ex: a push on the master in this repo will only update the BZFE component of the DEV/STAGING environment of the Unee-T installation.
 
-# Environment
+# Backup and Restore:
+https://github.com/unee-t/frontend/blob/master/backup/ to backup and restore
+Mongo.
 
-Secrets are managed in [AWS's parameter
-store](https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home?region=ap-southeast-1#Parameters:sort=Name).
+# Misc. - Things to keep in mind:
 
-## About email
+## Bugzilla configuration notes:
 
-`SES*` is required for email notifications. [SES dashboard](https://us-west-2.console.aws.amazon.com/ses/home?region=us-west-2#dashboard:)
+We rely on Bugzilla for several things.
 
-How to test if email is working:
+Bugzilla is setup by a variety of sources:
 
-	echo -e "Subject: Test Mail\r\n\r\nThis is a test mail" | msmtp --debug -t [email protected]
+* the initial [vanilla stable bugzilla base image](https://github.com/unee-t/bugzilla)
+* \*-params.json - seemingly just for URL and mailfrom address set via public URLs
+* localconfig - created with the start script to set database connection parameters
+* bugzilla_admin - for initial administrator user/pass (only used when starting from a blank slate)
+* custom skin and templates - set via the Dockerfile
 
-Video about testing email: https://s.natalian.org/2017-10-27/uneetmail.mp4
+Largely co-ordinated by environment varibles in:
 
-# Debug mysql queries locally
+* .env for local
+* aws-env.dev for development / testing /staging
+* aws-env.prod for production
 
-	innotop -h 127.0.0.1 -P 3306 -u root --password=$MYSQL_ROOT_PASSWORD
+Note that the **BUGZILLA_ADMIN_KEY** needs to be in place on the table **user_api_keys**. Please study how https://github.com/unee-t/reset-demo works.
 
-# How to check for mail when in test mode
 
-	docker exec -it bugzilla_bugzilla_1 /bin/bash
-	cat data/mailer.testfile
+## JSON API
+
+<https://bugzilla.readthedocs.io/en/latest/api/>
 
-# AWS ECS setup
+	curl http://localhost:8081/rest/bug/1?api_key=$(aws --profile uneet-dev ssm get-parameters --names BUGZILLA_ADMIN_KEY --with-decryption --query Parameters[0].Value --output text) | jq
+
+There are more examples in Postman.
+
+## AWS ECS setup
 
 * [ECS overview](https://unee-t-media.s3-accelerate.amazonaws.com/2017/ecs-overview.mp4)
 * [ECS deploy](https://unee-t-media.s3-accelerate.amazonaws.com/2017/ecs-deploy.mp4)
@@ -134,15 +187,25 @@ Refer to `ecs-cli compose service create -h` to create with a load balancer.
 * [Development account](https://812644853088.signin.aws.amazon.com/console)
 * [Production account](https://192458993663.signin.aws.amazon.com/console)
 
-# Logs on Cloudwatch
+## About email
+
+`SES*` is required for email notifications. [SES dashboard](https://us-west-2.console.aws.amazon.com/ses/home?region=us-west-2#dashboard:)
+
+How to test if email is working:
+
+	echo -e "Subject: Test Mail\r\n\r\nThis is a test mail" | msmtp --debug -t [email protected]
+
+Video about testing email: https://s.natalian.org/2017-10-27/uneetmail.mp4
+
+## Logs on Cloudwatch
 
 How to filter for 5xx errors:
 
 	[..., request = *HTTP*, status_code = 5**, , ,]
 
 https://media.dev.unee-t.com/2018-08-23/bugzilla-debug.mp4
 
-# Why slow?
+## Why slow?
 
 https://media.dev.unee-t.com/2018-08-23/targetresponsetime.mp4