|
| 1 | +# From Project to Productionized on Heroku |
| 2 | + |
| 3 | +###### Workshop presented by Casey Faist @ PyCon 2019 |
| 4 | + |
| 5 | +Updated for the PyCon 2020 Virtual Heroku Workshop, this project demonstrates the principles of 12 Factor Architecture and the basic steps most Django applications need to achieve that in your application for deploying to production. Heroku is our deployment server, but the principles demonstrated are applicable to any deployment schema and are a best practice for building robust, quick-recover applications in the cloud. |
| 6 | + |
| 7 | +This project utilizes the [Getting Started with Python on Heroku](https://github.com/heroku/python-getting-started) application. |
| 8 | + |
| 9 | +>Note: To clone and run this project locally from scratch, check out the related [Getting Started with Python on Heroku](https://devcenter.heroku.com/articles/getting-started-with-python) article on Dev Center. |
| 10 | +
|
| 11 | +## Prerequisites |
| 12 | + |
| 13 | +[ ] Signup for [Heroku](https://signup.heroku.com/) |
| 14 | +[ ] Install the [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli#download-and-install) |
| 15 | +[ ] Clone this Repo |
| 16 | + |
| 17 | +## Step 1: Add a .gitignore |
| 18 | + |
| 19 | +This will enable you to smoothly run your application locally and in production with minimal headache. |
| 20 | + |
| 21 | +In addition to whatever you'd normally place in your .gitignore, be sure to [untrack](https://git-scm.com/docs/git-rm#Documentation/git-rm.txt---cached) the following files. |
| 22 | + |
| 23 | +``` |
| 24 | +/your-venv-directory # Learn how to create a local environment in LOCALSETUP.md |
| 25 | +__pycache__ |
| 26 | +db.sqlite3 # not needed if you're using Postgres locally |
| 27 | +media/ |
| 28 | +yourapp/static/ |
| 29 | +``` |
| 30 | + |
| 31 | +>note: Don't add your migrations to .gitignore! We're not covering any app |
| 32 | +migrations here, but your migrations should live in your source control - see the [Django docs](https://docs.djangoproject.com/en/2.2/topics/migrations/#the-commands) for more details. |
| 33 | + |
| 34 | +Commit this and untrack files as necessary, and you're good to go. |
| 35 | + |
| 36 | +## Step 2: Modularize your settings |
| 37 | + |
| 38 | +### 2.a - Settings folder |
| 39 | + |
| 40 | +This step is pretty straight forward, but can lead to errors if you forget to update all the references. |
| 41 | + |
| 42 | +In the same directory that your `settings.py` file is located, create a new |
| 43 | +directory called `settings/`. Place your `settings.py` file into that new dir and rename it - I chose `local.py`, but you can use `dev.py` or whatever makes sense to you. |
| 44 | + |
| 45 | +### 2.b |
| 46 | + |
| 47 | +Now navigate to your `manage.py`. Somewhere around the 6th line, you'll see the main process setting the default `DJANGO_SETTINGS_MODULE` to `yourproject.settings`. Update that to `yourproject.settings.local`, or whatever you named your local settings file. |
| 48 | + |
| 49 | +### 2.c |
| 50 | + |
| 51 | +Navigate to `wsgi.py`, located in `yourproject/`. You'll see a similar line there setting the default `DJANGO_SETTINGS_MODULE`; change it the same way that you changed `manage.py`. |
| 52 | + |
| 53 | +>If you set the project up to run locally and have issues with the settings module after this step, you can use an untracked `.env` to set the `DJANGO_SETTINGS_MODULE` variable in your local environment. |
| 54 | +
|
| 55 | +## 3 Changes to local.py |
| 56 | + |
| 57 | +For twelve factor deployment and deployment on Heroku, a few changes are needed in your `local.py` file. |
| 58 | + |
| 59 | +### Whitenoise |
| 60 | + |
| 61 | +[Whitenoise](http://whitenoise.evans.io/en/stable/django.html#using-whitenoise-with-django) is a package for managing static assets, installed |
| 62 | +our `requirements.txt` file, so it'll automatically get installed when we deploy but we still have to install it as middleware. |
| 63 | + |
| 64 | +Scroll to the `MIDDLEWARE` list and place the following line at index 1, or 2nd in the list: |
| 65 | + |
| 66 | +`'whitenoise.middleware.WhiteNoiseMiddleware',` |
| 67 | + |
| 68 | +The order that middleware is loaded is important, which is why we need to add this change to our local.py file. More on that in a bit. |
| 69 | + |
| 70 | +For local-prod parity, let's enable WhiteNoise's [caching and compression](http://whitenoise.evans.io/en/stable/django.html#add-compression-and-caching-support) locally by adding the following line to the bottom of your `local.py`: |
| 71 | + |
| 72 | +`STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'` |
| 73 | + |
| 74 | +## Step 4: heroku.py |
| 75 | + |
| 76 | +We're splitting out our local and production settings, and now it's time to add a file specifying our production settings. I named this file `heroku.py` since that's where I'll be deploying, but `prod.py` works just as well. |
| 77 | + |
| 78 | +To successfully deploy on Heroku, you can copy paste the following into your `heroku.py` file: |
| 79 | + |
| 80 | +``` |
| 81 | +""" |
| 82 | +Production Settings for Heroku |
| 83 | +""" |
| 84 | +
|
| 85 | +import environ |
| 86 | +
|
| 87 | +# If using in your own project, update the project namespace below |
| 88 | +from dynowiki.settings.local import * |
| 89 | +
|
| 90 | +env = environ.Env( |
| 91 | + # set casting, default value |
| 92 | + DEBUG=(bool, False) |
| 93 | +) |
| 94 | +# # reading .env file |
| 95 | +# environ.Env.read_env() |
| 96 | +
|
| 97 | +# False if not in os.environ |
| 98 | +DEBUG = env('DEBUG') |
| 99 | +
|
| 100 | +# Raises django's ImproperlyConfigured exception if SECRET_KEY not in os.environ |
| 101 | +SECRET_KEY = env('SECRET_KEY') |
| 102 | +
|
| 103 | +ALLOWED_HOSTS = env.list('ALLOWED_HOSTS') |
| 104 | +
|
| 105 | +# Parse database connection url strings like psql://user:[email protected]:8458/db |
| 106 | +DATABASES = { |
| 107 | + # read os.environ['DATABASE_URL'] and raises ImproperlyConfigured exception if not found |
| 108 | + 'default': env.db(), |
| 109 | +} |
| 110 | +``` |
| 111 | + |
| 112 | +### 4.a Django-Environ |
| 113 | + |
| 114 | +Django-Environ, installed and imported as `environ`, is a third party package that enables you to easily pull config variables from your environment. You'll see it listed in the `requirements.txt` file, which means it'll get installed automatically on Heroku. To learn more about it's features, you can check out [the django-environ docs](https://django-environ.readthedocs.io/en/latest/) |
| 115 | + |
| 116 | +### 4.b Database connection |
| 117 | + |
| 118 | +The `settings.py` file that Django automatically generates for you utilizes a `sqlite3` database, but that's not ideal for production because data stores should be treated as attached resources. Using the sqlite database means that all your data will disappear on each deploy and dyno restart! |
| 119 | + |
| 120 | +Luckily, Heroku automatically provisions you a Postgres database and supplies it's connection via config var when you create a new Heroku app. Our friend `environ` can pull the database environment variable that Heroku automatically gives so that our application can permanently store data with minimal fuss. |
| 121 | + |
| 122 | +### 4.c Speaking of Databases |
| 123 | + |
| 124 | +You'll also see in our `requirements.txt` file that we have [`psycopg2`](http://initd.org/psycopg/docs/). This is a helper package and while it requires no code setup, it's crucial to include (and keep up to date) to use the Postgres databases supplied by Heroku. |
| 125 | + |
| 126 | +### 4.d Speaking of Dependencies |
| 127 | + |
| 128 | +A key factor of a 12 Factor App is that the dependencies needed for the application are explicitly declared. We've done that with two files. One is the standard Pip dependency file `requirements.txt`, which we've already looked at. |
| 129 | + |
| 130 | +The other is `runtime.txt`, here specifying `python-3.7.3`. This tells the Heroku Python Buildpack to install that specific version of Python 3 into your application's environment before the application is built. You can leave out this file - but the buildpack would then install the default, which is currently set as the latest patchved version of 3.6. |
| 131 | + |
| 132 | +## 5 The Procfile |
| 133 | + |
| 134 | +Our application's settings have been configured, but we still need to pass some information to our environment. In 12 Factor Architecture, we want to execute our codebase as one (or more) processes. |
| 135 | + |
| 136 | +Heroku will automatically start these processes for us - but we have to tell it which ones using the `Procfile`. |
| 137 | + |
| 138 | +Create a file named `Procfile` at your project's root directory (the same level as your `manage.py` file) and place the following in it: |
| 139 | + |
| 140 | +``` |
| 141 | +release: python3 manage.py migrate |
| 142 | +web: gunicorn dynowiki.wsgi --preload --log-file - |
| 143 | +``` |
| 144 | + |
| 145 | +There! We've defined two processes. |
| 146 | + |
| 147 | +### The Release phase |
| 148 | + |
| 149 | +The [release phase](https://devcenter.heroku.com/articles/release-phase) occurs after the build but before the application's other workers are booted. This is a great place for maintenance tasks that should be done each time the application is re-deployed, like our `migrate` task. |
| 150 | + |
| 151 | +### The Web Process |
| 152 | + |
| 153 | +This process is what actually serves our application. [Gunicorn](https://devcenter.heroku.com/articles/python-gunicorn) is a WSGI Server package, you'll see it declared in our `requirements.txt` file. The command specified here is pretty basic - you can specify max connections, number of workers and a host of other features and Heroku will run them automatically. Check out the [Gunicorn docs](https://docs.gunicorn.org/en/latest/settings.html) for more settings options that your application likely does not need. |
| 154 | + |
| 155 | + |
| 156 | +## Heroku Create! |
| 157 | + |
| 158 | +It's finally time! First, make sure that all these changes are checked into git. |
| 159 | + |
| 160 | +Then, then from the root directory, run the following command: |
| 161 | + |
| 162 | +``` |
| 163 | +heroku create |
| 164 | +``` |
| 165 | + |
| 166 | +You have an app! |
| 167 | + |
| 168 | +This app will receive your codebase as a slug. It's where all the environment variables get set, all the connections are specified. Once we've pushed your code to your app, you'll be able to scale up dynos - which run those separate processes from above! |
| 169 | + |
| 170 | +Before moving on, go ahead and grab the name of your site - it'll be something like `serene-caverns-99999.com`. |
| 171 | + |
| 172 | +## Heroku Config |
| 173 | + |
| 174 | +Before we push the code up, let's set the config vars (or we'll see errors!) |
| 175 | + |
| 176 | +You can remind yourself what environment variables we need by looking at your `heroku.py` file, but to get this project ready to run, you can use the CLI `heroku config` command. |
| 177 | + |
| 178 | +``` |
| 179 | +heroku config ALLOWED_HOSTS=your-app-name.com |
| 180 | +heroku config DJANGO_SETTINGS_MODULE=dynowiki.settings.heroku |
| 181 | +``` |
| 182 | +For the SECRET_KEY, you'll need to generate a new secret. For this demo, it doesn't matter what it is - it's great to use a secure hash generator, or a password manager's generator. Just be sure to keep this value secure, don't reuse it, and NEVER check it into source code! |
| 183 | + |
| 184 | +``` |
| 185 | +heroku config SECRET_KEY=YOURSECUREGENERATEDPASSWORD |
| 186 | +``` |
| 187 | +Lastly, Heroku will automatically detect the number of concurrent processes you want to run on each dyno. Depending on the resource usage of your process, this can make each dyno handle more requests more quickly - but for now, let's stick to one process. |
| 188 | + |
| 189 | +``` |
| 190 | +heroku config WEB_CONCURRENCY=1 |
| 191 | +``` |
| 192 | + |
| 193 | +## Heroku Deploy |
| 194 | + |
| 195 | +Alright. It's all led to this. It is time to [deploy to Heroku!](https://devcenter.heroku.com/articles/deploying-python) |
| 196 | + |
| 197 | +Double check all your changes are checked into git on the master branch, then run: |
| 198 | + |
| 199 | +``` |
| 200 | +git push heroku master |
| 201 | +``` |
| 202 | +Did you add your changes to another branch? No worries! (And great branch hygiene.) Use this command instead: |
| 203 | + |
| 204 | +``` |
| 205 | +git push heroku yourbranch:master |
| 206 | +``` |
| 207 | + |
| 208 | +## Scale Up |
| 209 | + |
| 210 | +Now, [scale up](https://devcenter.heroku.com/articles/getting-started-with-python#scale-the-app) your web [process](https://12factor.net/processes) to 1 dyno: |
| 211 | + |
| 212 | +``` |
| 213 | +heroku ps:scale web=1 |
| 214 | +heroku open |
| 215 | +``` |
| 216 | + |
| 217 | +Tada!! It's a dynowiki! |
| 218 | + |
| 219 | +## Logs |
| 220 | + |
| 221 | +To see your website's activity, you can run the following command: |
| 222 | + |
| 223 | +``` |
| 224 | +heroku logs --tail |
| 225 | +``` |
| 226 | + |
| 227 | +## Some Optional Commandline Things |
| 228 | + |
| 229 | +Because of COURSE we want to play with it right? |
| 230 | + |
| 231 | +To make a superuser and create the first article on your wiki: |
| 232 | + |
| 233 | +``` |
| 234 | +heroku run python manage.py createsuperuser |
| 235 | +``` |
| 236 | + |
| 237 | +Then login with the credentials you supplied on https://your-app-name.com and make your first article. |
| 238 | + |
| 239 | +Click on the article to edit, and you can even upload a photo! Or can you? |
| 240 | + |
| 241 | +## S3 Buckets |
| 242 | + |
| 243 | +Because Heroku uses an ephemeral system, the media you upload will disappear after your next deploy, config update or daily reboot. |
| 244 | + |
| 245 | +Heroku has some excellent docs on [how to get set up with S3](https://devcenter.heroku.com/articles/s3), and while preparing for this talk I came across a [thorough guide](https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html) on how to use another third-party package, `django-storages`. |
| 246 | + |
| 247 | +Another option you have is to use a Heroku Addon like [Bucketeer](https://elements.heroku.com/addons/bucketeer). This removes the headache of managing the AWS bucket setup manually, and automatically adds the AWS config vars to your Heroku environment. |
| 248 | + |
| 249 | +Either way, some code changes are required to make use of them. We won't go into detail on S3 setup here since both services cost money - but we will talk about the changes needed to get one hooked up if we have time. |
| 250 | + |
| 251 | +### When to S3 |
| 252 | + |
| 253 | +Until your application needs caching at the level of a CDN, you don't need to host your static files on S3 - they'll be automatically generated and re-collected on each rebuild. |
| 254 | + |
| 255 | +Media files - those uploaded by users or generated by API - are another story, and should be hosted in S3. |
| 256 | + |
| 257 | +### Project Changes for S3 |
| 258 | + |
| 259 | +This assumes you're using Bucketeer, but the changes are quite similar if you've configured an S3 bucket yourself. |
| 260 | + |
| 261 | +Example updated from [this Stack Overflow post](https://stackoverflow.com/questions/19915116/setting-django-to-serve-media-files-from-amazon-s3), a nicely straightforward overview. |
| 262 | + |
| 263 | +First, if you look in your `requirements.txt`, you'll see the dependencies `boto3` and [django-storages](https://django-storages.readthedocs.io/en/1.7.1/backends/amazon-S3.html). We'll need both of them to connect to S3. If you're working in your own project, pip install these packages. |
| 264 | + |
| 265 | +Next, add a file `s3utils.py` to your project inside the `myproject` directory - in this project, it should be inside `dynowiki/` |
| 266 | + |
| 267 | +``` |
| 268 | +from storages.backends.s3boto3 import S3Boto3Storage |
| 269 | +
|
| 270 | +MediaRootS3BotoStorage = lambda: S3Boto3Storage(location='media') |
| 271 | +
|
| 272 | +``` |
| 273 | + |
| 274 | +Then, add the following to your `heroku.py` settings file: |
| 275 | + |
| 276 | +``` |
| 277 | +# S3 Config |
| 278 | +INSTALLED_APPS += ('storages',) |
| 279 | +
|
| 280 | +AWS_STORAGE_BUCKET_NAME = env('BUCKETEER_BUCKET_NAME') |
| 281 | +# Bucketeer requires media files to be in /public |
| 282 | +S3_URL = f"http://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com/public/" |
| 283 | +MEDIA_URL = f"{S3_URL}{MEDIA_ROOT}/" |
| 284 | +DEFAULT_FILE_STORAGE = 'dynowiki.s3utils.MediaRootS3BotoStorage' |
| 285 | +AWS_ACCESS_KEY_ID = env('BUCKETEER_AWS_ACCESS_KEY_ID') |
| 286 | +AWS_SECRET_ACCESS_KEY = env('BUCKETEER_AWS_SECRET_ACCESS_KEY') |
| 287 | +``` |
0 commit comments