Getting Started with Chef

Getting Started with Chef

This brief guide is given as an alternative to Opscode's official Getting Started guide, since I find it confusing for newcomers in a handful of ways. Here we'll walk through the first steps of using Chef, with our established BarCamp Bangkok Hosted Chef account lending some specific examples.

Overview

There are three general elements to be aware of to understand the workflow for Chef Server:

1. The Chef Server
2. Chef clients -- the machine(s) you're configuring
3. Your workstation, where you edit Chef code and run the knife utility

The Server in our case is Hosted Chef. Under the hood, Chef Server stores information about clients and their configuration in a CouchDB data store, runs a Solr-based indexing service to make all of this information searchable, and exposes a REST API where clients report their information and ask for updates. It also provides the web app you see at manage.opscode.com for humans to browse the information.

Knife is a command-line tool that also communicates with the Server's API to query information and change some aspects of it on-the-fly. Knife is installed by the chef RubyGem and you will run it from your local workstation where you are developing. You don't need to read the linked guide on Knife at this time, but you'll find yourself referencing it (and the tool's built-in help) a lot soon.

System Requirements

Chef is a Ruby tool and you will thus need Ruby on your system in order to work with it, along with the RubyGems package system for installing Ruby libraries. If you're using a recent version of Mac OS X, you probably already have everything you need (just install Xcode if by some chance you haven't already).

Installing Ruby on various other platforms is beyond the scope of this tutorial but is well-documented on the official Ruby site. Likewise, the official RubyGems site has clear info on installing or updating to a current version of RubyGems.

From this point, installing Chef is as simple as:

$ gem install chef

Since the Chef repository describing our infrastructure is kept in the Git version control system, you'll need Git as well. GitHub has nice instructions for installing Git and setting things up to work with GitHub.

Authentication

The last setup steps needed are for authentication with the Chef Server. You'll need three things:

1. The verification key for the barcampbangkok organization sent to you by the web team after you signed up for Hosted Chef (see the Home page on this wiki if you missed signing up).
2. Your personal key that you can download while logged into Hosted Chef. Find the instructions for obtaining this on the Opscode guide -- only follow the part about getting your User Key, we'll do the rest in a better way below.
3. A key for decrypting server passwords that are stored in encrypted form on the Chef Server. See Encrypted Data Bags on the Chef wiki, and our Managing Sensitive Passwords document.

Now we'll look at what you need to do with these files. I assume Linux/OS X here, hoping Windows users can adapt for themselves. Start by cloning our project Git repository and move into the directory:

$ cd projects/barcampbkk  # wherever you keep yours :-)
$ git clone [email protected]:barcampbangkok/chef-repo.git chef
$ cd chef

The repository contains a Knife configuration file that will be used to connect you to our Chef Server whenever you execute knife while you are inside the project directory. We just need to put the authentication keys in place on your system where the config file expects to find them:

$ mkdir -p ~/.chef/barcampbangkok
$ mv USERNAME.pem ~/.chef/
$ mv barcampbangkok-validator.pem ~/.chef/
# A good idea to make these private:
$ chmod 600 ~/.chef/USERNAME.pem ~/.chef/barcampbangkok-validator.pem
# A team member will give you the contents to put in this file:
$ vim ~/.chef/barcampbangkok/encrypted_data_bag_secret

USERNAME.pem is of course your private User Key that you downloaded. You should rename it to your login name on your computer if different from your user name on Hosted Chef, or set the CHEF_USER environment variable referred to in the knife.rb in your shell config like .bashrc if you prefer.

This setup is a little different from what Opscode's Getting Started guide suggests -- it is intended to consolidate your private information in one sane place, outside of version control, and let us share a common knife.rb that is in version control to simplify setup steps.

That's it! You should be able to run Knife now, try:

$ knife node list
  barcampbangkok.org

See the Conclusion section below for some pointers on where to go from here, but let's look at one simple example to get you started: adding your own public SSH key so that you can log into servers.

A Chef Example: Adding Your Public SSH Key

Though you probably shouldn't be logging into servers very often that are managed by Chef since you should use Chef to do the work for you of changing configurations, you'll undoubtedly still want to log in sometimes to tail some log output, check top, etc., so let's add your SSH pubkey so that you're authorized to log in as the barcamp user.

Our public SSH keys are managed with data bags, which are bits of JSON data that are particularly useful for pieces of configuration used across several different systems in your infrastructure, and that you may want to change frequently without needing to change code. Edit the pubkeys data bag and upload your changes to the Chef Server:

$ vim data_bags/pubkeys/barcampbkk.json
# Add your public key following the simple structure you find in the file
$ knife data bag from file pubkeys barcampbkk.json
$ git commit -a -m 'Added my public SSH key'
$ git push origin master

If there is any problem with your JSON syntax, Knife will tell you and refuse to upload the changes. The knife command reads pretty naturally, but see knife help data to learn more about working with data bags.

Once you've uploaded the change, it will be found by chef-client the next time it periodically runs on the server(s) to apply configuration. A recipe in our custom cookbook for our application looks in this data bag for keys to put in the barcamp user's authorized_keys.

You don't want to wait for chef-client to run on its own to test your change and log in though, so let's force a Chef run right now:

$ knife ssh 'fqdn:*.barcampbangkok.org' 'sudo chef-client' -x barcamp

FIXME: This won't work at this time because our new server for 2011 has it's domain name configured as www.barcampbangkok.org, but DNS is still pointing at the old server. This will be fixed once that switch is made -- apologies for the inconvenience!

You should see the detailed output of Chef checking and applying configuration. Once completed, you should be able to access the server:

$ ssh [email protected]

We only have one server, but knife ssh is a powerful way to run a command on or log into multiple servers in larger infrastructures.

Note that we keep the data bag stored in version control. It's possible to do many things like editing data bags by direct communication with the Chef Server, in this case with knife data bag edit. Chef allows flexibility in your workflow -- throughout the documentation, you will find references to using a "maximal version control" approach, or using more on-the-fly Knife changes. I advocate using version control as much as possible, so code and git commits make it as clear as possible to other team members what has been done. The Environments wiki documentation has some discussion for a good example of this.

Conclusion

Now is the point where you'll want to check out the Knife documentation and the rest of the Chef wiki to learn how to fully use Chef.

One of the things I dislike about Opscode's official Getting Started guide for Chef is that it uses your own local workstation as a client node to be configured as part of your infrastructure in the examples. While nobly intended to shorten the setup needed to start experimenting, it conflates the role of your local workstation with the systems you are configuring, which is confusing and not something you'd typically do in real life. For this reason I don't recommend following that guide further for learning about cookbooks, etc. -- head straight for the wiki, and look at our project's existing code and git commit history for some examples.

I recommend using virtual machines to experiment with Chef and to test changes before applying them to production (Virtualbox works well and is free and open source). This may be documented further on this wiki in the future. Vagrant offers a nice quick way to get started if you want to try on your own.