Background<\/p>\n
At uShip, we’ve been moving to an AMI deployment strategy for standing up web servers that houses our main application. We made the decision as part of a larger strategy to ensure our environments (dev, qa, prod, etc.) were as similar as possible. We figured that if we could build a single AMI that is deployed to every environment, that would be a huge step in accomplishing environment parity. While the process has mostly been straight-forward, we have run into a problem and the new\u00a0Effortless Infrastructure Pattern\u00a0from Chef provided an elegant solution.<\/p>\n
The Problem<\/p>\n
Chef Infra\u00a0is a great way of managing configuration for servers. One of the biggest reasons that we reached for Chef versus something else is the Windows support. While other options have gotten better, Chef had the support back in 2015 when we were evaluating configuration management solutions. Chef’s client\/server model allowed us to get visibilty into our fleet. However, that visibility comes at a cost.<\/p>\n
The cost has to do with bootstrapping nodes into the Chef Server. Traditionally, this process works well as you’d usually have long-lived nodes and if you wanted to remove one, you could do that manually using the\u00a0chef-server-ctl. With our AMI deployment strategy, we were creating and destroying nodes every deployment so we were left with many missing nodes and no easy way of cleaning them up. Before we get into the effortless pattern, let’s look at the traditional way of bootstrapping a node.<\/p>\n
Bootstrapping Chef Nodes<\/p>\n
In Chef, bootstrapping is the process that installs the Chef Infra Client and sets up the node to communicate with the Chef Server. This can either be done using the\u00a0knife bootstrap\u00a0command from your workstation or, in the case of AWS, with a user data script. Here’s an example of what we were using for an unattended bootstrap:<\/p>\n
Write-Output “Pull the encrypted_data_bag_secret key from S3”
\n& “C:\/Program Files\/Amazon\/AWSCLI\/bin\/aws.exe” s3 cp s3:\/\/<my-super-real-s3-bucket>\/default-validator.pem C:\/chef\/
\n& “C:\/Program Files\/Amazon\/AWSCLI\/bin\/aws.exe” s3 cp s3:\/\/<my-super-real-s3-bucket>\/encrypted_data_bag_secret C:\/chef\/encrypted_data_bag_secret<\/p>\n
Write-Output “Create first-boot.json for Chef bootstrap into $environment policy_group”
\n$firstBoot = @{“policy_name” = “web”; “policy_group” = “$environment” }
\nSet-Content -Path C:\/chef\/first-boot.json -Value ($firstboot | ConvertTo-Json -Depth 10)<\/p>\n
Write-Output “Create client.rb file for Chef using a dynamically-generated node name”
\n$nodeName = “$(hostname)-{0}” -f ( -join ((65..90) + (97..122) | Get-Random -Count 4 | % { [char]$_ }))<\/p>\n
$clientrb = @”
\n chef_server_url ‘https:\/\/chef-server.example.com\/organizations\/default’
\n validation_client_name ‘default-validator’
\n validation_key ‘C:\/chef\/default-validator.pem’
\n node_name ‘{0}’
\n“@ -f $nodeName
\nSet-Content -Path C:\/chef\/client.rb -Value $clientrb<\/p>\n
Write-Output “Run Chef client first time”
\nC:\/opscode\/chef\/bin\/chef-client.bat -j C:\/chef\/first-boot.json<\/p>\n
<\/p>\n
I’d like to note that we were originally using\u00a0Chef Vault\u00a0to store secrets but there doesn’t appear to be a way for a node to bootstrap itself and then give itself permissions to a vault item and so we’re using\u00a0encrypted data bags\u00a0here.<\/p>\n
Assuming that you’ve set up your S3 bucket policy and EC2 instance role, this solution works well to bring up instances. But, as mentioned earlier, if you boot up four new servers in each environment every time you deploy, you’ll have an increasing number of missing nodes. There is a\u00a0Lambda\u00a0out on the interwebs for cleaning up nodes in the Chef Server, but this is kinda of a pain to do and only addresses the Chef Server; it does nothing for the ones in\u00a0Chef Automate.<\/p>\n
Effortless Infrastructure<\/p>\n
If you missed the session from ChefConf 2019, there’s an excellent talk by David Echols about\u00a0what effortless config is. Essentially, the effortless pattern is a way to build and run your cookbooks as a single, deployable package. It accomplishes this using\u00a0Habitat,\u00a0Policyfiles, and\u00a0Chef Solo. Before reading further, I urge you to check out that video and the track on\u00a0Learn Chef Rally.<\/p>\n
Prerequisites<\/p>\n
Chef Workstation
\nHabitat<\/p>\n
Generate a Cookbook<\/p>\n
The first thing we need to do is generate a new cookbook. I’m going to deploy a cookbook that sets up IIS on a Windows server but the concepts should be similar if you’re deploying Linux servers.<\/p>\n
PS C:\\Users\\uship\\Projects> chef generate cookbook webserver
\nGenerating cookbook webserver
\n– Ensuring correct cookbook content
\n– Committing cookbook files to git<\/p>\n
Your cookbook is ready. To setup the pipeline, type `cd webserver`, then run `delivery init`<\/p>\n
<\/p>\n
Let’s check out the content of the\u00a0webserver\u00a0cookbook:<\/p>\n
PS C:\\Users\\uship\\Projects> cd webserver
\nPS C:\\Users\\uship\\Projects\\webserver> tree
\n.
\n\u251c\u2500\u2500 CHANGELOG.md
\n\u251c\u2500\u2500 LICENSE
\n\u251c\u2500\u2500 Policyfile.rb
\n\u251c\u2500\u2500 README.md
\n\u251c\u2500\u2500 chefignore
\n\u251c\u2500\u2500 kitchen.yml
\n\u251c\u2500\u2500 metadata.rb
\n\u251c\u2500\u2500 recipes
\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 default.rb
\n\u251c\u2500\u2500 spec
\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 spec_helper.rb
\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 unit
\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 recipes
\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 default_spec.rb
\n\u2514\u2500\u2500 test
\n \u2514\u2500\u2500 integration
\n \u2514\u2500\u2500 default
\n \u2514\u2500\u2500 default_test.rb<\/p>\n
7 directories, 11 files<\/p>\n
<\/p>\n
To set up IIS, we’re going to leverage the\u00a0iis cookbook. Add the following to the\u00a0metadata.rb\u00a0file:<\/p>\n
name ‘webserver’
\nmaintainer ‘The Authors’
\n.
\n.
\n.
\n# source_url ‘https:\/\/github.com\/<insert_org_here>\/webserver’<\/p>\n
depends ‘iis’, ‘~> 7.2.0’<\/p>\n
<\/p>\n
We’ll need to go ahead and install the dependencies. For this, we’ll leverage Policyfiles. If you are unfamiliar, they’re basically what replaces\u00a0Berkshelf\u00a0and environments\/roles. Check out the\u00a0documentation\u00a0but you should just need to run the following:<\/p>\n
PS C:\\Users\\uship\\Projects\\webserver> chef install
\nBuilding policy webserver
\nExpanded run list: recipe[webserver::default]
\nCaching Cookbooks…
\nInstalling webserver >= 0.0.0 from path
\nInstalling iis 7.2.0
\nInstalling windows 6.0.1<\/p>\n
Lockfile written to \/Users\/uship\/Documents\/effortless_ami_deployments\/webserver\/Policyfile.lock.json
\nPolicy revision id: c2746cac28e13e1dae4fa99f4b9f9d56e5b7bf11894f1cce1e8940a2f4de42c3<\/p>\n
<\/p>\n
Now that we have our dependencies installed, let’s update the Chef recipe to install IIS.<\/p>\n
#
\n# Cookbook:: webserver
\n# Recipe:: default
\n#
\n# Copyright:: 2019, The Authors, All Rights Reserved.<\/p>\n
include_recipe ‘iis’<\/p>\n
<\/p>\n
This will install IIS on the server and enable the W3SVC service. At this point, if you boot up a\u00a0Test Kitchen\u00a0instance to test and then browse to the IP address, you should see the default Internet Information Services page.<\/p>\n
Package the Cookbook<\/p>\n
As I said earlier, the effortless infrastructure pattern leverages\u00a0Habitat\u00a0to package and run your Chef cookbook like an application. To package this up, we’ll need to habitatize our application and create a basic structure. Note that this is going to be deployed and run on a Windows server so it needs to be built on a Windows box to work properly. If you’re working on Mac or Linux, the concepts are the same but you’d use Bash instead of Powershell for writing your plan. Again, I’ll defer to the\u00a0Habitat documentation\u00a0for the specifics.<\/p>\n
From the root of your cookbook directory, initialize the Habitat plan, using your\u00a0origin:<\/p>\n
PS C:\\Users\\uship\\Projects\\webserver> hab plan init -o uship
\n\u00bb Constructing a cozy habitat for your app…<\/p>\n
\u03a9 Creating file: habitat\/plan.ps1
\n `plan.sh` is the foundation of your new habitat. It contains metadata,
\n dependencies, and tasks.<\/p>\n
\u03a9 Creating file: habitat\/default.toml
\n `default.toml` contains default values for `cfg` prefixed variables.<\/p>\n
\u03a9 Creating file: habitat\/README.md
\n `README.md` contains a basic README document which you should update.<\/p>\n
\u03a9 Creating directory: habitat\/config\/
\n `\/config\/` contains configuration files for your app.<\/p>\n
\u03a9 Creating directory: habitat\/hooks\/
\n `\/hooks\/` contains automation hooks into your habitat.<\/p>\n
For more information on any of the files:
\n https:\/\/www.habitat.sh\/docs\/reference\/plan-syntax\/<\/p>\n
\u2192 Using existing file: habitat\/..\/.gitignore (1 lines appended)
\n\u2261 An abode for your code is initialized!<\/p>\n
<\/p>\n
For the effortless infrastructure, we’ll lean on the\u00a0Habitat Scaffolding\u00a0provided by the Habitat core team. You can see what the scaffolding is doing by looking in the\u00a0repository, but all we need to do is update the\u00a0habitat\/plan.ps1\u00a0file:<\/p>\n
# This is the name of our Habitat package
\n$pkg_name=”webserver”<\/p>\n
# Update this with your origin
\n$pkg_origin=”uship”<\/p>\n
# Package version. Typically follomws Semantic Versioning
\n$pkg_version=”0.0.1″<\/p>\n
# Update this per your preferences
\n$pkg_maintainer=”uShip, Inc. <devops@uship.com>”<\/p>\n
# We need these dependencies for our application to run
\n$pkg_deps=@(
\n “core\/cacerts”
\n “stuartpreston\/chef-client” # https:\/\/github.com\/habitat-sh\/habitat\/issues\/6671
\n)<\/p>\n
# Use the scaffolding-chef-infra scaffolding
\n$pkg_scaffolding=”chef\/scaffolding-chef-infra”<\/p>\n
# Name of our Policyfile
\n$scaffold_policy_name=”Policyfile”<\/p>\n
# Location of the Policyfile. In this case, habitat\/..\/Policyfile.rb
\n$scaffold_policyfile_path=”$PLAN_CONTEXT\/..\/”<\/p>\n
<\/p>\n
The last thing we need to do before we can build our Habitat package is update the configuration for the Chef Client that will be running. Habitat uses\u00a0Toml\u00a0for configuration and the default config is in\u00a0habitat\/default.toml:<\/p>\n
# Use this file to templatize your application’s native configuration files.
\n# See the docs at https:\/\/www.habitat.sh\/docs\/create-packages-configure\/.
\n# You can safely delete this file if you don’t need it.<\/p>\n
# Run the Chef Client every 5 minutes
\ninterval = 300<\/p>\n
# Offset the Chef Client runs by 30 seconds
\nsplay = 30<\/p>\n
# No offset for the first run
\nsplay_first_run = 0<\/p>\n
# Wait for Chef Client run lock file to be deleted
\nrun_lock_timeout = 300<\/p>\n
<\/p>\n
Go ahead and remove the\u00a0habitat\/config\u00a0and\u00a0habitat\/hooks\u00a0directories as these aren’t needed and tend to cause errors with the build:<\/p>\n
PS C:\\Users\\uship\\Projects\\webserver> rmdir habitat\/config
\nPS C:\\Users\\uship\\Projects\\webserver> rmdir habitat\/hooks<\/p>\n
<\/p>\n
To build our Habitat package, we’ll enter the Habitat studio. The studio is a clean room which only packages up the dependencies that have been specified and nothing else.<\/p>\n
PS C:\\Users\\uship\\Projects\\webserver> hab studio enter
\nWARNING: Using a local Studio. To use a Docker studio, use the -D argument.
\n hab-studio: Creating Studio at C:\\hab\\studios\\Users–uship–Projects–webserver
\n\u00bb Importing origin key from standard input
\n\u2261 Imported public origin key uship-20190919164651.
\n\u00bb Importing origin key from standard input
\n\u2261 Imported secret origin key uship-20190919164651.
\n** The Habitat Supervisor has been started in the background.
\n** Use ‘hab svc start’ and ‘hab svc stop’ to start and stop services.
\n** Use the ‘Get-SupervisorLog’ command to stream the Supervisor log.
\n** Use the ‘Stop-Supervisor’ to terminate the Supervisor.<\/p>\n
hab-studio: Entering Studio at C:\\hab\\studios\\Users–uship–Projects–webserver
\n[HAB-STUDIO] Habitat:\\src><\/p>\n
<\/p>\n
Inside the studio, we’ll run\u00a0build\u00a0which will use the default location of the plan file in\u00a0habitat\/plan.ps1:<\/p>\n
[HAB-STUDIO] Habitat:\\src> build
\n : Loading C:\\hab\\studios\\Users–uship–Projects–webserver\\src\\habitat\\plan.ps1
\n webserver: Plan loaded
\n webserver: Validating plan metadata
\n webserver: hab-plan-build.ps1 setup
\n webserver: Using HAB_BIN=C:\\hab\\pkgs\\core\\hab-studio\\0.83.0\\20190712234514\\bin\\hab\\hab.exe for installs, signing, and hashing
\n webserver: Resolving scaffolding dependencies
\n\u00bb Installing chef\/scaffolding-chef-infra
\n\u2302 Determining latest version of chef\/scaffolding-chef-infra in the ‘stable’ channel
\n\u2192 Using chef\/scaffolding-chef-infra\/0.16.0\/20191028151207
\n\u2261 Install of chef\/scaffolding-chef-infra\/0.16.0\/20191028151207 complete with 0 new packages installed.
\n webserver: Resolved scaffolding dependency ‘chef\/scaffolding-chef-infra’ to C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\chef\\scaffolding-chef-infra\\0.16.0\\20191028151207
\n webserver: Loading Scaffolding C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\chef\\scaffolding-chef-infra\\0.16.0\\20191028151207\/lib\/scaffolding.ps1
\n\u00bb Installing chef\/scaffolding-chef-infra
\n\u2302 Determining latest version of chef\/scaffolding-chef-infra in the ‘stable’ channel
\n\u2192 Using chef\/scaffolding-chef-infra\/0.16.0\/20191028151207
\n\u2261 Install of chef\/scaffolding-chef-infra\/0.16.0\/20191028151207 complete with 0 new packages installed.
\n webserver: Resolved build dependency ‘chef\/scaffolding-chef-infra’ to C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\chef\\scaffolding-chef-infra\\0.16.0\\20191028151207
\n\u00bb Installing core\/chef-dk\/2.5.3\/20180416182816
\n\u2192 Using core\/chef-dk\/2.5.3\/20180416182816
\n.
\n.
\n.
\n webserver: Preparing to build
\n webserver: Building
\nBuilding policy webserver
\nExpanded run list: recipe[webserver::default]
\nCaching Cookbooks…
\nInstalling webserver >= 0.0.0 from path
\nUsing iis 7.2.0
\nUsing windows 6.0.1<\/p>\n
Lockfile written to C:\/hab\/studios\/Users–uship–Projects–webserver\/src\/Policyfile.lock.json
\nPolicy revision id: f8a3f2d55e079328c164d2c0250854348cdb7900e89c4c8e9cbe155825d7635b
\n webserver: Installing
\nExported policy ‘webserver’ to C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\uship\\webserver\\0.0.1\\20191114064617<\/p>\n
To converge this system with the exported policy, run:
\n cd C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\uship\\webserver\\0.0.1\\20191114064617
\n chef-client -z<\/p>\n
Directory: C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\uship\\webserver\\0.0.1\\20191114064617<\/p>\n
Mode LastWriteTime Length Name
\n—- ————- —— —-
\nd—– 11\/14\/2019 6:47 AM config
\n webserver: Writing configuration
\n webserver: Writing default.toml
\nd—– 11\/14\/2019 6:47 AM hooks
\n webserver: Creating manifest
\n webserver: Building package metadata
\n webserver: Generating package artifact
\n\u00bb Signing C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\cache\\artifacts\\.uship-webserver-0.0.1-20191114064617-x86_64-windows.tar.xz
\n\u2192 Signing C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\cache\\artifacts\\.uship-webserver-0.0.1-20191114064617-x86_64-windows.tar.xz with uship-20190919164651 to create C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\cache\\artifacts\\uship-webserver-0.0.1-20191114064617-x86_64-windows.hart
\n\u2261 Signed artifact C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\cache\\artifacts\\uship-webserver-0.0.1-20191114064617-x86_64-windows.hart.
\n webserver: hab-plan-build.ps1 cleanup
\n webserver:
\n webserver: Source Cache: C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\cache\\src\\webserver-0.0.1
\n webserver: Installed Path: C:\\hab\\studios\\Users–uship–Projects–webserver\\hab\\pkgs\\uship\\webserver\\0.0.1\\20191114064617
\n webserver: Artifact: C:\\hab\\studios\\Users–uship–Projects–webserver\\src\\results\\uship-webserver-0.0.1-20191114064617-x86_64-windows.hart
\n webserver: Build Report: C:\\hab\\studios\\Users–uship–Projects–webserver\\src\\results\\last_build.ps1
\n webserver: SHA256 Checksum:
\n webserver: Blake2b Checksum:
\n webserver:
\n webserver: I love it when a plan.ps1 comes together.
\n webserver:<\/p>\n
<\/p>\n
If everything is successful, the newly-built package will be in the\u00a0results\u00a0directory. Let’s go ahead and push it to the\u00a0Habitat Bldr Service. We can use the\u00a0results\/last_build.ps1\u00a0file to set variables so we don’t need to specify the full path to the artifact. Note that you’ll need to make sure your\u00a0auth token\u00a0is set up.<\/p>\n
PS C:\\Users\\uship\\Projects\\webserver\\results> . .\\last_build.ps1
\nPS C:\\Users\\uship\\Projects\\webserver\\results> hab pkg upload $pkg_artifact
\n 79 B \/ 79 B | [=====================================================================================================================================================================================] 100.00 % 654 B\/s
\n\u2192 Using existing public origin key uship-20190919164651.pub
\n\u2192 Using existing core\/cacerts\/2019.08.28\/20190829172945
\n\u2192 Using existing stuartpreston\/chef-client\/14.11.21\/20190328012639
\n\u2191 Uploading uship-webserver-0.0.1-20191114064617-x86_64-windows.hart
\n 70.89 KB \/ 70.89 KB | [===========================================================================================================================================================================] 100.00 % 1.45 MB\/s
\n\u221a Uploaded uship\/webserver\/0.0.1\/20191114064617
\n\u2261 Upload of uship\/webserver\/0.0.1\/20191114064617 complete.<\/p>\n
<\/p>\n
You should now have a public “webserver” package available in the “unstable” channel of your Habitat origin. In the next part of this blog post series, we’ll build an AMI and deploy our new package to a server using that AMI. If you want to see the code for this, it’s available at\u00a0https:\/\/github.com\/uShip\/effortless_ami_deployments\u00a0and the Habitat package is at\u00a0https:\/\/bldr.habitat.sh\/#\/pkgs\/uship\/webserver.<\/p>\n","protected":false},"excerpt":{"rendered":"
Background At uShip, we’ve been moving to an AMI deployment strategy for standing up web servers that houses our main application. We made the decision as part of a larger strategy to ensure our environments (dev, qa, prod, etc.) were as similar as possible. We figured that if we could build a single AMI that… Read More<\/a><\/p>\n","protected":false},"author":51,"featured_media":14947,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[297],"class_list":["post-14871","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-company-news","tag-shipping-code"],"acf":{"blog_post_content":null},"_links":{"self":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts\/14871","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/users\/51"}],"replies":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/comments?post=14871"}],"version-history":[{"count":0,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts\/14871\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/media\/14947"}],"wp:attachment":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/media?parent=14871"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/categories?post=14871"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/tags?post=14871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}