|
| 1 | +# Waffles from Scratch |
| 2 | + |
| 3 | +To illustrate Waffles's design, I'll walk through the creation of a Bash script that can successfully run multiple times on a single server and only make changes to the server when required. |
| 4 | + |
| 5 | +## Let's Create a Simple memcached Server in Bash |
| 6 | + |
| 7 | +`memcached` is a very simple service. It's a single daemon with a simple configuration file installed from a single package. |
| 8 | + |
| 9 | +Let's say we want to create a `memcached` server on a Linux container or virtual machine. Rather than running the commands manually, we'll create a Bash script to do the work. This will serve two purposes: |
| 10 | + |
| 11 | +1. Documentation |
| 12 | +2. Repeatable process |
| 13 | + |
| 14 | +### The First Draft |
| 15 | + |
| 16 | +The initial Bash script would look something like: |
| 17 | + |
| 18 | +```shell |
| 19 | +#!/bin/bash |
| 20 | + |
| 21 | +apt-get install -y memcached |
| 22 | +``` |
| 23 | + |
| 24 | +### The Second Draft |
| 25 | + |
| 26 | +This works, and doing `ps aux` shows that `memcached` is indeed running. But then we notice that `memcached` is only listening on `localhost`: |
| 27 | + |
| 28 | +```shell |
| 29 | +$ netstat nap | grep 11211 |
| 30 | +``` |
| 31 | + |
| 32 | +Since this `memcached` service will be used by other services on the network, we need to change `memcached`'s interface binding to `0.0.0.0`. The following should work: |
| 33 | + |
| 34 | +```shell |
| 35 | +$ sed -i -e '/^-l 127.0.0.1$/c -l 0.0.0.0' /etc/memcached.conf |
| 36 | +$ /etc/init.d/memcached restart |
| 37 | +``` |
| 38 | + |
| 39 | +And once that's tested on the command line, we add it to our script: |
| 40 | + |
| 41 | +```shell |
| 42 | +#!/bin/bash |
| 43 | + |
| 44 | +apt-get install -y memcached |
| 45 | +sed -i -e '/^-l 127.0.0.1$/c -l 0.0.0.0' /etc/memcached.conf |
| 46 | +/etc/init.d/memcached restart |
| 47 | +``` |
| 48 | + |
| 49 | +Astute readers will see an issue. In order for us to test this script, we need to run it again. However, the script is going to report that `memcached` is already installed and an unnecessary restart of `memcached` will take place. |
| 50 | + |
| 51 | +There are two ways to resolve this issue: |
| 52 | + |
| 53 | +The first is by starting over from scratch and running the script on a new server. There's a lot of merit to this method. For example, you can be sure that the exact steps work in sequence on new servers. However, the entire process could take a long time for some situations. Also, what if this `memcached` service was in production? Either you'd have to take the `memcached` service down temporarily while the new service builds or you'd have to find some way of seamlessly adding in the new service while removing the old. While there's benefit to this (which is similar to the current popularity of "microservices"), it may not always be a possible solution. |
| 54 | + |
| 55 | +The second way is to alter the script so that changes are only made if required. If a change does not need to be made, nothing happens. |
| 56 | + |
| 57 | +Let's say it's not possible for us to rebuild from scratch. Therefore, we'll opt for the second option. |
| 58 | + |
| 59 | +### The Third Draft |
| 60 | + |
| 61 | +In order to run our Bash script against a running service without causing (too much of) a disruption, we must ensure that each step is executed only if it needs to be. This means that before any command has run, we must check to see what the current state of the system is, compare it to the change we want to make, and only make the change if the system state does not match. |
| 62 | + |
| 63 | +By doing this, our Bash script becomes a "state declaration" that describes how the `memcached` service should be configured when the script is done running. This is known as [Idempotence](http://en.wikipedia.org/wiki/Idempotence) in Configuration Management. |
| 64 | + |
| 65 | +So let's make our basic Bash script more idempotent: |
| 66 | + |
| 67 | +```shell |
| 68 | +dpkg -s memcached &>/dev/null |
| 69 | +if [ $? == 1 ]; then |
| 70 | + echo "Installing memcached" |
| 71 | + apt-get install -y memcached |
| 72 | +fi |
| 73 | + |
| 74 | +grep -q '^-l 127.0.0.1' /etc/memcached.conf |
| 75 | +if [ $? == 0 ]; then |
| 76 | + echo "Updating memcached.conf and restarting it." |
| 77 | + sed -i -e '/^-l 127.0.0.1$/c -l 0.0.0.0' /etc/memcached.conf |
| 78 | + /etc/init.d/memcached restart |
| 79 | +fi |
| 80 | +``` |
| 81 | + |
| 82 | +With this in place, we can now execute this script multiple times on the same server, virtual machine, or container, and if a step has already been done it will not happen again. |
| 83 | + |
| 84 | +### The Fourth Draft |
| 85 | + |
| 86 | +Having to do a bunch of `grep`s and other checks can become very tedious. Waffles tries to resolve this by including a Standard Library of common tasks. Using the Waffles Standard Library, the above script can be re-written as: |
| 87 | + |
| 88 | +```shell |
| 89 | +#!/bin/bash |
| 90 | + |
| 91 | +source ./waffles.conf |
| 92 | +source ./lib/init.sh |
| 93 | + |
| 94 | +stdlib.apt --package memcached |
| 95 | +stdlib.file_line --name memcached.conf/listen --file /etc/memcached.conf --line "-l 0.0.0.0" --match "^-l" |
| 96 | +stdlib.sysvinit --name memcached |
| 97 | + |
| 98 | +if [ "$stdlib_state_change" == true ]; then |
| 99 | + /etc/init.d/memcached restart |
| 100 | +fi |
| 101 | +``` |
| 102 | + |
| 103 | +There's nothing magical about these commands. They're a collection of standard Bash functions that sweep all of the messy `grep`s under the carpet. You can see the full collection of Standard Library functions in the `lib/` directory. |
| 104 | + |
| 105 | +### The Fifth Draft |
| 106 | + |
| 107 | +The core `memcached` service is up and running, but there's still a few more tasks that need to be done. For example, maybe we want to create some users: |
| 108 | + |
| 109 | +```shell |
| 110 | +stdlib.groupadd --group jdoe --gid 999 |
| 111 | +stdlib.useradd --user jdoe --uid 999 --gid 999 --comment "John" --shell /bin/bash --homedir /home/jdoe --createhome true |
| 112 | +``` |
| 113 | + |
| 114 | +`stdlib.useradd` is another Waffles Standard Library function that enables an easy way to create and manage a user on a server. |
| 115 | + |
| 116 | +Looking at the above command, there are a lot of settings that are hard-coded. If we end up creating a `redis` server that also needs the `jdoe` user, we could just copy that line verbatim, but what about a scenario where the `uid` must be changed to `500`? Then we'd need to change every occurrence of `999` to `500`. In large environments, there's a chance some changes would be missed. |
| 117 | + |
| 118 | +To resolve this issue, Waffles allows settings such as this (known as _data_) to be stored in data files. |
| 119 | + |
| 120 | +A simple way of using data is to just throw all settings into a file called `site/data/common.sh`. |
| 121 | + |
| 122 | +Let's add a user: |
| 123 | + |
| 124 | +```shell |
| 125 | +data_users=( |
| 126 | + jdoe |
| 127 | +) |
| 128 | + |
| 129 | +declare -Ag data_user_info |
| 130 | +data_user_info=( |
| 131 | + [jdoe|uid]=999 |
| 132 | + [jdoe|gid]=999 |
| 133 | + [jdoe|comment]="John doe" |
| 134 | + [jdoe|homedir]="/home/jdoe" |
| 135 | + [jdoe|shell]="/bin/bash" |
| 136 | + [jdoe|create_home]=true |
| 137 | +) |
| 138 | +``` |
| 139 | + |
| 140 | +Waffles data variables can be named anything, but if you want to follow the project standards, have the variables start with `data_`. |
| 141 | + |
| 142 | +With all of this in place, the fifth draft now looks like: |
| 143 | + |
| 144 | +```shell |
| 145 | +#!/bin/bash |
| 146 | + |
| 147 | +source ./waffles.conf |
| 148 | +source ./lib/init.sh |
| 149 | + |
| 150 | +stdlib.data common |
| 151 | + |
| 152 | +for user in "${data_users[@]}"; do |
| 153 | + |
| 154 | + homedir="${data_user_info[${user}|homedir]}" |
| 155 | + uid="${data_user_info[${user}|uid]}" |
| 156 | + gid="${data_user_info[${user}|gid]}" |
| 157 | + comment="${data_user_info[${user}|comment]}" |
| 158 | + shell="${data_user_info[${user}|shell]}" |
| 159 | + create_home="${data_user_info[${user}|create_home]}" |
| 160 | + |
| 161 | + stdlib.groupadd --group "$user" --gid "$gid" |
| 162 | + stdlib.useradd --state present --user "$user" --uid "$uid" --gid "$gid" --comment "$comment" --homedir "$homedir" --shell "$shell" --createhome "$createhome" |
| 163 | + |
| 164 | +done |
| 165 | + |
| 166 | +stdlib.apt --package memcached |
| 167 | +stdlib.file_line --name memcached.conf/listen --file /etc/memcached.conf --line "-l 0.0.0.0" --match "^-l" |
| 168 | +stdlib.sysvinit --name memcached |
| 169 | + |
| 170 | +if [ "$stdlib_state_change" == true ]; then |
| 171 | + /etc/init.d/memcached restart |
| 172 | +fi |
| 173 | +``` |
| 174 | + |
| 175 | +### The Sixth Draft |
| 176 | + |
| 177 | +The block of user data can be re-used in other scripts. It'd be best if we just moved it out into its own separate script. By repeating this process, we can create a library of re-usable components. Final scripts then become "compositions" of the collection of scripts. |
| 178 | + |
| 179 | +Create the directory structure `site/profiles/common/scripts` and add the following to `site/profiles/common/scripts/users.sh` |
| 180 | + |
| 181 | +```shell |
| 182 | +for user in "${data_users[@]}"; do |
| 183 | + |
| 184 | + homedir="${data_user_info[${user}|homedir]}" |
| 185 | + uid="${data_user_info[${user}|uid]}" |
| 186 | + gid="${data_user_info[${user}|gid]}" |
| 187 | + comment="${data_user_info[${user}|comment]}" |
| 188 | + shell="${data_user_info[${user}|shell]}" |
| 189 | + create_home="${data_user_info[${user}|create_home]}" |
| 190 | + |
| 191 | + stdlib.groupadd --group "$user" --gid "$gid" |
| 192 | + stdlib.useradd --state present --user "$user" --uid "$uid" --gid "$gid" --comment "$comment" --homedir "$homedir" --shell "$shell" --createhome "$createhome" |
| 193 | + |
| 194 | +done |
| 195 | +``` |
| 196 | + |
| 197 | +And so the sixth draft now looks like: |
| 198 | + |
| 199 | +```shell |
| 200 | +#!/bin/bash |
| 201 | + |
| 202 | +source ./waffles.conf |
| 203 | +source ./lib/init.sh |
| 204 | + |
| 205 | +stdlib.data common |
| 206 | +stdlib.profile common/users |
| 207 | + |
| 208 | +stdlib.apt --package memcached |
| 209 | +stdlib.file_line --name memcached.conf/listen --file /etc/memcached.conf --line "-l 0.0.0.0" --match "^-l" |
| 210 | +stdlib.sysvinit --name memcached |
| 211 | + |
| 212 | +if [ "$stdlib_state_change" == true ]; then |
| 213 | + /etc/init.d/memcached restart |
| 214 | +fi |
| 215 | +``` |
| 216 | + |
| 217 | +You can create this script inside the Waffles directory (where `waffles.conf` is located), and run it like so: |
| 218 | + |
| 219 | +```shell |
| 220 | +bash test.sh |
| 221 | +``` |
| 222 | + |
| 223 | +When you run it for the first time on a new server, it'll add the group, user, and set up `memcached`. Run it multiple times and note how those same actions were not performed since the script detected that no changes needed to be made. |
| 224 | + |
| 225 | +## Conclusion |
| 226 | + |
| 227 | +At this point, we've effectively recreated the core of Waffles. The rest of controls how Waffles runs and where to find various files that Waffles needs to read. |
0 commit comments