Prevent Unencrypted Ansible Vaults from Being Pushed to Git

Ansible Vault is a nice tool that allows you to store sensitive data (such as passwords and application secrets) securely along with your Ansible Playbooks, so you have all your configuration in a single place. Obviously, you don’t want to store unencrypted secrets in your repository, so that’s why Ansible Vault encrypts them with AES-256 encryption by default.

However, a big drawback is that it can easily happen that people forget to re-encrypt the vault file after editing it. You can argue that this shouldn’t happen if you use the ansible-vault edit command: this decrypts the file, opens it in your default editor (based on the $EDITOR environment variable), and takes care of re-encrypting it for you when you’re done editing. When making a lot of changes though, it can be easier to decrypt it with ansible-vault decrypt, load it in your favorite GUI editor, and the manually re-encrypt it when done. Unfortunately, in this case, no one will remind you to perform the last step. Everything will continue to work just fine. This is because your vault is just a regular YAML file that Ansible will accept as a source of configuration variables—encrypted or not.

What Can Go Wrong, Will Go Wrong

Recently, I found an unencrypted Ansible Vault file in one of our Git repositories at work that contained lots of sensitive data. It turned out that four(!) months earlier, someone forgot to re-encrypt it after adding a new private key to it. No one noticed it because all deployment processes continued to work.

So whenever something bad happens, I try to think of ways how to improve the workflow to prevent it from happening again. In my point of view, it won’t help to indoctrinate people to never, ever do it again. Instead, I see an incident like this as a flaw in system: it should not have allowed this mistake to happen in the first place.

Git Hooks to the Rescue

So why don’t we create a Git Hook that ensures that the vault is properly encrypted, and if it’s not, rejecting the commit? I’m certainly not the first one having this idea. As you will notice when searching Google, most people are suggesting a pre-commit hook to tackle this issue. A big advantage of this approach is indeed that the commit is checked on the client, before it is even leaving the developer’s machine. This sounds like what we want, but there is a drawback: We cannot enforce a client-side Git hook. A developer cloning the Git repository will manually need to enable the pre-commit hook before it will be executed, which, again, is something that can be easily forgotten.

The pre-receive Hook

So in addition to a client-side hook, we also want to have a server-side hook as a last resort.

This could look like this:

This will reject the push if it contains files called credentials.yml or matching *vault*.yml. You may want to extend or change this pattern to the file names your team is using.

This hook has been battle-tested for more than a month now and will work with paths or filenames containing spaces as well as submodules. Those were two cases I ran into after creating the hook based on some examples I found on GitHub.

Installation for GitLab

To install this hook globally on your GitLab server, store the check-vault-encryption script in the /etc/gitlab/hooks/pre-receive.d folder and ensure it is executable.

When done, try pushing something to the server. You should see the following output:

aaron/test - [master] » git push
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 647 bytes | 647.00 KiB/s, done.
Total 5 (delta 1), reused 0 (delta 0)
remote: [vault-check] Looking for unencrypted Ansible Vault files
To www.example.com:aaron/test.git
   c03c2b4..fb005d3  master -> master

And in case you tried to commit something that looks like an unencrypted vault file, it will reject your commit:

aaron/test - [master] » git push
Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 254 bytes | 254.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)
remote: [vault-check] Looking for unencrypted Ansible Vault files
remote: ERROR: credentials.yml must be encrypted
To www.example.com:aaron/test.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:aaron/test.git'

Caveats

This approach isn’t perfect. We should keep the following in mind:

• While this will prevent any file matching the specified pattern from being added to the Git repository, the file will still be temporarily stored on the Git server (unencrypted, of course) while it is being checked. This can be a problem.
• Depending on the size of the data being pushed, the check can take a couple of seconds to complete, delaying all pushes (also those that don’t contain files matching the pattern).
• This hook relies on the file names. If someone doesn’t adhere to the naming convention, the file won’t be checked. An improvement to this hook could a heuristic that looks for certain patterns within any file being pushed. However, this would further slow down the delay on every push.