Use with git-filter config - filter.*.clean and filter.*.smudge

[jinnko]

I've been able to integrate sops with git such that files are decrypted/encrypted on checkout/commit. This was achieved like this:

1. Set up git-filter config

   git config --local filter.sops-json.clean "sops --input-type json --output-type json --encrypt /dev/stdin"
   git config --local filter.sops-json.smudge "sops --input-type json --output-type json --decrypt /dev/stdin"
   git config --local filter.sops-json.required true
   

2. Set up .gitattributes to pass files through the filter

   *.json filter=sops-json diff=sops-json
   

3. Have a .sops.yaml configuration with default creation_rules:

   creation_rules:
     - kms: arn:aws:kms:...:...:key/2305235902
   

Checkout and commit work well. Unfortunately the files are always considered changed, I believe because the IV is new on every pass.

Is it necessary for the IV to be ephemeral? Is there a way the random IV could be avoided so this workflow is viable - i.e. so the file isn't always marked as modified by git?

[mtoohey31]

This is the best I could come up with:

#!/usr/bin/env -S bash -euo pipefail

# we need $1 to be the path of the file so we can check the previous version
# via git-show to prevent the encryption's non-determinism from resulting in
# unnecessary changes
if test $# -ne 1; then
  echo "Usage: $0 FILE" >&2
  exit 1
fi

if ! git cat-file -e "HEAD:$1" &>/dev/null; then
  # if git cat-file -e fails, then the file doesn't exist at HEAD, so it's new,
  # meaning we need to encrypt it for the first time
  echo "$0: no previous version found while cleaning $1" >&2
  sops --input-type binary --output-type binary --encrypt /dev/stdin

  # TODO: figure out a better way to open fd 3
elif exec 3< <(echo -n) && diff \
  <(git cat-file -p "HEAD:$1" | sops --input-type binary --output-type binary --decrypt /dev/stdin) \
  <(cat /dev/stdin | tee /dev/fd/3) >/dev/null; then
  # if there's no difference between the decrypted version of the file at HEAD
  # and the new contents, then we re-use the previous version to prevent
  # unnecessary file updates
  echo "$0: no changes found while cleaning $1" >&2
  git cat-file -p "HEAD:$1"
else
  # if there is a difference then we re-encrypt it from fd 3, where we
  # duplicated stdin to
  echo "$0: found changes while cleaning $1" >&2
  sops --input-type binary --output-type binary --encrypt /dev/fd/3
fi

Basically, what this does is check if a previous version exists, and if it does, compares the decrypted contents of the previous version with the latest version being fed to stdin. If there's no difference, then it re-uses the old version. Otherwise, it re-encrypts the file.

If you swap out your clean command to be the path to this script, plus a %f, then you shouldn't get unecessary update commits. For example, something like the following:

[filter "sops"]
	required = true
	smudge = sops --input-type binary --output-type binary --decrypt /dev/stdin
	clean = scripts/git-filter-sops-clean %f

You might want to swap out the *-type binary arguments for JSON if that's what your use-case requires.

If someone has pointers on how I can resolve that TODO about the wierd opening of fd 3, please let me know. I don't really know what I'm doing with bash file descriptors.

Edit: just realized, this won't behave properly if you change the keys without changing the file contents, so keep that in mind.

[archite]

Thanks, @mtoohey31. I was trying to get something working for age and your script was the answer. I did away with creating file descriptors base populating a variable with stdin. This would work equally well with sops which I'm also using.

#!/usr/bin/env bash

PS4='${LINENO}: '

set -euo pipefail

# Exit if no file given
test $# -eq 1

# Exit if no stdin
test -t 0 && exit 1

decrypt() {
  age -d -i ~/.config/sops/age/keys.txt
}

encrypt() {
  age -r someagekey -a
}

show() {
  printf "%s\n" "${@}"
}

INPUT=$(cat)
: ${ENCRYPTED:=$(encrypt <<<${INPUT})}
: ${CONTENTS:=$(git cat-file -p "HEAD:${1}" 2>/dev/null)}
: ${DECRYPTED=$(decrypt <<<${CONTENTS} 2>/dev/null)}

if [[ -z "${CONTENTS}" || "${DECRYPTED}" != "${INPUT}" ]]
then
  show "${ENCRYPTED}"
else
  show "${CONTENTS}"
fi

[prskr]

I built a prototype for something similar based on age: git-age. If anyone is brave enough to give it a try I'd highly appreciate feedback 😄

I only started last week and I intentionally built a rough PoC without any tests (so far) but I hope I can clean out the rough edges "soon" ™️

[archite]

@prskr looks interesting. I'll checkout when I have some free cycles!

[bphenriques]

Was someone able to use sops --input-type json --output-type json --encrypt /dev/stdin with a --filename-override ?

It seems to be a illegal argument if first argument:

$ cat Makefile | sops --input-type binary --filename-override Makefile --encrypt /dev/stdin
...
FATA[0000] flag provided but not defined: -filename-override 

But seems to accept, but ignore when last:

$ cat Makefile | sops --input-type binary --encrypt /dev/stdin --filename-override Makefile

creation_rules:
  - path_regex: Makefile
    key_groups:
      - age:
          - *desktop
          - *bphenriques

Does not seem to possible 🤔 I am trying to make this work with path_regex as I am using in my dotfiles to setup different machines with different sets of secrets.

[felixfontein]

@bphenriques --filename-override is only availae on the main branch, not yet in any release. It will appear in the 3.9.0 release (whenever that will happen). (Feature added in #1332.)

But seems to accept, but ignore when last:

$ cat Makefile | sops --input-type binary --encrypt /dev/stdin --filename-override Makefile

Basically sops [some opions here] filename ignores everything after the filename. That's an unfortunate result of how the argument parsing currently works. The next release (3.9.0) will print some warnings in case something is found after the filename. (Warning added in #1342.)

[bphenriques]

The following worked for me:

• smudge: sops decrypt --input-type json --output-type binary --filename-override %f /dev/stdin
• clean: sops encrypt --input-type binary --output-type json --filename-override %f /dev/stdin

[kantum]

How would you updatekeys in this setup?

I've spend too much time trying to setup this, I will go back to a more manual thing 😢

I would love to have an official solution using git-filter.

[bphenriques]

How would you updatekeys in this setup?

I've spend too much time trying to setup this, I will go back to a more manual thing 😢

I would love to have an official solution using git-filter.

You can use age-keygen to generate a pair:

• Put the private key under ~/.config/sops/age/keys.txt (if I remember the path correctly, albeit you can override the path using a environment variable) - to decrypt the content.
• Put the public key under sops.yaml - to encrypt the content.

I no longer use this setup as I was using it to encrypt confidential information in my nix setup (I opted to use private repos). For reference this is what I had working:

• The script. edit: the path in the init() method in lines 25 and 26 should be ./bin/sops-git-filter.sh
• The sops.yaml
• The .gitattributes.

Hope it helps.

Edit: Oh, updateKeys, sorry lack of ☕ . I can try again, but it should't be too tricky. Perhaps you can:

• Introduce another key pair
• Re-encrypt (using the two keys)
• Remove the old one

I think I did that but I dont remember the details.

[prskr]

One thing to keep in mind when talking about rotating keys is that this will obviously only work for everything committed after the rotation but it won't affect anything in the history unless you rewrite it of course.

I guess for most this is obvious but better be sure 😅 if you're rotating because you want to revoke someone's access you also have to rewrite or it will only affect changes after you revoked the key.

Apart from that the 'tricky' part is really only to encrypt all files again with the new set of keys.

[bphenriques]

One thing to keep in mind when talking about rotating keys is that this will obviously only work for everything committed after the rotation but it won't affect anything in the history unless you rewrite it of course.

I guess for most this is obvious but better be sure 😅 if you're rotating because you want to revoke someone's access you also have to rewrite or it will only affect changes after you revoked the key.

Apart from that the 'tricky' part is really only to encrypt all files again with the new set of keys.

➕

That is actually a very good point. I was quite concerned as what I was encrypting is not really "rotatable" (e.g., bookmarks and wallpapers), therefore I opted out as there is no real advantage in rotating the keys as the old files will still be encrypted using the old one:

• Makes it harder for me to go back to a previous point in time as I lost the private key by then (faced that issue)
• If anyone finds a way to decrypt keys they can go back to my history and decrypt the same file which today is encrypted using another key.

[alcroito]

Thanks to @archite's script, and the other commenters (and looking at other projects like https://github.com/uw-labs/strongbox) , I was able to come up with a slightly modified script that handles newlines better, and should also show the diff in git show.

I haven't tested it much yet, but it seems working so far.

Might be useful for future travellers.

It uses sops + age to encrypt just dotenv files in the repo.

Unfortunately the .git/config needs to be adjusted for each checked out repo, because it's not committable afaik.

One also needs to export

export SOPS_AGE_KEY_FILE=/path/to/age/key.txt

So encryption works.

I do wish something like git-crypt + sops + age existed out of the box.

File contents below.

$ cat .git-sops.sh

#!/usr/bin/env bash

PS4='${LINENO}: '

set -euo pipefail

# Exit if the operation and file names were not given
test $# -eq 2

# Exit if no stdin available.
# stdin is used to fed sops with the content to encrypt / decrypt.
test -t 0 && exit 1

# First arg passed to script.
# clean is meant to call sops encrypt
# smudge is meant to call sops decrypt
OP="${1}"

# Second arg passed to script.
# The file name is fed to sops --filename-override so that sops can apply the creation_rules
# based on .sops.yaml file in the root of the repo.
FILE_NAME="${2}"

decrypt() {
  #printf "begin decrypt\n" 1>&2
  sops --decrypt --filename-override "${FILE_NAME}" /dev/stdin
  #printf "end decrypt\n" 1>&2
}

encrypt() {
  sops --encrypt --filename-override "${FILE_NAME}" /dev/stdin
}

show() {
  printf "%s\n" "${@}"
}

# Gets the unencrypted content of stdin, without stripping the last newline.
INPUT=$(cat; echo x)
INPUT=${INPUT%x}

#printf "INPUT contains\n'%s'\n\n" "${INPUT}" 1>&2

# If OP is equal to smudge, just decrypt the stdin contents.
if [[ "${OP}" == "smudge" ]]
then
  #printf "OP is smudge / decrypt\n\n" 1>&2
  TMP=$(mktemp)
  : ${DECRYPTED=$(decrypt <<< "${INPUT}" 2> "$TMP" )}
  err=$(cat "$TMP")
  rm "$TMP"
  wrong_key_error_message="age: no identity matched any of the recipients"
  if [[ $err == *"${wrong_key_error_message}"* ]]; then
    #printf "%s" "${err}" 1>&2
    :
  else
    show "${DECRYPTED}"
  fi

  exit 0
fi

# If the OP is not "clean", exit
if [[ "${OP}" != "clean" ]]
then
  exit 1
fi

# The OP is "clean".
# Either the file was not committed yet, or the existing decrypted content is different
# from the input, in which case we output the new encrypted input.
# If the file was commited and its decrypted content is the same as the new input,
# output the old encrypted content.

#printf "OP is clean / encrypt\n\n" 1>&2

: ${NEW_ENCRYPTED_INPUT:=$(encrypt <<< "${INPUT}" 2>/dev/null )}

: ${ENCRYPTED_HEAD_CONTENTS:=$(git cat-file -p "HEAD:${FILE_NAME}" 2>/dev/null)}

# Bash command expansion trims trailing newlines. We add an "x" to the end of the string
# and then filter it out to preserve the last newline.
# This is not applied to all the cases, because sops is inconsistent in that it always ensures
# a newline, if one was not present before.
: ${DECRYPTED_HEAD_CONTENTS=$(decrypt <<< "${ENCRYPTED_HEAD_CONTENTS}" 2>/dev/null; echo x )}
DECRYPTED_HEAD_CONTENTS=${DECRYPTED_HEAD_CONTENTS%x}

#printf "current  ENCRYPTED_HEAD_CONTENTS contains\n'%s'\n\n" "${ENCRYPTED_HEAD_CONTENTS}" 1>&2
#printf "current  DECRYPTED_HEAD_CONTENTS contains\n'%s'\n\n" "${DECRYPTED_HEAD_CONTENTS}" 1>&2
#printf "previous NEW_ENCRYPTED_INPUT contains\n'%s'\n\n" "${NEW_ENCRYPTED_INPUT}" 1>&2

if [[ -z "${ENCRYPTED_HEAD_CONTENTS}" || "${DECRYPTED_HEAD_CONTENTS}" != "${INPUT}" ]]
then
  show "${NEW_ENCRYPTED_INPUT}"
else
  show "${ENCRYPTED_HEAD_CONTENTS}"
fi

cat .gitattributes

*.env filter=git-sops diff=git-sops

$ cat .git/config

...
[filter "git-sops"]
	; clean is meant to encrypt
	; smudge is meant to decrypt
	clean = ./.git-sops.sh clean %f
	smudge = ./.git-sops.sh smudge %f
	required = true
[diff "git-sops"]
	textconv = cat

$ cat .sops.yaml

cat .sops.yaml
creation_rules:
    - path_regex: \.env$
      age: >-
        MY_RECIPIENT_ID

[austinbutler]

@alcroito that works for you even on a fresh clone with existing encrypted files? For me I end up with a modified file that shows as the encrypted SOPS JSON instead of the expected decrypted file contents.

EDIT: I'm not totally clear on it all yet, but whether any of these solutions are "working" on a fresh clone is complicated by the fact that not all Git tools actually use/implement Git filters, for example in my case gitui (extrawurst/gitui#1468 and extrawurst/gitui#1089) and gitsigns.nvim (lewis6991/gitsigns.nvim#480).

[alcroito]

@alcroito that works for you even on a fresh clone with existing encrypted files? For me I end up with a modified file that shows as the encrypted SOPS JSON instead of the expected decrypted file contents.

EDIT: I'm not totally clear on it all yet, but whether any of these solutions are "working" on a fresh clone is complicated by the fact that not all Git tools actually use/implement Git filters, for example in my case gitui and gitsigns.nvim.

I created a new repo, created an .env file, encrypted it manually with sops, committed the file. At this stage both the file contents on-disk and in the repo and in the diff is the encrypted content.

Then I set up .git/config, .gitattributes and .git-sops.sh, followed by rm .env && git checkout -- .env, as described at
https://github.com/uw-labs/strongbox?tab=readme-ov-file#existing-project

This reruns the clean filter on the existing file, storing the plain-text on disk, while keeping it encrypted in the git repo.
Committing a new commit with the file added doesn't show the plain-text diff. But any subsequent modifications and commits show the plain-text diff.

And indeed, the diff is only plain-text for CLI tools like git show and tig.
Other tooling that shows diffs need to delegate to the CLI to ensure it works, or reimplement the filter feature and the textconv feature.