base64 each line of a file

Requirements are to parse a file to:

• base64 encode the values (right of the =)
• treat values verbatim, AKA handle:
  • empty values
  • dollar signs
  • quotes
  • equals signs
  • backslashes (could be interpreted as escape sequence)
  • don’t add a newline

Here’s our test file (input.env):

AAAA=
BBBB=abc\ndef
CCCC=tuv$xyz
DDDD="with quotes"
EEEE=has-another=equals

Here’s the program:

# using: GNU awk 5.2.2, paste 9.3, sed 4.9, xargs 4.9.0
keysFile=$(mktemp)
sed "./input.env" \
  -e 's/=/\t/' \
  -e 's/\t.*//' \
  > "$keysFile"

valsFile=$(mktemp)
sed "./input.env" \
  -e 's/=/\t/' \
  -e 's/.*\t//' \
  | xargs -d'\n' -I'{}' bash -c "[ -z '{}' ] && echo || { echo -n '{}' | base64; }" \
  > "$valsFile"

finalFile=$(mktemp)
paste -d'=' "$keysFile" "$valsFile" > "$finalFile"
rm -f "$keysFile" "$valsFile"
cat "$finalFile"

General approach:

• for each line in the file (using sed), convert the first = to a tab character. There’s no g at the end of the regex so we stop at the first match. If we tried to use s/.*=/... it would be greedy so if the value contains an equals, things would go wrong. We could also try to limit the chars before the =, like [A-Z_] but you need to know all the possible chars.
• either grab everything up to, or after, the tab character
• for the keys file, just write the keys to a file
• for the values file
  • we use xargs to run a command for each line in the input
  • specifying the -d'\n flag means (from the man page) “every character in the input is taken literally”. The default behaviour also splits on newlines but would treat quotes and escapes specially. Using this flag also means empty lines are passed through. Without, xargs collects the required number (1 for us) of non-empty vals.
  • specifying the -I'{}' flag means we can drop the value into the command string mutliple times.
  • check if the value is empty, and if so, write a newline
  • if the value is not empty, send it to base64 without a newline. The base64 command will write a newline after the encoded output for us.
• join the two files together on the = char. The join command uses a field to join on, which isn’t what we want, so we use paste which joins line 1 of the first file to line 1 of the second file, and so on.

The output is

AAAA=
BBBB=YWJjXG5kZWY=
CCCC=dHV2JHh5eg==
DDDD=IndpdGggcXVvdGVzIg==
EEEE=aGFzLWFub3RoZXI9ZXF1YWxz

You can check the values are correct with echo YWJjXG5kZWY= | base64 -d and confirm they’re all identical to the input.

I was worried doing echo "$something" would run into issues with escaping, quotes, spaces, interpolation and all those other fun shell things. I search for a way to get each line of input to be fed to stdin of base64 and the closest I found was base64 <<< '{}', but it turns out the echo approach above works fine, so there was no need to stress about that.