Encrypting secrets and committing them seems very convenient but I'm paranoid about these sorts of things. Can anyone tell me why this would be a bad idea?
One reason I can think of is that normally with secrets I actually don't keep any copies of them. I just set them in whatever secret manager my cloud environment uses and never touch them again unless I need to rotate them. Meaning there is no way to accidentally expose them other than by the secret vault being hacked or my environment being hacked.
With this approach if someone gets access to the encryption key all secrets are exposed.
The biggest issue with storing secrets in version control with the code is that past secrets are never relevant after they have been rotated. This makes rollbacks risky. Consider:
1. Create secret v1
2. Code v1
3. Deploy
4. Secret v2 (rotation)
5. Code v2
6. Deploy
7. Oops, need to roll back to v1 (from step 2)
8. Outage, because the secrets in step 2 are not the secrets from step 4
Inserting deployments between steps 1 and 2 and between steps 4 and 5 still results in an outage when the deploy for code v2 needs to be rolled back to the older commit, not the commit before. Arguing that rollbacks means you only ever need to rollback to commit #n-1 is like arguing you should never retain more than one backup; there's always a chance that you need to rollback even further to fix things.
I disagree. Using vaults isn’t that bad. And I’d also like developers to never actually know the secrets.
It’s a learning curve, but I think it’s best to just bite the bullet and use a vault rather than trusting developers to know and manage secrets properly.
Another benefit is that debugging secret changes is a lot easier. We've had a couple of cases where someone changes the secret in a vault and that causes problems and no one can tell what changed between two deploys
I've used git-crypt and sealed-secrets and the problem is always backing up the master key. sealed-secrets rotates it every so often so you need to go find it and copy it to 1password or whatever you use as your root of trust. (We used a calendar invite for this.)
git-crypt is easy, the master key doesn't rotate, so don't leak it. (Secret encryption key rotation is kind useless; it's nice that if you leak an old key newer secrets aren't leaked, but it depends on your underlying secret rotation policy as to whether or not that saves you any work. I have tended to do them in bulk in the past.)
On my last project we did disaster recovery exercises every 6 months, sometimes with the master key intentionally lost, and it wasn't that big of a deal. Restoring your infra-as-code involves creating the secret encryption service manually, though, which is kind of a pain, but not like days of downtime pain or anything. Of course, if the secrets encrypted your database or something like that, then losing the master key equals losing all your data. Hopefully your database backup has some solution for that problem.
One reason I can think of is that normally with secrets I actually don't keep any copies of them. I just set them in whatever secret manager my cloud environment uses and never touch them again unless I need to rotate them. Meaning there is no way to accidentally expose them other than by the secret vault being hacked or my environment being hacked.
With this approach if someone gets access to the encryption key all secrets are exposed.