Postgres password rotated, can no longer connect

[JamesB41]

Summary

Set up Mattermost using the operator and RDS rotated the password, and I'm no longer able to establish a connection.

Steps to reproduce

Chart version is 0.3.5. Running pod for the instance is mattermost/mattermost-enterprise-edition:5.28.0. Home lab install of Mattermost running on RKE2 1.28.4.

I set up an RDS instance using postgres 15.4 on a completely fresh install of Mattermost. Worked great. The values I specified in my secret were (obviously not real data):

DB_CONNECTION_CHECK_URL: postgres://postgres:<actual pw>@<actual host>:5432/mattermost

DB_CONNECTION_STRING: postgres://postgres:<actual pw>@<actual host>:5432/mattermost

Expected behavior

Changing the database password, modifying the secret accordingly, and cycling the pod would allow it to reconnect.

Observed behavior (that appears unintentional)

When I configured my RDS instance, I had set it to use Secrets Manager and rotate the password every 7 days. Whoops. Password rotated on me. No big deal. Went to RDS and changed it to a static value just to get back up and running. I made very, very certain to have no escape-required passwords to eliminate encoding being an issue. I am able to connect to the mattermost database and query everything using the new connection string on the command line. Modified the secret accordingly and cycled the pod, figuring that would be that.

Instead I get this no matter what I do:

mattermost {"level":"info","ts":1702949419.059532,"caller":"sqlstore/supplier.go:232","msg":"Pinging SQL","database":"master"}                                                                                                                                                       
mattermost {"level":"error","ts":1702949419.2492898,"caller":"sqlstore/supplier.go:244","msg":"Failed to ping DB","error":"pq: password authentication failed for user \"postgres\"","retrying in seconds":10}

I made certain encoding wasn't a problem and that it didn't have a trailing \n in the base64 version in the secret, because that's a problem. Eventually I gave up and I just removed the database portion of the postgres URI (i.e. I trimmed off the /mattermost portion). Pod came up and installed a new copy of the Mattermost database in the default "postgres" database instead. Curious.

To test this behavior again, I deployed a completely identical RDS postgres host. Set the secret without the database on the end of the string, and it put it in "postgres" again, no problem. I then changed the connection string to add "/mattermost" on the end and that worked too! But then I changed the password (from literally "mattermost" to "mattermost2") on the RDS side. Updated the secret, rolled the pod, and same error about authentication failing. Changed the password back to "mattermost", reverted the secret change and right back in.

Seems to me that for some reason, changing the database user's password in the secret doesn't seem to have any impact. I can't imagine where that would be cached or persisted, but I'm open to any ideas.

Additional note: When I exec into the pod and look at the environment variables, I do see the updated password so I'm even more confused.

Another note: When I create a brand new database in my first RDS instance called "bob" and I modify the secret to point to '/bob' and leave the password exactly the same, it creates yet another mattermost deployment of the database in "bob", so the credential is definitely right. I just can't re-connect to the initial database named "mattermost".

MM_CONFIG=postgres://postgres:mattermost@<hostname>:5432/mattermost

Possible fixes

n/a