private keys still showing up in log despite indicating sensitive_var_conn_names and custom masking doesn't support wildcards

[dasphillipbrau]

I'm experiencing this very annoying issue where airflow is configured to mask sensitive info from key names such as private_key (there is both a list of names to mask and hide_sensitive_var_conn_fields is set to true and yet task logs are printing 'private_key': 'foo'

I tried using the mask_secret method in the docs linked above. with the following results.
If I just pass the name of the key, only the key gets masked, not the value, so mask_secret("private_key") only results in '***':'foo' which of course doesn't do much if we assume foo is actually a private key.

If I use mask_secret({'private_key':'.+'}) then unfortunately nothing happens, because the implementation of the method escapes the value before converting it into a regex pattern, so instead of getting a wildcard pattern .+ I get a string that's literally \\.\\+

Is there any reason why the provided value is ALWAYS escaped? I don't see much utility in needing to explicitly pass the exact value that you want to mask instead of just giving it a key name and masking whatever its value is.

[potiuk]

You misunderstood it. You should use "mask_secret(SECRET_VALUE)". You have to explicitly add the SECRET you want to msl as masked value.

Taking your example. you should call mask_secret("foo") (where foo is the content of the secret you want to replace with *** when logged.

[dasphillipbrau]

But I would say it's still very restrictive to not allow the user to write a pattern, especially because you can further specify a specific step where you want the masking to occur. As I said if you have a key in the logged json called private_key_id and you pass the dictionary {'private_key_id': '.+'} it would mask private_key_id but not it's value, because it would look for a string that literally contains ".+".

I just feel it would be much more useful to allow the user to pass a pattern, because it seems like a really bad idea (and also not practical) to write the whole secret in order to mask it in the logs.

This is being a headache for us because the GCP operators print all this sensitive info whenever you explicitly pass a connection ID other than google_cloud_default

[potiuk]

It really depends how you log your data. The "sensitive_var_conn_fields" - work this way if you pass logger the structured data that the logger can traverse and find out the key. You just need to set the field name there that you should hide. For example you have connection extra names "prvatke_key_id" - you can add `private_key_id" to the list and it's value will be hidden. This is how this variable works. so maybe in your case you are simply not logging structured data that can be redacted but you convert things to string before logging it.

Thisis where "mask_secret" comes in. the reason it works the way it works is that if you do not have structured data, you have no idea that particular value is part of actual structure - because it is a string. And you want to mask this string you just specify the string to be masked - because it can be done fast. Regexp matches like that are way slower and it is important for logs to be filtered as fast as possible - and because masking value will remove it regardless of whether it has been printed to the logs as part of dictionary or standalone, or maybe as part of another structure.

But you have an idea that you would like to add a regular expression to be masked, then feel free to implement it. We rejected that idea because of performance and the fact that it is something that is notoriously easy to get wrong (there is the famous saying - "if you have problem, introduce regexp and you will have two problems"). But maybe you can prove we were wrong.

You can implement and start using such masker even now.

Following https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html you can extend standard logging configuration, extend SecretsMasker or write your own and add such option - SecretsMasker is nothing else that standard Python Logging Filter and advanced logging allows you to extend the logging configuration as you see fit. Then you coulkd optimise it, test the performance and if you find it working well for you - maybe you can even contribute it back to airlfow - you could become one of > 2500 contributors who - like you - missed something, added it, and contributed it back (this is how airflow is created).