Allow Airflow to run in FIPs enabled environment

[longslvr]

Currently, we are running a forked Airflow 1.10.15 with additional changes to make Airflow runnable in FIPs enabled production environment.

With fips enabled you can't use md5 hash. Related to the issues mentioned here. #14966

This is proven to be very problematic whenever we want to upgrade the airflow version and we would have to reapply the changes with a fresh fork of Airflow.

To avoid the hassle of forking every time we want to upgrade airflow, we can submit a PR to apply the changes we make so that Airflow is runnable in FIPs-enabled environments. Let me know if anyone has any thoughts.

[potiuk]

I believe (please correct me if I am wrong, pointing to specific FIPS requirements), FIPS mandates not using some algorithms if they are used for cryptography, but is does not tell you that it is forbidden to use certain algorithms if they are not used for security.

Even python 3.9 introduced a flag "usedforsecurity" when hashlib is used:

https://docs.python.org/3/library/hashlib.html

Changed in version 3.9: All hashlib constructors take a keyword-only argument usedforsecurity with default value True. A false value 
allows the use of insecure and blocked hashing algorithms in restricted environments. False indicates that the hashing algorithm is not 
used in a security context, e.g. as a non-cryptographic one-way compression function.

I think md5 in general is so universally used for many non-security context, that getting rid of it completely is a bad idea.

So there is nothing wrong to use md5 hash if it is not used in security context. I believe in any kind of certification you can mark certain usages of certain algorithms for security.

We are still Airlfow 3.7 and there is no "useforsecurity" and also we are using md5 probably in some javascript. I am not aware of any "security" context where we would use it, however if you are using it for security, then yeah, it should be changed. But I see no reason why we should change the algorithm if we are not using it in security context.

In this context, I would be very open (but of course it should be revidwed by others) if you provide a PR (or several of them), but I tihnk it should be not straight replacing md5 use but make it more "pluggable" and prepared for "useforsecurity" parameter once we go Python 3.9+.

I think the sequence/scope of implementation shoudl be:

• inventory of all usages and extracting them to a common "util" library
• adding "useforsecurity" parameter in our util, anticipating that we will replace the util in Python 3.9+ directly (and we should actually pass this parameter to hashlib when we are on 3.9+. Likely similar approach for javascript.
• if you have some specific requirements of not using md5 at all, you could add an option to use other algorithms instead in this "util" of ours. I understand there might be some hardware/environment limits (which useforsecurity and 3.9+ should generally address) but maybe it's not perfect
• the behaviour should be controlled by a configuration parameter on a global level if added (security/fips_compliance).

I think if you do it in fully backwards-compatible way so that nothing changes for the users, this should be possible.

[longslvr]

@potiuk you are correct about the FIPS requirement. Our change is mainly to use "useforsecurity=False" to still use MD5 and introduce a CACHE_OPTIONS in flask_caching to use SHA256 instead of MD5.
Thank you for the suggestion on the changes and I think that is a great suggestion! We used a force repository so we can enforce airflow is running on Python 3.9. I will derive a solution that is backward compatible with all Airflow V2 versions and python from 3.7 and up!

[potiuk]

Fantastic! Looking forward to the PR!

[ngearhart]

Any progress on this? I am interested in running in a FIPS-enabled environment.

[ngearhart]

Actually, I was able to get my production environment running with the following modifications:

• Flask caching - add option to use hashlib.sha256 instead of the default md5
• Airflow scheduler - find and replace all hashlib.md5 calls with hashlib.sha256

Obviously, this may not work in every environment, and I have not tested it thoroughly outside of my own environment. Use at your own risk.

You can quickly accomplish this by running:
sed -i "s/cache_config = {'CACHE_TYPE': 'flask_caching.backends.filesystem', 'CACHE_DIR': gettempdir()}/import hashlib\n cache_config = {'CACHE_TYPE': 'flask_caching.backends.filesystem', 'CACHE_DIR': gettempdir(), 'CACHE_OPTIONS': {'hash_method': hashlib.sha256}}/g" ./venv/lib/python3.8/site-packages/airflow/www/app.py \ && find ./venv/lib/python3.8/site-packages/airflow -type f -exec sed -i "s/hashlib.md5/hashlib.sha256/g" {} \;
(if you have Airflow installed at ./venv).

[r0ck37]

latest versions use flask-2.2.5 which still use sha1 for digest.
Flask-3.0.3 has a fix.
pallets/flask#5448

[potiuk]

Airflow 3 is going to get rid of Flask Application Builder as dependency. Currently Even 5.0* alpha/rc releases have flask<3. If you would like to help with Airflow 3 @r0ck37 development , that might be a good idea.