Format Python and Bash scripts using ruff and shfmt

[eramongodb]

Followup to mongodb/mongo-cxx-driver#1454 which applies the same configuration and tools for formatting Python and Bash scripts to the C Driver using ruff and shfmt respectively.

As described in mongodb/mongo-cxx-driver#1454, the script formatter dependencies are defined in a separate format-scripts dependency group from clang-format to avoid impacting existing EVG lint tasks. Format configuration options are the same as for the C++ Driver (i.e. 2-space indentation for Bash scripts, 120 column width for Python scripts, etc.).

Despite the high file diff count, most changes are trivial whitespace or sorting/grouping of import statements. Notable exceptions include:

• Replacing `cmd` -> $(cmd) in Bash and sh scripts.
• Adding missing ;; in some case statements in Bash scripts.
• Changing "" -> '' for non-multiline strings in Python scripts.
• Changing ''' -> """ for multiline strings in Python scripts.
• Replacing x = lambda ... -> def x(): ... in some Python scripts.
• Re-indenting multiline raw strings in Python scripts to match new outer-level indentation.
• Addressing E402 warnings due to non-top-level import statements. Those which depend on preceding sys.path manipulation are silenced instead with # noqa: E403.
• Addressing E722 warnings by replacing except: -> except Exception:.
• Addressing F403 warnings by replacing * -> mongoc_common_setup for from mongoc_common import *. Silenced F403 and F405 warnings due to the import * pattern in Sphinx conf.py files.

There was also the following linter error triggered by the ruff check --select I --fix command used to sort-and-group import statements:

invalid-syntax: Cannot use `type` alias statement on Python 3.10 (syntax was added in Python 3.12)
  --> src/libbson/tests/validate-tests.py:52:1
   |
52 | type _ByteIter = bytes | Iterable[_ByteIter]
   | ^^^^
53 | """A set of bytes, or an iterable that yields more sets of bytes"""
   |

Found 1 error.

This seems to be due to the use of a Python 3.12 feature in src/libbson/tests/validate-tests.py, which is accurate to the inline script metadata Python requirement of >=3.12, but is inconsistent with the pyproject's Python version requirement of >=3.10. Ruff currently only understands the pyproject requirements; it does not yet support inline script metadata (see: astral-sh/ruff#10457). Therefore, validate-tests.py is manually excluded from the regular command and formatted separately with the --target-version py312 flag instead. All other ruff check warnings are addressed by this PR as a drive-by improvement.

[eramongodb]

The make-docs task was failing due to the attempted resolution of F403 warnings:

- from mongoc_common import *
+ from mongoc_common import mongoc_common_setup

Prior to changes in this PR, the _build/for-docs/src/libbson/doc/man directory contains bson_*.3 component doc files. With the changes in this PR, it contains a single libbson.1 file instead. Replacing * with mongoc_common_setup failed to account for the fact there are many more names imported by this statement which controls Sphinx configuration and behavior (Sphinx is sensitive to their presence in conf.py). Therefore, reverted to importing * and silencing both F403 and F405 warnings instead to fix make-docs task behavior.

To diagnose the cause, removed silent: true from the upload-man-docs EVG function, which seems to have been unquestioningly carried over through multiple major EVG config refactors and can be traced all the way back to its initial introduction in Feb 2017. I do not think it is worth keeping anymore. This revealed the following error message which hinted at the root cause of the make-docs task failure:

No .3 files in _build/for-docs/src/libbson/doc/man!

---

CC @rcsanchez97 It seems the debian-package-build task will need to be updated to account for changes to the formatting of some files which are being patched for Debian packaging, e.g.:

patching file src/libmongoc/doc/conf.py
Hunk #1 FAILED at 50.
1 out of 1 hunk FAILED

[connorsmacd]

LGTM

[vector-of-bool]

LGTM with comments about some vendored files

[vector-of-bool]

This file should probably be excluded from formatting, as it's a vendored external library.

[vector-of-bool]

This file should be excluded from formatting, or the formatting should be added to the upstream copy.

[vector-of-bool]

Could this be rewritten to instead import the variables that are actually relevant for the sphinx config, rather than star-importing everything?

[eramongodb]

I considered it, but opted to leave it as-is, since it seems the intended purpose of mongoc_common is to act like an #include rather than an import (contents "copied" into each conf.py file).