Adds cache invalidation logic

[cgranleese-r7]

This pull request fixes #20352

This pull request adds a caching validation. The goal is to force cache metadata to be reloaded if any changes have been made to lib, modules and ~/.msf4/modules. If changes have been made the DeferModuleLoads loads option will be overwritten to force a reload of the metadata cache.

Performance

CRC (2.91)

❯ hyperfine --export-json /dev/stdout --warmup 3 "bundle exec ruby ./msfconsole -qx 'exit'"
Benchmark 1: bundle exec ruby ./msfconsole -qx 'exit'
  Time (mean ± σ):      2.916 s ±  0.129 s    [User: 1.341 s, System: 0.643 s]
  Range (min … max):    2.821 s …  3.239 s    10 runs

{
  "results": [
    {
      "command": "bundle exec ruby ./msfconsole -qx 'exit'",
      "mean": 2.91607380178,
      "stddev": 0.12927006626381535,
      "median": 2.86691152268,
      "user": 1.3407359399999998,
      "system": 0.6426045199999999,
      "min": 2.82105658468,
      "max": 3.2393112516800002,
      "times": [
        2.8925379186800004,
        2.87763075168,
        2.8487470436800004,
        2.83313408468,
        2.82105658468,
        2.8561922936800004,
        2.90355996068,
        2.84913075168,
        3.2393112516800002,
        3.03943737668
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

md5 (3.0)

❯ hyperfine --export-json /dev/stdout --warmup 3 "bundle exec ruby ./msfconsole -qx 'exit'"
Benchmark 1: bundle exec ruby ./msfconsole -qx 'exit'
  Time (mean ± σ):      3.058 s ±  0.126 s    [User: 1.403 s, System: 0.682 s]
  Range (min … max):    2.911 s …  3.264 s    10 runs

{
  "results": [
    {
      "command": "bundle exec ruby ./msfconsole -qx 'exit'",
      "mean": 3.0578842767000003,
      "stddev": 0.12572988979906158,
      "median": 3.0178097435,
      "user": 1.40309968,
      "system": 0.6822578599999998,
      "min": 2.9105510145,
      "max": 3.2637436395000003,
      "times": [
        3.0477420145000003,
        2.9842478885,
        3.0173730565000003,
        2.9105510145,
        3.0182464305,
        2.9727870565,
        3.2020727225,
        2.9390091805000003,
        3.2230697635000003,
        3.2637436395000003
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

Verification

To test this easier add some logging to the new methods for when the cache is being invalidated

• Verify that changes made to lib, modules and ~/.msf4/modules are reflected when restarting console
• Adding a new module to ~/.msf4/modules is able to be used after a reboot of console
• Editing any files in lib, modules and ~/.msf4/modules cause the cache to be invalidated
• Cache file is created locally when the cache is invalidated e.g. should be under ~/.msf4/store/cache_metadata_base.json
• If no changes are made the cache should not be invalidated
• Code changes are sane

[cgranleese-r7]

CI failures are more Nokogiri issues, I believe they are safe to ignore.

[adfoster-r7]

It might be worthwhile using fibers, but they're bit of a mystery in terms of stability and whether they'll be supported long-term or not - your call on investigating the performance differences between the two 🙌 (cc @sjanusz-r7)