Add extensible source providers

[ashutosh-narkar]

Why

OCP's source types are hard-coded. Projects that embed OCP as a
library and need to build bundles from other kinds of sources currently
have to fork OCP or upstream every new type.

This adds a way to register custom source types programmatically. Providers are
Go code registered through the public API of pkg/service. They then work like
the built-in sources: they're configured on a source, synced by the bundle workers,
and built into the same bundles.

Some changes

• pkg/sync: SourceProvider and SourceProviderRegistry. A provider
  declares its Type, a JSON Schema for its configuration, Parse for
  validation, and New to create a Synchronizer. Its synchronizer can
  optionally implement BundleContributor to add manifest metadata, claim
  roots it has no files for, or set the Rego version of its content.
• Config: a providers: list on sources. name and type are
  required; path optionally prefixes the entry's content; all other keys
  go to the provider.
• Persistence: a new sources_providers table (migration 29). As with
  requirements, a source without providers leaves stored entries in place,
  and providers: [] removes them.
• Breaking change: pkg/sync.Synchronizer.Execute now returns
  (map[string]any, error) instead of error.

Example

Registering a provider:

reg := pkgsync.NewSourceProviderRegistry()
if err := reg.Register(myprovider.New()); err != nil { // Type() == "my-source"
    return err
}

svc := service.New().
    WithConfig(cfg).
    WithSourceProviders(reg)

Using it in a source:

sources:
  app:
    git:
      repo: https://github.com/example/policies.git
    providers:
      - name: users
        type: my-source
        path: users          # data files go under data.users
        endpoint: https://example.com/users

bundles:
  app:
    requirements:
      - source: app
    revision: $"{input.sources.app.git.commit}-{input.sources.app.providers.users.version}"

Fixes: #361

[srenatus]

Small fry only, thanks!

[srenatus]

Would it be possible to have a provider registered for E2E test purposes? So we could have a positive tests here, too.

[ashutosh-narkar]

Added tests.

[srenatus]

Suggested change

	addBundlesStatusesUpdatedAt(27, dialect), // adds 2, next is 29.
	addBundlesStatusesUpdatedAt(27, dialect),

[ashutosh-narkar]

Done.

[srenatus]

[nit] So we still need this these days? new(10) might do the same 🤔

[ashutosh-narkar]

Updated.

[srenatus]

nested claimed roots fail when a .manifest sets roots. One provider claims ["lazy/users", "lazy"], and another source's .manifest sets roots: ["app"]. The build fails with:

source "provider" claims root "lazy", which overlaps root "lazy/users" from the .manifest of source "git"

[ashutosh-narkar]

Good catch. Fixed and added tests to cover this.

[srenatus]

Thanks!