Support for splitting a FrozenRecord across multiple files?

[eapache-opslevel]

Hi @byroot / @casperisfine, been a while!

At my new place we've been thinking about a use case with a few very large records (with very long list properties) and it feels kind of natural to put each record in its own file, e.g. something like:

config/frozen_records/role/viewer.yml
config/frozen_records/role/editor.yml
config/frozen_records/role/admin.yml
...

where the actual ruby class would be class Role < FrozenRecord::Base.

This is sort-of / hackily achievable already by writing a custom backend, but the backend API and main gem code still assumes there's a single backing file. I guess you could also fake it with ERB code that uses ruby to iterate / load / inline all the files in the directory, but it runs into similar issues with reloading in dev, etc.

Thoughts on the use case? Any interest or tips for a PR for something like this, if we move forward with the idea? I don't know how deep the single-file assumption is baked right now.

cc @qr8r

[casperisfine]

Hey 👋

This is sort-of / hackily achievable already by writing a custom backend, but the backend API and main gem code still assumes there's a single backing file

Right, but I guess you could feed it a directory path?

Then in the load method you can just list all the files of the directory. Just an idea like this, but at a quick glance it seems possible.

[eapache-opslevel]

I will take a look at that. I was concerned that something would break if backend.filename returned a path to a directory, but maybe not.

I guess reloading in dev would still be an issue since the mtime of the directory itself won't update when a file gets updated, but that's fairly minor.

[casperisfine]

since the mtime of the directory itself won't update when a file gets updated

We can refactor this to go though the backend as well, this way the backend can return the max(mtime) from all the files in the directory.

[marcoroth]

Just sharing the approach I took:

# app/models/backends/multi_file.rb

module Backends
  class MultiFile
    def initialize(glob, backend: FrozenRecord::Backends::Yaml)
      @glob = glob
      @backend = backend
    end

    def filename(_model_name = nil)
      @glob
    end

    def load(file_path = @glob)
      Dir.glob(file_path).flat_map { |file| @backend.load(file) }
    end
  end
end

For the role model it would look something like:

# app/models/role.rb

class Role < FrozenRecord::Base
  self.backend = Backends::MultiFile.new("role/*.yml")
  self.base_path = Rails.root.join("config", "frozen_records")
end