Support for importing multiple bootstrap.cfg files?

[jsonmurphy]

Hi, great framework you guys have here, I almost built something similar before I found it.

As the title says, do you guys have any plans to support importing bootstrap.cfg files? At the moment it seems that if I package an application with a bootstrap.cfg file that has some services (which all depend on each other in some way) declared within it and then publish it. If i wanted to use all the services I'd have to re-declare the all of them in the current project's bootstrap.cfg. So you can imagine that if i had multiple groups of services packaged externally (and these service may have external dependencies themselves) this may become a bit tedious (or maybe I'm lazy :).

I've given it some thought and here's an ideal example of what a bootstrap.cfg file imports could look like:

trapper-demo.trapper-demo-web-service/hello-web-service
trapper-demo.trapper-demo-service/hello-service
puppetlabs.trapperkeeper.services.webserver.jetty9-service/jetty9-service
puppetlabs.trapperkeeper.services.webrouting.webrouting-service/webrouting-service

# import section
[:import]
# specify namespace(package?) and optional bootstrap-config file name (default: bootstrap.cfg)  
my.tottally-awesome-group-of-services:myboot.cfg

So in the bootstrap phase, the classpath for any namespaces (or package names?) specified in that section would be searched for the config file and they could be parsed recursively for their imports which are then merged and duplicates are removed.

From my initial walk through of the code it seems like only the parse-bootstrap-config! method would need to become a bit more sophisticated but I'm unsure of any other implications of changing it to support the above. (Shouldn't cause any performance issues since it only happens once)

I actually don't mind submitting a PR if this is something being considered. On the other hand, my alternative is to build a layer on top of trapperkeeper to support this for my own needs and just pass the options that trapperkeeper requires as seen in the wiki

[KevinCorcoran]

Hi,

Just to answer your initial question - this is not something we've considered. I don't believe we have quite run into the exact situation you're describing, where you are publishing libraries which include multiple services that depend on each other, and then rolling them up. Instead, we tend to extract each of our reusable services into their own projects/repositories (for example, https://github.com/puppetlabs/trapperkeeper-webserver-jetty9). So, we tend to just have 1 service per-library. And we generally don't include any bootstrap.cfg files in the .jars for these libraries, instead choosing to maintain the bootstrap.cfg in the project that actually pulls in the libraries.

I've got an idea to remove bootstrap.cfg entirely, and move the set of services into the rest of the configuration instead - there is no good reason, I think, for it to be in separate file in a different (and non-standard!) format. We use HOCON for most of our config files, so what I'm thinking would look something like

services : {
  my.service/service1
  another.service/awesome-service
  ... etc ...
}
... rest of the config

We already support loading multiple configuration files via --config /path/to/directory/of/config/files, so perhaps this would work for your use case as well?

(Just to be clear, that change is not 100% blessed yet. It will definitely need consideration from others, namely @cprice404.)

In any case, glad you like Trapperkeeper, and thanks a lot for the feature request!

[cprice404]

I'm with @KevinCorcoran in that I think if we decide to do anything fancier with bootstrap.cfg, we'd probably want to migrate it to use a "normal" config file format first. I've been thinking of something close to what @KevinCorcoran described, but probably a slightly more complex data structure in order to allow for easier future extensibility:

trapperkeeper : {
   boot-services : [
      "my.service/service1"
      "another.service/awesome-service"
   ]
}

This would allow us to add other sections alongside boot-services to provide additional capabilities, without breaking backwards compatibility. So, for your use case, that addition section might be something like additional-boot-resources, which could then be implemented to do something like you're suggesting, where it'd search the classpath for additional config data and then add things to the service list before booting.

Would probably want to brainstorm a bit more on the syntax of that construct, but it sounds like something we'd probably be in favor of adding :) I do think the first step would be to move the bootstrap config into a 'normal' config file, though. That's near the top of my wishlist for a 2.0 TK release.

Note that while we're using HOCON config snippets for this example, I don't think there's any reason why EDN or JSON wouldn't work just as well.

And yeah, as @KevinCorcoran said, I think the reason that this hasn't hit us too much so far is that we ship our apps as uberjars, so we know at packaging time what services will be available and we can just have our packaging code generate the correct bootstrap.cfg. I can see how anyone trying to ship an app as multiple jar files would run into what you're describing, so I'm definitely in favor of coming up with a good solution for that use case.

[cprice404]

And thanks for the input!

[jsonmurphy]

Gratitude for the quick responses, I completely agree with the idea of changing/merging the bootstrap-config into a normal config (especially for the structural and extensibility benefits).

Just to give a bit more insight into my use case:

I'd like to build a framework (though it's really just another service) that can easily compose full fledged applications (group of services) as well as any random service that might be global across them. So a concrete example might be, composing my application with a CMS application and a Document Management System. However, both these systems could function on their own as single applications. As for examples of services that might be global to both of these: a Database Service (or DAL Service) , DB Migration/Evolution Service, Theming Service etc. Both applications may also intersect in terms of the services they need but that shouldn't be a problem if the declarations are merged and duplicates removed.

Its just an initial/rough idea right now that i still need to validate, but I've made up my mind to use trapperkeeper as a base; Not only because its so great :) but also to allow the services built for the framework (or any application) to be used in other unrelated trapperkeeper applications. I'm no expert in Clojure as yet so let me know what you think and no lines of code have been spit out as yet so I'm open to any suggestions.

Also please let me know if/how i can assist with getting things to 2.0!

[KevinCorcoran]

Hi,

It sounds like Trapperkeeper is a good fit for what you have in mind, and I think the change to consolidate the configuration would be an easy one, if you are looking for something to hack on. I suspect the changes to the production code will be quite minimal, actually, and the bulk of the work will probably be to update all of the test cases, examples, documentation, etc.

[cprice404]

FWIW: we're still not quite ready to make the leap to 2.0, but we are adding support for multiple bootstrap configs as an interim step.

[MikaelSmith]

Multiple bootstrap configs were implemented. Documentation at https://github.com/puppetlabs/trapperkeeper/blob/master/documentation/Command-Line-Arguments.md#multiple-bootstrap-files.