How can I use this for a non General registry?

[davidanthoff]

We have our own registry for a bunch of lab internal stuff. Can I use Registrator for that use case? How would I configure it so that it works on our own registry, and not the General public one?

Is it general ready to be used?

[StefanKarpinski]

@nkottary, @tanmaykm, any thoughts on this?

[nkottary]

We support multiple registries. You have to say register(target=your-registry-id) where your-registry-id is configured in the configuration file

[StefanKarpinski]

@davidanthoff: please give it a try and file issues if there are any problems.

[davidanthoff]

So that would require that we run a server somewhere?

[tanmaykm]

Yes. Credentials to your private registry and repositories, that Registrator.jl needs to work with, would then remain on your server.

[davidanthoff]

Hm, I actually would much prefer a setup where we could just use the normal registrator bot, and for example have a config file in the root of our packages where we can specify the URL of the registry for that specific package.

If the choice is between a) giving access to our private repos to the central registrator bot and b) running our own server, a) easily wins out. Running our own infrastructure for this kind of thing is really kind of a no-starter for us...

[davidanthoff]

I'm actually wondering whether JuliaLang/PkgDev.jl#144 wouldn't be a perfectly fine workflow for our use-case...

[nkottary]

It isn't necessary that you run your own server. Just install the normal registrator bot on both your private repo and the private registry. However, Registrator will not know which registry you are referring to when you just say register(). By default, it thinks General.

[davidanthoff]

Is there a way to then tell it about the private registry?

[nkottary]

We need to update the config on our end to whitelist your registry.

[StefanKarpinski]

We can't really have the public registrator instance be an "open relay" to arbitrary private registries since that opens the server up to all sorts of mischief. The way this will have to work is that people can run their own registrator instances which are configured to run against their private registries.

[TransGirlCodes]

Is this simple to deploy on Heroku? I'd love to deploy this to manage the planned BioJulia registry

[StefanKarpinski]

Not sure... the only way to find out is to give it a try and see.

[StefanKarpinski]

Also, Julia Computing will host and manage an instance of this for General and we would be happy to run instances for any public registry of open source packages too. (Private registries of closed-source packages are a different story, of course.)

[TransGirlCodes]

@StefanKarpinski I'm having an experiment now to see how I get on, but I may well take you up on the offer if Julia Computing would be happy to run an instance for a BioJulia registry.

[davidanthoff]

We have a public registry (https://github.com/mimiframework/MimiRegistry) that contains some public and some private packages. The private packages are all research work-in-progress that will be made public once we have the associated research published in journals. If there was a way to get a hosted version of Registrator for that, it would be awesome. But I also totally understand if that doesn't work, given that this is somewhere in the middle between public and private...

[TransGirlCodes]

What goes in the token field in the [github] section of the config? I've made a "BioJulia Registrar" GitHub app and am filling out a config so as to test this server locally on my laptop before I try any pushing to Heroku business. I've filled in:

[github]
user = "BioJulia"
email = "[email protected]"
name = "BioJulia Registrar"
token = ""
secret = "## The Client secret"
app_id = 27978
priv_pem = "private-key.pem"

Presumably I can just provide the private-key.pem file for the app in the scripts folder and the Dockerfile will include it in any built image.

[nkottary]

token is the access token for the user BioJulia. You can get this from the settings page of the user.

[TransGirlCodes]

I can see tokens for my user BenJWard in Settings/Developer Settings/Personal Access tokens, but I can't see anything like that for the BioJulia org's settings. Do I have to set the [github] info to use my personal account details?

[nkottary]

Yes the user needs to be an actual user not an org. You don't have to use your personal token. You can create a new user on GitHub say "BioJuliaRegistrar" and use that token. Also note that permissions can be set to the token. The token has to have permissions to create a pull request on the registry repo and to comment on the package repo issues and commits.

[StefanKarpinski]

The private packages are all research work-in-progress that will be made public once we have the associated research published in journals. If there was a way to get a hosted version of Registrator for that, it would be awesome. But I also totally understand if that doesn't work, given that this is somewhere in the middle between public and private...

We'll figure something out for this as well. The intention is to support the open source ecosystem and the private development of soon-to-be-public packages is part of that. But obviously we have to figure out using this for General first and then work on other registries...

[davidanthoff]

“Soon-to-be-public” of course often means years, because of the wonderful joys of the extremely slow peer review process in many (including mine) fields :)

[TransGirlCodes]

@nkottary

How do I need to edit this section? Some of the fields like the trigger look self explanatory, but I'm not sure which of the others I need to alter.

[registrator]
trigger = "quxregister"
report_issue = false
issue_repo = "JuliaComputing/Registrator.jl"
reply_comment = true
set_status = true
enc_key = "1234567890123456"
disable_pull_request_trigger = false
disable_approval_process = false

[nkottary]

You want to set trigger = "BioJuliaRegistrar" and enc_key to an encryption key of your liking, it must be of length 16 characters. You can leave the other options as they are.

[nkottary]

Note that the trigger is the @ prefix to the command, Ex: @BioJuliaRegistrar register(). You can set it to whatever you like, it needn't be a user.

[timholy]

Just to chime in that we're in the exact same boat as @davidanthoff. As long as there are clear instructions about how to set this up, I don't care very much what the solution looks like (as long as it isn't heroically difficult).

To me it seems another option would be a PkgDev-like solution that prepares a commit with the appropriate changes to the 4 TOML files, e.g.,

julia> Registrator.commit("HolyLabRegistry", "SomeCoolPkg", v"0.4.1")

where all changes to "SomeCoolPkg"'s Project.toml file, other than the version bump, are assumed to have already been made. This concurs with my workflow in that when developing I think in terms of PRs, and the question of when to prepare a new release is handled separately.

We can handle pushing these commits up ourselves; the annoying part is the fiddly detail involved in getting the correct changes to those 4 TOML files in the registry. It seems there must be code out there to do this now, it's "just" a question of packaging it.

[davidanthoff]

I actually always thought that the PkgDev workflow was in theory pretty good, and that I almost preferred that over a bot. The reason the bot then was just way nicer was that PkgDev was pretty unreliable and the bot always worked, but that (at least to me) seems more a question of code quality than design.

[adamslc]

See #45 for a PkgDev like workflow.

[StefanKarpinski]

PkgDev was pretty unreliable

What was unreliable about it?

[davidanthoff]

It's been a while, but I have vague memories that it interacted with github, and that often failed because of credential problems and stuff like that. I might well misremember.

[StefanKarpinski]

That makes sense. I would imagine that for a private registry you may want to just generate a registry commit for the new version and just push it, assuming every one who would be tagging new versions would have push rights to the registry?

[davidanthoff]

Exactly, or it could create a local branch with the commit and then they could open a PR, if they don't have push access.

[TransGirlCodes]

@nkottary I've tried to deploy the docker image to Heroku free tier, with some success, however, I think I need to refine how the port for the webhook is set:

Here's a log:

2019-04-07T13:50:25.733014+00:00 app[web.1]: └ @ Registrator.RegServer /home/registrator/.julia/packages/Registrator/q1eJ6/src/Server.jl:907

2019-04-07T13:50:26.641031+00:00 app[web.1]: ┌ Info: GitHub webhook starting...

2019-04-07T13:50:26.641046+00:00 app[web.1]: │   trigger = r"@quxregister `(.*?)`"

2019-04-07T13:50:26.641049+00:00 app[web.1]: │   http_ip = 0.0.0.0

2019-04-07T13:50:26.641050+00:00 app[web.1]: │   http_port = 8001

2019-04-07T13:50:26.641052+00:00 app[web.1]: └ @ Registrator.RegServer /home/registrator/.julia/packages/Registrator/q1eJ6/src/Server.jl:894

2019-04-07T13:51:01.112956+00:00 heroku[web.1]: State changed from starting to crashed

2019-04-07T13:51:00.998531+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch

2019-04-07T13:51:00.998708+00:00 heroku[web.1]: Stopping process with SIGKILL

2019-04-07T13:51:01.096033+00:00 heroku[web.1]: Process exited with status 137

Can we add a way for the app to get the port number from the $PORT environment variable, if it is not set in the config? Using the docker EXPOSE command does not work for Heroku.

I can see a few ways of doing it in the github_webhook function.
One could be to check the value of http_port=config["server"]["http_port"], and detect if it is an environment variable e.g. "$PORT", and if it is to get the variable before passing it to anything else.

[nkottary]

The key http_port should be checked for and if it doesn't exist then take the port number from ENV["PORT"] if that exists, else set the default to say 8080. Can you send a PR for that? May be useful for others deploying to Heroku.

[nickrobinson251]

in case it is useful, there's an example of registering a package in a private registry in #197 (although this is not very nice, and I interact with a private registry via a hosted Registrator instance because the REPL's not so nice... hence the issue).