Tag a release of ev3dev-lang to correspond with kernel release

[WasabiFan]

Now that we have gotten the libraries in this repo updated so that (aside from some last changes) they are all ready for the newest kernel release, I think we should get the readme and spec updated, set the submodules to be pointing at the proper commits, and then merge develop into master and tag a release.

TODO list

• Update for breaking changes in new ev3dev release
• Autogen the markdown spec and update it for LED changes and new sensors
• Update the "currently supported kernel" in the markdown spec
• Decide on fate of current lua binding
  • I think that, unless @rhempel's new lua binding is ready for prime time, we should hold off on including it (more on that below)
  • Either mark it as old/broken or remove it for release
• Work through issues and close any that can be closed
  • Make sure that we haven't forgotten about anything important
• Modify repo to use "inverted" model
• Run an auto-formatter over JSON spec (and murderize any evil tabs)

As for the new bindings (C# and Lua v2), I think we should hold off on those until after we have a release, and then add them in and tag a new release with those included after those have been tested.

We should also figure out the versioning scheme to use on the main ev3dev-lang repo. We had started versioning the spec at v0.9 and continued from there (0.9.1, 0.9.2). We did this because we did not have enough feature coverage at the time to feel reasonable calling it a "v1". At this point, I think that jumping to v1 might be a good idea, but I think this too needs more discussion.

What am I missing? Thoughts, additions, corrections? @ddemidov @rhempel

[rhempel]

@WasabiFan - please leave the Lua binding out of the picture for now, and mark the old one as broken - actually, just take the old binding out too. It's not going to work at all.
@ddemidov and @ensonic - please check the master branch at [https://github.com/rhempel/ev3dev-lang-python] for the latest autogenerated Python binding - I believe that the only significant change is the port-name to address.
@dlech - we'll need to coordinate bumping the Debian package version for the Python binding, and also the release to PyPi.
@WasabiFan, the spec.json will need to be updated with the correct kernel version.

Once we get this done, I think we'll have a decent checklist for the next major bump!

[WasabiFan]

I have created a vNext milestone and added to it the currently-open issues that I think are important to get finished before the next release. If anyone spots something else that we need to do or notices something that's mis-tagged, please open a new issue, comment on this issue, or just make the modification (whichever seems appropriate).

[ddemidov]

I'll try to update C++ bindings next week.

[WasabiFan]

Sounds good.

[WasabiFan]

Unless anyone else sees differently, I think I'll go with v1.0.0 for the main repo version. If we want to have the bindings themselves use the same version number as the spec, we can do that with this scheme on pretty much any packaging system; We can use the first two digits for the main repo, and the last can be updated for maintenance releases on the individual libraries. And of course, we can keep the libraries versioned separately if that's the preference of the author.

[ddemidov]

With lua bindings removed (and to be added as a submodule later), the only in-repo binding is C++. I think it makes sense to split that out as well, so that primary purpose of this repo is maintaining specification (and providing links to bindings supporting that specification). Then tagging of this repo would not have to be negotiated with the bindings authors.

[WasabiFan]

Then tagging of this repo would not have to be negotiated with the bindings authors.

I think it still will, at least to some extent, because there are references to specific versions of each library in this repo. How are you thinking we should handle that?

[ddemidov]

That's a good point. In my old python bindings I had version string that contained both last tagged version of the bindings and the spec version it was generated with. May be this could be somehow extended onto other languages?

[rhempel]

I have to disagree with tying the main repo version to the language bindings version as you have described - but maybe there's a way to do it so that it does work. The main purpose of ev3dev-lang is to specify what is required in a compliant language binding. We need to deal with breaking changes and enhancements in the ev3dev-lang repo and still have a way to indicate compatibility with individual language bindings.

If we adopt a major.minor.inc numbering scheme, then I think we can use major.0.inc for spec.json and the corresponding ev3dev-lang repo version.

For compatibility with a language binding, I'll use the ev3dev-lang-python example. I would make the next version 1.minor.inc which means that it's compatible with 1.0.x of spec.json. This allows me to introduce breaking changes to ev3dev-lang-python as 1.1.x even if spec.json does not change.

On the other hand, tying all of the binding versions together is a huge pain - can we just keep the individual bindings separate and introduce a mandatory value that returns the minimum versoin of spec.json that the binding is based on and another one for the minimum kernel version?

[WasabiFan]

Now we have a discussion!

I think that, optimally, the versions of the individual bindings would be completely separate -- especially when we start getting more bindings. But at the current time, our libraries are inherently tied to the spec and main repo: We need the submodules to be able to easily use autogen.

So, that brings us to a new question: Is there a way that we can fundamentally change our architecture so that we can separately version the spec? We could, for example, move the markdown spec into the autogen folder and then separate autogen into its own repo; then the repo with the folder structure as we currently have it is never released, and we only release the autogen repo and spec.

[rhempel]

Take everything I say with a grain of salt - in my mind, the version of spec.json IS the version of the ev3dev-lang repo. It's the first thing we need to update when the kernel introduces a breaking change to attributes of devices.

We generate portions of wrapper-specification.md from spec.json so it makes sense to base the major.minor.x version of the ev3dev-lang on the major.minor version of spec.json - minor changes to correct spelling bump x. Introducing new things to spec.json in a non-breaking way bumps minor, and breaking changes bump major

The autogen folder can be its own repo, but then we move spec.json, package.json and autogen-list.json UP to the ev3dev-lang top level. Then we can change the underlying implementation of autogen but we keep JSON as our binding definition language.

Individual language bindings must somehow allow the programmer to retrieve the version of spec.json, the version and name of the autogen method, and the minimum kernel version that the binding is compatible with. Binding version bumps have the standard semantics of the ev3dev-lang version string. Breaking vs new vs minor changes.

How does that sound?

[rhempel]

Ah, I see the hole in the argument - the bindings may change quite frequently - so how do we bump the overall ev3dev-lang repo version? In that case, it's the same as a minor change since nothing new was added to spec.json - just bump the x every time someone issues a PR for an update to a compatible binding.

[WasabiFan]

I like that scheme, but there's a flaw that I can't tell if you have considered.

Let's say that we follow the plan as you described. We make a change to the spec and release a version of the main repo. Now we have a tagged release of the newest spec that has links to libraries that implement the previous version. Now, let's say that the libraries start updating and, at an interim stage when some may be broken, we make changes to the spec and release again. Now this release is pointing to broken code.

I don't think that this is an acceptable result; so how do you think we can mitigate this? Ideally in my view, we would have a repo that just has the submodules and a different one that just has the spec. We version the one with the spec, and don't release the one with the submodules.

[rhempel]

I think you're saying that the binding - spec.json and friends is one repository, autogen is another repository, and ev3dev-lang is the meta repo that binds them together?

[WasabiFan]

Yes, that is what I am imagining. We could also put autogen in a repo along with the spec if we want to eliminate the need for the extra repo and submodule.

Or, we could invert the model, and instead have each of the libraries use a submodule to include the ev3dev-lang repo, which would include the autogen and spec and such. This would make it more free-form in the sense that anyone could include autogen and the spec in their repo; this is an idea that we'd have to think about more if we want to consider it.

[rhempel]

Inverting the model has the distinct advantage of decoupling the language bindings from each other and allowing each author to manage their releases independently. Based on my inability to merge coherently this may be a good thing.

I still think we want to require that a language binding is able to report the spec.json and autogen and kernel versions!

[WasabiFan]

I still think we want to require that a language binding is able to report the spec.json and autogen and kernel versions!

Sorry if I didn't make it clear: I just considered it a done deal and forgot to mention it. I think that, whatever we choose for the repo structure and versioning, we should make sure to prominently display the spec and code versions both in the repos and and on the API surface.

[ddemidov]

I like the inverted model idea, it should scale well when we have more languages.

[WasabiFan]

I like the inverted model idea, it should scale well when we have more languages.

Do we want to make this change? If so, I can create a branch and in that branch I can remove the submodules and modify autogen to work with this other structure.

[rhempel]

I am going to be AFK until Sunday, most likely - so not much more input from me until then, but ...

The key problems that the inverted model solves are:

1. Allows each binding to be released separately - no waiting for everyone before we tag ev3dev-lang
2. Allows each binding to use an autogen mechanism that works well for that language.

Are there any disadvantages?

[WasabiFan]

@ddemidov I just pushed a branch in which I removed the JS and Python submodules in preparation for inverting the submodule system. How do you want to deal with the C++ library? We'll need to relocate it into its own repo for this to work, but you're currently updating it for spec changes. So are you going to get it updated in develop, move it into a submodule, then I merge that into my branch and delete the submodule? Or how do you want to do it?

I need to delete the C++ folder at some point in my branch, but I will keep it there for now until I get the rest of my stuff done.

[ddemidov]

I've just tested and pushed the updated C++ library to develop.

I think there is no need to add the C++ bindings as a submodule to the current repo; we can move it to its own repo right away. The only thing I would like to discuss is who would be the formal owner of the new repo. Can we create it under ev3dev? I am not sure how long I will be able to support it, so I would prefer that to creating it under my account.

[WasabiFan]

I think there is no need to add the C++ bindings as a submodule to the current repo; we can move it to its own repo right away

So, when we get a new repo to house it we should just delete the folder from develop and wait until we get the inverted setup integrated.

Can we create it under ev3dev? I am not sure how long I will be able to support it, so I would prefer that to creating it under my account.

That'll have to be a discussion with @dlech and @rhempel.

[WasabiFan]

Can you guys take a look at what I have done in the separate branch and see if it seems right?

Here's the diff of the JS binding after modifying it to work with the new structure. When I want to tell it to update, I run the following from the root of my repo:

$ node ev3dev-lang/autogen/autogen.js

Is this what you had imagined? How do you think it should be changed?

[rhempel]

We can do this a couple of ways - the owner/maintainer of the repo can keep the official version in their own GitHub space, and we can point the language specific binding withing the meta-repo for ev3dev.lang at the maintainer's version.

Or we can give the maintainer access to a language specific binding within the ev3dev organization on GitHub.

@dlech, what's your opinion on this topic? Is the "inversion" idea a reasonable way to split up the bindings?

[dlech]

I like the inversion idea. Makes things a bit simpler. If you published the autogen stuff on npm then you wouldn't even have to add ev3dev-lang as a submodule. It doesn't make a difference to me if individual languages are under the ev3dev org or personal accounts. GitHub lets you transfer owneship, so it doesn't matter where the new c++ repo starts.

[WasabiFan]

If you published the autogen stuff on npm then you wouldn't even have to add ev3dev-lang as a submodule.

If I did it that way, I could also make it so that you could npm install -g and get it as a global utility; that would be helpful. The great part as I see it is that we can start out using a submodule and then replace that with just the npm module once we know that our new system works. The code as it is in my decentralization branch should work from anywhere, as there are no hard-coded paths (absolute or relative).

[ddemidov]

It doesn't make a difference to me if individual languages are under the ev3dev org or personal accounts. GitHub lets you transfer owneship, so it doesn't matter where the new c++ repo starts.

Ok, this makes sense. I've split the cpp bindings into https://github.com/ddemidov/ev3dev-lang-cpp.

[WasabiFan]

@dlech how do we specify a kernel version in a way that is cross-platform in issues and docs? The only shared values between the three platforms' version numbers are 9 (the release cycle) and ev3dev. Will the release cycle always increase for all future releases? Or do we need to specify all three full version numbers if we want to be completely sure?

[dlech]

As far as the LEGO-specific drivers are concerned, the 9-ev3dev is what matters.