Migrating telamonian/jupyterlab-hdf to @jupyterlab/jupyterlab-hdf5

[telamonian]

I would like to move telamonian/jupyter-hdf into the @jupyterlab org. That will involve:

• Creating a jupyterlab/jupyterlab-hdf5 github repo in the org
• Adding me to the people who can publish in the @jupyterlab org on npmjs

I will be the primary maintainer of jupyterlab/jupyterlab-hdf5.

[saulshanabrook]

This sounds good to me! I have had many people request an extension like this.

[athornton]

Yum. I probably haven't requested the extension but I sure could use it.

[ian-r-rose]

I'm in favor, I know there is a lot of interest in well-supported HDF extension. There has also been several false starts (at least one of them mine :) )

[jasongrout]

Can you say a bit about how it works? In particular, the problems we've faced before are that double-clicking in the filebrowser automatically requests the content of the file, and that the notebook server file api doesn't support asking for more specialized content (like a url to some server-side service). How does your extension get around these limitations of these notebook server and contents api?

[jasongrout]

Process-wise, here are steps for incorporation into a Jupyter org: https://github.com/jupyter/governance/blob/master/newsubprojects.md#incorporation

[telamonian]

Can you say a bit about how it works?

@jasongrout The builtin file handling stuff has a baked-in constraint that a file path point to a real, existing resource available on a local (to the server) filesystem. Since I wanted to be able to path into the directory-like group objects within an .hdf5 file, I ended up having to side step the builtin file handling completely:

• Via a server extension, I supplied a new /hdf/contents api to replace the standard /api/contents api

  • The new api also enabled lazy loading of data from datasets. With the standard api, a single eager download of the entire file is the only option

• On the labextension side, I implemented my own Contents.IDrive. In particular, Contents.IDrive.get is what makes the actual api requests

Integration of hdf5 into the browser (both the default filebrowser one and the hdf browser supplied by my extension) proved tricky, for two reasons:

• There's no way to tell the default browser to use a different IDrive when opening a file (or to switch to another browser associated with a different IDrive)

• The type of file-like objects within an .hdf5 file are not determined by file extension suffixes, and in general their internal paths don't have any suffix. This makes it impossible to resolve these objects to a filetype within our current stack of open methods (which only uses path suffix when resolving filetype)

For now, the problem of browser integration is "solved" via monkey patches (shoutout to @ian-r-rose) that enable the correct opening behavior when .hdf5 files are double-clicked on in the default browser, and when groups/datasets are double-clicked on in the hdf browser. Improving the integration will require changes to core. I already have some reasonable ideas about what changes will be required, and will create PRs for them over the next couple of weeks.

[telamonian]

Integration of the HDF5 stuff with the data browser from jupyterlab/jupyterlab-data-explorer proved to be much more robust and concise (if not exactly simple). Longer term (Jlab 3.0, say), it may be worthwhile to consider refactoring out the current Contents and Filebrowser stack in favor of something more flexible. jupyterlab-data-explorer definitely shows some possible directions we could take it.

[vidartf]

Just for keeping this discussion on topic.

I would like to move telamonian/jupyter-hdf into the @jupyterlab org

Could you please share some motivating factors for moving this to the jupyterlab org? Given that the extension already exists, I don't think arguments of the kind "I would like this to exist" are enough?

Some standard pros and cons:

• Pro: It will increase visibility of the project, possibly contributing to the health of the extension.
• Pro: With it being in the lab org, lab maintainers will feel a sense of responsibility in maintaining and releasing the package, helping increase the health of the extension (especially if the original maintainer disappears).
• Con: With it being in the lab org, lab maintainers will feel a sense of responsibility in maintaining and releasing the package, further adding to their workload (especially if the original maintainer disappears).
• Con: Having too many packages in the org can feel overwhelming and bewildering to newcomers exploring the org.
• Pro/con: As the current owner of the package, you will lose full control of the package, as the opinions of other org members will need to be taken into account (i.e. lose BDFL status 😉).

Another general point for discussion: Do we currently have a policy for which extensions to accept into the jupyterlab org? Do we welcome all extensions? Which criteria are needed to be accepted? How do we determine which extension go into the main repo, and which are kept separate.

Hopefully if we answer these questions (in written form), it can be used as a guidance for other extension authors considering if their extension should be moved to the lab org/repo as well :)

[vidartf]

Some relevant points to the above questions from the Jupyter governance:

Subprojects should:

• Have an active developer community that offers a sustainable model for future development.
• Have an active user community.
• Use solid software engineering with documentation and tests hosted with appropriate technologies (Read The Docs and Travis are examples of technologies that can be used).
• Demonstrate continued growth and development.
• Integrate well with other official Subprojects.
• Be developed according to the Jupyter governance and contribution model that is documented here.
• Have a well-defined scope.
• Be packaged using appropriate technologies such as pip, conda, npm, bower, docker, etc.
• The most important question in evaluating Subprojects for incorporation as official Subprojects is this: is there broad agreement in the community and Steering Council that we are going to develop and maintain the Subproject in an official capacity?

[telamonian]

@vidartf

• Pro: It will increase visibility of the project, possibly contributing to the health of the extension.

This is my main reason for migrating. Currently the extension works, but its functionality is somewhat basic. As I start to add features I want to get as much feedback as possible so that I know what's of interest to the user-base. I also in general want more people to use it.

There's also a bunch of good synergies between jupyterlab-hdf, core, and some of the extensions in the core repo:

• jupyterlab-hdf provides a working example of how one could handle very large files in the default filebrowser

• In order for juptyerlab-hdf to achieve full functionality Jlab core is going to need a couple of changes (Make "Open" item in HDF browser context menu work correctly with datasets jupyterlab-hdf5#2, Make the "Open" filebrowser context menu item work with ".hdf5" files jupyterlab-hdf5#3). I want there to be a robust discussion about said changes, not least because they may lead into some long term plans to improve the file browser.

• juptyerlab-hdf provides a working example of how plugins can make use of/add support for @jupyterlab/dataregistry

[telamonian]

The timeline of the incorporation process, according to the governance docs:

• upon proposal, allow a week for potential objections (explicit agreement welcome, of course). If no objections, treat silence as agreement.

On Wednesday it will have been a week since I posted this proposal here. If no one has any objections, can we please pull the trigger on migrating jupyterlab-hdf -> @jupyterlab/hdf5 at the weekly dev meeting?

[saulshanabrook]

I have opened #12 to write down our process for adding a new repo to the jupyterlab org.

[blink1073]

I did a run-through of the source repo, and I think it is ready to bring in now. I'd say let's give a day and if no one disagrees go ahead and bring it in.

[jasongrout]

For the record, we had a long discussion this morning about adopting packages into the official JupyterLab github org. As I remember it, we went over the points @vidartf brought out above. We are exercising subjective judgement here for a pre-1.0 project that has generated good enthusiasm in the community and among core devs.

Have an active developer community that offers a sustainable model for future development.

Max is taking lead for now, and interest is wide enough among other developers that we would be willing to help.

Have an active user community.

Several people in the community (representing larger organizations) have expressed broad interest. As mentioned elsewhere, there have been a few attempts at this already, and having a single extension combine efforts on an hdf5 reader is important.

Use solid software engineering with documentation and tests hosted with appropriate technologies (Read The Docs and Travis are examples of technologies that can be used).

Max said testing would be an important next step in this repo. Given that it is pre-1.0, we feel it can come in now and we can work on it in the jlab repo as an experimental pre-1.0 project.

Demonstrate continued growth and development.

We feel the future is bright here for this one.

Integrate well with other official Subprojects.

It integrates with JupyterLab, and Max is working on PRs for JupyterLab to make it work much better.

Be developed according to the Jupyter governance and contribution model that is documented here.

Max has been an outstanding contributor demonstrating he understands what this means, and is willing to do so in this repo as well.

Have a well-defined scope.

This project does have well-defined scope of an extension in JupyterLab providing hd5 reading capabilities.

Be packaged using appropriate technologies such as pip, conda, npm, bower, docker, etc.

Yes.

The most important question in evaluating Subprojects for incorporation as official Subprojects is this: is there broad agreement in the community and Steering Council JupyterLab project that we are going to develop and maintain the Subproject in an official capacity?

This was a definite yes from the discussion in the dev meeting today.

[jasongrout]

We also decided to make the decision guidelines for this process more clear in #12.

We also decided that someone besides Max would look over the code (@blink1073 did today). Once the repo is moved in, it then falls under the Jupyter governance model, and would need to:

1. Change its license to the standard Jupyter license and copyright
2. Update the pypi and other package repositories to allow at least Brian and Fernando, as well as other relevant people in JupyterLab governance, to publish packages from this repo
3. Max would need admin permission on this repo in the JLab organization.

[blink1073]

I spoke to @telamonian today and we are going to move ahead with the migration now.

[jasongrout]

+1

[jasongrout]

It is done: https://github.com/jupyterlab/jupyterlab-hdf5

[jasongrout]

Thanks again @telamonian!

[jasongrout]

It is done

To be clear, the repo is in the jupyterlab org.

We also decided that someone besides Max would look over the code (@blink1073 did today). Once the repo is moved in, it then falls under the Jupyter governance model, and would need to:

Change its license to the standard Jupyter license and copyright

This is jupyterlab/jupyterlab-hdf5#13

Update the pypi and other package repositories to allow at least Brian and Fernando, as well as other relevant people in JupyterLab governance, to publish packages from this repo

This is jupyterlab/jupyterlab-hdf5#14

Max would need admin permission on this repo in the JLab organization.

To my understanding, this is done.

[telamonian]

I finally got around to updating all of the licenses, and to adding more Jupyter people as maintainers of the jupyterab-hdf package on pypi.

I still haven't tried doing a release of the migrated @jupyterlab/hdf5 package on npmjs. Do I need permission to upload a new package into the @jupyterlab org on npmjs?

[blink1073]

@telamonian, once you accept the npm org invite I will add you as an owner. Before then you won't be able to create new packages under that org.

[telamonian]

@jupyterlab/hdf5 has now been released on npmjs, and this migration is done done. Yaaaay!

[tgeorgeux]

Super cool!