Add support for custom widgets loaded from CDN

[motin]

So exciting to see initial widgets support in recently released 0.18!

Application or Package Used
nteract desktop

Describe the bug
The Jupyter widget Qgrid does not render in nteract.

To Reproduce
Steps to reproduce the behavior:

1. Open interact
2. Run !pip install pandas numpy qgrid
3. Run qgrid's official Example 1 code:

import numpy as np
import pandas as pd
import qgrid
randn = np.random.randn
df_types = pd.DataFrame({
    'A' : pd.Series(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
               '2013-01-05', '2013-01-06', '2013-01-07', '2013-01-08', '2013-01-09'],index=list(range(9)),dtype='datetime64[ns]'),
    'B' : pd.Series(randn(9),index=list(range(9)),dtype='float32'),
    'C' : pd.Categorical(["washington", "adams", "washington", "madison", "lincoln","jefferson", "hamilton", "roosevelt", "kennedy"]),
    'D' : ["foo", "bar", "buzz", "bippity","boppity", "foo", "foo", "bar", "zoo"] })
df_types['E'] = df_types['D'] == 'foo'
qgrid_widget = qgrid.show_grid(df_types, show_toolbar=True)
qgrid_widget

4. No output is rendered. Error thrown in the console:

Uncaught (in promise) Module qgrid@undefined not found

Source line in widget-manager.ts, in the loadClass function:

                return Promise.reject(`Module ${moduleName}@${moduleVersion} not found`);

Expected behavior
The widget (Qgrid) should display / render.

Desktop:

• OS: MacOS Catalina
• Browser: n/a
• Version: n/a

Additional context
Add any other context about the problem here.

[captainsafia]

Our ipywidgets implementation currently only supports the default set of widgets provided in @jupyter-widgets/controls. We aren't set up to support custom widgets. This would require adding some logic that understands how to load widget dependencies outside the standard set.

If you have any ideas on how to approach this, feel free to share in this thread.

[DonJayamanne]

@captainsafia Back in our prototype for the Python extension, we did manage to get this working. Unfortunately that prototype hasn't seen the light of day for the past 2 months due to other priorities.

The widget manager supports loading dependencies asynchronously, hence I used the pouplar amd library requirejs to load the packages within the widget manager (mixing amd with commonjs):

FYI - this works very well (in our prototype), and I've managed to get most of the 3rd party widgets up and running:

// Source borrowed from https://github.com/jupyter-widgets/ipywidgets/blob/master/examples/web3/src/manager.ts
// tslint:disable: no-any no-console

const cdn = 'https://unpkg.com/';

function moduleNameToCDNUrl(moduleName: string, moduleVersion: string) {
    let packageName = moduleName;
    let fileName = 'index'; // default filename
    // if a '/' is present, like 'foo/bar', packageName is changed to 'foo', and path to 'bar'
    // We first find the first '/'
    let index = moduleName.indexOf('/');
    if (index !== -1 && moduleName[0] === '@') {
        // if we have a namespace, it's a different story
        // @foo/bar/baz should translate to @foo/bar and baz
        // so we find the 2nd '/'
        index = moduleName.indexOf('/', index + 1);
    }
    if (index !== -1) {
        fileName = moduleName.substr(index + 1);
        packageName = moduleName.substr(0, index);
    }
    return `${cdn}${packageName}@${moduleVersion}/dist/${fileName}`;
}

async function requirePromise(pkg: string | string[]): Promise<any> {
    return new Promise((resolve, reject) => {
        const requirejs = (window as any).requirejs;
        if (requirejs === undefined) {
            reject('Requirejs is needed, please ensure it is loaded on the page.');
        } else {
            requirejs(pkg, resolve, reject);
        }
    });
}

function requireLoader(moduleName: string, moduleVersion: string) {
    const requirejs = (window as any).requirejs;
    if (requirejs === undefined) {
        throw new Error('Requirejs is needed, please ensure it is loaded on the page.');
    }
    const conf: { paths: { [key: string]: string } } = { paths: {} };
    conf.paths[moduleName] = moduleNameToCDNUrl(moduleName, moduleVersion);
    requirejs.config(conf);

    return requirePromise([`${moduleName}`]);
}

export class WidgetManager extends HTMLManager {
    public kernel: Kernel.IKernelConnection;
    public el: HTMLElement;
    constructor(kernel: Kernel.IKernelConnection, el: HTMLElement) {
        super({ loader: requireLoader });

[DonJayamanne]

@captainsafia I'm happy to submit a PR if this solution is acceptable.

[captainsafia]

@DonJayamanne Thanks for posting this update and sharing the code snippet!

I like the approach of mapping the package names to an unpkg-based CDN URL and loading client-side dependencies.

Generally, I'd like us to shy away from using RequireJS for module loading but I think it's sensible to have this contributed for now as a compatibility layer until there is a more permanent solution to the widget loading problem.

Would it be possible to scope it so that requirejs isn't registered on the window object?

cc: @rgbkrk @jdfreder

[jasongrout]

Just FYI (as indicated above in the credit to the ipywidgets web3 example), this loading from a CDN using requirejs is also the approach taken by the ipywidgets html manager:

• The loader can be given as a constructor argument
  https://github.com/jupyter-widgets/ipywidgets/blob/c998db931d1e25ba09d210d7d186e13dd4fec2ba/packages/html-manager/src/htmlmanager.ts#L100
• Here is the loader used by the embedding code: https://github.com/jupyter-widgets/ipywidgets/blob/c998db931d1e25ba09d210d7d186e13dd4fec2ba/packages/html-manager/src/libembed-amd.ts#L49-L83

Voila also uses a similar approach, but IIRC can also load the amd modules supplied by custom widgets for the classic notebook. CC @maartenbreddels and @SylvainCorlay.

@SylvainCorlay and I were also talking this last week at the widgets sprint about strengthening this story around loading custom widgets from the web. @wolfv also has a nice demo showing how to load custom widgets from the web in JupyterLab without having to install them.

[jasongrout]

Also, one more thing - we will probably switch from unpkg to jsdelivr as the default cdn for these sorts of things in ipywidgets 8: jupyter-widgets/ipywidgets#1627

[captainsafia]

Updating the issue title and moving this to the new repo where the outputs transforms live.

With @jasongrout's comments in mind, I think we can add fallback logic in the widget implementation to load widgets from jsdeliver. That should address most of the scenarios users have.

[vivek1729]

@captainsafia, just following up on the constructive discussion on this thread. We are looking into adding support for custom ipywidgets and it's great to see some work already in that direction. I am currently doing some learning spikes to get more clarity in the space and would be happy to take it forward. Stay tuned for updates.

[captainsafia]

Sounds good. You might find some of the stuff in #11 helpful.