build metadata connections between dataunderground & GitHub - 't21-hack-connections'

[JustinGOSSES]

This is a fuzzy sketch of a possible hackathon project to make it easier to:

GOAL

Find easy ways to grow connections between open-data and open code that:

• grow community
• shorten "time-to-start"
• more effectively share reusable knowledge across things owned by different people / groups

context

When one goes to an open data site like data underground, there are pages that describe different open-dataset. These provide some information to get you started. There may even be capabilities to preview tabular data. However, it is very common that the potential user has to spend significant time to:

1. Figure out how to load the data
2. Do some exploratory data analysis
   In order to figure out if the data is useful for their goals.

This lengths the "time to start" a lot of side projects that rely on open data as it means a potential user has to spend significant time doing the steps above on potentially multiple datasets before they find one that will work for their project.

This project is built around the idea that aspects of existing platforms for open data (CKAN) and open code (GitHub) could be leveraged to better connect open data with open code that leverages that dataset even if the data and code are owned/created by entirely separate individuals/orgs operating on different platforms. Specifically topics, APIs, and unique dataset identifiers.

If successful, it could have benefits for existing users of datasets, new users of the datasets, and dataset suppliers.

Key resources

• Dataunderground **
• github.com

The rough idea is to leverage:

• Topics tags on GitHub: to tag code repositories as using a dataset from https://dataunderground.org/
• Github API to return all code repositories with a "dataunderground" topic tag.
• Existing code from Lawrence Livermore National Laboratory that leverages the GitHub API and pumps the resulting metadata into a front end webpage that shows statistics on usage, engagement, and shared dependencies across projects could be leveraged to give indications of code similarities and chance of usefulness.
• Either topic tag or README section that can be parsed to return the specofic dataset on dataunderground that is being used. Unsure at this time if CKAN has unique identifiers for datasets.
• Links added to "dataunderground" that point from the dataset page to the static code catalog that has links to all the code repos that use that dataset.

In order to create the user experience:

USER LOOKING FOR DATASETS TO USE

• User goes to dataunderground and finds potentially useful datasets
• User can click from dataset page and go to another webpage (likely github pages page deployed on github) with short content extracted from metadata via API that describes what code has been written that leverages that dataset.
• User now has code to use that shows how to load and prep the dataset even if they aren't going the exact same thing.
• User may now have a connection they can reach out to that is doing similar things.

USER WHO HAS CREATED A CODE PROJECT WITH A DATASET DESCRIBED FROM DATAUNDERGOUND

• This user now gets additional social credit from using a dataset from dataunderground as it directs potential users to his project who might star, fork, or contribute to it.
• This user has to add a topic tag to their site to be cataloged in the catalog website and get more visibility.

People

None so far but Justin Gosses might contribute. Feel free to say something in the discussion section of possibly interested.

Skills

• Being able to think like different users
• Be able to spend time on something that might build community or might fail.
• Little HTML & CSS would help
• Knowledge or willingness to learn about GitHub API would help.
• Knowledge or willingness to learn about GitHubActions would help.

Will need people willing to add topic tags to their existing code repositories that use datasets described on dataunderground.

For non-coders, we need people to:

• Might need someone with access to make edits to dataset page(s) on dataunderground.
• Find existing code repositories on github.com that use datasets on dataunderground on add issues or discussion asking them to add the "dataunderground" topic tag.
• Write documentation describing the context and why code project owners should add the "dataunderground" tag to their code projects.

[filippo82]

T21 Hackathon Idea Mashup today (or tomorrow depending on your time zone) Wednesday April 14th at 16:00 UTC. Come to Slack to find the link for the meeting!

[JustinGOSSES]

Added non-coders needs section