Improve Portal Documentation (Data Catalog, How it Works, ...)

[davidgasquez]

We should expose both schemas and samples for all the curated datasets.

This will improve UX and make choosing datasets easier!

[davidgasquez]

@DistributedDoge mentioned to publish a notebook that loops over tables or whatever export dir is, fetches schema from .parquetfiles and surfaces that.

[DistributedDoge]

The code has landed, catalog will be built next time you update github pages by doing make publish.

Just remember to do make run first so that notebook can access table schema from which catalog is being built.

[davidgasquez]

The code has landed, catalog will be built next time you update github pages by doing make publish. 🤦

I thought the website was published with each push! 🤷 Got confussed with the Filecoin one.

Created an issue now: #36

[davidgasquez]

Working! 🎉

https://davidgasquez.github.io/gitcoin-grants-data-portal/catalog.html

[DistributedDoge]

Pretty neat. Three things I will try to add later:

• download link to each file, made easier thanks to stable links in Add IPNS support #18
• file size + row count so folks know how much data they have on hand
• better display for columns with nested schema

[davidgasquez]

Sharing it here so I remember in the future.

Would be awesome to aim for something like this: https://py-code.org/datasets

Nice UX and UI!

[DistributedDoge]

Also, on catalog side,

• Detailed descriptions for 3 most used tables (rounds, projects, rounds_votes)
• Introductory paragraph (data is mainly from indexer, updated at least weekly etc.)

Inspiration:

https://docs.passport.gitcoin.co/building-with-passport/passport-api/data-dictionary

[davidgasquez]

Cool find! I think we can do something similar with Dagster assets metadata.

Similar to what Subsets does.

Not sure how to deal with dbt models though! Perhaps we can extract the docs from the YAML files or even better, make Dagster understand dbt docs. 🤔