Skip to content

Add WikiCommons Data Source (Phase 1) #183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
+1,954 −0
Open

Add WikiCommons Data Source (Phase 1) #183

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
69c8982
Add WikiCommons fetch script (Phase 1)
najuna-brian Oct 12, 2025
70340cd
Add WikiCommons process script (Phase 2)
najuna-brian Oct 12, 2025
6e580c8
Add WikiCommons report script (Phase 3)
najuna-brian Oct 12, 2025
c1ddb36
Include Phase 1 CSV and updated quarterly README
najuna-brian Oct 12, 2025
1380858
Remove report files and updates
najuna-brian Oct 14, 2025
0998bbd
refactor: Move LICENSE_NORMALIZATION to shared module
najuna-brian Oct 14, 2025
6f4c732
Restoring report scripts for GCS, GitHub, and notes (keeping WikiComm…
najuna-brian Oct 14, 2025
8105eae
Merge branch 'creativecommons:main' into main
najuna-brian Oct 15, 2025
0c6650d
docs(shared): updated the license normalization comment to clarify SP…
najuna-brian Oct 17, 2025
7e4e532
Moving and Renaming RETRY_STATUS_CODES to shared.RETRY_STATUS_FORCELIST
najuna-brian Oct 20, 2025
f4bbc81
Moving USER_AGENT constant to shared library for reuse
najuna-brian Oct 20, 2025
9c08e3f
Merge branch 'creativecommons:main' into main
najuna-brian Oct 21, 2025
e2d1d97
Raise exceptions in fetch_wikicommons.py to stop execution on errors
najuna-brian Oct 21, 2025
6d51ff4
Updating WikiCommons error handling
najuna-brian Oct 22, 2025
a2b62da
Sorting Constants
najuna-brian Oct 27, 2025
3b01e2d
Using correct Creative Commons legal tool identifiers
najuna-brian Oct 27, 2025
2c3fe37
Skipping categories with no data instead of returning zero counts
najuna-brian Oct 27, 2025
8cfd2b8
Specifying encoding=utf-8 to ensure cross-platform compatibility
najuna-brian Oct 27, 2025
4b58f1e
Removing redundant try/except in query_wikicommons
najuna-brian Oct 27, 2025
8ed1f57
Limit WikiCommons implementation to Phase 1 (Fetch) only
najuna-brian Oct 28, 2025
633e4f1
Removing WikiCommons Phase 1 (Fetch) generated data files
najuna-brian Oct 28, 2025
25f73b3
Merge branch 'creativecommons:main' into main
najuna-brian Oct 28, 2025
3bb2638
Remove unnecessary try block causing syntax error
najuna-brian Oct 30, 2025
1ec4388
Sort constants alphabetically
najuna-brian Oct 30, 2025
c16d094
Move script execution log to main function
najuna-brian Oct 30, 2025
75c6b87
Update backoff_factor to 10 for consistency
najuna-brian Oct 30, 2025
73cf1b6
Remove newline parameter for reading to use universal newlines
najuna-brian Oct 30, 2025
8403519
use explicit newline for writing CSV files
najuna-brian Oct 30, 2025
b46057a
Rename to wikicommons_fetch.py
najuna-brian Oct 30, 2025
3fcb3c7
Name correction
najuna-brian Oct 30, 2025
ed4775d
Fixed Indention
najuna-brian Oct 30, 2025
df71aa1
Removed data/2025Q4/README.md to from the PR
najuna-brian Oct 30, 2025
63158e6
Change to return a default dictionary instead of None
najuna-brian Oct 30, 2025
f4a793a
Added encoding parameter to file reading in check_for_completion and …
najuna-brian Oct 30, 2025
0153e81
Correct Wikimedia Commons API queries to return accurate file counts
najuna-brian Nov 6, 2025
8177996
Restoring backoff_factor to 10
najuna-brian Nov 6, 2025
bdc8c8e
Merge branch 'main' into main
najuna-brian Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions data/2025Q4/1-fetch/wikicommons_1_count.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
"LICENSE","FILE_COUNT","PAGE_COUNT"
"CC-BY-4.0","4938662","1"
"CC-BY-SA-4.0","31157944","2"
"CC-BY-NC-4.0","3","1"
"CC-BY-NC-SA-4.0","235","0"
"CC-BY-NC-ND-4.0","0","0"
"CC-BY-ND-4.0","0","0"
"CC-BY-3.0","3524336","124"
"CC-BY-SA-3.0","14711405","1"
"CC-BY-NC-3.0","7360","2"
"CC-BY-NC-SA-3.0","0","0"
"CC-BY-NC-ND-3.0","0","0"
"CC-BY-ND-3.0","0","0"
"CC-BY-2.5","460944","0"
"CC-BY-SA-2.5","2290286","0"
"CC-BY-NC-2.5","0","0"
"CC-BY-NC-SA-2.5","0","0"
"CC-BY-NC-ND-2.5","0","0"
"CC-BY-ND-2.5","0","0"
"CC-BY-2.0","7750438","0"
"CC-BY-SA-2.0","15475190","13"
"CC-BY-NC-2.0","0","0"
"CC-BY-NC-SA-2.0","0","0"
"CC-BY-NC-ND-2.0","0","0"
"CC-BY-ND-2.0","0","0"
"CC-BY-1.0","28737","0"
"CC-BY-SA-1.0","1922661","1"
"CC-BY-NC-1.0","0","0"
"CC-BY-NC-SA-1.0","0","0"
"CC-BY-NC-ND-1.0","0","0"
"CC-BY-ND-1.0","0","0"
"CC0-1.0","0","0"
"PDM-1.0","0","0"
34 changes: 34 additions & 0 deletions data/2025Q4/2-process/wikicommons_2_processed.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
"LICENSE","LICENSE_TYPE","LICENSE_VERSION","BASE_LICENSE","FILE_COUNT","FILE_PERCENTAGE","PAGE_COUNT","PAGE_PERCENTAGE","IS_FREE_CULTURAL","IS_PUBLIC_DOMAIN"
"CC BY-SA 4.0","CC BY-SA","4.0","BY-SA","31157944","37.87","2","1.38","True","False"
"CC BY-SA 2.0","CC BY-SA","2.0","BY-SA","15475190","18.81","13","8.97","True","False"
"CC BY-SA 3.0","CC BY-SA","3.0","BY-SA","14711405","17.88","1","0.69","True","False"
"CC BY 2.0","CC BY","2.0","BY","7750438","9.42","0","0.0","True","False"
"CC BY 4.0","CC BY","4.0","BY","4938662","6.0","1","0.69","True","False"
"CC BY 3.0","CC BY","3.0","BY","3524336","4.28","124","85.52","True","False"
"CC BY-SA 2.5","CC BY-SA","2.5","BY-SA","2290286","2.78","0","0.0","True","False"
"CC BY-SA 1.0","CC BY-SA","1.0","BY-SA","1922661","2.34","1","0.69","True","False"
"CC BY 2.5","CC BY","2.5","BY","460944","0.56","0","0.0","True","False"
"CC BY 1.0","CC BY","1.0","BY","28737","0.03","0","0.0","True","False"
"CC BY-NC 3.0","CC BY-NC","3.0","BY-NC","7360","0.01","2","1.38","False","False"
"CC BY-NC-SA 4.0","CC BY-NC-SA","4.0","BY-NC-SA","235","0.0","0","0.0","False","False"
"CC BY-NC 4.0","CC BY-NC","4.0","BY-NC","3","0.0","1","0.69","False","False"
"CC BY-NC 2.0","CC BY-NC","2.0","BY-NC","0","0.0","0","0.0","False","False"
"CC BY-NC 2.5","CC BY-NC","2.5","BY-NC","0","0.0","0","0.0","False","False"
"CC BY-NC 1.0","CC BY-NC","1.0","BY-NC","0","0.0","0","0.0","False","False"
"CC BY-NC-SA 1.0","CC BY-NC-SA","1.0","BY-NC-SA","0","0.0","0","0.0","False","False"
"CC BY-NC-ND 4.0","CC BY-NC-ND","4.0","BY-NC-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-ND 3.0","CC BY-NC-ND","3.0","BY-NC-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-ND 2.5","CC BY-NC-ND","2.5","BY-NC-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-ND 2.0","CC BY-NC-ND","2.0","BY-NC-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-ND 1.0","CC BY-NC-ND","1.0","BY-NC-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-SA 2.5","CC BY-NC-SA","2.5","BY-NC-SA","0","0.0","0","0.0","False","False"
"CC BY-NC-SA 2.0","CC BY-NC-SA","2.0","BY-NC-SA","0","0.0","0","0.0","False","False"
"CC BY-ND 3.0","CC BY-ND","3.0","BY-ND","0","0.0","0","0.0","False","False"
"CC BY-ND 2.5","CC BY-ND","2.5","BY-ND","0","0.0","0","0.0","False","False"
"CC BY-ND 2.0","CC BY-ND","2.0","BY-ND","0","0.0","0","0.0","False","False"
"CC BY-ND 1.0","CC BY-ND","1.0","BY-ND","0","0.0","0","0.0","False","False"
"CC BY-NC-SA 3.0","CC BY-NC-SA","3.0","BY-NC-SA","0","0.0","0","0.0","False","False"
"CC BY-ND 4.0","CC BY-ND","4.0","BY-ND","0","0.0","0","0.0","False","False"
"CC0 1.0","CC0","1.0","CC0","0","0.0","0","0.0","False","True"
"PDM 1.0","PDM","1.0","PDM","0","0.0","0","0.0","False","True"
"TOTAL","TOTAL","ALL","ALL","82268201","100.0","145","100.0","False","False"
Binary file added data/2025Q4/3-report/wikicommons_summary.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
29 changes: 29 additions & 0 deletions data/2025Q4/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Quantifying the Commons 2025Q4

<!-- WikiCommons Start -->


## WikiCommons


<!-- Overview Start -->

### Overview

![WikiCommons Creative Commons License Distribution](3-report\wikicommons_summary.png)

WikiCommons Creative Commons License Distribution

WikiCommons data represents Creative Commons licensed media files (images, videos, audio) from Wikimedia Commons.

**The results indicate there are a total of 82,268,201 CC-licensed media files in WikiCommons.**

The most popular license is CC BY-SA 4.0 with 31,157,944 files (37.9%).

Thank you to the WikiCommons community for providing this valuable collection of openly licensed media!

<!-- Overview End -->


<!-- WikiCommons End -->

Loading