`gecko-dev` full checkout takes 10m

[abitrolly]

A shallow clone of https://github.com/mozilla/gecko-dev repo took 10m using this step (as of 2022-06-03)

      - uses: actions/checkout@v3
        with:
          repository: mozilla/gecko-dev
          path: gecko-dev
          fetch-depth: 0

https://github.com/abitrolly/firefox-lang-stats/runs/6713143940?check_suite_focus=true

Is it possible to speed up it somehow? Like through https://docs.github.com/en/rest/repos/contents#download-a-repository-archive-zip

[abitrolly]

Explicitly mentioning branch makes clone 1m slower.

      - uses: actions/checkout@v3
        with:
          repository: mozilla/gecko-dev
          path: gecko-dev
          ref: master
          fetch-depth: 0

But the clone commands are the same https://github.com/abitrolly/firefox-lang-stats/runs/6716850948?check_suite_focus=true

[abitrolly]

Like through https://docs.github.com/en/rest/repos/contents#download-a-repository-archive-zip

These ZIP repository archives are missing .git dir.

$ curl -L \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/abitrolly/firefox-lang-stats/zipball/main --output wow.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 53643    0 53643    0     0  88864      0 --:--:-- --:--:-- --:--:-- 88864                                                                                       /0.6s
$ unzip wow.zip 
Archive:  wow.zip
eea523c6772f8b6342a9121a175be9bd8be16c87
   creating: abitrolly-firefox-lang-stats-eea523c/
   creating: abitrolly-firefox-lang-stats-eea523c/.github/
   creating: abitrolly-firefox-lang-stats-eea523c/.github/workflows/
  inflating: abitrolly-firefox-lang-stats-eea523c/.github/workflows/deploy.yml  
 extracting: abitrolly-firefox-lang-stats-eea523c/.gitignore  
  inflating: abitrolly-firefox-lang-stats-eea523c/01stats.sh  
  inflating: abitrolly-firefox-lang-stats-eea523c/README.md  
   creating: abitrolly-firefox-lang-stats-eea523c/dev/
  inflating: abitrolly-firefox-lang-stats-eea523c/dev/build-data  
  inflating: abitrolly-firefox-lang-stats-eea523c/dev/git-file-stats  
  inflating: abitrolly-firefox-lang-stats-eea523c/index.mustache  
  inflating: abitrolly-firefox-lang-stats-eea523c/rustacean-orig-noshadow.ico  
  inflating: abitrolly-firefox-lang-stats-eea523c/rustacean-orig-noshadow.png

[abitrolly]

Maybe it is possible to cache repo clone? https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

[MartyIX]

• uses: actions/checkout@v3
  with:
  repository: mozilla/gecko-dev
  path: gecko-dev
  fetch-depth: 0

Should the fetch-depth be 1 to download just shallow copy? 0 means to download full git history but I understand you just need the latest snapshot.

https://git-scm.com/docs/git-clone explains --depth (which is what fetch-depth set, I hope):

(I'm coming from 4e6/firefox-lang-stats#12)

[abitrolly]

@MartyIX it is not evident, but I wanted to build a graph of language use in Firefox day by day, that's why a full clone is needed.

[MartyIX]

Oh, I see. Given that the repo is big, I would guess that there is not really that much you can do. Even if you find a way to speed it up, I don't believe you should use Github Actions to download the complete repo every day. That feels quite wasteful resource-wise (even if it's somewhat free).

I thought you wanted to get shallow copy of the repo, do the stat for the day and write it somewhere. That's a bit better imho.

[knowledgebreak]

No check out done

[abitrolly]

I thought you wanted to get shallow copy of the repo, do the stat for the day and write it somewhere.

@MartyIX the problem is to get checkout for the day.

[MartyIX]

btw: The repo migrated from https://github.com/mozilla/gecko to https://github.com/mozilla/firefox

Measure-Command { git clone https://github.com/mozilla-firefox/firefox --depth=1 | Out-Default }

Cloning into 'firefox'...
remote: Enumerating objects: 402129, done.
remote: Counting objects: 100% (402129/402129), done.
remote: Compressing objects: 100% (294830/294830), done.
remote: Total 402129 (delta 129396), reused 247710 (delta 89380), pack-reused 0 (from 0)
Receiving objects: 100% (402129/402129), 859.50 MiB | 5.52 MiB/s, done.
Resolving deltas: 100% (129396/129396), done.
Updating files: 100% (387865/387865), done.

Days              : 0
Hours             : 0
Minutes           : 5
Seconds           : 7
Milliseconds      : 163
Ticks             : 3071635406
TotalDays         : 0,00355513357175926
TotalHours        : 0,0853232057222222
TotalMinutes      : 5,11939234333333      <<---
TotalSeconds      : 307,1635406
TotalMilliseconds : 307163,5406

On my machine it took 5 minutes to get a shallow copy of the repo.

One way to achieve the daily stats would be to have a github repo with the following structure:

• script.sh
• firefox <-- pre-cloned repo of https://github.com/mozilla-firefox/firefox

Now the script.sh would run every day and it would

1. fetch the latest changes for the firefox repo,
2. compute file stats and
3. published it somewhere.

[abitrolly]

btw: The repo migrated from https://github.com/mozilla/gecko to https://github.com/mozilla/firefox

What a coincidence.. :D