Clean-up: Remove binaries like tugboat from the repo (faster download/git-cloning)

[klonos]

Adding this as a follow-up to #648

...perhaps we should also use something like https://rtyley.github.io/bfg-repo-cleaner to clean the binary(ies) from the git index/history ...takes less time to clone 😉

[klonos]

This is a test "plain" git clone, to be used to compare against the next command:

git clone https://github.com/backdrop-ops/backdropcms.org
Cloning into 'backdropcms.org'...
remote: Enumerating objects: 955, done.
remote: Counting objects: 100% (955/955), done.
remote: Compressing objects: 100% (834/834), done.
remote: Total 34214 (delta 224), reused 403 (delta 94), pack-reused 33259
Receiving objects: 100% (34214/34214), 61.57 MiB | 1.37 MiB/s, done.
Resolving deltas: 100% (21280/21280), done.

The actual git clone --mirror, which clones a "bare" copy of the repo:

git clone --mirror https://github.com/backdrop-ops/backdropcms.org
Cloning into bare repository 'backdropcms.org.git'...
remote: Enumerating objects: 1103, done.
remote: Counting objects: 100% (1103/1103), done.
remote: Compressing objects: 100% (939/939), done.
remote: Total 35346 (delta 290), reused 500 (delta 133), pack-reused 34243
Receiving objects: 100% (35346/35346), 61.84 MiB | 1.37 MiB/s, done.
Resolving deltas: 100% (22104/22104), done.

Notice that the directory created has .git appended to it. It will not have a locally checked-out copy of any of the files under revision control. That is, all of the Git administrative and control files that would normally be present in a hidden .git sub-directory are directly present in a repository.git directory instead, and no other files are present and checked out. Also notice the difference in the number of objects.

[klonos]

Great, now that we have all the objects that have been committed over time, let's clean up the tugboat binary from the repo clone...

Download the bfg tool:

wget -O bfg.jar https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar
--2020-05-22 00:55:29--  https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar
Resolving repo1.maven.org (repo1.maven.org)... 151.101.16.209
Connecting to repo1.maven.org (repo1.maven.org)|151.101.16.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13465496 (13M) [application/java-archive]
Saving to: ‘bfg.jar’

bfg.jar   100%[=======================>]  12.84M  1.38MB/s    in 9.4s

2020-05-22 00:55:39 (1.37 MB/s) - ‘bfg.jar’ saved [13465496/13465496]

[klonos]

Remove the tugboat binary:

java -jar bfg.jar --delete-files tugboat backdropcms.org.git

Using repo : /Users/greg/backdrop/backdropcms.org.git

Found 3563 objects to protect
Found 125 commit-pointing refs : HEAD, refs/heads/301-new-theme-based-on-homepage-designs, refs/heads/314-tryout-backddrop-page, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 25532976 (protected by 'HEAD')

Cleaning
--------

Found 1454 commits
Cleaning commits:       100% (1454/1454)
Cleaning commits completed in 594 ms.

Updating 56 Refs
----------------

	Ref                                    Before     After   
	----------------------------------------------------------
	refs/heads/544/fixDlCountsCore       | 06a13938 | c0e50f6c
	refs/heads/560-NewsFeedEndPoint      | a2c470db | 4f82ebf5
	refs/heads/678/latestDB              | dae477d6 | c942ee48
	refs/heads/json                      | 6b6cbe27 | 2d5e98e2
	refs/heads/klonos-demo-page-messages | 9861405a | 5c6f1a5e
	refs/heads/mailchimp-signup          | 585553b6 | e68a813e
	refs/heads/mailchimpSignupOnProfile  | 0fbc9189 | bc7391ba
	refs/heads/master                    | 25532976 | 49efab62
	refs/pull/456/merge                  | a01590bb | 00226cf7
	refs/pull/488/merge                  | 17f72fd9 | 9eba05dc
	refs/pull/495/head                   | 1eaa43ef | 9b0f3944
	refs/pull/496/head                   | 6bf6f104 | 5359c39c
	refs/pull/502/head                   | 737fc566 | 9aeb9fbf
	refs/pull/502/merge                  | 7814c4d3 | 16846ffc
	refs/pull/535/head                   | 95ae0a01 | a596977a
	...

Updating references:    100% (56/56)
...Ref update completed in 40 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	.............................................DDDDDDDDDDDDDmm

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After   
	-------------------------------------------
	First modified commit | 53a0eabd | 41c5a33e
	Last dirty commit     | 17d5c5e1 | bbc81810

Deleted files
-------------

	Filename   Git id                                     
	------------------------------------------------------
	tugboat  | 84405bf0 (36.5 MB), 64863f03 (36.5 MB), ...


In total, 573 object ids were changed. Full details are logged here:

	/Users/greg/backdrop/backdropcms.org.git.bfg-report/2020-05-22/00-58-59

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

[klonos]

Do as the message says: cd into the bare repo dir

cd backdropcms.org.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
Enumerating objects: 35324, done.
Counting objects: 100% (35324/35324), done.
Delta compression using up to 16 threads
Compressing objects: 100% (34375/34375), done.
Writing objects: 100% (35324/35324), done.
Selecting bitmap commits: 1439, done.
Building bitmaps: 100% (119/119), done.
Total 35324 (delta 23164), reused 10578 (delta 0)
git push

Now this is a part that will require some comms/coordination, so that everyone is across it:

At this point, you're ready for everyone to ditch their old copies of the repo and do fresh clones of the nice, new pristine data. It's best to delete all old clones, as they'll have dirty history that you don't want to risk pushing back into your newly cleaned repo.

[klonos]

Before:

After the tugboat binary has been removed:

I'm confused... What Tugboat binary? We dont use it and removed it.

[klonos]

@BWpanda when files are being committed to a repo, and then removed, the file remains in the git history. That's not an issue with small-sized files, but when huge binaries are being added/removed, then these remain in the git history, which is what is downloaded when one does git clone ...this is especially noticeable for people on slow connections.

You can test this by creating a test repo, then committing a large file (like a video or an iso image), then removing that file with another commit. If you try to clone the repo again, it will take ages. That's because the huge file is also being downloaded as an "object", despite having been deleted before.

I hope that this is clear now.

[klonos]

...this has been discussed around the web in various articles. Here's one example: https://support.acquia.com/hc/en-us/articles/360004334093-Removing-large-files-from-Git-without-losing-history

Ah OK, thanks for the clarification 😊

I'm all for this idea, but

Now this is a part that will require some comms/coordination, so that everyone is across it:

is the part that worries me... How would we practically do that?

I've posted this question to see if we can somehow 'automate' the checking of future PRs to make sure they don't muck up the work we do here: rtyley/bfg-repo-cleaner#421