TART_EXECUTOR_HOST_DIR breaking builds with Git LFS

[olejnjak]

Hi guys,

I wanted to utilize TART_EXECUTOR_HOST_DIR to see if we can manage to get some perf gains. But once set job fails when trying to fetch source code. It is the same when using --builds-dir in config stage.

All I get is this output

Running with gitlab-runner 17.5.3 (12030cf4)
  on <runner>, system ID: <runnerid>

Preparing the "custom" executor
Using Custom executor...
2024/11/16 15:49:01 Pulling the latest version of ghcr.io/cirruslabs/macos-sequoia-xcode:16.1...
2024/11/16 16:41:35 Cloning and configuring a new VM...
2024/11/16 16:41:35 Waiting for the VM to boot and be SSH-able...
2024/11/16 16:41:49 Was able to SSH!
2024/11/16 16:41:49 Mounting hostdir...
2024/11/16 16:41:50 VM is ready.

Preparing environment
Running on admins-Virtual-Machine.local...

Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in /Users/admin/hostdir/iOS/bazos/.git/
Created fresh repository.
Checking out e74a9577 as detached HEAD (ref is refs/merge-requests/1155/head)...
Could not pull
Errors logged to '/Users/admin/hostdir/iOS/bazos/.git/lfs/logs/20241116T074200.120681.log'.
Use `git lfs logs last` to view the log.
2024/11/16 16:42:00 Process exited with status 2

Cleaning up project directory and file based variables
ERROR: Job failed: exit status 1

The mentioned log file contains this, which seem extremely weird:

git-lfs/3.5.1 (GitHub; darwin arm64; go 1.23.1)
git version 2.39.5 (Apple Git-154)

$ git-lfs pull
Could not pull

Git can't resolve ref: "HEAD"

Current time in UTC:
2024-11-16 16:42:00

Environment:
LocalWorkingDir=/Users/admin/buildsdir/iOS/bazos
LocalGitDir=/Users/admin/buildsdir/iOS/bazos/.git
LocalGitStorageDir=/Users/admin/buildsdir/iOS/bazos/.git
LocalMediaDir=/Users/admin/buildsdir/iOS/bazos/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/admin/buildsdir/iOS/bazos/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/admin/buildsdir/iOS/bazos/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_DEPTH=10
GIT_EXEC_PATH=/Applications/Xcode-16.1.0.app/Contents/Developer/usr/libexec/git-core
GIT_LFS_SKIP_SMUDGE=1
GIT_STRATEGY=fetch

Client IP addresses:
fe80::18ce:aa2d:6336:bd8 fd26:5fb8:584e:1f6f:d9:5385:28a8:39f1 192.168.64.2
fe80::7cc9:987f:9153:5d0a
fe80::734f:657c:e782:baeb
fe80::3efa:4c3e:d361:8e5b
fe80::ce81:b1c:bd2c:69e

With builds dir inside of the VM, everything is working fine.

Do you have experience with such issue? Thanks :-)

[fkorotkov]

It's probably this issue we don't have a workaround for 😔 cirruslabs/tart#567 (comment)

[olejnjak]

Okay, sounds reasonable 🙃 might be worth a note in docs though 🤔

[olejnjak]

Hi @fkorotkov,

sorry for reopening, but I got an idea of using a ramdisk or tmpfs as build dir. Have you thought about it or tried, before I try implementing it? Storing build dir in RAM might be a pretty efficient solution at least for projects that are not I would say "giant"? 🤔

[fkorotkov]

@olejnjak do you mean pass a ramdisk from the host? Or use it inside the VM?

[olejnjak]

I meant inside the VM, I assume passing it from the host would trigger the same issue with virtiofs, wouldn't it?

[edigaryev]

I meant inside the VM, I assume passing it from the host would trigger the same issue with virtiofs, wouldn't it?

Are you using AWS EBS by a chance?

We've seen some reports where adding --root-disk-opts=sync=none,caching=cached to tart run significantly improves the speed when using EBS volumes: cirruslabs/tart#949.

[olejnjak]

Nope our CI is running on self-hosted macs.

[edigaryev]

Regarding the RAM disk, you can already have it with the following GitLab CI configuration, thanks to pre_get_sources_script:

image: ghcr.io/cirruslabs/macos-sonoma-base:latest

job:
  hooks:
    pre_get_sources_script: |
      # Preserve an already created builds directory as it might contain
      # important files like "gitlab_runner_env" and "CI_SERVER_TLS_CA_FILE"
      rsync -az "$CI_BUILDS_DIR/" "$CI_BUILDS_DIR.orig/"

      # Create 10 GiB RAM disk (note that the size is in 512-byte blocks, not in bytes)
      DISK=$(hdiutil attach -nomount ram://20971520 | tr -d '[:space:]')
      diskutil erasevolume HFS+ RAMDISK "$DISK"

      # Remount the created RAM disk to our builds directory
      diskutil umount "$DISK"
      diskutil mount -mountPoint "$CI_BUILDS_DIR" "$DISK"

      # Restore builds directory contents
      rsync -az "$CI_BUILDS_DIR.orig/" "$CI_BUILDS_DIR/"
      rm -rf "$CI_BUILDS_DIR.orig"
  script: |
    pwd
    mount

Regarding the --root-disk-opts I've mentioned in #92 (comment), you'll be able to customize it via TART_EXECUTOR_ROOT_DISK_OPTS environment variable after #96 is merged and released.

Also, in the future, please open a separate issue for such ideas: this will both allow us to track and for other users of GitLab Tart Executor to search them more easily. Thank you 🙌