Tebako fails to build if bundler version is different than in Gemfile.lock

[d4rky-pl]

Discovered while debugging #161

It seems to be a problem with the binary gems (ffi, jaro_winkler) and the Ruby interpreter used between bundle install and bundle exec gem build changing if your project has both gemspec and Gemfile. I did not check for other scenarios.

I've modified the tebako code to run bundle env before and after the install process and ran everything in verbose mode. This was the result:

The env before the install:

Bundler       2.4.22
  Platforms   ruby, arm64-darwin-23
Ruby          3.2.4p170 (2024-04-23 revision af471c0e0127eea0cafa6f308c0425bbfab0acf5) [arm64-darwin-23]
  Full Path   /Users/matyas/.tebako/o/s/bin/ruby
  Config Dir  /Users/matyas/.tebako/o/s/etc
RubyGems      3.4.19
  Gem Home    /Users/matyas/.tebako/o/s/lib/ruby/gems/3.2.0
  Gem Path    /Users/matyas/.tebako/o/s/lib/ruby/gems/3.2.0
  User Home   /Users/matyas
  User Path   /Users/matyas/.gem/ruby/3.2.0
  Bin Dir     /Users/matyas/.tebako/o/s/bin
Tools
  Git         2.42.1
  RVM         not installed
  rbenv       not installed
  chruby      not installed

The env after install:

Bundler       2.4.18
  Platforms   ruby, arm64-darwin-23
Ruby          3.2.4p170 (2024-04-23 revision af471c0e0127eea0cafa6f308c0425bbfab0acf5) [arm64-darwin-23]
  Full Path   /Users/matyas/.asdf/installs/ruby/3.2.4/bin/ruby
  Config Dir  /Users/matyas/.asdf/installs/ruby/3.2.4/etc
RubyGems      3.4.19
  Gem Home    /Users/matyas/.tebako/o/s/lib/ruby/gems/3.2.0
  Gem Path    /Users/matyas/.tebako/o/s/lib/ruby/gems/3.2.0
  User Home   /Users/matyas
  User Path   /Users/matyas/.gem/ruby/3.2.0
  Bin Dir     /Users/matyas/.tebako/o/s/lib/ruby/gems/3.2.0/bin
Tools
  Git         2.42.1
  RVM         not installed
  rbenv       not installed
  chruby      not installed

The bundler version change happened because my Gemfile.lock was bundled with 2.4.18 instead of 2.4.22. It seems that automatic bundler version switch messes up with which Ruby interpreter will be picked which in turn somehow breaks the binary gems resolution. In this case the ruby version is the same but in different path, on CI it was completely different (3.3.0, likely either system or rbenv default one)

Manually modifying Gemfile.lock and swapping the bundler version fixed the problem and allowed me to proceed further.

[d4rky-pl]

I'm no longer sure if it's just the bundler switching. I'm running into the same building issue again, even though I'm modifying Gemfile.lock on the fly before touching anything else to ensure bundler version is the same as system on which it's running (Semaphore currently has different bundler versions installed by default per environment so I was not able to just sync everything to ensure the same bundler).

Here's the log for one of the runs but the issue exists on both linux and Mac:

Tebako executable packager version 0.8.6
No prefix specified, using ~/.tebako
Running tebako press at /Users/semaphore/.tebako
   Ruby version:              '3.2.4'
   Project root:              '/Users/semaphore/helpling_toolbelt/release/'
   Application entry point:   'ht'
   Package file name:         '/Users/semaphore/helpling_toolbelt/release/ht'
   Loging level:              'error'
   Package working directory: '<Host current directory>'
-- Using brew environment at /opt/homebrew
-- Found bison keg installed by Homebrew at /opt/homebrew/opt/bison
-- Found flex keg installed by Homebrew at /opt/homebrew/opt/flex
-- Found bash keg installed by Homebrew at /opt/homebrew/opt/bash
Running tebako press script
Packaging /Users/semaphore/helpling_toolbelt/release/ folder with ht entry point to /Users/semaphore/helpling_toolbelt/release/ht
-- OSTYPE: 'darwin23.4.0'
-- NCORES: 4
Configuration summary:
-- ruby: v3.2.4 at /Users/semaphore/.tebako/deps/src/_ruby_3.2.4
-- dwarfs with tebako wrapper: @v0.5.9 at /Users/semaphore/.tebako/deps/src/_dwarfs_wr
-- DATA_SRC_DIR: /Users/semaphore/.tebako/o/s
-- DATA_PRE_DIR: /Users/semaphore/.tebako/o/r
-- DATA_BIN_DIR: /Users/semaphore/.tebako/o/p
-- DATA_BIN_FILE: /Users/semaphore/.tebako/o/p/fs.bin
-- Target binary directory: /Users/semaphore/.tebako/o/s/bin
-- Target library directory: /Users/semaphore/.tebako/o/s/lib
-- Target local directory: /Users/semaphore/.tebako/o/s/local
-- Target Gem directory:: /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0
-- FS_MOUNT_POINT: /__tebako_memfs__
-- Building for Win32 Ruby (RB_W32): OFF
-- Removing GLIBC_PRIVATE reference: OFF
-- Not building Ruby extensions: dbm,win32,win32ole,-test-/*
-- Ruby build cflags='-fPIC -I/Users/semaphore/.tebako/deps/include -I/Users/semaphore/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/tebako-0.8.6/include -fdeclspec -I/opt/homebrew/opt/openssl@3/include -I/opt/homebrew/opt/zlib/include -I/opt/homebrew/include'
-- Ruby build LDFLAGS='-L/Users/semaphore/.tebako/deps/lib -L/Users/semaphore/.tebako/o -L/opt/homebrew/opt/openssl@3/lib -L/opt/homebrew/opt/zlib/lib  -L/opt/homebrew/lib'
-- openssl Ruby option='--with-openssl-dir=/opt/homebrew/opt/openssl@3'
-- libyaml Ruby option=''
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/semaphore/.tebako/o
[ 42%] Built target _dwarfs_wr
[ 84%] Built target _ruby_3.2.4
Tebako setup has been verified
[ 84%] Built target setup
-- Running init script
   ... creating packaging environment at /Users/semaphore/.tebako/o/s
-- Running deploy script
'tebako-runtime' was not loaded.
RubyGems Environment:
  - RUBYGEMS VERSION: 3.4.19
  - RUBY VERSION: 3.2.4 (2024-04-23 patchlevel 170) [arm64-darwin23]
  - INSTALLATION DIRECTORY: /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0
  - USER INSTALLATION DIRECTORY: /Users/semaphore/.local/share/gem/ruby/3.2.0
  - RUBY EXECUTABLE: /Users/semaphore/.tebako/o/s/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /Users/semaphore/.tebako/o/s/bin
  - SPEC CACHE DIRECTORY: /Users/semaphore/.tebako/o/s/spec_cache
  - SYSTEM CONFIGURATION DIRECTORY: /Users/semaphore/.tebako/o/s/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - arm64-darwin-23
  - GEM PATHS:
     - /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /Users/semaphore/.rbenv/versions/3.3.0/bin
     - /opt/homebrew/Cellar/rbenv/1.3.0/libexec
     - /opt/homebrew/opt/bison/bin
     - /Users/semaphore/.nvm/versions/node/v18.20.1/bin
     - /Users/semaphore/flutter
     - /Users/semaphore/flutter/bin/
     - /Users/semaphore/flutter/bin/cache/dart-sdk/bin
     - /usr/local/bin/
     - /Users/semaphore/.pyenv/shims
     - /Users/semaphore/.rbenv/shims
     - /opt/homebrew/bin
     - /opt/homebrew/sbin
     - /usr/local/bin
     - /System/Cryptexes/App/usr/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
     - /Library/Apple/usr/bin
     - /Users/semaphore/android-sdk/cmdline-tools/latest/bin
     - /Users/semaphore/android-sdk/platform-tools
     - /Users/semaphore/android-sdk/emulator
   ... installing tebako-runtime gem
   ... @ /Users/semaphore/.tebako/o/s/bin/gem install tebako-runtime --no-document --install-dir /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0
   ... collecting gem from gemspec /Users/semaphore/helpling_toolbelt/release/helpling_toolbelt.gemspec and Gemfile
   ... @ /Users/semaphore/.tebako/o/s/bin/bundle config set --local build.ffi --disable-system-libffi
   ... @ /Users/semaphore/.tebako/o/s/bin/bundle config set --local build.nokogiri --no-use-system-libraries
   ... @ /Users/semaphore/.tebako/o/s/bin/bundle config set --local force_ruby_platform false
   *** It may take a long time for a big project. It takes REALLY long time on Windows ***
   ... @ /Users/semaphore/.tebako/o/s/bin/bundle install --jobs=4
   ... @ /Users/semaphore/.tebako/o/s/bin/bundle exec /Users/semaphore/.tebako/o/s/bin/gem build /Users/semaphore/helpling_toolbelt/release/helpling_toolbelt.gemspec
tebako-packager failed: Failed to run /Users/semaphore/.tebako/o/s/bin/bundle exec /Users/semaphore/.tebako/o/s/bin/gem build /Users/semaphore/helpling_toolbelt/release/helpling_toolbelt.gemspec (pid 82857 exit 1):
 Ignoring ffi-1.15.5 because its extensions are not built. Try: gem pristine ffi --version 1.15.5
Ignoring jaro_winkler-1.5.6 because its extensions are not built. Try: gem pristine jaro_winkler --version 1.5.6
Ignoring ffi-1.15.5 because its extensions are not built. Try: gem pristine ffi --version 1.15.5
Ignoring jaro_winkler-1.5.6 because its extensions are not built. Try: gem pristine jaro_winkler --version 1.5.6
/Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/definition.rb:596:in `materialize': Could not find jaro_winkler-1.5.6, ffi-1.15.5 in locally installed gems (Bundler::GemNotFound)
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/definition.rb:203:in `specs'
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/definition.rb:270:in `specs_for'
 
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler.rb:162:in `setup'
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/setup.rb:26:in `block in <top (required)>'
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/ui/shell.rb:159:in `with_level'
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/ui/shell.rb:111:in `silence'
	from /Users/semaphore/.tebako/o/s/lib/ruby/gems/3.2.0/gems/bundler-2.5.7/lib/bundler/setup.rb:26:in `<top (required)>'
	from <internal:/Users/semaphore/.tebako/o/s/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/Users/semaphore/.tebako/o/s/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /Users/semaphore/.tebako/o/s/lib/ruby/3.2.0/rubygems.rb:1370:in `<top (required)>'
	from <internal:gem_prelude>:2:in `require'
	from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
 [255]
make[3]: *** [CMakeFiles/source_filesystem] Error 255
make[2]: *** [CMakeFiles/source_filesystem.dir/all] Error 2
make[1]: *** [CMakeFiles/tebako.dir/rule] Error 2
make: *** [tebako] Error 2
Tebako script failed: 'tebako press' build step failed [104]

Unfortunately I'm on vacation for the next two weeks so I won't be able to debug this and try to narrow down the problem.

[d4rky-pl]

I was trying to fix the issue but unfortunately my knowledge of how bundler version switching works was not enough. The problem happened because when we run bundle install, it uses the current shell Ruby version, not the tebako one so the new bundler is installed in the wrong place. I've tried to add tebako Ruby (.tebako/o/s/bin) to the PATH for the duration of bundle install but it still failed elsewhere and I ran out of time to debug further.

Ultimately I've managed to workaround the issue with:

export BUNDLER_VERSION=$(bundle --version|cut -d' ' -f 3)

This forces bundler to use the current version instead of the one from Gemfile.lock so it's not ideal.

[maxirmx]

1. Tebako takes Ruby version from the command line and
2. Uses bundler version specified in the environment variables or applies some reasonable defaults

(1) is not correct if Ruby version is specified in Gemfile and it is normally specified for RoR projects
(2) is not correct if there is Gemfile.lock and solution was bundled with anpther version of bundler

So we need

• change Tebako::CliRubies to take Ruby version from Gemfile if set
• change Tebako::DeployHelper to take bundler version from Gemfile.lock if any