Error using on Windows

[apwojcik]

Recently stgit stopped working on Windows for me.

I can initialize the branch with metadata. Then, I can convert existing commits to patches. I can pop them one by one or all at once and push them back. After that, I can convert them to commits, too. Finally, I can create a new patch. stg undo works, too. That's it.

I cannot anymore stg refresh, change the order of patches, push anything when I delete a patch in the middle of the stack, etc. The error I am receiving is most of the time, for example: error: 'git update-index': fatal: not a git repository: '\\?\C:\Repo\.git'.

Stacked Git 2.1.0 (a5e19a6)
git version 2.37.3.windows.1

The Python and early Rust versions were working just fine on Windows. The issues probably started as soon as release 2.0.0 was published.

[jpgrayson]

Thanks for this issue report @apwojcik and thanks for trying to use StGit on Windows. That's one of us so far!

While I've tried to avoid any obvious incompatibilities with Windows and ensure that StGit builds for Windows, I've never even tried running it. I don't have or use Windows, so figuring out and fixing Windows-specific issues will require help from others.

To clarify, you are attempting to run a stg.exe built natively for Windows and not running stg from a WSL environment, right?

I also note that you are using the head of the master branch. It would be very helpful to know whether the same issue is present in v2.1.0 and/or v2.0.0.

My strong suspicion about the root of this are the Path::canonicalize() calls in StupidContext::git_in_work_root(). Those were introduced with the change to using gitoxide. That method converts the potentially relative git repo, working tree, and index paths into absolute paths using canonicalize() in order to account for the forthcoming git command being run from the working tree root. Seems like some alternative path manipulation is needed to avoid using canonicalize() which apparently produces paths that are not compatible with git.

[apwojcik]

That's correct. I am using stg.exe, natively built for Windows. Thanks for the hints. I will look at the code and figure out what is wrong. Then I will submit a PR.

[apwojcik]

You are right - the Path::canonicalize() calls cause the problem. Switching to std::fs::canonicalize() results in the same. There is std::path::absolute(), but absolute_path is unstable [92750] and requires a nightly build of rustc. So to fix my problem, I removed Path::canonicalize() locally for now. I will keep looking for a better solution.

[apwojcik]

The issue described above does not occur in the release version 2.1.0.

I found another issue with Windows (version 2.1.0 and newer).
Stack of patches: patch1 applied and patch2 unapplied. The stack pointer is on the patch1. So I am trying to execute stg push to apply patch2 on top of patch1 - the merge needs to be performed:

error: 'git read-tree': fatal: unable to write new index file.

The git exit code is 128.

Stacked Git 2.1.0
git version 2.39.2.windows.1

When there are no conflicts between patches, everything works fine.

[jpgrayson]

Thanks for the follow-ups on this issue, @apwojcik. It's now somewhere on my todo list to see if there is a way to do the needed path munging without using canonicalize(). I'm happy if you can find a solution first.

Note that I strongly prefer approaches that do not grow StGit's dependency tree. Or use any unstable features.

[bonsairobo]

I'd like to get stgit working on Windows; is there some way I can help? I've recommended it to my coworkers for helping to maintain well-organized PRs, but many of them use Windows, and I don't know of an alternative workflow for them.

[jpgrayson]

Hi @bonsairobo. I appreciate your interest in getting StGit working on Windows and I'm glad to hear it is helpful for your workflow. Happy to have help with getting StGit working on Windows.

The main problem with StGit on Windows is that I don't know what the problems are...because I don't use Windows and StGit's test suite is not running on Windows.

I've just pushed a fix for the particular problem that this issue identifies. I was able to reproduce the problem and verify the fix using a wine environment. I briefly attempted to run the entire StGit test suite using wine. It was a total hack job, but sufficient to run the test suite. This informed me as follows:

1. There are still at least several problems that prevent StGit from passing tests on Windows.
2. With some TLC, the test suite could perhaps have first-class support for running in a wine env.

However, it remains a big mystery to me as to whether this wine environment is truly representative of real world Windows environments that StGit may be run from. Would stg.exe be run from the bash environment that git ships? Or from a cmd.exe window? Or using powershell? I imagine all three are meant to be possible. Are there differences that StGit needs to comprehend?

Anyway, the most helpful thing for getting StGit working on Windows would be to enable the test suite to run natively on Windows. If we had that, then bugs could be identified and fixed and it could be incorporated into CI to keep Windows support from regressing as StGit evolves.

StGit's test suite is borrowed from Git's test suite. I.e. we use the same makefile structure, helper shell scripts, and test script structure. Ostensibly Git's test suite runs on Windows. So a good exercise would be to clone Git's git repo and attempt to build and test it in a Windows environment. I may have pruned Windows-specific bits from t/Makefile or the helper scripts that need to be restored.

If that sounds like too big of a chunk, I'm also happy to get one-off issue reports for anything not working on Windows.

[bonsairobo]

@jpgrayson Thanks for the detailed response. I agree that ideally the stgit test suite would be running on native Windows. WINE is a great start though! It looks like there is at least a Windows build CI action that's passing.

Would stg.exe be run from the bash environment that git ships? Or from a cmd.exe window? Or using powershell? I imagine all three are meant to be possible. Are there differences that StGit needs to comprehend?

It seems like the easiest option would be to use Git Bash, which claims to behave identically to bash on *NIX:

Git for Windows provides a BASH emulation used to run Git from the command line. *NIX users should feel right at home, as the BASH emulation behaves just like the "git" command in LINUX and UNIX environments.

So a good exercise would be to clone Git's git repo and attempt to build and test it in a Windows environment.

I can give this a try. I'll try running the test suite using Git Bash.

I may have pruned Windows-specific bits from t/Makefile or the helper scripts that need to be restored.

What's the easiest way to recover these bits? Should I just diff your t/ directory with Git's test suite? I imagine they might have diverged.

[bonsairobo]

I think the next step after Git Bash would be Powershell. That's what most of my coworkers use. I know very little about it though. Hopefully stgit is using enough cross-platform process APIs that there isn't much to fix.

[bonsairobo]

Well I've come to realize that building git inside of Git Bash for Windows would require that I install gcc somehow. I don't really want to do this, and it's starting to seem out of scope for testing STGit per se.

@jpgrayson What's the reason for using Git's test Makefiles? Is it not feasible to get your test coverage by writing Rust integration tests?

[jpgrayson]

What's the reason for using Git's test Makefiles? Is it not feasible to get your test coverage by writing Rust integration tests?

StGit's test suite predates the Rust implementation of StGit. The original implementation was in Python. So one answer is that the makefile/shell-script approach is language agnostic, which is good.

Another reason is that there are almost 1200 tests defined across 118 test files. We would need a pretty big incentive to migrate that mass of tests to a different test infrastructure.

For these kind of integration/regression tests (i.e. not unit tests), a key aspect of the test is to demonstrate StGit's behaviors in the target environment, so running stg (or git) from a shell allows that test environment to be tuned and well controlled.

Rust's test infra could be made to do the same things as the current test infrastructure, but it would be a big effort and I'm not sure that it would directly solve the problems with testing on Windows.

---

Following your link from above to https://gitforwindows.org/ revealed that there is Git for Windows SDK that apparently provides the MSYS2-based environment needed to build Git and run its test suite. I may have a go of trying to install and use that Git for Windows SDK from within Wine and see if it can work as an environment for running StGit's test suite.

I appreciate the steps you've taken thus far, @bonsairobo.

[bonsairobo]

So one answer is that the makefile/shell-script approach is language agnostic, which is good.

True, but it is not platform agnostic, hence the current predicament with Windows.

But I totally get not wanting to rewrite this massive test suite. I wouldn't suggest doing that, so if you can find a way to get these tests running in something resembling a Windows environment with minimal effort, it sounds worthwhile.

almost 1200 tests defined across 118 test files

Are these tests copied from Git? I.e. these tests are just showing that stg passthrough to git works when the commands need to behave identically?

If so, maybe it's not strictly necessary to port all of these tests to native Windows. I don't know the details of this passthrough logic, but maybe it would be easier to write some new targeted tests of that logic that can run on native Windows.

Following your link from above to https://gitforwindows.org/ revealed that there is Git for Windows SDK that apparently provides the MSYS2-based environment needed to build Git and run its test suite.

Cool. That would have been my next step if I wanted to get the unix build tools working in Windows. Wasn't too keen on going through that process last night though. Let me know how it goes!

[jpgrayson]

I have pushed a bunch of fixes for various issues affecting StGit on Windows. As of 342e5ab, the Windows build of StGit is passing the test suite and included in the CI suite.

Here's an enumeration of some of the issues that prevented StGit from working on Windows:

• Using Path::canonicalize() to get absolute paths yields paths that Git for Windows could not consume. Solved by using gix::path::realpath().
• gix-odb would fail when attempting to write an object to the repo that already existed. This happens frequently with empty tree and empty blob objects (among others). This was fixed upstream in gix 0.44.
• Running git read-tree and git write-tree from a work tree subdir seems to cause problems on Windows that do not appear on other platforms. Running these particular tools from the work tree root resolved these problems.
• Clap used the "stg.exe" binary name in its auto-generated usage lines in help output. We override that to "stg" for consistency with custom usage lines as well as consistency across platforms.
• If the StGit repository is checked out on Windows with core.autocrlf=true such that text files have CRLF line endings, the test suite breaks. Adding a rule to the toplevel .gitattributes file to force LF line endings helps ensure that StGit repos are checked out in a viable fashion on Windows.
• Hooks need to be run wrapped with sh on Windows. Trying to run hook scripts directly on Windows does not work.
• Running git interpret-trailers does not always work correctly. This issue is only partially mitigated. As things currently stand, StGit works with Git for Windows, but does not always work with the git executable distributed with MSYS2.
• Several test cases needed to be tweaked or made conditional on MINGW to accommodate the Windows/MSYS/MINGW/UCRT environment.

Anyway, StGit now seems to work well enough on Windows with the Git for Windows distribution. Closing.