Repository navigation
Conversation
Single-header test framework (MIT, licence kept in ThirdParty/doctest).
tests/BH.Tests.vcxproj builds a Win32 console exe that compiles BH sources unmodified, plus: - fakes/EnginePtrs.cpp: defines the D2Ptrs.h engine pointers like BH.cpp does, with Patch::GetDllOffset routing the engine functions tests need to fakes and everything else to zeroed memory, and D2Version fixed to 1.13c. - fakes/FakeEngine: a small model of the game state those functions read. - fakes/BHGlobals.cpp: globals owned by BH.cpp/Item.cpp/Module.cpp, which are not compiled. - support/LootFilter: helpers to build items and load a filter file through Config::Parse and ItemDisplay::InitializeItemRules. Game and BH state are reset before every test case. The project is not part of BH.sln.
A second job in build.yml builds tests/BH.Tests.vcxproj (MSVC Release|Win32) and runs it; any failing test fails the run. The doctest summary is in the log and a JUnit report is attached as an artifact. README explains how to run the tests locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds
tests/BH.Tests.vcxproj, a console program that compiles BH's own sources unmodified and runs 505 doctest test cases (6,078 assertions) against them. CI builds and runs it on every pull request, next to the existing BH.dll build, and the run fails if any test fails. No production code changed.The tests cover the loot filter end to end (conditions, formulas, actions, rule lists, aliases, filter levels, caches), the config layer (filter file parsing, BH.json load/save/backup recovery), JSONObject and the Mustache stash export templates, the game list filter, and a set of modules and UI controls (stats panel tables, automap info, party, chat colour, map notifications, Bnet, Keyhook/Checkhook/Combohook).
Along the way the suite found 21 real bugs. Each one has a test asserting the correct behaviour, marked
should_failwith a// BUG:comment, so CI stays green today and turns red once the bug is fixed (then the marker gets removed).Problem
BH has no automated tests. The loot filter alone is around 5,600 lines of parsing and evaluation that every PD2 player depends on, and the only way to check a change today is to load a filter in game and look at items. Regressions in keyword parsing, operator handling or name output are easy to ship and hard to spot.
Changes
ThirdParty/doctest/tests/BH.Tests.vcxprojtests/fakes/EnginePtrs.cpp_DEFINE_PTRS).Patch::GetDllOffsetis replaced: engine functions the tests need resolve to fakes (theROUTEmacro type-checks each fake against the real signature), everything else resolves to zeroed memory. D2Version is fixed to 1.13ctests/fakes/FakeEngine.*tests/fakes/BHGlobals.cpptests/support/LootFilter.*TestItem,Matches,LoadFilter(realConfig::Parse+ItemDisplay::InitializeItemRuleson a temp file),NameOf,DescriptionOftests/*Tests.cpp.github/workflows/build.ymltestsjob: build, run (fails the run on any failure), write and upload a JUnit report. The existing BH.dll job is untouchedREADME.mdAll game and BH state is reset before every test case, so tests are independent and also pass in random order (
--order-by=rand).Why a separate .vcxproj and not CMake
Maintainers build with Visual Studio and MSBuild, and CI already uses
msbuild. A plain .vcxproj next to the code needs nothing new to install and uses the same compiler and settings as BH.dll, so a test that passes is testing the same compilation. The CMakeLists.txt files in the repo are out of date (they list files that no longer exist), so a CMake target would mean maintaining a second build description. Keeping the project out of BH.sln means building BH in Visual Studio is exactly as before; anyone who wants the tests openstests\BH.Tests.vcxprojor runs one msbuild line.Test areas
Expected values come from Diablo II / PD2 rules and data (the PD2 Item Filtering wiki, breakpoint tables, Windows virtual-key codes, the Mustache spec, hand calculation), never from running BH and copying its output.
Bugs found (tests marked
should_fail)GetKeyCodeVK_FORWARDSLASHis 0xBD (same asVK_MINUS) andVK_TILDEis 0xBF (the/key). '/' should beVK_OEM_2, ''' can't be boundVK_OEM_3(0xC0), so these hotkeys fire on the wrong key and 'ParseGetToggle/GetKey/GetArray/GetAssocGetInt/GetBool/GetStringthey don't catchnlohmann::type_error, so a wrongly typed BH.json entry throws out ofLoadConfigFormulastrong) is dropped from the rule, so the rule matches items the formula rejects (wiki: only the leading F must be upper case)$f(-STAT5)without the stat) render as-0MAPTIERMAPTIER<Nmatches every non-map item)silently drops the rest of the rule (ETH ) UNIacts asETH)MULTIstd::stoithrow out ofInitializeItemRules, aborting the whole filter loadBASE*dmg(d + d/2) * 5/6, so odd values are low (1 gives 0, 5 gives 5) instead of x1.25+sums~ranges on sums ignore the upper bound (FRES+CRES~50-100never matches)%MAP%%MAP%picks the wrong minimap colour%TIER-n%%TIER-10%..%TIER-12%are ignored and shown as textJson_Escape\b\f\n\r\tare copied raw, producing invalid JSON0.000000SetYSizerejects the height instead of clamping, so the open panel no longer covers its rowsstd::stoi; a non-numeric colour throws out of the chat packet handlerflags & PARTY_NOT_IN_PARTY, which is 0, so it never fires and the player leaves their party in that windowAlso noticed but not tested (undefined behaviour or no deterministic way to test without changing code):
commaprintreturns a pointer to a stack buffer;string_format("")writes past a zero-length buffer; a self-referencingAliasloops forever inInitializeItemRules; ScreenInfo's exp/s rate wraps to billions after a death (unsigned subtraction);Task'scanceled_is never initialised.Production code seams
None. Everything is done from the test side: the engine pointers are defined by the test project, and the few BH globals and module classes that live in files the tests don't compile (BH.cpp, Item.cpp, Module.cpp, ModuleManager.cpp) get small stand-ins in
tests/fakes/BHGlobals.cpp.Testing
CI proof
tests/unit-suitetest cases: 505 | 505 passed | 0 failed, BH.dll job unchangedtests/gate-checkRun testsstep failed,test cases: 506 | 505 passed | 1 failedCoverage
Measured locally with clang source-based coverage (
-fprofile-instr-generate -fcoverage-mapping, llvm-cov) on a Win32 build of the same test project, for the BH files the tests target. Tooling is not part of this PR.The low numbers are mostly drawing code that needs the real renderer (StatsDisplay's
OnDrawis ~1,500 lines), helpers nothing in BH calls (left untested on purpose, e.g. the geometry helpers in Common.cpp), and in ItemDisplay.cpp the formula variable table, where most of the ~140 variables are not exercised one by one yet (a good follow-up: the wiki says each variable mirrors a condition keyword, which makes a table-driven check easy).Running locally
-ts=LootFilterRulesruns one suite,-tc="*TIER*"matches test names,--order-by=randshuffles.