World load refactoring

.gitignore

@@ -21,3 +21,22 @@ cmake-build-release-wsl/
 log/
 syslog
 cmake-build-*
+build/
+build_*/
+test/
+magic.mgc
+
+# Секретные данные - большой мир (только малый мир допустим)
+lib.template/world/mobs/*.yaml
+lib.template/world/objects/*.yaml
+lib.template/world/triggers/*.yaml
+lib.template/world/zones/*/
+!lib.template/world/zones/1/
+!lib.template/world/zones/2/
+!lib.template/world/zones/2000/
+!lib.template/world/zones/3/
+!lib.template/world/zones/40/
+
+# Test data generated by CMake
+data/
+misc/

CLAUDE.md

@@ -38,7 +38,7 @@ make tests -j$(nproc)
 
 ### Build Types
 - **Release** - Optimized production build (-O0 with debug symbols, -rdynamic, -Wall)
-- **Debug** - Debug build with ASAN (-fsanitize=address, -D_GLIBCXX_DEBUG)
+- **Debug** - Debug build with ASAN (-fsanitize=address, NO _GLIBCXX_DEBUG due to yaml-cpp ABI compatibility)
 - **Test** - Test build with optimizations (-O3, -DTEST_BUILD, -DNOCRYPT)
 - **FastTest** - Fast test build (-Ofast, -DTEST_BUILD)
 
@@ -49,6 +49,27 @@ docker run -d -p 4000:4000 -e MUD_PORT=4000 -v ./lib:/mud/lib --name mud mud-ser
 docker stop mud
 ```
 
+### Running the Server
+
+**CRITICAL**: Circle must ALWAYS be run from the build directory, NEVER from the source directory.
+
+**Correct usage**:
+```bash
+cd build_yaml        # Or build_legacy, build_sqlite, etc.
+./circle [-W] -d <world_directory> <port>
+```
+
+**Parameters**:
+- `-W` - Enable world checksum calculation (optional)
+- `-d <world_directory>` - Specify world data directory (e.g., `small`, `full`)
+- `<port>` - Port number to listen on (e.g., `4000`)
+
+**Example**:
+```bash
+cd build_yaml
+./circle -W -d small 4000
+```
+
 ### Running Tests
 ```bash
 # Run all tests
@@ -301,3 +322,128 @@ The heartbeat system includes built-in profiling:
 - **Pulse Timing**: Never use wall-clock delays; register actions with heartbeat system
 - **Script Depth**: DG Scripts limited to 512 recursion depth to prevent stack overflow
 - **Runtime Encoding**: While source files are KOI8-R, runtime text can be multiple encodings (Alt, Win, UTF-8, KOI8-R) based on client settings
+
+## Claude Code Workflow Rules
+
+### Build Directory Convention
+Use separate build directories for different CMake configurations to avoid lengthy rebuilds:
+```
+build/        - default build (without optional features)
+build_sqlite/ - build with -DHAVE_SQLITE=ON
+build_debug/  - debug build with -DCMAKE_BUILD_TYPE=Debug
+build_test/   - test data and converted worlds (not for compilation)
+```
+**Always warn the user when changing build directories or running cmake/make in a different directory.**
+
+### File Encoding - CRITICAL
+The Edit tool corrupts KOI8-R encoding in source files. To safely modify existing source files, use **unified diff patches**:
+
+```bash
+# Create and apply a unified diff patch
+cat > /tmp/fix.patch << 'PATCH'
+--- a/src/file.cpp
++++ b/src/file.cpp
+@@ -10,3 +10,4 @@
+ existing line
+-old line
++new line
++added line
+PATCH
+patch -p1 < /tmp/fix.patch
+```
+
+**NEVER use the Edit tool on existing .cpp/.h files that contain Russian text.**
+Only use Edit for newly created files or files known to be pure ASCII.
+
+**NEVER use sed for editing source files.** Sed has tendency to:
+- Modify files in unexpected places (matching wrong lines)
+- Lead to file corruption detection and accidental `git checkout` (losing all uncommitted work)
+- Cause cumulative errors from multiple sed operations
+
+Example of sed problem:
+```
+Problem: GetConfiguredThreadCount() defined 11 times due to multiple sed insertions.
+...
+File corrupted from multiple sed operations. Rolling back to last working version:
+...
+git checkout src/engine/db/yaml_world_data_source.cpp
+...
+😔 Critical error - accidentally reverted ALL changes with git checkout, losing all
+   parallel loading code we implemented.
+```
+
+**Use unified diff patches instead** - they are more reliable, preserve encoding, and fail cleanly if the context doesn't match.
+
+### _GLIBCXX_DEBUG Disabled in Debug Build
+**Important:** `_GLIBCXX_DEBUG` is intentionally **disabled** for Debug builds due to ABI incompatibility with external libraries (yaml-cpp, SQLite).
+
+**Why:**
+- External libraries (yaml-cpp) are compiled without `_GLIBCXX_DEBUG`
+- They return STL objects (`std::string`) to our code
+- If our code uses `_GLIBCXX_DEBUG`, ABI mismatch causes heap-buffer-overflow
+- Solution: disable the flag for all Debug builds
+
+**Trade-off:** We lose STL iterator/bounds checking in Debug mode, but gain ASAN (AddressSanitizer) which catches most memory errors.
+
+### Directory Change Notifications
+Always explicitly notify the user before:
+- Running cmake in a different build directory
+- Running make in a different build directory  
+- Changing the working directory for any build operation
+
+Example: "Switching to build_sqlite/ directory for SQLite-enabled build."
+
+### World Data Formats and Testing
+
+The project supports three world data formats:
+1. **Legacy** - Original CircleMUD text format (default, in lib/ + lib.template/)
+2. **SQLite** - World data in SQLite database (requires -DHAVE_SQLITE=ON)
+3. **YAML** - Human-readable YAML format (requires -DHAVE_YAML=ON)
+
+**CRITICAL: Never use lib/ from repository directly!**
+- `lib/` contains base configuration files only (NOT complete world data)
+- `lib.template/` contains world files, player data, and additional configs
+- To get a working world: copy lib/ to build directory, then overlay lib.template/
+
+**Preparing world for conversion:**
+```bash
+# Create working copy (example for YAML build)
+mkdir -p build_yaml/small
+cp -r lib build_yaml/small/
+cp -r lib.template/* build_yaml/small/lib/
+
+# Now build_yaml/small/lib contains complete world data ready for conversion
+```
+
+**Conversion Tool:**
+```bash
+# Convert legacy world to YAML (in-place conversion)
+./tools/convert_to_yaml.py --input build_yaml/small/lib/world --output build_yaml/small/world --format yaml --type all
+
+# Convert to SQLite database
+./tools/convert_to_yaml.py --input build_sqlite/small/lib/world --output build_sqlite/small/world.db --format sqlite --type all
+```
+
+**Automated Testing & Conversion:**
+```bash
+# Run world loading tests (automatically prepares and converts worlds)
+./tools/run_load_tests.sh              # Full test suite
+./tools/run_load_tests.sh --quick      # Quick test (Legacy + YAML checksums)
+./tools/run_load_tests.sh --help       # Show all options
+```
+
+The `run_load_tests.sh` script:
+- Builds all three variants (Legacy, SQLite, YAML) in separate build directories
+- Automatically prepares working worlds (copies lib + lib.template)
+- Converts worlds if missing or outdated
+- Runs boot tests with configurable timeout (default 5 minutes)
+- Calculates checksums (zones, rooms, mobs, objects, triggers) to verify correctness
+- Compares checksums between formats to detect discrepancies
+- Generates detailed reports with boot times and performance comparison
+
+**Important:**
+- Schema/format changes should be tested with `run_load_tests.sh`
+- Conversion script is in `tools/convert_to_yaml.py`
+- String enum values (like "kWorm") in YAML/SQLite are intentional for human readability - map them in loader
+- When fixing loader issues, check if the problem is in converter or loader
+

CMakeLists.txt

@@ -57,6 +57,7 @@ set(SOURCES
 		src/engine/scripting/dg_comm.cpp
 		src/engine/scripting/dg_domination_helper.cpp
 		src/engine/scripting/dg_db_scripts.cpp
+		src/engine/scripting/trigger_indenter.cpp
 		src/engine/scripting/dg_event.cpp
 		src/engine/scripting/dg_handler.cpp
 		src/engine/scripting/dg_misc.cpp
@@ -151,6 +152,7 @@ set(SOURCES
 		src/engine/structs/flags.hpp
 		src/utils/id_converter.cpp
 		src/utils/utils_time.cpp
+		src/utils/thread_pool.cpp
 		src/gameplay/mechanics/title.cpp
 		src/gameplay/statistics/top.cpp
 		src/utils/utils.cpp
@@ -160,6 +162,22 @@ set(SOURCES
 		src/utils/utils_string.cpp
 		src/engine/db/world_objects.cpp
 		src/engine/db/obj_prototypes.cpp
+		src/engine/db/world_checksum.cpp
+		src/engine/db/world_data_source.h
+		src/engine/db/world_data_source_manager.h
+		src/engine/db/world_data_source_manager.cpp
+		src/engine/db/null_world_data_source.h
+		src/engine/db/legacy_world_data_source.h
+		src/engine/db/legacy_world_data_source.cpp
+		src/engine/db/world_data_source_base.h
+		src/engine/db/world_data_source_base.cpp
+		$<$<BOOL:${HAVE_SQLITE}>:src/engine/db/sqlite_world_data_source.h>
+		$<$<BOOL:${HAVE_SQLITE}>:src/engine/db/sqlite_world_data_source.cpp>
+		$<$<BOOL:${HAVE_YAML}>:src/engine/db/yaml_world_data_source.h>
+		$<$<BOOL:${HAVE_YAML}>:src/engine/db/yaml_world_data_source.cpp>
+		$<$<BOOL:${HAVE_YAML}>:src/engine/db/dictionary_loader.h>
+		$<$<BOOL:${HAVE_YAML}>:src/engine/db/dictionary_loader.cpp>
+		# SQLite world data source (conditional)
 		src/engine/db/id.cpp
 		src/engine/db/utils_find_obj_id_by_vnum.cpp
 		src/engine/db/global_objects.cpp
@@ -503,6 +521,7 @@ set(HEADERS
 		src/engine/structs/radix_trie.h
 		src/engine/db/obj_prototypes.h
 		src/engine/db/world_objects.h
+		src/engine/db/world_checksum.h
 		src/utils/utils_string.h
 		src/gameplay/affects/affect_handler.h
 		src/gameplay/economics/auction.h
@@ -543,6 +562,7 @@ set(HEADERS
 		src/gameplay/mechanics/depot.h
 		src/engine/db/description.h
 		src/engine/scripting/dg_db_scripts.h
+		src/engine/scripting/trigger_indenter.h
 		src/engine/scripting/dg_domination_helper.h
 		src/engine/scripting/dg_event.h
 		src/engine/scripting/dg_olc.h
@@ -1132,6 +1152,8 @@ else ()
 	set(DEFAULT_HAVE_ZLIB "YES")
 endif ()
 option(HAVE_ZLIB "Should ZLib be compiled in. It is required to support MCCP." ${DEFAULT_HAVE_ZLIB})
+option(HAVE_SQLITE "Enable SQLite world data source for faster world loading." OFF)
+option(HAVE_YAML "Enable YAML world data source for human-readable world files." OFF)
 
 if (HAVE_ZLIB)
 	set(ZLIB_ROOT $ENV{ZLIB_ROOT})
@@ -1144,6 +1166,31 @@ else ()
 	message(STATUS "ZLib is turned off. Circle will NOT support MCCP.")
 endif ()
 
+# SQLite support
+if (HAVE_SQLITE)
+	find_package(SQLite3 REQUIRED)
+	add_definitions(-DHAVE_SQLITE)
+
+	# Create separate library for SQLite loader (without _GLIBCXX_DEBUG)
+	include_directories(${SQLite3_INCLUDE_DIRS})
+	target_link_libraries(circle.library SQLite::SQLite3)
+	message(STATUS "SQLite is turned ON. Circle will support SQLite world data source.")
+else ()
+	message(STATUS "SQLite is turned off.")
+endif ()
+
+# YAML support
+if (HAVE_YAML)
+	find_package(yaml-cpp REQUIRED)
+	add_definitions(-DHAVE_YAML)
+
+	include_directories(${YAML_CPP_INCLUDE_DIRS})
+	target_link_libraries(circle.library yaml-cpp)
+	message(STATUS "YAML is turned ON. Circle will support YAML world data source.")
+else ()
+	message(STATUS "YAML is turned off.")
+endif ()
+
 # Iconv
 option(HAVE_ICONV "Allows to enable search of iconv." OFF)
 if (HAVE_ICONV)
@@ -1207,7 +1254,7 @@ set(TESTBUILD_DEFINITIONS "-DNOCRYPT -DTEST_BUILD")
 set(ASAN_FLAGS)
 if (UNIX AND NOT CYGWIN)
 	set(DEFAULT_WITH_ASAN YES)
-elseif ()
+else ()
 	set(DEFAULT_WITH_ASAN NO)
 endif ()
 option(WITH_ASAN "Compile with ASAN" ${DEFAULT_WITH_ASAN})
@@ -1234,7 +1281,7 @@ endif ()
 if (CMAKE_HOST_UNIX)
 	if (CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang")
 		set(CMAKE_CXX_FLAGS_RELEASE  "-ggdb3 -O0 -rdynamic -Wall -Wextra -Wno-format-truncation ${DEBUG_CRYPT}")
-		set(CMAKE_CXX_FLAGS_DEBUG    "-ggdb3 -O0 -rdynamic -Wall -Wextra -Wno-format-truncation -D_GLIBCXX_DEBUG -D_GLIBXX_DEBUG_PEDANTIC ${ASAN_FLAGS} ${DEBUG_CRYPT}")
+		set(CMAKE_CXX_FLAGS_DEBUG    "-ggdb3 -O0 -rdynamic -Wall -Wextra -Wno-format-truncation ${ASAN_FLAGS} ${DEBUG_CRYPT}")
 		set(CMAKE_CXX_FLAGS_TEST     "-O3 -rdynamic -Wall -Wextra -Wno-format-truncation ${TESTBUILD_DEFINITIONS} -DLOG_AUTOFLUSH")
 		set(CMAKE_CXX_FLAGS_FASTTEST "-Ofast -Wall -Wextra -Wno-format-truncation ${TESTBUILD_DEFINITIONS}")
 
@@ -1307,3 +1354,55 @@ endif ()
 
 # vim: set ts=4 sw=4 ai tw=0 noet syntax=cmake :
 
+# =============================================================================
+# Data directories setup for running server from build directory
+# =============================================================================
+
+option(FULL_WORLD_PATH "Path to full world data directory (e.g., /path/to/full.world/lib)" "")
+
+# Only set up data directories if building out-of-source
+if (NOT "${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_BINARY_DIR}")
+    # Setup small world directory (lib + lib.template)
+    set(SMALL_WORLD_DIR "${CMAKE_BINARY_DIR}/small")
+
+    # Create small world directory by copying lib and overlaying lib.template
+    # First copy lib (base configuration)
+    execute_process(COMMAND ${CMAKE_COMMAND} -E remove_directory "${SMALL_WORLD_DIR}")
+    execute_process(COMMAND cp -r "${CMAKE_SOURCE_DIR}/lib" "${SMALL_WORLD_DIR}"
+        RESULT_VARIABLE copy_result)
+    if (NOT copy_result EQUAL 0)
+        message(FATAL_ERROR "Failed to copy lib directory")
+    endif()
+
+    # Overlay lib.template (don't overwrite existing files)
+    execute_process(COMMAND cp --update=none -r "${CMAKE_SOURCE_DIR}/lib.template/." "${SMALL_WORLD_DIR}/"
+        RESULT_VARIABLE copy_result)
+    if (NOT copy_result EQUAL 0)
+        message(FATAL_ERROR "Failed to overlay lib.template directory")
+    endif()
+    message(STATUS "Copied lib and overlaid lib.template to ${SMALL_WORLD_DIR}")
+
+    message(STATUS "Small world directory configured at: ${SMALL_WORLD_DIR}")
+
+    # Create lib symlink for running server from build directory
+    set(LIB_LINK "${CMAKE_BINARY_DIR}/lib")
+    if (NOT EXISTS "${LIB_LINK}")
+        execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink "${CMAKE_SOURCE_DIR}/lib.template" "${LIB_LINK}")
+        message(STATUS "Created lib symlink: ${LIB_LINK} -> ${CMAKE_SOURCE_DIR}/lib.template")
+    endif()
+
+    # Setup full world directory if path is specified
+    if (FULL_WORLD_PATH)
+        set(FULL_WORLD_DIR "${CMAKE_BINARY_DIR}/full")
+        if (EXISTS "${FULL_WORLD_PATH}" AND NOT EXISTS "${FULL_WORLD_DIR}")
+            execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink "${FULL_WORLD_PATH}" "${FULL_WORLD_DIR}")
+            message(STATUS "Full world directory configured at: ${FULL_WORLD_DIR} -> ${FULL_WORLD_PATH}")
+        elseif (NOT EXISTS "${FULL_WORLD_PATH}")
+            message(WARNING "FULL_WORLD_PATH specified but does not exist: ${FULL_WORLD_PATH}")
+        endif()
+    endif()
+else()
+    message(STATUS "In-source build detected, skipping data directory setup")
+endif()
+
+# vim: set ts=4 sw=4 ai tw=0 noet syntax=cmake :