Merge pull request #53 from hakril/winapi-wide

Full unicode compatibility

.github/workflows/mypytest.yml

@@ -67,4 +67,8 @@ jobs:
         uses: EnricoMi/publish-unit-test-result-action/composite@v1
         if: always()
         with:
-          files: junit/test-results.xml
+          files: junit/test-results.xml
+          check_name: PyTest Results for ${{ matrix.python-version}}-${{ matrix.python-bitness-to-test}}
+          secondary_rate_limit_wait_seconds: 90
+          seconds_between_github_writes: 10
+          seconds_between_github_reads: 1

CHANGELOG

@@ -1,3 +1,13 @@
+1.0.0:
+
+Unicode everywhere:
+	Whenever possible, PythonForWindows use W() (wide) APIs and returns unicode string.
+	This means `str` for python3 & `unicode` for python2.7
+
+windows.system.build_number now returns a int : the actual build number
+windows.system.build_number became windows.system.versionstr
+
+
 Between 0.4 & 0.5:
     == New features ==
         - windows.security

README.md

@@ -1,24 +1,23 @@
 # PythonForWindows
 
-[![Join the chat at https://gitter.im/PythonForWindows/general](https://badges.gitter.im/PythonForWindows/general.svg)](https://gitter.im/PythonForWindows/general)
 [![Pytest](https://github.com/hakril/PythonForWindows/actions/workflows/mypytest.yml/badge.svg?branch=master)](https://github.com/hakril/PythonForWindows/actions/workflows/mypytest.yml)
 
 PythonForWindows (PFW) is a base of code aimed to make interaction with `Windows` (on X86/X64) easier (for both 32 and 64 bits Python).
 Its goal is to offer abstractions around some of the OS features in a (I hope) pythonic way.
 It also tries to make the barrier between python and native execution thinner in both ways.
 There is no external dependencies but it relies heavily on the `ctypes` module.
 
-
-Some of this code is clean (IMHO) and some parts are just a wreck that works for now.
-Let's say that the codebase evolves with my needs and my curiosity.
+Let's say that the codebase evolves with my needs, my researches and my curiosity.
 
 Complete online documentation is available [here][ONLINE_DOC]
 You can find some examples of code in the [samples directory][SAMPLE_DIR] or [online][ONLINE_SAMPLE].
 
-PythonForWindows is principally known for its ALPC-RPC Client (see [samples](http://hakril.github.io/PythonForWindows/build/html/sample.html#windows-rpc)).
+PythonForWindows is principally known for:
+	- its ALPC-RPC Client (see [samples](http://hakril.github.io/PythonForWindows/build/html/sample.html#windows-rpc))
+	- its generated [ctypes definitions](https://github.com/hakril/PythonForWindows/tree/master/windows/generated_def).
 
 
-If you have any issue, question or suggestion do not hesitate to join [the Gitter channel](https://gitter.im/PythonForWindows/general).
+If you have any issue, question or suggestion do not hesitate to create an issue or reach me out.
 I am always glad to have feedbacks from people using this project.
 
 ## Installation
@@ -36,12 +35,17 @@ You can also install PythonForWindows by cloning it and using the ``setup.py`` s
 python setup.py install
 ``
 
-#### Python3
+## Encoding & unicode
+
+PythonForWindows support `python2.7` & `python3` and is currently tested for `Python2.7`, `3.6` & `3.11` via [Github Workflow](https://github.com/hakril/PythonForWindows/actions/workflows/mypytest.yml)
+
+Since 1.0.0, the code uses "wide APIs" whenever possible and accept/returns python3 `str` (py2.7 `unicode` type) almost everywhere. Any functions/APIs not accepting unicode string can be  considered a bug if its not stated explicitly in the documentation.
 
-PythonForWindows support python3 and is currently tested for Python2.7, 3.6 & 3.11 via [Github Workflow](https://github.com/hakril/PythonForWindows/actions/workflows/mypytest.yml)
-Regarding the handling of encoding in the project it's currently a mix of Ascii & Unicode that may be awkward on Python3 as automatic encoding/decoding is not present.
+### Python2
+
+PythonForWindows continues to support python2.7 as its the only way to have it running on `Windows XP` & `Windows Server 2003` which are sadly still seen in production.
+Encoding errors at print time might be awkward for unicode string on python2, see the [PythonForWindows encoding guide](http://hakril.github.io/PythonForWindows/build/html/encoding.html) in the documentation.
 
-The aim of passing the whole project under unicode is actvily ongoing.
 
 ## Overview
 
@@ -531,6 +535,52 @@ The local debugger handles
 * Standard breakpoint ``int3``
 * Hardware Execution breakpoint ``DrX``
 
+### Symbols
+
+Classes around the Symbols APIs of `dbghelp.dll` are also implemented and can be used independently of the Debugger.
+The path of `dbghelp.dll` can also be given via the `PFW_DBGHELP_PATH` environment variable.
+
+
+```python
+# Python3
+
+>>> from windows.debug import symbols
+>>> # symbols.set_dbghelp_path(MY_DBGHELP_PATH)
+>>> symbols.engine.options = 0 # Disable defered load
+>>> sh = symbols.VirtualSymbolHandler()
+>>> ntmod = sh.load_file(r"c:\windows\system32\ntdll.dll", addr=0x420000)
+>>> ntmod
+<SymbolModule name="ntdll" type=SymPdb pdb="ntdll.pdb" addr=0x420000>
+>>> ntmod.name
+'ntdll'
+>>> ntmod.path
+'c:\\windows\\system32\\ntdll.dll'
+>>> ntmod.pdb
+'c:\\Symbols\\ntdll.pdb\\8D5D5ED5D5B8AA609A82600C14E3004D1\\ntdll.pdb'
+>>> sym = sh["ntdll!LdrLoadDll"]
+>>> sym
+<SymbolInfoW name="LdrLoadDll" start=0x44a160 tag=SymTagFunction>
+>>> sym.fullname
+'ntdll!LdrLoadDll'
+>>> hex(sym.addr)
+'0x44a160'
+>>> sh.search("ntdll!*CreateFile")
+[<SymbolInfoW name="EtwpCreateFile" start=0x47d9ec tag=SymTagFunction>, <SymbolInfoW name="EtwpCreateFile" start=0x47d9ec tag=SymTagPublicSymbol>, <SymbolInfoW name="NtCreateFile" start=0x4c03e0 tag=SymTagPublicSymbol>, <SymbolInfoW name="ZwCreateFile" start=0x4c03e0 tag=SymTagPublicSymbol>, <SymbolInfoW name="__imp_NtCreateFile" start=0x55cb70 tag=SymTagPublicSymbol>]
+# Some types exploration
+>>> sh.get_type("ntdll!_PEB")
+<SymbolType name="_PEB" tag=_SymTagEnum.SymTagUDT(0xb)>
+>>> peb = _
+>>> peb = sh.get_type("ntdll!_PEB")
+>>> peb
+<SymbolType name="_PEB" tag=_SymTagEnum.SymTagUDT(0xb)>
+>>> peb.size
+2000
+>>> peb.children[:3]
+[<SymbolType name="InheritedAddressSpace" tag=_SymTagEnum.SymTagData(0x7)>, <SymbolType name="ReadImageFileExecOptions" tag=_SymTagEnum.SymTagData(0x7)>, <SymbolType name="BeingDebugged" tag=_SymTagEnum.SymTagData(0x7)>]
+>>> peb.children[2].offset
+2
+```
+
 ### Other stuff (see doc / samples)
 
 - Network

ctypes_generation/definitions/defines/kuser_shared_data.txt

@@ -0,0 +1,14 @@
+#define MM_SHARED_USER_DATA_VA 0x7FFE0000
+
+#define XSTATE_LEGACY_FLOATING_POINT        (0)
+#define XSTATE_LEGACY_SSE                   (1)
+#define XSTATE_GSSE                         (2)
+#define XSTATE_AVX                          (XSTATE_GSSE)
+#define XSTATE_MPX_BNDREGS                  (3)
+#define XSTATE_MPX_BNDCSR                   (4)
+#define XSTATE_AVX512_KMASK                 (5)
+#define XSTATE_AVX512_ZMM_H                 (6)
+#define XSTATE_AVX512_ZMM                   (7)
+#define XSTATE_IPT                          (8)
+#define XSTATE_LWP                          (62)
+#define MAXIMUM_XSTATE_FEATURES             (64)

ctypes_generation/definitions/defines/services.txt

@@ -38,13 +38,36 @@
 #define SERVICE_USER_OWN_PROCESS       0x00000050
 #define SERVICE_USER_SHARE_PROCESS     0x00000060
 
+
+
 #define SERVICE_WIN32                  (SERVICE_WIN32_OWN_PROCESS | SERVICE_WIN32_SHARE_PROCESS)
 
 #define SERVICE_INTERACTIVE_PROCESS    0x00000100
 
+/*
+Legacy value changed with Win10 build 14942
+https://github.com/processhacker/processhacker/issues/120
+*/
 #define SERVICE_TYPE_ALL               (SERVICE_WIN32 | SERVICE_ADAPTER | SERVICE_DRIVER | SERVICE_INTERACTIVE_PROCESS)
 
 
+/* New service Type from Win10 build 14942 */
+
+#define SERVICE_USER_SERVICE 0x00000040
+#define SERVICE_USERSERVICE_INSTANCE 0x00000080
+#define SERVICE_USER_SHARE_PROCESS (SERVICE_USER_SERVICE |
+SERVICE_WIN32_SHARE_PROCESS)
+#define SERVICE_USER_OWN_PROCESS (SERVICE_USER_SERVICE |
+SERVICE_WIN32_OWN_PROCESS)
+#define SERVICE_PKG_SERVICE 0x00000200
+
+
+/* Make a value for the new SERVICE_TYPE_ALL ? */
+
+
+
+
+
 #define SERVICE_BOOT_START             0x00000000
 #define SERVICE_SYSTEM_START           0x00000001
 #define SERVICE_AUTO_START             0x00000002

ctypes_generation/definitions/defines/windef.txt

@@ -668,4 +668,7 @@
 
 #define TXFS_MINIVERSION_COMMITTED_VIEW                  (0x0000)
 #define TXFS_MINIVERSION_DIRTY_VIEW                      (0xFFFF)
-#define TXFS_MINIVERSION_DEFAULT_VIEW                    (0xFFFE)
+#define TXFS_MINIVERSION_DEFAULT_VIEW                    (0xFFFE)
+
+
+#define PROCESSOR_FEATURE_MAX 64

ctypes_generation/definitions/functions/symbols.txt

@@ -29,6 +29,13 @@ BOOL WINAPI SymFromAddr(
   _Inout_   PSYMBOL_INFO Symbol
 );
 
+BOOL WINAPI SymFromAddrW(
+  [in]            HANDLE        hProcess,
+  [in]            DWORD64       Address,
+  [out, optional] PDWORD64      Displacement,
+  [in, out]       PSYMBOL_INFOW Symbol
+);
+
 
 BOOL WINAPI SymGetModuleInfo64(
   _In_  HANDLE             hProcess,
@@ -48,6 +55,12 @@ BOOL WINAPI SymInitialize(
   _In_     BOOL   fInvadeProcess
 );
 
+BOOL WINAPI SymInitializeW(
+  [in]           HANDLE hProcess,
+  [in, optional] PCWSTR UserSearchPath,
+  [in]           BOOL   fInvadeProcess
+);
+
 BOOL WINAPI SymFromName(
   _In_    HANDLE       hProcess,
   _In_    LPCSTR       Name,
@@ -111,12 +124,26 @@ BOOL WINAPI SymEnumTypesByName(
   _In_ PVOID                          UserContext
 );
 
+BOOL WINAPI SymEnumTypesByNameW(
+  [in]           HANDLE                          hProcess,
+  [in]           ULONG64                         BaseOfDll,
+  [in, optional] PCWSTR                          mask,
+  [in]           PVOID							 EnumSymbolsCallback,
+  [in]           PVOID                           UserContext
+);
+
 BOOL WINAPI SymEnumerateModules64(
   _In_ HANDLE                      hProcess,
   _In_ PVOID                        EnumModulesCallback,
   _In_ PVOID                       UserContext
 );
 
+BOOL WINAPI SymEnumerateModulesW64(
+  [in]           HANDLE                       hProcess,
+  [in]           PSYM_ENUMMODULES_CALLBACKW64 EnumModulesCallback,
+  [in, optional] PVOID                        UserContext
+);
+
 BOOL SymNext(
   HANDLE       hProcess,
   PSYMBOL_INFO si
@@ -266,6 +293,13 @@ BOOL WINAPI SymGetTypeFromName(
   _Inout_ PSYMBOL_INFO Symbol
 );
 
+BOOL WINAPI SymGetTypeFromNameW(
+  [in]      HANDLE        hProcess,
+  [in]      ULONG64       BaseOfDll,
+  [in]      PCWSTR        Name,
+  [in, out] PSYMBOL_INFOW Symbol
+);
+
 
 BOOL WINAPI SymGetTypeInfo(
   _In_  HANDLE                    hProcess,

ctypes_generation/definitions/functions/syscall.txt

@@ -182,6 +182,16 @@ NTSTATUS WINAPI NtQuerySystemInformation(
 );
 
 
+NTSTATUS WINAPI NtQuerySystemInformationEx(
+    _In_ SYSTEM_INFORMATION_CLASS SystemInformationClass,
+    _In_reads_bytes_(InputBufferLength) PVOID InputBuffer,
+    _In_ ULONG InputBufferLength,
+    _Out_writes_bytes_opt_(SystemInformationLength) PVOID SystemInformation,
+    _In_ ULONG SystemInformationLength,
+    _Out_opt_ PULONG ReturnLength
+);
+
+
 NTSTATUS WINAPI NtQueryInformationProcess(
   _In_      HANDLE           ProcessHandle,
   _In_      PROCESSINFOCLASS ProcessInformationClass,

ctypes_generation/definitions/functions/winfunc.txt

@@ -450,6 +450,9 @@ BOOL AllocConsole();
 
 BOOL FreeConsole();
 
+UINT WINAPI GetConsoleOutputCP();
+UINT WINAPI GetConsoleCP();
+
 HANDLE WINAPI GetStdHandle(
   _In_  DWORD nStdHandle
 );

ctypes_generation/definitions/simple_types.txt

@@ -240,6 +240,7 @@ PTRANSLATE_ADDRESS_ROUTINE64  = PVOID # StackWalk
 PSYMBOL_REGISTERED_CALLBACK64 = PVOID # Symbols
 PSYMBOL_REGISTERED_CALLBACK = PVOID # Symbols
 PSYM_ENUMPROCESSES_CALLBACK = PVOID # Symbols
+PSYM_ENUMMODULES_CALLBACKW64 = PVOID # Symbols
 ENUMRESNAMEPROCA = PVOID # Resources
 ENUMRESNAMEPROCW = PVOID # Resources
 ENUMRESTYPEPROCA = PVOID # Resources