[export] minimal numpy support #429

evaleev · 2025-11-21T13:39:00Z

No description provided.

….g. 2 * ex<Constant>(2)

Copilot

Pull Request Overview

This PR adds minimal NumPy and PyTorch export support to SeQuant through a new PythonEinsumGenerator that generates Python code using einsum notation for tensor contractions. The implementation includes comprehensive validation tests that execute generated Python code and verify correctness by comparing results computed in C++ (Eigen) with Python (NumPy).

Key changes:

New PythonEinsumGenerator for generating Python/NumPy/PyTorch code with einsum operations
Custom NumPy I/O utilities for reading/writing Eigen tensors in .npy format with proper layout handling
Refactored test context setup by extracting to_export_context() to a shared header file
Enhanced expression operator overloads to support scalar multiplication with ExprPtr

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
SeQuant/core/export/python_einsum.hpp	Implements PythonEinsumGenerator and PythonEinsumGeneratorContext for generating Python einsum code with NumPy/PyTorch backend support
tests/unit/test_export_python.cpp	Comprehensive validation tests including NumPy I/O utilities and end-to-end Python execution tests
tests/unit/test_export.hpp	Shared test context setup extracted from test_export.cpp for reuse across test files
tests/unit/test_export.cpp	Updated to use shared context from test_export.hpp and added PythonEinsumGenerator to test suite
tests/unit/CMakeLists.txt	Added Python/NumPy/PyTorch detection logic with conditional compilation for validation tests
SeQuant/core/expressions/constant.hpp	Improved template constraints for Constant constructor to prevent ambiguous conversions
SeQuant/core/expressions/expr_operators.hpp	Added operator overloads for scalar * ExprPtr and ExprPtr * scalar
tests/unit/export_tests/binary_contract.export_test	Added expected Python einsum output for binary contraction test case
.github/workflows/cmake.yml	Updated CI to install numpy package on macOS and Ubuntu for validation tests

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/unit/test_export.hpp

tests/unit/test_export_python.cpp

SeQuant/core/export/python_einsum.hpp

.github/workflows/cmake.yml

…nger

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

SeQuant/core/utility/scope.hpp

tests/unit/test_export_python.cpp

SeQuant/core/export/python_einsum.hpp

Copilot · 2025-11-21T18:14:47Z

@evaleev I've opened a new pull request, #430, to work on those changes. Once the pull request is ready, I'll request review from you.

…nger

Co-authored-by: Copilot <[email protected]>

Implement proper POSIX shell escaping for script path using single quotes. This prevents command injection when the path contains special shell metacharacters like backticks, dollar signs, or spaces. Co-authored-by: evaleev <[email protected]>

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

SeQuant/core/utility/scope.hpp

SeQuant/core/export/python_einsum.hpp

SeQuant/core/utility/debug.hpp

tests/unit/test_export.hpp

SeQuant/core/utility/debug.hpp

Krzmbrzl

More thorough review will come in the coming days

tests/unit/export_tests/binary_contract.export_test

Krzmbrzl

The test cases need to be extended to cover a more diverse set of exports. At the very least, I believe we need a ternary contraction, a sum of expressions and an example using a named section (function).

SeQuant/core/export/python_einsum.hpp

Krzmbrzl · 2025-11-27T09:12:46Z

SeQuant/core/export/python_einsum.hpp

+  bool requires_named_sections() const override { return false; }
+
+  DeclarationScope index_declaration_scope() const override {
+    return DeclarationScope::Global;


I believe Section would be more appropriate. Index declarations can be (ab)used to add parameters representing index dimensions to function (aka: "named section") signatures. Otherwise, the variables holding the index dimensions would have to be global.

In python all declare() methods are no-ops ... don't have a good feel for this in absence of use cases. Julia seems to use Global ... please advise

If you want to make the generated code rely on global variables (as it currently does), whatever this function returns is insignificant. My suggestion was to make use of index declarations in order to generate functions that take the index dimensions as parameters:

def bla(n_occ, n_virt): pass

for which we would need DeclarationScope::Section

SeQuant/core/export/python_einsum.hpp

…ther backends

…merics

…ce code and numerics

evaleev · 2025-11-28T14:28:17Z

The test cases need to be extended to cover a more diverse set of exports. At the very least, I believe we need a ternary contraction, a sum of expressions and an example using a named section (function).

added the cited tests via 142bb86 663b35c dc3cf69 ... ideally need to validate everything that ITF exporter. Unfortunately we only have codegen tests for non-Python backends, so Python needs to have the most extensive testset, with matching numerical tests

evaleev · 2025-11-28T14:29:21Z

@Krzmbrzl is the person who added Julia codegen still around? Would it be possible to implement support for numerical testing of Julia code along the lines of test_export_python.cpp?

Krzmbrzl · 2025-11-28T15:03:15Z

we only have codegen tests for non-Python backends, so Python needs to have the most extensive testset, with matching numerical tests

I think we could get a long way by having Python do integration tests in addition to unit tests by e.g. generating a CCSD residual code and then evaluating that for some non-zero amplitudes and then comparing to a pre-computed (and validated) result. That should contain most of the individual elements that the unit tests for current exporters test.

is the person who added Julia codegen still around? Would it be possible to implement support for numerical testing of Julia code along the lines of test_export_python.cpp?

Unfortunately, no. He was only there for a 3 month summer internship. If I find the time, I might be able to add (some of) that myself but there are a couple other things on my plate rn that have a higher priority for me 👀

evaleev · 2025-11-28T15:45:46Z

I think we could get a long way by having Python do integration tests in addition to unit tests by e.g. generating a CCSD residual code and then evaluating that for some non-zero amplitudes and then comparing to a pre-computed (and validated) result. That should contain most of the individual elements that the unit tests for current exporters test.

For the export paper (or whatever paper includes it) might be useful. I'll focus on unit tests first.

Unfortunately, no. He was only there for a 3 month summer internship. If I find the time, I might be able to add (some of) that myself but there are a couple other things on my plate rn that have a higher priority for me 👀

understood. AI bots are pretty good at things like this, especially since we already have the template for python (with Eigen::Tensor as the source of truth). It's just matter of getting CI configured to pull Julia, have CMake harness detect it, and then generating the numerical tests themselves.

…of the julia counterpart (base -> {numpy,pytorch} variants}

…nning of the export

Krzmbrzl · 2025-11-29T08:52:16Z

SeQuant/core/export/python_einsum.hpp

 };

+/// Generator for NumPy einsum
+class NumPyEinsumGenerator


All generators should be templated based on their context type. This is needed in order to make inheritance easier. The reason being that we want don't want to require the correct context class to be used with the generator (see also https://stackoverflow.com/q/18671062).

That is, we don't want to allow NumPyEinsumGenerator to be fed a PythonEinsumGeneratorContext (which could also be the PyTorch one). This is already given thanks to CRTP on the base class. However, it would be nice to just keep the option of inheriting from NumPyEinsumGenerator as well (which requires it to have a CRTP for the context class).

For "final" generators the context<->generator is 1<->1, removing ability to change context limits customization but until we run into the need to override the default I'll leave as is.

Yes but the problem is that handling of classes vs. template classes can be different in certain contexts. Hence, I would strongly suggest we make it a template as all other generator classes to ensure we don't have to refactor call-sides should we ever encounter the need.

…h installation

evaleev · 2025-11-29T13:25:27Z

many other issues remain (e.g. how not to hardwire to fp64, etc.) but good enough for the first version @Krzmbrzl

evaleev added 3 commits November 20, 2025 22:00

allow use of arithmetic constants alongside expressions in products e…

7a42611

….g. 2 * ex<Constant>(2)

preliminary implementation of PythonEinsumGenerator

8a8c94f

added 1 einsum ref output

72504a8

evaleev requested review from Krzmbrzl and Copilot November 21, 2025 13:39

Copilot started reviewing on behalf of evaleev November 21, 2025 13:39 View session

Copilot finished reviewing on behalf of evaleev November 21, 2025 13:42

Copilot AI reviewed Nov 21, 2025

View reviewed changes

ajay-mk reviewed Nov 21, 2025

View reviewed changes

.github/workflows/cmake.yml Outdated Show resolved Hide resolved

evaleev added 4 commits November 21, 2025 12:12

port over scope_exit wrapper from mpqc

66dac48

test_export: Implement RAII cleanup for temporary test directories

5af85f1

test_export_python.cpp: Fix command injection vulnerability

75213c9

PythonEinsumGenerator: index mapping does not use global state any lo…

54556c0

…nger

evaleev requested a review from Copilot November 21, 2025 18:08

Copilot started reviewing on behalf of evaleev November 21, 2025 18:08 View session

Copilot finished reviewing on behalf of evaleev November 21, 2025 18:11

Copilot AI reviewed Nov 21, 2025

View reviewed changes

Initial plan

eb8637b

Copilot AI mentioned this pull request Nov 21, 2025

Fix shell injection vulnerability in test_export_python.cpp #430

Merged

evaleev added 4 commits November 21, 2025 13:18

port over scope_exit wrapper from mpqc

9deec68

test_export: Implement RAII cleanup for temporary test directories

785edeb

test_export_python.cpp: Fix command injection vulnerability

ec5ca90

PythonEinsumGenerator: index mapping does not use global state any lo…

1f3c2e6

…nger

evaleev force-pushed the feature/export-numpy branch from 54556c0 to 1f3c2e6 Compare November 21, 2025 18:18

evaleev and others added 4 commits November 21, 2025 13:20

Apply suggestions from code review

cbe70f5

Co-authored-by: Copilot <[email protected]>

Merge branch 'feature/export-numpy' into copilot/sub-pr-429

2702b2c

Apply suggestions from code review

b11a07b

Co-authored-by: Copilot <[email protected]>

evaleev force-pushed the feature/export-numpy branch from cbe70f5 to b11a07b Compare November 21, 2025 18:24

Copilot AI reviewed Nov 23, 2025

View reviewed changes

evaleev added 3 commits November 23, 2025 00:14

switch to actions/setup-python

8b36281

[dox] fixup

32d3367

sequant::wprintf fixup format string

05d149f

Krzmbrzl reviewed Nov 23, 2025

View reviewed changes

tests/unit/export_tests/binary_contract.export_test Outdated Show resolved Hide resolved

Krzmbrzl requested changes Nov 27, 2025

View reviewed changes

evaleev added 5 commits November 27, 2025 10:31

Merge branch 'master' into feature/export-numpy

e66f924

[export] python tensor names are annotated by space tags, just like o…

d33818b

…ther backends

[export/python] validate ternary contraction, both source code and nu…

142bb86

…merics

[export/python] validate tensor + tensor*tensor expression, both sour…

663b35c

…ce code and numerics

[export/python] added reference code to named_expression_group

dc3cf69

evaleev added 2 commits November 28, 2025 10:28

[export/python] adjust sanitize_python_name to deal with wide chars

7cc9c3a

[export/python] remove files in destroy

9cd18e9

evaleev added 6 commits November 28, 2025 12:06

[export/python] preserve the case when mapping Index to einsum index

113ce2f

[export/python] redesign generator/context hierarchy along the lines …

d768e3c

…of the julia counterpart (base -> {numpy,pytorch} variants}

[export/python] (optionally, on by detail) import modules at the begi…

cea5f55

…nning of the export

[export] unname unused vars instead of (void)

f7f2b53

[export/python] make layout (row vs col) configurable

b2394ce

[export/python] fixup + validate torch unit tests

c0a1fa2

Krzmbrzl reviewed Nov 29, 2025

View reviewed changes

[ci] free up disk space on ubuntu runners to be able to afford PyTorc…

7901dcc

…h installation

evaleev force-pushed the feature/export-numpy branch from a3294de to 7901dcc Compare November 29, 2025 12:27

[skip ci] cleanup

12d5788

evaleev merged commit 01de514 into master Nov 29, 2025

evaleev deleted the feature/export-numpy branch November 29, 2025 14:37

[export] minimal numpy support #429

[export] minimal numpy support #429

Uh oh!

Conversation

evaleev commented Nov 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Nov 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Krzmbrzl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Krzmbrzl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Krzmbrzl Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

evaleev Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Krzmbrzl Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evaleev commented Nov 28, 2025

Uh oh!

evaleev commented Nov 28, 2025

Uh oh!

Krzmbrzl commented Nov 28, 2025

Uh oh!

evaleev commented Nov 28, 2025

Uh oh!

Krzmbrzl Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

evaleev Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Krzmbrzl Nov 29, 2025