Fix IMT skipping

[MartinOndejka]

Fixes of #350
fix ZK-341

Circuit changes:

• check that leaf before y in IMT is non-empty
• add signature check for IMT

DA changes:

• send acc set sparse ledger to da node to check that order of new accounts are the same as in acc set, so that there is no skipping

[gemini-code-assist]

Summary of Changes

Hello @MartinOndejka, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant enhancements to the Indexed Merkle Tree (IMT) and the Data Availability (DA) layer, focusing on data integrity and verifiability. It strengthens circuit-level guarantees for IMT operations, integrates the IMT's state into the DA layer's cryptographic signatures, and adds robust checks to ensure the consistent ordering of accounts. These changes are supported by necessary data model updates, RPC versioning, a new database migration, and comprehensive logging improvements across the system.

Highlights

• Indexed Merkle Tree (IMT) Circuit Enhancements: Introduced a new are_paths_neighbors function and added circuit-level checks to ensure that the leaf before a new entry (y) in the IMT is non-empty and that paths are correctly ordered. This significantly improves the integrity and correctness of IMT operations within the circuits.
• IMT Signature Integration in DA Layer: The Data Availability (DA) layer now includes the Merkle root of the account set (IMT) in the data signed for diffs. This strengthens the cryptographic guarantees by ensuring that the state of the account set is also attested to by the DA node's signature.
• DA Layer Account Set Ordering Verification: A critical new check has been implemented in the DA layer to verify that the order of new accounts in the sparse ledger matches their order in the account set openings. This prevents potential skipping or reordering of accounts, ensuring data consistency and integrity.
• DA Layer Data Structure and RPC Updates: The DA layer's internal data structures (Diff_table) and RPCs have been updated to store and transmit acc_set_openings (sparse IMT data) alongside ledger openings. This required introducing a new Diff.Stable.V3 version and a corresponding database migration to accommodate the additional data.
• Improved Logging and Observability: Widespread addition of ~logger parameters to Gql_client functions and the Utils.retry mechanism has been implemented across the sequencer and DA layer. This provides better observability, debugging capabilities, and context for operations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[linear]

ZK-341 Index Merkel tree benchmark (post vs change account)

[gemini-code-assist]

Code Review

This pull request introduces several changes to fix an issue with IMT skipping. The changes span across circuits and the DA layer. In the circuits, a new check is added to ensure the leaf before a new entry y is non-empty, and adjacency is verified. In the DA layer, the account set sparse ledger is now sent to the DA node to verify the order of new accounts, preventing skipping. The signature for DA checks now also covers the account set.

The changes are quite extensive and touch many files to propagate necessary data like acc_set_openings. The introduction of versioned RPC fallbacks is a good improvement for backward compatibility.

I've found two critical issues that need to be addressed:

1. An incorrect assertion in indexed_merkle_tree.ml that compares two different Merkle roots that should not be equal.
2. A typo in da_layer/lib/core.ml that makes a security check always pass.

Apart from these issues, the changes look good and consistent.