Add `[ink::e2e-test]` with a new E2E testing crate à la `ink_waterfall`

[cmichi]

Persisting the results from our discussion on Saturday:

• We want to remove the existing off-chain testing environment (engine) and replace it with an E2E testing framework.
• This E2E testing framework will be a new crate based on ink_waterfall. So we want to make the waterfall a product as part of ink!.
• We will retain the [ink::test] syntax, but it will then instead spawn a substrate-contracts-node (if one is not already spawned) and execute real transactions via cargo-contract instantiate/call/….

The reason behind this is that we found that we need a proper E2E testing framework to keep up with competing products. In the past users found it very irritating that they can't test e.g. the call builder in our off-chain testing environment. As we move more towards interoperability (XCM & Co.) the need for E2E testing will need to be prioritized. By making the waterfall part of ink! in this way we can in the future also use this setup to test contracts that interact with parachains.

It will also remove the need of duplicating the API and logic of pallet-contracts in ink!.

[ascjones]

I would add that there are potentially benefits to removing some abstractions required for the offchain env in the ink! codebase. It would certainly remove a lot of code, and allow more flexibility for more radical reductions in code size.

[h4x3rotab]

E2E testing framework is good. However there are many things to consider. For example:

1. Performance. Usually running the unit tests takes less than one second. If we keep only e2e tests, afaik, we need at least minutes to bring up the node, submit the transactions, and setup the browser. It slows down the development. One solution I can come up with is to build a customized node with instant seal engine (just like Ganache), and run polkadot.js based test scripts without involving the browser. Although I haven't tried RedSpot, I guess it go with this way. Generally I would be happy if the test can be done in 30s.

2. Customization. A lot of parachains will build their own ChainExtension. Any E2E test framework should allow the the developer to use their own configuration in pallet-contracts. Taking Phala Fat Contract as a special example, it has a heavily tweaked runtime environment with its own RPC API outside the substrate node. It would be nice if the E2E test framework should be flexible enough to support our use case.

3. Interpretability of the debugging process. The best part of the current unit test framework is that it allows us to access all the internal states, functions and get a understandable stack backtrace output. Once we compile it as WASM and run in a VM (especially WASMI), a lot of information for debugging will be lost. It will also be hard to access the contract internals at the rust level. I didn't come up with ideas to mitigate this problem.

[kayabaNerve]

I would like to re-iterate customization.

What if instead of natively running the Rust, as it does now, and instead of spawning a node, as proposed, an e2e-test setup and ran a full instance of wasmi? This may be best done via modularizing the contracts pallet to support such instances (if not already sufficiently modular). That could then be wrapped by a test, directly feeding TXs, or the pallet, feeding in all TXs from a block.