Merge pull request #44 from substrate-developer-hub/sl/tx-doc-and-review

Transaction and execution doc

Author: Sacha Lansky

old-docs-migration/docs/02-concepts/d-transaction-pool.md

@@ -7,119 +7,15 @@ category: concepts
 keywords: transaction, pool, order, ordering, sorting, validity
 ---
 
-The transaction pool contains all transactions ([signed](/v3/concepts/extrinsics#signed-transactions)
-and [unsigned](/v3/concepts/extrinsics#unsigned-transactions)) broadcasted to the network that have been
-received and validated by the local node.
-
-## Validity
-
-The transaction pool checks for transaction validity.
-Note that `validity` of the transaction is not hard-wired to the transaction pool, but is defined by the runtime.
-Example validity checks are:
-
-- Checking if the Transaction Index (nonce) is correct.
-- Checking if the account has enough funds to pay for the associated fees.
-- Checking if the signature is valid.
-
-The transaction pool also regularly checks validity of existing transactions within the pool.
-A transaction will be dropped from the pool if it is found to be invalid or is an expired mortal transaction.
-
-[`validate_transaction`](/rustdocs/latest/sp_transaction_pool/runtime_api/trait.TaggedTransactionQueue.html#method.validate_transaction) is called from the runtime to checks for a valid signature and nonce (or output for a UTXO chain) and returns a `Result`.
+## Validity 
+[content not used / to remove] 
 It does this check in isolation, so it will not catch errors such as the same output being spent twice.
+`validate_transaction` does not check whether calls to a pallet will succeed.
 
-Although it is possible, `validate_transaction` does not check whether calls to pallets will succeed.
-It is a potential DoS vector since all transactions in the network will be passed into `validate_transaction`.
-
-The `validate_transaction` function should focus on providing the necessary information for the pool to order and prioritize transactions, and quickly reject all transactions that are invalid or outdated.
-The function will be called frequently, potentially multiple times for the same transaction.
-It is also possible for `validate_transaction` to fail a dependent transaction that would pass `execute_block` if it were executed in the correct order.
-
-## Sorting
-
-If the transaction is valid, the transaction queue sorts transactions into two groups:
-
-- Ready Queue - Contains transactions that can be included in a new pending block.
-  For runtimes built with FRAME, the transactions must follow the exact order in the ready queue.
-- Future Queue - Contains transactions that may become valid in the future.
-  For example, a transaction may have a `nonce` that is too high for its account.
-  This transaction will wait in the future queue until the preceding transactions are included in the chain.
-
-Although it's possible to design a custom runtime to remove the queues' strict transaction ordering requirements.
+[ todo : put this in architecture? or design? ]
+It's possible to design a custom runtime to remove the strict ordering requirements of the queue.
 This would allow full nodes to implement different strategies on transaction propagation and block inclusion.
 
-## Transaction dependencies
-
-The [`ValidTransaction` struct](/rustdocs/latest/sp_runtime/transaction_validity/struct.ValidTransaction.html) defines the `requires` and `provides` parameters to build a dependency graph of transactions.
-Together with `priority` (discussed below), this dependency graph allows the pool to produce a valid linear ordering of transactions.
-
-For runtimes built with FRAME, the nodes order transactions with an account-based system.
-Every signed transaction needs to contain a nonce, which is incremented by 1 every time a new transaction is made.
-For example, the first transaction from a new account will have `nonce = 0` and the second transaction will have `nonce = 1`.
-
-At minimum, FRAME transactions have a `provides` tag of `encode(sender ++ nonce)` and a `requires` tag of `encode(sender ++ (nonce -1)) if nonce > 1`.
-Transactions do not require anything if `nonce=0`.
-As a result, all transactions coming from a single sender will form a sequence in which they should be included.
-
-Substrate supports multiple `provides` and `requires` tags, so custom runtimes can create alternate dependency (ordering) schemes.
-
-## Transaction priority
-
-Transaction `priority` in the `ValidTransaction` struct determines the ordering of transactions that are in the ready queue.
-If a node is the next block author, it will order transactions from high to low priority in the next block until it reaches the weight or length limit of the block.
-
-`priority` defines the linear ordering of a graph in the case of one transaction unlocking multiple dependent transactions.
-For example, if we have two (or more) transactions that have their dependencies satisfied, then we use `priority` to choose the order for them.
-
-For runtimes built with [FRAME](/v3/runtime/frame), `priority` is defined as the `fee` that the transaction is going to pay.
-For example:
-
-- If we receive 2 transactions from _different_ senders (with `nonce=0`), we use `priority` to
-  determine which transaction is more important and included in the block first.
-- If we receive 2 transactions from the _same_ sender with an identical `nonce`, only one
-  transaction can be included on-chain. We use `priority` to choose the transaction with a higher
-  `fee` to store in the transaction pool.
-
-Note that the pool does not know about fees, accounts, or signatures — it only deals with the validity of the transaction and the abstract notion of the `priority`, `requires`, and `provides` parameters.
-All other details are defined by the runtime via the `validate_transaction` function.
-
-## Transaction lifecycle
-
-A transaction can follow two paths:
-
-### Block produced by our node
-
-1. Our node listens for transactions on the network.
-1. Each transaction is verified and valid transactions are placed in the transaction pool.
-1. The pool is responsible for ordering the transactions and returning ones that are ready to be included in the block.
-   Transactions in the ready queue are used to construct a block.
-1. Transactions are executed and state changes are stored in local memory.
-   Transactions from the `ready` queue are also propagated (gossiped) to peers over the network.
-   We use the exact ordering as the pending block since transactions in the front of the queue have a higher priority and are more likely to be successfully executed in the next block.
-1. The constructed block is published to the network.
-   All other nodes on the network receive and execute the block.
-
-Notice that transactions are not removed from the ready queue when blocks are authored, but removed _only_ on block import.
-This is due to the possibility that a recently-authored block may not make it into the canonical chain.
-
-### Block received from network
-
-The block is executed and the entire block either succeeds or fails.
-
-## Signed extensions
-
-[`SignedExtension`](/rustdocs/latest/sp_runtime/traits/trait.SignedExtension.html) is a trait by which a transaction can be extended with additional data or logic.
-Signed extensions are used anywhere you want some information about a transaction prior to execution.
-This is heavily used in the transaction pool.
-
-The runtime can use some of this data, for example the `Call` that will be dispatched, to calculate transaction fees.
-Signed extensions also include an `AdditionalSigned` type that can hold any encodable data, and therefore allow you to perform any custom logic prior to including or dispatching a transaction.
-The transaction queue regularly calls functions from `SignedExtension` to validate transactions prior to block construction to avoid including transactions that will fail in blocks.
-
-Despite the name, `SignedExtension` can also be used to verify [unsigned transactions](/v3/concepts/extrinsics#unsigned-transactions).
-The `*_unsigned` set of methods can be implemented to encapsulate validation, spam, and replay protection logic that is needed by the transaction pool.
-
-- [Signed Extension API](/rustdocs/latest/sp_runtime/traits/trait.SignedExtension.html)
-
 ## Further Reading
 
 - [Extrinsics](/v3/concepts/extrinsics)

old-docs-migration/docs/unused-material.md

@@ -0,0 +1,145 @@
+A place for content that's not been used anywhere.
+Perhaps to scrap, haven't decided yet. 
+
+# Transactions 
+
+## Where transactions are defined
+
+As discussed in [Runtime development](), the Substrate runtime contains the business logic that defines valid transactions, determines whether the transactions are sent as signed or unsigned, and how transactions change the state of the chain.
+
+Typically, you use pallets to compose the runtime functions and to implement the transactions you want your chain to support.
+After you compile the runtime, users interact with the blockchain to submit requests that are processed as transactions.
+For example, a user might submit a request to transfer funds from one account to another.
+The request becomes a signed transaction that contains the signature for that user account and if there are sufficient funds in the user's account to pay for the transaction, the transaction executes successfully, and the transfer is made.
+
+<!-- Although inherents are very different by nature compared to signed and unsigned transaction types, they are treated like any other transaction from the perspective of the transaction pool. -->
+
+<!-- See how FRAME implements and used these types of transactions:
+- #[pallet::inherent]: 
+- #[pallet::validate_unsigned]:
+- dispatch origins: signed, unsigned, none (inherent) -->
+
+A `CheckedExtrinsic` is quite different on the other hand.
+It's format requires:
+
+- a `function` corresponding to the call being made.
+- a `signed` field which carries information on where this extrinsic comes from (`AccountId`) and the number of extrinsics that have come before from the same source (defined by the `CheckedExtrinsic` signed extension).
+
+
+<!-- All signed transactions in a FRAME runtime have the following formatted around the `UncheckExtrinsics` type and 
+
+- tip
+- period
+- current_block 
+- era 
+- extra 
+- raw_payload 
+- signature
+- address 
+
+And look like this when decoded:
+
+```text
+Extra: (CheckNonZeroSender, CheckSpecVersion, CheckTxVersion, CheckGenesis, CheckEra(Era::Mortal(2048, 99)), CheckNonce(0), CheckWeight, ChargeTransactionPayment<0>)
+Additional Signed: (259, 1, 0x0000000000000000000000000000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000000000000000000000000000, (), (), ())
+Raw Payload: [6, 0, 255, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125, 145, 1, 58, 6, 0, 0, 3, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+```
+
+Notice some other [signed extensions](#signed-extensions) which perform certain checks on the format of the transaction.
+ -->
+
+## Validity
+
+The validity of a transaction is verified by checks defined in the runtime. 
+Before a transaction is propagated to other nodes, it is "tagged" and checked for whether:
+
+- The signature is valid.
+- The account has enough funds to pay for the associated fees.
+- The transaction index (nonce) is correct.
+
+The transaction pool also regularly checks for transaction validity of existing transactions within the pool.
+Every transaction is defined in the runtime using the [`ValidTransaction`](https://docs.substrate.io/rustdocs/latest/sp_runtime/transaction_validity/struct.ValidTransaction.html) struct.
+A transaction will be dropped from the pool if it is found to be invalid or expired, for example:
+
+- If another transaction contains the same transaction dependencies (see: [`requires`](https://docs.substrate.io/rustdocs/latest/sp_runtime/transaction_validity/struct.ValidTransaction.html#structfield.requires) and [`TransactionTag`](https://docs.substrate.io/rustdocs/latest/sp_runtime/transaction_validity/type.TransactionTag.html)).
+- If the transaction has exceeded the [`TransactionLongevity`](https://docs.substrate.io/rustdocs/latest/sp_runtime/transaction_validity/type.TransactionLongevity.html#), i.e. the amount of blocks it can remain valid for.
+
+Other causes for invalid transactions are listed [here](https://docs.substrate.io/rustdocs/latest/sp_runtime/transaction_validity/enum.InvalidTransaction.html).
+
+The [`TaggedTransactionQueue`](https://docs.substrate.io/rustdocs/latest/sp_transaction_pool/runtime_api/trait.TaggedTransactionQueue.html) runtime API provides a [`validate_transaction`](/rustdocs/latest/sp_transaction_pool/runtime_api/trait.TaggedTransactionQueue.html#method.validate_transaction) method whose job is to verify the correctness of the transaction against the current state (i.e. the signature and nonce).
+The function will be called frequently, potentially multiple times for the same transaction.
+In doing so, it provides the necessary information for the transaction pool to order and prioritize transactions.
+It is also possible for `validate_transaction` to reject a dependent transaction that would pass `execute_block` if it were executed in the correct order.
+
+
+A decoded signed transaction from a JSON-RPC call would look like:
+
+```json
+"method":{
+            "pallet":"balances",
+            "method":"transfer"
+         },
+         "signature":{
+            "signature":"0x94b63112648e8e692f0076fa1ccab3a04510c269d1392c1df2560503865e144e3afd578f1e37e98063b64b98a77a89a9cdc8ade579dcac0984e78d90646a052001",
+            "signer":{
+               "id":"Gr5sBB1EgdmQ7FG3Ud2BdECWQTMDXNgGPfdHMMtDsmT4Dj3"
+            }
+         },
+         "nonce":"12",
+         "args":{
+            "dest":{
+               "id":"J6ksma2jVeHRcRoYPZBkJRzRbckys7oSmgvjKLrVbj1U8bE"
+            },
+            "value":"100000000"
+         },
+         "tip":"0",
+         "hash":"0xfbc5e5de75d64abe5aa3ee9272a3112b3ce53710664f6f2b9416b2ffda8799c2",
+         "info":{
+            "weight":"201217000",
+            "class":"Normal",
+            "partialFee":"2583332634"
+         }
+}
+```
+
+An inherent would look like:
+
+```json
+"method":{
+            "pallet":"timestamp",
+            "method":"set"
+         },
+         "signature":null,
+         "nonce":null,
+         "args":{
+            "now":"1620636072000"
+         },
+         "tip":null,
+         "hash":"0x8b853f49b6543e4fcbc796ad3574ea5601d2869d80629e080e501da4cb7b74b4",
+         "info":{
+
+         }
+}
+```
+
+
+The [`ValidTransaction` struct](/rustdocs/latest/sp_runtime/transaction_validity/struct.ValidTransaction.html) defines two parameters used to determine the ordering of transactions in the pool: `requires` and `provides`. 
+Together they create a dependency graph which allows the pool to produce a valid linear ordering of transactions.
+
+Substrate supports multiple `provides` and `requires` tags.
+This enables runtime engineers to create alternate ordering schemes.
+
+
+## FRAME implementation
+
+See how the core mechanisms described in this article are implemented in FRAME:
+- `pallet_transaction_payment_rpc`: RPC interface for the transaction payment pallet.
+- `pallet_transaction_payment`: Transaction Payment Pallet 
+- `pallet_transaction_storage`: Transaction storage pallet. Indexes transactions and …
+- `pallet_transaction_payment_rpc_runtime_api`: Runtime API definition for transaction payment pallet.
+
+`Operational` transactions are getting additional priority bump (3/4 * `u64::max()`), because that prevents other transactions from getting in front of them in the pool, but now equal to the `OperationalVirtualTip` amount configurable by a runtime developer.
+
+Notes:
+
+- We need some note in the "welcome" section that says that our docs describe building parachains with Substrate, specifically for runtimes built with FRAME.

source/docs/main-docs/02-fundamentals/architecture.md

@@ -32,16 +32,6 @@ It defines what transactions are valid and invalid and determines how the chain'
 The client on the other hand, or everything part of the node that's outside the runtime, is responsible for handling peer discovery, transaction pools, consensus and answering RPC calls from the outside world. 
 While performing these tasks, the outer node sometimes needs to query the runtime for information, or provide information to the runtime. 
 
-Communication of this sort is made possible by runtime APIs and host functions.
-
-### Runtime APIs and host functions
-
-A **host function** is a function that is expressed outside of the runtime whose job is to provide data required by the runtime, like providing hardware information about a node or network status.
-A **runtime API** provides a way to query information about the runtime, useful to make calls for querying a runtime's metadata or querying some storage item.
-
-Although designed to be constraint-free, any Substrate runtime must implement a runtime API for basic block execution and runtime versioning (see: [`Core`](/rustdocs/latest/sp_api/trait.Core.html)) and another for providing metadata to external applications (see: [`Metadata`](/rustdocs/latest/sp_api/trait.Metadata.html)).
-Other [runtime APIs]() are available out of the box.
-
 Learn more about [availble runtime APIs](./link-todo-design) in Substrate.
 
 ### Wasm runtimes