Merge development

.circleci/config.yml

@@ -48,8 +48,8 @@ jobs:
           name: Run WebAssembly tests in a headless browser
           command: make test-wasm-browser
       - run:
-          name: Run integration tests
-          command: make test-integration
+          name: Run browser integration tests
+          command: make test-browser-integration
       - run:
           name: Test installing Mesh without CGO
           command: CGO_ENABLED=0 go install ./...

.gitignore

@@ -13,7 +13,6 @@ browser/lib
 browser/ts/generated
 examples/browser/dist/bundle.js
 integration-tests/browser/dist/bundle.js
-integration-tests/data/standalone-0/db/
-integration-tests/data/standalone-0/p2p/
+!integration-tests/data/standalone-0/keys/privKey
+integration-tests/data/standalone-*/
 integration-tests/data/bootstrap-0/p2p/
-

.prettierrc

@@ -1,6 +1,6 @@
 {
     "tabWidth": 4,
     "printWidth": 120,
-    "trailingComma": all,
+    "trailingComma": "all",
     "singleQuote": true
 }

CHANGELOG.md

@@ -2,6 +2,30 @@
 
 This changelog is a work in progress and may contain notes for versions which have not actually been released. Check the [Releases](https://github.com/0xProject/0x-mesh/releases) page to see full release notes and more information about the latest released versions.
 
+## v8.0.0-beta-0xv3
+
+### Breaking changes 🛠 
+
+- Changed the response from `@0x/mesh-ts-client`'s `getOrdersAsync` endpoint to include the `snapshotID` and `snapshotTimestamp` at which the Mesh DB was queried along with the orders found. ([#591](https://github.com/0xProject/0x-mesh/pull/591))
+- Increased the default `ETHEREUM_RPC_MAX_REQUESTS_PER_24_HR_UTC` from 100k to 200k ([#596](https://github.com/0xProject/0x-mesh/pull/596)).
+
+### Features ✅
+
+- Instead of progressing Mesh forward by a single block on every invocation of the `BLOCK_POLLING_INTERVAL`, we now attempt to sync as many blocks as necessary to reach the latest block available. This will reduce the chances of Mesh becoming out-of-sync with it's backing Ethereum node ([#564](https://github.com/0xProject/0x-mesh/pull/564))
+- Added a new environment variable `ENABLE_ETHEREUM_RPC_RATE_LIMITING` and config option `enableEthereumRPCRateLimiting` which can be used to completely disable Mesh's internal Ethereum RPC rate limiting features. By default it is enabled, and disabling can have some consequences depending on your RPC provider. ([#584](https://github.com/0xProject/0x-mesh/pull/584))
+- Added a `SnapshotTimestamp` field to `GetOrdersResponse`, the return type of the `mesh_getOrders` RPC method. This way, the caller can know at what point in time the snapshot had been created. ([#591](https://github.com/0xProject/0x-mesh/pull/591))
+- Improved batching of events emitted from order events subscriptions ([#566](https://github.com/0xProject/0x-mesh/pull/566))
+- Added timestamp to order events ([#602](https://github.com/0xProject/0x-mesh/pull/602))
+
+### Bug fixes 🐞
+
+- Fixed an issue where order updates could have been missed if Mesh discovered blocks but didn't have time to process them before getting shut down. Now, blocks are only persisted to the DB once any order updates resulting from it have been processed. ([#566](https://github.com/0xProject/0x-mesh/pull/566)).
+- Fixed a race-condition when adding new orders to Mesh which could result in order-relevant events being missed if they occured very soon after the order was submitted and the order validation RPC call took a long time ([#566](https://github.com/0xProject/0x-mesh/pull/566)).
+- Upgraded the `web3-provider` dependency used by `@0x/mesh-rpc-client` in order to fix a bug where it was requiring either `process` OR `window` to exist in the global scope ([#601](https://github.com/0xProject/0x-mesh/pull/601)).
+- Fixed an issue where the internal Ethereum RPC rate limiter could be too aggressive in certain scenarios ([#596](https://github.com/0xProject/0x-mesh/pull/596)).
+- Add a default RPC request timeout of 30sec to all non-subscription requests sent by `@0x/mesh-rpc-client` to avoid the client from hanging endlessly if it misses a response due to network disruption ([#603](https://github.com/0xProject/0x-mesh/pull/603)).
+
+
 ## v7.2.1-beta-0xv3
 
 ### Bug fixes 🐞

CONTRIBUTING.md

@@ -49,11 +49,15 @@ make deps
 Some of the tests depend on having a test Ethereum node running. Before running
 the tests, make sure you have [Docker](https://docs.docker.com/install/)
 installed locally and start
-[0xorg/ganache-cli](https://hub.docker.com/r/0xorg/ganache-cli):
+[0xorg/ganache-cli](https://hub.docker.com/r/0xorg/ganache-cli). In these commands,
+`$GANACHE_VERSION` should be set to the version of ganache-cli that is used in the mesh project's
+CI found [here](https://github.com/0xProject/0x-mesh/blob/development/.circleci/config.yml#L10):
 
 ```
 docker pull 0xorg/ganache-cli
-docker run -ti -p 8545:8545 -e VERSION=4.3.0 0xorg/ganache-cli
+
+# Run the $GANACHE_VERSION image of ganache-cli.
+docker run -ti -p 8545:8545 -e VERSION=$GANACHE_VERSION 0xorg/ganache-cli
 ```
 
 There are various Make targets for running tests:
@@ -72,6 +76,20 @@ make test-wasm-browser
 make test-all
 ```
 
+### Potential Issues
+The default maximum number of open files is too low in some operating systems
+for the tests to be run successfully. If an error that reads like "Too many open files,"
+it may be necessary to increase this limit. On Unix-like operating systems, the `ulimit`
+command can be used as follows to accomplish this change:
+
+```bash
+# Increase number of open files that are tolerated to 2048 (a big number)
+ulimit -S -n 2048
+```
+
+It may be convenient to add this line to the `.bashrc` (or `.bash_profile` for MacOs users)
+file so that the change will go into effect whenever a new shell is created.
+
 ## Running the Linters
 
 0x Mesh is configured to use linters for both Go and TypeScript code. To run all

Makefile

@@ -59,9 +59,9 @@ test-go-serial:
 	go test ./zeroex/ordervalidator ./zeroex/orderwatch ./core -race -timeout 90s -p=1 --serial
 
 
-.PHONY: test-integration
-test-integration:
-	go test ./integration-tests -timeout 185s --integration
+.PHONY: test-browser-integration
+test-browser-integration:
+	go test ./integration-tests -timeout 185s --enable-browser-integration-tests -run BrowserIntegration
 
 
 .PHONY: test-wasm-node

browser/go/main.go

@@ -89,6 +89,7 @@ func convertConfig(jsConfig js.Value) (core.Config, error) {
 		EthereumRPCMaxContentLength:      524288,
 		EthereumRPCMaxRequestsPer24HrUTC: 100000,
 		EthereumRPCMaxRequestsPerSecond:  30,
+		EnableEthereumRPCRateLimiting:    true,
 		MaxOrdersInStorage:               100000,
 	}
 
@@ -126,6 +127,9 @@ func convertConfig(jsConfig js.Value) (core.Config, error) {
 	if ethereumRPCMaxRequestsPerSecond := jsConfig.Get("ethereumRPCMaxRequestsPerSecond"); !isNullOrUndefined(ethereumRPCMaxRequestsPerSecond) {
 		config.EthereumRPCMaxRequestsPerSecond = ethereumRPCMaxRequestsPerSecond.Float()
 	}
+	if enableEthereumRPCRateLimiting := jsConfig.Get("enableEthereumRPCRateLimiting"); !isNullOrUndefined(enableEthereumRPCRateLimiting) {
+		config.EnableEthereumRPCRateLimiting = enableEthereumRPCRateLimiting.Bool()
+	}
 	if customContractAddresses := jsConfig.Get("customContractAddresses"); !isNullOrUndefined(customContractAddresses) {
 		config.CustomContractAddresses = customContractAddresses.String()
 	}
@@ -212,7 +216,7 @@ func (cw *MeshWrapper) AddOrders(rawOrders js.Value, pinned bool) (js.Value, err
 	if err := json.Unmarshal([]byte(encodedOrders), &rawMessages); err != nil {
 		return js.Undefined(), err
 	}
-	results, err := cw.app.AddOrders(rawMessages, pinned)
+	results, err := cw.app.AddOrders(cw.ctx, rawMessages, pinned)
 	if err != nil {
 		return js.Undefined(), err
 	}

browser/ts/index.ts

@@ -86,10 +86,18 @@ export interface Config {
     // Parity, feel free to double the default max in order to reduce the number
     // of RPC calls made by Mesh. Defaults to 524288 bytes.
     ethereumRPCMaxContentLength?: number;
+    // Determines whether or not Mesh should limit the number of Ethereum RPC
+    // requests it sends. It defaults to true. Disabling Ethereum RPC rate
+    // limiting can reduce latency for receiving order events in some network
+    // conditions, but can also potentially lead to higher costs or other rate
+    // limiting issues outside of Mesh, depending on your Ethereum RPC provider.
+    // If set to false, ethereumRPCMaxRequestsPer24HrUTC and
+    // ethereumRPCMaxRequestsPerSecond will have no effect.
+    enableEthereumRPCRateLimiting?: boolean;
     // A cap on the number of Ethereum JSON-RPC requests a Mesh node will make
-    // per 24hr UTC time window (time window starts and ends at 12am UTC). It
-    // defaults to the 100k limit on Infura's free tier but can be increased
-    // well beyond this limit for those using alternative infra/plans.
+    // per 24hr UTC time window (time window starts and ends at midnight UTC).
+    // It defaults to 200k but can be increased well beyond this limit depending
+    // on your infrastructure or Ethereum RPC provider.
     ethereumRPCMaxRequestsPer24HrUTC?: number;
     // A cap on the number of Ethereum JSON-RPC requests a Mesh node will make
     // per second. This limits the concurrency of these requests and prevents
@@ -168,6 +176,7 @@ interface WrapperConfig {
     ethereumRPCMaxContentLength?: number;
     ethereumRPCMaxRequestsPer24HrUTC?: number;
     ethereumRPCMaxRequestsPerSecond?: number;
+    enableEthereumRPCRateLimiting?: boolean;
     customContractAddresses?: string; // json-encoded instead of Object.
     maxOrdersInStorage?: number;
 }
@@ -439,6 +448,7 @@ export enum OrderEventEndState {
 }
 
 interface WrapperOrderEvent {
+    timestamp: string;
     orderHash: string;
     signedOrder: WrapperSignedOrder;
     endState: OrderEventEndState;
@@ -451,6 +461,7 @@ interface WrapperOrderEvent {
  * or filled.
  */
 export interface OrderEvent {
+    timestampMs: number;
     orderHash: string;
     signedOrder: SignedOrder;
     endState: OrderEventEndState;
@@ -836,6 +847,7 @@ function signedOrderToWrapperSignedOrder(signedOrder: SignedOrder): WrapperSigne
 function wrapperOrderEventToOrderEvent(wrapperOrderEvent: WrapperOrderEvent): OrderEvent {
     return {
         ...wrapperOrderEvent,
+        timestampMs: new Date(wrapperOrderEvent.timestamp).getTime(),
         signedOrder: wrapperSignedOrderToSignedOrder(wrapperOrderEvent.signedOrder),
         fillableTakerAssetAmount: new BigNumber(wrapperOrderEvent.fillableTakerAssetAmount),
         contractEvents: wrapperContractEventsToContractEvents(wrapperOrderEvent.contractEvents),

cmd/mesh/main.go

@@ -62,7 +62,7 @@ func main() {
 	go func() {
 		defer wg.Done()
 		log.WithField("rpc_addr", config.RPCAddr).Info("starting RPC server")
-		if err := listenRPC(app, config, ctx); err != nil {
+		if err := listenRPC(ctx, app, config); err != nil {
 			rpcErrChan <- err
 		}
 	}()

cmd/mesh/rpc_handler.go

@@ -28,15 +28,17 @@ const orderEventsBufferSize = 8000
 
 type rpcHandler struct {
 	app *core.App
+	ctx context.Context
 }
 
 // listenRPC starts the RPC server and listens on config.RPCAddr. It blocks
 // until there is an error or the RPC server is closed.
-func listenRPC(app *core.App, config standaloneConfig, ctx context.Context) error {
+func listenRPC(ctx context.Context, app *core.App, config standaloneConfig) error {
 	// Initialize the JSON RPC WebSocket server (but don't start it yet).
 	rpcAddr := fmt.Sprintf("%s", config.RPCAddr)
 	rpcHandler := &rpcHandler{
 		app: app,
+		ctx: ctx,
 	}
 	rpcServer, err := rpc.NewServer(rpcAddr, rpcHandler)
 	if err != nil {
@@ -85,8 +87,11 @@ func (handler *rpcHandler) GetOrders(page, perPage int, snapshotID string) (resu
 		if _, ok := err.(core.ErrSnapshotNotFound); ok {
 			return nil, err
 		}
+		if _, ok := err.(core.ErrPerPageZero); ok {
+			return nil, err
+		}
 		// We don't want to leak internal error details to the RPC client.
-		log.WithField("error", err.Error()).Error("internal error in AddOrders RPC call")
+		log.WithField("error", err.Error()).Error("internal error in GetOrders RPC call")
 		return nil, constants.ErrInternal
 	}
 	return getOrdersResponse, nil
@@ -114,7 +119,7 @@ func (handler *rpcHandler) AddOrders(signedOrdersRaw []*json.RawMessage, opts rp
 			err = errors.New("method handler crashed in AddOrders RPC call (check logs for stack trace)")
 		}
 	}()
-	validationResults, err := handler.app.AddOrders(signedOrdersRaw, opts.Pinned)
+	validationResults, err := handler.app.AddOrders(handler.ctx, signedOrdersRaw, opts.Pinned)
 	if err != nil {
 		// We don't want to leak internal error details to the RPC client.
 		log.WithField("error", err.Error()).Error("internal error in AddOrders RPC call")

constants/constants.go

@@ -2,6 +2,7 @@ package constants
 
 import (
 	"errors"
+	"fmt"
 	"math/big"
 
 	"github.com/ethereum/go-ethereum/common"
@@ -30,16 +31,16 @@ var (
 	// GanacheAccount0 is the first account exposed on the Ganache test Ethereum node
 	GanacheAccount0           = common.HexToAddress("0x5409ed021d9299bf6814279a6a1411a7e866a631")
 	ganacheAccount0PrivateKey = common.Hex2Bytes("f2f48ee19680706196e2e339e5da3491186e0c4c5030670656b0e0164837257d")
-	// GanacheAccount1 is the first account exposed on the Ganache test Ethereum node
+	// GanacheAccount1 is the second account exposed on the Ganache test Ethereum node
 	GanacheAccount1           = common.HexToAddress("0x6ecbe1db9ef729cbe972c83fb886247691fb6beb")
 	ganacheAccount1PrivateKey = common.Hex2Bytes("5d862464fe9303452126c8bc94274b8c5f9874cbd219789b3eb2128075a76f72")
-	// GanacheAccount2 is the first account exposed on the Ganache test Ethereum node
+	// GanacheAccount2 is the third account exposed on the Ganache test Ethereum node
 	GanacheAccount2           = common.HexToAddress("0xe36ea790bc9d7ab70c55260c66d52b1eca985f84")
 	ganacheAccount2PrivateKey = common.Hex2Bytes("df02719c4df8b9b8ac7f551fcb5d9ef48fa27eef7a66453879f4d8fdc6e78fb1")
-	// GanacheAccount3 is the first account exposed on the Ganache test Ethereum node
+	// GanacheAccount3 is the fourth account exposed on the Ganache test Ethereum node
 	GanacheAccount3           = common.HexToAddress("0xe834ec434daba538cd1b9fe1582052b880bd7e63")
 	ganacheAccount3PrivateKey = common.Hex2Bytes("ff12e391b79415e941a94de3bf3a9aee577aed0731e297d5cfa0b8a1e02fa1d0")
-	// GanacheAccount4 is the first account exposed on the Ganache test Ethereum node
+	// GanacheAccount4 is the fifth account exposed on the Ganache test Ethereum node
 	GanacheAccount4           = common.HexToAddress("0x78dc5d2d739606d31509c31d654056a45185ecb6")
 	ganacheAccount4PrivateKey = common.Hex2Bytes("752dd9cf65e68cfaba7d60225cbdbc1f4729dd5e5507def72815ed0d8abc6249")
 )
@@ -82,3 +83,6 @@ const MaxOrderSizeInBytes = 8192
 // node stores archive-level state. One cannot make `eth_call` requests specifying blocks earlier than
 // 128 blocks ago on non-archive nodes.
 const MaxBlocksStoredInNonArchiveNode = 128
+
+// ErrMaxMessageSize is the error emitted when a message exceeds it's max size
+var ErrMaxMessageSize = fmt.Errorf("message exceeds maximum size of %d bytes", MaxOrderSizeInBytes)