Parallel 2.0 dev by lunarblock · Pull Request #1 · lunarblock/bsc

lunarblock · 2022-04-08T08:02:22Z

Description

add a description of your changes here...

Rationale

tell us why we need these changes...

Example

add an example CLI or API response...

Changes

Notable changes:

add each change in a bullet point here
...

yutianwu · 2022-04-08T08:23:06Z

core/state/journal.go

 func (ch suicideChange) revert(s *StateDB) {
-	obj := s.getStateObject(*ch.account)
+	var obj *StateObject
+	if s.parallel.isSlotDB {


we should abstract this into one function

Yes, I will try to remove this code.

yutianwu · 2022-04-08T12:21:52Z

core/state/state_object.go

 			meter = &s.db.StorageReads
 		}
-		if enc, err = s.getTrie(db).TryGet(key.Bytes()); err != nil {
+		if enc, err = s.getTrie(db).TryGet(key.Bytes()); err != nil { // fixme: handle trie concurrent safe


that‘s fine , we will disable parallel mode if snap is nil

yutianwu · 2022-04-08T12:23:55Z

core/state/state_object.go

 			return common.Hash{}, false
 		}
-		s.originStorage[key] = val.(common.Hash)
+		// s.originStorage[key] = val.(common.Hash)


why do we comment this

there will be a concurrent issue.

yutianwu · 2022-04-11T06:06:13Z

core/state/state_object.go

 			return common.Hash{}, false
 		}
-		s.originStorage[key] = val.(common.Hash)
+		// s.originStorage[key] = val.(common.Hash)


why delete this

yutianwu · 2022-04-11T08:18:42Z

core/state/statedb.go

+				"addr", addr)
+			// 1.Try to get from dirty
+			if val, ok := obj.dirtyStorage[hash]; ok {
+				log.Info("GetState key in dirty", "SlotIndex", s.parallel.SlotIndex, "txIndex", s.txIndex,


don't think directly read dirtyStorage and pendingStorage from StateObject is a good idea in StateDB

yutianwu · 2022-04-11T08:19:56Z

core/state/statedb.go

+				return val
+			}
+			// 2.Try to get from pending, it is for merge to get KV from finalized StateObject.
+			if val, ok := obj.pendingStorage[hash]; ok {


in which case we will read pendingStorage

yutianwu · 2022-04-11T09:11:02Z

core/state/statedb.go

+func (s *StateDB) unconfirmedLightCopy(obj *StateObject) *StateObject {
+	newObj := obj.lightCopy(s)
+	// do balance fixup
+	if balance := s.getBalanceFromUnconfirmedDB(obj.address); balance != nil {


newStateObject := s.unconfirmedLightCopy(stateObject) newStateObject.AddBalance(amount) s.parallel.dirtiedStateObjectsInSlot[addr] = newStateObject

since we will set dirtiedStateObjectsInSlot after unconfirmedLightCopy, why getBalanceFromUnconfirmedDB since it will always return nil.

if obj, exist := db.parallel.dirtiedStateObjectsInSlot[addr]; exist {

yutianwu · 2022-04-11T12:01:37Z

core/state/statedb.go

+			if obj, ok := db.parallel.dirtiedStateObjectsInSlot[addr]; ok { // if deleted on merge, can get from main StateDB, ok but fixme: concurrent safe
+				if _, ok := db.parallel.stateChangesInSlot[addr]; ok {
+					// fixme: dirtyStorage will be cleared on StateObject.finalize, to keep the KV in stateChangesInSlot?
+					if val, exist := obj.dirtyStorage[key]; exist {


we also should avoid directly using the dirtyStorage and pendingStorage. we can abstract these to a function like obj.GetDirtiedState()

yutianwu · 2022-04-11T12:16:18Z

core/state/statedb.go

+	BalanceChangeReferred map[common.Address]*big.Int
+	CodeChangeReferred    map[common.Address]struct{} // fixme: codehash to compare? it may not necessary
+	// NonceChangeSet      map[common.Address]struct{}  // nonce should be reliable, it is not necessary
+	AddrStateReferred map[common.Address]struct{} // the address of StateObject we refer, or the state change?


is there any possibility that we add an address to AddrStateReferred and then we get the changed code but not add it to CodeChangeReferred. or do we check it in the conflict check.

yutianwu · 2022-04-11T12:22:16Z

core/state/statedb.go

+}
+
+// for addr state check of: Exist(), Empty() and HasSuicided()
+func (s *StateDB) getAddrStateFromUnconfirmedDB(addr common.Address) *StateObject {


is there any difference between getStateObjectFromUnconfirmedDB and getAddrStateFromUnconfirmedDB

yutianwu · 2022-04-11T12:25:14Z

core/state/statedb.go

+					KVChangeReferred:      make(map[common.Address]Storage),
+				}
+			}
+			if obj, ok := db.parallel.dirtiedStateObjectsInSlot[addr]; ok {


why check addrStateChangesInSlot here

yutianwu · 2022-04-11T12:26:12Z

core/state/statedb.go

+					KVChangeReferred:      make(map[common.Address]Storage),  // the KV pair should be exist
+				}
+			}
+			if obj, exist := db.parallel.dirtiedStateObjectsInSlot[addr]; exist {


why check addrStateChangesInSlot here

yutianwu · 2022-04-11T12:29:50Z

core/state/state_object.go

 		//      have been handles via pendingStorage above.
 		//   2) we don't have new values, and can deliver empty response back
-		if _, destructed := s.db.snapDestructs[s.address]; destructed {
+		s.db.snapParallelLock.RLock()


maybe we need to check if we are in a slotdb before adding a lock

yutianwu · 2022-04-11T12:32:02Z

core/state/state_object.go

+	// fixme: if s is light copied, access mainStateDB's stateobject's dirty, if lightCopy's dirty missed
+	val := s.GetCommittedState(db, key)
+	var mainVal common.Hash
+	if !s.inMainDB {


as discussed, we can come up with a better way to replace inMainDB

yutianwu · 2022-04-11T12:35:18Z

core/state/state_object.go

+			log.Info("StateObject::GetState loadStateObj")
+			// obj = mainObj
+			mainVal = mainObj.GetCommittedState(db, key)
+			if mainVal != val {


a bit confusing here why we need to check the committed state here

* add sync pool for slotdb and remove deepCopy for mergeSlotDB * don't panic if there is anything wrong reading state * use map in sequential mode and sync.map in parallel mode * fix the comments

1.features of 2.0: ** Streaming Pipeline ** Implement universal unconfirmed state db reference, try best to get account object state. ** New conflict detect, check based on what it has read. ** Do parallel KV conflict check for large KV read ** new Interface StateDBer and ParallelStateDB ** shared memory pool for parallel objects ** use map in sequential mode and sync.map in parallel mode for concurrent StateObject access ** replace DeepCopy by LightCopy to avoid redundant memory copy of StateObject ** do trie prefetch in advance ** dispatcher 2.0 Static Dispatch & Dynamic Dispatch Stolen mode for TxReq when a slot finished its static dispatched tasks RealTime result confirm in Stage2, when most if the tx have been executed at least once Make it configurable 2.Handle of corner case: ** don't panic if there is anything wrong reading state ** handle system address, skip its balance check ** handle WBNB contract to reduce conflict rate by balance make up WBNB balance makeup by GetBalanceOpCode & depth add a lock to fix WBNB make up concurrent crash add a new interface GetBalanceOpCode

yutianwu reviewed Apr 8, 2022

View reviewed changes

yutianwu reviewed Apr 11, 2022

View reviewed changes

setunapo force-pushed the parallel_2.0_dev branch from e495214 to fdd2e52 Compare April 12, 2022 04:04

setunapo deleted the parallel_2.0_dev branch April 13, 2022 00:11

setunapo force-pushed the parallel_2.0_dev branch 6 times, most recently from 4a6a8c0 to f3863f3 Compare April 13, 2022 10:18

setunapo force-pushed the parallel_2.0_dev branch 6 times, most recently from 9465788 to 2b8f975 Compare April 21, 2022 10:05

setunapo force-pushed the parallel_2.0_dev branch 9 times, most recently from 9d2f607 to be5e6eb Compare April 26, 2022 05:44

setunapo force-pushed the parallel_2.0_dev branch 16 times, most recently from db72bc5 to 594eac2 Compare May 6, 2022 02:08

setunapo force-pushed the parallel_2.0_dev branch from 06561f3 to 8d29534 Compare May 18, 2022 02:16

improve snap mutex

8c4232f

setunapo force-pushed the parallel_2.0_dev branch from d71911a to fcf9572 Compare May 20, 2022 07:02

setunapo force-pushed the parallel_2.0_dev branch from fcf9572 to ac492c6 Compare May 20, 2022 07:10

Conversation

lunarblock commented Apr 8, 2022

Description

Rationale

Example

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yutianwu Apr 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yutianwu Apr 11, 2022 •

edited

Loading