Boost: Testing: Feature - Remote commP

[LexLuthr]

Overview

Boost team has recently created a new feature that allows the commP to be calculated on the seal workers. Currently, commP calculations are done on the boost side and this becomes a chokepoint when onboarding large amounts of data. With the new feature we expect to see performance improvement when onboarding large number of concurrent deals and lower resource requirements on the boost side.

Expectations

• Improve the data onboarding performance
  • We should see improvements in the time taken by simultaneous(parallel) deals from download to addpiece()
  • There would be more data movement between boost and sealing pipeline as payload will be sent twice with remote commP and AP calls separately.
  • We should see some performance improvement in offline-deals as well.

Instructions

• The remote commP feature should be tested in Calibration Network. If you need help with sending deals in parallel in calibnet, please reach out to LexLuthr in filecoin slack in #boost-help channel and you will receive the deals.
• Please ensure that you can have enough storage to stage 10 sectors in parallel
• Please take observations of current performance before upgrading to the testing release. We are expecting to capture the time taken by 10 deals to reach addpiece() state when onboarded simultaneously.
• Please collect at least 2 observations each time before and after the upgrade
• Please update Boost to v1.4.0-rc1
• All instructions on using boost can be found on the boost website under migrating from lotus to boost section.
• Enable remote commP by making changes to the boost configuration as below

// Whether to do commp on the Boost node (local) or on the Sealer (remote)
	RemoteCommp true

• Start the Boost node and make 10 simultaneous deals. Make observations of time taken to reach to addpiece() state.

Reporting results

Title	Metric
Total time taken to onboard 10 concurrent deals before upgrade
Average time taken by concurrent deals to reach addpiece() before upgrade
Number of deals onboarded in 24 hours before upgrade
Total time taken to onboard 10 concurrent deals after upgrade
Average time taken by concurrent deals to reach addpiece() after upgrade
Number of deals onboarded in 24 hours after upgrade

We would also like an observation of load changes within the sealing pipeline before and after enabling the feature. We are looking for network and resource usage changes along with a broader performance impact.

Reminders

• Risk Mitigation
  • Please backup everything in the repositories before making any changes.
  • Roll back to legacy markets
• Reporting Issues
  • Please fill out this template
• Questions
  • Start here
  • @LexLuthr will be monitoring the relevant slack channels and Github

[TippyFlitsUK]

Reporting results

• All tests performed with 28GiB test CAR files
• Minimal server load and immediate mpool processing
• All legacy CARs & CommPs produced with Lotus
• All boost CARs & CommPs produced with Boost
• Online deals timed from the moment the deal starts downloading to AP
• Offline deals timed from import-data command to AP
• No unusual or excessive resource usage observed

Daemon:  1.17.1-dev+mainnet+git.fa497e81a+api1.5.0
Local: lotus-miner version 1.17.1-dev+mainnet+git.fa497e81a

boost version 1.2.0+git.3e33bfa

First Pass:

Title	Metric
Current time taken for online boost deals	Total time = 1140s (download portion = 740s)
Current time taken for offline boost deals	Total time = 386s
Current time taken for online legacy deals	Total time = 753s (download portion = 341s)
Current time taken for offline legacy deals	Total time = 495s
New time taken for online boost deals	Total time = 1198s (download portion = 796s)
New time taken for offline boost deals	Total time = 408s
New time taken for online legacy deals	Total time = 760s (download portion = 342s)
New time taken for offline legacy deals	Total time = 536s

Second Pass:

Title	Metric
Current time taken for online boost deals	Total time = 1146s (download portion = 727s)
Current time taken for offline boost deals	Total time = 408s
Current time taken for online legacy deals	Total time = 749s (download portion = 340s)
Current time taken for offline legacy deals	Total time = 504s
New time taken for online boost deals	Total time = 1134s (download portion = 737s)
New time taken for offline boost deals	Total time = 434s
New time taken for online legacy deals	Total time = 757s (download portion = 338s)
New time taken for offline legacy deals	Total time = 561s

[stuberman]

What is the methodology for We are expecting to capture the time taken by 10 deals to reach addpiece() state when onboarded simultaneously.?

• Do you have a boost client that can send 10 simultaneous deals to use as a control?
• Or should I use my lotus client to propose 10 deals at the same time?
• Or should I download 10 Evergreen deals and then import them, however the imports occur serially....
• Or some other method? ('on boarding simultaneously' is unclear to me)

[stuberman]

What I am looking for is how to get those 10 simultaneous deals. Ideally you would reconfigure dealbot (a PL deal bot) to send us deals when we are ready, that way you can control the environment with deal types and speed.

[stuberman]

In the meantime, I enabled remote commP and even under load I am not seeing any problems.

[stuberman]

Remote commP has worked like a charm. It offloads my remote commP “DC” to a new server. I tried to find the task limit for a worker for “DC” but did not find the syntax (I also do remote AP on the same new server and I was able to limit the tasks).

As a suggestion, it would be a nice feature for the scheduler to assign a task to a server and if the server is unresponsive then dynamically reroute the task to an available server.

I noticed this when my new server was offline but the scheduler did not reassign the task to an available server that was also capable of the AP task.

[LexLuthr]

@stuberman Can you checked the second worker's capabilities? Below logic is used to allow DC and AP together on any worker.

		if (workerType == sealtasks.WorkerSealing || cctx.IsSet("addpiece")) && cctx.Bool("addpiece") {
			taskTypes = append(taskTypes, sealtasks.TTAddPiece, sealtasks.TTDataCid)
		}

Either the second worker does not have the DC enabled correctly or maybe it did not had open windows. Let me know what your investigation reveals. Which version of lotus is this?

[stuberman]

The new server (AP-Worker) has no problem doing both AP tasks and DC tasks (remote commP).
I disabled other tasks on this new server.

I run Lotus v1.17.1 and boost 1.4.0-rc2

[scaseye]

do you have some handy cmd line greps to do the time calculations?

[brendalee]

@LexLuthr anything you can share for this? ty!

[LexLuthr]

There is no direct method to do this. But you can either check using the deal page in the UI or use the below graphql query. Make sure to replace the dealID with correct ID or use a variable.

curl -s -X POST -H "Content-Type: application/json" -d '{"query":"query {deal(id: \"af9eb986-4397-499c-bb8b-0441674b25da\") {Logs{CreatedAt, LogMsg,}}}" }' http://localhost:8080/graphql/query | jq -r '.data.deal.Logs[] | select( (.LogMsg=="executing deal proposal received from network") or (.LogMsg=="deal successfully handed to the sealing subsystem")) | .CreatedAt'

[scaseye]

ok so time from download to AP for me varies on the online source too much... some shuttles are fast some are slow...

this seal task is new, assuming is related to remotecommp

enabled on 9/2/2022.
only online verified deals done since then.

boostd version 1.4.0-rc2+git.a425103

[donkabat]

Lotus 1.17.1+mainnet+git.8db6a939c+api1.5.0
boostd 1.4.0-rc2+git.52eda3d.dirty

Do only I have this problem: boostd makes offline legacy deals but AP worker makes online boost deals?

[LexLuthr]

Can you clarify a bit? Are boost deals (online and offline) not working with remote commp?

[donkabat]

I run one boostd and without any changes or even restart:

1. Estuary deals -> remote commP works
2. Offline legacy deal -> remote commP doesnt work

[LexLuthr]

The codepath to handle the legacy deals is same as lotus markets. Thus, remote commp is not working there. We encourage everyone to use boost offline deals(instead of legacy) to be able to use all the new market features. Where are the legacy deals coming from? Can we switch to boost client there? Let us know if we can do anything to help with the switch.

[scaseye]

ok just got done with 150 online deals through boost all downloaded within 5 hours... never had over 56 concurrent downloads.

throughput was considerably better than prior releases. so much so that I had a real problem with so many sectors in my pipeline at the same time.

DC jobs did noticeably slow down the AP jobs. When I had 10 DC jobs (which happened one time), the 3 AP jobs took over 5 times as long to complete as usual...