-
Notifications
You must be signed in to change notification settings - Fork 20
Feat: multi threaded scheduler #589
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
bmuddha
wants to merge
409
commits into
master
Choose a base branch
from
bmuddha/feat/multi-threaded-scheduler
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,001
−222
Open
Changes from 250 commits
Commits
Show all changes
409 commits
Select commit
Hold shift + click to select a range
04730fd
chore: remove obsolete fee payer test
thlorenz 2a8bf01
chore: remove obsolete escrow initialization in schedule commit tests
thlorenz 6b443e8
feat: add way to delegate airdropped account
thlorenz e6e0aff
chore: make task to compile sbf programs
thlorenz 4c1b478
chore: allow separate PR when inititing schedule commit PDA
thlorenz 863644f
chore: adapt schedule commit ix context initialization
thlorenz aef1606
chore: delegate committees working
thlorenz 459b58b
chore: more checks + lots as part of schedule commit setup
thlorenz d5d6f88
feat: set magic context account delegated
thlorenz f0e0305
chore: refresh ephem blockhash for each tx in schedule-commit tests
thlorenz b6b7506
chore: fix clippy warning
thlorenz 2548e79
chore: remove cached blockhashes
thlorenz 1f92a44
chore: colored backtraces
thlorenz 0d1e771
chore: comment clarification
thlorenz 5c54a9f
chore: logs + more fresh blockhashes
thlorenz a74077c
chore: simulate tx after confirming fails a few times
thlorenz 928547e
chore: move ouit static transaction methods from integration context
thlorenz a8b4d15
chore: add logs to schedule commit inc ix for debugging
thlorenz 5607063
chore: improved logging around tests
thlorenz da9a10e
chore: allow two possible errors when modifiying delegated acc on chain
thlorenz 5fecca1
feat: subscribe to account being undelegated
thlorenz 662944b
feat: unsubscribing when we receive sub update with _now_ delegated a…
thlorenz 4cfac7b
chore: cleanup tests which now passes, but inconsistently
thlorenz d15e29e
chore: commit and undelegate test passing
thlorenz 4a67d01
chore: fix more tests + turn on backtrace prettifier in more cases
thlorenz f5d5bbc
chore: include attempt to repro issue
thlorenz fc585a3
chore: add notes for improvements
thlorenz 9ad32f3
fix: guaranteeing scheduled commit tranaction uniqueness
thlorenz 14f1bc8
chore: update notes for babur
thlorenz 19738c0
chore: remove no longer needed repro attempt
thlorenz cadc7cd
chore: fix commit security tests
thlorenz fe02829
chore: fix chainlink tests
thlorenz 35d05e3
fix: issues tests setup
thlorenz 84546a1
chore: remove obsolete frequent commit issues test
thlorenz 9fd489b
chore: add some logs to config tests
thlorenz f83120b
chore: fix validator offline configs
thlorenz b66e8d5
chore: add task to start magicblock-api validators
thlorenz 41d59b5
chore: update progress docs
thlorenz e8ca627
chore: schedule intent test improvement note
thlorenz b7201f1
chore: update notes with test state
thlorenz b7efced
chore: pubsub tests startup two validators
thlorenz 014c7e3
chore: pubsub account tests passing
thlorenz 32ad8a9
chore: fix pubsub log tests
thlorenz 8a2eae5
chore: fix slot + program subscribe tests - cancel is brittle in all
thlorenz c3834ee
chore: stabilize pubsub cancel
thlorenz de8e5e3
fix: pubsub ephem setup
thlorenz 429b33f
chore: fix remaining pubsub tests
thlorenz b8cd767
feat: add locking coordinator to the scheduler
bmuddha 17dd52b
fixes: fix ledger/adb slot mismatch + update ix debuggin notes
bmuddha 9eee79d
chore: update work items
thlorenz 9766794
fixes: fix ledger/adb slot mismatch + update ix debuggin notes
bmuddha a962537
feat: persist failed transactions to the ledger
bmuddha aee453d
feat: allow storing zero lamport accounts
thlorenz d18fe12
chore: add optional collection to mark empty when not found
thlorenz 82edc21
chore: fix clippy in executor
thlorenz 7a19b24
chore: pass through mark as empty
thlorenz 0f1f25a
chore: remove obsolet force_refetch flag
thlorenz 247b693
Merge branch 'thlorenz/chainlink' into bmuddha/feat/multi-threaded-sc…
bmuddha f5523fb
chore: fix all calls to pass mark empty accounts
thlorenz cf227fb
feat: impl + testing mark empty behavior
thlorenz e2426cd
feat: mark feepayers as empty accounts when not found + ix test
thlorenz d4ee71b
fix: fair scheduling
bmuddha 4e84847
fix: cleanup the transaction to executor mapping
bmuddha 0d23617
refactor: improved documentation and code structure
bmuddha 6a3f27b
feat: prepare lookup tables when configured
thlorenz 4061362
chore: fix clippy
thlorenz 0603ee0
feat: remove all non-delegated accounts when resuming with existing a…
thlorenz 895887c
fix: correct contention resolution
bmuddha 04443c3
chore: first 2 ledger restore tests working
thlorenz 676f49d
chore: fix name
thlorenz 74bc3d7
chore: stage tests.rs file
bmuddha 61226ee
fix: divide the CPU resources between RPC and the execution
bmuddha 1c01030
chore: extract functionality + use to pass multi tranfers ledger rest…
thlorenz a031b7d
chore: fix minor typo in error message
thlorenz 8da42fc
chore: single block tx order test fixed
thlorenz ec0976e
chore: update progress
thlorenz 082d943
cleanup: remove stray prints and improve comments
bmuddha 74e8f09
chore: counter tests passing
thlorenz 0e499ea
feat: remove undelegate accounts _after_ ledger replay completed
thlorenz ae87eb6
chore: update progress
thlorenz 03bc653
chore: update deploy test skip message
thlorenz a6bff5d
chore: verify programs present during ledger replay
thlorenz 3af00bc
chore: delegated accounts test fixed
thlorenz c722016
chore: fix commit delegated restore test
thlorenz 766d161
chore: restore test surfaced clean bank issue
thlorenz 8d8cf49
chore: don't clean blacklisted accounts on restore
thlorenz 21b953c
chore: fix calls to updated chainlink method
thlorenz d632a57
chore: fix readonly update after test
thlorenz 7fa6dc7
chore: update undelegate before restart test indicating why this cann…
thlorenz ee55cba
chore: update notes for babur
thlorenz 3664298
chore: fix ledger account flush test
thlorenz b31f32a
chore: fix timestamp ledger test
thlorenz 70d9c43
chore: fix restore ledger with new keypair test
thlorenz 65365a1
chore: fix resume strategies test
thlorenz c1688ee
chore: ignore undelegate between restarts test for now
thlorenz 6cfca11
Merge branch 'master' into thlorenz/chainlink
bmuddha ce235d0
chore: fix merge conflicts in cargo lock
thlorenz f2bcc6c
chore: fix more merge issues to build validator
thlorenz 722f806
chore: fix merge issues for test-integration
thlorenz 0f61748
chore: fix compile issue due to merge
thlorenz 6928135
feat: ledger test validators start at different ports
thlorenz 071bfb0
chore: initial changes to allow running ledger tests in parallel
thlorenz 5ce2cbd
chore: true random port picking for ledger tests
thlorenz 16a46e8
chore: all non-skipped ledger tests passing in parallel
thlorenz 873e040
chore: allow overriding ephem port while running single test
thlorenz 7b17d39
chore: rename test to proper name
thlorenz 3e345a0
chore: update progress
thlorenz a50cbaa
fix: for failed txns use meta defaults that are explorer compatible
bmuddha 6f52e37
fix: sync accountsdb slot after ledger replay
bmuddha 7c7d175
fix: disable schedule intent tests
bmuddha 4b7b474
chore: apply c5f2d7c to fix config tests
thlorenz b3d922e
chore: update progress
thlorenz 7405eb3
fix: correcting resume slot instead of returning error
thlorenz 6fc6071
chore: use svm repo instead of local copy
thlorenz 2507acf
chore: give more context for clone errors
thlorenz 17e6207
chore: log _correctly_ canceled subs on trace to reduce debug noise
thlorenz 1e316c4
chore: log received/scheduled/executed transactions on trace
thlorenz d96cac9
chore: logs optimized to triage unsub behavior
thlorenz 3f45f40
chore: fix flexi counter ledger restore test
thlorenz 0f40459
chore: minor comment update
thlorenz aef5e32
fix: slot to reset accounts db from takes reset slot into account
thlorenz cf87239
chore: fix resume test case
thlorenz 8b0cd87
chore: link issue tracking undelegate between restarts regression
thlorenz 18d26f4
doc: add readme on how to run integration tests
thlorenz fe0a789
chore: update latest status
thlorenz 057f93d
chore: adapt escrow assertions to 'clone as placeholder' change
thlorenz cfd45f7
chore: fix API clocks match test
thlorenz c234448
chore: fix block timestamp stability test
thlorenz 3173865
chore: fix config test + disable auto airdrop test
thlorenz c6e425d
chore: first try fixing intent test via delegation of payer
thlorenz 530205d
chore: last failed attempt to fix intent tests, preparing them via es…
thlorenz 879cadd
chore: update progress
thlorenz 4d6c53d
fix: program re-clone after re-deploy
thlorenz 66f2309
chore: minor cleanup
thlorenz 0dea41d
chore: fix single test and update status on 9 more failing
thlorenz 61369e8
Merge branch 'master' into thlorenz/chainlink
thlorenz 8ad7de1
chore: fix issues caused by merge
thlorenz 8e85ea9
chore: adapt task scheduler
thlorenz cf0deb3
chore: fix context initialization
thlorenz 8e18721
chore: fix ix test merge conflicts
thlorenz 38b27f4
chore: fix phony section in Makefile
thlorenz 424d251
chore: fix compile error
thlorenz b67dd24
chore: add doc about running separate test suites
thlorenz 5039e8f
chore: update current integration test status
thlorenz 342c64e
chore: remove obsolete code step 1
thlorenz 6956b13
chore: remove magicblock-account-dumper
thlorenz 92d36e1
chore: remove magicblock-account-fetcher
thlorenz 84a69e7
chore: remove magicblock-account-updates
thlorenz 74be01b
chore: add moved bpf_loader_v1.rs
thlorenz 20bbc39
chore: remove magicblock-accounts-api
thlorenz 30226c8
chore: remove obsolete parts of magicblock-accounts step 1
thlorenz 12fc8b2
chore: weed out more obsolete code + conjunto use
thlorenz 10543f1
chore: remove obsolete magicblock-mutator
thlorenz 4fc0d58
refactor: clarifying names
Dodecahedr0x 9ae040e
fix: wait for tx to be processed
Dodecahedr0x 168a386
feat: use execution results
Dodecahedr0x 9de3825
feat: do not skip preflight
Dodecahedr0x 46e3436
docs: instructions on how to run task schedulder tests
thlorenz 600400b
chore: remove entry point hack for executables
thlorenz 8f110f4
chore: update progress
thlorenz 88c4a6e
Merge branch 'master' into thlorenz/chainlink
thlorenz 69a3946
Revert "chore: remove entry point hack for executables"
thlorenz bf40f37
chore: fix task tests running in parallel
thlorenz 44a9990
feat: properly enable/disable executable check via instructions
thlorenz 1d0f537
chore: fmt
thlorenz 7e1980f
chore: fix clippy
thlorenz 2541953
chore: run committor unit tests with preflight to show underlying issue
thlorenz 3d4ef66
chore: add details to failing committor tests
thlorenz ff870af
chore: add logs around account ensure
thlorenz 39e6eef
chore: disable tests requiring delegate account
thlorenz 39bc75a
chore: remove commented out account commit ticker code
thlorenz f33f131
chore: disable failing asserts for epoch schedule and supply tests fo…
thlorenz d49c0f4
chore: logs around fetch multiple accounts
thlorenz fdacb09
Revert "chore: logs around fetch multiple accounts"
thlorenz 5f90396
chore: ignore one more test failing due to delegate flag
thlorenz 10c0f46
chore: fmt integration tests
thlorenz 149bb3d
chore: test with no fail fast flag
thlorenz b924032
chore: temporarily disable intent tests assert that fails in CI
thlorenz 3637e10
chore: fix clippy issues in test-integration
thlorenz 74d9a5e
chore: use linux base64 compat flags only
thlorenz 1d457e4
chore: linux friendly way to create account json
thlorenz eda7c40
chore: remote account provider cleanup, logs, docs
thlorenz 984a1c2
tmp: add log when preparing transaction
thlorenz 0f91b12
tmp: also log on simulate transaction
thlorenz 27709b9
chore: update todos
thlorenz 33985ab
chore: allow more deviation for program size checks
thlorenz 1cacaf8
chore: check compiled program size
thlorenz c00fa4b
chore: calc prog size deviation via percentage
thlorenz 6b5f196
chore: minor fix
thlorenz f13fbd9
fix: test macros
thlorenz 6ac66b0
fix: macro issue
thlorenz 7c94e1f
chore: update progress notes
thlorenz 74a505c
chore: using node script to generate miniv2 json from so
thlorenz 15c5faf
chore: upgrade solana version used in CI
thlorenz b7a4566
chore: fix kill validator process name
thlorenz d5c03df
chore: wait for more slots after upgrade
thlorenz 6d855f3
chore: wait for a bit less slots
thlorenz 47d18a4
fix: use contains for blockhash check + fix tests
bmuddha 44089e2
fix: outdated assert messages
thlorenz 6d05032
chore: less brittle clone config test assertions
thlorenz 602ec16
chore: include task context in blacklisted accounts
thlorenz c5b1efb
chore: add details about undelegation race condition
thlorenz d609e63
chore: link issue about making LRU capacity configurable
thlorenz eb3cffd
chore: fix out of order sub updates
thlorenz 96c3098
chore: update progress
thlorenz 84b288d
Merge branch 'master' into thlorenz/chainlink
thlorenz 91a11e3
chore: fix invalid tmpdir assignments in ledger restore tests
thlorenz bb745fc
chore: update progress
thlorenz 79d3df3
fix: readme
thlorenz 9c04c09
chore: fmt
thlorenz dfe86ee
chore: remove progress tracking doc
thlorenz d0c8fe9
chore: wait a bit longer for program redeploy in cloning test
thlorenz 5a6a24e
fix: review related nitpick fixes
bmuddha 750e5be
chore: avoid Arc clone when possible
thlorenz 77aed49
chore: update solana-account crate
thlorenz 0826927
chore: remove unit tests that cannot run due to SVM changes
thlorenz 62db6bb
fix: don't allow rent epoch mutations in magic program
bmuddha 2cc4470
fix: intent tests
taco-paco b09e9c2
chore: remove duplicate log extraction code
thlorenz f955f74
chore: more resilient program upgrade test
thlorenz 0a9194c
chore: change warn to debug when starting with new ledger
thlorenz 9d604e9
fix: upgrade program test
thlorenz 79b4f97
chore: use proper deploy slot when cloning bpf_loader_v1 program
thlorenz 65628d9
fix: coderabbit post review fixes
bmuddha ffb5f99
chore: improve warning msg
thlorenz ba48d31
chore: not depending on joinset order when logging undelegate request…
thlorenz 19a65c4
chore: fmt
thlorenz 29ff17f
chore: improve escrow account cloning check
thlorenz 1a37103
chore: move not found message to debug
thlorenz 6b1a67e
chore: more efficient tuple extraction from vec
thlorenz 248ee2b
chore: wrap fetch cloner in Arc to avoid separate field cloning
thlorenz 23e9867
chore: address clippy issue
thlorenz cedcc16
chore: ran cargo fmt
bmuddha 7e6090e
Merge branch 'thlorenz/chainlink' into bmuddha/feat/multi-threaded-sc…
bmuddha a8876c2
chore: run cargo fmt
bmuddha e808bdb
Merge branch 'master' into bmuddha/feat/multi-threaded-scheduler
bmuddha 4e218fa
fix: post review fixes
bmuddha 76bb6e5
Merge branch 'master' into bmuddha/feat/multi-threaded-scheduler
bmuddha da6777d
fix: post merge cleanup
bmuddha 57ff682
fix: clippy
bmuddha 504bb1c
tests: integration tests for multi-threaded scheduler
bmuddha e19aee3
fix: add missing files
bmuddha f74bf51
fix: make executors configurable
bmuddha 8cb00c2
test: add integration tests to the scheduler
bmuddha 4584791
chore: fmt
bmuddha a3e0029
fix: clippy
bmuddha 55e62b5
Merge branch 'master' into bmuddha/feat/multi-threaded-scheduler
bmuddha b1e0933
fix: correct transaction queueing and rescheduling
bmuddha e077eab
fix: chore fmt
bmuddha c54053c
fix: typos and nits
bmuddha File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is never used?
On a side note, this weekend, I discovered
hashbrownbased on foldhash, which is faster on longer inputs than fxhash.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is used in the Lock contention tracking, since the keys are fixed (32 bytes) it's massively faster than siphash, and for a such a small values it's on par with foldhash, but that's an interesting algo to keep in mind, thanks.