[PERF]: parallelize applying log to segment types #3134
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
codetheweb
commented
Nov 14, 2024
codetheweb
commented
Nov 14, 2024
| @@ -787,11 +788,12 @@ impl<'me> LogMaterializer<'me> { | |||
|
|
|||
| // This needs to be public for testing | |||
| #[allow(async_fn_in_trait)] | |||
| pub trait SegmentWriter<'a> { | |||
There was a problem hiding this comment.
lifetime doesn't need to be on the struct
codetheweb
force-pushed
the
perf-parallelize-applying-log
branch
from
740db31 to
c40d5f2
Compare
codetheweb
commented
Nov 14, 2024
codetheweb
commented
Nov 15, 2024
| @@ -628,7 +630,7 @@ pub struct RecordSegmentReader<'me> { | |||
| user_id_to_id: BlockfileReader<'me, &'me str, u32>, | |||
| id_to_user_id: BlockfileReader<'me, u32, &'me str>, | |||
| id_to_data: BlockfileReader<'me, u32, DataRecord<'me>>, | |||
| curr_max_offset_id: Arc<AtomicU32>, | |||
| max_offset_id: u32, | |||
There was a problem hiding this comment.
I needed to change this because the shared state led to a bug in the log materialization logic when running multiple materializations in parallel (see comment in compact.rs). Readers should not expose mutable state.
codetheweb
force-pushed
the
perf-parallelize-applying-log
branch
from
88e8e29 to
bfd894f
Compare
rescrv
reviewed
Nov 15, 2024
rescrv
approved these changes
Nov 18, 2024
HammadB
reviewed
Nov 19, 2024
rust/worker/src/execution/operators/apply_log_to_segment_writer.rs
Outdated
Show resolved
Hide resolved
HammadB
reviewed
Nov 19, 2024
| panic!("Error creating record segment reader"); | ||
| } | ||
| RecordSegmentReaderCreationError::UserRecordNotFound(_) => { | ||
| _ => { |
There was a problem hiding this comment.
I think we should not panic here / can we use this as a moment to clean up
There was a problem hiding this comment.
this line is inside a test; were you referring to something else?
HammadB
approved these changes
Nov 19, 2024
There was a problem hiding this comment.
- Can we clean up the panics we bail out to in metadata segment?
- Can we expand the comment for the atomics dance that explains that we view materialization as idempotent and that it should result in the same results on each thread. Otherwise we can have horrible bugs.
- Is there anyway to aggressively test this atomics dance?
1 task
codetheweb
force-pushed
the
perf-parallelize-applying-log
branch
from
7be9113 to
cffe2a3
Compare
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Nov 20, 2024
codetheweb
force-pushed
the
perf-parallelize-applying-log
branch
2 times, most recently
from
e696cd7 to
851511a
Compare
codetheweb
force-pushed
the
feat-materialize-operator
branch
from
e30c087 to
cb43e94
Compare
codetheweb
force-pushed
the
perf-parallelize-applying-log
branch
from
92a0f28 to
f8ab4ac
Compare
rescrv
approved these changes
Dec 2, 2024
| @@ -309,7 +310,7 @@ impl CompactOrchestrator { | |||
| }, | |||
| }; | |||
|
|
|||
| self.num_write_tasks = partitions.len() as i32; | |||
| self.num_write_tasks = partitions.len() as i32 * 3; // 3 different segment types | |||
There was a problem hiding this comment.
This seems worth calling out as an assumption for when we add the index. I don't know the best way to log it. I'd do a TODO(feat), but we need to agree on what that would mean.
codetheweb
force-pushed
the
feat-materialize-operator
branch
from
cb43e94 to
5e19390
Compare
1 task
codetheweb
force-pushed
the
feat-materialize-operator
branch
3 times, most recently
from
1588bc4 to
b8d61ce
Compare
|
Superseded by #3359 (new branch was easier than rebasing as approach has changed). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Applies log updates to segment types in parallel rather than sequentially:
Pipelining flushes to S3/applying to blockfile is in the next PR in this stack.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustDocumentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?
n/a