Skip to content

Commit

Permalink
[Improvement](fragment) Use partitioned hash map to manage contexts (a…
Browse files Browse the repository at this point in the history
…pache#46235)

Contexts in `fragment_mgr` are managed by a global map and accessed by
multiple threads concurrently with a global lock. It introduced a
obvious overhead. To solve it , this PR use a partitioned hash table to
optimize the global lock.
  • Loading branch information
Gabriel39 committed Jan 3, 2025
1 parent b4c36d3 commit 9ced701
Show file tree
Hide file tree
Showing 4 changed files with 342 additions and 228 deletions.
1 change: 1 addition & 0 deletions be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ DEFINE_Int32(check_consistency_worker_count, "1");
DEFINE_Int32(upload_worker_count, "1");
// the count of thread to download
DEFINE_Int32(download_worker_count, "1");
DEFINE_Int32(num_query_ctx_map_partitions, "128");
// the count of thread to make snapshot
DEFINE_Int32(make_snapshot_worker_count, "5");
// the count of thread to release snapshot
Expand Down
2 changes: 2 additions & 0 deletions be/src/common/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -1350,6 +1350,8 @@ DECLARE_Int32(spill_io_thread_pool_queue_size);

DECLARE_mBool(check_segment_when_build_rowset_meta);

DECLARE_Int32(num_query_ctx_map_partitions);

DECLARE_mBool(enable_s3_rate_limiter);
DECLARE_mInt64(s3_get_bucket_tokens);
DECLARE_mInt64(s3_get_token_per_second);
Expand Down
Loading

0 comments on commit 9ced701

Please sign in to comment.