Skip to content

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated #9580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jojochuang wants to merge 4 commits into
base: master
Choose a base branch
from
Open

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated #9580

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8d43d43
docs: Correct EC pipeline calculation and limits
jojochuang Aug 15, 2025
c4deea7
Limit to 120 characters
jojochuang Jan 4, 2026
f8060d9
Update text for monitoring number of pipelines.
jojochuang Jan 6, 2026
b7eefa2
Update based on Nicholas's comments.
jojochuang Jan 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions hadoop-hdds/docs/content/feature/multi-raft-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.

### Calculating Ratis Pipeline Limits

The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... FACTOR_THREE ...

Let's use ReplicationFactor.THREE, which is a client API.

... three properties ... that define the maximum

three configuration properties ... that limit the

number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... a cluster-wide level, datanode level, and metadata disk level, respectively.

a cluster-wide level and a datanode level.

SCM will create pipelines until the most restrictive limit is met.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace this sentence with

  • "The number of pipelines created by SCM is restricted by these limits."


1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
* **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove FACTOR_THREE since the code actually includes ReplicationFactor.ONE pipeline; see

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 123 to 128 in 4fd8360

if (pipelineNumberLimit > 0) {
int factorOnePipelineCount = pipelineStateManager
.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
return openPipelines >= allowedOpenPipelines;
}

across the entire cluster. This acts as a final cap on the total number of pipelines.
* **Default Value**: `0` (which means no global limit is enforced by default).
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... no global limit is enforced by default).

no global limit by default).


2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
* **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
every datanode. This is one of two ways to calculate a cluster-wide target.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the sentence "This is one of two ways to calculate a cluster-wide target." since the Dynamic Limit
is not used in RatisPipelineProvider.exceedPipelineNumberLimit(..).

* **Default Value**: `2`
* **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

    *   **Cluster-wide Limit Calculation**: If this property is set,
        the number of pipelines in the cluster is in addition limited by
        `(<this value> * <number of healthy datanodes>) / 3`.

(revised)


3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
* **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... This property is used ... is explicitly set to 0.

This property takes effect ... is not set to a positive number.

It calculates a dynamic limit for each datanode based on its available metadata disks.
* **Default Value**: `2`
* **Calculation**: The limit for each datanode is
`(<this value> * <number of metadata disks on that datanode>)`.
The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
Copy link
Contributor

@szetszwo szetszwo Jan 7, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this section since it seems not true (or I cannot find the code enforcing it). I do see that the value is used for filtering datanodes.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 104 to 131 in 4fd8360

private boolean exceedPipelineNumberLimit(RatisReplicationConfig replicationConfig) {
// Apply limits only for replication factor THREE
if (replicationConfig.getReplicationFactor() != ReplicationFactor.THREE) {
return false;
}
PipelineStateManager pipelineStateManager = getPipelineStateManager();
int totalActivePipelines = pipelineStateManager.getPipelines(replicationConfig).size();
int closedPipelines = pipelineStateManager.getPipelines(replicationConfig, PipelineState.CLOSED).size();
int openPipelines = totalActivePipelines - closedPipelines;
// Check per-datanode pipeline limit
if (maxPipelinePerDatanode > 0) {
int healthyNodeCount = getNodeManager()
.getNodeCount(NodeStatus.inServiceHealthy());
int allowedOpenPipelines = (maxPipelinePerDatanode * healthyNodeCount)
/ replicationConfig.getRequiredNodes();
return openPipelines >= allowedOpenPipelines;
}
// Check global pipeline limit
if (pipelineNumberLimit > 0) {
int factorOnePipelineCount = pipelineStateManager
.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
return openPipelines >= allowedOpenPipelines;
}
// No limits are set
return false;
}


#### How Limits are Applied

SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
**lowest value** is used as the final target for the number of open pipelines.

**Example (Dynamic Limit):**

Consider a cluster with **10 healthy datanodes**.
* **8 datanodes** have 4 metadata disks each.
* **2 datanodes** have 2 metadata disks each.

And the configuration is:
* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)

**Calculation Steps:**
1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
4. Compare with the global limit: `min(24, 30) = 24`

SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.

**Production Recommendation:**

For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the
actual number of pipelines aligns with your configured targets.
Copy link
Contributor

@ashishkumar50 ashishkumar50 Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think NumOpenPipelines metrics doesn't exist. We can either use admin command to see number of open pipelines or may use recon as well to see the open pipelines.

Copy link
Contributor Author

@jojochuang jojochuang Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, the SCM web UI has a section Pipeline Statistics, and it pulls the metrics from:

{
"name": "Hadoop:service=SCMPipelineManager,name=SCMPipelineManagerInfo",
"modelerType": "org.apache.hadoop.hdds.scm.pipeline.PipelineManagerImpl",
"PipelineInfo": [
  {
    "key": "CLOSED",
    "value": 0
  },
  {
    "key": "ALLOCATED",
    "value": 0
  },
  {
    "key": "OPEN",
    "value": 1
  },
  {
    "key": "DORMANT",
    "value": 0
  }
  ]
},


## How to Use
1. Configure Datanode metadata directories:
```xml
Expand Down