Skip to content

Commit eac623c

Browse files
committed
CLVM: Added notes on CLVM Improvements and optimizations
1 parent 49d5339 commit eac623c

1 file changed

Lines changed: 52 additions & 5 deletions

File tree

‎source/adminguide/storage/clvm.rst‎

Lines changed: 52 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ CLVM_NG (QCOW2 format):
6767

6868
- Slight performance overhead from QCOW2 layer
6969
- More complex storage stack
70-
- Bitmaps lost during volume migration - hence the next snapshot will be a full snapshot (but subsequent snapshots will be incremental again)
70+
- After a VM migration with storage, the snapshot chain is ended: the next snapshot is a full snapshot and a new chain starts. Moving a volume to another volume group, even across clusters, keeps its bitmaps and the next snapshot stays incremental
7171

7272
Key features
7373
~~~~~~~~~~~~
@@ -87,7 +87,25 @@ CLVM/CLVM_NG use `lvmlockd` and `sanlock` to provide exclusive locking of logica
8787
- **Active Volumes**: Locked exclusively on the host running the VM
8888
- **Inactive Volumes**: Can be in shared mode or deactivated
8989
- **Lock Transfer**: Automatic during VM migration
90-
- **Lock Tracking**: CloudStack tracks which host owns each volume lock
90+
- **Lock Tracking**: CloudStack records which host owns each volume lock as a hint and verifies it against the real state, see *Finding the lock holder*
91+
92+
Finding the lock holder
93+
^^^^^^^^^^^^^^^^^^^^^^^
94+
95+
CloudStack records each volume's lock holder (the ``clvmLockHostId`` volume detail) only as a hint, and verifies it when an operation needs the holder:
96+
97+
#. It asks the recorded host whether the volume is active there.
98+
#. If not, it reads the volume's sanlock lease from the hidden ``lvmlock`` volume of the volume group, which any host can read even when the holder is down. The lease names the owner's sanlock ``host_id``, which is mapped to a CloudStack host through the ``host.sanlock.id`` host detail and confirmed with that host.
99+
#. If the lease is free, or its owner has restarted since it took the lock, the volume is activated exclusively on a chosen host, preferring the destination of the operation.
100+
#. Only when the lease cannot be read, or the owner cannot be mapped to exactly one host, CloudStack asks every host in parallel (see ``clvm.lock.query.threads``).
101+
102+
Right after a host failure or restart, an activation can be refused temporarily: ``locked by other host`` until the old holder's lease expires (140 seconds by default), and ``lock start in progress`` while the lockspace starts (about two minutes). These are retried until ``clvm.lock.recovery.retry.timeout``.
103+
104+
Requirements:
105+
106+
- Each host needs a unique numeric ``host_id`` in ``/etc/lvm/lvmlocal.conf`` (1 to 2000), the same for all volume groups of the host, and must have joined the lockspace of every volume group of its pools.
107+
- The KVM agent reports the ``host_id`` as ``host.sanlock.id`` each time it connects. After changing ``host_id``, make the host rejoin the lockspace (reboot it), then restart the agent.
108+
- A host without ``host.sanlock.id`` is logged with a warning, and a duplicate raises an alert. In both cases lookups ask every host.
91109

92110
Lightweight Migration
93111
^^^^^^^^^^^^^^^^^^^^^
@@ -201,6 +219,35 @@ clvm.secure.zero.fill
201219
- **Recommendation:** Enable for environments with strict security/compliance requirements (PCI-DSS, HIPAA). Disable for performance-critical environments where deletion speed matters more than data security
202220
- **Use Case:** Production environments handling sensitive data, multi-tenant environments with strict data isolation requirements
203221

222+
clvm.lock.recovery.retry.timeout
223+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
224+
225+
- **Type:** Integer
226+
- **Default:** `180`
227+
- **Scope:** Global
228+
- **Category:** Advanced
229+
- **Description:** Maximum time in seconds, counted from the start of the lock holder lookup, to keep retrying the exclusive activation of a volume that is refused only temporarily: the previous holder's lease has not expired yet, or the lockspace of the volume group is still starting on the host. `0` disables the retry.
230+
- **Recommendation:** The lease expires 8 x `io_timeout` + 60 seconds after the holder stops renewing it (140 seconds with the default lvmlockd sanlock `io_timeout` of 10). If lvmlockd runs with a different `--sanlock-timeout`, set this to at least that value plus about 40 seconds.
231+
232+
clvm.lock.recovery.retry.interval
233+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
234+
235+
- **Type:** Integer
236+
- **Default:** `10`
237+
- **Scope:** Global
238+
- **Category:** Advanced
239+
- **Description:** Time in seconds between retries of an activation that is refused only temporarily.
240+
241+
clvm.lock.query.threads
242+
^^^^^^^^^^^^^^^^^^^^^^^
243+
244+
- **Type:** Integer
245+
- **Default:** `5`
246+
- **Scope:** Global
247+
- **Category:** Advanced
248+
- **Description:** Number of threads a management server uses to send CLVM lock commands to several hosts in parallel: the fallback lookup that asks every host, and the deactivation of a template volume on all hosts before it is removed.
249+
- **Propagation:** Requires a management server restart.
250+
204251
For incremental snapshots to work with CLVM_NG enabled the `kvm.incremental.snapshot` global setting must also be set to `true`.
205252

206253
VM snapshots are currently not supported on CLVM/CLVM_NG volumes.
@@ -219,7 +266,7 @@ systemctl enable sanlock --now
219266
systemctl enable lvmlockd --now
220267

221268
# Update /etc/lvm/lvm.conf to set use_lvmlockd = 1
222-
# Update /etc/lvm/lvmlocal.conf set host_id to a unique value for each host (e.g. hostname or IP or ID (1, 2, 3...))
269+
# Update /etc/lvm/lvmlocal.conf set host_id to a unique number (1 to 2000) for each host (e.g. 1, 2, 3...); it must be numeric and is used for every volume group of the host
223270
```
224271

225272
Just on one of the hosts, initialize the shared physical volume (PV) that will be used for CLVM storage pools and start the VG in shared mode:
@@ -238,6 +285,6 @@ vgchange lockstart sharedvg # Start the VG in shared mode
238285

239286
.. note::
240287
It is reccommended to add CLVM / CLVM_NG storage pools as a Cluster scoped resource, so as to limited number of hosts that can access the pool for optimal performance and reliability.
241-
Due to the lack of support from `lvm` to provide a centralized way to query which host has the lock for a given volume, CloudStack relies on tracking this information in the database. However, if there is a drift in this information, for example due to a host failure, CloudStack will attempt to query each host for the lock status of the volume. This can lead to performance issues if there are a large number of hosts in the cluster as each host will need to be queried for each volume.
288+
`lvm` has no centralized way to query which host has the lock for a given volume, so CloudStack tracks it in the database and verifies it against the sanlock lease of the volume (see *Finding the lock holder*). Only when the lease cannot be read or mapped to a single host does CloudStack query each host of the pool, which is slower with a large number of hosts. This is another reason to add the pool as a cluster scoped resource.
242289

243-
VM snapshots and incremental volume snapshot are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0.
290+
VM snapshots are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0. Incremental volume snapshots are supported on CLVM_NG only, and need the ``kvm.incremental.snapshot`` global setting set to ``true``.

0 commit comments

Comments
 (0)