You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit eac623c
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: source/adminguide/storage/clvm.rst
+52-5Lines changed: 52 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -67,7 +67,7 @@ CLVM_NG (QCOW2 format):
67
67
68
68
- Slight performance overhead from QCOW2 layer
69
69
- More complex storage stack
70
-
- Bitmaps lost during volume migration - hence the next snapshot will be a full snapshot (but subsequent snapshots will be incremental again)
70
+
- After a VM migration with storage, the snapshot chain is ended: the next snapshot is a full snapshot and a new chain starts. Moving a volume to another volume group, even across clusters, keeps its bitmaps and the next snapshot stays incremental
71
71
72
72
Key features
73
73
~~~~~~~~~~~~
@@ -87,7 +87,25 @@ CLVM/CLVM_NG use `lvmlockd` and `sanlock` to provide exclusive locking of logica
87
87
- **Active Volumes**: Locked exclusively on the host running the VM
88
88
- **Inactive Volumes**: Can be in shared mode or deactivated
89
89
- **Lock Transfer**: Automatic during VM migration
90
-
- **Lock Tracking**: CloudStack tracks which host owns each volume lock
90
+
- **Lock Tracking**: CloudStack records which host owns each volume lock as a hint and verifies it against the real state, see *Finding the lock holder*
91
+
92
+
Finding the lock holder
93
+
^^^^^^^^^^^^^^^^^^^^^^^
94
+
95
+
CloudStack records each volume's lock holder (the ``clvmLockHostId`` volume detail) only as a hint, and verifies it when an operation needs the holder:
96
+
97
+
#. It asks the recorded host whether the volume is active there.
98
+
#. If not, it reads the volume's sanlock lease from the hidden ``lvmlock`` volume of the volume group, which any host can read even when the holder is down. The lease names the owner's sanlock ``host_id``, which is mapped to a CloudStack host through the ``host.sanlock.id`` host detail and confirmed with that host.
99
+
#. If the lease is free, or its owner has restarted since it took the lock, the volume is activated exclusively on a chosen host, preferring the destination of the operation.
100
+
#. Only when the lease cannot be read, or the owner cannot be mapped to exactly one host, CloudStack asks every host in parallel (see ``clvm.lock.query.threads``).
101
+
102
+
Right after a host failure or restart, an activation can be refused temporarily: ``locked by other host`` until the old holder's lease expires (140 seconds by default), and ``lock start in progress`` while the lockspace starts (about two minutes). These are retried until ``clvm.lock.recovery.retry.timeout``.
103
+
104
+
Requirements:
105
+
106
+
- Each host needs a unique numeric ``host_id`` in ``/etc/lvm/lvmlocal.conf`` (1 to 2000), the same for all volume groups of the host, and must have joined the lockspace of every volume group of its pools.
107
+
- The KVM agent reports the ``host_id`` as ``host.sanlock.id`` each time it connects. After changing ``host_id``, make the host rejoin the lockspace (reboot it), then restart the agent.
108
+
- A host without ``host.sanlock.id`` is logged with a warning, and a duplicate raises an alert. In both cases lookups ask every host.
91
109
92
110
Lightweight Migration
93
111
^^^^^^^^^^^^^^^^^^^^^
@@ -201,6 +219,35 @@ clvm.secure.zero.fill
201
219
- **Recommendation:** Enable for environments with strict security/compliance requirements (PCI-DSS, HIPAA). Disable for performance-critical environments where deletion speed matters more than data security
202
220
- **Use Case:** Production environments handling sensitive data, multi-tenant environments with strict data isolation requirements
203
221
222
+
clvm.lock.recovery.retry.timeout
223
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
224
+
225
+
- **Type:** Integer
226
+
- **Default:** `180`
227
+
- **Scope:** Global
228
+
- **Category:** Advanced
229
+
- **Description:** Maximum time in seconds, counted from the start of the lock holder lookup, to keep retrying the exclusive activation of a volume that is refused only temporarily: the previous holder's lease has not expired yet, or the lockspace of the volume group is still starting on the host. `0` disables the retry.
230
+
- **Recommendation:** The lease expires 8 x `io_timeout` + 60 seconds after the holder stops renewing it (140 seconds with the default lvmlockd sanlock `io_timeout` of 10). If lvmlockd runs with a different `--sanlock-timeout`, set this to at least that value plus about 40 seconds.
231
+
232
+
clvm.lock.recovery.retry.interval
233
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
234
+
235
+
- **Type:** Integer
236
+
- **Default:** `10`
237
+
- **Scope:** Global
238
+
- **Category:** Advanced
239
+
- **Description:** Time in seconds between retries of an activation that is refused only temporarily.
240
+
241
+
clvm.lock.query.threads
242
+
^^^^^^^^^^^^^^^^^^^^^^^
243
+
244
+
- **Type:** Integer
245
+
- **Default:** `5`
246
+
- **Scope:** Global
247
+
- **Category:** Advanced
248
+
- **Description:** Number of threads a management server uses to send CLVM lock commands to several hosts in parallel: the fallback lookup that asks every host, and the deactivation of a template volume on all hosts before it is removed.
249
+
- **Propagation:** Requires a management server restart.
250
+
204
251
For incremental snapshots to work with CLVM_NG enabled the `kvm.incremental.snapshot` global setting must also be set to `true`.
205
252
206
253
VM snapshots are currently not supported on CLVM/CLVM_NG volumes.
# Update /etc/lvm/lvm.conf to set use_lvmlockd = 1
222
-
# Update /etc/lvm/lvmlocal.conf set host_id to a unique value for each host (e.g. hostname or IP or ID (1, 2, 3...))
269
+
# Update /etc/lvm/lvmlocal.conf set host_id to a unique number (1 to 2000) for each host (e.g. 1, 2, 3...); it must be numeric and is used for every volume group of the host
223
270
```
224
271
225
272
Just on one of the hosts, initialize the shared physical volume (PV) that will be used for CLVM storage pools and start the VG in shared mode:
@@ -238,6 +285,6 @@ vgchange lockstart sharedvg # Start the VG in shared mode
238
285
239
286
.. note::
240
287
It is reccommended to add CLVM / CLVM_NG storage pools as a Cluster scoped resource, so as to limited number of hosts that can access the pool for optimal performance and reliability.
241
-
Due to the lack of support from `lvm` to provide a centralized way to query which host has the lock for a given volume, CloudStack relies on tracking this information in the database. However, if there is a drift in this information, for example due to a host failure, CloudStack will attempt to query each host for the lock status of the volume. This can lead to performance issues if there are a large number of hosts in the cluster as each host will need to be queried for each volume.
288
+
`lvm` has no centralized way to query which host has the lock for a given volume, so CloudStack tracks it in the database and verifies it against the sanlock lease of the volume (see *Finding the lock holder*). Only when the lease cannot be read or mapped to a single host does CloudStack query each host of the pool, which is slower with a large number of hosts. This is another reason to add the pool as a cluster scoped resource.
242
289
243
-
VM snapshots and incremental volume snapshot are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0.
290
+
VM snapshots are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0. Incremental volume snapshots are supported on CLVM_NG only, and need the ``kvm.incremental.snapshot`` global setting set to ``true``.
0 commit comments