Skip to content

check for active MSses before starting DB upgrade #12140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: 4.20
Choose a base branch
from
+342 −34
Draft

check for active MSses before starting DB upgrade #12140

Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
edcbbde
check for active MSses before starting DB upgrade
Nov 26, 2025
48af60a
fix sql
DaanHoogland Nov 27, 2025
46a6f7a
none may be up!
DaanHoogland Nov 27, 2025
2135e1e
Apply suggestions from code review
DaanHoogland Nov 27, 2025
42414f4
make condition not fatal
Dec 1, 2025
6d9a6c5
change moment of abort
Dec 2, 2025
0315005
more hard exit
Dec 2, 2025
335f144
some unit tests
Dec 2, 2025
e79dacb
imports
Dec 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 54 additions & 24 deletions engine/schema/src/main/java/com/cloud/upgrade/DatabaseUpgradeChecker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,10 @@
import com.cloud.utils.crypt.DBEncryptionUtil;
import com.cloud.utils.db.GlobalLock;
import com.cloud.utils.db.ScriptRunner;
import com.cloud.utils.db.Transaction;
import com.cloud.utils.db.TransactionCallback;
import com.cloud.utils.db.TransactionLegacy;
import com.cloud.utils.db.TransactionStatus;
import com.cloud.utils.exception.CloudRuntimeException;
import com.google.common.annotations.VisibleForTesting;

Expand Down Expand Up @@ -448,39 +451,66 @@ public void check() {
throw new CloudRuntimeException("Unable to acquire lock to check for database integrity.");
}

try {
initializeDatabaseEncryptors();
if (isStandalone()) {
doUpgrades(lock);
}
} finally {
lock.releaseRef();
}
}
private boolean isStandalone() throws CloudRuntimeException {
return Transaction.execute(new TransactionCallback<>() {
@Override
public Boolean doInTransaction(TransactionStatus status) {
String sql = "SELECT COUNT(*) FROM `cloud`.`mshost` WHERE `state` = 'UP'";
try (Connection conn = TransactionLegacy.getStandaloneConnection();
PreparedStatement pstmt = conn.prepareStatement(sql);
ResultSet rs = pstmt.executeQuery()) {
if (rs.next()) {
int count = rs.getInt(1);
return count == 0;
Copy link

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here has a subtle issue. The query counts management servers with state = 'UP', and returns true (standalone) when count is 0. However, this check is being performed by a management server that is currently running.

This means:

  1. If this MS has already registered itself in the mshost table as 'UP', the count would be at least 1, and it would never be considered standalone.
  2. If this MS hasn't registered yet, the count might be 0 even in a clustered environment if other MSes are down or not yet started.

The check should likely exclude the current management server from the count, or check for count <= 1 instead of count == 0, to properly detect if this is the only running MS.

Suggested change
return count == 0;
return count <= 1;

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

@DaanHoogland DaanHoogland Dec 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one valid issue is that an MS performing an upgrade is not yet registered as UP. so this fix is better than what we have but certainly not watertight yet.

}
} catch (SQLException e) {
String errorMessage = "Unable to check if the management server is running in standalone mode.";
LOGGER.error(errorMessage, e);
return false;
}
return true;
}
});
}

final CloudStackVersion dbVersion = CloudStackVersion.parse(_dao.getCurrentVersion());
final String currentVersionValue = this.getClass().getPackage().getImplementationVersion();
private void doUpgrades(GlobalLock lock) {
try {
initializeDatabaseEncryptors();

if (StringUtils.isBlank(currentVersionValue)) {
return;
}
final CloudStackVersion dbVersion = CloudStackVersion.parse(_dao.getCurrentVersion());
final String currentVersionValue = this.getClass().getPackage().getImplementationVersion();

String csVersion = SystemVmTemplateRegistration.parseMetadataFile();
final CloudStackVersion sysVmVersion = CloudStackVersion.parse(csVersion);
final CloudStackVersion currentVersion = CloudStackVersion.parse(currentVersionValue);
SystemVmTemplateRegistration.CS_MAJOR_VERSION = String.valueOf(sysVmVersion.getMajorRelease()) + "." + String.valueOf(sysVmVersion.getMinorRelease());
SystemVmTemplateRegistration.CS_TINY_VERSION = String.valueOf(sysVmVersion.getPatchRelease());
if (StringUtils.isBlank(currentVersionValue)) {
return;
}

LOGGER.info("DB version = " + dbVersion + " Code Version = " + currentVersion);
String csVersion = SystemVmTemplateRegistration.parseMetadataFile();
final CloudStackVersion sysVmVersion = CloudStackVersion.parse(csVersion);
final CloudStackVersion currentVersion = CloudStackVersion.parse(currentVersionValue);
SystemVmTemplateRegistration.CS_MAJOR_VERSION = String.valueOf(sysVmVersion.getMajorRelease()) + "." + String.valueOf(sysVmVersion.getMinorRelease());
SystemVmTemplateRegistration.CS_TINY_VERSION = String.valueOf(sysVmVersion.getPatchRelease());

if (dbVersion.compareTo(currentVersion) > 0) {
throw new CloudRuntimeException("Database version " + dbVersion + " is higher than management software version " + currentVersionValue);
}
LOGGER.info("DB version = " + dbVersion + " Code Version = " + currentVersion);

if (dbVersion.compareTo(currentVersion) == 0) {
LOGGER.info("DB version and code version matches so no upgrade needed.");
return;
}
if (dbVersion.compareTo(currentVersion) > 0) {
throw new CloudRuntimeException("Database version " + dbVersion + " is higher than management software version " + currentVersionValue);
}

upgrade(dbVersion, currentVersion);
} finally {
lock.unlock();
if (dbVersion.compareTo(currentVersion) == 0) {
LOGGER.info("DB version and code version matches so no upgrade needed.");
return;
}

upgrade(dbVersion, currentVersion);
} finally {
lock.releaseRef();
lock.unlock();
}
}

Expand Down
Loading