Webserver.service health-checks keep failing for many vrouters #12170

mwaag · 2025-12-01T20:53:34Z

mwaag
Dec 1, 2025

Cloudstack: 4.18.2.4 (in preparation to upgrade to a newer version)
Hypervisor: VMWare ESX 7.0.3
Management-Server OS: Ubuntu 24.04.3 LTS
vRouter version: 4.18.1
vRouter Software version: 4.18.2.4

Hi there,

we have a lot of virtual routers raising an alert because of a failing health check. The health check in question is always 'webserver.service' which keeps failing on many vrouters creating a lot of alerts :

We find this in managment.log as well:
2025-12-01 21:16:52,482 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-860476b6) (logid:7813569b) Health checks failed: 1 failing checks on router r-178445-VM / 826d1cc5-3a3e-486c-9731-5ea140d9f18c. Checking failed health checks to see if router needs recreate

We usually find nothing in cloud.log on the vrouter, tried to grep for 'health' 'fail' 'check' 'apache2' 'webserver', zero matches.

Systemctl-output:
root@r-178445-VM:# systemctl is-active apache2
failed
root@r-178445-VM:# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2025-12-01 20:21:10 UTC; 34s ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 115818 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
CPU: 11ms

Dec 01 20:21:10 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:10 r-178445-VM apachectl[115821]: no listening sockets available, shutting down
Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00015: Unable to open logs
Dec 01 20:21:10 r-178445-VM apachectl[115818]: Action 'start' failed.
Dec 01 20:21:10 r-178445-VM apachectl[115818]: The Apache error log may have more information.
Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:10 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.

apache2-journal:
Dec 01 20:18:11 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:18:11 r-178445-VM apachectl[115680]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:18:11 r-178445-VM apachectl[115680]: no listening sockets available, shutting down
Dec 01 20:18:11 r-178445-VM apachectl[115680]: AH00015: Unable to open logs
Dec 01 20:18:11 r-178445-VM apachectl[115677]: Action 'start' failed.
Dec 01 20:18:11 r-178445-VM apachectl[115677]: The Apache error log may have more information.
Dec 01 20:18:11 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:18:11 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:18:11 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.
Dec 01 20:21:06 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:06 r-178445-VM apachectl[115773]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:06 r-178445-VM apachectl[115773]: no listening sockets available, shutting down
Dec 01 20:21:06 r-178445-VM apachectl[115773]: AH00015: Unable to open logs
Dec 01 20:21:06 r-178445-VM apachectl[115770]: Action 'start' failed.
Dec 01 20:21:06 r-178445-VM apachectl[115770]: The Apache error log may have more information.
Dec 01 20:21:06 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:06 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:06 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.
Dec 01 20:21:07 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:07 r-178445-VM apachectl[115785]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:07 r-178445-VM apachectl[115785]: no listening sockets available, shutting down
Dec 01 20:21:07 r-178445-VM apachectl[115785]: AH00015: Unable to open logs
Dec 01 20:21:07 r-178445-VM apachectl[115782]: Action 'start' failed.
Dec 01 20:21:07 r-178445-VM apachectl[115782]: The Apache error log may have more information.
Dec 01 20:21:07 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:07 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:07 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.
Dec 01 20:21:08 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:08 r-178445-VM apachectl[115797]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:08 r-178445-VM apachectl[115797]: no listening sockets available, shutting down
Dec 01 20:21:08 r-178445-VM apachectl[115797]: AH00015: Unable to open logs
Dec 01 20:21:08 r-178445-VM apachectl[115794]: Action 'start' failed.
Dec 01 20:21:08 r-178445-VM apachectl[115794]: The Apache error log may have more information.
Dec 01 20:21:08 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:08 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:08 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.
Dec 01 20:21:09 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:09 r-178445-VM apachectl[115809]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:09 r-178445-VM apachectl[115809]: no listening sockets available, shutting down
Dec 01 20:21:09 r-178445-VM apachectl[115809]: AH00015: Unable to open logs
Dec 01 20:21:09 r-178445-VM apachectl[115806]: Action 'start' failed.
Dec 01 20:21:09 r-178445-VM apachectl[115806]: The Apache error log may have more information.
Dec 01 20:21:09 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:09 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:09 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.
Dec 01 20:21:10 r-178445-VM systemd[1]: Starting The Apache HTTP Server...
Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Dec 01 20:21:10 r-178445-VM apachectl[115821]: no listening sockets available, shutting down
Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00015: Unable to open logs
Dec 01 20:21:10 r-178445-VM apachectl[115818]: Action 'start' failed.
Dec 01 20:21:10 r-178445-VM apachectl[115818]: The Apache error log may have more information.
Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'.
Dec 01 20:21:10 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server.

Syntax seems to be ok:
root@r-178445-VM:~# apache2ctl configtest
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Syntax OK

Any other processes bound to port 80/443?
root@r-178445-VM:# ss -tulpn | grep -E ':80|:443'
(no output)
root@r-178445-VM:# lsof -nP -iTCP:80 -sTCP:LISTEN
(no output)

Checking rights on Log-dir:
root@r-178445-VM:# ls -ld /var/log/apache2
drwxr-x--- 2 root adm 4096 Mar 18 2025 /var/log/apache2
root@r-178445-VM:# test -w /var/log/apache2 && echo "log-dir writable" || echo "log-dir NOT writable"
log-dir writable

root@r-178445-VM:# uptime
20:31:29 up 265 days, 5:05, 1 user, load average: 0.00, 0.00, 0.00

Currently we have 52 of 398 vRouters in total showing this symptom.

We tried recreate one vrouter with vpc-restart with cleanup, but unfortunately with the same result.

Can we do some more troubleshooting? Unfortunately this keeps us away from filling alert.email.addresses because the traffic is just to high as every router send an alarm every 10 minutes or so.

Any hints, how we can further troubleshoot / eleminate this issues?

DaanHoogland · 2025-12-02T08:01:01Z

DaanHoogland
Dec 2, 2025
Collaborator

@mwaag, there were fixes on the healthchecks in newer versions. For instance, the webservice check would fail if a tier in a vpc would go from implemented to allocated due to no more VMs in the tier. Have a look at #10710 which tries to address an undlying fundemental problem that healthchecks may fail or succeed but also may not be possible at times.

3 replies

mwaag Dec 9, 2025
Author

Alright, will check, if this applies to our situation as well. Thank you!

mwaag Dec 9, 2025
Author

Ok. I'm still not completely sure, if the mentioned PR will address this particular problem, but if i followed the conversation in this PR correctly, the oldest release, that has this enhancement is 4.20.2, right?

DaanHoogland Dec 9, 2025
Collaborator

it is marked for milestone 4.22.0 @mwaag . usually that is correct (though a manual setting).

weizhouapache · 2025-12-16T12:24:23Z

weizhouapache
Dec 16, 2025
Collaborator

any error in /var/log/apache2/error.log ?

1 reply

mwaag Dec 16, 2025
Author

Hi Wei,
here is the whole last error.log from virtual router r-178445-VM:

[Tue Mar 18 10:40:23.373023 2025] [ssl:warn] [pid 282422:tid 140515846380864] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Tue Mar 18 10:40:23.399400 2025] [ssl:warn] [pid 282476:tid 140515846380864] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Tue Mar 18 10:40:23.400244 2025] [mpm_event:notice] [pid 282476:tid 140515846380864] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1w configured -- resuming normal operations
[Tue Mar 18 10:40:23.400265 2025] [core:notice] [pid 282476:tid 140515846380864] AH00094: Command line: '/usr/sbin/apache2'
[Wed Mar 19 09:18:04.256224 2025] [mpm_event:notice] [pid 282476:tid 140515846380864] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Mar 19 09:32:56.338027 2025] [ssl:warn] [pid 325671:tid 139644216470848] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Wed Mar 19 09:32:56.348395 2025] [ssl:warn] [pid 325699:tid 139644216470848] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Wed Mar 19 09:32:56.349297 2025] [mpm_event:notice] [pid 325699:tid 139644216470848] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1w configured -- resuming normal operations
[Wed Mar 19 09:32:56.349341 2025] [core:notice] [pid 325699:tid 139644216470848] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 28 14:33:54.499595 2025] [mpm_event:notice] [pid 325699:tid 139644216470848] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Oct 29 07:50:40.327203 2025] [ssl:warn] [pid 2198790:tid 140023026781504] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Wed Oct 29 07:50:40.384402 2025] [ssl:warn] [pid 2198821:tid 140023026781504] AH01909: vpcrouter.cs428cloud.internal:443:0 server certificate does NOT include an ID which matches the server name
[Wed Oct 29 07:50:40.385319 2025] [mpm_event:notice] [pid 2198821:tid 140023026781504] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1w configured -- resuming normal operations
[Wed Oct 29 07:50:40.385347 2025] [core:notice] [pid 2198821:tid 140023026781504] AH00094: Command line: '/usr/sbin/apache2'
[Thu Nov 06 12:52:29.333959 2025] [mpm_event:notice] [pid 2198821:tid 140023026781504] AH00492: caught SIGWINCH, shutting down gracefully

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Webserver.service health-checks keep failing for many vrouters #12170

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Webserver.service health-checks keep failing for many vrouters #12170

Uh oh!

mwaag Dec 1, 2025

Replies: 2 comments · 4 replies

Uh oh!

DaanHoogland Dec 2, 2025 Collaborator

Uh oh!

mwaag Dec 9, 2025 Author

Uh oh!

mwaag Dec 9, 2025 Author

Uh oh!

DaanHoogland Dec 9, 2025 Collaborator

Uh oh!

weizhouapache Dec 16, 2025 Collaborator

Uh oh!

mwaag Dec 16, 2025 Author

mwaag
Dec 1, 2025

Replies: 2 comments 4 replies

DaanHoogland
Dec 2, 2025
Collaborator

mwaag Dec 9, 2025
Author

mwaag Dec 9, 2025
Author

DaanHoogland Dec 9, 2025
Collaborator

weizhouapache
Dec 16, 2025
Collaborator

mwaag Dec 16, 2025
Author