Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default mDNS settings results in network flooding #5435

Open
sle118 opened this issue Nov 19, 2024 · 18 comments
Open

Default mDNS settings results in network flooding #5435

sle118 opened this issue Nov 19, 2024 · 18 comments
Labels

Comments

@sle118
Copy link

sle118 commented Nov 19, 2024

Describe the issue you are experiencing

The default NetworkManager configuration sets mDNS to yes which, in network having multiple vlans and a reflector (e.g. on a Unifi network), result in a flood of mDNS requests originating from the supervisor's systemd-resolved. I believe the network configuration originates from this line:
https://github.com/home-assistant/supervisor/blame/4e0de93096dc69c257056b36eff0c19b47006850/supervisor/dbus/network/setting/generate.py#L167

Typical flooding would look like this

2024-11-19 15:09:05.941 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant12746' to 'homeassistant12749'.
2024-11-19 15:09:06.214 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12749\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:09:06.214 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12749\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant12749.local
2024-11-19 15:09:06.214 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12749.local IN A 192.168.10.247
2024-11-19 15:09:06.214 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant12749' to 'homeassistant12758'.
2024-11-19 15:09:06.450 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12758\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:09:06.451 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12758\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant12758.local
2024-11-19 15:09:06.451 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12758.local IN A 192.168.10.247
2024-11-19 15:09:06.451 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant12758' to 'homeassistant12764'.
2024-11-19 15:09:06.704 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12764\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant12764.local
2024-11-19 15:09:06.704 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12764\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:09:06.704 homeassistant systemd-resolved[416]: Detected conflict on homeassistant12764.local IN A 192.168.10.247
2024-11-19 15:09:06.704 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant12764' to 'homeassistant12767'.

The issue was also reported under the operating system project, but got closed. Nevertheless, a long discussion continued after the issue got closed, with some hints on how to fix the issue.
home-assistant/operating-system#1163 (comment)

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. Run Home-Assistant on a network with multiple vlans
  2. Setup a mDNS reflector to reflect mDNS packets across vlans
  3. Open the system->logs->host (http://homeassistant.local:8123/config/logs?provider=host)
  4. Watch for systemd-resolved name collision entries in the log
    ...

Anything in the Supervisor logs that might be useful for us?

2024-11-19 15:00:45.209 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11362' to 'homeassistant11363'.
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11363\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11363.local
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11363\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11363\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11363\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11363.local
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11363.local IN A 192.168.10.247
2024-11-19 15:00:45.281 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11363' to 'homeassistant11371'.
2024-11-19 15:00:45.453 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11371\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11371.local
2024-11-19 15:00:45.453 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11371\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.453 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11371.local IN A 192.168.10.247
2024-11-19 15:00:45.453 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11371' to 'homeassistant11381'.
2024-11-19 15:00:45.515 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11381\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.515 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11381\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11381.local
2024-11-19 15:00:45.515 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11381.local IN A 192.168.10.247
2024-11-19 15:00:45.515 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11381' to 'homeassistant11382'.
2024-11-19 15:00:45.734 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11382\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.734 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11382\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11382.local
2024-11-19 15:00:45.735 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11382.local IN A 192.168.10.247
2024-11-19 15:00:45.735 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11382' to 'homeassistant11386'.
2024-11-19 15:00:45.812 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11386\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11386.local
2024-11-19 15:00:45.812 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11386\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.812 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11386.local IN A 192.168.10.247
2024-11-19 15:00:45.812 homeassistant systemd-resolved[416]: Hostname conflict, changing published hostname from 'homeassistant11386' to 'homeassistant11395'.
2024-11-19 15:00:45.957 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11395\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN TXT ""
2024-11-19 15:00:45.957 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11395\032\091431cf3e1addd46f5b14185b1bea39b0e\093._workstation._tcp.local IN SRV 0 0 0 homeassistant11395.local
2024-11-19 15:00:45.957 homeassistant systemd-resolved[416]: Detected conflict on homeassistant11395.local IN A 192.168.10.247
202

System Health information

System Information

version core-2024.11.2
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.54-haos
arch x86_64
timezone America/Toronto
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
HACS Data ok
GitHub API Calls Remaining 5000
Installed Version 2.0.1
Stage running
Available Repositories 1536
Downloaded Repositories 9
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 13.2
update_channel stable
supervisor_version supervisor-2024.11.2
agent_version 1.6.0
docker_version 27.2.0
disk_total 205.1 GB
disk_used 52.1 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization oracle
board ova
supervisor_api ok
version_api ok
installed_addons File editor (5.8.0), Terminal & SSH (9.15.0), Mosquitto broker (6.4.1), AppDaemon (0.16.7), SQLite Web (4.2.2), Z-Wave JS (0.9.0), ESPHome (2024.10.3), phpMyAdmin (0.10.0), rtl_433 MQTT Auto Discovery (0.8.2), Samba share (12.3.2), Z-Wave JS UI (3.17.0), MariaDB (2.7.1), Zigbee2MQTT (1.41.0-1), MQTT Explorer (browser-1.0.3), Node-RED (18.1.1), Studio Code Server (5.17.3), Frigate Proxy (1.5)
Dashboards
dashboards 3
resources 1
error /config/ui-lovelace.yaml not found
views 20
mode yaml
Recorder
oldest_recorder_run October 24, 2024 at 5:20 PM
current_recorder_run November 19, 2024 at 9:47 AM
estimated_db_size 3032.83 MiB
database_engine mysql
database_version 10.11.6
Spotify
api_endpoint_reachable ok

Supervisor diagnostics

config_entry-hassio-5dae251bf6ea118d92a240060ae3caf8.json

Additional information

In the meantime, I am trying the solution proposed in the following comment
home-assistant/operating-system#1163 (comment)

Which for me was as follow, while connected through ssh.

1- List all connections

> nmcli -m tabular c


NAME               UUID                                  TYPE      DEVICE
Supervisor enp0s3  502849d7-2609-3319-813c-1622970139ef  ethernet  enp0s3
lo                 d5b4fc51-dc86-4e86-99d4-b9652b9b3f59  loopback  lo

2- Set mdns mode to resolve on the Supervisor connection
> nmcli connection modify "Supervisor enp0s3" connection.mdns resolve

3- Restart Network Manager
> systemctl restart NetworkManager.service

4- Confirm it works

> resolvectl mdns
Global: yes
Link 2 (enp0s3): resolve
Link 3 (docker0): yes
Link 4 (hassio): yes

I am wondering if this is something that could be exposed to the UI, or if someone has a better solution?

@sle118 sle118 added the bug label Nov 19, 2024
@sle118
Copy link
Author

sle118 commented Nov 20, 2024

Following up on this: as expected, the change does not persist so before I looked at the logs this morning, the system-resolved service had a chance to try 55K times. Next I'm going to see if I can execute the command on boot somehow.

If this doesn't work, then... maybe it's time to roll out my own supervisor.

@sle118
Copy link
Author

sle118 commented Nov 21, 2024

I ended up creating a custom add-on for this. It monitors the primary interface at regular (configurable interval) and if mdns is not "resolve" on the primary (configurable too) interface, then it changes mdns automatically. If anyone is interested, I'm willing to share but at least for now this will do the trick.

@skorpion1298
Copy link

skorpion1298 commented Nov 23, 2024

Hey there :) I stumbled on this Thread here because my HA is using 30% CPU constantly and I was searching for a reason why. After scrolling through Logs I saw the Spamming of Hostname Conflicts. Can this be related to the CPU usage? If yes, I absolute would love to have your Add-On for this. High CPU Usage means High Electricity costs and here in Germany its expensive :D
I dont even have the ability to run nmcli. Its not recognized :/

Thank you a lot!

@t3hk0d3
Copy link

t3hk0d3 commented Nov 26, 2024

Problem is originated from this line:

CONF_ATTR_CONNECTION_MDNS: Variant("i", 2),

  • mdns | int32 | -1 | Whether mDNS is enabled for the connection. The permitted values are: "yes" (2) register hostname and resolving for the connection, "no" (0) disable mDNS for the interface, "resolve" (1) do not register hostname but allow resolving of mDNS host names and "default" (-1) to allow lookup of a global default in NetworkManager.conf. If unspecified, "default" ultimately depends on the DNS plugin. This feature requires a plugin which supports mDNS. Otherwise, the setting has no effect. Currently the only supported DNS plugin is systemd-resolved. For systemd-resolved, the default is configurable via MulticastDNS= setting in resolved.conf.

@agners
Copy link
Member

agners commented Nov 26, 2024

The default NetworkManager configuration sets mDNS to yes which, in network having multiple vlans and a reflector (e.g. on a Unifi network), result in a flood of mDNS requests originating from the supervisor's systemd-resolved.

[...]

The issue was also reported under the operating system project, but got closed.

Just to be clear, from what I can tell, the mDNS responder systemd-resolved behaves exactly how it is designed to behave: It uses the link-local mDNS protocol to announce its hostname, and validates the uniqueness on the network. The mDNS protocol is clearly defined as link-local only in RFC6762. Handling name conflicts is not explicitly specified, but mentioned too as possible behavior.

The "issue" here is that mDNS reflectors are a hack to extend mDNS outside of the link-local context. This is operating mDNS clearly outside of the specified context! I am not aware of a RFC which would define mDNS reflector behavior, and how a multi-homed host should behave in the face of looped back mDNS announcements.

FWIW, mDNS reflector also do not work if a device has IPv6 link-local addressing (fe80...). These addresses do not work across network segments by design. I haven't looked into specific mDNS reflectors behavior. But I'd expect that if there are routable address classes in the same response, a mDNS reflector filters out those link-local ones. Otherwise a resolving host might pick the link-local address (which is not valid in the local network segment) and fails to communicate with that particular device. I just want to point out that this potentially lead to issues also in other use cases (e.g. Matter), since mDNS is expected to be link-local.

That said, I understand that such Unifi network setups are quite popular. If folks feel comfortable to run mDNS in an non-standardized setup and deal with potential fallout (like this, I guess 🙈 ) I am fine adding features to help make Home Assistant work in such environments.

Also keep in mind that you ask for a specific behavior which suits your specific network setup. I use VLANs as well, but without mDNS reflectors. I actually want Home Assistant to announce itself on every network.

@agners
Copy link
Member

agners commented Nov 26, 2024

Btw, at least at one point Core itself also didn't like duplicate mDNS instances (see home-assistant/core#50695). Now I think today this no longer leads to a packet flood, but there is probably still an error in the logs in these setups.

@sle118
Copy link
Author

sle118 commented Nov 26, 2024

Also keep in mind that you ask for a specific behavior which suits your specific network setup. I use VLANs as well, but without mDNS reflectors. I actually want Home Assistant to announce itself on every network.

I guess me setting up a reflector actually defeats one of the original intent of reducing IoT broadcast packets such as the ones from mDNS amongst others. I am starting to lean towards a dedicated ethernet for the VirtualBox guest running home assistant so I can plug it on a dedicated VLAN port on my switch. At a bare minimum I could move home assistant to the IoT VLAN.

I am fine with turning off the reflector and keep all the esphome devices on the IoT VLAN, but I am not sure about other possible issues this could bring. For instance, how do you have home assistant "announce itself on every network"?

@agners
Copy link
Member

agners commented Nov 26, 2024

I am fine with turning off the reflector and keep all the esphome devices on the IoT VLAN, but I am not sure about other possible issues this could bring. For instance, how do you have home assistant "announce itself on every network"?

Yeah I think Home Assistant (the service) announces itself only on the primary interface currently. But since I have my phones etc. on the primary interface for me that is fine.

@patienttruth
Copy link

I ended up creating a custom add-on for this. It monitors the primary interface at regular (configurable interval) and if mdns is not "resolve" on the primary (configurable too) interface, then it changes mdns automatically. If anyone is interested, I'm willing to share but at least for now this will do the trick.

If you'd be willing to share that would be awesome. I get it's a bandaid, but right now I'm just trying to keep everything stable until a better options comes. (thanks for the quick PR - I wish I knew how to make something like that happen so quickly!)

I'm also open to other solutions. Right now I have an IOT VLAN (no egress for most devices via firewall rules) and a Chromecast VLAN, as well as my main VLAN. HAOS sits on IOT and Main. Perhaps I could turn off mDNS on my Unifi USG, and put HAOS on the Chromecast VLAN as well. I haven't experimented with that, and only understand mDNS a little.

Ultimately my goal is segregated network for the IOT devices, and everything to work.

@Stan-Gobien
Copy link

I have the same setup with HA on normal LAN and second interface on IOT VLAN.
The mDNS names are used frequently in HA, and when you can't use them or click on them it's frustrating (because your computer is in the normal LAN). Example ESPhome shows the mDNS name and not the IP address.

mDNS reflector/repeater was the solution, but triggered the above mentioned problem.

I'm interested in the add-on too, so I could use mDNS again.

@patienttruth
Copy link

I ended up creating a custom add-on for this.

If you'd be willing to share that would be awesome.

I went ahead and blocked mDNS on all except my main HOAS network interface via my router. No more log overload.

@sle118
Copy link
Author

sle118 commented Nov 29, 2024

We should monitor #5455

In the meantime, I went on a learning journey and created the following addon for anyone to use as a duck tape solution
https://github.com/sle118/addons-network-manager-fix

EDIT: Hold on with the installation please. It looks like I need to resolve a couple of issues with the published version; local was just fine

@patienttruth
Copy link

We should monitor #5455

In the meantime, I went on a learning journey and created the following addon for anyone to use as a duck tape solution
https://github.com/sle118/addons-network-manager-fix

EDIT: Hold on with the installation please. It looks like I need to resolve a couple of issues with the published version; local was just fine

Thank you! Will hold tight for now.

@sle118
Copy link
Author

sle118 commented Nov 29, 2024

ok Fixed. You can try now. The logs should output something like this:

 Add-on: NetworkManager Fix
 Fix NetworkManager configuration for mDNS to prevent flooding caused by reflectors across multiple vlans
-----------------------------------------------------------
 Add-on version: 1.0
 You are running the latest version of this add-on.
 System: Home Assistant OS 13.2  (amd64 / qemux86-64)
 Home Assistant Core: 2024.11.2
 Home Assistant Supervisor: 2024.11.4
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
s6-rc: info: service banner successfully started
s6-rc: info: service nmfix: starting
s6-rc: info: service nmfix successfully started
s6-rc: info: service legacy-services: starting
[Warning]: apparmor not available on host system!
No log_level set in config, fallback to info
s6-rc: info: service legacy-services successfully started
Warning: nmcli (1.44.4) and NetworkManager (1.44.2) versions don't match. Restarting NetworkManager is advised.
[12:10:39] INFO: If different than the primary interface, please configure a primary interface from the following list:
[12:10:39] INFO: "Supervisor enp0s3"
[12:10:39] INFO: "lo"
[12:10:39] INFO: Primary adapter automatically detected: enp0s3. Getting connection name...
[12:10:39] INFO: Primary interface automatically detected: Supervisor enp0s3
[12:10:39] INFO: Current mDNS status for enp0s3 (Supervisor enp0s3): resolve
[12:10:39] INFO: no changes needed for enp0s3 (Supervisor enp0s3)

If a change was needed, logs will indicate it. This typically occurs the first time the addon is executed and later when updates are installed (I believe only supervisor updates will do this, but I could be wrong).

hope this will help someone while a real fix is being worked on!

@coesy
Copy link

coesy commented Dec 16, 2024

This is great, I've just started using this to reduce the noise in the logs. How do you configure a primary interface to use?

@sle118
Copy link
Author

sle118 commented Dec 18, 2024

This is great, I've just started using this to reduce the noise in the logs. How do you configure a primary interface to use?

By default, the add-on binds to the primary interface. You can check the documentation on the add-on where everything is explained as I could

@arsenicks
Copy link

Just to add to this, I'm ha ving the same issue using opnsense and mdns repeater. Will try the add on provided but I feel hass should really clear things regarding the current state of support for multi nic, there's problem that seems to be around for some time, like this one or the "default" network interface flapping between those available even if you select only one of the interface the supervisor can select another one and from my experience this can change..

@patienttruth
Copy link

the current state of support for multi nic, there's problem that seems to be around for some time, like this one or the "default" network interface flapping between those available even if you select only one of the interface the supervisor can select another one and from my experience this can change..

I've noticed this too. I haven't concluded that it is causing a problem, although the only time I notice is when I'm into troubleshoot something networking related. It usually seems to sort itself out and assign the correct interface as "default", but it seems like that should be able to be manually defined.

I've also noticed that even though I disable a network interface in the HA settings, it will become re-enabled at some point. (Just checked, and my third interface is set to automatic again when I am certain I said it too disabled a couple days ago - perhaps it happens when I do a reboot).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants