wicked seems to ignore sysctl.d precedence conventions

[jonahbull]

Version information:

openSUSE Leap 42.3
wicked-0.6.52-12.1.x86_64

I'm running an openSUSE Leap 42.3 VM on GCE. For GCE images, Google adds a file with some network-related sysctl options to /etc/sysctl.d/11-gce-network-security.conf. One of those options is: net.ipv4.ip_forward=0. Since I do want ip forwarding to be enabled for docker-related reasons, I've added another file at /etc/sysctl.d/99-docker-sysctl.conf containing net.ipv4.ip_forward=1 to override the Google-created one. Given this configuration, I would expect the entry in the file with the lexicographically latest name to take precedence, as per man 5 sysctl.d. That is what I see when I run systemd-sysctl manually:

jonahbull@opensuse-42-3-test:~> sudo SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-sysctl
Skipping overridden file: /usr/lib/sysctl.d/99-sysctl.conf.
Parsing /etc/sysctl.d/11-gce-network-security.conf
Parsing /usr/lib/sysctl.d/50-coredump.conf
Parsing /usr/lib/sysctl.d/50-default.conf
Parsing /etc/sysctl.d/99-docker-sysctl.conf
Overwriting earlier assignment of net/ipv4/ip_forward in file '/etc/sysctl.d/99-docker-sysctl.conf'.
Parsing /etc/sysctl.d/99-sysctl.conf
Overwriting earlier assignment of kernel/core_pattern in file '/etc/sysctl.d/99-sysctl.conf'.
Setting 'net/ipv4/icmp_ignore_bogus_error_responses' to '1'
Setting 'vm/max_map_count' to '262144'
Setting 'net/ipv4/conf/all/send_redirects' to '0'
Setting 'kernel/core_pattern' to '/var/lib/jenkins/core-dumps/core-%t-%p'
Setting 'net/ipv4/conf/all/accept_redirects' to '0'
Setting 'net/ipv4/conf/all/log_martians' to '1'
Setting 'kernel/pid_max' to '4194303'
Setting 'net/bridge/bridge-nf-call-ip6tables' to '0'
Setting 'net/ipv4/ip_forward' to '1'
[snip...]

However, the behavior I am seeing after booting the VM each time is that ip forwarding is disabled for lo and eth0, while enabled globally:

jonahbull@opensuse-42-3-test:~> find /proc/sys/net/ipv4/conf/{all,default,lo,eth0}/forwarding | while read file; do printf "%s %s\n" $file $(cat $file); done
/proc/sys/net/ipv4/conf/all/forwarding 1
/proc/sys/net/ipv4/conf/default/forwarding 1
/proc/sys/net/ipv4/conf/lo/forwarding 0
/proc/sys/net/ipv4/conf/eth0/forwarding 0

I turned on debug logging for wicked and was somewhat surprised to see that wicked doesn't appear to read the contents of each directory in __NI_SUSE_SYSCTL_DIRS in any particular order, so the precedence conventions one would expect from man 5 sysctl.d are broken. Here's a log snippet showing that for some reason /etc/sysctl.d/11-gce-network-security.conf is the last sysctl file read:

Mar 29 19:32:04 linux wicked[1003]: eth0: changed state none -> device-exists
Mar 29 19:32:04 linux wicked[1003]: trying to discover netif config via firmware service "ibft"
Mar 29 19:32:04 linux wickedd-nanny[1001]: waiting for 0 devices to become ready (0 explicitly requested)
Mar 29 19:32:04 linux kernel: No iBFT detected.
Mar 29 19:32:04 linux wicked[1003]: subprocess 1007 (/etc/wicked/extensions/ibft) exited with status 0 [0m0.013s]
Mar 29 19:32:04 linux wicked[1003]: ni_netconfig_firmware_discovery: buffer has 0 bytes
Mar 29 19:32:04 linux wicked[1003]: ni_sysconfig_read(/etc/sysconfig/network/config)
Mar 29 19:32:04 linux wicked[1003]: ni_sysconfig_read(/etc/sysconfig/network/dhcp)
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/boot/sysctl.conf-4.4.175-89-default'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/usr/lib/sysctl.d/50-default.conf'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/etc/sysctl.conf'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/usr/lib/sysctl.d/50-coredump.conf'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/etc/sysctl.d/99-docker-sysctl.conf'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/etc/sysctl.conf'
Mar 29 19:32:04 linux wicked[1003]: Reading sysctl file '/etc/sysctl.d/11-gce-network-security.conf'
Mar 29 19:32:04 linux wicked[1003]: ni_sysconfig_read(/etc/sysconfig/network/ifcfg-lo)
Mar 29 19:32:04 linux wicked[1003]: ni_sysconfig_read(/etc/sysconfig/network/ifcfg-eth0)
Mar 29 19:32:04 linux wicked[1003]: ni_compat_generate_interfaces: compat:suse:/etc/sysconfig/network/ifcfg-lo:0
Mar 29 19:32:04 linux wicked[1003]: ni_compat_generate_interfaces: compat:suse:/etc/sysconfig/network/ifcfg-eth0:0
Mar 29 19:32:04 linux wicked[1003]: No valid configuration files found at /etc/wicked/ifconfig
Mar 29 19:32:04 linux wicked[1003]: wait 30 seconds for interfaces
Mar 29 19:32:04 linux wicked[1003]: Device hierarchy structure:
Mar 29 19:32:04 linux wicked[1003]: lo
Mar 29 19:32:04 linux wicked[1003]: eth0
Mar 29 19:32:04 linux wicked[1003]: lo: hiring nanny
Mar 29 19:32:04 linux wicked[1003]: generate policy match for lo (type unknown)
Mar 29 19:32:04 linux wicked[1003]: <interface>
Mar 29 19:32:04 linux wicked[1003]:   <name>lo</name>
Mar 29 19:32:04 linux wicked[1003]:   <control>
Mar 29 19:32:04 linux wicked[1003]:     <mode>boot</mode>
Mar 29 19:32:04 linux wicked[1003]:     <boot-stage>localfs</boot-stage>
Mar 29 19:32:04 linux wicked[1003]:     <persistent>true</persistent>
Mar 29 19:32:04 linux wicked[1003]:   </control>
Mar 29 19:32:04 linux wicked[1003]:   <link/>
Mar 29 19:32:04 linux wicked[1003]:   <ipv4>
Mar 29 19:32:04 linux wicked[1003]:     <enabled>true</enabled>
Mar 29 19:32:04 linux wicked[1003]:     <forwarding>false</forwarding>
Mar 29 19:32:04 linux wicked[1003]:   </ipv4>

Would it be possible to update wicked so that it follows the same precedence conventions users expect from man 5 sysctl.d?