Setting DNS parameters on Fedora

[DimitriPapadopoulos]

I suspect the official way to modify DNS parameters on CentOS >= 8 and Fedora is running resolvectl, at least when resolvectl is available.

Therefore I would recommend the following mechanism when --set-dns=1:

1. If resolvectl is available and works properly, run it to modify DNS parameters. Decide at build-time or run-time?
2. Else if resolvconf is available and works properly (either openresolv or resolvconf but not resolvectl), run it to modify DNS parameters. Decide at build-time or run-time?
3. Else fall back on modifying /etc/resolv.conf directly.

I recommend the above order, resolvectl before resolvconf, because resolvconf might be available but doesn't seem to be working properly on Fedora.

Man pages:

• resolvectl(1) from systemd
• resolvconf(8) from Debian
• resolvconf(8) from openresolv

Hopefully we can get some info from this bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=1815605

[grey-olli]

When using vpn nameservers I need nameserver records to be inserted rather than appended by openfortivpn. It is unclear for me is this mode is completely impossible or not in my configuration. I'm currently using fedora 31 xfce spin, /etc/resolv.conf is a symlink to systemd-resolved /usr/lib/systemd/resolv.conf , NetworkManager is in 'default' mode . It receives dns from dhcp and in /run/systemd/resolve/resolve.conf I see:

[root@dell15 ~]# tail -n 4 /run/systemd/resolve/resolv.conf 
nameserver 8.8.4.4
nameserver 10.126.126.137
nameserver 172.16.116.194
search dell15.localdomain
[root@dell15 ~]# ls -l /etc/resolv.conf 
lrwxrwxrwx. 1 root root 28 Mar 24 14:13 /etc/resolv.conf -> /usr/lib/systemd/resolv.conf
[root@dell15 ~]# tail -n 4 /usr/lib/systemd/resolv.conf 
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
[root@dell15 ~]# 

The problem I wish to solve - dns over vpn - how to configure openforticlient in that way, when these records received from vpn upstream:

nameserver 10.126.126.137
nameserver 172.16.116.194

are placed upper than record received from dhcp via NetworkManager when dell15 has ethernet plugged in:

nameserver 8.8.4.4

and remove inserted on vpn down.

This may appear to be a task of systemd, not openfortivpn? Can openfortivpn instruct systemd-resolved to insert records ?

[DimitriPapadopoulos]

I believe we agree but this issue is quite complex:

• For years openfortivpn has been updating /etc/resolv.conf directly and it has been "working". Nothing has really changed in recent versions on Fedora.
• You have a choice of 1) letting openfortivpn either call resolvconf or fall back on modifying /etc/resolv.conf directly (--set-dns=1 --pppd-use-peerdns=0) 2) instructing openfortivpn to let pppd handle DNS (--set-dns=0 --pppd-use-peerdns=1) but my experience is that doesn't work well on Fedora.
• The standard way of modifying DNS parameters of Debian/Ubuntu and other Linux vendors is resolvconf (as provided by resolveconf or openresolv). While a resolvconf program is provided by systemd on recent CentOS and Fedora versions, it is not compatible with the other two implementations and/or doesn't work.
• It looks like systemd-resolved is not installed by default, let alone enabled by default on all Fedora machines: it is not installed by default on Fedora build servers that create the RPM packages and it is not enabled by default on Fedora workstations (at least that's what I reproduce after a default installation of Fedora 31 "Workstation" in a virtual machine).
• On CentOS / Fedora it is not clear what the preferred / official / most robust way of modifying DNS parameters is. My guess is that it's resolvectl - which is why I have created this issue. Any clue?

[DimitriPapadopoulos]

@grey-olli That said the records are not appended on my Fedora 31 virtual machine, they are properly inserted in front of the rest:

Before:

$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 10.0.2.3
$ 

Running openfortivpn:

$ sudo /opt/openfortivpn/bin/openfortivpn -u xxxxxxx xxxxx.xxxx.de
VPN account password: 
INFO:   Connected to gateway.
INFO:   Authenticated.
INFO:   Remote gateway has allocated a VPN.
Using interface ppp0
Connect: ppp0 <--> /dev/pts/1
INFO:   Got addresses: [10.212.134.200], ns [xxx.xx.252.252, xxx.xx.252.202]
INFO:   negotiation complete
INFO:   negotiation complete
local  IP address 10.212.134.200
remote IP address 192.0.2.1
INFO:   Interface ppp0 is UP.
INFO:   Setting new routes...
INFO:   Adding VPN nameservers...
INFO:   Tunnel is up and running.

After:

$ cat /etc/resolv.conf 
nameserver xxx.xx.252.252
nameserver xxx.xx.252.202
search xxxx.de xxx.xxxx.de xxx.de
# Generated by NetworkManager
nameserver 10.0.2.3
$ 

Which version of openfortivpn are you using? Which options?

[DimitriPapadopoulos]

For what it's worth on a default installation of Fedora 31:

$ rpm -q systemd
systemd-243.7-1.fc31.x86_64
$ 
$ rpm -q openfortivpn
openfortivpn-1.12.0-1.fc31.x86_64
$ 

Here is what I see:

$ ls -l /etc/resolv.conf
-rw-r--r--. 1 root root 136 Mar 24 15:30 /etc/resolv.conf
$ 

whether systemd is enabled:

$ sudo systemctl enable --now systemd-resolved
Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → /usr/lib/systemd/system/systemd-resolved.service.
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.
$ 
$ $ sudo systemctl status --no-pager systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-03-24 15:14:00 CET; 34s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 3506 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 2332)
   Memory: 9.5M
      CPU: 274ms
   CGroup: /system.slice/systemd-resolved.service
           └─3506 /usr/lib/systemd/systemd-resolved

mars 24 15:13:59 localhost.localdomain systemd[1]: Starting Network Name Resolution...
mars 24 15:14:00 localhost.localdomain systemd-resolved[3506]: Positive Trust Anchors:
mars 24 15:14:00 localhost.localdomain systemd-resolved[3506]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bb…7f8ec8d
mars 24 15:14:00 localhost.localdomain systemd-resolved[3506]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.i…
mars 24 15:14:00 localhost.localdomain systemd-resolved[3506]: Defaulting to hostname 'linux'.
mars 24 15:14:00 localhost.localdomain systemd[1]: Started Network Name Resolution.
Hint: Some lines were ellipsized, use -l to show in full.
$ 

or not:

$ sudo systemctl disable --now systemd-resolved
$ 
$  sudo systemctl status --no-pager systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

mars 24 14:54:21 localhost.localdomain systemd[1]: Starting Network Name Resolution...
mars 24 14:54:21 localhost.localdomain systemd-resolved[716]: Positive Trust Anchors:
mars 24 14:54:21 localhost.localdomain systemd-resolved[716]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc…7f8ec8d
mars 24 14:54:21 localhost.localdomain systemd-resolved[716]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in…
mars 24 14:54:21 localhost.localdomain systemd-resolved[716]: Defaulting to hostname 'linux'.
mars 24 14:54:21 localhost.localdomain systemd[1]: Started Network Name Resolution.
mars 24 14:54:24 localhost.localdomain systemd-resolved[716]: Server 10.0.2.3 does not support DNSSEC, downgrading to non-DN…C mode.
mars 24 15:12:34 localhost.localdomain systemd[1]: Stopping Network Name Resolution...
mars 24 15:12:34 localhost.localdomain systemd[1]: systemd-resolved.service: Succeeded.
mars 24 15:12:34 localhost.localdomain systemd[1]: Stopped Network Name Resolution.
Hint: Some lines were ellipsized, use -l to show in full.
$ 

[DimitriPapadopoulos]

Also (while openfortivpn is running):

$ ls -l /etc/resolv.conf 
-rw-r--r--. 1 root root 136 Mar 24 15:49 /etc/resolv.conf
$ 
$ cat /etc/resolv.conf
nameserver xxx.xx.252.252
nameserver xxx.xx.252.202
search xxxx.de xxx.xxxx.de xxx.de
# Generated by NetworkManager
nameserver 10.0.2.3
$ 

$ ls -l /run/systemd/resolve/
total 8
drwx------. 2 systemd-resolve systemd-resolve  60 Mar 24 14:54 netif
-rw-r--r--. 1 systemd-resolve systemd-resolve 585 Mar 24 15:14 resolv.conf
-rw-r--r--. 1 systemd-resolve systemd-resolve 708 Mar 24 15:14 stub-resolv.conf
$ 
$ cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.0.2.3
$ 

$ ls -l /usr/lib/systemd/resolv.conf
-rw-r--r--. 1 root root 692 Feb  7 16:51 /usr/lib/systemd/resolv.conf
$ 
$ cat /usr/lib/systemd/resolv.conf
# This file belongs to man:systemd-resolved(8). Do not edit.
#
# This is a static resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists no search
# domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
$ 

[DimitriPapadopoulos]

In any case:

• openfortivpn is correct in trying to access /etc/resolv.conf rather than /usr/lib/systemd/resolv.conf.
• It looks like resolvectl is the proper command to use.
• We probably need to read carefully systemd-resolved.service(8) for more information.

I am still unable to grasp how DNS is configured on Fedora and how to modify the DNS parameters. My experience is that it has always been complicated on Fedora / CentOS.

[grey-olli]

@DimitriPapadopoulos
Well, after all above I have tested the following:

presets:

• the systemd and openfortivpn are same versions you mentioned.
• On each change of /etc/resolv.conf symlink NetworkManager was restarted, but not systemd-resolved as man page claims it should detect the changes seamlessly (automatic).

tested combinations:

1. /etc/resolve.conf as a file "generated by network manager" , not symlink, systemd-resolved enabled
2. /etc/resolve.conf symlinked to /usr/lib/systemd/resolv.conf
3. /etc/resolve.conf symlinked to /run/systemd/resolve/resolv.conf
4. /etc/resolve.conf symlinked to /run/systemd/resolve/stub-resolv.conf

Above 4 combinations with openfortivpn using all 3 combinations of settings:

[root@dell15 etc]# grep "dns =" /etc/openfortivpn/config
set-dns = 1
pppd-use-peerdns = 0
[root@dell15 etc]#

I mean 1/0 ; 0/1 ; 1/1 . 0/0 seem to be useless for my task.

Result:

"pppd-use-peerdns" seem to have no effect on /run/systemd/resolve/resolve.conf

"set-dns = 1" make my real dell15 host (not VM) to APPEND dns nameserver records in /run/systemd/resolve/resolve.conf . I need them inserted replacing "global" dns record.

If I stop NetworkManager, make /etc/resolv.conf a file again and reconnecct with openfortivpn - the systemd-resolved is detected and managed by openfortivpn - /run/systemd/resolve/resolve.conf has updated nameserver records (but again appended , not inserted).

systemd-resolved resolvectl multi-call binary tells that there is global dns and it is always the same - openfortivpn does NOT change 'resolvectl dns' output in "global" line - it changes specific "link " line with additional DNS servers.

systemd-resolved seem to configure "fall back" DNS w/o my explicit request:

[olli@dell15 ~]$ resolvectl | head -n 13
Global
LLMNR setting: yes
MulticastDNS setting: yes
DNSOverTLS setting: no
DNSSEC setting: allow-downgrade
DNSSEC supported: yes
Current DNS Server: 8.8.4.4
DNS Servers: 8.8.4.4
Fallback DNS Servers: 1.1.1.1
8.8.8.8
1.0.0.1
8.8.4.4
2606:4700:4700::1111
[olli@dell15 ~]$

When vpn is up I have something like this:

[root@dell15 ~]# resolvectl dns
Global: 8.8.4.4
Link 14 (ppp0): 10.126.126.137 172.16.116.194
Link 3 (wlp6s0):
Link 2 (enp7s0):
[root@dell15 ~]#

Who should pickup at least one ppp0 DNS server and append it to Global list? systemd?

[mrbaseman]

@grey-olli did you notice that we have issue #555 with openfortivpn 1.12.0 on Fedora? It's fixed in openfortivpn 1.13.1 for which packages are on their way. I believe with those you can achieve what you want:

If you want to have the dns entries prepended, you could use openfortivpn --use-resolvconf=0 --pppd-use-peerdns=0 --set-dns=1 in version 1.13.1.

Since you are mentioning dhcp, where your lease might expire, you might also want to use --half-internet-routes=1 at least for longer lasting connections when a new default route through the vpn tunnel is pushed.

[DimitriPapadopoulos]

@grey-olli I'm terribly sorry but I have mistakenly selected "Edit" instead of "Quote reply" and modified your post. Don't know how to revert, did my best 😥

[DimitriPapadopoulos]

Result:

"pppd-use-peerdns" seem to have no effect on /run/systemd/resolve/resolve.conf

Yes, I interpret that as pppd being broken on Fedora but I may be wrong.

"set-dns = 1" make my real dell15 host (not VM) to APPEND dns nameserver records in /run/systemd/resolve/resolve.conf . I need them inserted replacing "global" dns record.

Strange. On my machine opnefortivpn prepends DNS records sent by the FortiGate appliance before all the rest so they should have priority. See my own example:

$ cat /etc/resolv.conf
nameserver xxx.xx.252.252
nameserver xxx.xx.252.202
search xxxx.de xxx.xxxx.de xxx.de
# Generated by NetworkManager
nameserver 10.0.2.3
$ 

If I stop NetworkManager, make /etc/resolv.conf a file again and reconnect with openfortivpn - the systemd-resolved is detected and managed by openfortivpn - /run/systemd/resolve/resolve.conf has updated nameserver records (but again appended , not inserted).

Strange. openfortivpn does not modify /run/systemd/resolve/resolve.conf unless maybe /etc/resolv.conf is a symlink to /run/systemd/resolve/resolve.conf. Does this happen with --set-dns=1 --pppd-use-peerdns=0? If so some other software is applying these changes, perhaps NetworkManager itself.

systemd-resolved resolvectl multi-call binary tells that there is global dns and it is always the same - openfortivpn does NOT change 'resolvectl dns' output in "global" line - it changes specific "link " line with additional DNS servers.

Yes. This is because there is a local DNS servers that hides the DNS servers behind it. It does not necessarily mean that the modified /etc/resolv.conf is not taken into account behind the scenes.

I really don't know. The DNS subsystem is extremely complex on Fedora / CentOS. Reading and understanding systemd-resolved.service(8) is beyond my current capabilities.

Our best bet is probably to support resolvectl when it is available.

systemd-resolved seem to configure "fall back" DNS w/o my explicit request:

    [olli@dell15 ~]$ resolvectl | head -n 13
    Global
           LLMNR setting: yes
    MulticastDNS setting: yes
      DNSOverTLS setting: no
          DNSSEC setting: allow-downgrade
        DNSSEC supported: yes
      Current DNS Server: 8.8.4.4be
             DNS Servers: 8.8.4.4
    Fallback DNS Servers: 1.1.1.1
                          8.8.8.8
                          1.0.0.1
                          8.8.4.4
                          2606:4700:4700::1111
    [olli@dell15 ~]$ 

Yes, same here. That's built-in by Fedora.

When vpn is up I have something like this:

    [root@dell15 ~]# resolvectl dns
    Global: 8.8.4.4
    Link 14 (ppp0): 10.126.126.137 172.16.116.194
    Link 3 (wlp6s0):
    Link 2 (enp7s0):
    [root@dell15 ~]#

Who should pickup at least one ppp0 DNS server and append it to Global list? systemd?

I don't know. I'm lost in the complexity of the DNS subsystem and the multiple cases above. I cannot help. Perhaps it would be easier to focus on a single case you think should work and investigate it thoroughly in a new issue. The new issue should:

• if possible run in a brand new Fedora virtual machine with default settings and fully updated,
• give the exact version of openfortivpn and systemd packages,
• give the version of Fedora,
• show how you run openfortivpn (options),
• show file status and file contents of /etc/resolv.conf, /run/systemd/resolve/resolv.conf, /usr/lib/systemd/resolv.conf before and after openfortivpn is started,
• show sudo systemctl status systemd-resolved (it seems to be stopped by default on a recently installed Fedora)
• show the output of resolvectl or resolvectl dns before and after,
• always copy/paste from the terminal instead of describing what is in the terminal in other words,
• make sure NetworkManager-fortisslvpn is not used, perhaps not even installed,
• whatever else I have forgotten here...

But then I cannot guarantee I will have time to look into it