OpenStack single-stack IPv6 support #1909

yasminvalim · 2024-07-15T21:01:52Z

In #1897 issue, we are requested to create support for IPv6 so Ignition can fetch the metadata in single-stack environments.

To achieve this, we added a new URL with the IPv6 endpoint. We also created logic to first attempt the IPv4 endpoint so If this fails, it will try the IPv6 one. If both endpoints fail, it will return the appropriate error.

travier · 2024-07-17T11:14:30Z

Not sure that will work. What happens if the instance is IPv6 only? Do requests fail differently than if the network is not up yet?

travier · 2024-07-17T11:15:46Z

One of the principle behind Ignition is that it will retry indefinitely until the server explicitly returns an error.

yasminvalim · 2024-07-17T17:04:53Z

Not sure that will work. What happens if the instance is IPv6 only? Do requests fail differently than if the network is not up yet?

I updated my code logic. In an IPv6-only instance, IPv4 will fail because it's not found and the resources will be returned from IPv6. If both fails it will return errors. The error will be handled with a message in dispatch function above. It makes sense?

One thing I need to consider is the possibility of having both IPv4 and IPv6 simultaneously. Is that possible? If yes, I need to handle this case as well.

yasminvalim · 2024-07-17T17:08:51Z

One of the principle behind Ignition is that it will retry indefinitely until the server explicitly returns an error.

Nice. I guess this is being handled in func fetchConfig, right? There is a loop in there like this:

Loop:
	for {
		select {
		case <-ctx.Done():
			break Loop
		case <-errChan:
			dispatchCount--
			if dispatchCount == 0 {
				f.Logger.Info("couldn't fetch config")
				break Loop
			}
		}
	}

	return util.ParseConfig(f.Logger, data)
}

Thanks!

travier · 2024-07-22T16:19:39Z

One thing I need to consider is the possibility of having both IPv4 and IPv6 simultaneously. Is that possible? If yes, I need to handle this case as well.

Yes, this is very common.

yasminvalim · 2024-07-23T15:01:25Z

One thing I need to consider is the possibility of having both IPv4 and IPv6 simultaneously. Is that possible? If yes, I need to handle this case as well.

Yes, this is very common.

I updated the code to cover this scenarios. So, now the code looks like this:

If both endpoints have valid data, the results are combined and returned
I used append, not sure if is the best approach.
If only one endpoint has valid data, that data is returned
If both endpoints fail, the appropriate error is returned with logs
If both endpoints return resource.ErrNotFound, the function returns nil

These should be the basic scenarios. Now, I'm working on running my code in the OpenStack environment and reading the necessary docs to test it correctly.

Thanks for reviewing and helping me with this task!

MaysaMacedo

Something to take into consideration:

MaysaMacedo · 2024-08-06T19:06:46Z

internal/providers/openstack/openstack.go

 	}
+	ipv6MetadataServiceUrl = url.URL{
+		Scheme: "http",
+		Host:   "[fe80::a9fe:a9fe]",


I wonder if we might need to specify the interface name like it's mentioned on the docs
https://docs.openstack.org/nova/latest/user/metadata.html#the-metadata-service
and how it's done elsewhere
https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceOpenStack.py#L76

HuijingHei · 2024-08-20T09:13:50Z

internal/providers/openstack/openstack.go

+		}
+		ipv6MetadataServiceUrl = url.URL{
+			Scheme: "http",
+			Host:   fmt.Sprintf("[fe80::a9fe:a9fe%%%s]", url.PathEscape(iface)),


Maybe we need to change to [fe80::a9fe:a9fe%25%s] refer to https://docs.openstack.org/nova/latest/user/metadata.html#the-metadata-service

prestist · 2025-01-31T19:46:24Z

internal/providers/openstack/openstack.go

 	}
+	metadataServiceUrlIPv6 = url.URL{
+		Scheme: "http",
+		Host:   "fe80::a9fe:a9fe%",


Have you tried having the host be [fe90::a9fe:a9fe%]? since you are getting an error on fetch due to improper escaping ?

I agree with @prestist . Looks like you're missing to enclose the host in square brackets.
At least it's documented here.
Maybe you will also need to add the interface name to the host once you have it, before calling FetchToBuffer

yasminvalim · 2025-02-06T13:59:53Z

Hey team, I'm experiencing an error while trying to connect the server. I added some log to the code, then I ran openstack console log show <server-id> command and received the following:

[   18.648461] NetworkManager[808]: <info>  [1738700037.3670] dhcp6 (enp3s0): state changed new lease, address=fd2e:6f44:5dd8:c957::1bd
[   24.819235] ignition[918]: GET result: OK
[   24.826066] ignition[918]: failed to fetch config from metadata service: IPv4 succeeded, but IPv6 failed: parse "http://fe80::a9fe:a9fe%25/openstack/latest/user_data": invalid port ":a9fe%25" after host
[   24.852271] ignition[918]: Successfully fetched configuration from IPv4.
[   24.862505] ignition[918]: Fetching IPv6 address for metadata service...
[   24.872374] ignition[918]: Fetching zone id...
[   24.879246] ignition[918]: Checking interface: lo
[   24.886322] ignition[918]: Checking interface: enp3s0
[   24.893800] ignition[918]: Verificando se o IP �� IPv6: 10.1.0.179, resultado: false
[   24.904868] ignition[918]: Verificando se o IP �� IPv6: fd2e:6f44:5dd8:c957::1bd, resultado: true
[   24.917265] ignition[918]: Active interface found: enp3s0
[   24.925264] ignition[918]: Fetching from IPv6 metadata service at http://[fe80::a9fe:a9fe%%25enp3s0]/openstack/latest/user_data...
[   24.941718] ignition[918]: Fetching from IPv6 metadata service at http://fe80::a9fe:a9fe%25/openstack/latest/user_data...
[   24.960088] ignition[918]: IPv6 metadata service failed: parse "http://fe80::a9fe:a9fe%25/openstack/latest/user_data": invalid port ":a9fe%25" after host

Not sure why is not parsing correctly.

yasminvalim · 2025-02-06T18:45:03Z

[SOLVED] Mobbing session output:

We utilized an OpenStack environment to fetch and curl the IPv6, but were unable to get a result. Here are the logs:

[cloud-user@jcapitao-cs9-metadata-test ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
	link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	inet 127.0.0.1/8 scope host lo
   	valid_lft forever preferred_lft forever
	inet6 ::1/128 scope host
   	valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
	link/ether fa:16:3e:1f:f8:90 brd ff:ff:ff:ff:ff:ff
	altname enp0s3
	altname ens3
	inet 10.0.79.79/23 brd 10.0.79.255 scope global dynamic noprefixroute eth0
   	valid_lft 41025sec preferred_lft 41025sec
	inet6 2620:52:0:4e:f816:3eff:fe1f:f890/64 scope global dynamic noprefixroute
   	valid_lft 2591943sec preferred_lft 604743sec
	inet6 fe80::f816:3eff:fe1f:f890/64 scope link noprefixroute
   	valid_lft forever preferred_lft forever
[cloud-user@jcapitao-cs9-metadata-test ~]$ curl http://169.254.169.254/openstack/
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
2020-10-14
latest[cloud-user@jcapitao-cs9-metadata-test ~]$
[cloud-user@jcapitao-cs9-metadata-test ~]$ curl -v -g -6 'http://[fe80::a9fe:a9fe%25eth0]'
*   Trying fe80::a9fe:a9fe:80...
#... timeouts.

[EDIT] I was able to reproduce this commands in other machine and worked. It might be a older openstack version, since IPV6 support only came after Victoria release.

yasminvalim · 2025-03-06T20:17:30Z

Hey team, it's finally working properly and I was able to test it correctly. I would be grateful if you could take one last look at my code. Thanks!

Steps to Test:

Create the FCOS image with Ignition changes
Enter the OpenStack environment
Upload the modified FCOS image to your OpenStack environment.
Create the OpenStack image
Create the Ignition file
Prepare an Ignition configuration file that contains the necessary setup for your FCOS server.
Use the following command to create the server, passing the Ignition file path as user-data:

openstack server create \
  --port <port> \
  --image fedora-coreos-example.20250211.dev.0-openstack.x86_64 \
  --flavor worker \
  --security-group default \
  --user-data <ignition_file_path> \
  <server_name>

Note: If you're using the Ignition file with SSH keys, you don't need to specify a keypair.

SSH into the server

Once the server is created, you can SSH into it using the following command:

ssh -o "IdentitiesOnly=yes" -i <private_key_path> core@<server_ip>

prestist

Really good start, and thank you for working on this, just a few questions and nits.

prestist · 2025-03-07T20:38:42Z

internal/providers/openstack/openstack.go

+	isIPv6 := ip.To4() == nil
+	return isIPv6
+}
+


I think it would be good to have some reference on why you are looking for the zone ID in the manner that you are. A simple link to the IPV6 metadata service docs should be good with a quick summary? wdyt?

internal/providers/openstack/openstack.go

prestist · 2025-03-07T21:06:22Z

internal/providers/openstack/openstack.go

+	fmt.Println("Fetching zone id...")
+	interfaces, err := net.Interfaces()
+	if err != nil {
+		return "", fmt.Errorf("error fetching zone id: %v", err)


I think it might be more appropriate to say network interfaces as this is not really the zone id, and could confuse debugging. wdyt?

"error fetching network interfaces: %v", err

prestist · 2025-03-07T21:09:46Z

internal/providers/openstack/openstack.go

+			continue
+		}
+
+		for _, addr := range addrs {


Can I get a small comment explaining how this is giving us the zoneID?

MaysaMacedo · 2025-03-17T21:12:08Z

internal/providers/openstack/openstack.go


-	// the metadata server exists but doesn't contain any actual metadata,
-	// assume that there is no config specified
-	if err == resource.ErrNotFound {


shouldn't we continue handling the not found error like it was done here?
I believe the same would need to happen for IPv6.

tormath1

Thanks for working on this and for the initial effort. This is less trivial than expected :D

Here's some feedback:

The whole implementation (ipv4 / ipv6 detection) should be common to all providers. E.g: there is currently a similar ask for Scaleway provider (#1897 (comment))
For the implementation itself: it might increase the provisioning time as we need to wait for IPv4 to be seen as "unreachable" before giving up and trying IPv6

ignition/internal/resource/http.go

Line 309 in b111af1

duration := initialBackoff

(not to mention that sometimes the interface itself is not directly available to the instance)

Something I would propose:

For a cloud provider, we define a map of URLs:

urls := map[IP]url.URL{
    IPv4: url.URL{
        Scheme: "http",
	Host:   "169.254.169.254",
	Path:   "openstack/latest/user_data",
    },
    IPv6: url.URL{
	Scheme: "http",
	Host:   "[...]",
	Path:   "openstack/latest/user_data",
    }
}

In a common implementation, we start two go routines sharing a success channel. One routine trying the IPv4 stack and the other one using the IPv6 stack (with the correct IP for each) - of course if the initial URLs map contains only IPv4, we start only the IPv4 function (same goes for IPv6).
The first routine to return a success (i.e an Ignition configuration) will notify the channel and we can continue as usual (similar to what @prestist suggested here: https://github.com/coreos/ignition/pull/1909/files#r1985704411)

I'm not a big fan of using Go routines, but I think this situation might be a good usecase.

tormath1 · 2025-03-27T14:54:25Z

internal/providers/openstack/openstack.go

-		return nil, nil
+	// Try fetching from IPv4 first
+	response, err = f.FetchToBuffer(metadataServiceUrlIPv4, resource.FetchOptions{})
+	f.Logger.Debug("IPv6 URL:", metadataServiceUrlIPv6.Host)


Suggested change

f.Logger.Debug("IPv6 URL:", metadataServiceUrlIPv6.Host)

f.Logger.Debug("IPv4 URL:", metadataServiceUrlIPv6.Host)

We're trying to fetch IPv4 first?

Signed-off-by: Mathieu Tortuyaux <[email protected]>

This defines a wrapper that will try in paralell both IPv4 and IPv6 when the provider declares those two IPs. Signed-off-by: Mathieu Tortuyaux <[email protected]>

yasminvalim · 2025-04-14T20:03:12Z

Hey @tormath1, thanks for reviewing and working on that! I went through your changes, and your approach makes complete sense. I agree with your comments. To align with your work, I’ve cherry-picked the changes you made in internal/resource/url.go, and I’ll adapt the provider side accordingly so we can implement something like what you did for Scaleway for OpenStack!

Thanks again!

yasminvalim · 2025-04-28T15:20:31Z

@tormath1 I just finished implementing the helper! I'd really appreciate it if you could take a look and share your feedback.

tormath1 · 2025-05-12T15:24:09Z

@tormath1 I just finished implementing the helper! I'd really appreciate it if you could take a look and share your feedback.

Thanks for the ping! I somehow missed it. I'll give a try to this implementation, I noticed some hiccups in the current approach but I think it should be easy to fix. I'll let you know :)

This defines a wrapper that will try in paralell both IPv4 and IPv6 when the provider declares those two IPs. Signed-off-by: Mathieu Tortuyaux <[email protected]>

prestist

I think this is super close, I thiiink the main breakage is the template we have for ipv6, but I think how we reach out and find our nic is a point of failure as well. If we do my comments I think this will work :)

prestist · 2025-10-06T21:10:40Z

internal/providers/openstack/openstack.go

+
+		resource.IPv6: {
+			Scheme: "http",
+			Host:   "[fe80::a9fe:a9fe%iface]",


I think we are missing a bit of the ipv6

[fe80::a9fe:a9fe%25iface]

prestist · 2025-10-06T21:19:58Z

internal/providers/openstack/openstack.go

+	return data, nil
+}
+
+func findInterfaceWithIPv6() (string, error) {


So I think it might be good to try all interfaces rather then just the first one we find. So lets return all IPV6 interfaces which are UP and not loopback. i.e return ( []string,error)

prestist · 2025-10-06T21:33:04Z

internal/providers/openstack/openstack.go

+		string(resource.IPv4): userdataURLs[resource.IPv4],
+	}
+
+	ifaceName, err := findInterfaceWithIPv6()


Now since we have a list of strings, rather then 1 lets reach out to all of them at the same time.

I think we can do this a few ways, though I think it would be cool to do it in parallel using fan-out pattern.

Golang has a few concepts that will help us to achieve that.

goroutines, channels, waitgroups

…attern using goroutines and waitgroups

yasminvalim self-assigned this Jul 15, 2024

yasminvalim marked this pull request as draft July 15, 2024 21:01

yasminvalim force-pushed the openstack-support branch 2 times, most recently from 9e7cba8 to 87e4a48 Compare July 16, 2024 18:30

yasminvalim force-pushed the openstack-support branch from 87e4a48 to 5fad1fd Compare July 17, 2024 16:53

yasminvalim force-pushed the openstack-support branch from 5fad1fd to 91f89b5 Compare July 23, 2024 14:55

yasminvalim force-pushed the openstack-support branch from 91f89b5 to a850f29 Compare July 23, 2024 18:26

MaysaMacedo reviewed Aug 6, 2024

View reviewed changes

yasminvalim force-pushed the openstack-support branch 2 times, most recently from d81e5fb to 6b8a11e Compare August 8, 2024 22:23

HuijingHei reviewed Aug 20, 2024

View reviewed changes

yasminvalim force-pushed the openstack-support branch 3 times, most recently from f2979b8 to fa50682 Compare October 16, 2024 22:35

yasminvalim force-pushed the openstack-support branch from fa50682 to ee0d2af Compare October 23, 2024 20:13

yasminvalim force-pushed the openstack-support branch 5 times, most recently from e105dea to 1569e4c Compare January 6, 2025 13:46

yasminvalim force-pushed the openstack-support branch from 1569e4c to 35c1993 Compare January 13, 2025 19:52

prestist reviewed Jan 31, 2025

View reviewed changes

yasminvalim force-pushed the openstack-support branch from 787e7b9 to 0bb5233 Compare March 6, 2025 19:42

yasminvalim force-pushed the openstack-support branch 2 times, most recently from 18f03e7 to b3d311d Compare March 7, 2025 17:41

yasminvalim marked this pull request as ready for review March 7, 2025 17:43

prestist reviewed Mar 7, 2025

View reviewed changes

yasminvalim force-pushed the openstack-support branch 8 times, most recently from dbb2567 to e6b084a Compare March 13, 2025 14:47

MaysaMacedo reviewed Mar 17, 2025

View reviewed changes

tormath1 reviewed Mar 27, 2025

View reviewed changes

tormath1 mentioned this pull request Apr 11, 2025

url: support IPv4 and IPv6 for IMDS #2052

Draft

yasminvalim closed this Apr 14, 2025

yasminvalim force-pushed the openstack-support branch from e6b084a to 50a4776 Compare April 14, 2025 15:16

release-notes: add new feature entry

b9f4f6e

yasminvalim reopened this Apr 14, 2025

tormath1 added 2 commits April 14, 2025 12:22

url: try local port on both IP stacks

9ad66ad

Signed-off-by: Mathieu Tortuyaux <[email protected]>

url: support both IPv4 and IPv6

d5f444c

This defines a wrapper that will try in paralell both IPv4 and IPv6 when the provider declares those two IPs. Signed-off-by: Mathieu Tortuyaux <[email protected]>

openstack.go: add FetchConfigDualStackin openstack

d030c07

url: support both IPv4 and IPv6

aabc1db

This defines a wrapper that will try in paralell both IPv4 and IPv6 when the provider declares those two IPs. Signed-off-by: Mathieu Tortuyaux <[email protected]>

prestist reviewed Oct 6, 2025

View reviewed changes

openstack: implement parallel IPv6 interface discovery with fan-out p…

b6f610c

…attern using goroutines and waitgroups

	f.Logger.Debug("IPv6 URL:", metadataServiceUrlIPv6.Host)
	f.Logger.Debug("IPv4 URL:", metadataServiceUrlIPv6.Host)

OpenStack single-stack IPv6 support #1909

Are you sure you want to change the base?

OpenStack single-stack IPv6 support #1909

Uh oh!

Conversation

yasminvalim commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Jul 17, 2024

Uh oh!

travier commented Jul 17, 2024

Uh oh!

yasminvalim commented Jul 17, 2024

Uh oh!

yasminvalim commented Jul 17, 2024

Uh oh!

travier commented Jul 22, 2024

Uh oh!

yasminvalim commented Jul 23, 2024

Uh oh!

MaysaMacedo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yasminvalim commented Feb 6, 2025

Uh oh!

yasminvalim commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yasminvalim commented Mar 6, 2025

Uh oh!

prestist left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tormath1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yasminvalim commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yasminvalim commented Apr 28, 2025

Uh oh!

tormath1 commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prestist left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yasminvalim commented Jul 15, 2024 •

edited

Loading

yasminvalim commented Feb 6, 2025 •

edited

Loading

yasminvalim commented Apr 14, 2025 •

edited

Loading

tormath1 commented May 12, 2025 •

edited

Loading