Implement CAPI IPAM Provider (and Equinix Metal VRF support)

[displague]

User Story

Equinix Metal is a provider where the new CAPI IPAM features could be beneficial.

In Equinix Metal, Layer2 and Hybrid modes give users more control over the IPs and routing of their nodes. The VRF feature allows for IPAM blocks to be defined for additional routing benefits, through interconnection virtual circuits and gateways. IP assignments to individual nodes from the VRF block of IPs is currently unmanaged. The IPs are node / ARP managed. The Cluster API IPAM provider could be used to create VRF blocks and assign them to nodes (IP Claims) by creating IP reservation assignments in the Equinix Metal API. These assignments can include metadata (customdata, tags) that would map the Equinix Metal resources to the CAPI resources (IP Claims).

https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#motivation
https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster#cluster-api-ipam-provider-in-cluster

Detailed Description

Implement the IPAM provider in the CAPP provider. Follow the guidance set by the CAPV (and perhaps Metal3, others?) implementation.

https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#implementing-an-ipam-provider

https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/docs/node-ipam-demo.md

Anything else you would like to add:

• https://deploy.equinix.com/developers/docs/metal/interconnections/vrf/

/kind feature

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cprivitere]

/reopen

[k8s-ci-robot]

@cprivitere: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cprivitere]

/remove-lifecycle rotten

[rahulii]

hi @displague, here is my understanding of the feature:
Once, we have integrated IPAM Provider with CAPP, here is how it would work (please correct me If I am wrong):

1. User installs CAPP with IPAM Provider.
2. User creates an IPPool Custom Resource with a specific IP subnet defined (eg: 10.10.10.0/24) from which the IPAM will allocate the IP addresses.
   For example:

    apiVersion: ipam.cluster.x-k8s.io/v1alpha1
    kind: IPPool
    metadata:
      name: some-pool
    spec:
    pools:
    - subnet: 10.10.10.0/24
      start: 10.10.10.100
      end: 10.10.10.200

3. The IPAM provider will use the EM API to create '10.10.10.0/24' IP address reservation in VRF. (how to know which VRF to add IP address reservations to ?) or will it create a new VRF altogether ??

4. Once the IP Address reservation is created - IPAM can start allocating IPAddress on IPAddressClaim Request.

Please correct me if I have misunderstood anything (mainly point no.3 )
Note: This does not talks about how to actually implement IPAM. that will be done seperately.

[displague]

(how to know which VRF to add IP address reservations to ?) or will it create a new VRF altogether ??

With all of our IP options, IP Reservations, LoadBalancers (#740), VRF, I think it is going to be helpful to give the options for users to select an existing resource or (default) create a new one. There will also be a need to be ways to indicate how deletions work. Whether these options can be exposed as spec or annotations will depend on how the Kubernetes resource is defined, whether our controllers define that spec or whether they are defined upstream.

The machine in the cluster will need to join the VLAN (through the Equinix Metal APIs) where the VRF is attached. The IPAM/VRF request/definition will need to specify this somehow.

The local OS networking will also need to be configured, this may mean setting up IPs and it may mean attaching to VLANs, again at an OS level. If the IPAM Provider framework offers general-purpose tools like the CSI Attacher (which manages OS level mounting), I would hope that VLAN options would be exposed through that tool.