lb and eip resource changes in terraform

[sbogar-shs]

I am running into three issues when customizing the security groups and using static IPs for a concourse network lb using bbl

1. aws_eip resources should now be using domain = 'vpc" instead of vpc = true

This one is more of a "nice to have"
2) aws_lb does not like the subnet entry when trying to use Elastic IPs:
subnets = ["${aws_subnet.lb_subnets.*.id}"]
This generates an error when you run bbl up using EIPs

The correct syntax for mapping subnets to EIPs in aws_lb:

dynamic "subnet_mapping" {
  for_each = aws_subnet.lb_subnets[*].id
  content {
    subnet_id = subnet_mapping.value
    allocation_id = aws_eip.nlb_eip[subnet_mapping.key].id
  }
}

new resource for lb eips

resource "aws_eip" "nlb_eip" {
  count  = length(var.availability_zones)
  domain = "vpc"
}

3. when running bbl destroy it seems to wipe out the bbl-template.tf which is very limiting to what can be merged with _override.tf files as you are not allowed to add any new inputs/objects...this is extremely limiting...or maybe I am missing something (very possible). Once you have done any customization I am finding I am stuck in a state where bbl can not destroy the infrastructure.

Is there a reason why bbl-template.tf is being replaced during a bbl destroy?

[sbogar-shs]

here is one example for a typical use case (which should probably just be a feature of bbl instead of a customization) would be adding a set of input variables that hold CIDRs for various security groups.
then you might want to update security group ingress rules so that instead of 0.0.0.0/0 they use a custom cidr built up with a concat(my_cidr1, mycidr2, ..., mycidrN)

Right now that does not seem possible as the only input variable we have to use is bosh_director_cidr which only hits the jumpbox.

[sbogar-shs]

error message on aws_lb (this is corrected with the dynamic subnet_mapping chunk posted above):

Error: Incorrect attribute value type
│
│ on bbl-template.tf line 907, in resource "aws_lb" "concourse_lb":
│ 907: subnets = ["${aws_subnet.lb_subnets.*.id}"]
│ ├────────────────
│ │ aws_subnet.lb_subnets is tuple with 6 elements
│
│ Inappropriate value for attribute "subnets": element 0: string required.

[sbogar-shs]

quick note: I was able to get a terraform destroy to run and cleanup a customized infrastructure by just renaming my bbl_override.tf to bbl_custom.tf, copying the terraform.tfstate and variable files into the terraform directory and then running a terraform destroy manually.

[sbogar-shs]

ok final update from my testing today:

the --lb-type concourse switch breaks bbl up, even vanilla, without modifications, it simply does not work in AWS right now due to the dynamic subnet_mapping issue and fails with the Error: Incorrect attribute value type

Not 100% sure if this is related to version of terraform, but I am using the latest version 1.10.5
latest bbl version 9.0.30

[ramonskie]

the terraform is provided with the bootloader cli.
the current version 1.4.5 see https://github.com/cloudfoundry/bosh-bootloader/blob/main/ci/pipelines/bosh-bootloader.yml#L426
unless you specify a local terrafrom which was added in #603
there is also a pr #597 not sure if this is related though

[sbogar-shs]

I did not specify a local version so it must have been using the version of terraform included in BBL 9.0.30

I don't think this issue is related to that PR as that is related to CF ALB and this is regarding the Concourse NLB.

To replicate the issue, simply try to stand up a fresh vanilla bbl up with --lb-type concourse in AWS, it just fails.

[sbogar-shs]

It looks like another way to solve this error would be to use the 'flatten' function on the subnets property sort of like this:

subnets = flatten("${aws_subnet.lb_subnets.*.id}")

I haven't tested this code but did see it out there somewhere as a workaround for this issue

realized i saw this on the PR #597 you had referenced....

[ramonskie]

if you could confirm this. you can create a PR and we can merge your fix.
as it seems some changes has been made to aws. so it makes sense in a way

[sbogar-shs]

PR has been created
#629