module source dynamically set

[amarjothi]

In git, we have branches and release strategy,
A usecase, I want to have modules that I pull from use ref from released branches, but I will always want those to use same branch, within a project:

prod_git_tag = "v.0.0.1"
staging_git_tag = "v.0.5.7"
and then

module "ec2-instance" {
  source = "${git_repo_path}?ref=${git_tag}"
}

module "kubernetes-cluster" {
  source = "${git_repo_path}?ref=${git_tag}"
}

etc...

which seems pretty reasonable to me - when I pass in git_tag=prod_git_tag, now they all reference the same git_tag and can be updated with one line, rather than in all the various places.

At the moment Terraform, is throwing error line 2:
Variables may not be used here.

[pkolyvas]

Heya! Totally reasonable request, just one very hard to do with Terraform's current design. In a very basic sense, because Modules and Providers are initialized in a separate operation, we don't have a way to interpolate values for their source attributes, they have to be known at initialization.

We'd like to get to this and more dynamic work with providers, but we don't have it on our roadmap at the moment because we laser-focused on delivering a stable 1.0 product.

[ryanfelder]

Somehow I didn't find this ticket when I searched and then created #26888 . Sorry about that.

One way this might be implemented without totally breaking the Terraform design would be an init script.

I am now a Terraform Enterprise user, but in my previous role I was using the non-enterprise version on the command line.
I miss some of the flexibility I had then. To simplify things in that environment, I wrote ./plan.sh and ./apply.sh scripts. In both of these, I ran 'terraform init' every time. Brute force, but it never caused problems.

If Terraform Enterprise had support for a pre-run script, terraform.tfinit or something similar, I could run sed on the module source before the terraform plan and swap out the branch before the normal terraform init occurs.

Just a suggestion, and one that might be useful in other ways for terraform enterprise as well.

[fuzzyjulz]

Very keen on this - The version number specifically is a real pain if I use a constant version of a module I need to do a massive search and replace to update all my instances where I reference that module - even if we could setup a module alias somehow a bit like the terraform default provider works that would solve a lot of issues

[dimisjim]

this issue could be extended for the version module attribute, which also doesn't support variables' usage.

[tony-kerz]

i get that arbitrary interpolation is particularly challenging with the current flow of terraform plan/apply resolution.

would the complexity of the ask be mitigated if it were only allowed with special parameters similar to the way backends are initialized?

e.g. -backend-config

[solarmosaic-kflorence]

My use case is using the same module in several places in a single project (e.g. in /project/main.tf, as well as /project/modules/sub-module/main.tf as well as in a fixture for the tests). When updating a module version, it would be great to do it in a single place instead of multiple places. The best workaround I have found is to actually wrap any modules that are used multiple times into a sub-module, and reference that everywhere instead. But it is extra boilerplate that can be annoying to maintain (e.g. having to copy inputs/outputs).

[neugeeug]

Would be really nice to have the possibility ...
We use one Github repository as a kind of library for several modules and reference the modules in the way:

module "rds" {
  source = "github.com/repo_name//rds?ref=v1.0.16"

When we release a new version of the "TF library" we need to go through all the modules and increase the version :(

[jkblume]

A dynamic version property that could be filled on terraform init, would be super helpful on gitops approaches, where one updates version parameter automatically in a pipeline. Currently we need to search and replace the version string where the module is used.

If we got a possibility to define e.g. sources or versions in a .json or something like that, would be very good. This file can be used on init process to follow the currently implemented workflow of terraform. If you work with local terraform you can implement this on your own, with some pre-init shell scripts that do the search and replace work. Sadly this approach is not possible in the very integrated terraform enterprise.

[spkane]

Wouldn't it be possible to at least read in the auto.tfvars and any supplied tfvars files and use those to populate variables in a modules source = line?

I haven't dug into the code, but on the surface, it feels like this would at least make it possible to pass in a constant that doesn't, in theory, really need any additional computing to be referenced.

It might only provide minimal functionality but it seems like it would serve the common need to simply make sure that all your references to a module are using the same git ref, without needing to do a search and replace across your code base.

Maybe, even a new constant variable type, that is handled in a much simpler way, would help unlock this capability.

[ayashjorden]

Adding a use-case where TF is ran in a distributed environment and need to pull modules from a remote location.
Surely you can understand that pulling modules from with the same region and not having cross-region traffic is preferred.
For example, s3 object URL can be interpolated with different region names.

Best,
Jordan

[byteknacker]

I really need a way to dynamically interpolate the content of the module source value. Without that feature, I have to resort to wrapper tools such as Terragrunt. Rather prefer to have it implemented directly in Terraform. Is this a high prio issue now or will it be put off for later implementation?

[amarjothi]

For Versioning, I have made use of override.tf file that seems to have worked, although does not reference dynamic tags.

The override.tf file contains the module definition alone, with the source referencing the required tagged version. eg.
module "rds" {
source = "github.com/repo_name/rds?ref=v1.0.16"
}
module "kubernetes-cluster" {
source = "github.com/repo_name/kc?ref=v1.0.16"
}

[ayashjorden]

Hi all,
In my use-case, I want to pull modules from configurable location, mostly like same-region to avoid cross region traffic.

So source = "s3::https://s3-${var.region}.amazonaws.com/artifacts-${var.region}-dev/common-aws.1.0.0.tar.xz" , makes sense to me that should be supported.

Can anyone link here to the area in the code :

• Where I can specify input arguments? (I guess that's in the main TF binary not a provider, I'd like to experiment
• Where is the init functionality happens so I can try to support -var or -var-file ?

My logic tells me that input variables or var-files would be similar if not identical to the input of the rest of the configuration. in most cases.
In anyway, even if not, experimenting with that would support the discussion...
Best,
Jordan

[byteknacker]

Using override, is a good idea. Have not thought about this. But I guess it is still a kind of hack. It would help to implement dynamic source values though.

[lemoo5sg]

Hi, same need here, we have plenty of repos & modules & tf files refering to modules from a common remote repo on github and we need a way without an additional wrapper script to have a dynamic remote url & ref value. If the price to pay is to run init each time, that's ok for us.
source = "${var.common_repo_url/common_module1?ref=${var.common_repo_ref}"

[jlforester]

This would also be of tremendous value to me. I'd also like to combine this with the ability to override the default public Terraform registry host. Some of our environments exist in air-gapped networks that have to bring everything with them (modules and providers, among other things), and others don't, and it would be nice to not have to modify the source for each environment.

[amarjothi]

@jlforester suggest using override.tf, you can use a script to update the file dynamically, before the run.
Override.tf contains all the referenced modules, where you can specify the versions. It will just be one file to update rather than files all over the place.
override.tf content below
module "rds" {
source = "github.com/repo_name/rds?ref=${rds_version}"
}
module "kubernetes-cluster" {
source = "github.com/repo_name/kc?ref=${kube_version}"
}

[jlforester]

@amarjothi Thanks for that information. I will give it a test drive.

[jlforester]

@amarjothi I've tried the override.tf method, and it's not going to work for us. It looks like the registry host has to be valid in the original source = and that's not going to be the case in an air gapped environment. Ideally, we'd simply like to specify the module source without the registry host, and have the ability to override the default registry host to one of our choosing via a command-line option or an environment variable.

[byteknacker]

I am just curious if this is fundamentally not doable or rather not on the priority list of features right now. Having dynamic module source values is really very helpful for many users. I hope that this will be soon a feature in Terraform. But for now, I have to literally change my code every time when I merge from test to main…

[amarjothi]

@amarjothi I've tried the override.tf method, and it's not going to work for us. It looks like the registry host has to be valid in the original source = and that's not going to be the case in an air gapped environment. Ideally, we'd simply like to specify the module source without the registry host, and have the ability to override the default registry host to one of our choosing via a command-line option or an environment variable.

Ours was an air-gapped environment. Source can take different types, ours was a reference to another git repository,
module "vpc" {
source = "git::https://example.com/vpc.git"
}

module "storage" {
source = "git::ssh://[email protected]/storage.git"
}

we had the global modules downloaded and installed on local gitlab.

[crw]

Hi @byteknacker, thanks for the question. This issue is essentially a duplicate of #23948, although scoped a bit more broadly in terms of use case. @apparentlymart has a series of comments on that issue which also cover this issue.

Regardless of feasibility, it is not on the priority list at the moment.

[byteknacker]

Ok understood. Thanks for the answer though!

[arLevi]

We personally need it, because each ENV of server cluster can access different git provider

# ENV X
module "rds" {
   source = "bitbucket.com/projectKey/repo.git//rds?ref=v1.0.0"
                            ^^^ 
                   bitbucket has a projectKey 
                   which git server doesn't
}

# ENV Y
module "rds" {
   source = "personal-git-server.com/repo.git//rds?ref=v1.0.0"
}

[HenriBlacksmith]

Same need as stated before, this could help moving away from Terragrunt.

On the feasibility, it could be some kind of self-wrapper as Terragrunt does in a first approach and be improved later without changing the user interface.

[arLevi]

What about giving the option to clone from local git bare repo on disk ?
why it has to be either http/s and ssh ?

This way, at least, i can sync the right repo -> to remote machine, then terraform will fetch it via local files ?

# not talking about this, where revisioning isn't possible:
source = "/path/to/file.tf"

# Talking about something like this
source   = "git::local://path/to/repo.git//myfile?ref=v1.0.2"

[salewski]

What about giving the option to clone from local git bare repo on disk ? why it has to be either http/s and ssh ?

This way, at least, i can sync the right repo -> to remote machine, then terraform will fetch it via local files ?

# not talking about this, where revisioning isn't possible:
source = "/path/to/file.tf"

# Talking about something like this
source   = "git::local://path/to/repo.git//myfile?ref=v1.0.2"

Hi @arLevi, I implemented a while back the ability to use local git repos via both relative and absolute path names:

• Add support for 'git::' force token on local filepaths, both absolute and relative go-getter#269
• salewski@1b0953b

Needs some polishing, which I haven't had time to get back to. My use case was using Git "submodules" within a repo (so the relative path locations to the other Git repos could be known in advance). But there's nothing in the implementation that required the use of Git submodules -- any referenced Git repos would work.

That ability solves a different problem than the current issue, however. It does not provide the ability to use Terraform variables in the source string.

[maha51ece-tech]

Any Update on this issue ..Do we have provision to dynamically set "source" and "version" during run time for variable passed from pipeline

We can test different version behaviour for particular terraform module registry.

[HenriBlacksmith]

Any updates on this?

I do not know if the announced Terraform stacks will also allow to use different tags for different deployments of modules stored in git (not just in TFE/TFC) but this would be nice.

Otherwise the better option to propagate a new module version among environments without repeating a lot of code remains Terragrunt. (I do hope bot solving this issue and stacks will allow us to drop Terragrunt)

[Rusty-1]

Try using the git commit ID in the ?ref=xxxxxxxxxxxx

[rybakovanton-metta]

it also can be useful for "abstract" or common modules.
right now it need to be set like this
common module:

module "vpc" {
  count   = var.cloud == "aws" ? 1 : 0
  source  = "./aws_vpc"
}
module "azr" {
  count         = var.cloud == "azr" ? 1 : 0
  source        = "./azr_vpc"
}

when you need to add another cloud provider or delete it, you need not only change your config with clouds definitions but you need to change the TF code that already in use, not just add a new module, not affecting and not changing modules somewhere in prod.
so much better something like this

module "vpc" {
  source = "./${clouds[var.vpc_config.cloud]}"
}