Guidance on running on AWS ECS

[code-bunny]

Running Thruster on HTTP is all grand and lovely, but when I went to set this up on HTTPS on ECS I had a few issues leading to it just not accepting any requests.

The setup I tried was an AWS ALB to a ip https target group which I then connected to my cluster with an ECS task. On the target group and ECS task I tried both http/2 and http/1 combinations and even tried public IP's. In all cases the Rails application and Thruster start up and then no traffic can get to the task, not even the health checks. I did check and even have work devops check the security groups. Best we can tell is that it can't open a connection.

Current theory is maybe it isn't compatible with a ALB.

Is there a recommended setup? Having HTTPS at the task level would be amazing.

Use Ctrl-C to stop
staff
20 November 2024 at 19:52 (UTC)
Puma starting in single mode...
staff
20 November 2024 at 19:52 (UTC)
* Puma version: 6.4.3 (ruby 3.3.6-p108) ("The Eagle of Durango")
staff
20 November 2024 at 19:52 (UTC)
* Min threads: 3
staff
20 November 2024 at 19:52 (UTC)
* Max threads: 3
staff
20 November 2024 at 19:52 (UTC)
* Environment: staging
staff
20 November 2024 at 19:52 (UTC)
* PID: 84
staff
20 November 2024 at 19:52 (UTC)
* Listening on http://0.0.0.0:3000
staff
20 November 2024 at 19:52 (UTC)
=> Booting Puma
staff
20 November 2024 at 19:52 (UTC)
=> Rails 8.0.0 application starting in staging
staff
20 November 2024 at 19:52 (UTC)
=> Run `bin/rails server --help` for more startup options
staff
20 November 2024 at 19:52 (UTC)
{"time":"2024-11-20T19:52:02.825479254Z","level":"DEBUG","msg":"No custom 502 page found","path":"./public/502.html"}
staff
20 November 2024 at 19:52 (UTC)
{"time":"2024-11-20T19:52:02.825585699Z","level":"DEBUG","msg":"TLS: initializing","directory":"https://acme-v02.api.letsencrypt.org/directory","using_eab":false}
staff
20 November 2024 at 19:52 (UTC)
{"time":"2024-11-20T19:52:02.825680509Z","level":"INFO","msg":"Server started","http":":80","https":":443","tls_domain":["stage-staff.codebunnies.com"]}```

[pas256]

You may have run into the issue in AWS ECS where does not allow a process running as a non-root user to bind to privileged ports (<1024). I was banging my head against the wall because Thruster fails silently. I had to get a shell inside the container and take a look at netstat to see Thurster wasn't listening at all.

It works when you run locally through Docker because Docker sets net.ipv4.ip_unprivileged_port_start=0, basically allowing you to bind any port as an unprivileged user (see: moby/moby#41030). This has been brought up on the ECS roadmap, but no resolution so far: aws/containers-roadmap#1721

The workaround is to have Thruster use a higher port number by setting an environment variable. For HTTP that is HTTP_PORT=8080, and for TLS that is HTTPS_PORT=8443 or whatever you like.

I'm not sure why Thruster fails silently under these conditions (super frustrating), but at least we know how to work around it.

[code-bunny]

Ah thanks for the insight! I'll give it a go on a higher port as you suggested. I return to work tomorrow so will give it a try and see if this resolves my issue.