Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Github Action getting tailscale cversion 1.42.0 instead of default 1.72.1; ssh timed out until version manually set #147

Open
jhbrown-apeiron opened this issue Dec 9, 2024 · 10 comments

Comments

@jhbrown-apeiron
Copy link

First off, thank you for tailscale, it's the first VPN I've ever used that has truly great UX.

This morning our GHA deployments via ssh over tailscale stopped working. As far as I can tell, this was because:

  1. The V2 tag for tailscale's action points at a commit where the default version is 1.42.0; 1.4.2.0 has known vulnerabilities;
  2. Although you can see warnings on the tailscale admin console, the action completed successfully without complaint;
  3. But after that, ssh wasn't working, it was just timing out -- maybe tailscale was refusing to route for this vultnerable version?

When I hard-coded 1.78.1 as the required version, everything started working for us. I'm reporting this because:

  1. The Tailscale action should fail noisily if the version of tailscale specified is too old to be usable
  2. The default action configuration should not specify a vulnerable version of tailscale.

Again, thanks for Tailscale!

@jhbrown-apeiron jhbrown-apeiron changed the title Github Action getting tailscale cversion 1.42.0 instead of default 1.72.1; ssh timed out until manually set input Github Action getting tailscale cversion 1.42.0 instead of default 1.72.1; ssh timed out until version manually set Dec 9, 2024
@matanbaruch
Copy link

@jhbrown-apeiron I swiched from @v2 to @main on the action which uses 1.72.1.

But +1 on this. This should be maintained as this official package.

@evilhamsterman
Copy link

@matanbaruch @jhbrown-apeiron I don't necessarily agree but that is expected behavior. https://tailscale.com/kb/1276/tailscale-github-action?q=github+action#tailscale-github-action-version

Though it does look like they've got enough complaints they may actually be changing it #131

@jhbrown-apeiron
Copy link
Author

@matanbaruch @jhbrown-apeiron I don't necessarily agree but that is expected behavior. https://tailscale.com/kb/1276/tailscale-github-action?q=github+action#tailscale-github-action-version

I'd argue that "breaks all the workflows" counts as "meaningful" for "It changes only when there is a meaningful reason to do so." But snark aside, I'm happy it's getting fixed.

@matanbaruch
Copy link

@jaxxstorm merged #131 which actually solve our issues.

I think it worth the v3 of the action.

@evilhamsterman
Copy link

It doesn't necessarily need to be a v3. In fact I would recommend it not be a v3. The typical way that actions work is they have a major and a minor tag, but the major tag alone is the latest version of the minor release. So v2 includes 2.0, 2.1, etc. This would be a v2.1 to me and v2 should also pick it up. If someone wants to stick with v2.0 they would include v2.0 anyone using v2 would expect to get the latest version. If they're very paranoid they can use the commit hash

@fabn
Copy link

fabn commented Dec 16, 2024

This morning our GHA deployments via ssh over tailscale stopped working. As far as I can tell, this was because:

We're experiencing this as well with lot of timeout errors on GH like:

# DNS timeouts
error looking up IP of "sts.eu-north-1.amazonaws.com.": lookup sts.eu-north-1.amazonaws.com. on 127.0.0.53:53: read udp 127.0.0.1:54485->127.0.0.53:53: i/o timeout
Error: getaddrinfo EAI_AGAIN sts.eu-north-1.amazonaws.com
# Of if I disable DNS with args: '--accept-dns=false'
Error: Get "https://<redacted>.gr7.eu-north-1.eks.amazonaws.com/api/v1/namespaces/test": dial tcp 172.33.25.157:443: i/o timeout

Unfortunately the suggested fix didn't worked for us, even with 1.78.1 version the issue persists.

What should we do? We're not able to deploy some of our workloads because of this.

@smitmartijn
Copy link

Unfortunately the suggested fix didn't worked for us, even with 1.78.1 version the issue persists.

Same here, Github runners aren't able to connect to the tailnet and I can't figure out why. I've tried several tailscale versions (ones that match working machines), different ways of connecting with using --accept-routes and internal networks, the tailscale DNS hosts, and tailscale IPs directly, but none are working.

The weird part is that tailscale status and tailscale netcheck say everything is fine and the action also runs cleanly. I even see the internal subnets come past in ~/tailscaled.log. The GitHub runner also shows up briefly in the tailscale dashboard. Yet, no connection is possible.

I'm thinking it might be the GH runners, but I'm not sure. Hoping anyone here has ideas..

@garry-t
Copy link

garry-t commented Dec 16, 2024

relevant for us. Also if you try SSH from UI most probably you also will not able to connect. After manually restart tailscale on hosts I tried SSH from tailscale UI and it started work and also after this it started work from github actions.

@jaxxstorm
Copy link
Contributor

Unfortunately the suggested fix didn't worked for us, even with 1.78.1 version the issue persists.

Same here, Github runners aren't able to connect to the tailnet and I can't figure out why. I've tried several tailscale versions (ones that match working machines), different ways of connecting with using --accept-routes and internal networks, the tailscale DNS hosts, and tailscale IPs directly, but none are working.

The weird part is that tailscale status and tailscale netcheck say everything is fine and the action also runs cleanly. I even see the internal subnets come past in ~/tailscaled.log. The GitHub runner also shows up briefly in the tailscale dashboard. Yet, no connection is possible.

I'm thinking it might be the GH runners, but I'm not sure. Hoping anyone here has ideas..

what does your ACL file look like?

172.33.25.157

This is T-mobile IP address, so not sure why it would be in your DNS path, and likely isn't a Tailscale thing.

I'd recommend opening support tickets for these issues.

@mpminardi
Copy link
Member

mpminardi commented Dec 18, 2024

On the versioning front here we've released a v3 which:

  • Includes Always pull latest version #131 which allows for opting in to fetching the latest stable version of Tailscale
  • Will be a floating tag that is updated to point to the latest v3.Y.Z releases of the action as they come out to have behaviour that is more expected of actions releases / tags

Note that the guidance provided here around the default version for the Tailscale client is otherwise still true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants