Postgres authentication not working

[brandli]

(total rookie writing)

I tried to follow the tutorial for installing flyte on my desktop machine which has microk8s on it. I can set up the dependencies, minio seems to be running fine and I can also log into the postgres pod and see the according databases.

But whenever I try to run "helm install flyte-binary flyteorg/flyte-binary --values onprem-flyte-binary-values.yaml -n flyte", the pods keeps crashing with following log:

time="2024-07-15T09:37:59Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/013-storage-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml /etc/flyte/config.d/202-database-secrets.yaml]"
2024/07/15 09:37:59 /flyteorg/build/datacatalog/pkg/repositories/config/postgres.go:75
[error] failed to initialize database, got error failed to connect to `host=postgres.flyte.svc.cluster.local user=flyte database=flyte`: server error (FATAL: password authentication failed for user "flyte" (SQLSTATE 28P01))
panic: failed to connect to `host=postgres.flyte.svc.cluster.local user=flyte database=flyte`: server error (FATAL: password authentication failed for user "flyte" (SQLSTATE 28P01))
goroutine 136 [running]:
github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x4360f40, 0xc0007f2f50})
 /flyteorg/build/datacatalog/pkg/repositories/initialize.go:63 +0x786
github.com/flyteorg/flyte/cmd/single.startDataCatalog({0x4360f40, 0xc0007f2f50}, {0x0?})
 /flyteorg/build/cmd/single/start.go:48 +0x1d
github.com/flyteorg/flyte/cmd/single.glob..func4.3()
 /flyteorg/build/cmd/single/start.go:235 +0x31
golang.org/x/sync/errgroup.(*Group).Go.func1()
 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96

I am using the default "postgres" password and I already figured out that there is a username mismatch in the two files provided: in "onprem-flyte-dependencies .yaml" the username is "flyte" but in "onprem-flyte-binary-values.yaml" the username is "postgres". After I changed the config for the binary, it now just keeps complaining about the password. I tried it with providing the passwork together with the username but also with the local-secret.yaml ... neither is working. Could you please help me and point me in the right direction?

(There also seems to be a typo in point 3 of "002-single-node-onprem-install.md" -> the command is "kubectl apply -f local-flyte-resources.yaml" but the file downloaded is called "onprem-flyte-dependencies.yaml" ... or am I missing something?)

[brandli]

Adding the username to the secrect (acc. this bug report) fixed the issue but now I have another problem. No clue why the pod is not running:

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:514 +0x258
{"json":{"src":"start.go:226"},"level":"panic","msg":"Failed to start Propeller, err: failed to create FlyteWorkflow CRD: Post \"https://10.152.183.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","ts":"2024-07-15T10:33:10Z"}
panic: (*logrus.Entry) 0xc0006f0ee0
goroutine 146 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0006f0e00, 0x0, {0xc003db1d40, 0x110})
 /go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:260 +0x491
github.com/sirupsen/logrus.(*Entry).Log(0xc0006f0e00, 0x0, {0xc0017a1e68?, 0x1?, 0x1?})
 /go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Entry).Logf(0xc0006f0e00, 0x0, {0x34d6a61?, 0x0?}, {0xc0040a67b0?, 0x0?, 0x0?})
 /go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:349 +0x7c
github.com/sirupsen/logrus.(*Entry).Panicf(0x4360f40?, {0x34d6a61?, 0x414325?}, {0xc0040a67b0?, 0x2cf9f20?, 0x1?})
 /go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:387 +0x2e
github.com/flyteorg/flyte/flytestdlib/logger.Panicf({0x4360f40?, 0xc000697ef0?}, {0x34d6a61, 0x22}, {0xc0040a67b0, 0x1, 0x1})
 /flyteorg/build/flytestdlib/logger/logger.go:189 +0x5e
github.com/flyteorg/flyte/cmd/single.glob..func4.2()
 /flyteorg/build/cmd/single/start.go:226 +0xb1
golang.org/x/sync/errgroup.(*Group).Go.func1()
 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96

[nkwangleiGIT]

it seems that flyte propeller failed to connect to K8S apiserver using https://10.152.183.1:443, normally it should be the internal svc address 10.96.0.1:443

[davidmirror-ops]

@brandli thank you for pointing the problems in the guide and sorry for the poor experience.
I'll work on reproducing/fixing these problems soon. Thanks!
Also if you want to contribute, please feel free.

[davidmirror-ops]

@brandli there was a mismatch in the db username between the Helm values and the actual db deployment. It only affected flyte-binary and is now fixed.

Could you redeploy and report back any issues?
Thanks and sorry for the inconveniences