KFP Object Storage Configuration

[piatsko]

We are deploying kubeflow on our local cluster. One of the modifications we did is to use s3 as an artifact storage. Simply following the guide did not help as pipeline pod was still trying to connect to minio-service.kubeflow:9000. After some investigation we found out that one more ConfigMap needs to be changed -- workflow-controller-configmap as it is hardcoded to use local minio. We patched artifactRepository field in that config and only then pipelines started to use s3 as an artifact storage. Something similar is shown in sample aws installation

The question is: are we actually supposed to patch this ConfigMap which is related to argo workflows? If so, then maybe it should be reflected in documentation as well.

[juliusvonkohout]

Hello, is this still relevant in Kubeflow 1.11? I replaced Minio with Seaweedfs.

[piatsko]

Yes, this is still relevant. Without this change pipelines will try to save artifacts to seaweedfs, as it's hardcoded here. So, even if pipelineRoot and S3 provider are configured as per KFP documentation, runs will still fail

[piatsko]

Or maybe you mean that seaweedfs can sync with remote storage?

[juliusvonkohout]

Plese check https://github.com/kubeflow/pipelines/tree/master/manifests/kustomize/third-party/seaweedfs#gateway-to-remote-object-storage. I stongly recommend the seaweedfs gateway for easier compatibility. Othwerwise I can offer consulting https://www.kubeflow.org/docs/started/support/#support-from-commercial-providers-in-the-kubeflow-ecosystem. And please make sure that you are on Kubeflow 26.03+.

You could also raise a PR to update the documentation.