Docs for multiple storage backend support #8758

talSofer · 2025-03-05T09:16:23Z

Closes #8765
and https://github.com/treeverse/lakeFS-Enterprise/issues/57

github-actions · 2025-03-05T09:17:56Z

♻️ PR Preview 8b2f30f has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

github-actions · 2025-03-05T09:21:28Z

E2E Test Results - DynamoDB Local - Local Block Adapter

github-actions · 2025-03-05T09:21:40Z

E2E Test Results - Quickstart

ozkatz

Looks GREAT! A few nitpicks but nothing blocking

ozkatz · 2025-03-05T14:51:21Z

docs/howto/multiple-storage-backends.md

+
+This example setup configures lakeFS to manage data across two separate MinIO instances.
+
+```yaml


indentation is wrong (2 spaces everywhere)

ozkatz · 2025-03-05T14:51:32Z

docs/howto/multiple-storage-backends.md

+
+  <div markdown="1" id="on-prem">
+
+This example setup configures lakeFS to manage data across two separate MinIO instances.


Suggested change

This example setup configures lakeFS to manage data across two separate MinIO instances.

This example setup configures lakeFS to manage data across two separate MinIO instances:

ozkatz · 2025-03-05T14:51:49Z

docs/howto/multiple-storage-backends.md

+
+  <div markdown="2" id="multi-cloud">
+
+This example setup configures lakeFS to manage data across two public cloud providers: AWS and Azure.


Suggested change

This example setup configures lakeFS to manage data across two public cloud providers: AWS and Azure.

This example setup configures lakeFS to manage data across two public cloud providers: AWS and Azure:

ozkatz · 2025-03-05T14:52:34Z

docs/howto/multiple-storage-backends.md

+    * If static credentials are provided, lakeFS will use them. Otherwise, it will fall back to the AWS credentials chain. 
+      This means that for setups with multiple storages of type `s3`, static credentials are required for all but one.
+
+### Upgrading from Single to Multi-Store


Suggested change

### Upgrading from Single to Multi-Store

### Upgrading from a single storage backend to Multiple Storage backends

ozkatz · 2025-03-05T14:53:37Z

docs/howto/multiple-storage-backends.md

+The [Get Config](https://docs.lakefs.io/reference/api.html#/config/getConfig) API endpoint now returns a list of storage
+configurations. In multi-store setups, this is the recommended method to list connected storage backends and view their details.
+
+### Common Configuration Errors & Fixes


Suggested change

### Common Configuration Errors & Fixes

### Troubleshooting

ozkatz · 2025-03-05T14:54:43Z

docs/howto/multiple-storage-backends.md

+* Amazon S3
+* Local storage
+
+While this feature is designed to support any blockstore combination, testing for Azure and GCS in multi-store setups is 


I'd remove this line for now. It's not a limitation, it just wasn't tested.

guy-har

Added some comments, Approving in order not to be a blocker

guy-har · 2025-03-05T12:49:22Z

docs/howto/multiple-storage-backends.md

+---
+title: Multiple Storage Backends
+description: How to manage data across multiple storage systems with lakeFS 
+parent: How-To


Not sure this is the parent

I agree. I opened this task to move features to the right place on a separate pr.

guy-har · 2025-03-05T12:50:45Z

docs/howto/multiple-storage-backends.md

+{: .label .label-purple }
+
+{: .note}
+> Multi-storage backend support is only available for licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.<br>


Just a thought: Is this correct? IIUC it's not supported for them either

why not? it is available for licensed customers

guy-har · 2025-03-05T12:54:55Z

docs/howto/multiple-storage-backends.md

+With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including AWS S3,
+Azure Blob, Google Cloud Storage, other S3-compatible storage, and even local storages. 


Just a suggestion

Suggested change

With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including AWS S3,

Azure Blob, Google Cloud Storage, other S3-compatible storage, and even local storages.

With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including:

- AWS S3

- Azure Blob

- Google Cloud Storage

- other S3-compatible storage (such as minIO)

- local storage

guy-har · 2025-03-05T16:41:53Z

docs/howto/multiple-storage-backends.md

+* Use the new `blockstores` structure, **replacing** the existing `blockstore` configuration. Note that `blockstore` and `blockstores` 
+  configurations are mutually exclusive - lakeFS does not support both simultaneously. 
+* Define all previously available [single-blockstore settings](../reference/configuration.md#blockstore) under their respective storage backends.
+* The `signing.secret_key` remains a required global setting.


What do you mean by remains?

I changed wording, thx

guy-har · 2025-03-05T16:44:38Z

docs/howto/multiple-storage-backends.md

+* Restart the server.
+
+{: .warning}
+> Repositories linked to a removed storage backend will result in unexpected behavior. Ensure all necessary cleanup is done before removal.


lakeFS will fail to start, it's not un-expected

thx, changed it

Adding to Guy's comment - it will also make the repositories associated with the storage ID unusable

guy-har · 2025-03-05T16:47:17Z

docs/howto/multiple-storage-backends.md

+The [Get Config](https://docs.lakefs.io/reference/api.html#/config/getConfig) API endpoint now returns a list of storage
+configurations. In multi-store setups, this is the recommended method to list connected storage backends and view their details.


Not sure if you want to be concrete here, but the API returns both a string and a list and the user should use the relevant

guy-har · 2025-03-05T16:48:26Z

docs/howto/multiple-storage-backends.md

+* Restart the server.
+
+{: .warning}
+> Repositories linked to a removed storage backend will result in unexpected behavior. Ensure all necessary cleanup is done before removal.


I think we would like this kind of warning to replacing IDs, it's mentioned above but maybe we would like to emphesize

guy-har · 2025-03-05T16:51:51Z

docs/howto/multiple-storage-backends.md

+### Unsupported storage backends
+
+Multi-storage backend support has been validated on:
+* Self-managed S3-compatible object storage (e.g., MinIO)


I don't think e.g fits here, it's only tested on MinIO

guy-har · 2025-03-05T16:54:20Z

docs/howto/multiple-storage-backends.md

+still in progress.
+
+{: .note}
+> **Note:** Other untested combinations may still work. You are encouraged to try them and share feedback.


I would replace the encouraged to try them, prefer something like if you are interested please contact us.
No one can really try it

good call, done

Strongly agree

nopcoder · 2025-03-05T16:42:45Z

docs/enterprise/index.md

@@ -34,8 +34,11 @@ With lakeFS Enterprise you’ll receive access to the security package containin

 ## What additional functionality does lakeFS Enterprise provide?

-1. [lakeFS Mount]({% link reference/mount.md %}) allows users to virtually mount a remote lakeFS repository onto a local directory. Once mounted, users can access the data as if it resides on their local filesystem, using any tool, library, or framework that reads from a local filesystem.
+1. [lakeFS Mount]({% link reference/mount.md %}) - allows users to virtually mount a remote lakeFS repository onto a local directory. Once mounted, users can access the data as if it resides on their local filesystem, using any tool, library, or framework that reads from a local filesystem.


depends on how we communicate mount - we mount a branch

This is outside the scope of this prd, let's differ for later?

nopcoder · 2025-03-05T16:46:02Z

docs/assets/img/msb/msb_repo_settings_ui.png

Probably a different issue - the docs we specify the additional field as "stoarge id" and in the ui is is labeled "storage"

nopcoder · 2025-03-05T16:51:08Z

docs/howto/multiple-storage-backends.md

+for all organizational data assets, which is especially critical in AI/ML environments that rely on diverse datasets stored
+in multiple locations.
+
+With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including AWS S3,


multi-storage backend

nopcoder · 2025-03-05T16:52:22Z

docs/howto/multiple-storage-backends.md

+For a complete list of available options, refer to the [server configuration reference](../reference/configuration.md#blockstores).   
+
+{: .note}
+> **Note:** If you're upgrading from a single-store lakeFS setup, refer to the [upgrade guidelines](#upgrading-from-single-to-multi-store)


I think we need to use the same term "multi-storage" instead of store in all the doc

nopcoder · 2025-03-05T16:52:49Z

docs/howto/multiple-storage-backends.md

+### Listing Connected Storage Backends
+
+The [Get Config](https://docs.lakefs.io/reference/api.html#/config/getConfig) API endpoint now returns a list of storage
+configurations. In multi-store setups, this is the recommended method to list connected storage backends and view their details.


multi-storage

nopcoder · 2025-03-05T17:03:28Z

docs/howto/multiple-storage-backends.md

+**Note**: The `--storage-id` flag is currently hidden in the CLI. 
+* UI: Select a storage backend from the dropdown menu.
+![create repo with storage id](../assets/img/msb/msb_create_repo_ui.png)
+* High-level Python SDK: TODO


do we need to include todo here? is it not limitation or we can update the docs

nopcoder · 2025-03-05T17:05:02Z

docs/howto/multiple-storage-backends.md

+### Viewing Repository Details
+
+To check which storage backend is associated with a repository:
+* API – The [List Repositories](https://docs.lakefs.io/reference/api.html#/repositories/listRepositories) response includes the storage ID.


the link needs to be to a markdown related to the doc

nopcoder · 2025-03-05T17:06:09Z

docs/reference/configuration.md

+* `blockstores.stores[].s3.pre_signed_endpoint` `(string : )` - Custom endpoint for pre-signed URLs.
+* `blockstores.stores[].s3.disable_pre_signed` `(bool : false)` - Disable use of pre-signed URL.
+* `blockstores.stores[].s3.disable_pre_signed_ui` `(bool : true)` - Disable use of pre-signed URL in the UI.
+* `blockstores.stores[].s3.disable_pre_signed_multipart` `(bool : )` - Disable use of pre-signed multipart upload **experimental**, enabled on s3 block adapter with presign support.


Suggested change

* `blockstores.stores[].s3.disable_pre_signed_multipart` `(bool : )` - Disable use of pre-signed multipart upload **experimental**, enabled on s3 block adapter with presign support.

* `blockstores.stores[].s3.disable_pre_signed_multipart` `(bool : )` - Disable use of pre-signed multipart upload **experimental**, enabled on S3 block adapter with presign support.

nopcoder · 2025-03-05T17:19:57Z

docs/howto/multiple-storage-backends.md

+  * Access data across multiple storage backends using a single, consistent [URI format](../understand/model.md#lakefs-protocol-uris).
+
+3. **Centralized Access Control & Governance**:
+   * Access permissions and policies can be centrally managed across all connected storage systems using lakeFS [RBAC](../security/rbac.md).


note sure we need to specify - but we currently do not provide a mechanism to control access based on storage id

Thanks. I think that we shouldn't specify this, at least not here where we are walking through the high level use cases.
for now, I'd like not to mention this

nopcoder · 2025-03-05T17:20:51Z

docs/howto/multiple-storage-backends.md

+blockstores:
+  signing:
+    secret_key: "some-secret"
+    stores:


wrong level

This is for stores and everything below it.

done, thank you!

not require to block on docs changes

talSofer · 2025-03-05T09:17:40Z

docs/howto/multiple-storage-backends.md

+**Note**: The `--storage-id` flag is currently hidden in the CLI. 
+* UI: Select a storage backend from the dropdown menu.
+![create repo with storage id](../assets/img/msb/msb_create_repo_ui.png)
+* High-level Python SDK: TODO


@N-o-Z can you please assist in completing this part?

docs/howto/multiple-storage-backends.md

talSofer · 2025-03-05T15:41:35Z

docs/howto/multiple-storage-backends.md

+
+  <div markdown="1" id="on-prem">
+
+This example setup configures lakeFS to manage data across two separate MinIO instances.


talSofer · 2025-03-05T16:02:00Z

docs/howto/multiple-storage-backends.md

+
+This example setup configures lakeFS to manage data across two separate MinIO instances.
+
+```yaml


talSofer · 2025-03-05T16:04:22Z

docs/howto/multiple-storage-backends.md

+
+  <div markdown="2" id="multi-cloud">
+
+This example setup configures lakeFS to manage data across two public cloud providers: AWS and Azure.


talSofer · 2025-03-05T17:04:02Z

docs/howto/multiple-storage-backends.md

+With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including AWS S3,
+Azure Blob, Google Cloud Storage, other S3-compatible storage, and even local storages. 


talSofer · 2025-03-05T17:11:23Z

docs/howto/multiple-storage-backends.md

+* Use the new `blockstores` structure, **replacing** the existing `blockstore` configuration. Note that `blockstore` and `blockstores` 
+  configurations are mutually exclusive - lakeFS does not support both simultaneously. 
+* Define all previously available [single-blockstore settings](../reference/configuration.md#blockstore) under their respective storage backends.
+* The `signing.secret_key` remains a required global setting.


I changed wording, thx

talSofer · 2025-03-05T17:12:13Z

docs/howto/multiple-storage-backends.md

+* Restart the server.
+
+{: .warning}
+> Repositories linked to a removed storage backend will result in unexpected behavior. Ensure all necessary cleanup is done before removal.


thx, changed it

talSofer · 2025-03-05T17:24:02Z

docs/howto/multiple-storage-backends.md

+still in progress.
+
+{: .note}
+> **Note:** Other untested combinations may still work. You are encouraged to try them and share feedback.


good call, done

talSofer · 2025-03-05T17:28:39Z

docs/howto/multiple-storage-backends.md

+* Restart the server.
+
+{: .warning}
+> Repositories linked to a removed storage backend will result in unexpected behavior. Ensure all necessary cleanup is done before removal.


N-o-Z

Very detailed and well phrased.
Some comments/suggestions

N-o-Z · 2025-03-05T17:14:09Z

docs/howto/multiple-storage-backends.md

+1. **Distributed Data Management**:
+   * Eliminate data silos and enable seamless cross-cloud collaboration.
+   * Maintain version control across different storage providers for consistency and reproducibility.
+   * Ideal for AI/ML environments where datasets are distributed across multiple storage locations.


Is this for SEO purposes?
I think removing the AI/ML makes this bullet more inclusive

Not for SEO but to make it clear to what we think is our target customers.
I think that making this more inclusive is not necessarily a good thing here. I want people to easily understand what's in it for them and that's one way

N-o-Z · 2025-03-05T17:15:54Z

docs/howto/multiple-storage-backends.md

+To configure your lakeFS server to connect to multiple storage backends, define them under the `blockstores`
+section in your server configurations. The `blockstores.stores` field is an array of storage backends, each with its own configuration.  


Suggested change

To configure your lakeFS server to connect to multiple storage backends, define them under the `blockstores`

section in your server configurations. The `blockstores.stores` field is an array of storage backends, each with its own configuration.

To configure your lakeFS server to connect to multiple storage backends, define them under the `blockstores` section in your server configurations.

The `blockstores.stores` field is an array of storage backends, each with its own configuration.

N-o-Z · 2025-03-05T17:19:29Z

docs/howto/multiple-storage-backends.md

+* S3 Authentication Handling:
+    * All standard S3 authentication methods are supported.
+    * If static credentials are provided, lakeFS will use them. Otherwise, it will fall back to the AWS credentials chain. 
+      This means that for setups with multiple storages of type `s3`, static credentials are required for all but one.


Maybe we want to caveat that with the usage of profiles
Not necessarily defined as static but still can be used to define multiple aws blockstores

N-o-Z · 2025-03-05T17:20:56Z

docs/howto/multiple-storage-backends.md

+  configurations are mutually exclusive - lakeFS does not support both simultaneously. 
+* Define all previously available [single-blockstore settings](../reference/configuration.md#blockstore) under their respective storage backends.
+* The `signing.secret_key` remains a required global setting.
+* Set `backward_compatible: true` for the existing storage backend to ensure:


I feel like we need to be much more explicit here to the fact that things will not work correctly without it and that it cannot be removed

good call, I added your points

N-o-Z · 2025-03-05T17:23:54Z

docs/howto/multiple-storage-backends.md

+* Restart the server.
+
+{: .warning}
+> Repositories linked to a removed storage backend will result in unexpected behavior. Ensure all necessary cleanup is done before removal.


Adding to Guy's comment - it will also make the repositories associated with the storage ID unusable

N-o-Z · 2025-03-05T17:33:08Z

docs/howto/multiple-storage-backends.md

+### Importing Data into a Repository
+
+Importing data into a repository is supported when the following conditions are met:
+* The credentials used for the repository's backing blockstore allow access to the storage location.


Suggested change

* The credentials used for the repository's backing blockstore allow access to the storage location.

* The credentials used for the repository's backing blockstore allow access (read, list) to the storage location.

N-o-Z · 2025-03-05T17:33:35Z

docs/howto/multiple-storage-backends.md

+
+Importing data into a repository is supported when the following conditions are met:
+* The credentials used for the repository's backing blockstore allow access to the storage location.
+* The storage location is in the same region as the repository's backend.


I don't believe that is true. I don't think we enforce region

N-o-Z · 2025-03-05T17:34:02Z

docs/howto/multiple-storage-backends.md

+still in progress.
+
+{: .note}
+> **Note:** Other untested combinations may still work. You are encouraged to try them and share feedback.


Strongly agree

N-o-Z · 2025-03-05T17:37:17Z

docs/reference/configuration.md

@@ -211,6 +211,70 @@ Configuration section when using `database.type="local"`
 * `blockstore.gs.server_side_encryption_customer_supplied` `(string : )` - Server side encryption with AES key in hex format, exclusive with key ID below
 * `blockstore.gs.server_side_encryption_kms_key_id` `(string : )` - Server side encryption KMS key ID, exclusive with above

+### blockstores


I think that the best thing we could do will be to create a separate documentation for enterprise configurations. However, if not possible perhaps separate the configuration documentation clearly and have an "Enterprise" section where the enterprise configurations are detailed.
I wouldn't want as an OSS reader to weave through enterprise configuration as I'm trying to understand what configurations I need

I like the idea of separating Enterprise only configurations until we have separate docs - thanks!

itaigilo

Thanks for this great doc @talSofer !

Added some comments.

itaigilo · 2025-03-05T17:34:18Z

docs/enterprise/index.md

 2. [Transactional Mirroring]({% link howto/mirroring.md %}) - allows replicating lakeFS repositories into consistent read-only copies in remote locations.
+3. [Multiple Storage Backends]({% link howto/multiple-storage-backends.md %}) - allows managing data stored across multiple storage locations: on-prem, hybrid, or multi-cloud.        
+
+<br>


Nit: <br> is html, shouldn't belong in md.

itaigilo · 2025-03-05T17:35:48Z

docs/howto/multiple-storage-backends.md

+parent: How-To
+---
+
+# Multi-Storage Backend Support


Suggested change

# Multi-Storage Backend Support

# Multi-Storage Backend

itaigilo · 2025-03-05T17:37:32Z

docs/howto/multiple-storage-backends.md

+
+{% include toc.html %}
+
+## What is Multi-storage Backend Support? 


Here also - every feature can have Support in it.
I think we should simply use Multi-storage Backend.

I disagree. I changed the title but the sentence "What is Multi-storage Backend?" has different answer than I expect for "What is Multi-storage Backend support?" the later is related to what it means in lakeFS

itaigilo · 2025-03-05T17:38:37Z

docs/howto/multiple-storage-backends.md

+
+## What is Multi-storage Backend Support? 
+
+lakeFS multi-storage backend support enables seamless data management across multiple storage systems — 


Suggested change

lakeFS multi-storage backend support enables seamless data management across multiple storage systems —

Using lakeFS multi-storage backend enables seamless data management across multiple storage systems —

disagree here too

itaigilo · 2025-03-05T17:47:33Z

docs/howto/multiple-storage-backends.md

+   * Ideal for AI/ML environments where datasets are distributed across multiple storage locations.
+
+2. **Unified Data Access**:
+  * Access data across multiple storage backends using a single, consistent [URI format](../understand/model.md#lakefs-protocol-uris).


Nit: this has a different indentation than 1 and 3,

docs/howto/multiple-storage-backends.md

itaigilo · 2025-03-05T18:06:08Z

docs/howto/multiple-storage-backends.md

+After setting up lakeFS Enterprise to connect with multiple storage backends, this section explains how to use these 
+connected storages when working with lakeFS.
+
+With multiple storage backends configured, lakeFS repositories are now linked to a specific storage backend. Together with


Suggested change

With multiple storage backends configured, lakeFS repositories are now linked to a specific storage backend. Together with

With multiple storage backends configured, lakeFS repositories are now linked to a specific storage. Together with

itaigilo · 2025-03-05T18:07:21Z

docs/howto/multiple-storage-backends.md

+
+In a multi-store setup, users must specify a storage ID when creating a repository. This can be done using the following methods:
+* API: Use the `storage_id` parameter in the [Create Repository endpoint](https://docs.lakefs.io/reference/api.html#/repositories/createRepository).  
+* CLI: Use the `storage-id` flag with the [repo create](../reference/cli.md#lakectl-repo-create) command:


--storage-id maybe?

itaigilo · 2025-03-05T18:09:54Z

docs/howto/multiple-storage-backends.md

+### Creating a Repository
+
+In a multi-store setup, users must specify a storage ID when creating a repository. This can be done using the following methods:
+* API: Use the `storage_id` parameter in the [Create Repository endpoint](https://docs.lakefs.io/reference/api.html#/repositories/createRepository).  


Suggestion - make these (API, CLI, WebUI, Py HL SDK) to headers:

#### API Use the... ### CLI Use the... ...

itaigilo · 2025-03-05T18:11:02Z

docs/howto/multiple-storage-backends.md

+
+## Limitations 
+
+### Unsupported storage backends


Suggested change

### Unsupported storage backends

### Supported storages

agree, done

arielshaqed

Thanks! Impressive path through all the docs...

arielshaqed · 2025-03-06T06:54:29Z

docs/howto/multiple-storage-backends.md

+{: .label .label-purple }
+
+{: .note}
+> Multi-storage backend support is only available for licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.<br>


Nit:

Suggested change

> Multi-storage backend support is only available for licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.<br>

> Multi-storage backend support is only available to licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.

arielshaqed · 2025-03-06T06:58:30Z

docs/howto/multiple-storage-backends.md

+## What is Multi-storage Backend Support? 
+
+lakeFS multi-storage backend support enables seamless data management across multiple storage systems — 
+on-premises, across public clouds, or hybrid environments. This capability makes lakeFS a unified data management platform


Cross-cloud support may be a trap. Obviously it will work, but most use cases will involve paying egress fees to at least one of the cloud providers. Do we want to go there with our customers?

Good point, this feature sets the grounds for supporting cross cloud in a way that is egress fees aware - I can now manage cross cloud data for a single platform. making it more efficient in cost is another (very reasonable) feature to build

arielshaqed · 2025-03-06T07:02:42Z

docs/howto/multiple-storage-backends.md

+
+### Example Configurations
+
+<div class="tabs">


"tabs"... this is some CSS thing that renders multiple selectable tabs.

arielshaqed · 2025-03-06T07:04:46Z

docs/howto/multiple-storage-backends.md

+            access_key_id: "prod_access_key"
+            secret_access_key: "prod_secret_key"
+      - id: "minio-backup"
+        description: "Backup MinIO storage for disaster recovery"


We should document how disaster recovery works.

Can you please elaborate on that?

arielshaqed · 2025-03-06T07:07:01Z

docs/howto/multiple-storage-backends.md

+* Define all previously available [single-blockstore settings](../reference/configuration.md#blockstore) under their respective storage backends.
+* The `signing.secret_key` is a required setting global to all connected stores.
+* Set `backward_compatible: true` for the existing storage backend to ensure:
+  * Existing repositories continue using the original storage backend.


Suggested change

* Existing repositories continue using the original storage backend.

* Existing repositories continue to use the original storage backend.

arielshaqed · 2025-03-06T07:12:42Z

docs/howto/multiple-storage-backends.md

+* The `signing.secret_key` is a required setting global to all connected stores.
+* Set `backward_compatible: true` for the existing storage backend to ensure:
+  * Existing repositories continue using the original storage backend.
+  * Newly created repositories default to this backend unless explicitly assigned a different one, to ensure a non-breaking upgrade process. 


We are re-opening the whole default vs. b/c debate here. I would much prefer not to say "upgrade" here, given that upgrades might not be possible when using external tools, or for customers whose users do not know their storage IDs.

@arielshaqed i'm not sure I understand your suggestion. can you please help me understand?

arielshaqed · 2025-03-06T07:14:56Z

docs/howto/multiple-storage-backends.md

+
+### Creating a Repository
+
+In a multi-store setup, users must specify a storage ID when creating a repository. This can be done using the following methods:


Typically we put equivalent methods (CLI / API / UI / ...) in tabs.

Do we want do document how to do this in HTTP REST (which many people seem to do), or in the Java SDK?

Typically we put equivalent methods (CLI / API / UI / ...) in tabs.

great idea - I changed it!

Do we want do document how to do this in HTTP REST (which many people seem to do), or in the Java SDK?

I'm differing for later if ok with you

itaigilo

Great detailed doc 👍

talSofer added 18 commits March 2, 2025 11:52

msb docs outline

89a1793

msb intro

488b649

tmp

8fdae1f

tmp2

a78bdf1

tmp2

963d117

use cases

074ab4d

partial configurations

6e94ced

configurations

de4e87a

add/remove

478fc86

working with repos

fe951f1

change menu title

8508261

working with repos + limitations

10ee3a6

storage listing

56d5be2

server configuration reference

e6cb80c

limitations

37d5e4f

reference msb accross docs

4dc2747

reference msb accross docs

919e9c5

lakeFS version

2652a42

talSofer added docs Improvements or additions to documentation exclude-changelog PR description should not be included in next release changelog labels Mar 5, 2025

talSofer requested review from nopcoder, N-o-Z, arielshaqed, itaigilo and guy-har March 5, 2025 09:16

talSofer requested a review from ozkatz March 5, 2025 09:20

ozkatz approved these changes Mar 5, 2025

View reviewed changes

fix cr comments and add HL SDK example

39f33c4

guy-har approved these changes Mar 5, 2025

View reviewed changes

use default client

b68b504

nopcoder previously requested changes Mar 5, 2025

View reviewed changes

fix cr comments

2189bbc

talSofer commented Mar 5, 2025

View reviewed changes

N-o-Z reviewed Mar 5, 2025

View reviewed changes

itaigilo requested changes Mar 5, 2025

View reviewed changes

arielshaqed reviewed Mar 6, 2025

View reviewed changes

talSofer added 2 commits March 6, 2025 10:37

fix cr comments

77011bd

fix cr comments

412e401

talSofer requested a review from itaigilo March 6, 2025 09:43

talSofer added 2 commits March 6, 2025 11:59

tmp

3254ac4

fix cr comments

8b2f30f

itaigilo approved these changes Mar 6, 2025

View reviewed changes

talSofer merged commit c92f605 into master Mar 6, 2025
43 of 44 checks passed

talSofer deleted the docs/multi-storage-backends branch March 6, 2025 11:56


		This example setup configures lakeFS to manage data across two separate MinIO instances.

		```yaml


		<div markdown="1" id="on-prem">

		This example setup configures lakeFS to manage data across two separate MinIO instances.


		<div markdown="2" id="multi-cloud">

		This example setup configures lakeFS to manage data across two public cloud providers: AWS and Azure.

	### Upgrading from Single to Multi-Store
	### Upgrading from a single storage backend to Multiple Storage backends

		With a multi-store setup, lakeFS can connect to and manage any combination of supported storage systems, including AWS S3,
		Azure Blob, Google Cloud Storage, other S3-compatible storage, and even local storages.

		The [Get Config](https://docs.lakefs.io/reference/api.html#/config/getConfig) API endpoint now returns a list of storage
		configurations. In multi-store setups, this is the recommended method to list connected storage backends and view their details.

	* `blockstores.stores[].s3.disable_pre_signed_multipart` `(bool : )` - Disable use of pre-signed multipart upload experimental, enabled on s3 block adapter with presign support.
	* `blockstores.stores[].s3.disable_pre_signed_multipart` `(bool : )` - Disable use of pre-signed multipart upload experimental, enabled on S3 block adapter with presign support.

		To configure your lakeFS server to connect to multiple storage backends, define them under the `blockstores`
		section in your server configurations. The `blockstores.stores` field is an array of storage backends, each with its own configuration.

	* The credentials used for the repository's backing blockstore allow access to the storage location.
	* The credentials used for the repository's backing blockstore allow access (read, list) to the storage location.


		{% include toc.html %}

		## What is Multi-storage Backend Support?


		## What is Multi-storage Backend Support?

		lakeFS multi-storage backend support enables seamless data management across multiple storage systems —

	lakeFS multi-storage backend support enables seamless data management across multiple storage systems —
	Using lakeFS multi-storage backend enables seamless data management across multiple storage systems —

	With multiple storage backends configured, lakeFS repositories are now linked to a specific storage backend. Together with
	With multiple storage backends configured, lakeFS repositories are now linked to a specific storage. Together with

	> Multi-storage backend support is only available for licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.<br>
	> Multi-storage backend support is only available to licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.

	* Existing repositories continue using the original storage backend.
	* Existing repositories continue to use the original storage backend.


		### Creating a Repository

		In a multi-store setup, users must specify a storage ID when creating a repository. This can be done using the following methods:

Docs for multiple storage backend support #8758

Docs for multiple storage backend support #8758

Conversation

talSofer commented Mar 5, 2025 • edited Loading

github-actions bot commented Mar 5, 2025 • edited Loading

github-actions bot commented Mar 5, 2025

E2E Test Results - DynamoDB Local - Local Block Adapter

github-actions bot commented Mar 5, 2025

E2E Test Results - Quickstart

ozkatz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guy-har left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

N-o-Z left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

talSofer commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading