diff --git a/bottomless-cli/README.md b/bottomless-cli/README.md new file mode 100644 index 0000000000..326ad5a0e9 --- /dev/null +++ b/bottomless-cli/README.md @@ -0,0 +1,99 @@ +# Bottomless CLI + +The command-line interface helps inspect and manage replicated snapshots stored in S3-compatible storage. + +It can be installed as a standalone executable with: + +```sh +RUSTFLAGS="--cfg uuid_unstable" cargo install bottomless-cli +``` + +> [!WARNING] +> The [CLI crate](https://crates.io/crates/bottomless-cli) hasn't been updated in over a year, and while it's being heavily worked on, we recommend building against the source for now. + +In order to build from the repository: + +```sh +git clone git@github.com:tursodatabase/libsql.git +cargo build -p bottomless-cli --release +cargo install --path . +``` + +You will now be able to use the `bottomless-cli` command. + +Available commands: +``` +$ bottomless-cli +Bottomless CLI + +Usage: bottomless-cli [OPTIONS] + +Commands: + copy Copy bottomless generation locally + create Create new generation from database + ls List available generations + restore Restore the database + verify Verify integrity of the database + rm Remove given generation from remote storage + snapshot Generate and upload a snapshot for a given generation or timestamp + help Print this message or the help of the given subcommand(s) + +Options: + -e, --endpoint + -b, --bucket + -d, --database + -n, --namespace + --encryption-key + --db-name + -h, --help Print help +``` + +## Examples + +### Listing generations + +```sh +$ bottomless-cli -e http://localhost:9000 ls -v -l3 +e4eb3c21-ff53-7b2e-a6ea-ca396f4df9b1 + created at (UTC): 2022-12-23 08:24:52.500 + change counter: [0, 0, 0, 51] + consistent WAL frame: 0 + WAL frame checksum: 0 + main database snapshot: + object size: 408 + last modified: 2022-12-23T08:24:53Z + +e4eb3c22-0359-7af6-9acb-285ed7b6ed59 + created at (UTC): 2022-12-23 08:24:51.470 + change counter: [0, 0, 0, 51] + consistent WAL frame: 1 + WAL frame checksum: 5335f2a044d2f455 + main database snapshot: + object size: 399 + last modified: 2022-12-23T08:24:52Z + +e4eb3c22-0941-73eb-85df-4e8552a0e88c + created at (UTC): 2022-12-23 08:24:49.958 + change counter: [0, 0, 0, 50] + consistent WAL frame: 10 + WAL frame checksum: 6ac65882f9a2dba7 + main database snapshot: + object size: 401 + last modified: 2022-12-23T08:24:51Z +``` + +### Restoring the database + +```sh +$ RUST_LOG=info bottomless-cli -e http://localhost:9000 restore +2022-12-23T10:16:10.703557Z INFO bottomless::replicator: Bucket bottomless exists and is accessible +2022-12-23T10:16:10.709526Z INFO bottomless_cli: Database: test.db +2022-12-23T10:16:10.713070Z INFO bottomless::replicator: Restoring from generation e4eb3c29-fe84-7347-a0c0-b9a3a71d0fc2 +2022-12-23T10:16:10.727646Z INFO bottomless::replicator: Restored the main database file +``` + +### Removing old snapshots +```sh +$ bottomless-cli -e http://localhost:9000 rm -v --older-than 2022-12-15 +Removed 4 generations +``` diff --git a/bottomless/README.md b/bottomless/README.md index 5ac2764b94..3ac34f93a9 100644 --- a/bottomless/README.md +++ b/bottomless/README.md @@ -1,5 +1,7 @@ # Bottomless S3-compatible virtual WAL for libSQL -##### Work in heavy progress! + +> [!WARNING] +> This project is in heavy development! This project implements a virtual write-ahead log (WAL) which continuously backs up the data to S3-compatible storage and is able to restore it later. @@ -45,88 +47,13 @@ A short demo script is in `test/smoke_test.sh`, and can be executed with: LIBSQL_DIR=/path/to/your/libsql/directory make test ``` -## CLI -The command-line interface supports browsing, restoring and removing snapshot generations. -It can be installed as a standalone executable with: -```sh -RUSTFLAGS="--cfg uuid_unstable" cargo install bottomless-cli -``` -Alternatively, bottomless-cli is available from the repository by running `cargo run`. -Available commands: -``` -$ bottomless-cli --help -Bottomless CLI - -Usage: bottomless-cli [OPTIONS] - -Commands: - ls List available generations - restore Restore the database - rm Remove given generation from remote storage - help Print this message or the help of the given subcommand(s) - -Options: - -e, --endpoint - -b, --bucket - -d, --database - -h, --help Print help information -``` - -### Examples - -#### Listing generations -``` -[sarna@sarna-pc test]$ bottomless-cli -e http://localhost:9000 ls -v -l3 -e4eb3c21-ff53-7b2e-a6ea-ca396f4df9b1 - created at (UTC): 2022-12-23 08:24:52.500 - change counter: [0, 0, 0, 51] - consistent WAL frame: 0 - WAL frame checksum: 0 - main database snapshot: - object size: 408 - last modified: 2022-12-23T08:24:53Z - -e4eb3c22-0359-7af6-9acb-285ed7b6ed59 - created at (UTC): 2022-12-23 08:24:51.470 - change counter: [0, 0, 0, 51] - consistent WAL frame: 1 - WAL frame checksum: 5335f2a044d2f455 - main database snapshot: - object size: 399 - last modified: 2022-12-23T08:24:52Z - -e4eb3c22-0941-73eb-85df-4e8552a0e88c - created at (UTC): 2022-12-23 08:24:49.958 - change counter: [0, 0, 0, 50] - consistent WAL frame: 10 - WAL frame checksum: 6ac65882f9a2dba7 - main database snapshot: - object size: 401 - last modified: 2022-12-23T08:24:51Z -``` - -#### Restoring the database -``` -$ RUST_LOG=info bottomless-cli -e http://localhost:9000 restore -2022-12-23T10:16:10.703557Z INFO bottomless::replicator: Bucket bottomless exists and is accessible -2022-12-23T10:16:10.709526Z INFO bottomless_cli: Database: test.db -2022-12-23T10:16:10.713070Z INFO bottomless::replicator: Restoring from generation e4eb3c29-fe84-7347-a0c0-b9a3a71d0fc2 -2022-12-23T10:16:10.727646Z INFO bottomless::replicator: Restored the main database file -``` - -#### Removing old snapshots -``` -$ bottomless-cli -e http://localhost:9000 rm -v --older-than 2022-12-15 -Removed 4 generations -``` - ## Details All page writes committed to the database end up being asynchronously replicated to S3-compatible storage. On boot, if the main database file is empty, it will be restored with data coming from the remote storage. If the database file is newer, it will be uploaded to the remote location with a new generation number. If a local WAL file is present and detected to be newer than remote data, it will be uploaded as well. -### Tests +## Tests A fully local test can be performed by using a local S3-compatible server, e.g. [Minio](https://min.io/). Assuming the server is available at HTTP port 9000, you can use the following scripts: ```sh @@ -140,3 +67,6 @@ The `smoke_test` script sets up a new database in WAL mode and 64KiB page size - The `restore_test` script syncs with the replication server and fetches the newest database if necessary. Once `smoke_test` ran at least once, `restore_test` should always be able to fetch the database data, even if the local `test.db` file is removed. The same set of tests also work with remote servers. In case of AWS S3, just make sure that the AWS SDK credentials are valid and the user has permissions for managing the chosen bucket. + +## CLI +[See dedicated CLI's documentation]: ../bottomless-cli/README.md diff --git a/libsql-server/README.md b/libsql-server/README.md index 9700e23bc1..6ef5c3b616 100644 --- a/libsql-server/README.md +++ b/libsql-server/README.md @@ -96,21 +96,9 @@ LIBSQL_BOTTOMLESS_AWS_ACCESS_KEY_ID= # ... to set up auth, regions LIBSQL_BOTTOMLESS_AWS_REGION= # . ``` -### bottomless-cli +### Bottomless CLI -Replicated snapshots can be inspected and managed with the official command-line -interface. - -The tool can be installed via `cargo`: - -```bash -RUSTFLAGS='--cfg uuid_unstable' cargo install bottomless-cli -``` - -For usage examples and description, refer to the [bottomless-cli -documentation]. - -[bottomless-cli documentation]: ../bottomless#cli +[See dedicated CLI's documentation]: ../bottomless-cli/README.md ## License