Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(bottomless-cli): move CLI documentation to a single README #1707

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions bottomless-cli/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Bottomless CLI

The command-line interface helps inspect and manage replicated snapshots stored in S3-compatible storage.

It can be installed as a standalone executable with:

```sh
RUSTFLAGS="--cfg uuid_unstable" cargo install bottomless-cli
```

> [!WARNING]
> The [CLI crate](https://crates.io/crates/bottomless-cli) hasn't been updated in over a year, and while it's being heavily worked on, we recommend building against the source for now.
Copy link
Contributor Author

@jeremybarbet jeremybarbet Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and while it's being heavily worked on, we recommend building against the source for now

Just my assumption btw (related to #857)


In order to build from the repository:

```sh
git clone [email protected]:tursodatabase/libsql.git
cargo build -p bottomless-cli --release
cargo install --path .
```

You will now be able to use the `bottomless-cli` command.

Available commands:
```
$ bottomless-cli
Bottomless CLI

Usage: bottomless-cli [OPTIONS] <COMMAND>

Commands:
copy Copy bottomless generation locally
create Create new generation from database
ls List available generations
restore Restore the database
verify Verify integrity of the database
rm Remove given generation from remote storage
snapshot Generate and upload a snapshot for a given generation or timestamp
help Print this message or the help of the given subcommand(s)

Options:
-e, --endpoint <ENDPOINT>
-b, --bucket <BUCKET>
-d, --database <DATABASE>
-n, --namespace <NAMESPACE>
--encryption-key <ENCRYPTION_KEY>
--db-name <DB_NAME>
-h, --help Print help
```

## Examples

### Listing generations

```sh
$ bottomless-cli -e http://localhost:9000 ls -v -l3
e4eb3c21-ff53-7b2e-a6ea-ca396f4df9b1
created at (UTC): 2022-12-23 08:24:52.500
change counter: [0, 0, 0, 51]
consistent WAL frame: 0
WAL frame checksum: 0
main database snapshot:
object size: 408
last modified: 2022-12-23T08:24:53Z

e4eb3c22-0359-7af6-9acb-285ed7b6ed59
created at (UTC): 2022-12-23 08:24:51.470
change counter: [0, 0, 0, 51]
consistent WAL frame: 1
WAL frame checksum: 5335f2a044d2f455
main database snapshot:
object size: 399
last modified: 2022-12-23T08:24:52Z

e4eb3c22-0941-73eb-85df-4e8552a0e88c
created at (UTC): 2022-12-23 08:24:49.958
change counter: [0, 0, 0, 50]
consistent WAL frame: 10
WAL frame checksum: 6ac65882f9a2dba7
main database snapshot:
object size: 401
last modified: 2022-12-23T08:24:51Z
```

### Restoring the database

```sh
$ RUST_LOG=info bottomless-cli -e http://localhost:9000 restore
2022-12-23T10:16:10.703557Z INFO bottomless::replicator: Bucket bottomless exists and is accessible
2022-12-23T10:16:10.709526Z INFO bottomless_cli: Database: test.db
2022-12-23T10:16:10.713070Z INFO bottomless::replicator: Restoring from generation e4eb3c29-fe84-7347-a0c0-b9a3a71d0fc2
2022-12-23T10:16:10.727646Z INFO bottomless::replicator: Restored the main database file
```

### Removing old snapshots
```sh
$ bottomless-cli -e http://localhost:9000 rm -v --older-than 2022-12-15
Removed 4 generations
```
84 changes: 7 additions & 77 deletions bottomless/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Bottomless S3-compatible virtual WAL for libSQL
##### Work in heavy progress!

> [!WARNING]
> This project is in heavy development!

This project implements a virtual write-ahead log (WAL) which continuously backs up the data to S3-compatible storage and is able to restore it later.

Expand Down Expand Up @@ -45,88 +47,13 @@ A short demo script is in `test/smoke_test.sh`, and can be executed with:
LIBSQL_DIR=/path/to/your/libsql/directory make test
```

## CLI
The command-line interface supports browsing, restoring and removing snapshot generations.
It can be installed as a standalone executable with:
```sh
RUSTFLAGS="--cfg uuid_unstable" cargo install bottomless-cli
```
Alternatively, bottomless-cli is available from the repository by running `cargo run`.
Available commands:
```
$ bottomless-cli --help
Bottomless CLI

Usage: bottomless-cli [OPTIONS] <COMMAND>

Commands:
ls List available generations
restore Restore the database
rm Remove given generation from remote storage
help Print this message or the help of the given subcommand(s)

Options:
-e, --endpoint <ENDPOINT>
-b, --bucket <BUCKET>
-d, --database <DATABASE>
-h, --help Print help information
```

### Examples

#### Listing generations
```
[sarna@sarna-pc test]$ bottomless-cli -e http://localhost:9000 ls -v -l3
e4eb3c21-ff53-7b2e-a6ea-ca396f4df9b1
created at (UTC): 2022-12-23 08:24:52.500
change counter: [0, 0, 0, 51]
consistent WAL frame: 0
WAL frame checksum: 0
main database snapshot:
object size: 408
last modified: 2022-12-23T08:24:53Z

e4eb3c22-0359-7af6-9acb-285ed7b6ed59
created at (UTC): 2022-12-23 08:24:51.470
change counter: [0, 0, 0, 51]
consistent WAL frame: 1
WAL frame checksum: 5335f2a044d2f455
main database snapshot:
object size: 399
last modified: 2022-12-23T08:24:52Z

e4eb3c22-0941-73eb-85df-4e8552a0e88c
created at (UTC): 2022-12-23 08:24:49.958
change counter: [0, 0, 0, 50]
consistent WAL frame: 10
WAL frame checksum: 6ac65882f9a2dba7
main database snapshot:
object size: 401
last modified: 2022-12-23T08:24:51Z
```

#### Restoring the database
```
$ RUST_LOG=info bottomless-cli -e http://localhost:9000 restore
2022-12-23T10:16:10.703557Z INFO bottomless::replicator: Bucket bottomless exists and is accessible
2022-12-23T10:16:10.709526Z INFO bottomless_cli: Database: test.db
2022-12-23T10:16:10.713070Z INFO bottomless::replicator: Restoring from generation e4eb3c29-fe84-7347-a0c0-b9a3a71d0fc2
2022-12-23T10:16:10.727646Z INFO bottomless::replicator: Restored the main database file
```

#### Removing old snapshots
```
$ bottomless-cli -e http://localhost:9000 rm -v --older-than 2022-12-15
Removed 4 generations
```

## Details
All page writes committed to the database end up being asynchronously replicated to S3-compatible storage.
On boot, if the main database file is empty, it will be restored with data coming from the remote storage.
If the database file is newer, it will be uploaded to the remote location with a new generation number.
If a local WAL file is present and detected to be newer than remote data, it will be uploaded as well.

### Tests
## Tests
A fully local test can be performed by using a local S3-compatible server, e.g. [Minio](https://min.io/). Assuming the server is available at HTTP port 9000,
you can use the following scripts:
```sh
Expand All @@ -140,3 +67,6 @@ The `smoke_test` script sets up a new database in WAL mode and 64KiB page size -
The `restore_test` script syncs with the replication server and fetches the newest database if necessary. Once `smoke_test` ran at least once, `restore_test` should always be able to fetch the database data, even if the local `test.db` file is removed.

The same set of tests also work with remote servers. In case of AWS S3, just make sure that the AWS SDK credentials are valid and the user has permissions for managing the chosen bucket.

## CLI
[See dedicated CLI's documentation]: ../bottomless-cli/README.md
16 changes: 2 additions & 14 deletions libsql-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,21 +96,9 @@ LIBSQL_BOTTOMLESS_AWS_ACCESS_KEY_ID= # ... to set up auth, regions
LIBSQL_BOTTOMLESS_AWS_REGION= # .
```

### bottomless-cli
### Bottomless CLI

Replicated snapshots can be inspected and managed with the official command-line
interface.

The tool can be installed via `cargo`:

```bash
RUSTFLAGS='--cfg uuid_unstable' cargo install bottomless-cli
```

For usage examples and description, refer to the [bottomless-cli
documentation].

[bottomless-cli documentation]: ../bottomless#cli
[See dedicated CLI's documentation]: ../bottomless-cli/README.md

## License

Expand Down