Skip to content

Commit

Permalink
Various documentation updates (awslabs#129)
Browse files Browse the repository at this point in the history
* Various documentation updates

- Document how to configure logging
- Update contribution guidelines to point to logging docs and generally
  be more specific to Mountpoint
- Update readme with more details about the alpha release
- Update semantics with more notes about concurrent mutations

Signed-off-by: James Bornholt <[email protected]>

* This has been bothering me

Signed-off-by: James Bornholt <[email protected]>

---------

Signed-off-by: James Bornholt <[email protected]>
  • Loading branch information
jamesbornholt authored Mar 13, 2023
1 parent 82d8e03 commit 3590a0f
Show file tree
Hide file tree
Showing 6 changed files with 87 additions and 39 deletions.
4 changes: 1 addition & 3 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,4 @@

---

By submitting this pull request,
I confirm that my contribution is made under the terms of the Apache 2.0 license
and I agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/).
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/).
45 changes: 33 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
## Mountpoint for Amazon S3
# Mountpoint for Amazon S3

![CI Status](https://github.com/awslabs/mountpoint-s3/actions/workflows/rust.yml/badge.svg?branch=main)

Mountpoint for Amazon S3 is a simple, high-throughput connector for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through basic file operations like `open` and `read`. Mountpoint for Amazon S3 gives these applications access to the elastic storage and throughput of Amazon S3 without rewriting them to use AWS SDKs and object APIs.
Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like `open` and `read`. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.

**This is an alpha release and not yet ready for production use.** We're especially interested in early feedback on missing features, performance, and compatibility. See [Current Status](#current-status) for more limitations.
Mountpoint for Amazon S3 is optimized for read-heavy workloads that need high throughput. It intentionally does not implement the full POSIX specification for file systems. See [SEMANTICS.md](doc/SEMANTICS.md) for a detailed description of the client's behavior and POSIX support.

**This is an alpha release and not yet ready for production use.** We're especially interested in early feedback on features, performance, and compatibility. See [Current status](#current-status) for more limitations.

## Getting started

Mountpoint for Amazon S3 is currently only available by building from source, and only supports Linux. These instructions are for Amazon Linux 2, but other Linux distributions should be similar.
The alpha release of Mountpoint for Amazon S3 is only available by building from source, and only supports Linux. These instructions are for Amazon Linux 2, but other Linux distributions should be similar.

First, install dependencies, including the Rust compiler via [rustup](https://rustup.rs/):

Expand All @@ -19,36 +21,55 @@ Now clone this repository and its submodules:

git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git

Finally, compile the connector:
Finally, compile the client:

cd mountpoint-s3
cargo build --release

The final binary will be at `target/release/mount-s3`.

To use the connector, first ensure you have [access to valid AWS credentials](https://docs.aws.amazon.com/sdkref/latest/guide/access.html). For example, you could [create a new IAM user and add it to the `~/.aws/credentials` file](https://docs.aws.amazon.com/sdkref/latest/guide/access-users.html), or [configure an EC2 instance with an IAM role](https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-roles-for-ec2.html).
To use the client, first ensure you have [access to valid AWS credentials](https://docs.aws.amazon.com/sdkref/latest/guide/access.html). For example, you could [create a new IAM user and add it to the `~/.aws/credentials` file](https://docs.aws.amazon.com/sdkref/latest/guide/access-users.html), or [configure an EC2 instance with an IAM role](https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-roles-for-ec2.html).

Then run the connector, specifying the directory in which your S3 bucket should be mounted (here, `~/mnt`):
Then run the client, specifying the directory in which your S3 bucket should be mounted (here, `~/mnt`):

mkdir ~/mnt
mount-s3 my-s3-bucket-name ~/mnt

The `~/mnt` directory now gives access to the objects in your S3 bucket.
The client will run in the background by default, and the `~/mnt` directory now gives access to the objects in your S3 bucket.

### Configuration

For the alpha release, some additional configuration is required to use Mountpoint for Amazon S3 in some circumstances:

* **High-bandwidth EC2 instances**: manually specify the available network bandwidth in Gbps with the `--throughput-target-gbps` command-line argument. Defaults to 10 Gbps if not configured. This configuration will be automated in the future (https://github.com/awslabs/mountpoint-s3/issues/3).
* **AWS GovCloud, China (Beijing), and China (Ningxia) regions**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument **and** the region name with the `--region` argument. For example, for the AWS GovCloud (US-West) region, specify `--endpoint-url https://s3.us-gov-west-1.amazonaws.com --region us-gov-west-1`. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These regions will be supported via only the `--region` command-line argument in the future (https://github.com/awslabs/mountpoint-s3/issues/4).
* **Access points**: use the [bucket-style access point alias](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html) for the access point as the bucket name to mount.
* **FIPS, dualstack, and transfer acceleration endpoints**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These endpoints will be configurable in the future (https://github.com/awslabs/mountpoint-s3/issues/4).

## Current status

Mountpoint for Amazon S3 is optimized for read-heavy workloads that need high throughput. It intentionally does not implement the full POSIX specification for file systems. See [SEMANTICS.md](doc/SEMANTICS.md) for a detailed description of the connector's behavior and POSIX support.
Mountpoint for Amazon S3 is currently an alpha release and should not be used in production. We're tracking its production readiness and future features on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We welcome suggestions for new features or use cases!

The [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84) gives an overview of features we're planning to support in the future.
Our thinking around features is guided by our [tenets for Mountpoint for Amazon S3](doc/SEMANTICS.md#semantics-tenets). We don't intend to implement (and likely won't accept contributions of) features that cannot be efficiently executed against S3's object APIs. This rules out Mountpoint for Amazon S3 supporting some POSIX file system functionality like ACLs or file locking. For workloads that need these features, we recommend AWS fully managed file system services like [Amazon FSx for Lustre](https://aws.amazon.com/fsx/lustre/) and its [support for linking S3 buckets](https://docs.aws.amazon.com/fsx/latest/LustreGuide/create-dra-linked-data-repo.html).

## Contributing

We welcome contributions to Mountpoint for Amazon S3! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for more information on how to report bugs or submit pull requests.
We welcome contributions to Mountpoint for Amazon S3! Please see [CONTRIBUTING.md](doc/CONTRIBUTING.md) for more information on how to report bugs or submit pull requests, and [LOGGING.md](doc/LOGGING.md) for details on how to capture logging data for bug reports.

Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.

### Security

If you discover a potential security issue in this project we ask that you notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue.

### Code of conduct

This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). See [CODE_OF_CONDUCT.md](doc/CODE_OF_CONDUCT.md) for more details.

## License

This project is licensed under the Apache-2.0 License.
This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily:
* The [fuser](https://github.com/cberner/fuser) bindings for FUSE (MIT license)
* The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache-2.0 license)

See [deny.toml](deny.toml) for a list of licenses used by our dependencies.
File renamed without changes.
23 changes: 10 additions & 13 deletions CONTRIBUTING.md → doc/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
# Contributing Guidelines

Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
documentation, we greatly value feedback and contributions from our community.
Thank you for your interest in contributing to Mountpoint for Amazon S3. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community.

Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
information to effectively respond to your bug report or contribution.


## Reporting Bugs/Feature Requests

We welcome you to use the GitHub issue tracker to report bugs or suggest features.
Please use the GitHub issue tracker to [report bugs](https://github.com/awslabs/mountpoint-s3/issues/new?assignees=&labels=bug&template=bug-report.yml) or [suggest features](https://github.com/awslabs/mountpoint-s3/issues/new?assignees=&labels=enhancement&template=feature-request.yml).

When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
Expand All @@ -19,21 +17,23 @@ reported the issue. Please try to include as much information as you can. Detail
* Any modifications you've made relevant to the bug
* Anything unusual about your environment or deployment

Logs are also valuable for bug reports. Please read the [logging documentation](LOGGING.md) for details on how to capture logs.

## Contributing via Pull Requests

Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:

1. You are working against the latest source on the *main* branch.
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
3. You open an issue to discuss any significant work -- we would hate for your time to be wasted.

To send us a pull request, please:

1. Fork the repository.
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
2. Modify the source. Please focus on the specific change you are contributing; we value multiple small pull requests that are easier to review.
3. Ensure local tests pass.
4. Commit to your fork using clear commit messages.
5. Send us a pull request, answering any default questions in the pull request interface.
5. Send us a pull request, answering the default questions in the pull request interface.
6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.

GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
Expand All @@ -43,21 +43,18 @@ We require all contributors to agree to the terms of the [Developer Certificate
and to indicate this agreement by signing their commits using [Git's `Signed-off-by` support](https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---signoff).
To add this trailer to your commits, use the `-s, --sign-off` option to `git commit`.


## Finding contributions to work on
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.

Looking at the existing issues is a great way to find something to contribute on. We use issue labels to identify [good first issues](https://github.com/awslabs/mountpoint-s3/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) as a great place to start.

## Code of Conduct
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
[email protected] with any additional questions or comments.


## Security issue notifications
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.

If you discover a potential security issue in this project we ask that you notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue.

## Licensing

See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
Mountpoint for Amazon S3 is licensed under the Apache-2.0 license. The pull request template will ask you to confirm the licensing of your contribution and to agree to the [Developer Certificate of Origin (DCO)](https://developercertificate.org/).
25 changes: 25 additions & 0 deletions doc/LOGGING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Logging

Mountpoint for Amazon S3 uses the [tracing](https://docs.rs/tracing/latest/tracing/) ecosystem for logging. This makes it easy to configure the verbosity and target of log output. By default we output minimal log information, but for reporting issues or debugging application problems, you may want to customize this logging behavior.

## Log outputs

By default, Mountpoint for Amazon S3 outputs logs to the `~/.mountpoint-s3/` directory, creating it if it doesn't exist. This destination can be changed using the `-l, --log-directory` command-line argument.

A new log file is created for each execution of `mount-s3`. Log files are never automatically rotated or cleaned up.

When running in foreground mode (`-f, --foreground`), `mount-s3` also emits the same log information to stdout.

## Log details

By default, we output minimal (error-level) information to the log file. For reporting issues or debugging application problems, it can be helpful to increase this verbosity. We use the common `RUST_LOG` environment variable for controlling log verbosity and subjects.

To control the log verbosity, set the `RUST_LOG` environment variable. If unset, it defaults to `error` to enable only error-level log messages. Verbosity can be increased by instead setting `RUST_LOG` to `warn`, `info`, `debug`, or `trace`.

`RUST_LOG` can also control the subjects that are included in the log output. By default, all subjects are included, but verbosity can be configured on a per-subject basis. For example, setting `RUST_LOG` to `trace,awscrt=off` turns on trace-level logging for all subjects except `awscrt`. See the [tracing documentation](https://docs.rs/tracing-subscriber/0.3.16/tracing_subscriber/struct.EnvFilter.html) for more details on how to configure `RUST_LOG`.

### Logging suggestions

For interactive debugging, we often set `RUST_LOG=debug,awscrt=off` to enable debug-level errors for everything except the AWS Common Runtime (the `awscrt` subject). The Common Runtime logging is more verbose, so it's helpful to filter those messages out interactively.

For reporting issues, we suggest `RUST_LOG=trace,awscrt=debug` as a good default choice.
Loading

0 comments on commit 3590a0f

Please sign in to comment.