Skip to content

Commit

Permalink
str: Work in progress on Orchid Storage framework.
Browse files Browse the repository at this point in the history
  • Loading branch information
patniemeyer committed Nov 17, 2023
1 parent ecae545 commit b581b6e
Show file tree
Hide file tree
Showing 50 changed files with 2,024 additions and 434 deletions.
9 changes: 6 additions & 3 deletions str-twincoding/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@ venv
file_1MB.dat
*.encoded
*.dat
recovered
a.py
a.sh
a.svg
recovered
twin_coding_nogf.py


output.html
repository
providers.jsonc
75 changes: 75 additions & 0 deletions str-twincoding/README-in.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@

## Orchid Storage Project

_*Orchid Storage is a **work in progress*** - Orchid is an open source project. Help
us build a truly decentralized storage system._

***TODO: Link to whitepaper***

This repository contains work in progress on the file encoding CLI and server framework.

![monitor](docs/monitor.png "Monitor")

A key aspect of the Orchid Storage project is the development of an efficient encoding scheme that minimizes
bandwidth costs incurred during migration of distributed data through providers over time.

**Twin Coding** is a hybrid encoding scheme that works with any two linear coding schemes and combines
them to achieve a space-bandwidth tradeoff, minimizing the amount of data that must be transferred
between storage nodes in order to recover a lost shard of data. In contrast to a traditional
erasure scheme, in which restoration of a lost node requires a full reconstruction of the original
file, Twin Coding allows for the recovery of a lost data shard with data transfer totalling exactly
the size of the lost data shard, with no additional transfer overhead.


This repository contains an implementation of Twin Coding, as well as a command line API for encoding
files, decoding files with erasures, and optimally recovering lost shards. There is also a

See [`twin_coding.py`](encoding/twin_coding.py) for an explanation of the algorithm, example code, and a link to the original paper.


## Installation

```
# Create a virtual environment
python3 -m venv venv
```

```
# Activate the virtual environment
# For macOS and Linux:
source venv/bin/activate
# For Windows:
.\venv\Scripts\activate
```

```
# Install the dependencies
pip install -r requirements.txt
```

## Example Usage
```
INSERT_USAGE
```

## Encoding CLI Examples

See also [`examples.sh`](examples/examples.sh)

```
INSERT_EXAMPLES
```

## CLI Docs

INSERT_STORAGE_DOCS

## Server Docs
```
INSERT_SERVER_DOCS
```

## Monitor Docs
```
INSERT_MONITOR_DOCS
```
Loading

0 comments on commit b581b6e

Please sign in to comment.