|
1 | | -# js-ipfs-migrator |
| 1 | +# Migration tool for JS IPFS Repo |
| 2 | + |
| 3 | +[](https://travis-ci.com/ipfs/js-ipfs-repo-migrations) |
| 4 | +[](https://codecov.io/gh/ipfs/js-ipfs-repo-migrations) |
| 5 | +[](https://david-dm.org/ipfs/js-ipfs-repo-migrations) |
| 6 | +[](http://ipn.io) |
| 7 | +[](http://ipfs.io/) |
| 8 | +[](http://webchat.freenode.net/?channels=%23ipfs) |
| 9 | +[](https://github.com/RichardLitt/standard-readme) |
| 10 | +[](https://github.com/feross/standard) |
| 11 | + |
| 12 | + |
| 13 | + |
| 14 | +> Migration framework for versioning of JS IPFS Repo |
| 15 | +
|
| 16 | +This package is inspired by the [go-ipfs repo migration tool](https://github.com/ipfs/fs-repo-migrations/) |
| 17 | + |
| 18 | +## Lead Maintainer |
| 19 | + |
| 20 | +[Adam Uhlíř](https://github.com/auhau/) |
| 21 | + |
| 22 | +## Table of Contents |
| 23 | + |
| 24 | +- [Background](#background) |
| 25 | +- [Install](#install) |
| 26 | + - [npm](#npm) |
| 27 | + - [Use in Node.js](#use-in-nodejs) |
| 28 | + - [Use in a browser with browserify, webpack or any other bundler](#use-in-a-browser-with-browserify-webpack-or-any-other-bundler) |
| 29 | + - [Use in a browser Using a script tag](#use-in-a-browser-using-a-script-tag) |
| 30 | +- [Usage](#usage) |
| 31 | + - [Writing migration](#writing-migration) |
| 32 | + - [Migrations matrix](#migrations-matrix) |
| 33 | +- [API](#api) |
| 34 | +- [CLI](#cli) |
| 35 | +- [Versioning](#versioning) |
| 36 | +- [Contribute](#contribute) |
| 37 | +- [License](#license) |
| 38 | + |
| 39 | +## Background |
| 40 | + |
| 41 | + |
| 42 | +As js-ipfs evolves and new technologies, algorithms and data structures are incorporated it is necessary to |
| 43 | +enable users to transition between versions. Different versions of js-ipfs may expect a different IPFS repo structure or content (see: [IPFS repo spec](https://github.com/ipfs/specs/tree/master/repo), [JS implementation](https://github.com/ipfs/js-ipfs-repo) ). |
| 44 | +So the IPFS repo is versioned, and this package provides a framework to create migrations to transition |
| 45 | +from one version of IPFS repo to the next/previous version. |
| 46 | + |
| 47 | +This framework: |
| 48 | + * Handles locking/unlocking of repository |
| 49 | + * Defines migrations API |
| 50 | + * Executes and reports migrations in both directions: forward and backward |
| 51 | + * Simplifies creation of new migrations |
| 52 | + * Works on the browser too! |
| 53 | + |
| 54 | +## Install |
| 55 | + |
| 56 | +### npm |
| 57 | + |
| 58 | +```sh |
| 59 | +> npm install ipfs-repo-migrations |
| 60 | +``` |
| 61 | + |
| 62 | +### Use in Node.js |
| 63 | + |
| 64 | +```js |
| 65 | +const migrations = require('ipfs-repo-migrations') |
| 66 | +``` |
| 67 | + |
| 68 | +### Use in a browser with browserify, webpack or any other bundler |
| 69 | + |
| 70 | +```js |
| 71 | +const migrations = require('ipfs-repo-migrations') |
| 72 | +``` |
| 73 | + |
| 74 | +## Usage |
| 75 | + |
| 76 | +Example: |
| 77 | + |
| 78 | +```js |
| 79 | +const migrations = require('ipfs-repo-migrations') |
| 80 | +const getVersion = require('ipfs-repo-migrations/repo/version') |
| 81 | + |
| 82 | +const repoPath = 'some/repo/path' |
| 83 | +const repoVersion = await getVersion(repoPath) |
| 84 | + |
| 85 | +if(repoVersion < migrations.getLatestMigrationVersion()){ |
| 86 | + // Old repo! Lets migrate to latest version! |
| 87 | + await migrations.migrate(repoPath) |
| 88 | +} |
| 89 | +``` |
| 90 | + |
| 91 | +To migrate your repository using the CLI, see the [how to run migrations](./run.md) tutorial. |
| 92 | + |
| 93 | +## API |
| 94 | + |
| 95 | +### `.migrate(path, {toVersion, ignoreLock, repoOptions, onProgress, isDryRun}) -> Promise<void>` |
| 96 | + |
| 97 | +Executes a forward migration to a specific version, or to the latest version if a specific version is not specified. |
| 98 | + |
| 99 | +**Arguments:** |
| 100 | + |
| 101 | + * `path` (string, mandatory) - path to the repo to be migrated |
| 102 | + * `options` (object, optional) - options for the migration |
| 103 | + * `options.toVersion` (int, optional) - version to which the repo should be migrated. Defaults to the latest migration version. |
| 104 | + * `options.ignoreLock` (bool, optional) - if true will not lock the repo when applying migrations. Use with caution. |
| 105 | + * `options.repoOptions` (object, optional) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo). |
| 106 | + * `options.onProgress` (function, optional) - callback that is called after finishing execution of each migration to report progress. |
| 107 | + * `options.isDryRun` (bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes. |
| 108 | + |
| 109 | +#### `onProgress(migration, counter, totalMigrations)` |
| 110 | + |
| 111 | +Signature of the progress callback. |
| 112 | + |
| 113 | +**Arguments:** |
| 114 | + * `migration` (object) - object of migration that just successfully finished running. See [Architecture of migrations](#architecture-of-migrations) for details. |
| 115 | + * `counter` (int) - index of current migration. |
| 116 | + * `totalMigrations` (int) - total count of migrations that will be run. |
| 117 | + |
| 118 | +### `.revert(path, toVersion, {ignoreLock, options, onProgress, isDryRun}) -> Promise<void>` |
| 119 | + |
| 120 | +Executes backward migration to a specific version. |
| 121 | + |
| 122 | +**Arguments:** |
| 123 | + |
| 124 | + * `path` (string, mandatory) - path to the repo to be reverted |
| 125 | + * `toVersion` (int, mandatory) - version to which the repo should be reverted to. |
| 126 | + * `options` (object, optional) - options for the reversion |
| 127 | + * `options.ignoreLock` (bool, optional) - if true will not lock the repo when applying migrations. Use with caution. |
| 128 | + * `options.options` (object, optional) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo). |
| 129 | + * `options.onProgress` (function, optional) - callback that is called after finishing execution of each migration to report progress. |
| 130 | + * `options.isDryRun` (bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes. |
| 131 | + |
| 132 | +### `getLatestMigrationVersion() -> int` |
| 133 | + |
| 134 | +Return the version of the latest migration. |
| 135 | + |
| 136 | +## CLI |
| 137 | + |
| 138 | +The CLI is a NodeJS binary named `jsipfs-repo-migrations`. |
| 139 | +It has several commands: |
| 140 | + |
| 141 | + * `migrate` - performs forward/backward migration to specific or latest version. |
| 142 | + * `status` - check repo for migrations that should be run. |
| 143 | + * `add` - bootstraps new migration. |
| 144 | + |
| 145 | +For further details see the `--help` pages. |
| 146 | + |
| 147 | +## Creating a new migration |
| 148 | + |
| 149 | +Migrations are one of those things that can be extremely painful on users. At the end of the day, we want users never to have to think about it. The process should be: |
| 150 | + |
| 151 | +- SAFE. No data lost. Ever. |
| 152 | +- Revertible. Tools must implement forward and backward (if possible) migrations. |
| 153 | +- Tests. Migrations have to be well tested. |
| 154 | +- To Spec. The tools must conform to the spec. |
| 155 | + |
| 156 | +If your migration has several parts, it should be fail-proof enough that if one part of migration fails the previous changes |
| 157 | +are reverted before propagating the error. If possible then the outcome should be consistent repo so it migration could |
| 158 | +be run again. |
| 159 | + |
| 160 | +### Architecture of a migration |
| 161 | + |
| 162 | +All migrations are placed in the `/migrations` folder. Each folder there represents one migration that follows the migration |
| 163 | +API. |
| 164 | + |
| 165 | +All migrations are collected in `/migrations/index.js`, which should not be edited manually. It is regenerated on |
| 166 | +every run of `jsipfs-migrations add` (manual changes should follow the same style of modifications). |
| 167 | +**The order of migrations is important and migrations must be sorted in ascending order**. |
| 168 | + |
| 169 | +Each migration must follow this API. It must export an object in its `index.js` that has following properties: |
| 170 | + |
| 171 | + * `version` (int) - Number that represents the version which the repo will migrate to (eg. `migration-8` will move the repo to version 8). |
| 172 | + * `description` (string) - Brief description of what the migrations does. |
| 173 | + * `migrate` (function) - Function that performs the migration (see signature of this function below) |
| 174 | + * `revert` (function) - If defined then this function will revert the migration to the previous version. Otherwise it is assumed that it is not possible to revert this migration. |
| 175 | + |
| 176 | +#### `.migrate(repoPath, isBrowser)` |
| 177 | + |
| 178 | +_Do not confuse this function with the `require('ipfs-repo-migrations').migrate()` function that drives the whole migration process!_ |
| 179 | + |
| 180 | +Arguments: |
| 181 | + * `repoPath` (string) - absolute path to the root of the repo |
| 182 | + * `options` (object, optional) - object containing `IPFSRepo` options, that should be used to construct a datastore instance. |
| 183 | + * `isBrowser` (bool) - indicates if the migration is run in a browser environment (as opposed to NodeJS) |
| 184 | + |
| 185 | +#### `.revert(repoPath, isBrowser)` |
| 186 | + |
| 187 | +_Do not confuse this function with the `require('ipfs-repo-migrations').revert()` function that drives the whole backward migration process!_ |
| 188 | + |
| 189 | +Arguments: |
| 190 | + * `repoPath` (string) - path to the root of the repo |
| 191 | + * `options` (object, optional) - object containing `IPFSRepo` options, that should be used to construct the datastore instance. |
| 192 | + * `isBrowser` (bool) - indicates if the migration is run in a browser environment (as opposed to NodeJS) |
| 193 | + |
| 194 | +### Browser vs. NodeJS environments |
| 195 | + |
| 196 | +The migration might need to distinguish in which environment it runs (browser vs. NodeJS). For this reason there is an argument |
| 197 | +`isBrowser` passed to migrations functions. But with simple migrations it should not be necessary to distinguish between |
| 198 | +these environments as the datastore implementation will handle the main differences. |
| 199 | + |
| 200 | +There are currently two main datastore implementations: |
| 201 | + 1. [`datastore-fs`](https://github.com/ipfs/js-datastore-fs) that is backed by file system and is used mainly in the NodeJS environment |
| 202 | + 2. [`datastore-level`](https://github.com/ipfs/js-datastore-level) that is backed by LevelDB and is used mainly in the browser environment |
| 203 | + |
| 204 | + Both implementations share the same API and hence are interchangeable. |
| 205 | + |
| 206 | + When the migration is run in a browser environment, `datastore-fs` is automatically replaced with `datastore-level` even |
| 207 | + when it is directly imported (`require('datastore-fs')` will return `datastore-level` in a browser). |
| 208 | + So with simple migrations you shouldn't worry about the difference between `datastore-fs` and `datastore-level` |
| 209 | + and by default use the `datastore-fs` package (as the replace mechanism does not work vice versa). |
| 210 | + |
| 211 | +### Guidelines |
| 212 | + |
| 213 | +The recommended way to write a new migration is to first bootstrap a dummy migration using the CLI: |
| 214 | + |
| 215 | +```sh |
| 216 | +> npm run new-migration |
| 217 | +``` |
| 218 | + |
| 219 | +A new folder is created with the bootstrapped migration. You can then simply fill in the required fields and |
| 220 | +write the rest of the migration! |
| 221 | + |
| 222 | +### Integration with js-ipfs |
| 223 | + |
| 224 | +When a new migration is created, the repo version in [`js-ipfs-repo`](https://github.com/ipfs/js-ipfs-repo) should be updated with the new version, |
| 225 | +together with updated version of this package. Then the updated version should be propagated to `js-ipfs`. |
| 226 | + |
| 227 | +### Tests |
| 228 | + |
| 229 | +If a migration affects any of the following functionality, it must provide tests for the following functions |
| 230 | + to work under the version of the repo that it migrates to: |
| 231 | + |
| 232 | +* `/src/repo/version.js`:`getVersion()` - retrieving repository's version |
| 233 | +* `/src/repo/lock.js`:`lock()` - locking repository that uses file system |
| 234 | +* `/src/repo/lock-memory.js`:`lock()` - locking repository that uses memory |
| 235 | + |
| 236 | +Every migration must have test coverage. Tests for migrations should be placed in the `/test/migrations/` folder. Most probably |
| 237 | +you will have to plug the tests into `browser.js`/`node.js` if they require specific bootstrapping on each platform. |
| 238 | + |
| 239 | +### Empty migrations |
| 240 | + |
| 241 | +For interop with go-ipfs it might be necessary just to bump a version of a repo without any actual |
| 242 | +modification as there might not be any changes needed in the JS implementation. For that purpose you can create an "empty migration". |
| 243 | + |
| 244 | +The easiest way to do so is with the CLI: |
| 245 | + |
| 246 | +```sh |
| 247 | +> npm run new-migration -- --empty |
| 248 | +``` |
| 249 | + |
| 250 | +This will create an empty migration with the next version. |
| 251 | + |
| 252 | +### Migrations matrix |
| 253 | + |
| 254 | +| IPFS repo version | JS IPFS version | |
| 255 | +| -----------------: |:----------------:| |
| 256 | +| 7 | v0.0.0 - latest | |
| 257 | + |
| 258 | +## Developer |
| 259 | + |
| 260 | +### Module versioning notes |
| 261 | + |
| 262 | +In order to have good overview of what version of package contains what kind of migrations, this package follows this versioning schema: `0.<versionOfLastMigration>.<patches>`. |
| 263 | + |
| 264 | +## Contribute |
| 265 | + |
| 266 | +There are some ways you can make this module better: |
| 267 | + |
| 268 | +- Consult our [open issues](https://github.com/ipfs/js-ipfs-repo/issues) and take on one of them |
| 269 | +- Help our tests reach 100% coverage! |
| 270 | + |
| 271 | +This repository falls under the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md). |
| 272 | + |
| 273 | +[](https://github.com/ipfs/community/blob/master/contributing.md) |
| 274 | + |
| 275 | +## License |
| 276 | + |
| 277 | +[MIT](LICENSE) |
0 commit comments