High performance distributed data processing engine
Skale-engine is a fast and general purpose distributed data processing system. It provides a high-level API in Javascript and an optimized parallel execution engine on top of NodeJS.
Word count using skale:
var sc = require('skale-engine').context();
sc.textFile('/path/...')
.flatMap(line => line.split(' '))
.map(word => [word, 1])
.reduceByKey((a, b) => a + b, 0)
.count().then(console.log);npm install skale-engine
- In-memory computing
- Controlled memory usage, spill to disk when necessary
- Fast multiple distributed streams
- realtime lazy compiling and running of execution graphs
- workers can connect through TCP or websockets
- skale-engine API
- Gitter for support and discussions
The best and quickest way to get started with skale-engine is to use skale to create, test and deploy skale applications.
To run the examples, clone the skale-engine repository and install the dependencies:
$ git clone git://github.com/skale-me/skale-engine.git --depth 1
$ cd skale-engine
$ npm install
Then start a skale-engine server and workers on local host:
$ npm start
Then run whichever example you want
$ ./examples/core/wordcount.js /etc/hosts
To run the test suite, first install the dependencies, then run npm test:
$ npm install
$ npm test
The original authors of skale-engine are Cedric Artigue and Marc Vertes.