Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github action to update benchmarks #106

Merged
merged 3 commits into from
Jan 20, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Node benchmarks

on:
schedule:
# * is a special character in YAML so you have to quote this string
- cron: '0 0 1 * *'

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Use Node.js ${{ matrix.node-version }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Use Node.js ${{ matrix.node-version }}
- name: Use Node.js 12

uses: actions/setup-node@v1
with:
node-version: 12
- name: run benchmarks
run: |
npm install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is error prone and hard to manage. Put it in a separate file and let it be part of the linter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StarpTech I have removed the inline node code to run the benchmarks, this required a fix to benchmark-bench.js to avoid the prompts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea is to move the entire code in a file and just run e.g scripts/bench.js

npm start y 100 10 40
- name: commit benchmarks
run: |
node_version=$(node --version)
benchmark_title=$(cat << EOF
# Benchmarks
* __Machine:__ $(uname -a) | $(node -r os -p "\`\${os.cpus().length} vCPUs | \${Math.ceil(os.totalmem() / (Math.pow(1024, 3)))}GB\`").
* __Method:__ \`autocannon -c 100 -d 40 -p 10 localhost:3000\` (two rounds; one to warm-up, one to measure).
* __Node:__ \`$node_version\`
* __Run:__ $(date)
EOF)
benchmark_table=$(node benchmark-compare.js -t -c)
strip_readme=$(node -r fs -p 'fs.readFileSync("./README.md", "utf-8").split(/# Benchmarks/)[0]')
git checkout master
echo -e "${strip_readme:?}\n${benchmark_title:?}\n\n${benchmark_table}" > README.md
git add README.md
git config user.name 'Github Actions'
git config user.email '<>'
git commit -m "Add new benchmarks to README.md"
- name: push benchmark changes
uses: ad-m/github-push-action@master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,3 @@ yarn.lock
package-lock.json

# benchmark results
results
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice this right now: why we removed from gitignore? I don't find an answer

118 changes: 67 additions & 51 deletions benchmark-bench.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,75 @@
#!/usr/bin/env node
'use strict'

const inquirer = require('inquirer')
const bench = require('./lib/bench')
const { choices, list } = require('./lib/packages')
const argv = process.argv.slice(2)

function select (callback) {
inquirer.prompt([
run().catch(err => {
console.error(err)
process.exit(1)
})

async function run () {
const options = await getBenchmarkOptions()
const modules = options.all ? choices : await select(list)
return bench(options, modules)
}

async function getBenchmarkOptions () {
if (argv.length) return parseArgv()
return inquirer.prompt([
{
type: 'confirm',
name: 'all',
message: 'Do you want to run all benchmark tests?',
default: false
},
{
type: 'input',
name: 'connections',
message: 'How many connections do you need?',
default: 100,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'pipelining',
message: 'How many pipelines do you need?',
default: 10,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'duration',
message: 'How long should it take?',
default: 40,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
}
])
}

function parseArgv () {
const [all, connections, pipelining, duration] = argv
return {
all: all === 'y',
connections: +connections,
pipelining: +pipelining,
duration: +duration
}
}

async function select () {
const result = await inquirer.prompt([
{
type: 'checkbox',
message: 'Select packages',
Expand All @@ -25,52 +88,5 @@ function select (callback) {
}
}
])
.then(function (answers) {
callback(answers.list)
})
return result.list
}

inquirer.prompt([
{
type: 'confirm',
name: 'all',
message: 'Do you want to run all benchmark tests?',
default: false
},
{
type: 'input',
name: 'connections',
message: 'How many connections do you need?',
default: 100,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'pipelining',
message: 'How many pipelines do you need?',
default: 10,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'duration',
message: 'How long should it take?',
default: 40,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
}
]).then((opts) => {
if (!opts.all) {
select(list => bench(opts, list))
} else {
bench(opts, choices)
}
})
2 changes: 1 addition & 1 deletion benchmark-compare.js
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ if (!choices.length) {
head: ['', 'Router', 'Requests/s', 'Latency', 'Throughput/Mb']
})
if (commander.commandlineMdTable) {
table.push([':--', '--:', ':-:', '--:', '--:', '--:'])
table.push([':--', '--:', ':-:', '--:', '--:'])
}

choices.forEach((result) => {
Expand Down
Loading