Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github action to update benchmarks #106

Merged
merged 3 commits into from
Jan 20, 2020
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Node benchmarks

on:
schedule:
# * is a special character in YAML so you have to quote this string
- cron: '0 0 1 * *'

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Use Node.js ${{ matrix.node-version }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Use Node.js ${{ matrix.node-version }}
- name: Use Node.js 12

uses: actions/setup-node@v1
with:
node-version: 12
- name: benchmarks
run: |
npm install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is error prone and hard to manage. Put it in a separate file and let it be part of the linter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StarpTech I have removed the inline node code to run the benchmarks, this required a fix to benchmark-bench.js to avoid the prompts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea is to move the entire code in a file and just run e.g scripts/bench.js

npm start y 100 10 40
node_version=$(node --version)
benchmark_title=$(cat << EOF
# Benchmarks
* __Machine:__ $(uname -a) | $(node -r os -p "\`\${os.cpus().length} vCPUs | \${Math.ceil(os.totalmem() / (Math.pow(1024, 3)))}GB\`").
* __Method:__ \`autocannon -c 100 -d 40 -p 10 localhost:3000\` (two rounds; one to warm-up, one to measure).
* __Node:__ \`$node_version\`
* __Run:__ $(date)
EOF)
benchmark_table=$(node benchmark-compare.js -t -c)
strip_readme=$(node -r fs -p 'fs.readFileSync("./README.md", "utf-8").split(/# Benchmarks/)[0]')
git checkout master
echo -e "${strip_readme:?}\n${benchmark_title:?}\n\n${benchmark_table}" > README.md
git add README.md
git config user.name 'Github Actions'
git config user.email '<>'
git commit -m "Add new benchmarks to README.md"
git remote set-url --push origin https://${{ secrets.GITHUB_USERNAME }}:${{ secrets.GITHUB_REPO_TOKEN }}@github.com/fastify/benchmarks
git push
mcollina marked this conversation as resolved.
Show resolved Hide resolved
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,3 @@ yarn.lock
package-lock.json

# benchmark results
results
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice this right now: why we removed from gitignore? I don't find an answer

118 changes: 67 additions & 51 deletions benchmark-bench.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,75 @@
#!/usr/bin/env node
'use strict'

const inquirer = require('inquirer')
const bench = require('./lib/bench')
const { choices, list } = require('./lib/packages')
const argv = process.argv.slice(2)

function select (callback) {
inquirer.prompt([
run().catch(err => {
console.error(err)
process.exit(1)
})

async function run () {
const options = await getBenchmarkOptions()
const modules = options.all ? choices : await select(list)
return bench(options, modules)
}

async function getBenchmarkOptions () {
if (argv.length) return parseArgv()
return inquirer.prompt([
{
type: 'confirm',
name: 'all',
message: 'Do you want to run all benchmark tests?',
default: false
},
{
type: 'input',
name: 'connections',
message: 'How many connections do you need?',
default: 100,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'pipelining',
message: 'How many pipelines do you need?',
default: 10,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'duration',
message: 'How long should it take?',
default: 40,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
}
])
}

function parseArgv () {
const [all, connections, pipelining, duration] = argv
return {
all: all === 'y',
connections: +connections,
pipelining: +pipelining,
duration: +duration
}
}

async function select () {
const result = await inquirer.prompt([
{
type: 'checkbox',
message: 'Select packages',
Expand All @@ -25,52 +88,5 @@ function select (callback) {
}
}
])
.then(function (answers) {
callback(answers.list)
})
return result.list
}

inquirer.prompt([
{
type: 'confirm',
name: 'all',
message: 'Do you want to run all benchmark tests?',
default: false
},
{
type: 'input',
name: 'connections',
message: 'How many connections do you need?',
default: 100,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'pipelining',
message: 'How many pipelines do you need?',
default: 10,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
},
{
type: 'input',
name: 'duration',
message: 'How long should it take?',
default: 40,
validate (value) {
return !Number.isNaN(parseFloat(value)) || 'Please enter a number'
},
filter: Number
}
]).then((opts) => {
if (!opts.all) {
select(list => bench(opts, list))
} else {
bench(opts, choices)
}
})
2 changes: 1 addition & 1 deletion benchmark-compare.js
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ if (!choices.length) {
head: ['', 'Router', 'Requests/s', 'Latency', 'Throughput/Mb']
})
if (commander.commandlineMdTable) {
table.push([':--', '--:', ':-:', '--:', '--:', '--:'])
table.push([':--', '--:', ':-:', '--:', '--:'])
}

choices.forEach((result) => {
Expand Down
Loading