-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(benchmark-test): suggestions for #10975 #10982
base: ahmad-benchmark-test
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,28 @@ | ||
{ | ||
"name": "benchmark", | ||
"version": "1.0.0", | ||
"type": "module", | ||
"scripts": { | ||
"test": "yarn rollup -c && eshost -h v8,xs dist/bundle.js" | ||
}, | ||
"license": "MIT" | ||
"name": "@agoric/benchmark-test", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Generated the correct entry in sdk-package-names.js |
||
"version": "1.0.0", | ||
"type": "module", | ||
"scripts": { | ||
"build": "exit 0", | ||
"prepack": "tsc --build tsconfig.build.json", | ||
"postpack": "git clean -f '*.d.ts*' '*.tsbuildinfo'", | ||
"test": "yarn rollup -c && eshost -h 'V8,Moddable XS' dist/bundle.js", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't do any of the "Pre-reqs of running yarn test" instructions from #10975 , because I was curious how far I'd get with my existing configuration. I still have not tried that, so I don't know what difference it would make. That said: By debugging into eshost code to figure out why it was failing, it was not finding the names "v8" or "xs". The names it seemed to want instead were "V8" and "Moddable XS". I tried it and it worked, but I suspect this is not a good change. I'm just reporting it as the outcome of my experiment. |
||
"bench": "yarn rollup -c && node --jitless --cpu-prof dist/bundle.js", | ||
"lint": "run-s --continue-on-error lint:*", | ||
"lint:eslint": "eslint .", | ||
"lint:types": "tsc" | ||
}, | ||
"author": "Agoric", | ||
"license": "Apache-2.0", | ||
"bugs": { | ||
"url": "https://github.com/Agoric/agoric-sdk/issues" | ||
}, | ||
"homepage": "https://github.com/Agoric/agoric-sdk#readme", | ||
Comment on lines
+15
to
+20
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All blindly imitated from patterns found in other packages. Note the license change from "MIT" to "Apache-2.0". It is our policy to use this license by default, unless there is a compelling reason for another choice. That said, I do like "MIT" ;) |
||
"devDependencies": { | ||
"tsd": "^0.31.1", | ||
"@rollup/plugin-node-resolve": "^16.0.0", | ||
"@rollup/plugin-commonjs": "^28.0.2", | ||
"@endo/init": "^1.1.8", | ||
"@endo/pass-style": "^1.4.8" | ||
} | ||
Comment on lines
+21
to
+27
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Needed by the top level "yarn build" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll note that this package does not depend on anything in the agoric-sdk repo, and many of the things we'd like to benchmark reside into the endo repo. In general, it is fine for packages in agoric-sdk to depend on things in endo, but not vice versa. Thus, at some point before this is announced for general use (even internally), it should be moved to the endo repository. But feel free to do it at any earlier time as is convenient. IMO, you'll find it most convenient to move it asap, because the longer we develop this in the wrong place, the more unknown assumptions we'll accumulate that will make this migration difficult. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I already tried moving it to endo repo. But I was not able to do
Let me try again and paste more findings here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You may find the endo ./scripts/create-package.sh useful. That's what I always start with to create a new endo package. Attn @kriskowal There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the script. I used it and I was able to create the new package. |
||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,16 +2,12 @@ import resolve from '@rollup/plugin-node-resolve'; | |
import commonjs from '@rollup/plugin-commonjs'; | ||
|
||
export default { | ||
input: 'benchmark.test.js', | ||
input: 'test/benchmark.test.js', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I generally don't like substantive source files in the top level directory, but rather in subdirectories "src/", "test/", or "tools/". Putting this package's own internal tests into "test/" is natural. Putting the tool itself into "tools/" rather than "src/" is a bit strange. But "tools/" generally contains those things that are appropriate for others to use via a "devDependencies:" rather than a "dependencies:". In other words, it is to support use for testing by other modules. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make sense. Test should be in the |
||
output: { | ||
file: 'dist/bundle.js', | ||
format: 'iife', | ||
name: 'bundle', | ||
sourcemap: false, | ||
}, | ||
plugins: [ | ||
resolve(), | ||
commonjs(), | ||
], | ||
plugins: [resolve(), commonjs()], | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think I made this format change manually. I suspect it was done by a top level "yarn format", but I am not certain. |
||
}; | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,58 @@ | ||
import '@endo/init'; | ||
import { passStyleOf } from '@endo/pass-style'; | ||
import { test, benchmark } from '../tools/benchmark.js'; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Github is supposed to be better at detecting file rename and motion as different from a deletion and addition for purposes of review. Oh well. In any case, I think this is the only line I changed from your original. |
||
|
||
async function runBenchmark() { | ||
await test('Benchmark', async t => { | ||
await benchmark( | ||
'Empty object', | ||
t, | ||
async () => { | ||
passStyleOf(harden({})); | ||
}, | ||
0.01, | ||
); | ||
}); | ||
|
||
await test('Another Benchmark', async t => { | ||
await benchmark( | ||
'Alphabets object', | ||
t, | ||
async () => { | ||
passStyleOf( | ||
harden({ | ||
a: 12, | ||
b: 13, | ||
c: 14, | ||
d: 15, | ||
e: 16, | ||
f: 17, | ||
g: 18, | ||
h: 19, | ||
i: 20, | ||
j: 21, | ||
k: 22, | ||
l: 23, | ||
m: 24, | ||
n: 25, | ||
o: 26, | ||
p: 27, | ||
q: 28, | ||
r: 29, | ||
s: 30, | ||
t: 31, | ||
u: 32, | ||
v: 33, | ||
w: 34, | ||
x: 35, | ||
y: 36, | ||
z: 37, | ||
}), | ||
); | ||
}, | ||
0.001, | ||
); | ||
}); | ||
} | ||
|
||
void runBenchmark(); |
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,39 @@ | ||||||||||
// import { performance } from 'perf_hooks'; | ||||||||||
const performance = { now: () => Date.now() }; | ||||||||||
Comment on lines
+1
to
+2
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did not change anything in this file beyond moving it to "tools/". However, I considered changing this line to
Suggested change
Since your naming it "performance.now" suggests that we're using it as a low resolution replacement on those platforms on which it is absent. I did not make this change because I'm not sure what else I would then need to change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am of this opinion that we should start writing tests with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok. But should we multiply the output of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For now, I am keeping it consistent with both engines. This means that even if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since the benchmarking is in our control, why not multiply by |
||||||||||
|
||||||||||
async function benchmark(name, t, fn, expedtedTime, iterations = 10000) { | ||||||||||
await null; | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added this line due to our lint checking rules for await safety |
||||||||||
const start = performance.now(); // 3 | ||||||||||
for (let i = 0; i < iterations; i += 1) { | ||||||||||
await fn(); | ||||||||||
} | ||||||||||
const end = performance.now(); // 8 | ||||||||||
const avgTime = (end - start) / iterations; | ||||||||||
|
||||||||||
console.log(`${name} | Average time: ${avgTime}ms`); | ||||||||||
t.assert( | ||||||||||
avgTime < expedtedTime, | ||||||||||
`Expected ${avgTime} to be less than ${expedtedTime}`, | ||||||||||
); | ||||||||||
} | ||||||||||
|
||||||||||
async function test(name, fn) { | ||||||||||
await null; | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added this line due to our lint checking rules for await safety |
||||||||||
try { | ||||||||||
console.log('Running test: ', name); | ||||||||||
await fn({ assert, truthy }); | ||||||||||
console.log(`✅ Passed`); | ||||||||||
} catch (err) { | ||||||||||
console.log(`❌ Failed: ${err.message}`); | ||||||||||
} | ||||||||||
} | ||||||||||
|
||||||||||
function assert(condition, message = 'Assertion failed') { | ||||||||||
if (!condition) throw new Error(message); | ||||||||||
} | ||||||||||
|
||||||||||
function truthy(value, message = 'Expected a truthy value') { | ||||||||||
if (!value) throw new Error(message); | ||||||||||
} | ||||||||||
|
||||||||||
export { benchmark, test }; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"extends": [ | ||
"./tsconfig.json", | ||
"../../tsconfig-build-options.json" | ||
] | ||
} | ||
Comment on lines
+1
to
+6
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. blindly imitated |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
// This file can contain .js-specific Typescript compiler config. | ||
{ | ||
"extends": "../../tsconfig.json", | ||
"compilerOptions": {}, | ||
"include": [ | ||
"tools", | ||
"test", | ||
"rollup.config.js" | ||
Comment on lines
+6
to
+8
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did not move "rollup.config.js" to a subdirectory, because it is directly used by "yarn test" at the top. Aside from the contents of "includes", I blindly imitated the rest. |
||
] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
{ | ||
"extends": [ | ||
"../../typedoc.base.json" | ||
], | ||
"entryPoints": [ | ||
"tools/benchmark.js" | ||
], | ||
"validation": { | ||
"notExported": false, | ||
"invalidLink": false, | ||
} | ||
} | ||
Comment on lines
+1
to
+12
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. blindly imitated |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was automatically generated from the change to the packages's package.json
name:
. This was one of the impediments to doing ayarn build
at the root directory.