Questions or feature requests for use in unit testing

[jaydenseric]

Hi there! Thanks for working on this; it's great to see finally a Deno native package for working with a headless browser. Puppeteer has been very frustrating with Deno, and we're at the point now that it's not working at all. Attempting to use npm:[email protected] results type errors due to globals being set by npm Node.js types conflicting with globals set by Deno types. Attempting to use https://deno.land/x/[email protected] results in strange runtime errors now with modern Deno; and it's a huge number of major versions behind the latest Puppeteer version on npm. It seems to be unmaintained.

Consider this issue a question about how Astral can be used to achieve some things I previously had working with Puppeteer (when it was working with earlier Deno versions), and if there is no way, or an inconvenient way, consider it a feature request.

There are 2 main reasons I have been using headless browsers over the years:

1. Screenshotting web pages, or parts of pages. This is probably straightforward with your current Astral API, so I don't have questions about that right now.
2. Unit testing modules of a published library to check they function as intended in a browser. You want to be able to functionally create a browser page, do unit tests with it in an async callback, and then after it automatically does cleanup:
   • Code coverage is collected by the browser, and then combined with V8 code coverage data collected by Node.js/Deno when the same modules/functions were run directly in tests, to be able to have an accurate code coverage report of the library after the tests have run.
   • Any console messages emitted in the browser page are forwarded to the Node.js/Deno process and logged via console.group under a message indicating the messages came from the headless browser, using what API (warn, error, log, etc.).
   • Page errors are collected and after the tests callback completes, assert the array of page errors is empty. You don't want to end up in a situation where the unit tests are passing but there are uncaught page errors going on undetected.

How can Astral be used to achieve use-case number 2? Here is how it was achieved using Puppeteer:

https://github.com/jaydenseric/ruck/blob/aa068afa7f0630c4e3e8617b2936210c14815964/test/testPuppeteerPage.mjs

It was very difficult for me to work out at first, but that's just because the Puppeteer API doesn't provide a ergonomic ways to do it. Surely it would not be complicated to provide a specific API for this in Astral; an easy way to create a browser page for use in unit testing.

[lino-levan]

Thanks for the issue! You're right that at the moment, use-case number 1 is quite easy to fulfill. Astral could technically achieve use-case number 2 using the raw (but fully typed) celestial bindings but I totally agree that a more ergonomic API would be fitting to have. In an ideal world, how would you imagine such an API to work? I would love to work with you on designing something to fit your usecase.

[lowlighter]

I would also be interested into some way of collecting the code coverage from functions passed to page.evaluate() as they're currently left out by deno coverage despite being ran.

For 2.1, I feel like it'd be nice if we just had to pass something like newPage({coverage: "coverageDir"}) and it'll launch Profiler.startPreciseCoverage() and when page.close() it'd call Profiler.takePreciseCoverage() and Profiler.stopPreciseCoverage() and save the coverage to the specified directory automatically

For 2.2 and 2.3, I think it's pretty much #22 so maybe we could just have two more options console: "log" | "collect" | "null" (for Runtime.consoleAPICalled) and same for exceptions Runtime.exceptionThrown, where "log" would just log them to console, and "collect" would stores them into the page instance so they can be asserted or handled manually later on

[lino-levan]

I'm in favor of your solution to 2.1 and 2.2. I'm a little skeptical about 2.3, but I think I could be convinced.

[lowlighter]

I've just looked quickly for the coverage of Page.evaluate() functions

While it seems possible to map back to where the function originate from (using hacks on Error.prepareStackTrace to get back the caller filename/line number through v8 internals), unfortunately I don't think it's possible to merge it back with deno coverage, at least not without post-processing.

Thing is, if a coverage for a same module appears twice, deno seems to only take the first occurence so it'll still appears as uncovered, so you need to merge them (and coverage for deno is only written at the end of the tests). I feel unless there's a way to inject coverage status to deno's v8 it won't be possible to make straightforward way of doing it

Ideally it'd be nice also if there was a way to detect in deno if it's running in test mode and if coverage is enabled, so astral would be automatically able to transfer the coverage of function running in the browser and it'd just be as simple as using how to print coverage for deno currently. But for this I assume it requires some upstream changes, I don't think it's possible to do it seamlessly in userland

[lino-levan]

Completely agree with your vision here. I think this would be great but it does seam unfeasible at the current moment.

[jaydenseric]

@lino-levan npm:puppeteer isn't working with Deno v2, due to a Deno bug and also 3 seperate memory leaks. I'm now desperate to get things working with Astral.

Astral could technically achieve use-case number 2 using the raw (but fully typed) celestial bindings

Are you able to point me in the right direction? Basically given a page instance, I need to be able to send via CDP commands to start the profiler, start precise coverage:

https://github.com/puppeteer/puppeteer/blob/83449567c7013ae1c87c54e42501aa74575640d0/packages/puppeteer-core/src/cdp/Coverage.ts#L247-L251

And then, after running my tests with the page instance, take the precise V8 coverage data:

https://github.com/jaydenseric/ruck/blob/aa068afa7f0630c4e3e8617b2936210c14815964/test/testPuppeteerPage.mjs#L97

I'm not sure there is a point to then stopping the precise coverage, because the next step is to close the page anyway.

I've looked at some of the Astral source code. It seems you call the CDP client "Celestial", and you can access an instance of it off a Page instance via the method unsafelyGetCelestialBindings:

astral/src/page.ts

Lines 282 to 287 in 0a4ed05

	/**
	* Returns raw celestial bindings for the page. Super unsafe unless you know what you're doing.
	*/
	unsafelyGetCelestialBindings(): Celestial {
	return this.#celestial;
	}

It's not clear from there though if it's possible to send custom CDP commands because the method #sendReq is private:

astral/bindings/celestial.ts

Lines 13808 to 13818 in 0a4ed05

	#sendReq(method: string, params?: unknown): Promise<any> {
	this.ws.send(JSON.stringify({
	id: ++this.#id,
	method,
	params,
	}));
	return new Promise((res) => {
	this.#handlers.set(this.#id, res);
	});
	}

But there are some predefined functions to do this?

astral/bindings/celestial.ts

Line 23852 in 0a4ed05

startPreciseCoverage: async (opts: {

astral/bindings/celestial.ts

Line 23897 in 0a4ed05

takePreciseCoverage: async (): Promise<{

astral/bindings/celestial.ts

Line 23888 in 0a4ed05

stopPreciseCoverage: async (): Promise<void> => {

Side question: The Celestial module is 24,538 lines long. Did you write all that by hand, or is it based on code from somewhere else?

[lino-levan]

It's not clear from there though if it's possible to send custom CDP commands because the method #sendReq is private:

Yep, this is intentional. You should use the predefined functions instead. They're fully typed. They're guaranteed to work with the version of chromium Astral ships with.

The Celestial module is 24,538 lines long. Did you write all that by hand, or is it based on code from somewhere else?

It's generated automatically. You can check out the code in https://github.com/lino-levan/astral/blob/main/bindings/_tools/generate/mod.ts for how that is done.

Do you have any questions on how to approach this? I think you have the core idea down:

1. Get the celestial bindings
2. Use the methods on the bindings to start / stop and take the precise coverage data
3. Done!