Add caching #179

gladyshcodes · 2024-12-26T23:45:29Z

Issue #124

This PR introduces basic caching mechanism to reduce costs and increase effectiveness of running test suites.

Performance Boost: Achieves an average speedup of 400%-600%, automations like "Find Lionel Messi Wikipedia page" now completing about 5x faster

Flow diagram

Benchmarks

No caching

With caching

Perf boost: 6x

vercel · 2024-12-26T23:45:32Z

@gladyshcodes is attempting to deploy a commit to the Antiwork Team on Vercel.

A member of the Team first needs to authorize it.

CLAassistant · 2024-12-26T23:45:34Z

All committers have signed the CLA.

gladyshcodes · 2024-12-27T00:08:45Z

One challenge is:

async getComponentStringByCoords(x: number, y: number) {
    return await this.getPage().evaluate(
      ([x, y]) => {
        const elem = document.elementFromPoint(x, y);
        return elem?.outerHTML.trim().replace(/\s+/g, ' ');
      },
      [x, y]
    );
  }

The function above retrieves normalized component string given X and Y coordinates. The reason it's implemented this way is that currently, Playwright does not support selecting DOM elements using coordinates (see this closed issue). The only way I have found was to use native document func. If you have any other ideas in mind, please suggest 👍

vercel · 2024-12-27T07:25:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
shortest	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 30, 2024 7:50pm

m2rads · 2024-12-27T07:34:29Z

Would be nice to resolve the conflicts and then I can test your branch locally.

gladyshcodes · 2024-12-27T16:02:13Z

Would be nice to resolve the conflicts and then I can test your branch locally.

@m2rads I have resolved conflicts now. You can check it out.

gladyshcodes · 2024-12-27T17:01:11Z

Refinements / Improvements:

Write new text execution to cache file only after it has been successfully completed. That way we can be sure cached tests are only successful ones
Delete cache entry if it fails to run (e.g UI change) and rewrite it

m2rads · 2024-12-27T22:29:21Z

@gladyshcodes sorry we made some more changes. Please resolve the conflicts again and I will test soon. Thank you :)

gladyshcodes · 2024-12-28T19:02:21Z

@gladyshcodes sorry we made some more changes. Please resolve the conflicts again and I will test soon. Thank you :)

Hi there @m2rads. Those conflicts actually do not interfere with functionality added. I resolved them 👍 .

m2rads · 2024-12-29T11:40:43Z

packages/shortest/src/types/browser.ts

-export type BrowserAction =
-  | "mouse_move"
-  | "left_click"
-  | "left_click_drag"
-  | "right_click"
-  | "middle_click"
-  | "double_click"
-  | "screenshot"
-  | "cursor_position"
-  | "github_login"
-  | "clear_session"
-  | "type"
-  | "key"
-  | "run_callback"
-  | "navigate"
-  | "sleep"
-  | "check_email";
+export enum BrowserActionEnum {
+  MouseMove = "mouse_move",
+  LeftClick = "left_click",
+  LeftClickDrag = "left_click_drag",
+  RightClick = "right_click",
+  MiddleClick = "middle_click",
+  DoubleClick = "double_click",
+  Screenshot = "screenshot",
+  CursorPosition = "cursor_position",
+  GithubLogin = "github_login",
+  ClearSession = "clear_session",
+  Type = "type",
+  Key = "key",
+  RunCallback = "run_callback",
+  Navigate = "navigate",
+  Sleep = "sleep",
+  CheckMail = "check_mail",
+}
+
+export type BrowserAction = `${BrowserActionEnum}`;


Why this change?

I needed to use screenshot browser action and founded it no good to make hard code comparison. That's why I created enum from interface, not touching the interface itself. Now we can do: BrowserActionEnum.Screenshot

m2rads · 2024-12-29T11:43:30Z

Seems like Vercel deployment is failing. I am also getting some build errors locally.

Steps to reproduce this error:

Remove these files:
pnpm-lock.json
node_modules
packages/shortest/node_modules
packages/shortest/dist

Then run pnpm install

Let me know if you need Vercel logs for troubleshooting.

gladyshcodes · 2024-12-29T15:00:56Z

@m2rads Build issue resolved, but Vercel pipeline does not re-run. I guess your approval is needed

…fix issue with recursion

m2rads · 2024-12-29T21:27:12Z

packages/shortest/src/cache/cache.ts

+    this.cacheFile = path.join(process.cwd(), ".cache", "cache.json");
+    this.lockFile = path.join(process.cwd(), ".cache", "cache.lock");


I think it would be better to put this inside .shortest folder. Also does this only cache passing test cases?

I think it would be better to put this inside .shortest folder.

Good point. I will rename it

Also does this only cache passing test cases?

Not entirely sure what you mean. If you are asking whether test file only contains tests that succeeded, then yes

m2rads · 2024-12-29T22:14:41Z

Thanks the build issue is resolved now but I think there might be another issue. When running a test, it caches the steps successfully, but on a consecutive run, it deletes the cache and does the same process. It doesn't seem like the consecutive test is being executed from the cache as it takes the same amount of time to run.

cache.mp4

gladyshcodes · 2024-12-29T22:22:03Z

@m2rads Hmm that's weird. I have just tried running several tests and it worked fine for me. I didn't try it with new dashboard test though. I will see now and let you know

shortest("Open Google"); runs:

m2rads · 2024-12-29T22:25:42Z

I see. The new test needs setup with Mailosaur. Maybe try a more complicated test that has multiple steps and see if this issue happens.

gladyshcodes · 2024-12-29T22:33:33Z

@m2rads I have tried with this test, it works:

shortest("Open first article on Hackernews");

m2rads · 2024-12-30T09:03:23Z

@m2rads I have tried with this test, it works:

shortest("Open first article on Hackernews");

Hhhm seems like this happens when you have multiple test cases. Can you try with more than one test case?

gladyshcodes · 2024-12-30T16:34:55Z

@m2rads Thank you for caching the bug! I've realized the problem was caused by several factors:

Missing sleep time after each browserTool.execute. Similar to how it's done in runner, we need to address this. However, I believe we should eliminate such code altogether since it affects the performance of both cached and uncached tests. Sometimes, waiting less than a second is sufficient for the next action, while other times, potentially, a longer wait might be needed
For dashboard.test.ts:

I managed to get it running by referencing to the source code of Feature: Email Validation with Mailosaur #183, as the relevant documentation appears to be missing. It would be helpful under the services configuration section section. Also, Clerk configuration needs updating: you’ll need to disable the “Require the same device and browser” checkmark in the Clerk dashboard for the tests with mailosaur to work (at least in my case)
Issue itself was the difference in components from email letter as their href attrs differ every time. We now recursively clean up href attributes from the tags. The idea that if the URL changes and we need to click the link, the next step will fail

I ran several tests and they pass now:

Perf boost: 5.6x

@slavingia If you have time, take a look as well, maybe any suggestions?

packages/shortest/src/ai/client.ts

slavingia · 2024-12-30T17:14:35Z

Merged in auto fix which led to some new failures

slavingia

Minor comments, looks great otherwise!

…at/cache

slavingia · 2024-12-31T18:19:52Z

Bug report:

however I've removed the button I was testing and it still pass

maybe I did something wrong, my test is just this:
shortest("Open the home page and signin via email").expect("A message to check your email");

what I did now is that I removed the "Signing via Email" link and it still pass

it seems the second time I run the tests they are run against the screenshots but don't hit the server

You can see the video here: https://x.com/madarco/status/1874141497404363128

I guess there needs to be some logic to destroy the cache, so that the spec starts to fail. We'll need to think more on the best way to do that dynamically; we can start by making it very easy (one command) to nuke the cache?

add caching

710b172

gladyshcodes mentioned this pull request Dec 26, 2024

Performance bottleneck for non-cached tests #181

Open

vercel bot deployed to Preview December 27, 2024 07:26 View deployment

gladyshcodes added 4 commits December 27, 2024 14:57

Merge branch 'main' into feat/cache

ee112f9

run eslint

8aff3ea

refine cache file

9900099

update component str retrieval logic

be78be0

vercel bot deployed to Preview December 27, 2024 22:30 View deployment

Merge remote-tracking branch 'upstream/main' into feat/cache

3aa7f00

vercel bot had a problem deploying to Preview December 29, 2024 11:36 Failure

m2rads reviewed Dec 29, 2024

View reviewed changes

fix build issue

485cfa8

gladyshcodes mentioned this pull request Dec 29, 2024

Cache tests #124

Closed

gladyshcodes added 6 commits December 29, 2024 18:10

add batch set cache

8d23a75

implement cache deletion

18af38e

fix typo

829b40d

add --debug-ai logging for cached tests

c0fbcdd

return window to initial state when cached test fails to execute and …

3c384f1

…fix issue with recursion

self-review refinements

60917ca

vercel bot deployed to Preview December 29, 2024 21:17 View deployment

m2rads reviewed Dec 29, 2024

View reviewed changes

change cache file path to .shortest

b0c2d2c

gladyshcodes requested a review from m2rads December 29, 2024 21:43

vercel bot deployed to Preview December 29, 2024 21:47 View deployment

Merge remote-tracking branch 'upstream/main' into feat/cache

3b81b26

gladyshcodes added 2 commits December 30, 2024 15:35

fix lint issues

87bbfb2

fix critical issue

008461a

Merge branch 'main' into feat/cache

f102c29

slavingia reviewed Dec 30, 2024

View reviewed changes

packages/shortest/src/ai/client.ts Outdated Show resolved Hide resolved

vercel bot deployed to Preview December 30, 2024 17:06 View deployment

[autofix.ci] apply automated fixes

2dd5914

Update .gitignore

cf6ab19

slavingia reviewed Dec 30, 2024

View reviewed changes

gladyshcodes added 3 commits December 30, 2024 19:58

remove duplicate lines

02f4fa1

Merge branch 'feat/cache' of github.com:gladyshcodes/shortest into fe…

11cae11

…at/cache

remove .cache dir from gitignore

7f44822

gladyshcodes requested a review from slavingia December 30, 2024 19:12

slavingia approved these changes Dec 30, 2024

View reviewed changes

m2rads approved these changes Dec 30, 2024

View reviewed changes

vercel bot deployed to Preview December 30, 2024 19:50 View deployment

slavingia merged commit c43a054 into anti-work:main Dec 30, 2024
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add caching #179

Add caching #179

gladyshcodes commented Dec 26, 2024 •

edited

Loading

vercel bot commented Dec 26, 2024

CLAassistant commented Dec 26, 2024 •

edited

Loading

gladyshcodes commented Dec 27, 2024 •

edited

Loading

vercel bot commented Dec 27, 2024 •

edited

Loading

m2rads commented Dec 27, 2024

gladyshcodes commented Dec 27, 2024

gladyshcodes commented Dec 27, 2024

m2rads commented Dec 27, 2024

gladyshcodes commented Dec 28, 2024

m2rads Dec 29, 2024

gladyshcodes Dec 29, 2024 •

edited

Loading

m2rads commented Dec 29, 2024 •

edited

Loading

gladyshcodes commented Dec 29, 2024

m2rads Dec 29, 2024

gladyshcodes Dec 29, 2024

m2rads commented Dec 29, 2024

gladyshcodes commented Dec 29, 2024 •

edited

Loading

m2rads commented Dec 29, 2024

gladyshcodes commented Dec 29, 2024

m2rads commented Dec 30, 2024

gladyshcodes commented Dec 30, 2024

slavingia commented Dec 30, 2024

slavingia left a comment

slavingia commented Dec 31, 2024

		this.cacheFile = path.join(process.cwd(), ".cache", "cache.json");
		this.lockFile = path.join(process.cwd(), ".cache", "cache.lock");

Add caching #179

Add caching #179

Conversation

gladyshcodes commented Dec 26, 2024 • edited Loading

vercel bot commented Dec 26, 2024

CLAassistant commented Dec 26, 2024 • edited Loading

gladyshcodes commented Dec 27, 2024 • edited Loading

vercel bot commented Dec 27, 2024 • edited Loading

m2rads commented Dec 27, 2024

gladyshcodes commented Dec 27, 2024

gladyshcodes commented Dec 27, 2024

m2rads commented Dec 27, 2024

gladyshcodes commented Dec 28, 2024

m2rads Dec 29, 2024

Choose a reason for hiding this comment

gladyshcodes Dec 29, 2024 • edited Loading

Choose a reason for hiding this comment

m2rads commented Dec 29, 2024 • edited Loading

gladyshcodes commented Dec 29, 2024

m2rads Dec 29, 2024

Choose a reason for hiding this comment

gladyshcodes Dec 29, 2024

Choose a reason for hiding this comment

m2rads commented Dec 29, 2024

gladyshcodes commented Dec 29, 2024 • edited Loading

m2rads commented Dec 29, 2024

gladyshcodes commented Dec 29, 2024

m2rads commented Dec 30, 2024

gladyshcodes commented Dec 30, 2024

slavingia commented Dec 30, 2024

slavingia left a comment

Choose a reason for hiding this comment

slavingia commented Dec 31, 2024

gladyshcodes commented Dec 26, 2024 •

edited

Loading

CLAassistant commented Dec 26, 2024 •

edited

Loading

gladyshcodes commented Dec 27, 2024 •

edited

Loading

vercel bot commented Dec 27, 2024 •

edited

Loading

gladyshcodes Dec 29, 2024 •

edited

Loading

m2rads commented Dec 29, 2024 •

edited

Loading

gladyshcodes commented Dec 29, 2024 •

edited

Loading