Refactor grep functionality and enhance documentation tools across co…

…debase (#757)

* Refactor grep functionality and enhance documentation tools across codebase

* Refactor documentation querying and tool functionality, enhance logging, and update error handling.

* Update description of system.agent_docs to clarify its querying capability

* Add option to truncate text from the end in truncateTextToTokens function

* Enhance token handling with new utilities for counting and truncating logs

* Update script paths and descriptions, and revise git diff handling logic

* Add maxTokens option and enhance tool usage for pull request reviewer

* Update script references and improve error handling in tool calls

* Fix typos and add note on pull requests in GitHub agent documentation

* Fix typo by changing "globs" to "glob" in workspace.grep call

* Remove excludedPaths in git diff config and update commit prompt with emoji usage ✨

* Add note about using the description for pull requests 📝

* Refactor error handling, update image model, and enhance logging 📦🛠️

* Update model references from gpt-4-turbo-v to gpt-4o throughout the project 🚀

* Remove debugger statement from cli.ts 🚀

* Remove `workspace grep` command and related functionality 🗑️

* Add PR descriptor script for pull request summaries 📝✨

* Add git branch default tool and enhance PR descriptor tools 🚀

* Improve logging, add options handling, and fix typos in code and docs 📄🔧

* Update PR description generation: add GPT-4o model and increase token limit 🚀

* Remove workspace grep tests from cli.test.ts 🧹

* Update fs_diff_files description for clarity 📄✨

.gitignore.genai

@@ -1,3 +1,4 @@
+.vscode/*
 **/genaiscript.d.ts
 **/yarn.lock
 THIRD_PARTY_LICENSES.md

.vscode/tasks.json

@@ -32,21 +32,6 @@
                 "showReuseMessage": false,
                 "clear": true
             }
-        },
-        {
-            "label": "rv",
-            "type": "shell",
-            "command": "node packages/cli/built/genaiscript.cjs run rv",
-            "detail": "genai reviewer",
-            "problemMatcher": "$tsc",
-            "presentation": {
-                "echo": true,
-                "reveal": "always",
-                "focus": false,
-                "panel": "shared",
-                "showReuseMessage": false,
-                "clear": true
-            }
         }
     ]
 }

README.md

@@ -93,7 +93,7 @@ The quick brown fox jumps over the lazy dog.
 Grep or fuzz search [files](https://microsoft.github.io/genaiscript/reference/scripts/files).
 
 ```js
-const { files } = await workspace.grep(/[a-z][a-z0-9]+/, "**/*.md")
+const { files } = await workspace.grep(/[a-z][a-z0-9]+/, { globs: "*.md" })
 ```
 
 ### LLM Tools

docs/genaisrc/image-alt-text.genai.js

@@ -1,7 +1,7 @@
 script({
     title: "Image Alt Text generator",
     description: "Generate alt text for images",
-    model: "openai:gpt-4-turbo-v",
+    model: "openai:gpt-4o",
     group: "docs",
     maxTokens: 4000,
     temperature: 0,

docs/src/components/BuiltinAgents.mdx

@@ -6,6 +6,7 @@ import { LinkCard } from '@astrojs/starlight/components';
 
 ### Builtin Agents
 
+<LinkCard title="agent docs" description="query the documentation" href="/genaiscript/reference/scripts/system#systemagent_docs" />
 <LinkCard title="agent fs" description="query files to accomplish tasks" href="/genaiscript/reference/scripts/system#systemagent_fs" />
 <LinkCard title="agent git" description="query a repository using Git to accomplish tasks. Provide all the context information available to execute git queries." href="/genaiscript/reference/scripts/system#systemagent_git" />
 <LinkCard title="agent github" description="query GitHub to accomplish tasks" href="/genaiscript/reference/scripts/system#systemagent_github" />

docs/src/components/BuiltinTools.mdx

@@ -9,6 +9,7 @@ import { LinkCard } from '@astrojs/starlight/components';
 <LinkCard title="fs_diff_files" description="Computes a diff between two files." href="/genaiscript/reference/scripts/system#systemfs_diff_files" />
 <LinkCard title="fs_find_files" description="Finds file matching a glob pattern. Use pattern to specify a regular expression to search for in the file content." href="/genaiscript/reference/scripts/system#systemfs_find_files" />
 <LinkCard title="fs_read_file" description="Reads a file as text from the file system. Returns undefined if the file does not exist." href="/genaiscript/reference/scripts/system#systemfs_read_file" />
+<LinkCard title="git_branch_default" description="Gets the default branch using git." href="/genaiscript/reference/scripts/system#systemgit" />
 <LinkCard title="git_branch_current" description="Gets the current branch using git." href="/genaiscript/reference/scripts/system#systemgit" />
 <LinkCard title="git_branch_list" description="List all branches using git." href="/genaiscript/reference/scripts/system#systemgit" />
 <LinkCard title="git_diff" description="Computes file diffs using the git diff command. If the diff is too large, it returns the list of modified/added files." href="/genaiscript/reference/scripts/system#systemgit" />
@@ -28,6 +29,7 @@ import { LinkCard } from '@astrojs/starlight/components';
 <LinkCard title="github_pulls_get" description="Get a single pull request by number." href="/genaiscript/reference/scripts/system#systemgithub_pulls" />
 <LinkCard title="github_pulls_review_comments_list" description="Get review comments for a pull request." href="/genaiscript/reference/scripts/system#systemgithub_pulls" />
 <LinkCard title="math_eval" description="Evaluates a math expression" href="/genaiscript/reference/scripts/system#systemmath" />
+<LinkCard title="md_find_files" description="Get the file structure of the documentation markdown/MDX files. Retursn filename, title, description for each match. Use pattern to specify a regular expression to search for in the file content." href="/genaiscript/reference/scripts/system#systemmd_find_files" />
 <LinkCard title="md_read_frontmatter" description="Reads the frontmatter of a markdown or MDX file." href="/genaiscript/reference/scripts/system#systemmd_frontmatter" />
 <LinkCard title="python_code_interpreter_run" description="Executes python 3.12 code for Data Analysis tasks in a docker container. The process output is returned. Do not generate visualizations. The only packages available are numpy, pandas, scipy. There is NO network connectivity. Do not attempt to install other packages or make web requests." href="/genaiscript/reference/scripts/system#systempython_code_interpreter" />
 <LinkCard title="python_code_interpreter_copy_files" description="Copy files from the host file system to the container file system" href="/genaiscript/reference/scripts/system#systempython_code_interpreter" />

docs/src/components/CreateScript.mdx

@@ -4,16 +4,16 @@ description: Learn how to create a new script using Visual Studio Code or other
 keywords: create script, VSCode, command palette, CLI, script generation
 ---
 
-import { Tabs, TabItem } from '@astrojs/starlight/components';
+import { Tabs, TabItem } from "@astrojs/starlight/components"
 
 <Tabs>
 <TabItem label="Visual Studio Code, Cursor">
 
-Use the `> GenAiScript: Create new script...` command in the [command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) 
+Use the `> GenAiScript: Create new script...` command in the [command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette)
 (`Ctrl+Shift+P` on Windows/Linux, `⇧⌘P` on Mac)
 to create a new script.
 
-![The command palette and the create script command](../assets/vscode-create-new-script.png)
+![A command palette with a search bar showing ">createn" and a suggestion below for "GenAIScript: Create new script..." highlighted in blue.](../assets/vscode-create-new-script.png)
 
 </TabItem>
 <TabItem label="Other Editors">

docs/src/content/docs/blog/drafts/code-review-with-genai.md

@@ -6,13 +6,12 @@ tags: ["code review", "development", "programming", "automation"]
 authors: genaiscript
 canonical_url: https://microsoft.github.io/genaiscript/blog/code-review-with-genai
 description: Dive into automated code review processes with GenAI, enhancing
-  developer efficiency and code quality.
-
+    developer efficiency and code quality.
 ---
 
 ## Introducing "Code Review with GenAI" 🧐
 
-Have you ever wished for an extra set of eyes while coding? Well, GenAI has got your back! Let's delve into the "Reviewer" script, which automates the code review process, making it a breeze for developers. This powerful script for the GenAIScript platform can be found [here on GitHub](https://github.com/microsoft/genaiscript/blob/main/packages/vscode/genaisrc/rv.genai.mts).
+Have you ever wished for an extra set of eyes while coding? Well, GenAI has got your back! Let's delve into the "Reviewer" script, which automates the code review process, making it a breeze for developers. This powerful script for the GenAIScript platform can be found [here on GitHub](https://github.com/microsoft/genaiscript/blob/main/packages/vscode/genaisrc/prr.genai.mts).
 
 ### What is the Script About?
 

docs/src/content/docs/guides/ask-my-image.mdx

@@ -16,7 +16,7 @@ The quick-start guide illustrates how to write a GenAIScript that takes input fr
     ```js
     script({
         title: "Apply a script to an image",
-        model: "openai:gpt-4-turbo-v",
+        model: "openai:gpt-4o",
     })
     ```
 4. Use [defImages](/genaiscript/reference/scripts/images/) to ingest the image file into the model context:

docs/src/content/docs/guides/search-and-transform.mdx

@@ -69,9 +69,9 @@ that allows to efficiently search for a pattern in files (this is the same searc
 that powers the Visual Studio Code search).
 
 ```js "workspace.grep"
-const { pattern, glob } = env.vars
+const { pattern, globs } = env.vars
 const patternRx = new RegExp(pattern, "g")
-const { files } = await workspace.grep(patternRx, glob)
+const { files } = await workspace.grep(patternRx, { globs })
 ```
 
 ## Compute Transforms

docs/src/content/docs/index.mdx

@@ -249,7 +249,7 @@ The quick brown fox jumps over the lazy dog.
 Grep or fuzz search [files](/genaiscript/referen/script/files)
 
 ```js wrap
-const { files } = await workspace.grep(/[a-z][a-z0-9]+/, "**/*.md")
+const { files } = await workspace.grep(/[a-z][a-z0-9]+/, { globs: "*.md" })
 ```
 
 </Card>

docs/src/content/docs/reference/cli/commands.md

@@ -407,30 +407,6 @@ Options:
   -h, --help          display help for command
 ```
 
-## `workspace`
-
-```
-Usage: genaiscript workspace [options] [command]
-
-Workspace tasks
-
-Options:
-  -h, --help                 display help for command
-
-Commands:
-  grep <pattern> [files...]
-  help [command]             display help for command
-```
-
-### `workspace grep`
-
-```
-Usage: genaiscript workspace grep [options] <pattern> [files...]
-
-Options:
-  -h, --help  display help for command
-```
-
 ## `info`
 
 ```

docs/src/content/docs/reference/scripts/images.md

@@ -6,7 +6,7 @@ sidebar:
     order: 10
 ---
 
-Images can be added to the prompt for models that support this feature (like `gpt-4-turbo-v`).
+Images can be added to the prompt for models that support this feature (like `gpt-4o`).
 Use the `defImages` function to declare the images. Supported images will vary
 with models but typically include `PNG`, `JPEG`, `WEBP`, and `GIF`. Both local files and URLs are supported.