Refactor grep functionality and enhance documentation tools across codebase

README.md

@@ -93,7 +93,7 @@ The quick brown fox jumps over the lazy dog.
 Grep or fuzz search [files](https://microsoft.github.io/genaiscript/reference/scripts/files).
 
 ```js
-const { files } = await workspace.grep(/[a-z][a-z0-9]+/, "**/*.md")
+const { files } = await workspace.grep(/[a-z][a-z0-9]+/, { globs: "*.md" })
 ```
 
 ### LLM Tools

docs/src/components/BuiltinAgents.mdx

@@ -6,6 +6,7 @@ import { LinkCard } from '@astrojs/starlight/components';
 
 ### Builtin Agents
 
+<LinkCard title="agent docs" description="query the documentation" href="/genaiscript/reference/scripts/system#systemagent_docs" />
 <LinkCard title="agent fs" description="query files to accomplish tasks" href="/genaiscript/reference/scripts/system#systemagent_fs" />
 <LinkCard title="agent git" description="query a repository using Git to accomplish tasks. Provide all the context information available to execute git queries." href="/genaiscript/reference/scripts/system#systemagent_git" />
 <LinkCard title="agent github" description="query GitHub to accomplish tasks" href="/genaiscript/reference/scripts/system#systemagent_github" />

docs/src/components/BuiltinTools.mdx

@@ -28,6 +28,7 @@ import { LinkCard } from '@astrojs/starlight/components';
 <LinkCard title="github_pulls_get" description="Get a single pull request by number." href="/genaiscript/reference/scripts/system#systemgithub_pulls" />
 <LinkCard title="github_pulls_review_comments_list" description="Get review comments for a pull request." href="/genaiscript/reference/scripts/system#systemgithub_pulls" />
 <LinkCard title="math_eval" description="Evaluates a math expression" href="/genaiscript/reference/scripts/system#systemmath" />
+<LinkCard title="md_find_files" description="Get the file structure of the documentation markdown/MDX files. Retursn filename, title, description for each match. Use pattern to specify a regular expression to search for in the file content." href="/genaiscript/reference/scripts/system#systemmd_find_files" />
 <LinkCard title="md_read_frontmatter" description="Reads the frontmatter of a markdown or MDX file." href="/genaiscript/reference/scripts/system#systemmd_frontmatter" />
 <LinkCard title="python_code_interpreter_run" description="Executes python 3.12 code for Data Analysis tasks in a docker container. The process output is returned. Do not generate visualizations. The only packages available are numpy, pandas, scipy. There is NO network connectivity. Do not attempt to install other packages or make web requests." href="/genaiscript/reference/scripts/system#systempython_code_interpreter" />
 <LinkCard title="python_code_interpreter_copy_files" description="Copy files from the host file system to the container file system" href="/genaiscript/reference/scripts/system#systempython_code_interpreter" />

docs/src/content/docs/guides/search-and-transform.mdx

@@ -68,11 +68,11 @@
 that allows to efficiently search for a pattern in files (this is the same search engine
 that powers the Visual Studio Code search).
 
 ```js "workspace.grep"
-const { pattern, glob } = env.vars
+const { pattern, globs } = env.vars
 const patternRx = new RegExp(pattern, "g")
-const { files } = await workspace.grep(patternRx, glob)
+const { files } = await workspace.grep(patternRx, { globs })
 ```
 
 ## Compute Transforms
 

docs/src/content/docs/index.mdx

@@ -248,9 +248,9 @@
 
 Grep or fuzz search [files](/genaiscript/referen/script/files)
 
 ```js wrap
-const { files } = await workspace.grep(/[a-z][a-z0-9]+/, "**/*.md")
+const { files } = await workspace.grep(/[a-z][a-z0-9]+/, { globs: "*.md" })
 ```
 
 </Card>
 

docs/src/content/docs/reference/scripts/system.mdx

@@ -96,6 +96,58 @@
 `````
 
 
+### `system.agent_docs`
+
+Agent that can query on the documentation.
+
+
+
+
+
+`````js wrap title="system.agent_docs"
+system({
+    title: "Agent that can query on the documentation.",
+})
+
+const docsRoot = env.vars.docsRoot || "docs"
+const samplesRoot = env.vars.samplesRoot || "packages/sample/genaisrc/"
+
+defAgent(
+    "docs",
+    "query the documentation",
+    async (ctx) => {
+        ctx.$`Your are a helpfull LLM agent that is an expert at Technical documentation. You can provide the best analyzis to any query about the documentation.
+
+        Analyze QUERY and respond with the requested information.
+
+        ## Tools
+
+        The 'md_find_files' can perform a grep search over the documentation files and return the title, description, and filename for each match.
+        To optimize search, conver the QUERY request into keywords or a regex pattern.
+
+        Try multiple searches if you cannot find relevant files.
+
+        ## Context
+
+        - the documentation is stored in markdown/MDX files in the ${docsRoot} folder
+        ${samplesRoot ? `- the code samples are stored in the ${samplesRoot} folder` : ""}
+        `
+    },
+    {
+        system: ["system.explanations", "system.github_info"],
+        tools: [
+            "md_find_files",
+            "md_read_frontmatterm",
+            "fs_find_files",
+            "fs_read_file",
+        ],
+        maxTokens: 5000,
+    }
+)
+
+`````
+
+
 ### `system.agent_fs`
 
 Agent that can find, search or read files to accomplish tasks
@@ -615,8 +667,8 @@
         context.log(
             `ls ${glob} ${pattern ? `| grep ${pattern}` : ""} ${frontmatter ? "--frontmatter" : ""}`
         )
         const res = pattern
-            ? (await workspace.grep(pattern, glob, { readText: false })).files
+            ? (await workspace.grep(pattern, { glob, readText: false })).files
             : await workspace.findFiles(glob, { readText: false })
         if (!res?.length) return "No files found."
 
@@ -989,8 +1041,12 @@
         let log = await github.downloadWorkflowJobLog(job_id, {
             llmify: true,
         })
-        if (parsers.tokens(log) > 1000)
-            log = "...(truncated, tool long)...\n" + log.slice(-3000)
+        if ((await tokenizers.count(log)) > 1000) {
+            log = await tokenizers.truncate(log, 1000, { last: true })
+            const annotations = await parsers.annotations(log)
+            if (annotations.length > 0)
+                log += "\n\n" + YAML.stringify(annotations)
+        }
         return log
     }
 )
@@ -1105,11 +1161,11 @@
 })
 
 const info = await github.info()
 if (info?.owner) {
-    const { auth, owner, repo, baseUrl } = info
+    const { owner, repo, baseUrl } = info
     $`- current github repository: ${owner}/${repo}`
     if (baseUrl) $`- current github base url: ${baseUrl}`
 }
 
 `````
 
@@ -1448,6 +1504,95 @@
 `````
 
 
+### `system.md_find_files`
+
+Tools to help with documentation tasks
+
+
+
+-  tool `md_find_files`: Get the file structure of the documentation markdown/MDX files. Retursn filename, title, description for each match. Use pattern to specify a regular expression to search for in the file content.
+
+`````js wrap title="system.md_find_files"
+system({
+    title: "Tools to help with documentation tasks",
+})
+
+const model = (env.vars.mdSummaryModel = "gpt-4o-mini")
+
+defTool(
+    "md_find_files",
+    "Get the file structure of the documentation markdown/MDX files. Retursn filename, title, description for each match. Use pattern to specify a regular expression to search for in the file content.",
+    {
+        type: "object",
+        properties: {
+            path: {
+                type: "string",
+                description: "root path to search for markdown/MDX files",
+            },
+            pattern: {
+                type: "string",
+                description:
+                    "regular expression pattern to search for in the file content.",
+            },
+            question: {
+                type: "string",
+                description: "Question to ask when computing the summary",
+            },
+        },
+    },
+    async (args) => {
+        const { path, pattern, context, question } = args
+        context.log(
+            `docs: ls ${path} ${pattern ? `| grep ${pattern}` : ""} --frontmatter ${question ? `--ask ${question}` : ""}`
+        )
+        const matches = pattern
+            ? (await workspace.grep(pattern, { path, readText: true })).files
+            : await workspace.findFiles(path + "/**/*.{md,mdx}", {
+                  readText: true,
+              })
+        if (!matches?.length) return "No files found."
+        const q = await host.promiseQueue(5)
+        const files = await q.mapAll(matches, async ({ filename, content }) => {
+            const file = {
+                filename,
+            }
+            try {
+                const fm = await parsers.frontmatter(content)
+                if (fm) {
+                    file.title = fm.title
+                    file.description = fm.description
+                }
+                const { text: summary } = await runPrompt(
+                    (_) => {
+                        _.def("CONTENT", content, { language: "markdown" })
+                        _.$`As a professional summarizer, create a concise and comprehensive summary of the provided text, be it an article, post, conversation, or passage, while adhering to these guidelines:
+                        ${question ? `* ${question}` : ""}
+                        * The summary is intended for an LLM, not a human.
+                        * Craft a summary that is detailed, thorough, in-depth, and complex, while maintaining clarity and conciseness.
+                        * Incorporate main ideas and essential information, eliminating extraneous language and focusing on critical aspects.
+                        * Rely strictly on the provided text, without including external information.
+                        * Format the summary in one single paragraph form for easy understanding. Keep it short.
+                        * Generate a list of keywords that are relevant to the text.`
+                    },
+                    {
+                        label: `summarize ${filename}`,
+                        cache: "md_find_files_summary",
+                        model,
+                    }
+                )
+                file.summary = summary
+            } catch (e) {}
+            return file
+        })
+        const res = YAML.stringify(files)
+        return res
+    },
+    { maxTokens: 20000 }
+)
+
+`````
+
+
 ### `system.md_frontmatter`
 
 Markdown frontmatter reader