Running from javascript/typescript

[ChuckJonas]

I have a use case where I would like to process a folder of files and use the output of a genai-script to write rows in a DB.

Pseudo code would look something like this:

const previouslyProcess = getProcessedHashes();
const folder = getFolder();
for(file of folder){
   const hash = md5(file)
   if(previouslyProcess.contains(hash)) 
       continue;
   const output = /npx --yes genaiscript@^1.80.10 run my_script file
   writeResults(output)
}

Is the only way to do this to spawn a process and parse the output for stdout? Or can it be invoked directly via a javascript api?

Seems like this would be a very common usecase, so sorry if this is in the docs. Searched and couldn't find it.

[github-actions]

To address the issue of running genai-script from JavaScript/TypeScript without spawning a separate process, you can leverage the child_process module's functionality to interact with the process directly. However, there are more elegant and efficient ways to achieve this using JavaScript APIs if available.

Option 1: Using JavaScript APIs (if available)

Firstly, check if genai-script provides any official JavaScript/TypeScript SDK or API that you can use directly. This would be the most straightforward solution as it would avoid the overhead of spawning a separate process and parsing its output.

For example:

import { run, execute } from 'genaiscript'; // Hypothetical SDK

async function processFiles() {
    const previouslyProcessed = getProcessedHashes();
    const folder = getFolder();

    for (const file of folder) {
        const hash = md5(file);
        if (previouslyProcessed.contains(hash)) continue;

        try {
            const output = await run('my_script', { file });
            await writeResults(output);
        } catch (error) {
            console.error(`Error processing file ${file}:`, error);
        }
    }
}

processFiles();

Option 2: Spawning a Process and Parsing Output

If the SDK is not available, you will need to spawn the process using child_process and handle the output.

const { exec } = require('child_process');
const fs = require('fs');

async function processFile(file) {
    const hash = md5(file);
    if (previouslyProcessed.contains(hash)) return;

    try {
        await new Promise((resolve, reject) => {
            exec(`npx --yes genaiscript@^1.80.10 run my_script ${file}`, async (error, stdout, stderr) => {
                if (error) {
                    console.error(`Error executing command:`, error);
                    reject(error);
                } else if (stderr) {
                    console.error(`Stderr:\n${stderr}`);
                    reject(stderr);
                } else {
                    const output = stdout.trim();
                    await writeResults(output);
                    resolve();
                }
            });
        });
    } catch (error) {
        console.error(`Error processing file ${file}:`, error);
    }
}

async function processFiles() {
    const previouslyProcessed = getProcessedHashes();
    const folder = getFolder();

    for (const file of folder) {
        await processFile(file);
    }
}

processFiles();

Notes:

1. Security: Ensure that the paths and arguments passed to exec are properly sanitized to prevent command injection attacks.
2. Error Handling: The script should handle errors gracefully, logging appropriate messages and continuing with other files if one fails.

If genai-script does not have a JavaScript SDK or API, spawning a process is the only way to execute it from JavaScript/TypeScript. However, if there are future updates or official support for Node.js, make sure to check for those.

generated by issue-reviewer

[ChuckJonas]

related #450

[ChuckJonas]

This is my current workaround... Feels fragile and inefficient, especially if you are going to process a large number of files. Seems like there is a decent bit of overhead to spin up genaiscript each time.

const { exec } = require("child_process");
const fs = require("fs");

async function runGenAi(script: string, file: string) {
  return new Promise((resolve, reject) => {
    exec(
      `npx --yes --quiet genaiscript@^1.80.10 run ${script} "${file}" -j`,
      async (error, stdout, stderr) => {
        if (error) {
          console.error(`Error executing command:`, error);
          reject(error);
        } else {
          // trim to first and last {} as there was noise on the stdout
          const output = stdout.trim().match(/\{[\s\S]*\}/);
          try {
            const json = JSON.parse(output);
            if (json["json"]) {
              resolve(json["json"]);
            }
            reject("No JSON found in output");
          } catch (e) {
            console.error(`Error parsing JSON:`, e);
            reject(e);
          }
        }
      }
    );
  });
}

runGenAi("receipts", "purchase.pdf")
  .then(console.log)
  .catch(console.error);

[pelikhan]

I will export a "run" api

[pelikhan]

See #925

[pelikhan]

Available for testing in 1.83.0 https://microsoft.github.io/genaiscript/reference/cli/api/