Unyek, to re-write the files from a yek file

[AntouanK]

Not really an issue, but posting in case someone else needs it.

I wanted an ... unyek, to un-serialise the files, so I made one

https://github.com/AntouanK/unyek

[mohsen1]

https://github.com/codeinred/untree :)

Quick suggestion here write in parallel (I think Deno knows not to overwhelm the OS with too many writes in parallel but worth testing with a huge number to make sure)

[AntouanK]

There's a pattern :D

Since it awaits for every write, it will happen sequentially, right?

[mohsen1]

Something like this:

Details

async function unyekFile(yekPath: string): Promise<void> {
  try {
    const content = await Deno.readTextFile(yekPath);
    const files = parseYekContent(content);
    const groupedFiles = groupFilesByBasePath(files);

    const results = await Promise.allSettled(
      groupedFiles
      .map(async ([basePath, fileGroup]) =>
        combineFileParts(fileGroup.parts)
      )
      .map((combinedContent) =>
        writeFileContent(basePath, combinedContent)
      )
    );


    if (results.every((result) => result.status === "fulfilled")) {
      console.log("All files written successfully");
    } else {
      console.error("Failed to write some files");
    }
  } catch (error) {
    console.error(`Error processing yek file: ${error.message}`);
    Deno.exit(1);
  }
}

Note that from v0.14.0 yek is only writing one file so you might not need that. I realized chunking is not really useful for LLMs since even if you feed an LLM with 8K token context 2 8K token messages, the result is no different than just not providing the first (less important) chunk