Refactor input handling and config, remove chunking #51
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR significantly refactors the repository's core functionality, removing chunking in favor of single-file output and adding token-based processing.
- Critical bug in
src/parallel.rs: Both token mode and byte mode truncation logic have incorrect conditions that would prevent any files from being processed (if total_tokens_needed > 0andif header_size + content_size > 0) - Removal of
test_chunk_order.rswithout adequate replacement tests for file ordering validation in the new single-file output system - Shell scripts in root directory (
.sh) are now ignored in.gitignore, which could affect critical installation scripts - Significant behavioral change in
test_normalize_path.rswhere paths outside base directory now only return filename instead of full path
28 file(s) reviewed, 36 comment(s)
Edit PR Review Bot Settings | Greptile
| tokio = { version = "1.0", features = ["rt", "time", "macros"] } | ||
| env_logger = "0.11" |
There was a problem hiding this comment.
style: tokio is added as a dev dependency but its purpose isn't clear from the changes shown. Check if async testing is actually needed
README.md
Outdated
| ``` | ||
|
|
||
| > [!NOTE] | ||
| > When max-size is reached, `yek` will throw away all of the less important files and generate |
There was a problem hiding this comment.
logic: This line appears to be cut off mid-sentence. Complete the explanation of what happens when max-size is reached.
README.md
Outdated
| # Tokenizer model for token counting (defaults to 'deepseek-reasoner') | ||
| tokens = "deepseek" |
There was a problem hiding this comment.
logic: The default model is stated as 'deepseek-reasoner' here but 'openai' is shown as default in the CLI help (line 141). This inconsistency needs to be resolved.
| // Verify all models are listed | ||
| let models = [ | ||
| "openai", // OpenAI models | ||
| "claude", // Anthropic Claude models | ||
| "mistral", // Mistral models | ||
| "mixtral", // Mixtral models | ||
| "deepseek", // DeepSeek models | ||
| "llama", // Meta Llama models | ||
| "codellama", // CodeLlama models | ||
| ]; |
There was a problem hiding this comment.
style: Models array should be imported from model_manager.rs SUPPORTED_MODEL_FAMILIES to avoid duplication and ensure consistency
| "^.github/.*", | ||
| "^.cargo/.*", |
There was a problem hiding this comment.
logic: Redundant ignore pattern - .github/ is already covered by the pattern on line 13
mohsen1
force-pushed
the
refactor-input-handling-and-config
branch
from
1b27189 to
f10d33e
Compare
…ig-ai-fix-20250126_0319 AI Fix for refactor-input-handling-and-config (Attempts: 5)
…run/yek into refactor-input-handling-and-config
…ant ignore pattern
No description provided.