Change Internal to utils and build stuff

README.md

@@ -1,6 +1,6 @@
 # OneFile: Comprehensive Project and Package Management Tool
 
-OneFile is a versatile command-line tool designed to streamline various operations on project structures and package contents. It enables developers to easily dump local projects, reconstruct projects from JSON, convert JSON to Markdown, and fetch contents from GitHub repositories and various package managers.
+OneFile is a versatile command-line tool designed to streamline various operations on project structures and package contents. It enables developers to easily dump local projects, reconstruct projects from JSON, convert JSON to Markdown, and fetch contents from GitHub repositories and PyPI packages.
 
 ## Features
 
@@ -12,7 +12,8 @@ OneFile is a versatile command-line tool designed to streamline various operatio
 - **Flexible Output**: Choose between JSON and Markdown output formats.
 - **Progress Reporting**: View download progress for fetching operations.
 - **Customizable Inclusion/Exclusion**: Use patterns to include or exclude specific files.
-- **Individual Command Execution**: Use as a single command with subcommands or as separate standalone commands.
+- **Git Integration**: Option to use git clone for faster repository fetching.
+- **GitHub Token Support**: Authenticate with GitHub API for higher rate limits.
 
 ## Installation
 
@@ -26,9 +27,9 @@ go install github.com/gusanmaz/onefile@latest
 
 This installs the `onefile` binary in your `$GOPATH/bin` directory. Ensure this directory is in your system's PATH to run OneFile from anywhere.
 
-### Option 2: Build individual commands
+### Option 2: Build from source
 
-To use commands separately, clone the repository and build individual commands:
+To build the project from source, clone the repository and use the provided build script:
 
 ```bash
 git clone https://github.com/gusanmaz/onefile.git
@@ -37,7 +38,7 @@ chmod +x build.sh
 ./build.sh
 ```
 
-This creates individual executables for each command in the `bin` directory.
+This creates the main `onefile` executable and individual command executables in the `bin` directory.
 
 ## Usage
 
@@ -55,8 +56,9 @@ Flags:
 - `-p, --path`: Project root path (default: current directory)
 - `-o, --output`: Output file name (without extension)
 - `-t, --type`: Output type: 'json' or 'md' (default: 'json')
-- `-i, --include`: Patterns to include files (space-separated)
 - `-e, --exclude`: Patterns to exclude files (space-separated)
+- `--include-git`: Include .git files and directories
+- `--include-non-text`: Include non-text files
 
 #### 2. Reconstructing a Project from JSON
 
@@ -77,127 +79,60 @@ onefile json2md -j project_data.json -o project_structure.md
 Flags:
 - `-j, --json`: Input JSON file
 - `-o, --output`: Output Markdown file
+- `--include-git`: Include .git files and directories
+- `--include-non-text`: Include non-text files
 
 #### 4. Fetching GitHub Repository
 
 ```sh
-onefile github2file -u https://github.com/username/repo -t json -o output_file.json
+onefile github2file -u https://github.com/username/repo -t json -o output_file
 ```
 
 Flags:
 - `-u, --url`: Full GitHub repository URL
 - `-t, --type`: Output type: 'json' or 'md' (default: 'md')
-- `-o, --output`: Output file name (without extension)
+- `-o, --output-name`: Output file name (without extension)
 - `-d, --output-dir`: Output directory
-- `-i, --include`: Patterns to include files (space-separated)
 - `-e, --exclude`: Patterns to exclude files (space-separated)
 - `-a, --all-repos`: Fetch all repositories for a user
+- `-g, --use-git`: Use git clone if available (default: true)
+- `-k, --token`: GitHub API token
+- `--include-git`: Include .git files and directories
+- `--include-non-text`: Include non-text files
 
 #### 5. Fetching PyPI Package
 
 ```sh
-onefile pypi2file -p package_name -t json -o output_file.json
+onefile pypi2file -p package_name -t json -o output_file
 ```
 
 Flags:
 - `-p, --package`: PyPI package name
 - `-t, --type`: Output type: 'json' or 'md' (default: 'md')
-- `-o, --output`: Output file name (without extension)
+- `-o, --output-name`: Output file name (without extension)
 - `-d, --output-dir`: Output directory
-
-### Individual Command Usage
-
-If you've built individual commands, you can use them as follows:
-
-- Dump a local project:
-  ```
-  ./bin/dump -p /path/to/your/project -o output_file -t json
-  ```
-
-- Fetch a GitHub repository:
-  ```
-  ./bin/github2file -u https://github.com/username/repo -t json -o output_file.json
-  ```
-
-- Convert JSON to Markdown:
-  ```
-  ./bin/json2md -j project_data.json -o project_structure.md
-  ```
-
-- Fetch a PyPI package:
-  ```
-  ./bin/pypi2file -p package_name -t json -o output_file.json
-  ```
-
-- Reconstruct a project from JSON:
-  ```
-  ./bin/reconstruct -j project_data.json -o /path/to/output/directory
-  ```
+- `--include-git`: Include .git files and directories
+- `--include-non-text`: Include non-text files
 
 ## Use Cases
 
 1. **LLM Code Analysis**: Package entire projects for submission to Large Language Models for code review, refactoring suggestions, or documentation generation.
-
 2. **Project Snapshots**: Create snapshots of project states for version control or backup purposes.
-
 3. **Open Source Exploration**: Easily fetch and examine the structure of open-source projects on GitHub without cloning entire repositories.
-
 4. **Documentation Generation**: Automatically generate project structure documentation in Markdown format for wikis or README files.
-
 5. **Dependency Analysis**: Fetch PyPI packages to analyze their structure and contents before including them in your project.
-
 6. **Code Sharing**: Share project structures and contents with colleagues or in forum posts without zipping and uploading entire projects.
-
 7. **Project Comparisons**: Dump multiple project versions to JSON and use diff tools to compare structures and contents over time.
-
 8. **Automated Tooling**: Incorporate OneFile into CI/CD pipelines for automated project analysis, documentation updates, or dependency checks.
 
-## Example Workflow
-
-1. **Dump a local project to JSON:**
-
-```sh
-onefile dump -p /path/to/your/project -o project_data -t json
-```
-
-2. **Convert the JSON to Markdown:**
-
-```sh
-onefile json2md -j project_data.json -o project_structure.md
-```
-
-3. **Fetch a GitHub repository and create a Markdown file:**
-
-```sh
-onefile github2file -u https://github.com/octocat/Hello-World -t md -o github_project
-```
-
-4. **Fetch a PyPI package and create a JSON file:**
-
-```sh
-onefile pypi2file -p numpy -t json -o numpy_package
-```
-
 ## Contributing
 
-Contributions to OneFile are welcome! Please feel free to submit pull requests, open issues, or suggest new features. Here are some areas where contributions could be particularly valuable:
-
-- Adding support for more package managers (npm, Maven, RubyGems, etc.)
-- Improving error handling and user feedback
-- Enhancing performance for large projects or repositories
-- Adding more output format options
-- Creating comprehensive test suites
-
-Before contributing, please review our contribution guidelines (CONTRIBUTING.md).
+Contributions to OneFile are welcome! Please feel free to submit pull requests, open issues, or suggest new features.
 
 ## License
 
 This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
 
 ## Support
 
-If you encounter any issues or have questions about using OneFile, please open an issue on our GitHub repository. We'll do our best to assist you promptly.
-
-## Acknowledgments
-
-We'd like to thank all the contributors who have helped make OneFile a robust and useful tool for the developer community. Your efforts are greatly appreciated!
+If you encounter any issues or have questions about using OneFile, please open an issue on our GitHub repository.

build.sh

@@ -3,6 +3,9 @@
 # Ensure we're in the project root
 cd "$(dirname "$0")"
 
+# Run go mod tidy to ensure dependencies are up to date
+go mod tidy
+
 # Build the main onefile command
 go build -o bin/onefile main.go
 

go.mod

@@ -1,4 +1,4 @@
-module onefile
+module github.com/gusanmaz/onefile
 
 go 1.16
 
@@ -9,4 +9,3 @@ require (
 	github.com/spf13/cobra v1.2.1
 )
 
-replace github.com/gusanmaz/onefile => ./

go.mod.local

@@ -0,0 +1,12 @@
+module onefile
+
+go 1.16
+
+require (
+	github.com/gusanmaz/onefile v0.0.0-20240701201240-b18e40cc9eea
+	github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06
+	github.com/schollz/progressbar/v3 v3.8.2
+	github.com/spf13/cobra v1.2.1
+)
+
+replace github.com/gusanmaz/onefile => ./

go.sum

@@ -138,6 +138,7 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
+github.com/gusanmaz/onefile v0.0.0-20240701201240-b18e40cc9eea/go.mod h1:daN5GEfAcummIJXgBMTNr0XMnzWlGr/0QqEPPGkp2h0=
 github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
 github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8=
 github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=