Skip to content

en-US_project_structure.md

Latest commit

 

History

History
249 lines (207 loc) · 12 KB

en-US_project_structure.md

File metadata and controls

249 lines (207 loc) · 12 KB

Project Structure

This document provides a concise overview of the deep-solutions project structure, dependency management, and development environment requirements.

Table of Contents

Directory Structure

deep-solutions/
├── src/deep_solutions/          # Main package (src-layout)
│   ├── __init__.py              # Public API exports
│   ├── _version.py              # Auto-generated by setuptools-scm
│   ├── tools/                   # User-facing utility tools (public API)
│   │   ├── helloworld/          # Template/example tool
│   │   │   ├── core.py          # Core functionality (@public_api)
│   │   │   ├── utils.py         # Utility functions (@public_api)
│   │   │   └── __init__.py      # Public API exports
│   │   └── parameter_search/    # Parameter search library
│   │       ├── core/            # Core engine (ParamSearcher)
│   │       ├── epochs/          # Built-in epoch implementations
│   │       ├── analyzers/       # Result analyzers (chart, best param)
│   │       └── pytorch/         # PyTorch DataLoader wrapper
│   └── _utils/                  # Internal utilities (not public API)
│       ├── timer.py             # Performance timing utility
│       ├── metrics.py           # Metrics collection
│       ├── decorators.py        # @public_api decorator
│       └── __init__.py          # Internal exports
│
├── tests/                       # Unit tests (pytest)
│   ├── __init__.py
│   ├── test_tools/              # Tests for user-facing tools
│   │   ├── test_helloworld.py   # Tests for helloworld tool
│   │   └── test_parameter_search/  # Tests for parameter search
│   └── test__utils/             # Tests for internal utilities
│       └── test_utils.py        # Tests for timer, metrics, decorators
│
├── poc/                         # Proof-of-concept examples
│   └── param_selector/          # Parameter selection POC
│       └── dataloader_param_selector.py  # Runnable DataLoader example
│
├── docs/                        # Documentation (bilingual)
│   ├── en-US/                   # English documentation
│   │   ├── devs/                # Development guides (setup, standards, CI, release)
│   │   ├── user-guide/          # User tutorials and usage instructions
│   │   ├── design/              # Architecture and design documents
│   │   └── index.md             # EN documentation index
│   └── zh-CN/                   # Chinese documentation (same structure as en-US)
│       ├── devs/                # 开发指南
│       ├── user-guide/          # 用户指南
│       ├── design/              # 设计文档
│       └── index.md             # ZH documentation index
│
├── scripts/                     # Development & release scripts
│   ├── check.sh                 # Run all local checks (format, lint, type, language, test)
│   ├── ci-local.sh              # Simulate full CI pipeline locally
│   ├── check_language.py        # Verify English-only code (calls document_checker)
│   ├── document_checker.py      # Check bilingual docs, broken refs, structure
│   ├── test_release.sh          # Automate TestPyPI test release
│   └── final_release.sh         # Automate production PyPI release
│
├── .github/                     # GitHub configuration
│   ├── dependabot.yml           # Dependabot dependency update config
│   ├── PULL_REQUEST_TEMPLATE.md # PR description template
│   ├── ISSUE_TEMPLATE/          # Issue templates
│   │   ├── config.yml           # Template chooser config
│   │   ├── bug_report.md        # Bug report template
│   │   ├── feature_request.md   # Feature request template
│   │   ├── documentation.md     # Documentation issue template
│   │   └── custom.md            # Custom issue template
│   └── workflows/               # GitHub Actions workflows
│       ├── ci.yml               # CI: lint, type-check, test
│       ├── report.yml           # Post test results to PR comments
│       ├── publish-test.yml     # Publish to TestPyPI + validate
│       └── publish.yml          # Publish to PyPI + create GitHub Release
│
├── pyproject.toml               # Project metadata, dependencies, tool config
├── tox.ini                      # Multi-Python-version testing (3.8-3.12)
├── environment.yml              # Conda env (Python version only)
├── .pre-commit-config.yaml      # Pre-commit hooks config
├── .gitmessage                  # Git commit message template
├── .gitignore                   # Git ignore rules
├── README.md                    # Project README (English)
├── README.zh-CN.md              # Project README (Chinese)
├── CHANGELOG.md                 # Release history
└── LICENSE                      # Apache 2.0 License

Key Structure Principles

tools/ vs _utils/: The codebase distinguishes between user-facing utilities and internal utilities:

  • tools/: User-facing utilities that are part of the public API. Each tool is a complete, task-oriented package that users directly import and use. Examples: helloworld, parameter_search.
  • _utils/: Internal, project-agnostic utility code used across the codebase. NOT part of the public API. Users should not import from _utils directly. Examples: Timer, MetricsCollector, @public_api decorator.

@public_api Decorator: Functions and classes decorated with @public_api are explicitly marked as stable public APIs. The decorator:

  • Prepends [PUBLIC API] to the docstring for visibility
  • Signals to users and developers that the API is stable and supported
  • Used in tools/ modules to mark user-facing functionality
  • Example: @public_api\ndef hello_world() -> str: ... │ ├── en-US/ # English documentation │ │ ├── devs/ # Development guides (setup, standards, CI, release) │ │ ├── user-guide/ # User tutorials and usage instructions │ │ ├── design/ # Architecture and design documents │ │ └── index.md # EN documentation index │ └── zh-CN/ # Chinese documentation (same structure as en-US) │ ├── devs/ # 开发指南 │ ├── user-guide/ # 用户指南 │ ├── design/ # 设计文档 │ └── index.md # ZH documentation index │ ├── scripts/ # Development & release scripts │ ├── check.sh # Run all local checks (format, lint, type, language, test) │ ├── ci-local.sh # Simulate full CI pipeline locally │ ├── check_language.py # Verify English-only code (calls document_checker) │ ├── document_checker.py # Check bilingual docs, broken refs, structure │ ├── test_release.sh # Automate TestPyPI test release │ └── final_release.sh # Automate production PyPI release │ ├── .github/ # GitHub configuration │ ├── dependabot.yml # Dependabot dependency update config │ ├── PULL_REQUEST_TEMPLATE.md # PR description template │ ├── ISSUE_TEMPLATE/ # Issue templates │ │ ├── config.yml # Template chooser config │ │ ├── bug_report.md # Bug report template │ │ ├── feature_request.md # Feature request template │ │ ├── documentation.md # Documentation issue template │ │ └── custom.md # Custom issue template │ └── workflows/ # GitHub Actions workflows │ ├── ci.yml # CI: lint, type-check, test │ ├── report.yml # Post test results to PR comments │ ├── publish-test.yml # Publish to TestPyPI + validate │ └── publish.yml # Publish to PyPI + create GitHub Release │ ├── pyproject.toml # Project metadata, dependencies, tool config ├── tox.ini # Multi-Python-version testing (3.8-3.12) ├── environment.yml # Conda env (Python version only) ├── .pre-commit-config.yaml # Pre-commit hooks config ├── .gitmessage # Git commit message template ├── .gitignore # Git ignore rules ├── README.md # Project README (English) ├── README.zh-CN.md # Project README (Chinese) ├── CHANGELOG.md # Release history └── LICENSE # Apache 2.0 License

---

## Dependency Management

### Dependency Strategy: Pip-in-Conda

- **Conda**: Manages only Python version and pip (`environment.yml`)
- **pip / pyproject.toml**: Manages all package dependencies
- This ensures dev and production dependencies stay in sync

### Dependency Categories (`pyproject.toml`)

| Category | Section | Purpose |
|----------|---------|---------|
| Core | `[project] dependencies` | Runtime requirements (numpy, scipy, torch, matplotlib) |
| Dev | `[project.optional-dependencies] dev` | Testing, linting, formatting, building tools |
| Docs | `[project.optional-dependencies] docs` | Sphinx documentation building |

### Core Dependencies

The following dependencies are required for all users and are automatically installed:

- **NumPy** (>=1.17.0): Numerical computing foundation
- **SciPy** (>=1.5.0): Scientific computing algorithms
- **PyTorch** (>=1.7.0): Deep learning framework (required for parameter search tools)
- **Matplotlib** (>=3.3.0): Visualization and plotting (required for chart generation)

These are defined in `pyproject.toml` under `[project] dependencies`.

### Installation

```bash
# Core only
pip install .

# Core + dev (recommended for development)
pip install -e ".[dev]"

# All
pip install -e ".[dev,docs]"

Adding Dependencies

  1. Add to appropriate section in pyproject.toml
  2. Reinstall: pip install -e ".[dev]"

Note: Never add Python packages to environment.yml — it manages only the Python version.

Python Version Requirements

Python Version Status
3.8 ✅ Supported (development version)
3.9–3.12 ✅ Supported

⚠️ Development Must Use Python 3.8

Python 3.8 is the minimum supported version. Developing on 3.8 guarantees forward compatibility and prevents accidental use of newer syntax (match-case, X | Y unions, etc.).

conda create -n deep-solutions python=3.8 -y
conda activate deep-solutions
pip install -e ".[dev]"

Configuration Files

File Purpose
pyproject.toml Project metadata, dependencies, tool config (ruff, pytest, mypy, commitizen, setuptools-scm)
tox.ini Multi-version testing across Python 3.8–3.12
environment.yml Conda environment specification (Python version only)
.pre-commit-config.yaml Pre-commit hooks for code quality
.gitmessage Git commit message template

Version Management

The project uses setuptools-scm to derive versions from Git tags:

  • Tag format: v<MAJOR>.<MINOR>.<PATCH> (e.g., v1.0.0)
  • Auto-generated: src/deep_solutions/_version.py
  • Follows: Semantic Versioning

Related Documentation