refactor: split Dockerfiles into prod and dev files #95
Conversation
… management and development workflow - Updated Dockerfiles for services and libs to use Python 3.13 and 3.11.7. - Introduced separate Dockerfile.dev for development environments with optimized caching. - Enhanced dependency installation logic in Dockerfiles to streamline dev and prod setups. - Updated Poetry version to 2.1.3 across all services and libs. - Refined pyproject.toml files to include TestPyPI sources and group dependencies for better management. - Improved logging setup in Dockerfiles for better runtime diagnostics.
- Moved test dependencies to a dedicated group and added pytest and coverage. - Created a separate lint group with flake8 and related plugins. - Added debugpy and rag-core libraries to the dev group. - Updated production dependencies for rag-core libraries to version 2.2.1.
- Updated `pyproject.toml` and `poetry.lock` files to enforce `setuptools` version to be less than 80.9 in multiple services. - Adjusted per-file ignores for linting in `admin-backend` and `document-extractor`. - Streamlined Dockerfiles for `document-extractor`, `rag-backend`, and `mcp-server` to improve build efficiency and user permissions. - Ensured consistent formatting in dependency declarations across services. - Updated versioning for `pdfextractor_server` to 2.2.1. - Added minimal library structures in Dockerfiles to facilitate local package recognition by Poetry.
- Moved `pytest` and `pytest-asyncio` to a new test dependency group in `rag-core-lib`. - Added `coverage` to the test dependencies in `rag-core-lib`. - Created a separate lint dependency group in `rag-core-lib` and included `flake8` and related packages. - Updated markers in `poetry.lock` for better platform compatibility. - Removed `pytest-asyncio` from the `rag-backend` dependencies and adjusted its group to only include `test`.
… prod-local group for development
… Dockerfile cleanup and user permissions; adjust pyproject.toml for local dependencies
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the Docker build structure to separate development and production concerns, implementing dedicated dev and production Dockerfiles across all services and libraries. The changes introduce a new development mode flag for Tilt, reorganize Poetry dependency groups for better separation of concerns, and update various configuration files to support the new dual-dockerfile approach.
- Split Dockerfiles into production-optimized builds and development-optimized builds with live code updates
- Reorganized Poetry dependency groups into test, lint, dev, and production groups across all services and libraries
- Updated Tilt configuration to support development mode with live updates and production mode for realistic testing
Reviewed Changes
Copilot reviewed 23 out of 32 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| services/*/pyproject.toml | Reorganized Poetry dependency groups and added setuptools compatibility constraint |
| services/*/Dockerfile.dev | New development-optimized Dockerfiles with fast iteration and live code updates |
| services/*/Dockerfile | Updated production Dockerfiles with conditional dependency groups and multi-stage builds |
| libs/*/pyproject.toml | Reorganized dependency groups and updated library versions to 2.2.1 |
| libs/Dockerfile | Updated Poetry version and improved conditional installation logic |
| Tiltfile | Added development mode support and updated build configurations |
| .github/workflows/* | Updated CI workflows to use new dev Dockerfiles and removed build arguments |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| | .mypy_cache | ||
| | .nox | ||
| | .pants.d | ||
| | .pants.dgit |
There was a problem hiding this comment.
The exclude pattern '.pants.dgit' appears to be a typo. It should likely be '.pants.d' as in the other pyproject.toml files.
| | .pants.dgit | |
| | .pants.d |
Copilot uses AI. Check for mistakes.
| RUN mkdir -p log && chmod 700 log \ | ||
| && touch /app/services/admin-backend/log/logfile.log && chmod 600 /app/services/admin-backend/log/logfile.log | ||
|
|
||
| ENV PYTHONPATH="/app/services/admin-backend/src:/app/libs/rag-core-api/src:/app/libs/rag-core-lib/src" |
There was a problem hiding this comment.
The PYTHONPATH includes '/app/libs/rag-core-api/src' but this service only depends on 'admin-api-lib' and 'rag-core-lib' according to the pyproject.toml. This incorrect path could cause import issues.
| ENV PYTHONPATH="/app/services/admin-backend/src:/app/libs/rag-core-api/src:/app/libs/rag-core-lib/src" | |
| ENV PYTHONPATH="/app/services/admin-backend/src:/app/libs/rag-core-lib/src" |
Copilot uses AI. Check for mistakes.
| COPY services/document-extractor/pyproject.toml services/document-extractor/poetry.lock /app/services/document-extractor/ | ||
| COPY libs/extractor-api-lib/pyproject.toml libs/extractor-api-lib/poetry.lock /app/libs/extractor-api-lib/ | ||
|
|
||
| COPY --chown=nonroot:nonroot libs/extractor-api-lib/src /app/libs/extractor-api-lib/src |
There was a problem hiding this comment.
The COPY command uses '--chown=nonroot:nonroot' but the nonroot user is created later in the Dockerfile (line 35). This will cause the build to fail.
Copilot uses AI. Check for mistakes.
This pull request introduces significant improvements to the development and build workflows for both the backend services and Python libraries. The main changes include a clearer separation between development and production Docker builds, enhanced configuration for local development via Tilt, and improved dependency management and testing setup for Python libraries. These updates streamline live development, testing, and linting processes, making them more reliable and easier to maintain.
Build and workflow improvements:
Tiltfile, allowing dynamic switching betweenDockerfile.dev(for live code updates and fast iteration) and productionDockerfile(for optimized builds). Also updated Docker ignore lists for more targeted builds. [1] [2] [3] [4] [5] [6] [7]Dockerfile.devfor service builds during linting and testing, removed unnecessary build arguments, and set Python version to 3.13 for consistency. [1] [2] [3] [4]Python library dependency and testing enhancements:
test,lint, anddevdependencies, addedsetuptools <80.9for compatibility, and updated library versions. Also improvedpytestconfiguration for better logging and path setup. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]Docker build improvements for Python libraries:
libs/Dockerfileto use Poetry 2.1.3, improved conditional installation logic for dev/test/lint dependencies, and ensured correct installation of shared dependencies for interdependent libraries.Frontend workflow fixes:
npm installruns before frontend linting and testing in Tilt, and expanded dependency tracking for these local resources. [1] [2]Dependency version updates:
fasttextto a stable release and updating internal library versions to 2.2.1. [1] [2] [3] [4]Let me know if you want to walk through any of these changes in detail or discuss how they impact your local development workflow!