Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
include README.md LICENSE requirements.txt main.py
recursive-include src/library *.yml
Copy link
Contributor

@elad-pticha elad-pticha Jun 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to library *.yml

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the whole point of this PR, right now you are not including the library within the Python package, and the culprit is that the library it outside the package source, you could maybe do a build hook with setuptools but that is way more complicated than just moving it into the source tree.

See below the build log, the yml files are never included:

$ python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/src
copying src/__init__.py -> build/lib/src
copying src/cmdline.py -> build/lib/src
creating build/lib/src/common
copying src/common/__init__.py -> build/lib/src/common
copying src/common/ignore_warnings.py -> build/lib/src/common
copying src/common/utils.py -> build/lib/src/common
creating build/lib/src/config
copying src/config/__init__.py -> build/lib/src/config
copying src/config/config.py -> build/lib/src/config
creating build/lib/src/downloader
copying src/downloader/__init__.py -> build/lib/src/downloader
copying src/downloader/download.py -> build/lib/src/downloader
copying src/downloader/gh_api.py -> build/lib/src/downloader
copying src/downloader/utils.py -> build/lib/src/downloader
creating build/lib/src/indexer
copying src/indexer/__init__.py -> build/lib/src/indexer
copying src/indexer/index.py -> build/lib/src/indexer
creating build/lib/src/logger
copying src/logger/__init__.py -> build/lib/src/logger
copying src/logger/log.py -> build/lib/src/logger
creating build/lib/src/queries
copying src/queries/__init__.py -> build/lib/src/queries
creating build/lib/src/reporter
copying src/reporter/__init__.py -> build/lib/src/reporter
copying src/reporter/report.py -> build/lib/src/reporter
copying src/reporter/slack_reporter.py -> build/lib/src/reporter
creating build/lib/src/storage
copying src/storage/__init__.py -> build/lib/src/storage
copying src/storage/neo4j_graph.py -> build/lib/src/storage
copying src/storage/neo4j_utils.py -> build/lib/src/storage
copying src/storage/redis_connection.py -> build/lib/src/storage
copying src/storage/redis_utils.py -> build/lib/src/storage
creating build/lib/src/workflow_components
copying src/workflow_components/__init__.py -> build/lib/src/workflow_components
copying src/workflow_components/composite_action.py -> build/lib/src/workflow_components
copying src/workflow_components/dependency.py -> build/lib/src/workflow_components
copying src/workflow_components/parsing_utils.py -> build/lib/src/workflow_components
copying src/workflow_components/workflow.py -> build/lib/src/workflow_components
/usr/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/src
copying build/lib/src/__init__.py -> build/bdist.linux-x86_64/wheel/src
copying build/lib/src/cmdline.py -> build/bdist.linux-x86_64/wheel/src
creating build/bdist.linux-x86_64/wheel/src/common
copying build/lib/src/common/__init__.py -> build/bdist.linux-x86_64/wheel/src/common
copying build/lib/src/common/ignore_warnings.py -> build/bdist.linux-x86_64/wheel/src/common
copying build/lib/src/common/utils.py -> build/bdist.linux-x86_64/wheel/src/common
creating build/bdist.linux-x86_64/wheel/src/config
copying build/lib/src/config/__init__.py -> build/bdist.linux-x86_64/wheel/src/config
copying build/lib/src/config/config.py -> build/bdist.linux-x86_64/wheel/src/config
creating build/bdist.linux-x86_64/wheel/src/downloader
copying build/lib/src/downloader/__init__.py -> build/bdist.linux-x86_64/wheel/src/downloader
copying build/lib/src/downloader/download.py -> build/bdist.linux-x86_64/wheel/src/downloader
copying build/lib/src/downloader/gh_api.py -> build/bdist.linux-x86_64/wheel/src/downloader
copying build/lib/src/downloader/utils.py -> build/bdist.linux-x86_64/wheel/src/downloader
creating build/bdist.linux-x86_64/wheel/src/indexer
copying build/lib/src/indexer/__init__.py -> build/bdist.linux-x86_64/wheel/src/indexer
copying build/lib/src/indexer/index.py -> build/bdist.linux-x86_64/wheel/src/indexer
creating build/bdist.linux-x86_64/wheel/src/logger
copying build/lib/src/logger/__init__.py -> build/bdist.linux-x86_64/wheel/src/logger
copying build/lib/src/logger/log.py -> build/bdist.linux-x86_64/wheel/src/logger
creating build/bdist.linux-x86_64/wheel/src/queries
copying build/lib/src/queries/__init__.py -> build/bdist.linux-x86_64/wheel/src/queries
creating build/bdist.linux-x86_64/wheel/src/reporter
copying build/lib/src/reporter/__init__.py -> build/bdist.linux-x86_64/wheel/src/reporter
copying build/lib/src/reporter/report.py -> build/bdist.linux-x86_64/wheel/src/reporter
copying build/lib/src/reporter/slack_reporter.py -> build/bdist.linux-x86_64/wheel/src/reporter
creating build/bdist.linux-x86_64/wheel/src/storage
copying build/lib/src/storage/__init__.py -> build/bdist.linux-x86_64/wheel/src/storage
copying build/lib/src/storage/neo4j_graph.py -> build/bdist.linux-x86_64/wheel/src/storage
copying build/lib/src/storage/neo4j_utils.py -> build/bdist.linux-x86_64/wheel/src/storage
copying build/lib/src/storage/redis_connection.py -> build/bdist.linux-x86_64/wheel/src/storage
copying build/lib/src/storage/redis_utils.py -> build/bdist.linux-x86_64/wheel/src/storage
creating build/bdist.linux-x86_64/wheel/src/workflow_components
copying build/lib/src/workflow_components/__init__.py -> build/bdist.linux-x86_64/wheel/src/workflow_components
copying build/lib/src/workflow_components/composite_action.py -> build/bdist.linux-x86_64/wheel/src/workflow_components
copying build/lib/src/workflow_components/dependency.py -> build/bdist.linux-x86_64/wheel/src/workflow_components
copying build/lib/src/workflow_components/parsing_utils.py -> build/bdist.linux-x86_64/wheel/src/workflow_components
copying build/lib/src/workflow_components/workflow.py -> build/bdist.linux-x86_64/wheel/src/workflow_components
running install_egg_info
running egg_info
creating raven_cycode.egg-info
writing raven_cycode.egg-info/PKG-INFO
writing dependency_links to raven_cycode.egg-info/dependency_links.txt
writing entry points to raven_cycode.egg-info/entry_points.txt
writing requirements to raven_cycode.egg-info/requires.txt
writing top-level names to raven_cycode.egg-info/top_level.txt
writing manifest file 'raven_cycode.egg-info/SOURCES.txt'
reading manifest file 'raven_cycode.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'LICENSE'
adding license file 'LICENSE.md'
writing manifest file 'raven_cycode.egg-info/SOURCES.txt'
Copying raven_cycode.egg-info to build/bdist.linux-x86_64/wheel/raven_cycode-0.0.0-py3.12.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/raven_cycode-0.0.0.dist-info/WHEEL
creating 'dist/raven_cycode-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'src/__init__.py'
adding 'src/cmdline.py'
adding 'src/common/__init__.py'
adding 'src/common/ignore_warnings.py'
adding 'src/common/utils.py'
adding 'src/config/__init__.py'
adding 'src/config/config.py'
adding 'src/downloader/__init__.py'
adding 'src/downloader/download.py'
adding 'src/downloader/gh_api.py'
adding 'src/downloader/utils.py'
adding 'src/indexer/__init__.py'
adding 'src/indexer/index.py'
adding 'src/logger/__init__.py'
adding 'src/logger/log.py'
adding 'src/queries/__init__.py'
adding 'src/reporter/__init__.py'
adding 'src/reporter/report.py'
adding 'src/reporter/slack_reporter.py'
adding 'src/storage/__init__.py'
adding 'src/storage/neo4j_graph.py'
adding 'src/storage/neo4j_utils.py'
adding 'src/storage/redis_connection.py'
adding 'src/storage/redis_utils.py'
adding 'src/workflow_components/__init__.py'
adding 'src/workflow_components/composite_action.py'
adding 'src/workflow_components/dependency.py'
adding 'src/workflow_components/parsing_utils.py'
adding 'src/workflow_components/workflow.py'
adding 'raven_cycode-0.0.0.dist-info/LICENSE.md'
adding 'raven_cycode-0.0.0.dist-info/METADATA'
adding 'raven_cycode-0.0.0.dist-info/WHEEL'
adding 'raven_cycode-0.0.0.dist-info/entry_points.txt'
adding 'raven_cycode-0.0.0.dist-info/top_level.txt'
adding 'raven_cycode-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

recursive-include tests *.py
recursive-include src *
3 changes: 1 addition & 2 deletions deployment/test.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ RUN mkdir -p /raven/tests
WORKDIR /raven
COPY Makefile requirements.txt /raven/
COPY src /raven/src
COPY library /raven/library
COPY tests /raven/tests

# Install any needed packages specified in requirements.txt
RUN pip3 install -r requirements.txt

# Run RAVEN tests
CMD ["make", "test-run"]
CMD ["make", "test-run"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove extra line added

1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,5 @@
install_requires=REQUIRMENTS,
packages=find_packages(exclude=("tests", "tests.*")),
entry_points={"console_scripts": ["raven = src.cmdline:execute"]},
include_package_data=True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required?
I tried to build the package without this, and it worked. Is there something I am missing?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a safeguard, it looks like it's at True by default but the package wants the data to be included, so I'd suggest to keep it explicit.

)
4 changes: 3 additions & 1 deletion src/config/config.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
from src.storage.neo4j_graph import GraphDb
from pathlib import Path
import src

# Default Values
DEBUG_DEFAULT = False
MIN_STARS_DEFAULT = 1000
REDIS_CLEAN_DEFAULT = False
NEO4J_CLEAN_DEFAULT = False
QUERIES_PATH_DEFAULT = "library"
QUERIES_PATH_DEFAULT = Path(src.__file__).resolve().parent / "library"
REPORT_RAW_FORMAT = "raw"
REPORT_JSON_FORMAT = "json"
SLACK_REPORTER = "slack"
Expand Down