-
Notifications
You must be signed in to change notification settings - Fork 372
Added docs for new compression plugin and created a plugin subfolder #4107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
perseoGI
wants to merge
4
commits into
conan-io:develop2
Choose a base branch
from
perseoGI:pgi/compression_plugin
base: develop2
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
9e910f6
Added docs for new compression plugin and created a plugin/ subfolder…
perseoGI 178ff46
Update reference/extensions/plugins.rst
AbrilRBS 6bad714
Apply suggestions from code review
AbrilRBS e82e47c
Apply suggestions from code review
AbrilRBS File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
.. _reference_plugins: | ||
|
||
Plugins | ||
========== | ||
|
||
Conan plugins are a powerful mechanism to customize and extend Conan's built-in | ||
behavior. They are Python scripts loaded dynamically from the user's local | ||
Conan cache, allowing users to intercept and augment specific parts of the | ||
Conan workflow without modifying the Conan source code. | ||
|
||
Plugins are ideal for advanced scenarios where teams or organizations need to: | ||
|
||
- Enforce custom authorization or access control rules. | ||
- Dynamically modify profiles or settings based on complex logic. | ||
- Customize compression or packaging formats to optimize bandwidth or storage. | ||
|
||
Plugins offer a clean, maintainable, and shareable way to implement advanced | ||
behavior, particularly in CI/CD pipelines or large-scale deployments. | ||
|
||
They can be distributed and shared using ``conan config install``, making it easy to apply | ||
consistent behavior across all users in a team. | ||
|
||
The following types of plugins are currently supported: | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
plugins/authorization_plugins | ||
plugins/profile_plugin | ||
plugins/compression_plugin | ||
|
||
Each plugin type has a specific purpose and interface, which is documented in its corresponding section. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
.. _reference_extensions_compression_plugin: | ||
|
||
Compression plugin | ||
------------------ | ||
|
||
.. include:: ../../../common/experimental_warning.inc | ||
|
||
The ``compression.py`` plugin is a Conan extension that allows users to | ||
customize the compression and extraction processes for all files managed by Conan. | ||
|
||
To activate it, place the plugin at: ``extensions/plugins/compression.py``. | ||
|
||
This plugin provides flexibility in how Conan packages are compressed and | ||
extracted, making it especially useful in scenarios such as: | ||
|
||
- Replacing the default ``gzip`` compression algorithm with a more efficient one like ``zstd`` or ``xz``. | ||
- Embedding custom metadata into the compressed archive. | ||
- Modifying the internal structure of the package content. | ||
- Modifying the file or directory permissions inside the compressed archive. | ||
- Manage symbolic links inside archives. | ||
|
||
These capabilities can help organizations reduce bandwidth and storage usage, | ||
or enforce specific packaging policies. | ||
|
||
.. important:: | ||
|
||
Once this plugin is enabled, all operations involving ``conan upload`` and | ||
``conan download`` will use the user-defined compression and extraction functions. | ||
This implies that **all users** within the same organization **must install** the | ||
plugin to avoid incompatibilities during extraction. | ||
|
||
You can distribute and synchronize your configuration by packaging it and installing it via ``conan config install``. | ||
|
||
Plugin Interface | ||
++++++++++++++++ | ||
|
||
To implement a custom compression plugin, define the following two functions in ``compression.py``: | ||
|
||
.. code-block:: python | ||
|
||
def tar_extract(archive_path: str, dest_dir: str, conf=None, ref=None, *args, **kwargs) -> None: | ||
AbrilRBS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
... | ||
|
||
def tar_compress(archive_path: str, files: dict[str,str], recursive=False, conf=None, ref=None, *args, **kwargs) -> None: | ||
AbrilRBS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
... | ||
|
||
|
||
- ``archive_path``: Path to the final archive file. This value is immutable. | ||
|
||
.. important:: | ||
|
||
Even if you use a different compression algorithm, the output file must retain | ||
the ``.tgz`` extension. This is required for Conan to correctly handle archives. | ||
Changing the extension will **break** the workflow. | ||
|
||
- ``files``: Dictionary of files to be compressed in the form ``{name_in_tar: absolute_path}``. | ||
- ``recursive``: Whether to include subdirectories when adding files. | ||
- ``conf``: Conan configuration object with user-defined options. It can be used to retrieve custom settings, such as compression level in the following way: | ||
|
||
.. code-block:: python | ||
|
||
compresslevel = conf.get("core.gzip:compresslevel", check_type=int) if conf else None | ||
|
||
|
||
Also, the ``conf`` object can be used to retrieve other custom configuration options that might be relevant for the compression process. | ||
|
||
- ``ref``: Optional Conan reference (e.g., package or recipe reference) useful for logging. | ||
|
||
|
||
|
||
Example: Compression Plugin Using xz | ||
++++++++++++++++++++++++++++++++++++ | ||
|
||
This example shows how to implement a plugin using the ``xz`` compression format: | ||
|
||
.. code-block:: python | ||
|
||
import os | ||
import tarfile | ||
from conan.api.output import ConanOutput | ||
|
||
def tar_compress(archive_path, files, recursive, conf=None, *args, **kwargs): | ||
name = os.path.basename(archive_path) | ||
ConanOutput().info(f"Compressing {name} using compression plugin (xz)") | ||
compresslevel = conf.get("core.gzip:compresslevel", check_type=int) if conf else None | ||
tar_kwargs = {"preset": compresslevel} if compresslevel else {} | ||
with tarfile.open(archive_path, f"w:xz", **tar_kwargs) as txz: | ||
for filename, abs_path in sorted(files.items()): | ||
txz.add(abs_path, filename, recursive=True) | ||
|
||
def tar_extract(archive_path, dest_dir, conf=None, *args, **kwargs): | ||
ConanOutput().info(f"Decompressing {os.path.basename(archive_path)} using compression plugin (xz)") | ||
with open(archive_path, mode='rb') as file_handler: | ||
txz = tarfile.open(fileobj=file_handler) | ||
txz.extraction_filter = (lambda member, path: member) | ||
txz.extractall(path=dest_dir) | ||
txz.close() | ||
|
2 changes: 1 addition & 1 deletion
2
reference/extensions/profile_plugin.rst → ...nce/extensions/plugins/profile_plugin.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.