Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multiple mirror sync with extensive include/exclude pattern matching #639

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
cb4f948
start: allow multiple mirrors per package, extensive include/exclude …
YYYasin19 Jun 1, 2023
267b94a
use condas MatchSpec utility
YYYasin19 Jun 9, 2023
557cec0
package spec relies on = while our repo package names use '-'
YYYasin19 Jun 9, 2023
df8a217
validate that package was therefore before delete attempt is issued
YYYasin19 Jun 9, 2023
5815902
update: include condas matchspec
YYYasin19 Jun 15, 2023
de27969
refactor: extract repodata retrieval || add: package matching on more…
YYYasin19 Jun 15, 2023
dbd3f75
add: support for excludelist
YYYasin19 Jun 29, 2023
1d4af06
add: new database model for accessing channel urls
YYYasin19 Jun 29, 2023
0f07838
fix: not set mirror_channel_url
YYYasin19 Jun 29, 2023
a5e3847
fix: mirror_channel_urlS
YYYasin19 Jun 29, 2023
8217368
add: new tests for validating pattern checking
YYYasin19 Jun 29, 2023
c3c5467
refactor: channel membership is completely refactored
YYYasin19 Jul 18, 2023
b2af339
add + refactor: option to remove local packages that would _not_ fit …
YYYasin19 Jul 18, 2023
76e7c29
refactoring: clean up code, extract methods, document some others
YYYasin19 Aug 17, 2023
1f6be80
debug: update version to validate deployment
YYYasin19 Aug 22, 2023
8b4b965
add custom URLType for validating urls
YYYasin19 Aug 29, 2023
223e29a
fix rest models
YYYasin19 Aug 29, 2023
3ca604c
Update docs/source/using/mirroring.rst
YYYasin19 Sep 8, 2023
854d471
Update quetz/dao.py
YYYasin19 Sep 8, 2023
cf5cc7a
fix
YYYasin19 Sep 8, 2023
f6480b7
validate that proxy mode only allows a single mirror url
YYYasin19 Sep 8, 2023
e0f3bf1
add more commments
YYYasin19 Sep 8, 2023
ce554b6
add docs
YYYasin19 Sep 8, 2023
299d89b
add comment
YYYasin19 Sep 8, 2023
1afd522
rename method
YYYasin19 Sep 8, 2023
0c83779
add type hints
YYYasin19 Sep 8, 2023
a3b39db
use dist for parsing
YYYasin19 Sep 8, 2023
fc5c4bf
sooner package creation based on cond
YYYasin19 Sep 8, 2023
ff90938
simplify package removal from database
YYYasin19 Sep 8, 2023
5f951d0
update comments
YYYasin19 Sep 8, 2023
4108791
replaced Pydantic Field Validation with Annotated by constr
YYYasin19 Sep 8, 2023
2ed0e86
use enum instead of concrete values for mirror mode
YYYasin19 Sep 8, 2023
183f482
use enum instead of concrete values for mirror mode
YYYasin19 Sep 8, 2023
b51ef67
add docstring
YYYasin19 Sep 8, 2023
5ded7dd
add docstring
YYYasin19 Sep 8, 2023
c0acdbc
fix import
YYYasin19 Sep 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions quetz/rest_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,19 @@
import uuid
from datetime import date, datetime
from enum import Enum
from typing import Annotated, Dict, Generic, List, Optional, TypeVar, Union
from typing import Dict, Generic, List, NewType, Optional, TypeVar, Union

from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from pydantic import (
BaseModel,
ConfigDict,
Field,
constr,
field_validator,
model_validator,
)

T = TypeVar('T')
URLType = Annotated[str, Field(pattern="^(http|https)://.+")]
URLType = NewType('URLType', constr(pattern="^(http|https)://.+|None|^(\\[.*\\])+$"))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you have a URL type and define a pattern? https://docs.pydantic.dev/latest/usage/types/urls/

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, my thought was that due to implementation of AnyHttpUrl in Pydantic

AnyHttpUrl = Annotated[Url, UrlConstraints(allowed_schemes=['http', 'https'])]

this won't work as well.
But afaik, they only require Python 3.7 which we do as well.



class BaseProfile(BaseModel):
Expand Down Expand Up @@ -66,9 +73,7 @@ class ChannelBase(BaseModel):
private: bool = Field(True, title="channel should be private")
size_limit: Optional[int] = Field(None, title="size limit of the channel")
ttl: int = Field(36000, title="ttl of the channel")
mirror_channel_url: Optional[Union[URLType, List[URLType]]] = Field(
None,
)
mirror_channel_url: Optional[Union[URLType, List[URLType]]] = Field(None)
mirror_mode: Optional[MirrorMode] = Field(None, nullable=True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this an enum so that only proxy and mirror are allowed?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not already an enum..?


@field_validator("size_limit")
Expand Down