Cannot Replace Model Deployed at PrivateEndpoint with a New Model Version

[sapphire008]

• Package version: 1.73.0
• Python 3.12.4
• MacOS 14.6.1; (also encountered the same problem on Vertex AI Pipeline run as a custom component)

I am deploying models to a PrivateEndpoint that attaches to a VPC network, using the following script.

# already created PSA endpoint
endpoint = aiplatform.PrivateEndpoint(
    endpoint_name=endpoint_resource_name
)
# already uploaded model
model = aiplatform.Model('projects/xxxxx/locations/us-central1/models/test_model_name@7')

# deploy new version to existing private endpoint
model.deploy(
            endpoint=endpoint,
            deployed_model_display_name=model_display_name,
            machine_type=machine_type,
            traffic_percentage=100,
            min_replica_count=int(min_replica_count),
            max_replica_count=int(max_replica_count),
            network="projects/xxxxxx/global/networks/default",
        )

or alternatively, I have also attempted to use endpoint.deploy

endpoint.deploy(
           model=model,
           deployed_model_display_name=model_display_name,
            machine_type=machine_type,
            traffic_percentage=100,
            min_replica_count=int(min_replica_count),
            max_replica_count=int(max_replica_count),
)

When the endpoint was first created, this works fine. However, if there is a pre-existing model deployed, then deploying a new version of the model will throw the following error:

google.api_core.exceptions.FailedPrecondition: 400 There might be another deployed model or a failed deployed model that hasn't been cleaned up under the same private endpoint, please try again later or create another endpoint.

I also tried to use gcloud ai command line

gcloud ai endpoints deploy-model $ENDPOINT_NAME --project=$PROJECT_ID --region=us-central1 --model=$MODEL_NAME --display-name=$DISPLAY_NAME

The doc-string of the PrivateEndpoint.deploy method, the traffic_percentage states the following

traffic_percentage (int):
                Optional. Desired traffic to newly deployed model.
                Defaults to 0 if there are pre-existing deployed models.
                Defaults to 100 if there are no pre-existing deployed models.
                Defaults to 100 for PSA based private endpoint.
                Negative values should not be provided. Traffic of previously
                deployed models at the endpoint will be scaled down to
                accommodate new deployed model's traffic.
                Should not be provided if traffic_split is provided.

So it appears replacing a model with another model should have been supported. Setting traffic_percentage to 0 or commenting this argument out got me the same error as previously. I have also tried to set this traffic_percentage argument to None, and got

 if traffic_percentage > 100:
           ^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: '>' not supported between instances of 'NoneType' and 'int'

Expected behavior: The PrivateEndpoint should be able to deploy new model version replacing the traffic of older model versions.

Other considerations (but maybe out of scope of the ticket): I need to access a MemoryStore Redis instance on GCP, and so far, only deploying the endpoint and Redis with PSA worked.