Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation during AzureFunctionApp Task takes too long #17718

Open
vannomad opened this issue Feb 7, 2023 · 19 comments
Open

Validation during AzureFunctionApp Task takes too long #17718

vannomad opened this issue Feb 7, 2023 · 19 comments
Assignees
@FinVamp1
Labels
FunctionsBacklogPri1 FunctionsTaskBacklog question Task: AzureFunction

Comments

@vannomad
Copy link

vannomad commented Feb 7, 2023

Type: Question

Enter Task Name: AzureFunctionAppV1, AzureFunctionAppV2

Issue Description

Both of these tasks take a long time (e.g. 13 minutes) just for the validation step to complete.

azure-pipelines-tasks/Tasks/AzureFunctionAppV2/azure-arm-rest/azure-arm-app-service-kudu.ts

Line 653 in 30c99c8

public async validateZipDeploy(webPackage: string, queryParameters?: Array<string>): Promise<any> {

This is the call that takes so long for no apparent reason.
Is this potentially a bug or what could be the cause?

Task logs

2023-02-06T18:09:38.7269951Z ##[section]Starting: Azure functions app deploy
2023-02-06T18:09:38.7411299Z ==============================================================================
2023-02-06T18:09:38.7411725Z Task         : Azure Functions
2023-02-06T18:09:38.7412152Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2023-02-06T18:09:38.7412537Z Version      : 1.216.1
2023-02-06T18:09:38.7412790Z Author       : Microsoft Corporation
2023-02-06T18:09:38.7413114Z Help         : https://aka.ms/azurefunctiontroubleshooting
2023-02-06T18:09:38.7414623Z ==============================================================================
2023-02-06T18:09:39.6641983Z Got service connection details for Azure App Service:'Function-ScanBlobEvents-Prod'
2023-02-06T18:09:41.4890907Z (node:12608) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2023-02-06T18:10:43.7300482Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2023-02-06T18:10:43.7301278Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2023-02-06T18:10:43.7302201Z App Service Application settings are already present.
--> 2023-02-06T18:14:17.6081212Z Validating deployment package for functions app before Zip Deploy <-- This call took >13 minutes to complete, deployment itself only 2.5 minutes
2023-02-06T18:19:35.9223566Z Package deployment using ZIP Deploy initiated.
2023-02-06T18:22:09.9532786Z Deploy logs can be viewed at https://function-scanblobevents-prod.scm.azurewebsites.net/api/deployments/dfcf6fc0659649cf81efa707064b3346/log
2023-02-06T18:22:09.9535573Z The web package has been deployed to App Service. Please note that the package mount or extraction errors will be logged in the deployment logs in the location above.
2023-02-06T18:22:09.9537769Z NOTE: Run From Package makes wwwroot read-only, so you will receive an error when writing files to this directory.
@github-actions github-actions bot added question Area: Release environment:need-to-triage Issues need to be triage by environment-deployment team Task: AzureFunction labels Feb 7, 2023
@Taylor-Daly
Copy link

I have the same issue.

@FinVamp1 FinVamp1 assigned FinVamp1 and unassigned nadesu Mar 17, 2023
@FinVamp1
Copy link
Contributor

Hello, moved this issue to me and I will take a look.

@QwoaterTonkoBoekhoud
Copy link

QwoaterTonkoBoekhoud commented Mar 22, 2023 •
edited
Loading

In addition to this: we run several deploy pipelines that all have the same setup, and they all go to similar function apps. The raw publish log of them is normally this:

2023-03-21T12:46:21.8261947Z ##[section]Starting: Publish Function UpdateFromService.Customers
2023-03-21T12:46:21.8272169Z ==============================================================================
2023-03-21T12:46:21.8272425Z Task         : Azure Functions
2023-03-21T12:46:21.8272577Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2023-03-21T12:46:21.8272848Z Version      : 1.216.1
2023-03-21T12:46:21.8272977Z Author       : Microsoft Corporation
2023-03-21T12:46:21.8273143Z Help         : https://aka.ms/azurefunctiontroubleshooting
2023-03-21T12:46:21.8273334Z ==============================================================================
2023-03-21T12:46:22.4502841Z Got service connection details for Azure App Service:'qwtr-updatecustomers'
2023-03-21T12:46:23.4993748Z (node:1657) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2023-03-21T12:46:23.8723277Z NOTE: Function app is VNet integrated.
2023-03-21T12:46:33.0042232Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2023-03-21T12:46:33.0042750Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2023-03-21T12:46:33.0043511Z App Service Application settings are already present.
2023-03-21T12:46:41.0142651Z Validating deployment package for functions app before Zip Deploy
2023-03-21T12:46:43.3399757Z Package deployment using ZIP Deploy initiated.
2023-03-21T12:47:20.9470056Z Deploy logs can be viewed at https://qwtr-updatecustomers-staging.scm.azurewebsites.net/api/deployments/35cf65354a8048ac9640b86ebc36fd6c/log
2023-03-21T12:47:20.9472638Z The web package has been deployed to App Service. Please note that the package mount or extraction errors will be logged in the deployment logs in the location above.
2023-03-21T12:47:20.9474877Z NOTE: Run From Package makes wwwroot read-only, so you will receive an error when writing files to this directory.
2023-03-21T12:47:22.2026977Z Successfully added release annotation to the Application Insight : qwtr-weu-prd-appInsights
2023-03-21T12:47:22.5619332Z App Service Application URL: https://qwtr-updatecustomers-staging.azurewebsites.net
2023-03-21T12:47:22.5704061Z ##[section]Finishing: Publish Function UpdateFromService.Customers

So deploy takes normally a few minutes in total. Only one (new) pipeline is taking so long that a timeout occurs:

2023-03-22T08:43:44.5518660Z ##[section]Starting: Publish Function UnlockOfficeDocuments
2023-03-22T08:43:44.5530173Z ==============================================================================
2023-03-22T08:43:44.5530441Z Task         : Azure Functions
2023-03-22T08:43:44.5530585Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2023-03-22T08:43:44.5530895Z Version      : 1.216.1
2023-03-22T08:43:44.5531035Z Author       : Microsoft Corporation
2023-03-22T08:43:44.5531248Z Help         : https://aka.ms/azurefunctiontroubleshooting
2023-03-22T08:43:44.5531437Z ==============================================================================
2023-03-22T08:43:45.2210626Z Got service connection details for Azure App Service:'qwtr-unlockofficedocuments'
2023-03-22T08:43:46.5070235Z (node:1660) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2023-03-22T08:43:47.3525716Z NOTE: Function app is VNet integrated.
2023-03-22T08:44:47.7385285Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2023-03-22T08:44:47.7385803Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2023-03-22T08:44:47.7386586Z App Service Application settings are already present.
2023-03-22T08:56:10.4333163Z Validating deployment package for functions app before Zip Deploy
2023-03-22T09:22:47.1057019Z Package deployment using ZIP Deploy initiated.
2023-03-22T09:43:21.1983505Z ##[error]The operation was canceled.
2023-03-22T09:43:21.1989523Z ##[section]Finishing: Publish Function UnlockOfficeDocuments

I cannot find any reason why this pipeline should perform different from the others.
Hope this addition has some info that helps finding a solution.

@QwoaterTonkoBoekhoud
Copy link

Hello, moved this issue to me and I will take a look.

@FinVamp1, could you please give an update? We still have an undeployable function....

@FinVamp1
Copy link
Contributor

Hello there. It looks like when this app was going through validation the call returned a 503 Server error and the site was not able to start. As your Function App is running on a virtual network it appears that intermittently the content share hosting the app content was not available. Our guidance on configuring this is here. https://learn.microsoft.com/en-us/azure/azure-functions/configure-networking-how-to . Can you check the Storage account configuration related to the Virtual Network?

Finbar

@QwoaterTonkoBoekhoud
Copy link

QwoaterTonkoBoekhoud commented Mar 30, 2023 via email

@FinVamp1
Copy link
Contributor

Hi, I will add logging for this.

@QwoaterTonkoBoekhoud
Copy link

QwoaterTonkoBoekhoud commented Apr 19, 2023 via email

@woeterman94
Copy link

Any update on this issue? Facing the same issue with an Azure .NET Function App.

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

@github-actions github-actions bot added the stale label Dec 31, 2023
@github-actions github-actions bot closed this as completed Jan 5, 2024
@FinVamp1 FinVamp1 reopened this Jan 5, 2024
@FinVamp1
Copy link
Contributor

FinVamp1 commented Jan 5, 2024

Reopened as I have plans to help on this.

@github-actions github-actions bot removed the stale label Jan 5, 2024
@FinVamp1 FinVamp1 added FunctionsTaskBacklog and removed environment:need-to-triage Issues need to be triage by environment-deployment team labels Feb 6, 2024
@hybridtechie
Copy link

Following. I am facing similar issue

@Serfenia
Copy link

Serfenia commented Mar 5, 2024

Up, we're facing this issue in our deployments as well, using the AzureFunctionApp@2 version task. Validating takes around 10-15 minutes. It doesn't take always this long, but sometimes it does.

@srikanthreddyk098
Copy link

I'm too facing the similar issue, I have tried the most of the troubleshooting and still facing same any possible solutions here much appreciated

@bpons-mir
Copy link

For my case, as it is behind private endpoint with public access disabled, and as vnet integration is setup, those 2 settings made it work:
"WEBSITE_CONTENTOVERVNET" = "1"
"WEBSITE_VNET_ROUTE_ALL" = "1"

@FinVamp1
Copy link
Contributor

Can people seeing this issue please share their deployment logs? If you don't want to share these here then email me finbarr at microsoft.com? An app name would also be ideally? @bpons-mir Were you deploying to a Functions Elastic Premium plan?

@bpons-mir
Copy link

@FinVamp1 well actually I was deploying to a Logic App but I use AzureFunctionApp@2 task to do so.

@findajay
Copy link

We are facing similar issue on one of our pipeline.

@FinVamp1
Copy link
Contributor

FinVamp1 commented Jan 6, 2025

Triaged, add some guardrails and logging around this call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FinVamp1 FinVamp1

Labels
FunctionsBacklogPri1 FunctionsTaskBacklog question Task: AzureFunction
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests