Out of memorry errors in CICD pipelines freqent for large third party dependencies

[SteveKrisjanovsD365]

AL-Go version

6.3

Describe the issue

we're currently building an appsource app that exends appsource app LS Central (Publisher: LS Retail).

50% of the time CICD fails at the build step due to out of memory errors (LS for BC takes a large resource footprint). Currently we re-run the failed job(s) and cross our fingers.

Short of exploring self-hosted runners, is there anything that can be done to improve CICD reliability?

Expected behavior

CICD succeeds without any out of memory errors

Steps to reproduce

create an app that depends on LS Central

Additional context (logs, screenshots, etc.)

No response

[freddydk]

Could you include the log, so that we can determine whether you are using containers or compilerfolders?
Also the log will tell us how much memory is used.
If you are using containers, you could try to add a setting called "MemoryLimit": "16G" - or like - but without the log, it is hard to determine whether this will solve anything.

[SteveKrisjanovsD365]

alas I'm re-running the failed job(s) so I don't have a failed log to attach. if it fails again I'll attach the log.,

We are definately using containers however and NOT compilerfolders

wer'e aware that LS Central for BC has a huge memory footprint (both onprem and saas) which has presented its own chalenges in the past when too many onprem instances on a server have LS installed and onprem BC shell experiences frequent out of memory errors if there are too many instances with LS installed.,

[freddydk]

In github, you can select the prior re-run in the top right corner and see the log from the failing builds.

[SteveKrisjanovsD365]

duly noted!

[SteveKrisjanovsD365]

@freddydk

while I wait for the CICD to finish with the memorylimit set to 16G, I found an older log that has the out of memory exception. see attached. in the log zip it's the "CA" project that failed @ build step with out of memory

logs_35243457639.zip

[SteveKrisjanovsD365]

after setting memoryLimit to "16G" didn't help. the pipline still randomly throws out-of-memory errors during the build step, requiring re-running the failed workflow step 3-4x before succeeding.

below is a copy/paste of the log when it fails installing a large third party app

  Copy C:\ProgramData\BcContainerHelper\Extensions\bcUS14471464938\c2d0981a-73c8-4141-af2a-91789be4e109\LS Retail_LS Central_25.0.2.1232.app to D:\a\Cannabis-Retail-App\Cannabis-Retail-App\US\.buildartifacts\Dependencies
  Publishing C:\ProgramData\BcContainerHelper\Extensions\bcUS14471464938\c2d0981a-73c8-4141-af2a-91789be4e109\LS Retail_LS Central_25.0.2.1232.app
  Exception of type 'System.OutOfMemoryException' was thrown.
  
  Exception Script Stack Trace:
  at <ScriptBlock>, <No file>: line 55
  
  PowerShell Call Stack:
  at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 71
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 394
  at Publish-BcContainerApp, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 154
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1176
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1451
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1377
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1269
  at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1254
  at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.3\RunPipeline\RunPipeline.ps1: line 397
  at <ScriptBlock>, D:\a\_temp\8440e36b-6875-49bc-86db-13f2ba994c65.ps1: line 3
  at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.3\Invoke-AlGoAction.ps1: line 17
  at <ScriptBlock>, D:\a\_temp\8440e36b-6875-49bc-86db-13f2ba994c65.ps1: line 2
  at <ScriptBlock>, <No file>: line 1
  
  Container Free Physical Memory: 0.2Gb
  Disk C: Free 121Gb from 127Gb

and further down the log I see this:

Error: Unexpected error when running action. Error Message: Exception of type 'System.OutOfMemoryException' was thrown., StackTrace: at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 113 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 394 <- at Publish-BcContainerApp, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 154 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1176 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1451 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1377 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1269 <- at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.1.3\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1254 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.3\RunPipeline\RunPipeline.ps1: line 397 <- at <ScriptBlock>, D:\a\_temp\8440e36b-6875-49bc-86db-13f2ba994c65.ps1: line 3 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.3\Invoke-AlGoAction.ps1: line 17 <- at <ScriptBlock>, D:\a\_temp\8440e36b-6875-49bc-86db-13f2ba994c65.ps1: line 2 <- at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

@freddydk I asked copilot about the memorylimit=16G setting and it suggested to reduce the value to 14G due windows-based github-runner VM overhead (not that II think it'll help in this case as evidenced by the Container Free Physical Memory: 0.2Gb message in the error log

it would appear theres only two workarounds to this error:

• setting al-go settings "useCompilerFolder": true, "doNotPublishApps": true, unfortunately there are drawbacks to this workaround (cannot validate schema changes, cannot run test libraries, etc).
• using a self-hosted github runner to get us past the 16G github runner limit

thoughts??

[freddydk]

Yeah - the self-hosted runners might be your only option here.
GitHub also have larger runners - you can use those as GitHubRunner and keep runs-on as ubuntu-latest or like, so that only the build jobs will utilize larger runners.

[SteveKrisjanovsD365]

@freddydk thanks for the confirmation. I'm pivoting to self-hosted runners this week to prove it out (not all our runners need to be self-hosted but the ones that depend on LS Retail do)

[HazardMK]

@SteveKrisjanovsD365 did you managed to publish LS Central app via self-hosted runner? I'm facing the same issue on github-runner now, but getting timeout. Sometimes no error, but the CI/CD runs for hours, i need to stop it to not eat up the minutes. Btw LS app file size is ~30mb.

[SteveKrisjanovsD365]

@HazardMK we stopped using self-hosted runners. We now use github-hosted "large" runners for any apps that depend on LS.

[HazardMK]

could you explain a bit about these "large" runners? is there a documentation on it?

[SteveKrisjanovsD365]

@HazardMK I couldnt find documentation but I pieced together earlier this year how to set it up from my self-hosted runner experience

log into github >> org >> settings >> runners. here is where you'll create a new github-hosted runner (mind you, larger runners with more RAM cost more than the defalt runner). Note the runner label for the runner you created. In my case the label for my large runner is "windows-latest-l"

then in my AL-go settings, I added this:

"githubRunner": "windows-latest-l",
  "defaultNewContainerParameters": {
        "isolation": "hyperv"
  },
  "memoryLimit": "16G"

[HazardMK]

thank you for the info! i tried to create a new larger github hosted runner, but it gets "shutdown" state after a few minutes. probably enterprise does not allow it or they need to create separate billing account for larger runners. isn't it pay-as-you-go plan?