Skip to content
This repository was archived by the owner on Oct 28, 2024. It is now read-only.

Conversation

@v1v
Copy link
Member

@v1v v1v commented Feb 5, 2020

What does this PR do?

Try/Catch errors and retry with a sleep.

Why is it important?

It will speed up the CI builds as the sleep doesn't happen firstly but only if there are failures.
The sleep was in place to ensure DDOS attack is not detected when running a bunch of parallel stages to interact with our systems, such as docker.registry and vault.

Related issues

This could be implemented with the proposal in jenkinsci/workflow-basic-steps-plugin#97

How to test this

$ hub pr checkout 376
$ cd local
$ make build
$ make start
$ cd workers/linux
$ vagrant up --provision
$ open http://localhost:18080/job/it/job/gitCheckout/
$ open http://localhost:18080/job/it/job/withTotpVault/
$ open http://localhost:18080/job/it/job/gitBaseCommit/job/master/
## build the above pipelines a few times (the first time might take a bit longer as it does require to syncup the shared library)

Tests

  • Vault access
    From 12 seconds average to 2 seconds

image

  • Checkout in pipelines

From 30 seconds average to 18 seconds

image

  • Checkout in multibranch pipelines

From 1min and 40 seconds average to 24 seconds

image

@v1v v1v self-assigned this Feb 5, 2020
@v1v v1v added the automation label Feb 5, 2020
@botelastic botelastic bot added the groovy label Feb 5, 2020
@v1v v1v added the enhancement New feature or request label Feb 5, 2020
@v1v v1v changed the title feature: catch errors and retry with a sleep Speed up steps with less sleeps steps Feb 5, 2020
sleep randomNumber(min: 5, max: 10)
def token = getVaultToken(env.VAULT_ADDR, env.VAULT_ROLE_ID, env.VAULT_SECRET_ID)
props = getVaultSecretObject(env.VAULT_ADDR, secret, token)
retry(3) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

increased the number of retries a shorter sleep

vars/git.groovy Outdated
try {
body()
} catch(e) {
sleep(20)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sleep only if it fails, then let's honor those steps that don't have a timeout issue

def token = getVaultSecret.getVaultToken(env.VAULT_ADDR, env.VAULT_ROLE_ID, env.VAULT_SECRET_ID)
dir(path) {
writeFile file: tokenFile, text: token
}
try {
body()
} catch (err) {
error "withVaultToken: error ${err}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrequired error, throw err is the one in charge.

@v1v v1v marked this pull request as ready for review February 5, 2020 10:29
@ghost
Copy link

ghost commented Feb 5, 2020

💚 Build Succeeded

vars/git.groovy Outdated
@@ -36,7 +36,11 @@ def call(params) {
def retryWithSleep(int i, body) {
retry(i) {
log(level: 'DEBUG', text: "Let's git (${i} tries).")
sleep(20)
body()
try {
Copy link
Contributor

@kuisathaverat kuisathaverat Feb 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see a new step in the horizon 😄

sleepOnError(min: 0, max:30){
  echo "I am a command"
}
def call(Map params = [:], Closure body) {
  def min = params.containsKey('min') ? params.min : 0
  def max = params.containsKey('max') ? params.max : 30
  try{
    body()
  } catch (e){
    sleep randomNumber(min: min, max: max)
    throw e
  }
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, I actually was thinking to provide that step as a built-in step in the declarative pipeline:

retry(times: 3, sleep: 5)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jenkinsci/workflow-basic-steps-plugin#97 is the one for that particular requirement :)

@ghost
Copy link

ghost commented Feb 5, 2020

💚 Build Succeeded

@v1v v1v changed the title Speed up steps with less sleeps steps Speed up builds with less sleeps steps Feb 5, 2020
@v1v v1v merged commit 80e05a4 into elastic:master Feb 5, 2020
@v1v v1v deleted the feature/sleep_refactor branch February 5, 2020 11:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants