Change timeout for long-term experiments

[andreyvelich]

/kind feature

Users can run long-term Experiment where Suggestion needs to analyse information from more than 200-300 Trials.
In that situation, some Suggestion needs more time than 1 minute to generate new Trial set.

We specify the timeout of GetSuggestion request here: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/suggestion/suggestionclient/suggestionclient.go#L26.
We can provide functionality to change this information from katib-config if it necessary.

Ref: #1052.

What do you think @gaocegege @hougangliu @johnugeorge @richardsliu ?

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.95

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen

[robertzsun-dev]

I'm curious why no one has brought up this issue? The hardcoded 60 second timeout is problematic for me as there is no way to use Bayesian Optimization properly because at around 100 trials, the suggestion takes longer than 60 seconds to compute for me.

[tenzen-y]

Let's move the discussion forward.

We can provide functionality to change this information from katib-config if it necessary.

I agree with @andreyvelich.
Please let me know what others think.

cc: @andreyvelich @johnugeorge @gaocegege

[johnugeorge]

Yes. Lets target to add this in the katib-config

[LuukvandenBent]

Any update on this?

[andreyvelich]

Hi @LuukvandenBent, not yet, but contributions are always welcome!

/remove-lifecycle frozen
/good-first-issue

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

Hi @LuukvandenBent, not yet, but contributions are always welcome!

/remove-lifecycle frozen
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[LuukvandenBent]

The problem is happening here right?: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L1231
The function uses default value of constants.DEFAULT_TIMEOUT which is set at 120s. Could you give me some pointers on how/where we would want to fix this?

[andreyvelich]

That's right.
We need to find a way to dynamically inject this value using Katib Config: https://github.com/kubeflow/katib/blob/master/pkg/apis/config/v1beta1/types.go#L101
We can inject this timeout from the config before calling GetSuggestion() API.