Feature/adaptive capping

docs/advanced_usage/13_adaptive_capping.md

@@ -0,0 +1,107 @@
+# Adaptive Capping
+
+Adaptive capping is a feature that can be used to speedup the evaluation of candidate configurations when the objective
+is to minimize runtime of an algorithm across a set of instances. The basic idea is to terminate unpromising candidates
+early and adapting the timeout for solving a single instance dynamically based on the incumbent's runtime and the.
+runtime already used by the challenging configuration.
+
+## Theoretical Background
+
+When comparing a challenger configuration with the current incumbent for a (sub-)set of instances, we already know how
+much cost (in terms of runtime) was incurred by the incumbent to solve the set of instances. As soon as the challenger
+configuration exceeds the cost of the incumbent, it is evident that the challenger will not become the new incumbent
+since the costs accumulate over time and are strictly positive, i.e., solving an instance cannot have negative runtime.
+
+Example:
+*Let the incumbent be evaluated for two instances with observed runtimes 3s and 4s. When a challenger configuration is
+evaluated and compared against the incumbent, it is first evaluated on a first instance. For example, we observe a
+runtime of 2s. As the challenger appears to be a promising configuration, its evaluation is intensified and the budget
+is doubled, i.e., the budget is increased to 2. For solving the second instance, adaptive capping will allow a timeout
+of 5s since the sum of runtimes for the incumbent is 7s and the challenger used up 2s for solving the first instance so
+far so that 5s remain until the costs of the incumbent are exceeded. Even if the challenger configuration would need 10s
+to solve the second instance, its execution would be aborted. In this example, by adaptive capping we thus save 5s of
+evaluation costs for the challenger to notice that it will not replace the current incumbent.*
+
+In combination with random online aggressive racing, we can further speedup the evaluation of challenger configurations
+as we increase the horizon for adaptive capping step by step with every step of intensification. Note that
+intensification will double the number of instances to which the challenger configuration (and eventually also the
+incumbent configuration) are applied to. Furthermore, to increase the trust into the current incumbent, the incumbent is
+regularly subject to intensification.
+
+
+## Setting up Adaptive Capping
+
+To achieve this, the user must take active care in the termination of their target function.
+The capped problem.train will receive a budget keyword argument, detailing the seconds allocated to the configuration.
+Below is an example of a capped problem that will return the used budget if the computation exceeds the budget.
+
+
+```python
+
+    class TimeoutException(Exception):
+        pass
+
+
+    @contextmanager
+    def timeout(seconds):
+        def handler(signum, frame):
+            raise TimeoutException(f"Function call exceeded timeout of {seconds} seconds")
+
+        # Set the signal handler for the alarm signal
+        signal.signal(signal.SIGALRM, handler)
+        signal.alarm(seconds)  # Schedule an alarm after the given number of seconds
+
+        try:
+            yield
+        finally:
+            # Cancel the alarm if the block finishes before timeout
+            signal.alarm(0)
+
+
+    class CappedProblem:
+        @property
+        def configspace(self) -> ConfigurationSpace:
+            ...
+
+        def train(self, config: Configuration, instance:str, budget, seed: int = 0) -> float:
+
+            try:
+                with timeout(int(math.ceil(budget))):
+                    start_time = time.time()
+                    ... # heavy computation
+                    runtime = time.time() - start_time
+                    return runtime
+            except TimeoutException as e:
+                print(f"Timeout for configuration {config} with runtime budget {budget}")
+                return budget # here the runtime is capped and we return the used budget.
+```
+
+In order to enable adaptive capping in smac, we need to create [problem instances](4_instances.md) to optimize over and specify a
+global runtime cutoff in the intensifier. Then we optimize as usual.
+
+
+```python
+from smac.intensifier import Intensifier
+from smac.scenario.scenario import Scenario
+
+scenario = Scenario(
+    capped_problem.configspace,
+    ...
+    instances=['1', '2', '3'], # add problem instances we want to solve
+    instance_features={'1': [1], '2': [2], '3': [3]} # in the absence of actual features add dummy features for identification
+)
+
+intensifier = Intensifier(
+scenario,
+runtime_cutoff=10 # specify an absolute runtime cutoff (sum over instances) never to be exceeded
+)
+
+smac = HyperparameterOptimizationFacade(
+    scenario,
+    capped_problem.train,
+    intensifier=intensifier,
+    ...
+)
+
+incumbent = smac.optimize()
+```

examples/1_basics/8_adaptive_capping.py

@@ -0,0 +1,103 @@
+"""Adaptive Capping
+# Flags: doc-Runnable
+
+Adaptive capping is often used in optimization algorithms, particularly in
+scenarios where the time taken to evaluate solutions can vary significantly.
+For more details on adaptive capping, consult the [info page adaptive capping](../../../advanced_usage/13_adaptive_capping.html).
+
+"""
+import math
+import time
+
+import signal
+from contextlib import contextmanager
+
+from smac.runhistory import InstanceSeedBudgetKey, TrialInfo
+
+import warnings
+
+from ConfigSpace import Categorical, Configuration, ConfigurationSpace, Float
+
+
+class TimeoutException(Exception):
+    pass
+
+@contextmanager
+def timeout(seconds):
+    def handler(signum, frame):
+        raise TimeoutException(f"Function call exceeded timeout of {seconds} seconds")
+
+    # Set the signal handler for the alarm signal
+    signal.signal(signal.SIGALRM, handler)
+    signal.alarm(seconds)  # Schedule an alarm after the given number of seconds
+
+    try:
+        yield
+    finally:
+        # Cancel the alarm if the block finishes before timeout
+        signal.alarm(0)
+
+
+class CappedProblem:
+    @property
+    def configspace(self) -> ConfigurationSpace:
+        cs = ConfigurationSpace(seed=0)
+        x0 = Float("x0", (0, 5), default=5, log=False)
+        x1 = Float("x1", (0, 7), default=7, log=False)
+        cs.add_hyperparameters([x0, x1])
+        return cs
+
+    def train(self, config: Configuration, instance:str, budget, seed: int = 0) -> float:
+        x0 = config["x0"]
+        x1 = config["x1"]
+
+        try:
+            with timeout(int(math.ceil(budget))):
+                runtime = 0.5 * x1 + 0.5 * x0 * int(instance)
+                time.sleep(runtime)
+                return runtime
+        except TimeoutException as e:
+            print(f"Timeout for configuration {config} with runtime budget {budget}")
+            return budget  # FIXME: what should be returned here?
+
+
+if __name__ == '__main__':
+    from smac import HyperparameterOptimizationFacade, RunHistory
+    from smac import Scenario
+
+    from smac.intensifier import Intensifier
+
+    capped_problem = CappedProblem()
+
+    scenario = Scenario(
+        capped_problem.configspace,
+        walltime_limit=3600,  # After 200 seconds, we stop the hyperparameter optimization
+        n_trials=500,  # Evaluate max 500 different trials
+        instances=['1', '2', '3'],
+        instance_features={'1': [1], '2': [2], '3': [3]}
+    )
+
+    # We want to run five random configurations before starting the optimization.
+    initial_design = HyperparameterOptimizationFacade.get_initial_design(scenario, n_configs=5)
+
+    intensifier = Intensifier(scenario, runtime_cutoff=10, adaptive_capping_slackfactor=1.2)
+
+    # Create our SMAC object and pass the scenario and the train method
+    smac = HyperparameterOptimizationFacade(
+        scenario,
+        capped_problem.train,
+        initial_design=initial_design,
+        intensifier=intensifier,
+        overwrite=True,
+    )
+
+    # Let's optimize
+    incumbent = smac.optimize()
+
+    # Get cost of default configuration
+    default_cost = smac.validate(capped_problem.configspace.get_default_configuration())
+    print(f"Default cost ({intensifier.__class__.__name__}): {default_cost}")
+
+    # Let's calculate the cost of the incumbent
+    incumbent_cost = smac.validate(incumbent)
+    print(f"Incumbent cost ({intensifier.__class__.__name__}): {incumbent_cost}")

examples/1_basics/8_warmstart.py

@@ -3,7 +3,7 @@
 
 With the ask and tell interface, we can support warmstarting SMAC. We can communicate rich
 information about the previous trials to SMAC using `TrialInfo` and `TrialValue` instances.
-For more details on ask and tell consult the [info page ask-and-tell](../../../advanced_usage/5_ask_and_tell).
+For more details on ask and tell consult the [info page ask-and-tell](../../../advanced_usage/5_ask_and_tell.html).
 """
 from __future__ import annotations
 

mkdocs.yaml

@@ -207,6 +207,7 @@ nav:
     - "advanced_usage/10_continue.md"
     - "advanced_usage/11_reproducibility.md"
     - "advanced_usage/12_optimizations.md"
+    - "advanced_usage/13_adaptive_capping.md"
   # Auto generated with docs/examples_runner.py
   - Examples: "examples/"
   # Auto generated with docs/api_generator.py

smac/facade/abstract_facade.py

@@ -119,6 +119,7 @@ def __init__(
         multi_objective_algorithm: AbstractMultiObjectiveAlgorithm | None = None,
         runhistory_encoder: AbstractRunHistoryEncoder | None = None,
         config_selector: ConfigSelector | None = None,
+        runtime_cutoff: int | None = None,
         logging_level: int | Path | Literal[False] | None = None,
         callbacks: list[Callback] = None,
         overwrite: bool = False,
@@ -175,6 +176,7 @@ def __init__(
         self._runhistory = runhistory
         self._runhistory_encoder = runhistory_encoder
         self._config_selector = config_selector
+        self._runtime_cutoff = runtime_cutoff
         self._callbacks = callbacks
         self._overwrite = overwrite
 
@@ -485,4 +487,7 @@ def _get_signature_arguments(self) -> list[str]:
         if self._intensifier.uses_instances:
             arguments += ["instance"]
 
+        if self._intensifier.uses_cutoffs:
+            arguments += ["cutoff"]
+
         return arguments

smac/facade/algorithm_configuration_facade.py

@@ -99,12 +99,7 @@ def get_acquisition_maximizer(  # type: ignore
         return optimizer
 
     @staticmethod
-    def get_intensifier(
-        scenario: Scenario,
-        *,
-        max_config_calls: int = 2000,
-        max_incumbents: int = 10,
-    ) -> Intensifier:
+    def get_intensifier(scenario: Scenario, *, max_config_calls: int = 2000, max_incumbents: int = 10) -> Intensifier:
         """Returns ``Intensifier`` as intensifier. Supports budgets.
 
         Parameters
@@ -115,11 +110,7 @@ def get_intensifier(
         max_incumbents : int, defaults to 10
             How many incumbents to keep track of in the case of multi-objective.
         """
-        return Intensifier(
-            scenario=scenario,
-            max_config_calls=max_config_calls,
-            max_incumbents=max_incumbents,
-        )
+        return Intensifier(scenario=scenario, max_config_calls=max_config_calls, max_incumbents=max_incumbents)
 
     @staticmethod
     def get_initial_design(  # type: ignore

smac/intensifier/abstract_intensifier.py

@@ -213,14 +213,20 @@ def uses_seeds(self) -> bool:
     @abstractmethod
     def uses_budgets(self) -> bool:
         """If the intensifier needs to make use of budgets."""
-        raise NotImplementedError
+        return False
 
     @property
     @abstractmethod
     def uses_instances(self) -> bool:
         """If the intensifier needs to make use of instances."""
         raise NotImplementedError
 
+    @property
+    @abstractmethod
+    def uses_cutoffs(self) -> bool:
+        """If the intensifier needs to make use of cutoffs."""
+        raise NotImplementedError
+
     @property
     def incumbents_changed(self) -> int:
         """How often the incumbents have changed."""
@@ -530,13 +536,41 @@ def update_incumbents(self, config: Configuration) -> None:
         # 2) Highest budget: We only want to compare the configs if they are evaluated on the highest budget.
         # Here we do actually care about the budgets. Please see the ``get_instance_seed_budget_keys`` method from
         # Successive Halving to get more information.
-        # Noitce: compare=True only takes effect when subclass implemented it. -- e.g. in SH it
-        # will remove the budgets from the keys.
+        # Notice: compare=True only takes effect when subclass implemented it. -- e.g. in SH it will remove the budgets
+        # from the keys.
         config_isb_comparison_keys = self.get_instance_seed_budget_keys(config, compare=True)
         # Find the lowest intersection of instance-seed-budget keys for all incumbents.
         config_incumbent_isb_comparison_keys = self.get_incumbent_instance_seed_budget_keys(compare=True)
 
         # Now we have to check if the new config has been evaluated on the same keys as the incumbents
+        logger.debug(
+            f"Validate whether we have overlap in the keys evaluated for the challenger config "
+            f"{config_isb_comparison_keys} and the incumbent config {config_incumbent_isb_comparison_keys}. If the "
+            f"challenger is dominated, reject id."
+        )
+
+        if not self.uses_budgets and all(
+            [key in config_incumbent_isb_comparison_keys for key in config_isb_comparison_keys]
+        ):
+            logger.debug(
+                "Check on the currently evaluated instances whether the challenger is dominated by the incumbent"
+            )
+
+            # determine challenger costs
+            challenger_costs = self.runhistory.average_cost(config, config_isb_comparison_keys)
+
+            # check the list of incumbents whether any of the incumbents dominates the current challenger
+            for inc in incumbents:
+                # determine incumbent costs
+                inc_costs = self.runhistory.average_cost(inc, config_isb_comparison_keys)
+                # check dominance
+                is_dominated = not np.any(np.array([challenger_costs]) < np.array([inc_costs]))
+
+                # if challenger config is dominated by the incumbent, reject it
+                if is_dominated:
+                    logger.debug(f"Challenger config {config_hash} is dominated by incumbent {get_config_hash(inc)}.")
+                    self._add_rejected_config(config_id)
+
         if not all([key in config_isb_comparison_keys for key in config_incumbent_isb_comparison_keys]):
             # We can not tell if the new config is better/worse than the incumbents because it has not been
             # evaluated on the necessary trials