Skip to content

[Bug] Configs with different types (numpy vs native) are counted double #1241

@benjamc

Description

@benjamc

Steps/Code to Reproduce

Installation: pip install optuna optunahub rich

import optuna
import optunahub
from rich import inspect


def main():
    smac = optunahub.load_module("samplers/smac_sampler")
    bench = optunahub.load_module("benchmarks/nasbench201")

    # CIFAR 10
    dataset_id = 0
    seeds = 10
    n_trials = 100

    problem = bench.Problem(dataset_id=dataset_id)

    inspect(problem.search_space)

    for seed in range(3, seeds):
        print(seed)
        sampler = smac.SMACSampler(
            search_space=problem.search_space,
            n_trials=n_trials,
            seed=seed,
        )

        study = optuna.create_study(
            sampler=sampler,
            directions=problem.directions
        )

        study.optimize(problem, n_trials=n_trials)


if __name__ == "__main__":
    main()

Expected Results

No crash.

Actual Results

Config selector cannot find new configurations.

Reason

Configs are added to the runhistory twice because the runhistory sees the configs with different types but same values as distinct.

Fix

Convert all config values to built-in python types via json serialization.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

Status

No status

Relationships

None yet

Development

No branches or pull requests

Issue actions