Dynamically assert model temperature value in argparser

[dberardi99]

A new method get_model_temperature(args: argparse.Namespace) has been added to hardcode the model temperature value based on the model name. The chat session models (namely the ones ending with "-chat" or "-azure") have been treated as the corresponding base models. The temperature values can be found in the following data table #366 (comment)

The temperature value will be automatically aligned to the one of the model chosen only if no temperature has been set (args.temperature == TEMPERATURE). Instead, if a wrong model name is fed (args.model in models.LLM.all_llm_names() == 0), the above method is skipped and the temperature is left unchanged to its default value.

In addition, the default temperature has been changed to 1.0 since it is the one relative to the default model (vertex_ai_gemini-1-5)

Fix #366

[dberardi99]

Hi @DonggeLiu Could you please review it when you get a chance? Many thanks in advance!

[DonggeLiu]

Thanks @dberardi99.
I left a comment to clarify a bit more about the task.
Please let me know if that makes sense : )

[DonggeLiu]

Ops let's keep the default temperature the same for now to avoid causing surprising results in other people's recent experiments.
We can grid search for the best default values for our use case later.

[dberardi99]

Perfect, I got it. I'll reset it to its prior value

[DonggeLiu]

I reckon the issue we solve has 2 tasks:

1. Main: Different models have different temperature ranges, we want to assert that if user specified a temperature in args, then it should fall into the corresponding models range.
2. Minor: Define a default temperature for each model.

let's solve the main task first:
1.1. Define the temperature rate for each model class in https://github.com/google/oss-fuzz-gen/blob/main/llm_toolkit/models.py. Use inheritance to minimize the changes needed.
1.2. Replace this hardcoded assertion with dynamic assertion based on the model name:

oss-fuzz-gen/run_all_experiments.py

Lines 280 to 281 in 33bddff

	if args.temperature:
	assert 2 >= args.temperature >= 0, '--temperature must be within 0 and 2.'

Then we can work on the minor task:

1. Add default temperatures under each class.
2. Set the temperature as the default value here, if user did not specify it.

[dberardi99]

Everything's clear! Just one thing, do you prefer to solve only the different temperature ranges in this PR or to implement both temperature ranges and default temperatures here?

[DonggeLiu]

Thanks @dberardi99, some nits.

[DonggeLiu]

Let's keep this attribute but make the value a constant like TEMPERATURE defined at the top of the file:

temperature_range: list[float] = TEMPERATURE_RANGE

[DonggeLiu]

IIUC, you are replicating all_llm_names.
Would it be more maintainable and extensible to extract and reuse the repeating logic of these two functions? E.g., a function to return all models so that we can reuse it to acquire other attributes in the future.

    @classmethod
    def _all_llm_models(cls):
        """
        Returns a list of LLM model classes that have a `name` attribute
        and are not `AIBinaryModel`.
        """
        models = []
        for subcls in cls.all_llm_subclasses():
            # May need a different filter logic here.
            if subcls.name != AIBinaryModel.name:
                models.append(subcls)
        return models

    @classmethod
    def all_llm_names(cls) -> list[str]:
        """Returns the current model name and all child model names."""
        return [m.name for m in cls._all_llm_models()]

    @classmethod
    def all_llm_temperature_ranges(cls) -> dict[str, list[float, float]]:
        """Returns the current model name and all child model temperature ranges."""
        return {
            m.name: m.temperature_range
            for m in cls._all_llm_models()
            if hasattr(m, 'temperature_range')
        }

[DonggeLiu]

Feel free to adjust/simplify these functions as you see fit, particular the filtering logic.
These above are just examples.

[dberardi99]

Good idea! I've slightly modified it by introducing the all_llm_search method, which allows to output the desired attribute for all LLMs. If you don't like the idea, I can switch back to your implementation.

[DonggeLiu]

Add ... for model args.model specify the model name we parsed.

[dberardi99]

@DonggeLiu I'm so sorry for the great delay... If you like this final implementation I can open a new PR creating a new branch for this fix in order to leave the modification history clearer.

[DonggeLiu]

Thanks @dberardi99.
This is already pretty good, just a final question regarding reliability.

I don't think you have to create a new PR, as the modification history is pretty clear. But please let me know if you prefer otherwise.

Also, as your code is almost ready to merge, I will create a new base (#986) for your PR so that we can experiment with it before merging into main.

[DonggeLiu]

tr and names may mismatch.
This code assumes that all_llm_subclasses will always return the subclasses in the same order, but I don't think Python's __subclasses__ can guarantee that.

[dberardi99]

I've had the same your doubt regarding the order in which Python returns subclasses. I've done lots of tests and the order was always the same, but I can make additional researches and in case we are unsure about it propose a small fix to increase reliability

[DonggeLiu]

How about using dictionaries like we discussed earlier?
#856 (comment)

[dberardi99]

I've slightly adapted the filtering function to return also all the models names, so that now we guarantee the correct assignment between model name and temperature range

[dberardi99]

Thanks @dberardi99. This is already pretty good, just a final question regarding reliability.

I don't think you have to create a new PR, as the modification history is pretty clear. But please let me know if you prefer otherwise.

Also, as your code is almost ready to merge, I will create a new base (#986) for your PR so that we can experiment with it before merging into main.

Awesome! We can continue working in this PR with no problem in this case