JSON validation

[shubhamugare]

Hi,

I see that you are using a JSON validator with an exact match with the completion from the dataset. How would that make sense in case the completion is a string as in the following example.

{"reservationID": "AH-158394", "guestName": "Alexander Hamilton", "reservationTime": "2023-04-15T19:30:00Z", "specialRequests": ["Table by the window", "Surprise dessert for a special occasion"]}

This would match only if the output string is exactly "Table by the window" and "Surprise dessert for a special occasion", right?

[interstellarninja]

Hi that's a valid point, for strings we would probably need a fuzzy matching logic.

However for our eval set if there's a string value as a function argument, we make sure the query contains the exact strings in double quotes.

The ideal case for eval samples is to have unambiguous deterministic argument values instead of unconstrained string values.