Add Everest storage (port of seba_sqlite logic)

[yngve-sk]

Release notes:

• Stores constraint violation values
• Replaces seba sqlite storage

(Storage PR back in, for reviews, previous PR (accidentally merged): #9161)

Issue
Resolves #8811

Base idea/documentation:

Store datasets by [batch, realization, perturbation] x [controls, objectives, constraints, objective_gradient, constraint_gradient]:

Exhaustive list of data stored PER BATCH :

• batch.json - contains info about the batch, batch_id and whether it is an improvement (aka merit flag, but the concepts are now unified for dakota and non-dakota runs)
• batch_constraints constraint values (and violations) for constraints, batch-wide
• batch_objectives objective values, batch-wide
• realization_controls - control values for geo-realizations, also includes simulation_id
• realization_objectives - objective values per geo-realization
• realization_constraints - constraint values per geo-realization
• perturbation_objectives - objective and control values per perturbation
• perturbation_constraints - constraint and control values per perturbation (Note/discussion point: control values could be pulled into separate table to avoid redundancy)
• batch_objective_gradient - Partial derivatives of objectives, given different controls. This dataset has one column per objective, and one row per control value, and the intersecting cells represent the partial derivative of the objective wrt that control value.
• batch_constraint_gradient - Partial derivatives of constraints, given different controls. This dataset has one column per constraint, and one row per control value, and the intersecting cells represent the partial derivative of the constraint wrt that control value.

Example data from math_func/config_advanced.yml (json format)

Exhaustive list of data stored PER OPTIMIZATION

• controls.json - control values for this batch
• realization_weights.json - realization weights
• nonlinear_constraints - conditions for constraints to satisfy (on average over the batch)
• objective_functions - objective function names, weights, and normalization

Example data from math_func/config_advanced.yml

Potential simplifications

The everest_data_api is currently used for plotting, but could be used (probably expanded a bit) to avoid doing direct (polars) dataframe manipulations elsewhere in the code, but currently they are done directly in the code.

[codspeed-hq]

CodSpeed Performance Report

Merging #9763 will improve performances by 10.95%

_{Comparing yngve-sk:24.10.25.store-everest-opt-results-in-ertstorage (f45879a) with main (8bdb398)}

Summary

⚡ 1 improvements
✅ 23 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	yngve-sk:24.10.25.store-everest-opt-results-in-ertstorage	Change
⚡	test_direct_dark_performance_with_storage[gen_x: 20, sum_x: 20 reals: 10-gen_data_with_obs-get_record_observations]	2.5 ms	2.3 ms	+10.95%

[DanSava]

This looks good 👍 🏅

[StephanDeHoop]

Looks great! Most of my comments are just asking to clarify things I don't understand :). Amazing work, well done :) !!!