FEA ridge benchmarks

[fcharras]

• run scikit-learn vanilla benchmark for svd
• have consolidation script work
• push vanilla scikit-learn to the worksheet
• run scikit-learn vanilla benchmark for other solvers
• add cuml benchmark
• add scikit-learn-intelex benchmark
• add scikit-learn + array-api benchmark

Targeted devices will be 32 cores cpu and cuda gpu. Maybe intel max series too if access is not too complicated.

Probably a follow-up PR:

• factorize consolidation script (with this third example I have some clearer ideas)

[fcharras]

I'm not quite confident I wrote the objective evaluation well. Is that correct ?

        if self.sample_weight_ is not None:
            X, y, _ = _rescale_data(X, y, self.sample_weight_, inplace=False)

        y = y.reshape((y.shape[0], -1))
        weights = weights.reshape((-1, X.shape[1], 1))

        value = (
            (((X @ weights).squeeze(2) + (intercept - y).T) ** 2).sum()
            + (self.alpha * (weights**2).sum())
        ) / (X.shape[0] * len(y.T))

or is it better to just report r2_score ?

[ogrisel]

Some feed back.

[ogrisel]

or is it better to just report r2_score ?

r2_score is not the training objective (in particular it does not reflect the bias induced by the regularizer) so it's harder to interpret as a metric to check that models converge to comparable solutions. Unfortunately, scikit-learn does not yet expose the value of its internal training objective, nor a method to compute it from a fitted model.

What you wrote seems sensible at first glance but would require detailed code inspection to check that this is actually what is optimized in scikit-learn internally. Note that I started work to improve the convergence tests for Ridge models scikit-learn/scikit-learn#25948 but I could not yet find time to get back to it.

[fcharras]

So I added some solvers other than svd. Not sure it's appropriate though, since the initial purpose is rather comparing backends of same solvers. If we want to have different solvers, then maybe let's consider having one different folder and sheet per solver. I think it's still interesting to look at but more difficult to setup and read (because of max_iter and tol parameters mostly)

[ogrisel]

I browsed the results a bit and it's quite interesting.

Based on the results, here are some suggestions to the benchmark setting to make this even more interesting (in my opinion).

[ogrisel]

max_iter below 100 seems small to me. Based on the results it happens quite to often that the maximum is reached for lsqr, possibly with a very bad value of the objective.

[fcharras]

I reverted to None and set tol to 1e-4 (which are the defaults parameters), so iteration-based solvers might have different number of iterations now. For multi targets, lsqr report a different number of iterations for each target, but it's unconvenient to report all those... I settled for report of the maximum over the targets, after observing that it doesn't seem to differ much from one target to another.

[fcharras]

I have updated following your suggestions +sync'd the spreadsheet, but for some reason I can't run the cuml solver anymore:

RuntimeError: cuSOLVER error encountered at: file=/[...]/ridge_benchmark/include/raft/linalg/detail/svd.cuh line=78: 

edit: so apparently cuml svd solver uses a lot of vram and this is a vram issue. I add some more dataset dimensions that show what cuml supports or not. (no issue with torch+cuda svd on the other hand)

edit2: does not explain all occurences of the error

edit3: cuml svd solver does not accept n_features > n_samples apparently, and use up a lot of memory, will crash on biggest datasets.

[fcharras]

Somes fixes done, and all results pushed and synchronized to the sheet. Let's merge ?