ELG redshift failures due to bad sky model

[Jiaxi-Yu]

desispec_ELG_failure.txt
The attached figure is a typical ELG spectrum that was identified as a redshift catastrophic failure. The [OII] vertical dashed lines are at the correct position determined by visual inspection. The black line is the Redrock redshift, which misidentified the sky residual as [OII] doublets. This comprises 20% of the total catastrophic failure (<0.3% for EDR and Y1).
As the CTE problem has been solved (Stephen at the Hawaii meeting in 2024), shall we expect a suppressed sky-residual ELG redshift failure rate?
The attached ASCII file is a list of ELG catastrophic failures identified in daily samples (by comparing with SV3 redshifts) for your reference. Both use iron redrock.

[sbailey]

@Jiaxi-Yu thanks. Could you also provide the table with incorrect redshifts and correct redshifts for these targets? It's unclear to me whether the CTE problems/fixes explain these cases, but this is a useful set for testing.

@dylanagreen @moustakas @abrodze @abhi0395 heads up for the "is Redrock correct" group.

[Jiaxi-Yu]

@sbailey Here is the target list with true redshift and the catastrophical failure redshift. The first 5 elements are samples that probably don't have reliable redshift measurements as their [OII] overlap with the sky model.
desispec_ELG_skyresidual_ZTRUE.txt

[araichoor]

thanks @Jiaxi-Yu for sharing infos.

for what is worth, I ve spot-checked few cases: my understanding is that, for those cases, there always are 1-2 pixels around 8660A which have an insanely large snr, like >15.

i.e. another way to phrase the issue is that the flux error is severely underestimated for those pixels.
though I don t know if there is a way to improve that in the processing, as those pixels around sky emission lines are very hard to handle.

here s one example for TARGETID=39627444868617702 (same target as in the prospect screenshot in the initial message):

[moustakas]

I only checked this first object, but the good news is that the new (non-negative, physical) galaxy templates get the right redshift (in fact, the top three minima are at the right redshift, just with different galaxy subtypes and comparable chi2):

[moustakas]

@Jiaxi-Yu I'm going to close this ticket but I've ported it over to desihub/rrevaluator#4, where we're consolidating all the various "is Redrock right or wrong" tickets which are currently spread across many different repositories.

We should have a more substantive update for you regarding this issue soon.