Skip to content

Commit 6707838

Browse files
GeorgWaGeorgWaclaude
authored
[3/5] Refactor: extract the charged fragment type annotation and the pointer reannotation (#436)
* Characterize flatten_fragments before the memory refactor flatten_fragments had no unit test. Add black-box tests that pin its contract, so the refactoring that follows can be checked step by step. `_dense_library` builds dense frames with mz == 0 padding from both sources that produce it: unmodified precursors carry no modloss fragments, and some fragments fall outside the mz range. Intensities are distinct, so the top-k selection is unambiguous. `_expected_keep_mask` derives the expected mask per precursor, without flatten_fragments. The tests cover the kept fragments over five combinations of `keep_top_k_fragments` and `min_fragment_intensity`, every annotation column, the reannotated precursor pointers, `custom_df` and `custom_columns`, the path without intensities, an empty library, and a precursor long enough to need more than a uint8 fragment number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the fragment parsing helpers private `parse_fragment`, `fill_in_indices` and `calculate_fragment_numbers` only serve `flatten_fragments`, and nothing in alphabase, alphadia or peptdeep calls them. A leading underscore states that, and frees their signatures for the refactoring that follows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Extract the charged fragment type annotation and the pointer reannotation Move two concerns out of the body of `flatten_fragments`: the per-column annotation of the charged fragment types, and the reannotation of the precursor pointers. The function now reads as its steps, and the memory refactoring that follows changes a named helper instead of a long function body. `_annotate_charged_frag_types` returns typed arrays rather than four parallel lists of Python ints. Tiling a typed array keeps its dtype, so the dense type/loss_type/charge/direction arrays no longer pass through an int64 intermediate of 8 bytes per dense slot. No functional change. Output stays bit-identical to main across 5400 generated configurations of library shape, padding, top-k, intensity threshold and custom_columns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Pin flatten_fragments against a hand-written library _expected_keep_mask reimplemented the filtering logic, so it could agree with a broken flatten_fragments. Add a minimal_library fixture of two precursors over a forward and a reverse fragment type, with the kept slots written out per filter setting, and assert both the helper and flatten_fragments against it. Derive the long precursor row count from the uint8 boundary it tests instead of picking a number near the fill_in_indices limit, and guard the limit explicitly so exceeding it fails on an assertion rather than on a broadcast error inside numba. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Document the fragment numbering convention "fragment number" read as ambiguous between a count and an enumeration. Spell out that it is the MS series numbering, and contrast it with the fragment position that all series share. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Assert against a hand-written library The library was generated from a seeded rng and the expectations came from _expected_keep_mask, which reimplemented the filtering logic and so could agree with a broken flatten_fragments. Replace both with one library fixture over hand-written mz and intensity grids, and hand-written keep masks per filter setting. Two fragment types carry the whole annotation surface: y_z1 is reverse without loss and b_modloss_z1 is forward with loss, so direction, series and loss all vary over two columns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Assert the flat fragment frames with assert_frame_equal Comparing column by column left the column set, the column order and the dtypes unchecked. Assert the whole annotated frame against an expected frame instead, and fold the row count of the filter cases into a frame comparison of mz and intensity. type and charge come out as int8, so the expected frame spells that out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fold the repeated annotation tiling into a loop The three tile calls differed only in column name and source array, and the comment above them explained the dtype of the previous implementation rather than the code it sat on. Loop over the columns instead; the dtypes stay visible where they are set, in _annotate_charged_frag_types. Trim the two comments carried into the extracted helpers. The direction comment claimed a third value of 0 that DIRECTION_MAPPING never produces. Column order of the flat fragment dataframe is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Point the test at the private helper name The merged tests named fill_in_indices, which this branch renamed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Assert every flat frame with assert_frame_equal Replace the remaining column comparisons. Each one now also pins the column set, the column order and the dtypes: - without_intensity subsumes its separate missing-column assertion - selects_custom_columns checked only the column names - filters_custom_df_columns now pins where cardinality lands - reannotates_precursor_pointers compares hardcoded flat pointers instead of reslicing mz per precursor, which was the last place the tests recomputed an expectation - empty_precursor_df compares both frames instead of their lengths Five tests assert against the same unfiltered library, so it moves into a flat_unfiltered fixture. test_flatten_fragments_long_precursor keeps its property assertions. It builds 257 rows to push fragment numbers past 255, so a frame of 1542 generated values would bury what it tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Compare the long precursor maxima and dtypes as series The four scalar assertions become two equality comparisons. The dtypes need their own comparison, because .max() over columns of different dtypes upcasts to a common one and would hide a narrowed column. Also drop the tuple rebuild in the parse test, expected is already the tuple being compared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Merge refactor/private-fragment-parsing-helpers into refactor/annotate-charged-frag-types Brings main in behind it. The merge was textually clean but left the module unimportable: _annotate_charged_frag_types is annotated -> Tuple[...], and main dropped Python 3.8 and with it the typing import. Changed to the builtin tuple, matching the rest of the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: GeorgWa <georg@aplusia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 52e5821 commit 6707838

1 file changed

Lines changed: 83 additions & 44 deletions

File tree

‎alphabase/peptide/fragment.py‎

Lines changed: 83 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -980,6 +980,75 @@ def _parse_fragment(
980980
return frag_numbers, indices, excluded_indices
981981

982982

983+
def _annotate_charged_frag_types(
984+
charged_frag_types: np.ndarray,
985+
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
986+
"""Give the series id, loss id, charge and direction of every charged fragment type.
987+
988+
Parameters
989+
----------
990+
charged_frag_types : np.ndarray
991+
names of the charged fragment types, in the order of the dense columns
992+
993+
Returns
994+
-------
995+
Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]
996+
series ids, loss ids, charges and directions, one value per charged fragment type
997+
998+
"""
999+
series_ids = []
1000+
loss_ids = []
1001+
charges = []
1002+
directions = [] # a/b/c series count forward (1), x/y/z series backward (-1)
1003+
1004+
for charged_frag_type in charged_frag_types:
1005+
frag_type, charge = parse_charged_frag_type(charged_frag_type)
1006+
series_ids.append(FRAGMENT_TYPES[frag_type].series_id)
1007+
loss_ids.append(FRAGMENT_TYPES[frag_type].loss_id)
1008+
charges.append(charge)
1009+
directions.append(FRAGMENT_TYPES[frag_type].direction_id)
1010+
1011+
return (
1012+
np.array(series_ids, dtype=np.int8),
1013+
np.array(loss_ids, dtype=np.int16),
1014+
np.array(charges, dtype=np.int8),
1015+
np.array(directions, dtype=np.int8),
1016+
)
1017+
1018+
1019+
def _reannotate_precursor_pointers(
1020+
precursor_df: pd.DataFrame, excluded: np.ndarray, n_fragment_types: int
1021+
) -> None:
1022+
"""Add the flat fragment pointers to `precursor_df`, in place.
1023+
1024+
Parameters
1025+
----------
1026+
precursor_df : pd.DataFrame
1027+
precursor dataframe with the `frag_start_idx` and `frag_stop_idx` columns
1028+
1029+
excluded : np.ndarray
1030+
whether each dense slot is excluded from the flat fragment dataframe
1031+
1032+
n_fragment_types : int
1033+
number of charged fragment types, that is the number of dense columns
1034+
1035+
"""
1036+
precursor_df["flat_frag_start_idx"] = precursor_df.frag_start_idx
1037+
precursor_df["flat_frag_stop_idx"] = precursor_df.frag_stop_idx
1038+
precursor_df[["flat_frag_start_idx", "flat_frag_stop_idx"]] *= n_fragment_types
1039+
1040+
# exclusive prefix sum, so cum_sum_tresh[i] counts the slots removed before slot i
1041+
cum_sum_tresh = np.zeros(shape=len(excluded) + 1, dtype=np.int64)
1042+
cum_sum_tresh[1:] = np.cumsum(excluded)
1043+
1044+
precursor_df["flat_frag_start_idx"] -= cum_sum_tresh[
1045+
precursor_df.flat_frag_start_idx.values
1046+
]
1047+
precursor_df["flat_frag_stop_idx"] -= cum_sum_tresh[
1048+
precursor_df.flat_frag_stop_idx.values
1049+
]
1050+
1051+
9831052
def flatten_fragments(
9841053
precursor_df: pd.DataFrame,
9851054
fragment_mz_df: pd.DataFrame,
@@ -1060,37 +1129,23 @@ def flatten_fragments(
10601129
for col_name, df in custom_df.items():
10611130
frag_df[col_name] = df.values.reshape(-1)
10621131

1063-
frag_types = []
1064-
frag_loss_types = []
1065-
frag_charges = []
1066-
frag_directions = [] # 'abc': direction=1, 'xyz': direction=-1, otherwise 0
1132+
series_ids, loss_ids, charges, directions = _annotate_charged_frag_types(
1133+
fragment_mz_df.columns.values
1134+
)
10671135

1068-
for charged_frag_type in fragment_mz_df.columns.values:
1069-
frag_type, charge = parse_charged_frag_type(charged_frag_type)
1070-
frag_charges.append(charge)
1071-
frag_types.append(FRAGMENT_TYPES[frag_type].series_id)
1072-
frag_loss_types.append(FRAGMENT_TYPES[frag_type].loss_id)
1073-
frag_directions.append(FRAGMENT_TYPES[frag_type].direction_id)
1074-
1075-
if "type" in custom_columns:
1076-
frag_df["type"] = np.array(
1077-
np.tile(frag_types, len(fragment_mz_df)), dtype=np.int8
1078-
)
1079-
if "loss_type" in custom_columns:
1080-
frag_df["loss_type"] = np.array(
1081-
np.tile(frag_loss_types, len(fragment_mz_df)), dtype=np.int16
1082-
)
1083-
if "charge" in custom_columns:
1084-
frag_df["charge"] = np.array(
1085-
np.tile(frag_charges, len(fragment_mz_df)), dtype=np.int8
1086-
)
1136+
frag_type_annotations = {
1137+
"type": series_ids,
1138+
"loss_type": loss_ids,
1139+
"charge": charges,
1140+
}
1141+
for col_name, values in frag_type_annotations.items():
1142+
if col_name in custom_columns:
1143+
frag_df[col_name] = np.tile(values, len(fragment_mz_df))
10871144

1088-
frag_directions = np.array(
1089-
np.tile(frag_directions, (len(fragment_mz_df), 1)), dtype=np.int8
1090-
)
1145+
dense_directions = np.tile(directions, (len(fragment_mz_df), 1))
10911146

10921147
numbers, positions, excluded_indices = _parse_fragment(
1093-
frag_directions,
1148+
dense_directions,
10941149
precursor_df.frag_start_idx.values,
10951150
precursor_df.frag_stop_idx.values,
10961151
keep_top_k_fragments,
@@ -1104,12 +1159,6 @@ def flatten_fragments(
11041159
if "position" in custom_columns:
11051160
frag_df["position"] = positions.reshape(-1)
11061161

1107-
precursor_df["flat_frag_start_idx"] = precursor_df.frag_start_idx
1108-
precursor_df["flat_frag_stop_idx"] = precursor_df.frag_stop_idx
1109-
precursor_df[["flat_frag_start_idx", "flat_frag_stop_idx"]] *= len(
1110-
fragment_mz_df.columns
1111-
)
1112-
11131162
if use_intensity:
11141163
frag_df["intensity"][frag_df["mz"] == 0.0] = 0.0
11151164

@@ -1125,17 +1174,7 @@ def flatten_fragments(
11251174
frag_df = frag_df[~excluded]
11261175
frag_df = frag_df.reset_index(drop=True)
11271176

1128-
# cumulative sum counts the number of fragments before the given fragment which were removed.
1129-
# This sum does not include the fragment at the index position and has therefore len N +1
1130-
cum_sum_tresh = np.zeros(shape=len(excluded) + 1, dtype=np.int64)
1131-
cum_sum_tresh[1:] = np.cumsum(excluded)
1132-
1133-
precursor_df["flat_frag_start_idx"] -= cum_sum_tresh[
1134-
precursor_df.flat_frag_start_idx.values
1135-
]
1136-
precursor_df["flat_frag_stop_idx"] -= cum_sum_tresh[
1137-
precursor_df.flat_frag_stop_idx.values
1138-
]
1177+
_reannotate_precursor_pointers(precursor_df, excluded, len(fragment_mz_df.columns))
11391178

11401179
return precursor_df, frag_df
11411180

0 commit comments

Comments
 (0)