Skip to content

Add a modular preprocessing module for data preparation #32

Description

@iamprasadraju

Description

Introduce a trueml.preprocessing module that provides essential data preprocessing utilities for machine learning workflows. The module should be internally modular while exposing a simple, unified public API through trueml.preprocessing.

The goal is to provide lightweight, educational implementations that integrate naturally with the rest of TrueML.

Goals

  • Create the trueml.preprocessing package.
  • Design a modular internal package structure.
  • Support both NumPy arrays and Pandas DataFrames where appropriate.
  • Keep implementations simple, readable, and educational.
  • Expose a flat public API through trueml.preprocessing.
  • Maintain a consistent API across all preprocessing utilities.

Proposed Package Structure

trueml/
└── preprocessing/
    ├── __init__.py
    ├── scaling/
    │   ├── __init__.py
    │   ├── standard.py
    │   ├── minmax.py
    │   ├── robust.py
    │   ├── maxabs.py
    │   └── normalizer.py
    ├── encoding/
    │   ├── __init__.py
    │   ├── label.py
    │   ├── onehot.py
    │   └── ordinal.py
    ├── imputation/
    │   ├── __init__.py
    │   ├── mean.py
    │   ├── median.py
    │   ├── most_frequent.py
    │   └── constant.py
    ├── transformation/
    │   ├── __init__.py
    │   ├── log.py
    │   ├── polynomial.py
    │   └── power.py
    ├── feature_selection/
    │   ├── __init__.py
    │   ├── variance_threshold.py
    │   ├── correlation.py
    │   └── select_k_best.py
    ├── decomposition/
    │   ├── __init__.py
    │   └── pca.py
    ├── sampling/
    │   ├── __init__.py
    │   ├── oversampling.py
    │   └── undersampling.py
    └── splitting.py

Public API

from trueml.preprocessing import (
    StandardScaler,
    MinMaxScaler,
    RobustScaler,
    LabelEncoder,
    OneHotEncoder,
    MeanImputer,
    train_test_split,
)

Tasks

Scaling

  • StandardScaler
  • MinMaxScaler
  • RobustScaler
  • MaxAbsScaler
  • Normalizer

Encoding

  • LabelEncoder
  • OneHotEncoder
  • OrdinalEncoder

Missing Value Imputation

  • MeanImputer
  • MedianImputer
  • MostFrequentImputer
  • ConstantImputer

Data Splitting

  • train_test_split

Feature Transformation

  • LogTransformer
  • PolynomialFeatures
  • PowerTransformer

Feature Selection

  • VarianceThreshold
  • CorrelationFilter
  • SelectKBest

Dimensionality Reduction

  • PCA

Sampling

  • RandomOverSampler
  • RandomUnderSampler

Future Enhancements

  • KNNImputer
  • TargetEncoder
  • FeatureHasher
  • SMOTE
  • ADASYN
  • Recursive Feature Elimination (RFE)
  • Mutual Information feature selection
  • Column-wise transformations
  • Preprocessing pipelines

Acceptance Criteria

  • Modular internal architecture.
  • Flat public API via trueml.preprocessing.
  • Consistent API across all preprocessing components.
  • NumPy-first implementation with optional Pandas support.
  • Comprehensive documentation with examples.
  • Unit tests for every preprocessing component.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions