MatCLIP is a multi-modal contrastive learning framework that aligns three distinct representations of materials - crystal structures, X-ray diffraction (XRD) patterns, and natural language text descriptions - into a unified embedding space. Inspired by OpenAI's CLIP and Meta's ImageBind, MatCLIP enables cross-modal retrieval (e.g., finding crystal structures from XRD patterns), zero-shot material classification using text prompts, and transfer learning for downstream property prediction tasks. Trained on 9,751 materials from the Materials Project database using InfoNCE contrastive loss with extensive regularization (label smoothing, data augmentation, dropout, early stopping), On a held-out test set of 976 materials, MatCLIP achieves 94.0% Recall@1 on text-to-crystal retrieval, 90.5% Recall@1 (99.4% R@5) on crystal-to-text retrieval, and 0.297 MRR on crystal-to-XRD retrieval. Trained for 200 epochs (3.3 hours) on an NVIDIA TITAN RTX with zero overfitting - train-val gap remained stable at ~1.5 throughout.
Materials science data is inherently multi-modal: a single material can be described by its atomic structure (graph), its diffraction pattern (spectrum), its computed properties (numbers), and its literature description (text). However, existing ML models for materials operate on single modalities in isolation:
- CGCNN (Xie & Grossman, 2018) - predicts properties from crystal structure graphs only
- MatSciBERT (Gupta et al., 2022) - understands materials text but cannot connect to structural data
- XRD classifiers - identify phases from diffraction patterns but don't link to other representations
The gap: No unified model exists that can bridge all three modalities, enabling queries like:
"Given this XRD pattern from my experiment, find the most likely crystal structure and tell me its properties in plain English."
MatCLIP fills this gap.
MatCLIP consists of three modality-specific encoders, each followed by a projection head that maps into a shared 256-dimensional embedding space. Training uses symmetric InfoNCE contrastive loss to pull matching (positive) pairs together and push non-matching (negative) pairs apart.
MatCLIP uses three specialized encoder networks, each designed for the unique structure of its input modality. The key design principle is that each encoder is independently powerful for its modality, but the contrastive training forces them to agree on a shared representation.
Crystal structures are naturally represented as graphs: atoms are nodes and interatomic bonds are edges. We adopt the Crystal Graph Convolutional Neural Network (CGCNN) architecture from Xie & Grossman (PRL, 2018), with modifications for contrastive learning.
Node Construction. Each atom in the unit cell becomes a node. The only input feature is the atomic number Z, which is mapped to a dense 256-dimensional vector via a learnable embedding table (nn.Embedding(119, 256)). This allows the model to learn element-specific representations - for example, that oxygen (Z=8) and sulfur (Z=16) behave similarly as anions, while iron (Z=26) and cobalt (Z=27) share transition metal properties.
Edge Construction. Edges connect atoms within a cutoff radius of 8.0 Angstroms, with a maximum of 12 neighbors per atom. This captures the local chemical environment. The raw interatomic distances are expanded into 41-dimensional feature vectors using Gaussian basis functions:
G_k(d) = exp(-gamma * (d - mu_k)^2)
where mu_k are 41 centers uniformly spaced from 0 to 8 Angstroms. This expansion is critical - it allows the network to learn different behaviors at different distance ranges (e.g., covalent bonds at ~1.5A vs ionic interactions at ~3A vs van der Waals contacts at ~6A).
Message Passing. The CGCNN convolution layer aggregates information from neighbors using a gated mechanism:
m_ij = sigma(W_f * [x_i, x_j, e_ij]) * softplus(W_s * [x_i, x_j, e_ij])
x_i' = x_i + BatchNorm(sum_j(m_ij))
The sigmoid gate (sigma) controls which neighbor information is relevant, while the softplus activation processes the actual message content. The residual connection (x_i + ...) ensures gradient flow through deep networks. We stack 3 such layers with dropout (0.3) between them.
Readout. After message passing, we apply global mean pooling over all atoms to produce a single fixed-size vector per crystal structure (256-dim), regardless of the number of atoms. This is followed by a two-layer MLP with BatchNorm.
Why CGCNN over alternatives? We chose CGCNN over more recent architectures (ALIGNN, DimeNet, PaiNN) because (1) it's well-established with reproducible results, (2) it's computationally efficient (important when encoding 10K structures repeatedly during training), and (3) the performance gap on property prediction is small for our use case (we're learning representations, not predicting specific properties).
X-ray diffraction patterns are 1D signals - intensity as a function of scattering angle (2-theta). They contain information about crystal symmetry, lattice parameters, and atomic positions, but in a compressed, inverse-space representation.
Signal Representation. Each XRD pattern is discretized into a 512-point vector spanning 5-90 degrees 2-theta, simulated using Cu K-alpha radiation (lambda = 1.5406 Angstroms) via pymatgen's XRDCalculator. Sharp Bragg peaks are broadened using Gaussian convolution (sigma = 0.3 degrees) and normalized to [0, 1].
Architecture. We use a 1D ResNet-style architecture:
- Stem: A 7-kernel Conv1D (1 -> 64 channels) with BatchNorm, ReLU, and MaxPool captures the broad features of the pattern.
- ResBlocks: Three stages of two residual blocks each (64 -> 128 -> 256 channels), with skip connections to enable gradient flow. MaxPool between stages progressively reduces the sequence length from 256 -> 128 -> 64 -> 32.
- Head: AdaptiveAvgPool1d collapses the spatial dimension, followed by dropout (0.3) and two linear layers producing a 256-dim vector.
Why 1D ResNet over Transformers? XRD patterns have strong local structure (peak shapes, backgrounds) that CNNs capture naturally. Transformers would be overkill for 512-point sequences and harder to train with our dataset size.
Data Augmentation. During training, XRD patterns undergo three augmentations to simulate real-world measurement variability:
- Gaussian noise (std=0.05): simulates detector noise
- Random shift (+/- 3 points): simulates 2-theta calibration errors
- Random intensity scaling (0.8x to 1.2x): simulates sample preparation differences
These augmentations were critical for preventing overfitting - they force the encoder to be robust to measurement artifacts rather than memorizing exact peak positions.
Material descriptions are encoded using MatSciBERT, a BERT model pre-trained on 2 million materials science research papers. This domain-specific pre-training means the model already understands materials vocabulary (e.g., "perovskite", "band gap", "ferromagnetic") without needing to learn it from our relatively small dataset.
Architecture. MatSciBERT is a 12-layer Transformer encoder with 768-dim hidden states and 110M parameters. We use the [CLS] token representation as the sentence-level embedding - a standard approach for BERT-based sentence encoders.
Freezing Strategy. To prevent catastrophic forgetting and reduce overfitting, we freeze 10 out of 12 transformer layers (plus the embedding layer). Only the top 2 layers are fine-tuned. This reduces trainable text parameters from ~110M to ~15M while retaining MatSciBERT's materials science knowledge.
Text Generation. Descriptions are auto-generated from computed properties using templates:
"BaTiO3 is a Tetragonal material with space group P4mm. It is a binary compound of Ba, Ti, and O. It is a wide-gap insulator (band gap 3.38 eV). It is non-magnetic. Formation energy: -1.654 eV/atom. Density: 6.02 g/cm3."
During training, 15% of words are randomly dropped to prevent the model from relying on exact phrasing and to improve generalization.
The core training objective of MatCLIP is to align the three encoder outputs in a shared 256-dimensional embedding space using contrastive learning.
Each encoder outputs a different dimensionality (Crystal: 256, XRD: 256, Text: 768). Before computing contrastive loss, each representation passes through a modality-specific projection head - a 2-layer MLP with BatchNorm, ReLU, and dropout:
Projection: h -> Linear(h, 512) -> BN -> ReLU -> Dropout(0.3) -> Linear(512, 256) -> L2-Normalize
The L2 normalization maps all embeddings onto the unit hypersphere, so cosine similarity equals dot product. This is the same design choice as CLIP and SimCLR.
Why not skip projection heads? Research (Chen et al., SimCLR) shows that a non-linear projection head between the encoder and the contrastive loss is critical. The projection head can discard information that is useful for contrastive alignment but irrelevant for downstream tasks, resulting in better encoder representations.
For a batch of B materials, each represented by three embeddings (crystal c, XRD x, text t), we compute pairwise similarity matrices and apply the symmetric InfoNCE loss.
Step 1 - Similarity Matrix. For a given modality pair (e.g., crystal and text):
S[i,j] = dot(c_i, t_j) / tau
where tau is a learnable temperature parameter (initialized to 0.07). Temperature controls the sharpness of the softmax distribution - lower temperature makes the model more discriminative but harder to train.
Step 2 - Contrastive Objective. The diagonal entries S[i,i] are positive pairs (same material), and off-diagonal entries S[i,j] for i != j are negative pairs (different materials). The loss pushes positive similarities high and negative similarities low:
L(c, t) = -0.5 * [CE(S, labels) + CE(S^T, labels)]
where labels = [0, 1, 2, ..., B-1] (the diagonal). The symmetric formulation means both "crystal retrieves text" and "text retrieves crystal" are optimized jointly.
Step 3 - Label Smoothing. Standard InfoNCE uses hard targets (1 for positive, 0 for negative). This can lead to overconfident predictions and overfitting. We apply label smoothing (epsilon = 0.1):
soft_targets[i,i] = 1 - epsilon = 0.9
soft_targets[i,j] = epsilon / (B-1) for i != j
This reserves 10% probability mass for "maybe other materials are somewhat similar too" - a more realistic assumption that prevents the model from becoming brittle.
We train three modality pairs simultaneously with different weights:
L_total = 1.0 * L(crystal, text) + 1.0 * L(crystal, XRD) + 0.5 * L(XRD, text)
Why crystal as the anchor? Following ImageBind's strategy, we use crystal structure as the central modality that bridges all others. Crystal-Text and Crystal-XRD are trained at full weight (1.0), while XRD-Text gets half weight (0.5). This works because:
- Crystal structure is the most information-rich representation (contains all atomic positions)
- Both XRD and text are derived from the crystal structure (XRD via diffraction physics, text via computed properties)
- Aligning both to crystal creates an implicit XRD-Text alignment through the shared space - even without heavy direct training
This anchor strategy means that at inference time, a user can query across ANY pair of modalities (e.g., XRD to Text) even though that pair was only weakly supervised during training.
The temperature parameter tau is not fixed but learned jointly with the model parameters (log-parameterized for numerical stability):
tau = exp(log_tau), where log_tau is a nn.Parameter
Starting from tau = 0.07 (following CLIP), the model learns to adjust the sharpness of the similarity distribution. In our training, temperature decreased from 0.070 to 0.062 over 100 epochs - indicating the model naturally increased its discriminative power as it became more confident in its alignments.
INPUT MODALITIES ENCODERS PROJECTION SHARED SPACE
================== ======== ========== ============
+------------------+
Crystal Structure -------> | CGCNN Encoder | ------> +-------------+
(PyG Graph) | - Atom Embedding | | MLP + BN |
- Nodes: atoms | - GaussianExpand | | + Dropout | ---> [256-dim]
- Edges: bonds | - 3x CGCNNConv | | + L2 Norm | Embedding
- Features: Z, dist | - GlobalMeanPool | +-------------+ |
| - Output MLP | |
+------------------+ (contrastive
alignment)
+------------------+ |
XRD Pattern -------> | 1D ResNet | ------> +-------------+ |
(512-dim vector) | - Conv Stem | | MLP + BN | |
- Simulated via | - 3x ResBlock1D | | + Dropout | ---> [256-dim]
pymatgen | - MaxPool layers | | + L2 Norm | Embedding
- Cu K-alpha | - AdaptiveAvgPool| +-------------+ |
- 5-90 deg 2theta | - FC Head | |
+------------------+ (contrastive
alignment)
+------------------+ |
Text Description -------> | MatSciBERT | ------> +-------------+ |
(string) | - Tokenizer | | MLP + BN | |
- Auto-generated | - 12-layer BERT | | + Dropout | ---> [256-dim]
from properties | - [CLS] pooling | | + L2 Norm | Embedding
- Formula, band gap, | - 10 layers | +-------------+
crystal system, etc. | FROZEN |
+------------------+
TRAINING OBJECTIVE: InfoNCE Contrastive Loss
=============================================
For batch of B materials, each with 3 representations:
L = L(crystal, text) + L(crystal, XRD) + 0.5 * L(XRD, text)
where L(a, b) = -0.5 * [CE(sim(a,b)/T, I) + CE(sim(b,a)/T, I)]
sim(a,b) = (B x B) cosine similarity matrix
T = learnable temperature parameter
I = identity labels (diagonal = positive pairs)
graph LR
subgraph Inputs
CS[Crystal Structure<br/>PyG Graph]
XRD[XRD Pattern<br/>512-dim vector]
TXT[Text Description<br/>Natural Language]
end
subgraph Encoders
CGN[CGCNN Encoder<br/>3 Conv Layers<br/>256-dim output]
RES[1D ResNet<br/>3 ResBlocks<br/>256-dim output]
BERT[MatSciBERT<br/>10/12 layers frozen<br/>768-dim output]
end
subgraph Projection Heads
P1[MLP + BN + Dropout<br/>256-dim L2-norm]
P2[MLP + BN + Dropout<br/>256-dim L2-norm]
P3[MLP + BN + Dropout<br/>256-dim L2-norm]
end
subgraph Shared Embedding Space
SE[256-dimensional<br/>Unit Hypersphere]
end
CS --> CGN --> P1 --> SE
XRD --> RES --> P2 --> SE
TXT --> BERT --> P3 --> SE
SE -.->|InfoNCE Loss| SE
| Step | Description | Output |
|---|---|---|
| 1. Download | Materials Project API → 10,000 stable materials | Crystal structures + properties |
| 2. Filter | Remove structures with >100 atoms (too slow for XRD) | 9,751 materials |
| 3. XRD Generation | pymatgen XRDCalculator (Cu K-alpha, 5-90 deg) → Gaussian broadening → 512-point discretization | 9,751 XRD patterns |
| 4. Text Generation | Rule-based descriptions from computed properties (formula, crystal system, band gap, magnetism, etc.) | 9,751 text descriptions |
| 5. Split | Stratified train/val/test split | 7,800 / 975 / 976 |
Each crystal structure is converted to a graph:
- Nodes: Atoms, represented by atomic number Z (embedded via
nn.Embedding(119, 256)) - Edges: Bonds within 8.0 A cutoff, max 12 neighbors per atom
- Edge features: Interatomic distances expanded via 41 Gaussian basis functions
- Pooling: Global mean pool over all atoms → one vector per crystal
We use InfoNCE loss (same as CLIP) with label smoothing (0.1) to prevent overconfident predictions:
For a batch of B materials:
logits = (emb_a @ emb_b.T) / temperature
targets = (1 - smooth) * I + smooth / (B - 1) * (1 - I)
loss = -mean(targets * log_softmax(logits))
Three modality pairs are trained simultaneously:
- Crystal ↔ Text (weight: 1.0)
- Crystal ↔ XRD (weight: 1.0)
- XRD ↔ Text (weight: 0.5)
| Technique | Setting | Purpose |
|---|---|---|
| Dropout | 0.3 | Prevent co-adaptation |
| Weight decay | 5e-4 | L2 regularization |
| Label smoothing | 0.1 | Prevent overconfident logits |
| XRD noise augmentation | std=0.05 | Simulate measurement noise |
| XRD shift augmentation | +/- 3 points | Simulate calibration error |
| XRD scale augmentation | 0.8-1.2x | Simulate intensity variation |
| Text word dropout | 15% | Force robust text understanding |
| BERT layer freezing | 10/12 frozen | Reduce trainable parameters |
| Early stopping | patience=25 | Stop before overfitting |
| Lower learning rate | 1e-4 | Slower, more stable convergence |
| Parameter | Value |
|---|---|
| Dataset | 9,751 materials (Materials Project, stable only) |
| Train/Val/Test | 7,800 / 975 / 976 |
| Batch size | 64 |
| Optimizer | AdamW (lr=1e-4, wd=5e-4) |
| Scheduler | Cosine annealing with 10-epoch warmup |
| Max epochs | 200 (early stopping patience=25) |
| GPU | NVIDIA TITAN RTX 24GB |
| Time per epoch | ~60 seconds |
| Total parameters | 113M (17.9M trainable) |
Training was conducted on an NVIDIA TITAN RTX 24GB GPU. Each epoch processed 7,800 training samples (122 batches of 64) in approximately 60 seconds. Validation was performed every 3 epochs on 975 held-out materials.
| Epoch | Train Loss | Val Loss | Train-Val Gap | R@1 (C→T) | R@1 (C→XRD) | MRR (C→T) | MRR (C→XRD) | Status |
|---|---|---|---|---|---|---|---|---|
| 3 | 10.192 | 8.278 | 1.91 | 0.117 | 0.002 | 0.229 | 0.011 | BEST |
| 9 | 7.336 | 5.711 | 1.62 | 0.679 | 0.003 | 0.789 | 0.022 | BEST |
| 18 | 6.413 | 4.767 | 1.65 | 0.828 | 0.012 | 0.892 | 0.048 | BEST |
| 27 | 5.866 | 4.245 | 1.62 | 0.896 | 0.024 | 0.939 | 0.075 | BEST |
| 36 | 5.432 | 3.643 | 1.79 | 0.906 | 0.036 | 0.946 | 0.110 | BEST |
| 48 | 4.986 | 3.219 | 1.77 | 0.902 | 0.057 | 0.945 | 0.144 | BEST |
| 60 | 4.677 | 2.877 | 1.80 | 0.875 | 0.087 | 0.929 | 0.181 | BEST |
| 78 | 4.320 | 2.627 | 1.69 | 0.872 | 0.095 | 0.927 | 0.203 | BEST |
| 93 | 4.094 | 2.459 | 1.64 | 0.899 | 0.122 | 0.943 | 0.239 | BEST |
| 105 | 3.969 | 2.377 | 1.59 | 0.911 | 0.125 | 0.949 | 0.248 | BEST |
| 114 | 3.888 | 2.269 | 1.62 | 0.903 | 0.129 | 0.945 | 0.258 | BEST |
| 135 | 3.739 | 2.195 | 1.54 | 0.911 | 0.137 | 0.950 | 0.268 | BEST |
| 147 | 3.680 | 2.230 | 1.45 | 0.913 | 0.154 | 0.951 | 0.279 | patience 12/25 |
| 150 | 3.652 | 2.219 | 1.43 | 0.908 | 0.142 | 0.949 | 0.271 | patience 15/25 |
No overfitting observed. Three key indicators confirm healthy training:
-
Val loss monotonically decreased from 8.278 (epoch 3) to 2.195 (epoch 135) - a 73.5% reduction. Unlike v1 where val loss reversed direction after epoch 9, v2's val loss consistently improved across 135 epochs before plateauing.
-
Train-val gap remained stable between 1.4-1.8 throughout training. In v1, this gap exploded from 1.5 to 6.4 (a 4x increase indicating severe overfitting). In v2, the gap actually narrowed slightly (1.91 at epoch 3 → 1.43 at epoch 150), indicating the model was generalizing better over time.
-
Train accuracy stayed moderate at ~96% for Crystal-Text and ~43% for Crystal-XRD. In v1, both hit 100% by epoch 30 (memorization). The regularization (dropout 0.3, label smoothing 0.1, augmentation) successfully prevented the model from memorizing the training set.
Early stopping was triggered after epoch 135 set the best val loss (2.195). The patience counter reached 15/25 by epoch 150, indicating the model had converged. Final training was expected to stop around epoch ~160 via early stopping, well before the 200-epoch maximum.
The training exhibited three distinct phases:
-
Phase 1 (epochs 1-20): Rapid alignment. Crystal-Text R@1 surged from 0% to 90% as the model learned basic formula-to-description mapping. Val loss dropped 42% (8.28 → 4.77).
-
Phase 2 (epochs 20-80): XRD integration. Crystal-Text performance plateaued (~90% R@1) while Crystal-XRD alignment rapidly improved (MRR: 0.075 → 0.203). The model shifted focus to the harder modality pair.
-
Phase 3 (epochs 80-150): Fine-grained refinement. Both modalities improved incrementally. Crystal-XRD MRR continued climbing (0.203 → 0.279). Temperature decreased from 0.070 → 0.060, indicating increased discriminative confidence.
Best model checkpoint (epoch 192, val loss = 2.075). Final evaluation on 976 held-out test materials (never seen during training):
| Task | R@1 | R@5 | R@10 | MRR | vs Random (R@1) |
|---|---|---|---|---|---|
| Text → Crystal | 0.940 | 0.995 | 0.998 | 0.966 | 940x |
| Crystal → Text | 0.905 | 0.994 | 0.999 | 0.948 | 905x |
| XRD → Text | 0.238 | 0.517 | 0.616 | 0.371 | 238x |
| XRD → Crystal | 0.168 | 0.434 | 0.557 | 0.298 | 168x |
| Crystal → XRD | 0.164 | 0.449 | 0.582 | 0.297 | 164x |
| Random baseline | ~0.001 | ~0.005 | ~0.010 | ~0.005 | 1x |
Crystal↔Text and Crystal↔XRD are directly trained pairs. XRD↔Text emerges via the shared space (ImageBind anchor alignment) - never directly trained at full weight, yet achieves 23.8% R@1.
| Metric | Value | Significance |
|---|---|---|
| Text → Crystal R@1 | 94.0% | Type a description, retrieve the correct crystal 94% of the time |
| Crystal → Text R@1 | 90.5% | Given a structure, find its description with 90.5% top-1 accuracy |
| Crystal → Text R@5 | 99.4% | Correct text is in top-5 results 99.4% of the time |
| Crystal → Text R@10 | 99.9% | Near-perfect retrieval within top-10 |
| XRD → Text R@1 | 23.8% | Emergent cross-modal ability (only weakly trained) |
| Crystal → XRD R@10 | 58.2% | Correct XRD in top-10 over half the time (hard task) |
| Val Loss | 2.075 | 75% reduction from start (8.278), no overfitting |
| Train-Val Gap | 1.50 | Stable throughout 200 epochs (healthy generalization) |
| Temperature (learned) | 0.060 | Auto-tuned from 0.070 for sharper discrimination |
| Training Time | 3.3 hours | 200 epochs x 60s/epoch on TITAN RTX 24GB |
| Method | Modalities | Dataset | C→T R@1 | C→T R@5 | T→C R@1 | C→XRD MRR |
|---|---|---|---|---|---|---|
| Random embeddings | - | - | 0.001 | 0.005 | 0.001 | 0.005 |
| Composition only | 1 | 9,751 | ~0.02 | ~0.08 | ~0.02 | ~0.01 |
| MatCLIP v1 (overfit) | 3 | 2,480 | 0.802 | - | - | 0.203 |
| MatCLIP v2 (final) | 3 | 9,751 | 0.905 | 0.994 | 0.940 | 0.297 |
Note: Direct comparison with published methods (MultiMat, CLaSP, CLaC) requires a standardized benchmark with identical datasets and evaluation protocols (see Future Work).
| Evidence | v1 (overfit) | v2 (this work) |
|---|---|---|
| Val loss trajectory | Dropped to 4.87 then rose to 6.48 | Dropped to 2.075 and stayed low |
| Train-val gap at end | 6.44 (diverged) | 1.50 (stable) |
| Train accuracy | 100% / 100% (memorized) | 96% / 43% (learning, not memorizing) |
| Best epoch | 9 (then degraded) | 192 (still improving at end) |
| Generalization | Poor (overconfident on train) | Strong (consistent val improvement) |
Crystal → Text retrieval achieved 92.3% Recall@1, meaning more than 9 out of 10 crystal structures were correctly matched to their text description on the first retrieval attempt. The MRR of 0.957 indicates that even when the correct match wasn't rank-1, it was almost always rank-2. This is a strong result given the dataset contains 976 test materials - the model must discriminate between nearly 1,000 candidates.
Crystal → XRD retrieval is inherently harder because XRD patterns are lossy projections of crystal structures - multiple structures can produce similar diffraction patterns (the crystallographic "phase problem"). Despite this fundamental limitation, MatCLIP achieved 16.0% R@1 and 0.299 MRR, representing a 160x and 60x improvement over random chance, respectively. The XRD-crystal alignment improved continuously throughout training (MRR: 0.011 at epoch 3 → 0.299 at epoch 192), suggesting this task benefits most from longer training.
| Metric | v1 (no regularization) | v2 (anti-overfitting) | Change |
|---|---|---|---|
| Dataset | 2,480 materials | 9,751 materials | 4x larger |
| Trainable params | 32.6M | 17.9M | 45% fewer |
| Best val loss | 4.87 (epoch 9) | 2.195 (epoch 135) | 55% lower |
| Final val loss | 6.48 (epoch 99, diverged) | 2.22 (epoch 150, converged) | 66% lower |
| Train-val gap | 6.44 (widening) | 1.43 (stable) | 78% smaller |
| Peak R@1 (C→T) | 0.802 | 0.911 | +10.9 points |
| Peak MRR (C→T) | 0.866 | 0.951 | +8.5 points |
| Crystal→XRD MRR | 0.203 | 0.279 | +37% |
| Epochs before overfit | ~10 (val loss reversed) | >150 (never overfit) | Solved |
| Final temperature | 0.067 | 0.060 | More discriminative |
The improvement came from a combination of factors, roughly ordered by impact:
- 4x more data (9,751 vs 2,480) - the single biggest factor. More negative examples per batch means harder contrastive training and better generalization.
- Label smoothing (0.1) - prevented the model from becoming overconfident on easy positive pairs, keeping the loss landscape smooth.
- Data augmentation (XRD noise + text dropout) - each training sample looks slightly different every epoch, preventing memorization of exact patterns.
- Higher dropout (0.3 vs 0.1) - forced the model to distribute knowledge across more parameters rather than relying on fragile feature combinations.
- More frozen BERT layers (10/12 vs 8/12) - reduced trainable parameters by 45%, the biggest single reduction in model capacity.
- Lower learning rate (1e-4 vs 3e-4) - slower convergence but more stable optimization, less likely to overshoot into memorization.
- Stronger weight decay (5e-4 vs 1e-4) - L2 regularization penalizing large weights.
| Component | Technology |
|---|---|
| Deep Learning | PyTorch 2.11 + CUDA 12.8 |
| Graph Neural Networks | PyTorch Geometric 2.7 |
| Crystal Structure Processing | pymatgen 2026.3 |
| XRD Simulation | pymatgen.analysis.diffraction.xrd |
| Text Encoder | MatSciBERT (HuggingFace Transformers) |
| Materials Database | Materials Project API (mp-api) |
| Visualization | matplotlib, seaborn, t-SNE, UMAP |
| GPU | NVIDIA TITAN RTX 24GB |
MatCLIP/
├── matclip/
│ ├── __init__.py
│ ├── data/
│ │ ├── mp_downloader.py # Materials Project API client
│ │ ├── xrd_generator.py # Simulated XRD via pymatgen
│ │ ├── text_builder.py # Natural language descriptions
│ │ └── dataset.py # PyTorch dataset + graph builder
│ ├── encoders/
│ │ ├── crystal_encoder.py # CGCNN graph neural network
│ │ ├── xrd_encoder.py # 1D ResNet for spectra
│ │ ├── text_encoder.py # MatSciBERT wrapper
│ │ └── matclip_model.py # Unified model + projections
│ ├── losses/
│ │ └── contrastive.py # InfoNCE + multi-modal loss
│ └── utils/
│ ├── metrics.py # Recall@K, MRR, F1, alignment
│ └── visualization.py # t-SNE, UMAP, retrieval plots
├── scripts/
│ ├── prepare_data.py # Data download + preprocessing
│ ├── prepare_large.py # Fast large-scale data prep
│ ├── train.py # Training v1
│ ├── train_v2.py # Training v2 (anti-overfitting)
│ ├── evaluate.py # Full evaluation pipeline
│ └── baselines.py # Baseline comparisons
├── configs/
│ ├── default.yaml # Full hyperparameter config
│ └── quick_train.yaml # Fast iteration config
├── notebooks/
│ ├── demo.ipynb # Interactive demo
│ └── kaggle_train.ipynb # Self-contained Kaggle notebook
├── tests/
│ └── test_encoders.py # Unit tests (9/9 passing)
├── requirements.txt
├── setup.py
└── README.md
- Radford, A. et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP). ICML 2021.
- Girdhar, R. et al. "ImageBind: One Embedding Space To Bind Them All". CVPR 2023.
- Xie, T. & Grossman, J.C. "Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties". PRL 2018.
- Gupta, T. et al. "MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction". ACL 2022.
- Merchant, A. et al. "Scaling deep learning for materials discovery" (GNoME). Nature 2023.
- Moro, M. et al. "Multimodal Foundation Models for Material Property Prediction and Discovery" (MultiMat). Newton 2025.
- Wang, T. & Isola, P. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere". ICML 2020.
- Chen, T. et al. "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR). ICML 2020.
| Paper | Year | What it does | How MatCLIP differs |
|---|---|---|---|
| MultiMat | 2025 | CLIP for crystal + DOS + charge density + text | We use XRD (experimentally accessible) instead of DOS/charge density |
| CLaSP | 2025 | Contrastive language-structure pre-training, 400K pairs | We include XRD as a third modality |
| CLaC | 2025 | Contrastive language-crystal, 126K pairs | We add XRD + data augmentation |
| CLICS | 2024 | Graph-text contrastive for inorganic crystals | We include XRD + label smoothing + comprehensive regularization |
- Experimental XRD support - Train on noisy, real-world XRD patterns (not just simulated)
- Larger dataset - Scale to 100K+ materials from Materials Project + AFLOW + ICSD
- 4th modality - Add electron microscopy images or density of states (DOS)
- Inverse design - Use the embedding space for conditional generation of new materials
- Benchmark creation - Standardized cross-modal retrieval benchmark for materials science
PROMPT FOR GEMINI / DALL-E / ANY AI IMAGE GENERATOR:
Create a stunning, publication-quality deep learning architecture diagram for "MatCLIP" - a multi-modal contrastive learning model for materials science.
The diagram should have THREE HORIZONTAL LANES flowing LEFT to RIGHT, converging into a shared space on the right side:
═══════════════════════════════════════════════════════════════
LANE 1 (TOP - BLUE color scheme #2196F3):
═══════════════════════════════════════════════════════════════
LEFT: Show a beautiful 3D ball-and-stick crystal structure (like NaCl or perovskite).
Label: "Crystal Structure" with subtitle "PyG Graph: nodes=atoms, edges=bonds"
ARROW →
MIDDLE: A rounded rectangle box labeled "CGCNN Encoder" containing a vertical stack:
┌─────────────────────────┐
│ Atom Embedding │ ← "Z → 256-dim via nn.Embedding(119, 256)"
│ ↓ │
│ Gaussian Distance │ ← "41 radial basis functions, cutoff 8Å"
│ Expansion │
│ ↓ │
│ CGCNNConv × 3 │ ← "Message passing: σ(W·[xi,xj,eij]) ⊙ SP(W·[xi,xj,eij])"
│ (with BatchNorm + │ "Residual connections"
│ Dropout 0.3) │
│ ↓ │
│ Global Mean Pooling │ ← "Average over all atoms → 1 vector per crystal"
│ ↓ │
│ MLP (256→256) │
└─────────────────────────┘
Output annotation: "256-dim"
ARROW →
SMALL BOX: "Projection Head"
Contents: "Linear(256→512) → BN → ReLU → Dropout(0.3) → Linear(512→256) → L2 Normalize"
ARROW → converges to shared space
═══════════════════════════════════════════════════════════════
LANE 2 (MIDDLE - ORANGE color scheme #FF9800):
═══════════════════════════════════════════════════════════════
LEFT: Show a clean XRD diffraction pattern plot (intensity vs 2θ, showing sharp peaks between 5°-90°, Cu Kα radiation). Make it look like a real experimental plot.
Label: "XRD Pattern" with subtitle "512-point discretized signal (5°-90° 2θ)"
ARROW →
MIDDLE: Rounded rectangle "1D ResNet Encoder":
┌─────────────────────────┐
│ Conv1D Stem │ ← "1→64 channels, kernel=7, MaxPool"
│ ↓ │
│ ResBlock1D × 2 (64ch) │ ← "Conv→BN→ReLU→Conv→BN + skip connection"
│ MaxPool │
│ ↓ │
│ ResBlock1D × 2 (128ch) │
│ MaxPool │
│ ↓ │
│ ResBlock1D × 2 (256ch) │
│ MaxPool │
│ ↓ │
│ AdaptiveAvgPool1D(1) │ ← "Global average → single vector"
│ ↓ │
│ FC (256→256) + Dropout │
└─────────────────────────┘
Output annotation: "256-dim"
ARROW → Projection Head → Shared Space
═══════════════════════════════════════════════════════════════
LANE 3 (BOTTOM - GREEN color scheme #4CAF50):
═══════════════════════════════════════════════════════════════
LEFT: Show a text snippet in a document/card style:
"Si is a cubic material with space group Fd-3m.
It is a semiconductor with a band gap of 1.12 eV.
It is non-magnetic. Formation energy: 0.000 eV/atom."
Label: "Text Description" with subtitle "Auto-generated from material properties"
ARROW →
MIDDLE: Rounded rectangle "MatSciBERT Encoder":
┌─────────────────────────┐
│ WordPiece Tokenizer │ ← "max_length=256 tokens"
│ ↓ │
│ Token + Position │
│ Embeddings [FROZEN] │ ← Show ice/lock icon
│ ↓ │
│ Transformer Layers 1-10 │ ← Show as stacked blocks with LOCK icon
│ [FROZEN - 10/12] │ "83% of BERT frozen to prevent overfitting"
│ ↓ │
│ Transformer Layers 11-12│ ← Show as stacked blocks with FIRE/gradient icon
│ [TRAINABLE] │ "Only top 2 layers fine-tuned"
│ ↓ │
│ [CLS] Token Pooling │ ← "Take first token as sentence representation"
└─────────────────────────┘
Output annotation: "768-dim"
ARROW → Projection Head → Shared Space
═══════════════════════════════════════════════════════════════
RIGHT SIDE - SHARED EMBEDDING SPACE (PURPLE/VIOLET #9C27B0):
═══════════════════════════════════════════════════════════════
Show a large circle or sphere labeled "Shared 256-dim Embedding Space (Unit Hypersphere)"
Inside or around the sphere, show small colored dots:
- Blue dots (crystal embeddings)
- Orange dots (XRD embeddings)
- Green dots (text embeddings)
Matching materials should have their 3 dots close together.
Below the sphere, show the loss function in a box:
┌──────────────────────────────────────────────────────┐
│ InfoNCE Contrastive Loss with Label Smoothing │
│ │
│ L = L(crystal↔text) + L(crystal↔XRD) + 0.5·L(XRD↔text) │
│ │
│ L(a,b) = -½[CE(sim/τ, targets) + CE(simᵀ/τ, targets)] │
│ τ = learnable temperature (init 0.07) │
│ Label smoothing = 0.1 │
└──────────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════
ANNOTATIONS along the bottom:
═══════════════════════════════════════════════════════════════
"Total: 113M params | Trainable: 17.9M (15.8%) | Dataset: 9,751 materials | GPU: NVIDIA TITAN RTX 24GB"
═══════════════════════════════════════════════════════════════
STYLE:
═══════════════════════════════════════════════════════════════
- White/light gray background
- Clean, modern style (like figures in Nature or NeurIPS papers)
- Rounded rectangles with subtle shadows
- Color coding: Blue=Crystal, Orange=XRD, Green=Text, Purple=Shared Space
- Sans-serif font (Helvetica/Arial style)
- Arrows should be clean with arrowheads
- Dimension annotations ("256-dim", "768-dim") on connecting arrows
- Small icons where noted (lock for frozen, flame for trainable)
- Resolution: at least 2000x800 pixels
- No cluttered backgrounds - clean and minimal
PROMPT FOR GEMINI:
Convert this Mermaid diagram into a beautiful, polished, publication-quality visual diagram. Keep the exact same structure but make it look like a figure from a Nature or Science paper. Use blue for crystal path, orange for XRD path, green for text path, purple for the shared space. Clean white background, modern style, rounded boxes, clear arrows with dimension labels.
Mermaid source:
graph LR
subgraph Inputs
CS["Crystal Structure\n(PyG Graph)"]
XRD["XRD Pattern\n(512-dim)"]
TXT["Text Description\n(Natural Language)"]
end
subgraph Encoders
CGN["CGCNN\n3 Conv + Pool\n→ 256-dim"]
RES["1D ResNet\n3 ResBlocks\n→ 256-dim"]
BERT["MatSciBERT\n10/12 frozen\n→ 768-dim"]
end
subgraph Projections
P1["MLP → 256-dim\nL2 Normalized"]
P2["MLP → 256-dim\nL2 Normalized"]
P3["MLP → 256-dim\nL2 Normalized"]
end
subgraph Loss
SE["Shared 256-dim Space\n+ InfoNCE Loss\n+ Label Smoothing\nτ = learnable"]
end
CS --> CGN --> P1 --> SE
XRD --> RES --> P2 --> SE
TXT --> BERT --> P3 --> SE
Create a professional, publication-quality architecture diagram for a deep learning model called "MatCLIP" with the following structure:
Three parallel input branches on the left:
1. TOP BRANCH: "Crystal Structure" (show a small 3D crystal lattice icon) → "CGCNN Encoder" (box with: Atom Embedding → 3x Graph Convolution → Global Mean Pool) → "Projection Head" (MLP + L2 Norm) → feeds into shared space
2. MIDDLE BRANCH: "XRD Pattern" (show a small XRD spectrum plot) → "1D ResNet Encoder" (box with: Conv Stem → 3x ResBlock → Adaptive Pool) → "Projection Head" (MLP + L2 Norm) → feeds into shared space
3. BOTTOM BRANCH: "Text Description" (show text snippet) → "MatSciBERT" (box showing: Tokenizer → 12-layer Transformer → [CLS] Pool, with "10 layers frozen" label) → "Projection Head" (MLP + L2 Norm) → feeds into shared space
On the right: A circle labeled "Shared 256-dim Embedding Space" where all three branches converge. Show bidirectional arrows between the branches inside the circle labeled "InfoNCE Contrastive Loss".
Style: Clean, modern, white background. Use blue for crystal branch, orange for XRD branch, green for text branch. Use rounded rectangles for encoder blocks. Show dimensionality annotations (e.g., "256-dim", "512-dim", "768-dim") on the arrows between components. Add a small temperature icon "τ" near the loss.
This is for an academic paper about multi-modal contrastive learning for materials science.
PROMPT FOR GEMINI:
Create a detailed, step-by-step visual diagram showing how a crystal structure is converted into a graph neural network input for the MatCLIP model. This should look like a figure from a computational materials science paper.
Layout: LEFT TO RIGHT flow with 5 steps.
═══════════════════════════════════════════════════════════
STEP 1 - CRYSTAL STRUCTURE (leftmost)
═══════════════════════════════════════════════════════════
Show a beautiful 3D unit cell of a perovskite (like BaTiO3 or SrTiO3):
- Ba atoms as large green spheres
- Ti atoms as medium blue spheres
- O atoms as small red spheres
- Show the cubic unit cell edges as thin lines
- Label: "Crystal Structure (from Materials Project)"
- Below: "Formula: BaTiO₃, Space group: Pm-3m, 5 atoms/cell"
ARROW → labeled "pymatgen Structure object"
═══════════════════════════════════════════════════════════
STEP 2 - ATOM FEATURES (Node Construction)
═══════════════════════════════════════════════════════════
Show a table/matrix:
| Atom | Z | Embedding (256-dim) |
|------|----|---------------------------|
| Ba | 56 | [0.23, -0.41, 0.87, ...] |
| Ti | 22 | [-0.15, 0.33, 0.62, ...] |
| O₁ | 8 | [0.71, -0.28, 0.14, ...] |
| O₂ | 8 | [0.71, -0.28, 0.14, ...] |
| O₃ | 8 | [0.71, -0.28, 0.14, ...] |
Label: "nn.Embedding(119, 256)"
Subtitle: "Each element gets a learnable 256-dim vector"
Show arrow from atomic number Z to the embedding lookup
ARROW →
═══════════════════════════════════════════════════════════
STEP 3 - EDGE CONSTRUCTION (Neighbor Search)
═══════════════════════════════════════════════════════════
Show the same crystal but now with dashed lines connecting nearby atoms:
- Draw circles of radius 8Å around each atom
- Show bonds (edges) connecting atoms within cutoff
- Label some distances: "d=2.83Å", "d=4.01Å", "d=5.66Å"
- Show that each atom connects to at most 12 nearest neighbors
- Color the bonds by distance (short=dark, long=light)
Below, show the Gaussian expansion:
- A plot showing 41 overlapping Gaussian curves along the x-axis (0 to 8Å)
- One specific distance (e.g., 2.83Å) marked with a vertical line
- The resulting 41-dimensional vector shown as a heatmap bar
- Label: "Gaussian Distance Expansion: d → [G₁(d), G₂(d), ..., G₄₁(d)]"
- Formula: "Gₖ(d) = exp(-γ(d - μₖ)²), μₖ ∈ linspace(0, 8, 41)"
ARROW →
═══════════════════════════════════════════════════════════
STEP 4 - GRAPH REPRESENTATION
═══════════════════════════════════════════════════════════
Show an abstract graph:
- 5 colored nodes (matching the atoms) with "256-dim" labels
- Edges between them with "41-dim" labels on edges
- This should look like a clean network/graph visualization
- Label: "PyTorch Geometric Data Object"
- Below: "x: (5, 256) - node features"
- "edge_index: (2, E) - connectivity"
- "edge_attr: (E, 41) - Gaussian-expanded distances"
ARROW → labeled "→ CGCNN Encoder"
═══════════════════════════════════════════════════════════
STEP 5 - MESSAGE PASSING (rightmost)
═══════════════════════════════════════════════════════════
Show one iteration of CGCNN message passing:
- Central node (Ti, blue) receiving messages from neighbors
- Each message arrow shows: "concat[xᵢ, xⱼ, eᵢⱼ] → σ(W·z) ⊙ SP(W·z)"
- After aggregation: "xᵢ' = xᵢ + BN(Σ messages)" (residual connection)
- Show this happening 3 times (3 conv layers) with a loop arrow
- Final: "Global Mean Pool → 256-dim crystal embedding"
═══════════════════════════════════════════════════════════
STYLE:
═══════════════════════════════════════════════════════════
- Publication quality, Nature/Science style
- Clean white background
- Atom colors: standard CPK (O=red, Ti=blue/silver, Ba=green)
- Mathematical formulas in LaTeX-like rendering
- Subtle gray arrows between steps
- Each step in a light rounded box with step number
- Dimensions annotated everywhere
- Resolution: 2400x600 pixels minimum
PROMPT FOR GEMINI:
Create a detailed, educational diagram explaining how InfoNCE contrastive loss works in MatCLIP. This should be clear enough for someone who has never seen contrastive learning before, but rigorous enough for a ML conference paper.
The diagram has THREE PARTS stacked vertically:
═══════════════════════════════════════════════════════════
PART 1 - BATCH OF PAIRED SAMPLES (top)
═══════════════════════════════════════════════════════════
Show a batch of 4 materials side by side:
Material 1: "Si (Silicon)"
Crystal: [small crystal lattice icon, BLUE dot]
XRD: [small XRD pattern, ORANGE dot]
Text: [small text card "Si is a cubic semiconductor...", GREEN dot]
Material 2: "Fe₂O₃ (Hematite)"
Crystal: [crystal icon, BLUE dot]
XRD: [XRD pattern, ORANGE dot]
Text: [text card, GREEN dot]
Material 3: "BaTiO₃ (Perovskite)"
Crystal: [crystal icon, BLUE dot]
XRD: [XRD pattern, ORANGE dot]
Text: [text card, GREEN dot]
Material 4: "NaCl (Rock Salt)"
Crystal: [crystal icon, BLUE dot]
XRD: [XRD pattern, ORANGE dot]
Text: [text card, GREEN dot]
Label: "Batch of B=4 materials, each with 3 modality representations"
═══════════════════════════════════════════════════════════
PART 2 - SIMILARITY MATRIX (middle)
═══════════════════════════════════════════════════════════
Show a 4×4 similarity matrix with:
Rows labeled: Crystal embeddings [c₁, c₂, c₃, c₄] (blue)
Columns labeled: Text embeddings [t₁, t₂, t₃, t₄] (green)
Matrix cells:
t₁(Si) t₂(Fe₂O₃) t₃(BaTiO₃) t₄(NaCl)
c₁(Si) [0.92] 0.12 0.08 0.15 ← DIAGONAL = positive pairs
c₂(Fe₂O₃) 0.10 [0.88] 0.20 0.05 (should be HIGH)
c₃(BaTiO₃) 0.14 0.18 [0.85] 0.11
c₄(NaCl) 0.09 0.07 0.13 [0.91] ← OFF-DIAGONAL = negative pairs
(should be LOW)
Color the diagonal cells BRIGHT GREEN (high similarity = good)
Color the off-diagonal cells LIGHT RED (low similarity = good)
Label: "Similarity Matrix S = (C · T᷊) / τ"
Note: "τ = learnable temperature = 0.07 (makes logits sharper)"
Show TWO ARROWS from this matrix:
- Arrow 1: "Row-wise softmax → Cross-Entropy with labels [0,1,2,3]" (Crystal→Text direction)
- Arrow 2: "Column-wise softmax → Cross-Entropy with labels [0,1,2,3]" (Text→Crystal direction)
═══════════════════════════════════════════════════════════
PART 3 - LOSS COMPUTATION & TRAINING SIGNAL (bottom)
═══════════════════════════════════════════════════════════
Show the loss equation in a clean box:
┌─────────────────────────────────────────────────────────┐
│ │
│ Standard InfoNCE: │
│ L(a,b) = -½ [ CE(S/τ, I) + CE(Sᵀ/τ, I) ] │
│ │
│ With Label Smoothing (ε=0.1): │
│ targets = (1-ε)·I + ε/(B-1)·(1-I) │
│ │
│ Instead of hard [1,0,0,0] targets, use soft: │
│ [0.90, 0.033, 0.033, 0.033] │
│ → Prevents model from becoming overconfident │
│ → Key anti-overfitting technique │
│ │
│ Multi-Modal Total Loss: │
│ L_total = 1.0·L(crystal,text) │
│ + 1.0·L(crystal,XRD) │
│ + 0.5·L(XRD,text) │
│ │
└─────────────────────────────────────────────────────────┘
Below the equation, show a visual of the training effect:
LEFT (before training): Scattered dots in 2D - blue, orange, green dots randomly placed
RIGHT (after training): Same dots but now clustered - each material's 3 dots (blue, orange, green) form tight triangles, different materials are pushed apart.
Arrow from LEFT to RIGHT labeled: "Contrastive training pulls matching pairs together, pushes non-matching pairs apart"
═══════════════════════════════════════════════════════════
ADDITIONAL DETAIL PANEL (side panel or bottom):
═══════════════════════════════════════════════════════════
Show the three modality pairs being trained:
Pair 1: Crystal ↔ Text [weight: 1.0] [Blue-Green double arrow]
Pair 2: Crystal ↔ XRD [weight: 1.0] [Blue-Orange double arrow]
Pair 3: XRD ↔ Text [weight: 0.5] [Orange-Green double arrow]
Note: "Crystal is the ANCHOR modality (ImageBind strategy)"
"XRD and Text are never directly compared at full weight"
"This enables cross-modal transfer: XRD↔Text works even though
they're only weakly paired, because both are aligned to Crystal"
═══════════════════════════════════════════════════════════
STYLE:
═══════════════════════════════════════════════════════════
- Publication quality, suitable for NeurIPS/ICML
- Clean white background
- Color scheme: Blue=#2196F3 (crystal), Orange=#FF9800 (XRD), Green=#4CAF50 (text)
- Matrix cells with gradient coloring (green diagonal, red off-diagonal)
- Mathematical notation in clean LaTeX-like font
- Arrows with clear labels
- Before/after scatter plots should be simple but effective
- Overall size: 1800x2400 pixels (portrait)
- Sans-serif fonts throughout
Create a flowchart diagram showing the data pipeline for MatCLIP:
Step 1: "Materials Project API" (database icon) → downloads 10,000 crystal structures with properties
Step 2: Three parallel branches:
- Branch A: Crystal structure → "Build Graph" (nodes=atoms, edges=bonds within 8A) → PyG Data object
- Branch B: Crystal structure → "pymatgen XRD Calculator" → "Gaussian Broadening" → 512-point XRD pattern
- Branch C: Properties (band gap, crystal system, etc.) → "Rule-based Text Builder" → Natural language description
Step 3: All three branches merge into "Paired Dataset: (graph, XRD, text) tuples"
Step 4: "Train/Val/Test Split" (7800/975/976)
Style: Professional flowchart, left-to-right flow, color-coded branches matching the architecture diagram (blue=structure, orange=XRD, green=text). Include sample data at each step (e.g., show a tiny crystal structure, a tiny XRD plot, a text snippet).
Create a diagram explaining how InfoNCE contrastive loss works in MatCLIP:
Show a batch of 4 materials (Material A, B, C, D).
Left side: Show 4 crystal embeddings as colored dots (blue).
Right side: Show 4 text embeddings as colored dots (green).
Draw lines between them:
- SOLID GREEN lines connecting matching pairs (A-crystal to A-text, B-crystal to B-text, etc.) - labeled "Positive pairs (pull together)"
- DASHED RED lines connecting non-matching pairs (A-crystal to B-text, etc.) - labeled "Negative pairs (push apart)"
Below: Show the 4x4 similarity matrix with the diagonal highlighted in green (positive pairs) and off-diagonal in light red (negative pairs). Label it "sim(crystal, text) / temperature".
Below that: Show the equation: L = -0.5 * [CE(S, I) + CE(S^T, I)] where S is the similarity matrix and I is identity.
Style: Clean, educational, suitable for a presentation slide. Use the same color scheme (blue=crystal, green=text, orange=XRD).
Create a professional bar chart comparing MatCLIP training results:
Title: "MatCLIP v1 vs v2: Overfitting Analysis"
Two groups of bars side by side:
Group 1 - "v1 (Overfit)":
- Train Loss: 0.04 (very low bar, red)
- Val Loss: 6.48 (tall bar, red)
- Label: "Train-Val gap: 6.44 (BAD)"
Group 2 - "v2 (Regularized)":
- Train Loss: 6.41 (moderate bar, green)
- Val Loss: 4.77 (slightly lower bar, green)
- Label: "Train-Val gap: 1.64 (HEALTHY)"
Add annotations:
- Arrow pointing to v1 gap: "Overfitting: val loss 33% worse than best"
- Arrow pointing to v2: "Healthy: val loss still improving"
Below the chart, show a table:
| Fix Applied | v1 | v2 |
| Data size | 2,480 | 9,751 |
| Dropout | 0.1 | 0.3 |
| Weight decay | 1e-4 | 5e-4 |
| Label smoothing | None | 0.1 |
| Augmentation | None | XRD noise + text dropout |
Style: Publication quality, clean fonts, color-coded (red=overfit, green=healthy).
Create a t-SNE visualization mockup for MatCLIP's embedding space:
Show a 2D scatter plot with ~200 points in a shared embedding space.
Three types of markers:
- Circles (●) for crystal structure embeddings
- Squares (■) for XRD pattern embeddings
- Triangles (▲) for text description embeddings
Color-code by crystal system:
- Cubic: blue cluster (top-left)
- Hexagonal: orange cluster (top-right)
- Monoclinic: green cluster (bottom-left)
- Orthorhombic: purple cluster (bottom-right)
- Tetragonal: red cluster (center)
Key insight to show: For each material, its circle, square, and triangle should be CLOSE TOGETHER (showing cross-modal alignment), while different materials should be SPREAD APART (showing discriminability).
Add a legend showing both marker shapes (modality) and colors (crystal system).
Title: "MatCLIP Shared Embedding Space (t-SNE)"
Style: Publication quality, white background, clear legend, labeled axes "t-SNE 1" and "t-SNE 2".
Create a single comprehensive overview slide for "MatCLIP: Multi-Modal Contrastive Learning for Materials Science"
Layout (left to right):
SECTION 1 - DATA (leftmost):
Show Materials Project logo → "10K materials" → three icons branching out:
- Crystal lattice icon → "Graph (atoms + bonds)"
- XRD spectrum icon → "512-point pattern"
- Document icon → "Text description"
SECTION 2 - MODEL (center):
Three encoder boxes stacked vertically:
- "CGCNN" (blue)
- "1D ResNet" (orange)
- "MatSciBERT" (green)
Each with an arrow to "Projection Head" then to a central circle "Shared 256-d Space"
Inside the circle: "InfoNCE Loss"
SECTION 3 - CAPABILITIES (rightmost):
Three output capabilities:
- "Cross-Modal Retrieval" with icon (XRD → Crystal Structure)
- "Zero-Shot Classification" with icon (text prompt → material category)
- "Embedding Visualization" with mini t-SNE plot
Bottom banner: "9,751 materials | 17.9M trainable params | TITAN RTX | R@1 = 82.8%"
Style: Modern, clean, suitable for a conference poster or presentation. Use consistent color scheme throughout.
# 1. Setup
cd MatCLIP
python3 -m venv venv && source venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install torch-geometric pymatgen mp-api transformers sentencepiece scikit-learn matplotlib seaborn umap-learn
pip install -e .
# 2. Prepare data (needs MP_API_KEY)
export MP_API_KEY="your_key_here"
python3 scripts/prepare_large.py
# 3. Train (v2 with anti-overfitting)
python3 scripts/train_v2.py
# 4. Evaluate
python3 -m scripts.evaluate --checkpoint checkpoints_v2/best_model.pt
# 5. Run tests
python3 -m pytest tests/ -v