Train from step

[HURIMOZ]

Hi, Iʻm finetuning Unbabelʻs TowerInstruct and I want to know how to train from a step instead of training from scratch again.
Can you give me instructions for the yaml config please?

[francoishernandez]

Not sure what you mean. "finetuning" is the mutually exclusive with "training from scratch".

Training from scratch means you initialize the model with random weights, and go from there.
Finetuning means you load the weights of a pretrained model, and continue training from there.

If you are using the train_from flag to finetune TowerInstruct, then you are not training from scratch. You can check in the logs that it will load the checkpoint at the beginning of training.

Maybe your issue is more about learning rate schedulers? If so, give more details on what you want to achieve.

[HURIMOZ]

Yes I did figure out I need to use train_from but my problem is the model wonʻt continue where it left. The perplexity is soaring, the cross-entropy too. It looks like itʻs not resuming but actually starting from "scratch" again. It starts with a perplexity of about 32,000, which is the size of the vocab. I donʻt get it. Should I leave all hyperparameters as they were? (learning_rate, noam decay, warmup steps, etc).

[francoishernandez]

Share your logs and config.

[HURIMOZ]

Hi François, here is my config:

# General settings
seed: 1234
share_vocab: true
save_data: "./finetune/Dec31TowerInstruct-7b-v0.2"
src_vocab: "./models/TowerInstruct-7b-v0.2/vocab.txt"
src_vocab_size: 32000
overwrite: true  # Keeping consistent with original
report_every: 10

# datasets
data:
    enty_train: 
        path_src: "./processed_data/en-tah/src-tgt-prompts.enty"
        weight: 1
    enty_valid:
        path_src: "./processed_data/en-tah/src-tgt-prompts-val.enty"

    frty_train:
        path_src: "./processed_data/fr-ty/src-tgt-prompts.frty"
        weight: 1
    frty_valid:
        path_src: "./processed_data/fr-ty/src-tgt-prompts-val.frty"

    enmi_train:
        path_src: "./processed_data/en-mri/src-tgt-prompts.enmi"
        weight: 1
    enmi_valid:
        path_src: "./processed_data/en-mri/src-tgt-prompts-val.enmi"

    ento_train:
        path_src: "./processed_data/en-ton/src-tgt-prompts.enton"
        weight: 1
    ento_valid:
        path_src: "./processed_data/en-ton/src-tgt-prompts-val.enton"

    enhaw_train:
        path_src: "./processed_data/en-haw/src-tgt-prompts.enhaw"
        weight: 1
    enhaw_valid:
        path_src: "./processed_data/en-haw/src-tgt-prompts-val.enhaw"

skip_empty_level: silent

transforms: ['huggingface_tokenize']
transforms_configs:
  huggingface_tokenize:
    huggingface_model: "Unbabel/TowerInstruct-7B-v0.2"
    max_length: 512

training:
    # GPU dispatching
    world_size: 1
    gpu_ranks: [0]
    
    # Dropout settings
    dropout_steps: [0]
    dropout: [0.03]
    attention_dropout: [0.03]
    
    # Batching - optimized for memory and throughput
    bucket_size: 4096  # Reduced from 8192
    num_workers: 16    # Reduced from 48
    batch_type: "tokens"
    batch_size: 256    # Reduced from 512
    valid_batch_size: 256
    batch_size_multiple: 1
    
    # Optimization
    compute_dtype: "fp16"
    optim: "adam"
    learning_rate: 0.5
    warmup_steps: 200
    decay_method: "noam"
    
    # Quantization
    quant_layers: ['gate_up_proj', 'down_proj', 'up_proj', 'linear_values', 'linear_query', 'linear_keys', 'final_linear']
    quant_type: "bnb_NF4"
    
    # LoRA settings
    lora_layers: ['linear_values', 'linear_query', 'linear_keys', 'final_linear']
    lora_rank: 8
    lora_dropout: 0.05
    lora_alpha: 16
    lora_embedding: false
    
    # Training duration and checkpointing
    train_steps: 3000
    valid_steps: 100
    save_checkpoint_steps: 100
    keep_checkpoint: 10
    
    # Model paths
    reset_optim: "none"  # Don't reset optimizer state
    model_path: "./finetune/Jan01TowerInstruct-7b-v0.2"
    train_from: "./models/TowerInstruct-7b-v0.2"

# Tensorboard logging
tensorboard: true
tensorboard_log_dir: "logs"
log_file: "logs/training.log"

And hereʻs a sample of the training logs:

[2025-01-01 00:27:40,362 INFO] Loading checkpoint from ./finetune/Dec31TowerInstruct-7b-v0.2/step_1300
[2025-01-01 00:27:40,521 WARNING] You have a CUDA device, should run with -gpu_ranks
[2025-01-01 00:27:40,523 INFO] Option: model_path, value: ./models/TowerInstruct-7b-v0.2, overriding model: ./finetune/Dec31TowerInstruct-7b-v0.2
[2025-01-01 00:27:40,523 INFO] Option: train_from, value: ./finetune/Dec31TowerInstruct-7b-v0.2/step_1300, overriding model: ./models/TowerInstruct-7b-v0.2
[2025-01-01 00:27:40,524 INFO] Default transforms (might be overridden downstream): ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for enty_train data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for enty_valid data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for frty_train data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for frty_valid data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for enmi_train data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for enmi_valid data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for ento_train data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for ento_valid data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,524 INFO] Missing transforms field for enhaw_train data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,525 INFO] Missing transforms field for enhaw_valid data, set to default: ['huggingface_tokenize'].
[2025-01-01 00:27:40,525 INFO] Parsed 10 corpora from -data.
[2025-01-01 00:27:40,525 INFO] Get special vocabs from Transforms: {'src': [], 'tgt': []}.
[2025-01-01 00:27:40,966 INFO] Initialized tokenizers from HF model: Unbabel/TowerInstruct-7B-v0.2
[2025-01-01 00:27:40,966 INFO] Transforms applied: ['huggingface_tokenize']
[2025-01-01 00:27:40,966 INFO] Keeping checkpoint vocabulary
[2025-01-01 00:27:41,000 INFO] The first 10 tokens of the vocabs are:['<unk>', '<s>', '</s>', '<0x00>', '<0x01>', '<0x02>', '<0x03>', '<0x04>', '<0x05>', '<0x06>']
[2025-01-01 00:27:41,001 INFO] The decoder start token is: <s>
[2025-01-01 00:27:41,001 INFO] bos_token token is: <s> id: [1]
[2025-01-01 00:27:41,001 INFO] eos_token token is: <|im_end|> id: [32005]
[2025-01-01 00:27:41,001 INFO] pad_token token is: <PAD> id: [32004]
[2025-01-01 00:27:41,001 INFO] unk_token token is: <unk> id: [0]
[2025-01-01 00:27:41,001 INFO] Building model...
[2025-01-01 00:27:41,280 INFO] bnb_NF4 compression of layer ['gate_up_proj', 'down_proj', 'up_proj']
[2025-01-01 00:27:41,403 INFO] Adding LoRa layers for linear_values quant bnb_NF4
[2025-01-01 00:27:41,421 INFO] Adding LoRa layers for linear_query quant bnb_NF4
[2025-01-01 00:27:41,439 INFO] Adding LoRa layers for linear_keys quant bnb_NF4
[2025-01-01 00:27:41,462 INFO] Adding LoRa layers for final_linear quant bnb_NF4
[2025-01-01 00:27:41,495 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.input_layernorm.weight
[2025-01-01 00:27:41,495 INFO] Switching model to float32 for amp/apex_amp
[2025-01-01 00:27:41,495 INFO] Non quantized layer compute is torch.float16
[2025-01-01 00:27:41,683 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.self_attn.linear_keys.weight
[2025-01-01 00:27:41,776 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.self_attn.linear_values.weight
[2025-01-01 00:27:41,808 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.self_attn.linear_query.weight
[2025-01-01 00:27:41,840 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.self_attn.final_linear.weight
[2025-01-01 00:27:41,872 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.post_attention_layernorm.weight
[2025-01-01 00:27:41,872 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.mlp.gate_up_proj.weight
[2025-01-01 00:27:41,955 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.mlp.down_proj.weight
[2025-01-01 00:27:42,037 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.0.mlp.up_proj.weight
[2025-01-01 00:27:42,120 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.input_layernorm.weight
[2025-01-01 00:27:42,121 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.self_attn.linear_keys.weight
[2025-01-01 00:27:42,153 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.self_attn.linear_values.weight
[2025-01-01 00:27:42,185 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.self_attn.linear_query.weight
[2025-01-01 00:27:42,217 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.self_attn.final_linear.weight
[2025-01-01 00:27:42,249 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.post_attention_layernorm.weight
[2025-01-01 00:27:42,249 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.mlp.gate_up_proj.weight
[2025-01-01 00:27:42,333 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.mlp.down_proj.weight
[2025-01-01 00:27:42,416 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.1.mlp.up_proj.weight
[2025-01-01 00:27:42,499 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.input_layernorm.weight
[2025-01-01 00:27:42,499 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.self_attn.linear_keys.weight
[2025-01-01 00:27:42,531 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.self_attn.linear_values.weight
[2025-01-01 00:27:42,564 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.self_attn.linear_query.weight
[2025-01-01 00:27:42,596 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.self_attn.final_linear.weight
[2025-01-01 00:27:42,628 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.post_attention_layernorm.weight
[2025-01-01 00:27:42,628 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.mlp.gate_up_proj.weight
[2025-01-01 00:27:42,711 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.mlp.down_proj.weight
[2025-01-01 00:27:42,794 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.2.mlp.up_proj.weight
[2025-01-01 00:27:42,877 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.input_layernorm.weight
[2025-01-01 00:27:42,877 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.self_attn.linear_keys.weight
[2025-01-01 00:27:42,910 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.self_attn.linear_values.weight
[2025-01-01 00:27:42,942 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.self_attn.linear_query.weight
[2025-01-01 00:27:42,974 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.self_attn.final_linear.weight
[2025-01-01 00:27:43,006 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.post_attention_layernorm.weight
[2025-01-01 00:27:43,007 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.mlp.gate_up_proj.weight
[2025-01-01 00:27:43,090 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.mlp.down_proj.weight
[2025-01-01 00:27:43,173 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.3.mlp.up_proj.weight
[2025-01-01 00:27:43,256 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.input_layernorm.weight
[2025-01-01 00:27:43,256 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.self_attn.linear_keys.weight
[2025-01-01 00:27:43,288 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.self_attn.linear_values.weight
[2025-01-01 00:27:43,321 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.self_attn.linear_query.weight
[2025-01-01 00:27:43,353 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.self_attn.final_linear.weight
[2025-01-01 00:27:43,385 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.post_attention_layernorm.weight
[2025-01-01 00:27:43,385 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.mlp.gate_up_proj.weight
[2025-01-01 00:27:43,468 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.mlp.down_proj.weight
[2025-01-01 00:27:43,551 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.4.mlp.up_proj.weight
[2025-01-01 00:27:43,634 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.input_layernorm.weight
[2025-01-01 00:27:43,635 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.self_attn.linear_keys.weight
[2025-01-01 00:27:43,667 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.self_attn.linear_values.weight
[2025-01-01 00:27:43,699 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.self_attn.linear_query.weight
[2025-01-01 00:27:43,731 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.self_attn.final_linear.weight
[2025-01-01 00:27:43,763 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.post_attention_layernorm.weight
[2025-01-01 00:27:43,764 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.mlp.gate_up_proj.weight
[2025-01-01 00:27:43,847 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.mlp.down_proj.weight
[2025-01-01 00:27:43,931 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.5.mlp.up_proj.weight
[2025-01-01 00:27:44,014 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.input_layernorm.weight
[2025-01-01 00:27:44,015 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.self_attn.linear_keys.weight
[2025-01-01 00:27:44,047 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.self_attn.linear_values.weight
[2025-01-01 00:27:44,080 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.self_attn.linear_query.weight
[2025-01-01 00:27:44,112 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.self_attn.final_linear.weight
[2025-01-01 00:27:44,145 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.post_attention_layernorm.weight
[2025-01-01 00:27:44,145 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.mlp.gate_up_proj.weight
[2025-01-01 00:27:44,228 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.mlp.down_proj.weight
[2025-01-01 00:27:44,312 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.6.mlp.up_proj.weight
[2025-01-01 00:27:44,395 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.input_layernorm.weight
[2025-01-01 00:27:44,395 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.self_attn.linear_keys.weight
[2025-01-01 00:27:44,428 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.self_attn.linear_values.weight
[2025-01-01 00:27:44,460 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.self_attn.linear_query.weight
[2025-01-01 00:27:44,492 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.self_attn.final_linear.weight
[2025-01-01 00:27:44,524 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.post_attention_layernorm.weight
[2025-01-01 00:27:44,525 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.mlp.gate_up_proj.weight
[2025-01-01 00:27:44,607 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.mlp.down_proj.weight
[2025-01-01 00:27:44,690 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.7.mlp.up_proj.weight
[2025-01-01 00:27:44,778 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.input_layernorm.weight
[2025-01-01 00:27:44,779 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.self_attn.linear_keys.weight
[2025-01-01 00:27:44,811 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.self_attn.linear_values.weight
[2025-01-01 00:27:44,844 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.self_attn.linear_query.weight
[2025-01-01 00:27:44,876 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.self_attn.final_linear.weight
[2025-01-01 00:27:44,908 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.post_attention_layernorm.weight
[2025-01-01 00:27:44,908 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.mlp.gate_up_proj.weight
[2025-01-01 00:27:44,991 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.mlp.down_proj.weight
[2025-01-01 00:27:45,074 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.8.mlp.up_proj.weight
[2025-01-01 00:27:45,157 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.input_layernorm.weight
[2025-01-01 00:27:45,158 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.self_attn.linear_keys.weight
[2025-01-01 00:27:45,190 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.self_attn.linear_values.weight
[2025-01-01 00:27:45,222 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.self_attn.linear_query.weight
[2025-01-01 00:27:45,255 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.self_attn.final_linear.weight
[2025-01-01 00:27:45,287 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.post_attention_layernorm.weight
[2025-01-01 00:27:45,287 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.mlp.gate_up_proj.weight
[2025-01-01 00:27:45,370 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.mlp.down_proj.weight
[2025-01-01 00:27:45,454 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.9.mlp.up_proj.weight
[2025-01-01 00:27:45,538 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.input_layernorm.weight
[2025-01-01 00:27:45,538 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.self_attn.linear_keys.weight
[2025-01-01 00:27:45,571 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.self_attn.linear_values.weight
[2025-01-01 00:27:45,603 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.self_attn.linear_query.weight
[2025-01-01 00:27:45,636 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.self_attn.final_linear.weight
[2025-01-01 00:27:45,668 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.post_attention_layernorm.weight
[2025-01-01 00:27:45,668 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.mlp.gate_up_proj.weight
[2025-01-01 00:27:45,751 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.mlp.down_proj.weight
[2025-01-01 00:27:45,835 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.10.mlp.up_proj.weight
[2025-01-01 00:27:45,919 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.input_layernorm.weight
[2025-01-01 00:27:45,919 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.self_attn.linear_keys.weight
[2025-01-01 00:27:45,952 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.self_attn.linear_values.weight
[2025-01-01 00:27:45,984 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.self_attn.linear_query.weight
[2025-01-01 00:27:46,016 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.self_attn.final_linear.weight
[2025-01-01 00:27:46,049 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.post_attention_layernorm.weight
[2025-01-01 00:27:46,049 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.mlp.gate_up_proj.weight
[2025-01-01 00:27:46,132 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.mlp.down_proj.weight
[2025-01-01 00:27:46,215 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.11.mlp.up_proj.weight
[2025-01-01 00:27:46,299 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.input_layernorm.weight
[2025-01-01 00:27:46,299 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.self_attn.linear_keys.weight
[2025-01-01 00:27:46,331 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.self_attn.linear_values.weight
[2025-01-01 00:27:46,364 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.self_attn.linear_query.weight
[2025-01-01 00:27:46,396 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.self_attn.final_linear.weight
[2025-01-01 00:27:46,428 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.post_attention_layernorm.weight
[2025-01-01 00:27:46,429 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.mlp.gate_up_proj.weight
[2025-01-01 00:27:46,512 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.mlp.down_proj.weight
[2025-01-01 00:27:46,595 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.12.mlp.up_proj.weight
[2025-01-01 00:27:46,678 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.input_layernorm.weight
[2025-01-01 00:27:46,678 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.self_attn.linear_keys.weight
[2025-01-01 00:27:46,710 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.self_attn.linear_values.weight
[2025-01-01 00:27:46,743 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.self_attn.linear_query.weight
[2025-01-01 00:27:46,775 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.self_attn.final_linear.weight
[2025-01-01 00:27:46,807 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.post_attention_layernorm.weight
[2025-01-01 00:27:46,807 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.mlp.gate_up_proj.weight
[2025-01-01 00:27:46,891 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.mlp.down_proj.weight
[2025-01-01 00:27:46,974 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.13.mlp.up_proj.weight
[2025-01-01 00:27:47,057 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.input_layernorm.weight
[2025-01-01 00:27:47,057 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.self_attn.linear_keys.weight
[2025-01-01 00:27:47,090 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.self_attn.linear_values.weight
[2025-01-01 00:27:47,122 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.self_attn.linear_query.weight
[2025-01-01 00:27:47,158 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.self_attn.final_linear.weight
[2025-01-01 00:27:47,191 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.post_attention_layernorm.weight
[2025-01-01 00:27:47,191 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.mlp.gate_up_proj.weight
[2025-01-01 00:27:47,275 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.mlp.down_proj.weight
[2025-01-01 00:27:47,358 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.14.mlp.up_proj.weight
[2025-01-01 00:27:47,442 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.input_layernorm.weight
[2025-01-01 00:27:47,442 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.self_attn.linear_keys.weight
[2025-01-01 00:27:47,475 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.self_attn.linear_values.weight
[2025-01-01 00:27:47,507 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.self_attn.linear_query.weight
[2025-01-01 00:27:47,540 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.self_attn.final_linear.weight
[2025-01-01 00:27:47,572 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.post_attention_layernorm.weight
[2025-01-01 00:27:47,572 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.mlp.gate_up_proj.weight
[2025-01-01 00:27:47,655 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.mlp.down_proj.weight
[2025-01-01 00:27:47,739 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.15.mlp.up_proj.weight
[2025-01-01 00:27:47,822 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.input_layernorm.weight
[2025-01-01 00:27:47,823 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.self_attn.linear_keys.weight
[2025-01-01 00:27:47,855 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.self_attn.linear_values.weight
[2025-01-01 00:27:47,887 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.self_attn.linear_query.weight
[2025-01-01 00:27:47,920 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.self_attn.final_linear.weight
[2025-01-01 00:27:47,952 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.post_attention_layernorm.weight
[2025-01-01 00:27:47,952 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.mlp.gate_up_proj.weight
[2025-01-01 00:27:48,036 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.mlp.down_proj.weight
[2025-01-01 00:27:48,119 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.16.mlp.up_proj.weight
[2025-01-01 00:27:48,203 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.input_layernorm.weight
[2025-01-01 00:27:48,203 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.self_attn.linear_keys.weight
[2025-01-01 00:27:48,235 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.self_attn.linear_values.weight
[2025-01-01 00:27:48,268 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.self_attn.linear_query.weight
[2025-01-01 00:27:48,300 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.self_attn.final_linear.weight
[2025-01-01 00:27:48,333 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.post_attention_layernorm.weight
[2025-01-01 00:27:48,333 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.mlp.gate_up_proj.weight
[2025-01-01 00:27:48,416 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.mlp.down_proj.weight
[2025-01-01 00:27:48,500 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.17.mlp.up_proj.weight
[2025-01-01 00:27:48,583 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.input_layernorm.weight
[2025-01-01 00:27:48,583 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.self_attn.linear_keys.weight
[2025-01-01 00:27:48,616 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.self_attn.linear_values.weight
[2025-01-01 00:27:48,648 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.self_attn.linear_query.weight
[2025-01-01 00:27:48,680 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.self_attn.final_linear.weight
[2025-01-01 00:27:48,713 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.post_attention_layernorm.weight
[2025-01-01 00:27:48,713 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.mlp.gate_up_proj.weight
[2025-01-01 00:27:48,796 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.mlp.down_proj.weight
[2025-01-01 00:27:48,880 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.18.mlp.up_proj.weight
[2025-01-01 00:27:48,964 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.input_layernorm.weight
[2025-01-01 00:27:48,964 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.self_attn.linear_keys.weight
[2025-01-01 00:27:48,997 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.self_attn.linear_values.weight
[2025-01-01 00:27:49,029 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.self_attn.linear_query.weight
[2025-01-01 00:27:49,061 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.self_attn.final_linear.weight
[2025-01-01 00:27:49,094 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.post_attention_layernorm.weight
[2025-01-01 00:27:49,094 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.mlp.gate_up_proj.weight
[2025-01-01 00:27:49,177 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.mlp.down_proj.weight
[2025-01-01 00:27:49,260 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.19.mlp.up_proj.weight
[2025-01-01 00:27:49,343 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.input_layernorm.weight
[2025-01-01 00:27:49,343 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.self_attn.linear_keys.weight
[2025-01-01 00:27:49,376 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.self_attn.linear_values.weight
[2025-01-01 00:27:49,409 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.self_attn.linear_query.weight
[2025-01-01 00:27:49,442 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.self_attn.final_linear.weight
[2025-01-01 00:27:49,474 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.post_attention_layernorm.weight
[2025-01-01 00:27:49,474 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.mlp.gate_up_proj.weight
[2025-01-01 00:27:49,558 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.mlp.down_proj.weight
[2025-01-01 00:27:49,641 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.20.mlp.up_proj.weight
[2025-01-01 00:27:49,724 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.input_layernorm.weight
[2025-01-01 00:27:49,725 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.self_attn.linear_keys.weight
[2025-01-01 00:27:49,757 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.self_attn.linear_values.weight
[2025-01-01 00:27:49,790 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.self_attn.linear_query.weight
[2025-01-01 00:27:49,822 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.self_attn.final_linear.weight
[2025-01-01 00:27:49,854 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.post_attention_layernorm.weight
[2025-01-01 00:27:49,855 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.mlp.gate_up_proj.weight
[2025-01-01 00:27:49,938 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.mlp.down_proj.weight
[2025-01-01 00:27:50,021 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.21.mlp.up_proj.weight
[2025-01-01 00:27:50,105 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.input_layernorm.weight
[2025-01-01 00:27:50,105 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.self_attn.linear_keys.weight
[2025-01-01 00:27:50,137 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.self_attn.linear_values.weight
[2025-01-01 00:27:50,170 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.self_attn.linear_query.weight
[2025-01-01 00:27:50,202 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.self_attn.final_linear.weight
[2025-01-01 00:27:50,235 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.post_attention_layernorm.weight
[2025-01-01 00:27:50,235 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.mlp.gate_up_proj.weight
[2025-01-01 00:27:50,318 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.mlp.down_proj.weight
[2025-01-01 00:27:50,402 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.22.mlp.up_proj.weight
[2025-01-01 00:27:50,486 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.input_layernorm.weight
[2025-01-01 00:27:50,486 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.self_attn.linear_keys.weight
[2025-01-01 00:27:50,519 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.self_attn.linear_values.weight
[2025-01-01 00:27:50,552 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.self_attn.linear_query.weight
[2025-01-01 00:27:50,584 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.self_attn.final_linear.weight
[2025-01-01 00:27:50,616 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.post_attention_layernorm.weight
[2025-01-01 00:27:50,617 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.mlp.gate_up_proj.weight
[2025-01-01 00:27:50,700 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.mlp.down_proj.weight
[2025-01-01 00:27:50,783 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.23.mlp.up_proj.weight
[2025-01-01 00:27:50,867 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.input_layernorm.weight
[2025-01-01 00:27:50,867 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.self_attn.linear_keys.weight
[2025-01-01 00:27:50,900 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.self_attn.linear_values.weight
[2025-01-01 00:27:50,932 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.self_attn.linear_query.weight
[2025-01-01 00:27:50,965 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.self_attn.final_linear.weight
[2025-01-01 00:27:50,997 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.post_attention_layernorm.weight
[2025-01-01 00:27:50,997 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.mlp.gate_up_proj.weight
[2025-01-01 00:27:51,080 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.mlp.down_proj.weight
[2025-01-01 00:27:51,164 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.24.mlp.up_proj.weight
[2025-01-01 00:27:51,248 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.input_layernorm.weight
[2025-01-01 00:27:51,248 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.self_attn.linear_keys.weight
[2025-01-01 00:27:51,280 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.self_attn.linear_values.weight
[2025-01-01 00:27:51,313 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.self_attn.linear_query.weight
[2025-01-01 00:27:51,345 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.self_attn.final_linear.weight
[2025-01-01 00:27:51,378 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.post_attention_layernorm.weight
[2025-01-01 00:27:51,378 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.mlp.gate_up_proj.weight
[2025-01-01 00:27:51,462 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.mlp.down_proj.weight
[2025-01-01 00:27:51,545 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.25.mlp.up_proj.weight
[2025-01-01 00:27:51,628 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.input_layernorm.weight
[2025-01-01 00:27:51,628 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.self_attn.linear_keys.weight
[2025-01-01 00:27:51,661 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.self_attn.linear_values.weight
[2025-01-01 00:27:51,694 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.self_attn.linear_query.weight
[2025-01-01 00:27:51,726 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.self_attn.final_linear.weight
[2025-01-01 00:27:51,759 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.post_attention_layernorm.weight
[2025-01-01 00:27:51,759 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.mlp.gate_up_proj.weight
[2025-01-01 00:27:51,842 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.mlp.down_proj.weight
[2025-01-01 00:27:51,926 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.26.mlp.up_proj.weight
[2025-01-01 00:27:52,009 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.input_layernorm.weight
[2025-01-01 00:27:52,010 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.self_attn.linear_keys.weight
[2025-01-01 00:27:52,042 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.self_attn.linear_values.weight
[2025-01-01 00:27:52,075 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.self_attn.linear_query.weight
[2025-01-01 00:27:52,107 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.self_attn.final_linear.weight
[2025-01-01 00:27:52,140 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.post_attention_layernorm.weight
[2025-01-01 00:27:52,140 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.mlp.gate_up_proj.weight
[2025-01-01 00:27:52,223 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.mlp.down_proj.weight
[2025-01-01 00:27:52,307 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.27.mlp.up_proj.weight
[2025-01-01 00:27:52,391 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.input_layernorm.weight
[2025-01-01 00:27:52,392 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.self_attn.linear_keys.weight
[2025-01-01 00:27:52,424 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.self_attn.linear_values.weight
[2025-01-01 00:27:52,456 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.self_attn.linear_query.weight
[2025-01-01 00:27:52,489 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.self_attn.final_linear.weight
[2025-01-01 00:27:52,521 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.post_attention_layernorm.weight
[2025-01-01 00:27:52,521 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.mlp.gate_up_proj.weight
[2025-01-01 00:27:52,605 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.mlp.down_proj.weight
[2025-01-01 00:27:52,689 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.28.mlp.up_proj.weight
[2025-01-01 00:27:52,772 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.input_layernorm.weight
[2025-01-01 00:27:52,773 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.self_attn.linear_keys.weight
[2025-01-01 00:27:52,805 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.self_attn.linear_values.weight
[2025-01-01 00:27:52,837 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.self_attn.linear_query.weight
[2025-01-01 00:27:52,870 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.self_attn.final_linear.weight
[2025-01-01 00:27:52,902 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.post_attention_layernorm.weight
[2025-01-01 00:27:52,903 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.mlp.gate_up_proj.weight
[2025-01-01 00:27:52,986 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.mlp.down_proj.weight
[2025-01-01 00:27:53,069 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.29.mlp.up_proj.weight
[2025-01-01 00:27:53,153 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.input_layernorm.weight
[2025-01-01 00:27:53,153 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.self_attn.linear_keys.weight
[2025-01-01 00:27:53,186 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.self_attn.linear_values.weight
[2025-01-01 00:27:53,218 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.self_attn.linear_query.weight
[2025-01-01 00:27:53,251 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.self_attn.final_linear.weight
[2025-01-01 00:27:53,284 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.post_attention_layernorm.weight
[2025-01-01 00:27:53,284 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.mlp.gate_up_proj.weight
[2025-01-01 00:27:53,368 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.mlp.down_proj.weight
[2025-01-01 00:27:53,452 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.30.mlp.up_proj.weight
[2025-01-01 00:27:53,536 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.input_layernorm.weight
[2025-01-01 00:27:53,536 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.self_attn.linear_keys.weight
[2025-01-01 00:27:53,569 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.self_attn.linear_values.weight
[2025-01-01 00:27:53,601 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.self_attn.linear_query.weight
[2025-01-01 00:27:53,634 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.self_attn.final_linear.weight
[2025-01-01 00:27:53,666 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.post_attention_layernorm.weight
[2025-01-01 00:27:53,666 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.mlp.gate_up_proj.weight
[2025-01-01 00:27:53,750 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.mlp.down_proj.weight
[2025-01-01 00:27:53,833 INFO] Missing key in safetensors checkpoint: decoder.transformer_layers.31.mlp.up_proj.weight
[2025-01-01 00:27:53,917 INFO] Missing key in safetensors checkpoint: decoder.layer_norm.weight
[2025-01-01 00:27:53,917 INFO] Missing key in safetensors checkpoint: tgt_emb.embeddings.weight
[2025-01-01 00:27:54,149 INFO] Missing key in safetensors checkpoint: generator.weight
[2025-01-01 00:27:54,381 INFO] DecoderModel(
  (decoder): TransformerLMDecoder(
    (rope): RotaryPosition()
    (transformer_layers): ModuleList(
      (0-31): 32 x TransformerLMDecoderLayer(
        (input_layernorm): RMSNorm()
        (self_attn): SelfMHA(
          (linear_keys): QLoraLinear_cls(
            in_features=4096, out_features=4096, bias=False
            (lora_dropout): Dropout(p=0.05, inplace=False)
          )
          (linear_values): QLoraLinear_cls(
            in_features=4096, out_features=4096, bias=False
            (lora_dropout): Dropout(p=0.05, inplace=False)
          )
          (linear_query): QLoraLinear_cls(
            in_features=4096, out_features=4096, bias=False
            (lora_dropout): Dropout(p=0.05, inplace=False)
          )
          (softmax): Softmax(dim=-1)
          (dropout): Dropout(p=0.03, inplace=False)
          (final_linear): QLoraLinear_cls(
            in_features=4096, out_features=4096, bias=False
            (lora_dropout): Dropout(p=0.05, inplace=False)
          )
        )
        (dropout): Dropout(p=0.03, inplace=False)
        (post_attention_layernorm): RMSNorm()
        (mlp): MLP(
          (gate_up_proj): Linear4bit(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear4bit(in_features=11008, out_features=4096, bias=False)
          (dropout_1): Dropout(p=0.03, inplace=False)
          (dropout_2): Dropout(p=0.03, inplace=False)
          (up_proj): Linear4bit(in_features=4096, out_features=11008, bias=False)
        )
      )
    )
    (layer_norm): RMSNorm()
  )
  (tgt_emb): Embeddings(
    (embeddings): Embedding(32007, 4096, padding_idx=32004)
    (dropout): Dropout(p=0.03, inplace=False)
  )
  (generator): Linear(in_features=4096, out_features=32007, bias=False)
)
[2025-01-01 00:27:54,388 INFO] embeddings: 131100672
[2025-01-01 00:27:54,389 INFO] encoder: 0
[2025-01-01 00:27:54,389 INFO] decoder: 3246657536
[2025-01-01 00:27:54,389 INFO] generator: 131100672
[2025-01-01 00:27:54,389 INFO] other: 0
[2025-01-01 00:27:54,389 INFO] * number of parameters: 3508858880
[2025-01-01 00:27:54,389 INFO] Trainable parameters = {'torch.float32': 8388608}
[2025-01-01 00:27:54,389 INFO] Non trainable parameters = {'torch.float32': 262467584, 'torch.uint8': 3238002688}
[2025-01-01 00:27:54,389 INFO]  * src vocab size = 32007
[2025-01-01 00:27:54,389 INFO]  * tgt vocab size = 32007
[2025-01-01 00:27:56,537 INFO] Starting training on GPU: [0]
[2025-01-01 00:27:56,537 INFO] Start training loop without validation...
[2025-01-01 00:27:56,537 INFO] Scoring with: None
[2025-01-01 00:34:54,450 INFO] Step 1310/ 3000; acc: 0.0; ppl: 32007.01; xent: 10.37; aux: 0.000; lr: 3.45e-04; sents:     619; bsz:  238/ 238/ 2; 183/183 tok/s;    418 sec;

[francoishernandez]

Ok, your issue is related to the LoRa finetuning technique. This technique allows to finetune bigger models with limited VRAM by only finetuning part of the weights. But it requires some additional steps afterwards.
Check the lora_weights tool -- https://github.com/eole-nlp/eole/blob/main/eole/bin/model/lora_weights.py

The easiest is probably to merge your finetuned weights with the original model before continuing the training. I don't think we have an easier way right now. (Main idea is that saving the full model each time you save a checkpoint is not really efficient, so we only save the LoRa weights, and the merging happens later at the user's discretion.)

[HURIMOZ]

Ah now I understand. I merged the LoRa weights with the base model for inference but didnʻt realize I have to do that to resume the training as well.
Thank you François and Bonne annee.