The relevant code that caused the error is in the Controllable Text Generation section, after the model trained for 6 epochs and started evaluating, it raised a KeyError: 'eval_loss'

[Markkk111]

You're welcome! I'm glad to assist you with this question！

1. The error message is as follows：
   huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
   To disable this warning, you can either:
   - Avoid using tokenizers before the fork if possible
   - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
   huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
   To disable this warning, you can either:
   - Avoid using tokenizers before the fork if possible
   - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
   huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
   To disable this warning, you can either:
   - Avoid using tokenizers before the fork if possible
   - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
   huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
   To disable this warning, you can either:
   - Avoid using tokenizers before the fork if possible
   - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
   {'loss': 0.0185, 'learning_rate': 7.973102785782901e-06, 'epoch': 5.04}
   {'loss': 0.0185, 'learning_rate': 6.9724623759205894e-06, 'epoch': 5.16}
   {'loss': 0.0188, 'learning_rate': 5.971821966058277e-06, 'epoch': 5.28}
   {'loss': 0.0178, 'learning_rate': 4.971181556195966e-06, 'epoch': 5.4}
   wandb: Network error (ReadTimeout), entering retry loop.
   {'loss': 0.0183, 'learning_rate': 3.970541146333654e-06, 'epoch': 5.52}
   {'loss': 0.018, 'learning_rate': 2.9699007364713415e-06, 'epoch': 5.64}
   {'loss': 0.0179, 'learning_rate': 1.96926032660903e-06, 'epoch': 5.76}
   {'loss': 0.0174, 'learning_rate': 9.68619916746718e-07, 'epoch': 5.88}
   [INFO|trainer.py:1901] 2023-04-19 17:43:28,689 >>

Training completed. Do not forget to share your model on huggingface.co/models =)

{'train_runtime': 1749.401, 'train_samples_per_second': 142.815, 'train_steps_per_second': 14.281, 'train_loss': 0.023130248267032992, 'epoch': 6.0}
[INFO|trainer.py:2709] 2023-04-19 17:43:28,693 >> Saving model checkpoint to classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None
[INFO|configuration_utils.py:453] 2023-04-19 17:43:28,694 >> Configuration saved in classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None/config.json
[INFO|modeling_utils.py:1704] 2023-04-19 17:43:29,841 >> Model weights saved in classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None/pytorch_model.bin
***** train metrics *****
epoch = 6.0
train_loss = 0.0231
train_runtime = 0:29:09.40
train_samples = 41640
train_samples_per_second = 142.815
train_steps_per_second = 14.281
04/19/2023 17:43:29 - INFO - main - *** Evaluate ***
[INFO|trainer.py:710] 2023-04-19 17:43:29,848 >> The following columns in the evaluation set don't have a corresponding argument in Classifier_Tree.forward and have been ignored: chart_lst. If chart_lst are not expected by Classifier_Tree.forward, you can safely ignore this message.
[INFO|trainer.py:2964] 2023-04-19 17:43:29,850 >> ***** Running Evaluation *****
[INFO|trainer.py:2966] 2023-04-19 17:43:29,850 >> Num examples = 421
[INFO|trainer.py:2969] 2023-04-19 17:43:29,851 >> Batch size = 10
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
{'eval_runtime': 1.4868, 'eval_samples_per_second': 283.16, 'eval_steps_per_second': 28.921, 'epoch': 6.0}
Traceback (most recent call last):
File "/home/name/diffusion-LM/transformers/examples/pytorch/language-modeling/run_clm.py", line 1704, in
main()
File "/home/name/diffusion-LM/transformers/examples/pytorch/language-modeling/run_clm.py", line 1675, in main
perplexity = math.exp(metrics["eval_loss"])
KeyError: 'eval_loss'
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
Exception ignored in atexit callback: <function _Manager._atexit_setup.. at 0x7f2f280f1fc0>
Traceback (most recent call last):
File "/home/name/anaconda3/lib/python3.10/site-packages/wandb/sdk/wandb_manager.py", line 166, in
self._atexit_lambda = lambda: self._atexit_teardown()
File "/home/name/anaconda3/lib/python3.10/site-packages/wandb/sdk/wandb_manager.py", line 175, in _atexit_teardown
self._teardown(exit_code)
File "/home/name/anaconda3/lib/python3.10/site-packages/wandb/sdk/wandb_manager.py", line 186, in _teardown
result = self._service.join()
File "/home/name/anaconda3/lib/python3.10/site-packages/wandb/sdk/service/service.py", line 216, in join
ret = self._internal_proc.wait()
File "/home/name/anaconda3/lib/python3.10/subprocess.py", line 1204, in wait
return self._wait(timeout=timeout)
File "/home/name/anaconda3/lib/python3.10/subprocess.py", line 1938, in _wait
(pid, sts) = self._try_wait(0)
File "/home/name/anaconda3/lib/python3.10/subprocess.py", line 1896, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt:
(diffusion-LM) name@taizun-SYS-4029GP-TRT:/diffusion-LM$ wandb: - 0.010 MB of 0.010 MB uploaded (0.(diffusion-LM) name@taizun-SYS-4029GP-TRT:/diffusion-LM$ wandb: / 0.010 MB of 0.010 MB uploaded (0.wandb: \ 0.010 MB of 0.010 MB uploaded (0.000 MB deduped)

2. The relevant code that caused the error is as follows：

   Training

   if training_args.do_train:
   checkpoint = None
   if training_args.resume_from_checkpoint is not None:
   checkpoint = training_args.resume_from_checkpoint
   elif last_checkpoint is not None:
   checkpoint = last_checkpoint
   train_result = trainer.train(resume_from_checkpoint=checkpoint)
   trainer.save_model() # Saves the tokenizer too for easy upload

    metrics = train_result.metrics
   
   
    max_train_samples = (
        data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
    )
    metrics["train_samples"] = min(max_train_samples, len(train_dataset))
   
    trainer.log_metrics("train", metrics)
    trainer.save_metrics("train", metrics)
    trainer.save_state()
   

   Evaluation

   if training_args.do_eval:
   logger.info("*** Evaluate ***")

    metrics = trainer.evaluate()
   
    max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
    metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
    try:
        perplexity = math.exp(metrics["eval_loss"])
    except OverflowError:
        perplexity = float("inf")
    metrics["perplexity"] = perplexity
   
    trainer.log_metrics("eval", metrics)
    trainer.save_metrics("eval", metrics)
   

   kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-generation"}
   if data_args.dataset_name is not None:
   kwargs["dataset_tags"] = data_args.dataset_name
   if data_args.dataset_config_name is not None:
   kwargs["dataset_args"] = data_args.dataset_config_name
   kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
   else:
   kwargs["dataset"] = data_args.dataset_name

   if training_args.push_to_hub:
   trainer.push_to_hub(**kwargs)
   else:
   trainer.create_model_card(**kwargs)

[25018528927]

Hello! I see that you have the same problem as me, did you solve it? If solved, how to solve it? Looking forward to your answer, very anxious！ @Markkk111

[heychhavi]

Hi @Markkk111 @25018528927 , I am also facing this similar issue, anyone of you able to solve this?