Issue: CARLA Simulation Crashes with Signal 11 After Running for Some Time During VAD Closed-Loop Evaluation

[curryqka]

Description

During the closed-loop evaluation using VAD, both run_evaluation_debug.sh and multi-GPU evaluation scripts result in crashes after running for some time. The error messages indicate a segmentation fault and timeout issues related to the CARLA server.

Error Messages

The following errors occur during execution:

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)
Watchdog exception - Timeout of 601.0 seconds occurred
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/shitianyu/anaconda3/envs/b2d_zoo/lib/python3.8/threading.py", line 932, in _bootstrap_inner

Error during the simulation:Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 412, in _load_and_run_scenario
    self.run()
    self.manager.run_scenario()
  File "/home/shitianyu/anaconda3/envs/b2d_zoo/lib/python3.8/threading.py", line 870, in run
  File "/home/shitianyu/Carla/Bench2Drive/leaderboard/leaderboard/scenarios/scenario_manager.py", line 161, in run_scenario
    self._target(*self._args, **self._kwargs)
  File "/home/shitianyu/Carla/Bench2Drive/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in build_scenarios_loop
    self._tick_scenario()
  File "/home/shitianyu/Carla/Bench2Drive/leaderboard/leaderboard/scenarios/scenario_manager.py", line 168, in _tick_scenario
    self.scenario.spawn_parked_vehicles(self.ego_vehicles[0])
  File "/home/shitianyu/Carla/Bench2Drive/leaderboard/leaderboard/scenarios/route_scenario.py", line 236, in spawn_parked_vehicles
    CarlaDataProvider.get_world().tick(self._timeout)
RuntimeError: time-out of 600000ms while waiting for the simulator, make sure the simulator is ready and connected to localhost:30000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 564, in <module>
    for response in CarlaDataProvider.get_client().apply_batch_sync(new_parked_vehicles):
RuntimeError: time-out of 600000ms while waiting for the simulator, make sure the simulator is ready and connected to localhost:30000
    main()
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 554, in main
    crashed = leaderboard_evaluator.run(arguments)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 477, in run
    crashed = self._load_and_run_scenario(args, config)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 428, in _load_and_run_scenario
    print("\n\033[91mError during the simulation:", flush=True)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 144, in _signal_handler
    self.manager.signal_handler(signum, frame)
  File "/home/shitianyu/Carla/Bench2Drive/leaderboard/leaderboard/scenarios/scenario_manager.py", line 89, in signal_handler
    raise RuntimeError("The simulation took longer than {}s to update".format(self._timeout))
RuntimeError: The simulation took longer than 600.0s to update

[jayyoung0802]

The same to #138.