Hip memory issue in 9070xt with rocm-7.0.2

[druidican]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

a simpe genetration of an image,

Actual Behavior

during image genetration the following error message appears

Steps to Reproduce

install rocm-7.0.2, use a default workflow with ksampler -> ultimate SD upscale -> face detailer

after one image, the following will fail non stop,

Debug Logs

# ComfyUI Error Report
## Error Details
- **Node ID:** 116
- **Node Type:** FaceDetailer
- **Exception Type:** torch.AcceleratorError
- **Exception Message:** HIP error: an illegal memory access was encountered
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.


## Stack Trace

  File "/home/lasse/ComfyUI/execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-lora-manager/py/metadata_collector/metadata_hook.py", line 165, in async_map_node_over_list_with_metadata
    results = await original_map_node_over_list(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)

  File "/home/lasse/ComfyUI/execution.py", line 277, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_pack.py", line 876, in doit
    enhanced_img, cropped_enhanced, cropped_enhanced_alpha, mask, cnet_pil_list = FaceDetailer.enhance_face(
                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_pack.py", line 830, in enhance_face
    DetailerForEach.do_detail(image, segs, model, clip, vae, guide_size, guide_size_for_bbox, max_size, seed, steps, cfg,

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_pack.py", line 362, in do_detail
    enhanced_image, cnet_pils = core.enhance_detail(cropped_image, model, clip, vae, guide_size, guide_size_for_bbox, max_size,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/core.py", line 383, in enhance_detail
    refined_latent = impact_sampling.ksampler_wrapper(model2, seed2, steps2, cfg2, sampler_name2, scheduler2, positive2, negative2,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_sampling.py", line 209, in ksampler_wrapper
    refined_latent = separated_sample(model, True, seed, advanced_steps, cfg, sampler_name, scheduler,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_sampling.py", line 182, in separated_sample
    res = sample_with_custom_noise(model, add_noise, seed, cfg, positive, negative, impact_sampler, sigmas, latent_image, noise=noise, callback=callback)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/custom_nodes/comfyui-impact-pack/modules/impact/impact_sampling.py", line 126, in sample_with_custom_noise
    samples = comfy.sample.sample_custom(model, noise, cfg, sampler, sigmas, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/sample.py", line 50, in sample_custom
    samples = comfy.samplers.sample(model, noise, positive, negative, cfg, model.load_device, sampler, sigmas, model_options=model.model_options, latent_image=latent_image, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 1044, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 1029, in sample
    output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 997, in outer_sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 980, in inner_sample
    samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 752, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/k_diffusion/sampling.py", line 795, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 401, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 953, in __call__
    return self.outer_predict_noise(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 960, in outer_predict_noise
    ).execute(x, timestep, model_options, seed)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 963, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 381, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 206, in calc_cond_batch
    return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 214, in _calc_cond_batch_outer
    return executor.execute(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/samplers.py", line 326, in _calc_cond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/model_base.py", line 161, in apply_model
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/model_base.py", line 200, in _apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 831, in forward
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 873, in _forward
    h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 38, in forward_timestep_embed
    x = layer(x, emb)
        ^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 239, in forward
    return checkpoint(
           ^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 191, in checkpoint
    return func(*inputs)
           ^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 252, in _forward
    h = self.in_layers(x)
        ^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/container.py", line 244, in forward
    input = module(input)
            ^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ops.py", line 146, in forward
    return self.forward_comfy_cast_weights(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/comfy/ops.py", line 141, in forward_comfy_cast_weights
    return self._conv_forward(input, weight, bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 543, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^


## System Information
- **ComfyUI Version:** 0.3.65
- **Arguments:** main.py --listen 0.0.0.0 --output-directory /home/lasse/MEGA/ComfyUI --use-pytorch-cross-attention --reserve-vram 1 --lowvram --fast --disable-smart-memory
- **OS:** posix
- **Python Version:** 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
- **Embedded Python:** false
- **PyTorch Version:** 2.8.0+rocm7.0.2.git245bf6ed
## Devices

- **Name:** cuda:0 AMD Radeon Graphics : native
  - **Type:** cuda
  - **VRAM Total:** 17095983104
  - **VRAM Free:** 15807266816
  - **Torch VRAM Total:** 161480704
  - **Torch VRAM Free:** 17809408

## Logs

2025-10-16T15:00:55.196764 - 2025-10-16T15:00:55.196864 -   File "/home/lasse/ComfyUI/main.py", line 195, in prompt_worker
2025-10-16T15:00:55.197216 - 2025-10-16T15:00:55.197287 -     2025-10-16T15:00:55.197397 - e.execute(item[2], prompt_id, item[3], item[4])2025-10-16T15:00:55.197526 - 
2025-10-16T15:00:55.197670 - 2025-10-16T15:00:55.197738 -   File "/home/lasse/ComfyUI/execution.py", line 649, in execute
2025-10-16T15:00:55.198239 - 2025-10-16T15:00:55.198317 -     2025-10-16T15:00:55.198431 - asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))2025-10-16T15:00:55.198526 - 
2025-10-16T15:00:55.198680 - 2025-10-16T15:00:55.198785 -   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
2025-10-16T15:00:55.199070 - 2025-10-16T15:00:55.199147 -     2025-10-16T15:00:55.199274 - return runner.run(main)2025-10-16T15:00:55.199349 - 
2025-10-16T15:00:55.199456 - 2025-10-16T15:00:55.199528 -  2025-10-16T15:00:55.199681 -  2025-10-16T15:00:55.199816 -  2025-10-16T15:00:55.199893 -  2025-10-16T15:00:55.199987 -  2025-10-16T15:00:55.200055 -  2025-10-16T15:00:55.200146 -  2025-10-16T15:00:55.200238 -  2025-10-16T15:00:55.200311 -  2025-10-16T15:00:55.200410 -  2025-10-16T15:00:55.200501 -  2025-10-16T15:00:55.200600 - ^2025-10-16T15:00:55.200710 - ^2025-10-16T15:00:55.200800 - ^2025-10-16T15:00:55.200891 - ^2025-10-16T15:00:55.200983 - ^2025-10-16T15:00:55.201043 - ^2025-10-16T15:00:55.201137 - ^2025-10-16T15:00:55.201219 - ^2025-10-16T15:00:55.201318 - ^2025-10-16T15:00:55.201490 - ^2025-10-16T15:00:55.201644 - ^2025-10-16T15:00:55.201754 - ^2025-10-16T15:00:55.201816 - ^2025-10-16T15:00:55.201906 - ^2025-10-16T15:00:55.201979 - ^2025-10-16T15:00:55.202075 - ^2025-10-16T15:00:55.202149 - 
2025-10-16T15:00:55.202239 - 2025-10-16T15:00:55.202334 -   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
2025-10-16T15:00:55.202522 - 2025-10-16T15:00:55.202670 -     2025-10-16T15:00:55.202772 - return self._loop.run_until_complete(task)2025-10-16T15:00:55.202874 - 
2025-10-16T15:00:55.202987 - 2025-10-16T15:00:55.203060 -  2025-10-16T15:00:55.203155 -  2025-10-16T15:00:55.203227 -  2025-10-16T15:00:55.203328 -  2025-10-16T15:00:55.203411 -  2025-10-16T15:00:55.203515 -  2025-10-16T15:00:55.203671 -  2025-10-16T15:00:55.203776 -  2025-10-16T15:00:55.203849 -  2025-10-16T15:00:55.203940 -  2025-10-16T15:00:55.204007 -  2025-10-16T15:00:55.204097 - ^2025-10-16T15:00:55.204188 - ^2025-10-16T15:00:55.204276 - ^2025-10-16T15:00:55.204382 - ^2025-10-16T15:00:55.204471 - ^2025-10-16T15:00:55.204533 - ^2025-10-16T15:00:55.204684 - ^2025-10-16T15:00:55.204785 - ^2025-10-16T15:00:55.204878 - ^2025-10-16T15:00:55.204978 - ^2025-10-16T15:00:55.205080 - ^2025-10-16T15:00:55.205169 - ^2025-10-16T15:00:55.205229 - ^2025-10-16T15:00:55.205359 - ^2025-10-16T15:00:55.205448 - ^2025-10-16T15:00:55.205537 - ^2025-10-16T15:00:55.205611 - ^2025-10-16T15:00:55.205718 - ^2025-10-16T15:00:55.205806 - ^2025-10-16T15:00:55.205866 - ^2025-10-16T15:00:55.205960 - ^2025-10-16T15:00:55.206042 - ^2025-10-16T15:00:55.206148 - ^2025-10-16T15:00:55.206303 - ^2025-10-16T15:00:55.206435 - ^2025-10-16T15:00:55.206525 - ^2025-10-16T15:00:55.206648 - ^2025-10-16T15:00:55.206725 - ^2025-10-16T15:00:55.206818 - ^2025-10-16T15:00:55.206918 - ^2025-10-16T15:00:55.206991 - ^2025-10-16T15:00:55.207122 - ^2025-10-16T15:00:55.207184 - ^2025-10-16T15:00:55.207312 - ^2025-10-16T15:00:55.207407 - ^2025-10-16T15:00:55.207536 - 
2025-10-16T15:00:55.207613 - 2025-10-16T15:00:55.207712 -   File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
2025-10-16T15:00:55.208176 - 2025-10-16T15:00:55.208256 -     2025-10-16T15:00:55.208357 - return future.result()2025-10-16T15:00:55.208460 - 
2025-10-16T15:00:55.208652 - 2025-10-16T15:00:55.208728 -  2025-10-16T15:00:55.208858 -  2025-10-16T15:00:55.208998 -  2025-10-16T15:00:55.209071 -  2025-10-16T15:00:55.209147 -  2025-10-16T15:00:55.209247 -  2025-10-16T15:00:55.209406 -  2025-10-16T15:00:55.209499 -  2025-10-16T15:00:55.209572 -  2025-10-16T15:00:55.209687 -  2025-10-16T15:00:55.209787 -  2025-10-16T15:00:55.209848 - ^2025-10-16T15:00:55.209944 - ^2025-10-16T15:00:55.210071 - ^2025-10-16T15:00:55.210136 - ^2025-10-16T15:00:55.210233 - ^2025-10-16T15:00:55.210362 - ^2025-10-16T15:00:55.210423 - ^2025-10-16T15:00:55.210550 - ^2025-10-16T15:00:55.210695 - ^2025-10-16T15:00:55.210760 - ^2025-10-16T15:00:55.210889 - ^2025-10-16T15:00:55.210949 - ^2025-10-16T15:00:55.211074 - ^2025-10-16T15:00:55.211163 - ^2025-10-16T15:00:55.211259 - ^2025-10-16T15:00:55.211347 - 
2025-10-16T15:00:55.211471 - 2025-10-16T15:00:55.211560 -   File "/home/lasse/ComfyUI/execution.py", line 722, in execute_async
2025-10-16T15:00:55.211863 - 2025-10-16T15:00:55.211942 -     2025-10-16T15:00:55.212055 - comfy.model_management.unload_all_models()2025-10-16T15:00:55.212158 - 
2025-10-16T15:00:55.212296 - 2025-10-16T15:00:55.212396 -   File "/home/lasse/ComfyUI/comfy/model_management.py", line 1399, in unload_all_models
2025-10-16T15:00:55.212996 - 2025-10-16T15:00:55.213065 -     2025-10-16T15:00:55.213159 - free_memory(1e30, get_torch_device())2025-10-16T15:00:55.213268 - 
2025-10-16T15:00:55.213380 - 2025-10-16T15:00:55.213508 -  2025-10-16T15:00:55.213651 -  2025-10-16T15:00:55.213750 -  2025-10-16T15:00:55.213881 -  2025-10-16T15:00:55.214008 -  2025-10-16T15:00:55.214068 -  2025-10-16T15:00:55.214199 -  2025-10-16T15:00:55.214306 -  2025-10-16T15:00:55.214439 -  2025-10-16T15:00:55.214545 -  2025-10-16T15:00:55.214697 -  2025-10-16T15:00:55.214810 -  2025-10-16T15:00:55.214901 -  2025-10-16T15:00:55.214993 -  2025-10-16T15:00:55.215083 -  2025-10-16T15:00:55.215144 -  2025-10-16T15:00:55.215271 -  2025-10-16T15:00:55.215366 -  2025-10-16T15:00:55.215493 -  2025-10-16T15:00:55.215597 -  2025-10-16T15:00:55.215699 -  2025-10-16T15:00:55.215789 -  2025-10-16T15:00:55.215875 - ^2025-10-16T15:00:55.215962 - ^2025-10-16T15:00:55.216066 - ^2025-10-16T15:00:55.216210 - ^2025-10-16T15:00:55.216316 - ^2025-10-16T15:00:55.216407 - ^2025-10-16T15:00:55.216506 - ^2025-10-16T15:00:55.216644 - ^2025-10-16T15:00:55.216709 - ^2025-10-16T15:00:55.216836 - ^2025-10-16T15:00:55.216927 - ^2025-10-16T15:00:55.216988 - ^2025-10-16T15:00:55.217112 - ^2025-10-16T15:00:55.217232 - ^2025-10-16T15:00:55.217346 - ^2025-10-16T15:00:55.217407 - ^2025-10-16T15:00:55.217498 - ^2025-10-16T15:00:55.217633 - ^2025-10-16T15:00:55.217729 - 
2025-10-16T15:00:55.217836 - 2025-10-16T15:00:55.217977 -   File "/home/lasse/ComfyUI/comfy/model_management.py", line 187, in get_torch_device
2025-10-16T15:00:55.218166 - 2025-10-16T15:00:55.218231 -     2025-10-16T15:00:55.218364 - return torch.device(torch.cuda.current_device())2025-10-16T15:00:55.218464 - 
2025-10-16T15:00:55.218576 - 2025-10-16T15:00:55.218698 -  2025-10-16T15:00:55.218796 -  2025-10-16T15:00:55.218869 -  2025-10-16T15:00:55.218998 -  2025-10-16T15:00:55.219059 -  2025-10-16T15:00:55.219184 -  2025-10-16T15:00:55.219246 -  2025-10-16T15:00:55.219371 -  2025-10-16T15:00:55.219467 -  2025-10-16T15:00:55.219563 -  2025-10-16T15:00:55.219639 -  2025-10-16T15:00:55.219733 -  2025-10-16T15:00:55.219803 -  2025-10-16T15:00:55.219876 -  2025-10-16T15:00:55.219987 -  2025-10-16T15:00:55.220118 -  2025-10-16T15:00:55.220223 -  2025-10-16T15:00:55.220289 -  2025-10-16T15:00:55.220378 -  2025-10-16T15:00:55.220438 -  2025-10-16T15:00:55.220567 -  2025-10-16T15:00:55.220676 -  2025-10-16T15:00:55.220766 -  2025-10-16T15:00:55.220855 -  2025-10-16T15:00:55.220950 - ^2025-10-16T15:00:55.221046 - ^2025-10-16T15:00:55.221172 - ^2025-10-16T15:00:55.221266 - ^2025-10-16T15:00:55.221393 - ^2025-10-16T15:00:55.221482 - ^2025-10-16T15:00:55.221543 - ^2025-10-16T15:00:55.221650 - ^2025-10-16T15:00:55.221785 - ^2025-10-16T15:00:55.221846 - ^2025-10-16T15:00:55.221971 - ^2025-10-16T15:00:55.222057 - ^2025-10-16T15:00:55.222123 - ^2025-10-16T15:00:55.222200 - ^2025-10-16T15:00:55.222296 - ^2025-10-16T15:00:55.222436 - ^2025-10-16T15:00:55.222538 - ^2025-10-16T15:00:55.222678 - ^2025-10-16T15:00:55.222745 - ^2025-10-16T15:00:55.222833 - ^2025-10-16T15:00:55.222921 - ^2025-10-16T15:00:55.222981 - ^2025-10-16T15:00:55.223106 - ^2025-10-16T15:00:55.223200 - ^2025-10-16T15:00:55.223327 - ^2025-10-16T15:00:55.223414 - ^2025-10-16T15:00:55.223511 - ^2025-10-16T15:00:55.223656 - 
2025-10-16T15:00:55.223733 - 2025-10-16T15:00:55.223863 -   File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py", line 1072, in current_device
2025-10-16T15:00:55.224413 - 2025-10-16T15:00:55.224522 -     2025-10-16T15:00:55.224595 - return torch._C._cuda_getDevice()2025-10-16T15:00:55.224733 - 
2025-10-16T15:00:55.224819 - 2025-10-16T15:00:55.224897 -  2025-10-16T15:00:55.224997 -  2025-10-16T15:00:55.225137 -  2025-10-16T15:00:55.225239 -  2025-10-16T15:00:55.225307 -  2025-10-16T15:00:55.225403 -  2025-10-16T15:00:55.225504 -  2025-10-16T15:00:55.225576 -  2025-10-16T15:00:55.225699 -  2025-10-16T15:00:55.225795 -  2025-10-16T15:00:55.225885 -  2025-10-16T15:00:55.225957 - ^2025-10-16T15:00:55.226083 - ^2025-10-16T15:00:55.226179 - ^2025-10-16T15:00:55.226276 - ^2025-10-16T15:00:55.226371 - ^2025-10-16T15:00:55.226496 - ^2025-10-16T15:00:55.226634 - ^2025-10-16T15:00:55.226711 - ^2025-10-16T15:00:55.226837 - ^2025-10-16T15:00:55.226957 - ^2025-10-16T15:00:55.227047 - ^2025-10-16T15:00:55.227144 - ^2025-10-16T15:00:55.227240 - ^2025-10-16T15:00:55.227322 - ^2025-10-16T15:00:55.227503 - ^2025-10-16T15:00:55.227635 - ^2025-10-16T15:00:55.227736 - ^2025-10-16T15:00:55.227833 - ^2025-10-16T15:00:55.227960 - ^2025-10-16T15:00:55.228059 - ^2025-10-16T15:00:55.228155 - ^2025-10-16T15:00:55.228243 - ^2025-10-16T15:00:55.228336 - ^2025-10-16T15:00:55.228460 - ^2025-10-16T15:00:55.228554 - ^2025-10-16T15:00:55.228699 - ^2025-10-16T15:00:55.228804 - 
2025-10-16T15:00:55.228902 - 2025-10-16T15:00:55.229002 - torch2025-10-16T15:00:55.229121 - .2025-10-16T15:00:55.229247 - AcceleratorError2025-10-16T15:00:55.229308 - : 2025-10-16T15:00:55.229440 - HIP error: an illegal memory access was encountered
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
2025-10-16T15:00:55.229563 - 


## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.

Workflow too large. Please manually upload the workflow from local file system.


## Additional Context
(Please add any additional context or steps to reproduce the error here)

Other

No response

[CSFFlame]

I have an identical issue (9070XT),any Rocm 7 build. Both latest amd proprietary drivers and the ubuntu base amd drivers.

Kubuntu 24.04, latest nightly comfyui, but it's been over the past few weeks of comfyui builds.

I'm just using the pytorch rocm 6.4 for now because it doesn't crash like this.

[druidican]

How did you install rocm.. with the amdgpu-installer and with DKMS or in another way ?

[CSFFlame]

with the amdgpu-installer and with DKMS

yes,

and with pip3 using the pip3 install -U --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0

[crosson]

I have the same issue. Here is how I installed rocm. I've since reverted but these were my commands

From this guide

wget https://repo.radeon.com/amdgpu-install/7.0.2/ubuntu/noble/amdgpu-install_7.0.2.70002-1_all.deb
sudo apt install ./amdgpu-install_7.0.2.70002-1_all.deb
sudo apt update
sudo apt install rocm
apt list --installed | grep rocm #Confirming the version

#Then in my test venv
source newenv/bin/activate
pip list
pip uninstall torch torchaudio torchvision
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm7.0

For some reason this broke my other VENV for comfyui running the stable with rocm 6.4. But idk if that's a comfyui issue or my system just getting confused with rocm7 installed while using pytorch stable rocm 6.4.

My error

  hidden_states_slice = torch.bmm(attn_probs.to(value.dtype), value)
/home/auser/git/comfy/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py:180: UserWarning: HIP warning: an illegal memory access was encountered (Triggered internally at /pytorch/aten/src/ATen/hip/impl/HIPGuardImplMasqueradingAsCUDA.h:83.)
  hidden_states_slice = torch.bmm(attn_probs.to(value.dtype), value)
  0%|                                                                                                                                                                                                                      | 0/2 [00:06<?, ?it/s]
!!! Exception during processing !!! HIP error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 277, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/nodes.py", line 1559, in sample
    return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/nodes.py", line 1492, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/sample.py", line 45, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 1161, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 1051, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 1036, in sample
    output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 1004, in outer_sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 987, in inner_sample
    samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 759, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/k_diffusion/sampling.py", line 199, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 408, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 960, in __call__
    return self.outer_predict_noise(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 967, in outer_predict_noise
    ).execute(x, timestep, model_options, seed)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 970, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 388, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 206, in calc_cond_batch
    return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 214, in _calc_cond_batch_outer
    return executor.execute(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/samplers.py", line 333, in _calc_cond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/model_base.py", line 160, in apply_model
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/model_base.py", line 199, in _apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1791, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/wan/model.py", line 614, in forward
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/wan/model.py", line 634, in _forward
    return self.forward_orig(x, timestep, context, clip_fea=clip_fea, freqs=freqs, transformer_options=transformer_options, **kwargs)[:, :, :t, :h, :w]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/wan/model.py", line 579, in forward_orig
    x = block(x, e=e0, freqs=freqs, context=context, context_img_len=context_img_len, transformer_options=transformer_options)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1791, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/wan/model.py", line 235, in forward
    y = self.self_attn(
        ^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1791, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/wan/model.py", line 81, in forward
    x = optimized_attention(
        ^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/modules/attention.py", line 130, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/modules/attention.py", line 257, in attention_sub_quad
    hidden_states = efficient_dot_product_attention(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 268, in efficient_dot_product_attention
    compute_query_chunk_attn(
  File "/home/auser/git/comfy/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 180, in _get_attention_scores_no_kv_chunking
    hidden_states_slice = torch.bmm(attn_probs.to(value.dtype), value)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: HIP error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.


Exception in thread Thread-2 (prompt_worker):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/auser/git/comfy/ComfyUI/main.py", line 195, in prompt_worker
    e.execute(item[2], prompt_id, item[3], item[4])
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 649, in execute
    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/execution.py", line 722, in execute_async
    comfy.model_management.unload_all_models()
  File "/home/auser/git/comfy/ComfyUI/comfy/model_management.py", line 1402, in unload_all_models
    free_memory(1e30, get_torch_device())
                      ^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/comfy/model_management.py", line 187, in get_torch_device
    return torch.device(torch.cuda.current_device())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/auser/git/comfy/ComfyUI/newenv/lib/python3.12/site-packages/torch/cuda/__init__.py", line 1080, in current_device
    return torch._C._cuda_getDevice()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: HIP error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Side note

The output says

set AMD_SERIALIZE_KERNEL=3
But setting that gets
UserWarning: Ignoring invalid value for boolean flag AMD_SERIALIZE_KERNEL: 3valid values are 0 or 1.

I set these env variables as a part of my troubleshooting.

export HIP_VISIBLE_DEVICES=0
export HIP_LAUNCH_BLOCKING=1
export AMD_SERIALIZE_KERNEL=1
export TORCH_BLAS_PREFER_HIPBLASLT=0

Env details

  Name:                    AMD Ryzen 9 7950X3D 16-Core Processor
  Marketing Name:          AMD Ryzen 9 7950X3D 16-Core Processor
  Vendor Name:             CPU                                
  Name:                    gfx1201                            
  Marketing Name:          AMD Radeon RX 9070 XT              
  Vendor Name:             AMD                                
      Name:                    amdgcn-amd-amdhsa--gfx1201         
      Name:                    amdgcn-amd-amdhsa--gfx12-generic   
torch 2.10.0.dev20251016+rocm7.0
HIP runtime: 7.0.51831-a3e329ad8
ROCm detected: True
Device: AMD Radeon RX 9070 XT

I also double checked I have update comfyui to the latest

comfyui-embedded-docs      0.3.0
comfyui_frontend_package   1.28.7
comfyui_workflow_templates 0.1.95

I wouldn't be surprised if this also was a pytorch/rocm issue. But I didn't see any open issues on those repo's and I did see this one. Thank's fellas. I've since reverted back to 6.4 but I'm happy to do another upgrade test. I suppose my life would be a bit easier if I used docker.

[druidican]

this seems to be a growing issue with 9000 series
i see a lot of people complaining about rocm / python instability, but no resolution
a few says it works, but none of them have so far been willing to describe how they have done it (so I considder it unreliable for now)

[x5nder]

Same issue on the 7900GRE (gfx1100) with ROCm 7.x. Tried all the ROCm parameters in the book, nothing works.

[Nihlus]

Same here - 9070XT with ROCm 7.x. It's not happening every time, but frequently enough to be an interrupt to the workflow. It's always in the KSampler node for me.

System information: https://termbin.com/8e28
Output: https://gist.github.com/Nihlus/59edf3a7fc5ebb21c6bd5e243705b448

[slojosic-amd]

Can you try to install 7.0.2 official wheels from here: https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-pytorch.html#install-pytorch-via-pip

[Tureti]

Can you try to install 7.0.2 official wheels from here: https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-pytorch.html#install-pytorch-via-pip

Installing torch from there (rocm-7.0.2) does not fix the issue for me

[druidican]

Same for me... neither reinstall, upgrade or any of the wheels fixes it for me
same error every time

[x5nder]

Using Distorch2 MultiGPU samples solved all my problems.

[druidican]

How did you set it up ??
the
Distorch2 MultiGPU samples

cause i still get:

untered
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with TORCH_USE_HIP_DSA to enable device-side assertions.

Traceback (most recent call last):
File "/home/lasse/ComfyUI/execution.py", line 499, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/execution.py", line 316, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/custom_nodes/ComfyUI-Lora-Manager/py/metadata_collector/metadata_hook.py", line 165, in async_map_node_over_list_with_metadata
results = await original_map_node_over_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/execution.py", line 290, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/lasse/ComfyUI/execution.py", line 278, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/home/lasse/ComfyUI/nodes.py", line 1525, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/nodes.py", line 1492, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/sample.py", line 60, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 1163, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 1053, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 1035, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 997, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 980, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/samplers.py", line 752, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/lasse/ComfyUI/comfy/k_diffusion/sampling.py", line 800, in sample_dpmpp_2m
if old_denoised is None or sigmas[i + 1] == 0:
^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: HIP error: an illegal memory access was encountered
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with TORCH_USE_HIP_DSA to enable device-side assertions.

Prompt executed in 202.07 seconds
Exception in thread Thread-4 (prompt_worker):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/lasse/ComfyUI/main.py", line 240, in prompt_worker
comfy.model_management.soft_empty_cache()
File "/home/lasse/ComfyUI/comfy/model_management.py", line 1432, in soft_empty_cache
torch.cuda.empty_cache()
File "/home/lasse/ComfyUI/.venv/lib/python3.12/site-packages/torch/cuda/memory.py", line 224, in empty_cache
torch._C._cuda_emptyCache()
torch.AcceleratorError: HIP error: an illegal memory access was encountered
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with TORCH_USE_HIP_DSA to enable device-side assertions.