From e0ba75091fce76bdc7a5922fcdc213db381c8863 Mon Sep 17 00:00:00 2001 From: Lukas Steiger Date: Sat, 1 Jun 2024 11:42:21 +0200 Subject: [PATCH 001/132] Update README.md --- README.md | 69 ++++--------------------------------------------------- 1 file changed, 4 insertions(+), 65 deletions(-) diff --git a/README.md b/README.md index a514500c4..b5f2d1577 100644 --- a/README.md +++ b/README.md @@ -1,71 +1,10 @@ +# Experimental fork of the Deforum A1111 extension with the intent of making "turbo-frames" controllable directly. +Will require an experimental version of Parseq to supply new "is-turbo-frame" values, fork here: https://github.com/Tok/sd-parseq -# Deforum Stable Diffusion โ€” official extension for AUTOMATIC1111's webui +This is an experiment in progress and there's no good reason to install this as of now. +Check out the original project instead: https://github.com/deforum-art/sd-webui-deforum -

- Last Commit - GitHub issues - GitHub stars - GitHub forks - -

-## Need help? See our [FAQ](https://github.com/deforum-art/sd-webui-deforum/wiki/FAQ-&-Troubleshooting) - -## Getting Started - -1. Install [AUTOMATIC1111's webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui/). - -2. Now two ways: either clone the repo into the `extensions` directory via git commandline launched within in the `stable-diffusion-webui` folder - -```sh -git clone https://github.com/deforum-art/sd-webui-deforum extensions/deforum -``` - -Or download this repository, locate the `extensions` folder within your WebUI installation, create a folder named `deforum` and put the contents of the downloaded directory inside of it. Then restart WebUI. - -Or launch A1111, navigate to the Extensions tab, choose Available, find deforum in the list of available extensions and install it. Restart A1111 once the extension has been installed. -3. Open the webui, find the Deforum tab at the top of the page. - -4. Enter the animation settings. Refer to [this general guide](https://docs.google.com/document/d/1pEobUknMFMkn8F5TMsv8qRzamXX_75BShMMXV8IFslI/edit) and [this guide to math keyframing functions in Deforum](https://docs.google.com/document/d/1pfW1PwbDIuW0cv-dnuyYj1UzPqe23BlSLTJsqazffXM/edit?usp=sharing). However, **in this version prompt weights less than zero don't just like in original Deforum!** Split the positive and the negative prompt in the json section using --neg argument like this "apple:\`where(cos(t)>=0, cos(t), 0)\`, snow --neg strawberry:\`where(cos(t)<0, -cos(t), 0)\`" - -5. To view animation frames as they're being made, without waiting for the completion of an animation, go to the 'Settings' tab and set the value of this toolbar **above zero**. Warning: it may slow down the generation process. - -![adsdasunknown](https://user-images.githubusercontent.com/14872007/196064311-1b79866a-e55b-438a-84a7-004ff30829ad.png) - - -6. Run the script and see if you got it working or even got something. **In 3D mode a large delay is expected at first** as the script loads the depth models. In the end, using the default settings the whole thing should consume 6.4 GBs of VRAM at 3D mode peaks and no more than 3.8 GB VRAM in 3D mode if you launch the webui with the '--lowvram' command line argument. - -7. After the generation process is completed, click the button with the self-describing name to show the video or gif result right in the GUI! - -8. Join our Discord where you can post generated stuff, ask questions and more: https://discord.gg/deforum.
-* There's also the 'Issues' tab in the repo, for well... reporting issues ;) - -9. Profit! - -## Known issues - -* This port is not fully backward-compatible with the notebook and the local version both due to the changes in how AUTOMATIC1111's webui handles Stable Diffusion models and the changes in this script to get it to work in the new environment. *Expect* that you may not get exactly the same result or that the thing may break down because of the older settings. - -## Screenshots - -Amazing raw Deforum animation by [Pxl.Pshr](https://www.instagram.com/pxl.pshr): -* Turn Audio ON! - -(Audio credits: SKRILLEX, FRED AGAIN & FLOWDAN - RUMBLE (PHACE'S DNB FLIP)) - -https://user-images.githubusercontent.com/121192995/224450647-39529b28-be04-4871-bb7a-faf7afda2ef2.mp4 - -Setting file of that video: [here](https://github.com/deforum-art/sd-webui-deforum/files/11353167/PxlPshrWinningAnimationSettings.txt). - -
- -Main extension tab: - -![image](https://user-images.githubusercontent.com/121192995/226101131-43bf594a-3152-45dd-a5d1-2538d0bc221d.png) - -Keyframes tab: - -![image](https://user-images.githubusercontent.com/121192995/226101140-bfe6cce7-9b78-4a1d-be9a-43e1fc78239e.png) ## License From 6bd5e7c984e4d24954d99c81e63609ff07e7c164 Mon Sep 17 00:00:00 2001 From: Lukas Steiger Date: Sat, 1 Jun 2024 11:43:04 +0200 Subject: [PATCH 002/132] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b5f2d1577..7b0e49df8 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -# Experimental fork of the Deforum A1111 extension with the intent of making "turbo-frames" controllable directly. +# Experimental +Esperimental fork of the Deforum A1111 extension with the intention of making "turbo-frames" controllable directly. Will require an experimental version of Parseq to supply new "is-turbo-frame" values, fork here: https://github.com/Tok/sd-parseq This is an experiment in progress and there's no good reason to install this as of now. From 152f39622b8087e3a83e949ec07b3380099e8aca Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 12:50:02 +0200 Subject: [PATCH 003/132] Started refacoring the render_animation method. Extracted some logic into separate methods and regrouped data accordingly into new tuples. Replaced some comments with samely named method calls. Prefixed some booleans with "is_" for better readability. Reorganized imports. Marked some initialization code that seems dead and added a few TODOs for further consideration and optimization. --- scripts/deforum_helpers/render.py | 552 ++++++++++++++++++++---------- 1 file changed, 372 insertions(+), 180 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 964498c3a..f3c7d9855 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -14,83 +14,71 @@ # Contact the authors: https://deforum.github.io/ -import os -import pandas as pd -import cv2 -import numpy as np -import numexpr import gc +import os import random -import PIL import time +from collections import namedtuple + +import PIL +import cv2 +import numexpr +import numpy as np +import pandas as pd from PIL import Image, ImageOps -from .generate import generate, isJson -from .noise import add_noise +from deforum_api import JobStatusTracker +from modules import lowvram, devices, sd_hijack +from modules.shared import opts, cmd_opts, state, sd_model + +from .RAFT import RAFT from .animation import anim_frame_warp from .animation_key_frames import DeformAnimKeys, LooperAnimKeys -from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -from .depth import DepthModel from .colors import maintain_colors -from .parseq_adapter import ParseqAdapter -from .seed import next_seed -from .image_sharpening import unsharp_mask -from .load_images import get_mask, load_img, load_image, get_mask_from_file +from .composable_masks import compose_mask_with_check +from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled +from .depth import DepthModel +from .generate import generate, isJson from .hybrid_video import ( - hybrid_generation, hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, + hybrid_generation, hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, + get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, rel_flow_to_abs_flow) +from .image_sharpening import unsharp_mask +from .load_images import get_mask, load_img, load_image, get_mask_from_file +from .masks import do_overlay_mask +from .noise import add_noise +from .parseq_adapter import ParseqAdapter +from .prompt import prepare_prompt +from .resume import get_resume_vars from .save_images import save_image -from .composable_masks import compose_mask_with_check +from .seed import next_seed from .settings import save_settings_from_animation_run -from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from .subtitle_handler import init_srt_file, write_frame_subtitle, format_animation_params -from .resume import get_resume_vars -from .masks import do_overlay_mask -from .prompt import prepare_prompt -from modules.shared import opts, cmd_opts, state, sd_model -from modules import lowvram, devices, sd_hijack -from .RAFT import RAFT - -from deforum_api import JobStatusTracker - -def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - - # initialise Parseq adapter - parseq_adapter = ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) - - if opts.data.get("deforum_save_gen_info_as_srt", False): # create .srt file and set timeframe mechanism using FPS - srt_filename = os.path.join(args.outdir, f"{root.timestring}.srt") - srt_frame_duration = init_srt_file(srt_filename, video_args.fps) - - if anim_args.animation_mode in ['2D', '3D']: - # handle hybrid video generation - if anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']: - args, anim_args, inputfiles = hybrid_generation(args, anim_args, root) - # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False - hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') - # initialize prev_flow - if anim_args.hybrid_motion == 'Optical Flow': - prev_flow = None +from .video_audio_utilities import get_frame_name, get_next_frame, render_preview - if loop_args.use_looper: - print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") - if args.strength == 0: - raise RuntimeError("Strength needs to be greater than 0 in Init tab") - args.strength_0_no_init = False - args.seed_behavior = "schedule" - if not isJson(loop_args.init_images): - raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") - # handle controlnet video input frames generation - if is_controlnet_enabled(controlnet_args): - unpack_controlnet_vids(args, anim_args, controlnet_args) +# TODO Temporary tuples to group some data. May be replaced later.. +Srt = namedtuple('Srt', ['filename', 'frame_duration']) +Schedule = namedtuple('Schedule', [ + 'steps', + 'sampler', + 'clipskip', + 'noise_multiplier', + 'ddim_eta', + 'ancestral_eta', + 'mask', + 'noise_mask' +]) +AnimMode = namedtuple('AnimMode', ['hybrid_frame_path', 'prev_flow']) - # expand key frame strings to values - keys = DeformAnimKeys(anim_args, args.seed) if not parseq_adapter.use_parseq else parseq_adapter.anim_keys - loopSchedulesAndData = LooperAnimKeys(loop_args, anim_args, args.seed) if not parseq_adapter.use_parseq else parseq_adapter.looper_keys - # create output folder for the batch - os.makedirs(args.outdir, exist_ok=True) - print(f"Saving animation frames to:\n{args.outdir}") +def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): + parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + srt = prepare_subtitle_file_if_active(args.outdir, root.timestring, video_args.fps) + anim_mode, args, anim_args, inputfiles = apply_animation_mode_settings(anim_args, args, loop_args, root) + handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + # TODO rename loopSchedulesAndData to snake case? + keys, loopSchedulesAndData = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) + create_output_folder_for_the_batch(args) # save settings.txt file for the current run save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) @@ -104,50 +92,31 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if parseq_adapter.use_parseq: anim_args.flip_2d_perspective = True - # expand prompts out to per-frame - if parseq_adapter.manages_prompts(): - prompt_series = keys.prompts - else: - prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) - for i, prompt in root.animation_prompts.items(): - if str(i).isdigit(): - prompt_series[int(i)] = prompt - else: - prompt_series[int(numexpr.evaluate(i))] = prompt - prompt_series = prompt_series.ffill().bfill() - - # check for video inits - using_vid_init = anim_args.animation_mode == 'Video Input' + prompt_series = expand_prompts_out_to_per_frame(parseq_adapter, keys, anim_args, root) - # load depth model for 3D - predict_depths = (anim_args.animation_mode == '3D' and anim_args.use_depth_warping) or anim_args.save_depth_maps - predict_depths = predict_depths or (anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth']) - predict_depths = predict_depths and not args.motion_preview_mode - if predict_depths: - keep_in_vram = opts.data.get("deforum_keep_3d_models_in_vram") + # TODO bundle.. + is_using_init_video = anim_args.animation_mode == 'Video Input' # check for video inits + is_predicting_depths = load_depth_model_for_3d(args, anim_args) + is_keep_in_vram = None + if is_predicting_depths: + is_keep_in_vram = opts.data.get("deforum_keep_3d_models_in_vram") device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) - depth_model = DepthModel(root.models_path, device, root.half_precision, keep_in_vram=keep_in_vram, depth_algorithm=anim_args.depth_algorithm, Width=args.W, Height=args.H, + depth_model = DepthModel(root.models_path, device, root.half_precision, keep_in_vram=is_keep_in_vram, + depth_algorithm=anim_args.depth_algorithm, Width=args.W, Height=args.H, midas_weight=anim_args.midas_weight) # depth-based hybrid composite mask requires saved depth maps - if anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth': + if is_composite_with_depth_mask(anim_args): anim_args.save_depth_maps = True else: depth_model = None anim_args.save_depth_maps = False - raft_model = None - load_raft = (anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1) or \ - (anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT") or \ - (anim_args.optical_flow_redo_generation == "RAFT") - load_raft = load_raft and not args.motion_preview_mode - if load_raft: - print("Loading RAFT model...") - raft_model = RAFT() + is_raft_active, raft_model = load_raft(args, anim_args) # state for interpolating between diffusion steps - turbo_steps = 1 if using_vid_init else int(anim_args.diffusion_cadence) + turbo_steps = 1 if is_using_init_video else int(anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -197,11 +166,15 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 if anim_args.use_mask_video: - args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), + args) + root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), + args) - mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - noise_mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + mask_vals['video_mask'] = get_mask_from_file( + get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + noise_mask_vals['video_mask'] = get_mask_from_file( + get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) elif mask_image is None and args.use_mask: mask_vals['video_mask'] = get_mask(args) noise_mask_vals['video_mask'] = get_mask(args) # TODO?: add a different default noisc mask @@ -245,8 +218,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro "alpha": keys.hybrid_comp_alpha_schedule_series[frame_idx], "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], - "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), - "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), + "mask_auto_contrast_cutoff_low": int( + keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), + "mask_auto_contrast_cutoff_high": int( + keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), "flow_factor": keys.hybrid_flow_factor_schedule_series[frame_idx] } scheduled_sampler_name = None @@ -257,22 +232,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro mask_seq = None noise_mask_seq = None - if anim_args.enable_steps_scheduling and keys.steps_schedule_series[frame_idx] is not None: - args.steps = int(keys.steps_schedule_series[frame_idx]) - if anim_args.enable_sampler_scheduling and keys.sampler_schedule_series[frame_idx] is not None: - scheduled_sampler_name = keys.sampler_schedule_series[frame_idx].casefold() - if anim_args.enable_clipskip_scheduling and keys.clipskip_schedule_series[frame_idx] is not None: - scheduled_clipskip = int(keys.clipskip_schedule_series[frame_idx]) - if anim_args.enable_noise_multiplier_scheduling and keys.noise_multiplier_schedule_series[frame_idx] is not None: - scheduled_noise_multiplier = float(keys.noise_multiplier_schedule_series[frame_idx]) - if anim_args.enable_ddim_eta_scheduling and keys.ddim_eta_schedule_series[frame_idx] is not None: - scheduled_ddim_eta = float(keys.ddim_eta_schedule_series[frame_idx]) - if anim_args.enable_ancestral_eta_scheduling and keys.ancestral_eta_schedule_series[frame_idx] is not None: - scheduled_ancestral_eta = float(keys.ancestral_eta_schedule_series[frame_idx]) - if args.use_mask and keys.mask_schedule_series[frame_idx] is not None: - mask_seq = keys.mask_schedule_series[frame_idx] - if anim_args.use_noise_mask and keys.noise_mask_schedule_series[frame_idx] is not None: - noise_mask_seq = keys.noise_mask_schedule_series[frame_idx] + + schedule = apply_scheduling(keys, frame_idx, anim_args, args) #FIXME dead code?? if args.use_mask and not anim_args.use_noise_mask: noise_mask_seq = mask_seq @@ -284,12 +245,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - if predict_depths: depth_model.to(root.device) + if is_predicting_depths: depth_model.to(root.device) if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) params_string = format_animation_params(keys, prompt_series, frame_idx, params_to_print) - write_frame_subtitle(srt_filename, frame_idx, srt_frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") + write_frame_subtitle(srt.filename, frame_idx, srt.frame_duration, + f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") params_string = None # emit in-between frames @@ -309,68 +271,97 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': if keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: - cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, anim_args.optical_flow_cadence, raft_model) / 2 + cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, + anim_args.optical_flow_cadence, raft_model) / 2 turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) params_string = format_animation_params(keys, prompt_series, tween_frame_idx, params_to_print) - write_frame_subtitle(srt_filename, tween_frame_idx, srt_frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") + write_frame_subtitle(srt.filename, tween_frame_idx, srt.frame_duration, + f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None - print(f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") + print( + f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") if depth_model is not None: assert (turbo_next_image is not None) depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, keys, tween_frame_idx, + depth_model, depth=depth, device=root.device, + half_precision=root.half_precision) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, keys, tween_frame_idx, + depth_model, depth=depth, device=root.device, + half_precision=root.half_precision) # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion if tween_frame_idx > 0: if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, prev_img, anim_args.hybrid_motion) + matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), + inputfiles, prev_img, anim_args.hybrid_motion) if advance_prev: - turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, anim_args.hybrid_motion) + turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, + anim_args.hybrid_motion) if advance_next: - turbo_next_image = image_transform_ransac(turbo_next_image, matrix, anim_args.hybrid_motion) + turbo_next_image = image_transform_ransac(turbo_next_image, matrix, + anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, anim_args.hybrid_motion) + matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, + anim_args.hybrid_motion) if advance_prev: - turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, anim_args.hybrid_motion) + turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, + anim_args.hybrid_motion) if advance_next: - turbo_next_image = image_transform_ransac(turbo_next_image, matrix, anim_args.hybrid_motion) + turbo_next_image = image_transform_ransac(turbo_next_image, matrix, + anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, - anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, + anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, + anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, + anim_args.hybrid_consistency_blur, + anim_args.hybrid_comp_save_extra_frames) if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, + hybrid_comp_schedules['flow_factor']) if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - prev_flow = flow + turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, + hybrid_comp_schedules['flow_factor']) + anim_mode.prev_flow = flow else: - flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, anim_args.hybrid_flow_method, raft_model, - anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, + anim_mode.hybrid_frame_path, anim_mode.prev_flow, + anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, + anim_args.hybrid_consistency_blur, + anim_args.hybrid_comp_save_extra_frames) if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, + hybrid_comp_schedules['flow_factor']) if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - prev_flow = flow + turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, + hybrid_comp_schedules['flow_factor']) + anim_mode.prev_flow = flow # do optical flow cadence after animation warping if cadence_flow is not None: cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model, + depth=depth, device=root.device, + half_precision=root.half_precision) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.W, args.H) * tween if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, cadence_flow_inc, cadence_flow_factor) + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, cadence_flow_inc, + cadence_flow_factor) if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, cadence_flow_inc, cadence_flow_factor) + turbo_next_image = image_transform_optical_flow(turbo_next_image, cadence_flow_inc, + cadence_flow_factor) turbo_prev_frame_idx = turbo_next_frame_idx = tween_frame_idx @@ -398,13 +389,16 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro filename = f"{root.timestring}_{tween_frame_idx:09}.png" cv2.imwrite(os.path.join(args.outdir, filename), img) if anim_args.save_depth_maps: - depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), depth) + depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), + depth) # get color match for video outside of prev_img conditional - hybrid_available = anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective'] + hybrid_available = anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Optical Flow', 'Affine', + 'Perspective'] if anim_args.color_coherence == 'Video Input' and hybrid_available: if int(frame_idx) % int(anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name(anim_args.video_init_path) + f"{frame_idx:09}.jpg")) + prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name( + anim_args.video_init_path) + f"{frame_idx:09}.jpg")) prev_vid_img = prev_vid_img.resize((args.W, args.H), PIL.Image.LANCZOS) color_match_sample = np.asarray(prev_vid_img) color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) @@ -412,32 +406,45 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, args, anim_args, keys, frame_idx, depth_model, depth=None, device=root.device, half_precision=root.half_precision) + prev_img, depth = anim_frame_warp(prev_img, args, anim_args, keys, frame_idx, depth_model, depth=None, + device=root.device, half_precision=root.half_precision) # do hybrid compositing before motion if anim_args.hybrid_composite == 'Before Motion': - args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, hybrid_comp_schedules, root) + args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, + hybrid_comp_schedules, root) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, prev_img, anim_args.hybrid_motion) + matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, prev_img, + anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, anim_args.hybrid_motion) + matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, + anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, - anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, + anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, + anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, + anim_args.hybrid_consistency_blur, + anim_args.hybrid_comp_save_extra_frames) else: - flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, anim_args.hybrid_flow_method, raft_model, - anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, + anim_mode.hybrid_frame_path, + anim_mode.prev_flow, anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, + anim_args.hybrid_consistency_blur, + anim_args.hybrid_comp_save_extra_frames) prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) - prev_flow = flow + anim_mode.prev_flow = flow # do hybrid compositing after motion (normal) if anim_args.hybrid_composite == 'Normal': - args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, hybrid_comp_schedules, root) + args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, + hybrid_comp_schedules, root) # apply color matching if anim_args.color_coherence != 'None': @@ -455,12 +462,15 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro contrast_image = (prev_img * contrast).round().astype(np.uint8) # anti-blur if amount > 0: - contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, mask_image if args.use_mask else None) + contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, + mask_image if args.use_mask else None) # apply frame noising if args.use_mask or anim_args.use_noise_mask: - root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq, noise_mask_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) + root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq, noise_mask_vals, Image.fromarray( + cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) noised_image = add_noise(contrast_image, noise, args.seed, anim_args.noise_type, - (anim_args.perlin_w, anim_args.perlin_h, anim_args.perlin_octaves, anim_args.perlin_persistence), + (anim_args.perlin_w, anim_args.perlin_h, anim_args.perlin_octaves, + anim_args.perlin_persistence), root.noise_mask, args.invert_mask) # use transformed previous frame as init for current @@ -498,20 +508,24 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) # grab init image for current frame - if using_vid_init: + if is_using_init_video: init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") args.init_image = init_frame args.init_image_box = None # init_image_box not used in this case args.strength = max(0.0, min(1.0, strength)) if anim_args.use_mask_video: - args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), + args) + root.noise_mask = get_mask_from_file( + get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + mask_vals['video_mask'] = get_mask_from_file( + get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) if args.use_mask: - args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, root.init_sample) if root.init_sample is not None else None # we need it only after the first frame anyway + args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, + root.init_sample) if root.init_sample is not None else None # we need it only after the first frame anyway # setting up some arguments for the looper loop_args.imageStrength = loopSchedulesAndData.image_strength_schedule_series[frame_idx] @@ -522,7 +536,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro loop_args.use_looper = loopSchedulesAndData.use_looper loop_args.imagesToKeyframe = loopSchedulesAndData.imagesToKeyframe - if 'img2img_fix_steps' in opts.data and opts.data["img2img_fix_steps"]: # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations + if 'img2img_fix_steps' in opts.data and opts.data[ + "img2img_fix_steps"]: # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations opts.data["img2img_fix_steps"] = False if scheduled_clipskip is not None: opts.data["CLIP_stop_at_last_layers"] = scheduled_clipskip @@ -534,20 +549,22 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro opts.data["eta_ancestral"] = scheduled_ancestral_eta if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): - if predict_depths: depth_model.to('cpu') + if is_predicting_depths: depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - + optical_flow_redo_generation = anim_args.optical_flow_redo_generation if not args.motion_preview_mode else 'None' # optical flow redo before generation if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: stored_seed = args.seed args.seed = random.randint(0, 2 ** 32 - 1) - print(f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") + print( + f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) + disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + frame_idx, sampler_name=scheduled_sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) @@ -563,7 +580,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro for n in range(0, int(anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) + disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + frame_idx, sampler_name=scheduled_sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(anim_args.diffusion_redo): @@ -574,7 +592,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # generation - image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) + image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, + sampler_name=scheduled_sampler_name) if image is None: break @@ -586,11 +605,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, so we do an extra generation to avoid the corruption introduced by the color match of first output - if frame_idx == 0 and (anim_args.color_coherence == 'Image' or (anim_args.color_coherence == 'Video Input' and hybrid_available)): - image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, anim_args.color_coherence) + if frame_idx == 0 and (anim_args.color_coherence == 'Image' or ( + anim_args.color_coherence == 'Video Input' and hybrid_available)): + image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, + anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) elif color_match_sample is not None and anim_args.color_coherence != 'None' and not anim_args.legacy_colormatch: - image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, anim_args.color_coherence) + image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, + anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) # intercept and override to grayscale @@ -603,11 +625,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro image = do_overlay_mask(args, anim_args, image, frame_idx) # on strength 0, set color match to generation - if ((not anim_args.legacy_colormatch and not args.use_init) or (anim_args.legacy_colormatch and strength == 0)) and not anim_args.color_coherence in ['Image', 'Video Input']: + if ((not anim_args.legacy_colormatch and not args.use_init) or ( + anim_args.legacy_colormatch and strength == 0)) and not anim_args.color_coherence in ['Image', + 'Video Input']: color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not using_vid_init: + if not is_using_init_video: prev_img = opencv_image if turbo_steps > 1: @@ -633,18 +657,186 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro sd_hijack.model_hijack.hijack(sd_model) frame_idx += 1 - state.assign_current_image(image) + last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, + root, frame_idx, last_preview_frame) + update_tracker(root, frame_idx, anim_args) + cleanup(is_predicting_depths, is_keep_in_vram, depth_model, is_raft_active, raft_model) + + +def prepare_subtitle_file_if_active(outdir, timestring, fps): + if opts.data.get("deforum_save_gen_info_as_srt", False): + # create .srt file and set timeframe mechanism using FPS + filename = os.path.join(outdir, f"{timestring}.srt") + frame_duration = init_srt_file(filename, fps) + return Srt(filename, frame_duration) + + +def has_schedule(keys, i): + return keys.steps_schedule_series[i] is not None + + +def has_mask_schedule(keys, i): + return keys.mask_schedule_series[i] is not None + + +def has_noise_mask_schedule(keys, i): + return keys.noise_mask_schedule_series[i] is not None + + +def use_on_cond_if_scheduled(keys, i, value, cond): + return value if cond and has_schedule(keys, i) else None + + +def schedule_steps(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, int(keys.steps_schedule_series[i]), + anim_args.enable_steps_scheduling) + + +def schedule_sampler(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, keys.sampler_schedule_series[i].casefold(), + anim_args.enable_sampler_scheduling) + + +def schedule_clipskip(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, int(keys.clipskip_schedule_series[i]), + anim_args.enable_clipskip_scheduling) + + +def schedule_noise_multiplier(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, float(keys.noise_multiplier_schedule_series[i]), + anim_args.enable_noise_multiplier_scheduling) + + +def schedule_ddim_eta(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, float(keys.ddim_eta_schedule_series[i]), + anim_args.enable_ddim_eta_scheduling) + + +def schedule_ancestral_eta(keys, i, anim_args): + return use_on_cond_if_scheduled(keys, i, float(keys.ancestral_eta_schedule_series[i]), + anim_args.enable_ancestral_eta_scheduling) + + +def schedule_mask(keys, i, args): + #TODO can we have a mask schedule without a normal schedule? if so check and optimize + return keys.mask_schedule_series[i] \ + if args.use_mask and has_mask_schedule(keys, i) else None + + +def schedule_noise_mask(keys, i, anim_args): + #TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize + return keys.noise_mask_schedule_series[i] \ + if anim_args.use_noise_mask and has_noise_mask_schedule(keys, i) else None + + +def apply_scheduling(keys, i, anim_args, args): + """Apply various scheduling settings based on the current frame index.""" + return Schedule(schedule_steps(keys, i, anim_args), + schedule_sampler(keys, i, anim_args), + schedule_clipskip(keys, i, anim_args), + schedule_noise_multiplier(keys, i, anim_args), + schedule_ddim_eta(keys, i, anim_args), + schedule_ancestral_eta(keys, i, anim_args), + schedule_mask(keys, i, args), #TODO for some reason use_mask is in args instead of anim_args + schedule_noise_mask(keys, i, anim_args)) + + +def apply_animation_mode_settings(anim_args, args, loop_args, root): + hybrid_frame_path = None + prev_flow = None + inputfiles = None + if anim_args.animation_mode in ['2D', '3D']: + # handle hybrid video generation + if anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']: + args, anim_args, inputfiles = hybrid_generation(args, anim_args, root) + # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False + hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') + # initialize prev_flow + if anim_args.hybrid_motion == 'Optical Flow': + prev_flow = None #TODO prev_flow is always set to None in here, so probably remove this logic + + if loop_args.use_looper: + print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") + if args.strength == 0: + raise RuntimeError("Strength needs to be greater than 0 in Init tab") + args.strength_0_no_init = False + args.seed_behavior = "schedule" + if not isJson(loop_args.init_images): + raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") + return AnimMode(hybrid_frame_path, prev_flow), args, anim_args, inputfiles + + +def initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args): + return ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + + +def handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args): + if is_controlnet_enabled(controlnet_args): + unpack_controlnet_vids(args, anim_args, controlnet_args) + + +def create_output_folder_for_the_batch(args): + os.makedirs(args.outdir, exist_ok=True) + print(f"Saving animation frames to:\n{args.outdir}") + + +def use_value_or_parseq_value(value, parseq_value, parseq_adapter): + return value if not parseq_adapter.use_parseq else parseq_value + - args.seed = next_seed(args, root) +def expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args): + return (use_value_or_parseq_value(DeformAnimKeys(anim_args, args.seed), + parseq_adapter.anim_keys, parseq_adapter), + use_value_or_parseq_value(LooperAnimKeys(loop_args, anim_args, args.seed), + parseq_adapter.looper_keys, parseq_adapter)) - last_preview_frame = render_preview(args, anim_args, video_args, root, frame_idx, last_preview_frame) - JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx/anim_args.max_frames) +def load_depth_model_for_3d(args, anim_args): + is_depth_warped_3d = anim_args.animation_mode == '3D' and anim_args.use_depth_warping + is_composite_with_depth = anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] + is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth + return is_depth_used and not args.motion_preview_mode +def expand_prompts_out_to_per_frame(parseq_adapter, keys, anim_args, root): + if parseq_adapter.manages_prompts(): + return keys.prompts + else: + prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) + for i, prompt in root.animation_prompts.items(): + if str(i).isdigit(): + prompt_series[int(i)] = prompt + else: + prompt_series[int(numexpr.evaluate(i))] = prompt + return prompt_series.ffill().bfill() + + +def is_composite_with_depth_mask(anim_args): + return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' + + +def load_raft(args, anim_args): + is_cadenced_raft = anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1 + is_optical_flow_raft = anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT" + is_raft_redo = anim_args.optical_flow_redo_generation == "RAFT" + is_load_raft = (is_cadenced_raft or is_optical_flow_raft or is_raft_redo) and not args.motion_preview_mode + if is_load_raft: + print("Loading RAFT model...") + return is_load_raft, RAFT() if is_load_raft else None + + +def progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame): + state.assign_current_image(image) + args.seed = next_seed(args, root) + return render_preview(args, anim_args, video_args, root, frame_idx, last_preview_frame) + + +def update_tracker(root, frame_idx, anim_args): + JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) + + +def cleanup(predict_depths, keep_in_vram, depth_model, is_load_raft, raft_model): if predict_depths and not keep_in_vram: depth_model.delete_model() # handles adabins too - - if load_raft: + if is_load_raft: raft_model.delete_model() - From 091a1fb6ae47dcd1177b246e46be964496445d74 Mon Sep 17 00:00:00 2001 From: Lukas Steiger Date: Sat, 1 Jun 2024 13:01:24 +0200 Subject: [PATCH 004/132] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7b0e49df8..e8e9b13b7 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ # Experimental -Esperimental fork of the Deforum A1111 extension with the intention of making "turbo-frames" controllable directly. +Experimental fork of the Deforum A1111 extension with the intention of making "turbo-frames" controllable directly. Will require an experimental version of Parseq to supply new "is-turbo-frame" values, fork here: https://github.com/Tok/sd-parseq This is an experiment in progress and there's no good reason to install this as of now. From 3ec5a0de653d2d088b2d6880977ccf7e4bfb7a53 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 13:33:05 +0200 Subject: [PATCH 005/132] Extracted looper and color match initialization. Introduced some temp vars to avoid reassignments and consolidate calls. --- scripts/deforum_helpers/render.py | 69 +++++++++++++++++-------------- 1 file changed, 37 insertions(+), 32 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index f3c7d9855..b7e3cdd36 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -76,8 +76,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro srt = prepare_subtitle_file_if_active(args.outdir, root.timestring, video_args.fps) anim_mode, args, anim_args, inputfiles = apply_animation_mode_settings(anim_args, args, loop_args, root) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - # TODO rename loopSchedulesAndData to snake case? - keys, loopSchedulesAndData = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) + keys, loop_schedules_and_data = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) create_output_folder_for_the_batch(args) # save settings.txt file for the current run @@ -515,27 +514,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.init_image_box = None # init_image_box not used in this case args.strength = max(0.0, min(1.0, strength)) if anim_args.use_mask_video: - args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), - args) - root.noise_mask = get_mask_from_file( - get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - - mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + mask_init_frame = get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True) + temp_mask = get_mask_from_file(mask_init_frame, args) + args.mask_file = temp_mask + root.noise_mask = temp_mask + mask_vals['video_mask'] = temp_mask if args.use_mask: - args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, - root.init_sample) if root.init_sample is not None else None # we need it only after the first frame anyway - - # setting up some arguments for the looper - loop_args.imageStrength = loopSchedulesAndData.image_strength_schedule_series[frame_idx] - loop_args.blendFactorMax = loopSchedulesAndData.blendFactorMax_series[frame_idx] - loop_args.blendFactorSlope = loopSchedulesAndData.blendFactorSlope_series[frame_idx] - loop_args.tweeningFrameSchedule = loopSchedulesAndData.tweening_frames_schedule_series[frame_idx] - loop_args.colorCorrectionFactor = loopSchedulesAndData.color_correction_factor_series[frame_idx] - loop_args.use_looper = loopSchedulesAndData.use_looper - loop_args.imagesToKeyframe = loopSchedulesAndData.imagesToKeyframe + args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, root.init_sample) \ + if root.init_sample is not None else None # we need it only after the first frame anyway + setup_looper_arguments(loop_args, loop_schedules_and_data, frame_idx); if 'img2img_fix_steps' in opts.data and opts.data[ "img2img_fix_steps"]: # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations opts.data["img2img_fix_steps"] = False @@ -600,20 +589,16 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do hybrid video after generation if frame_idx > 0 and anim_args.hybrid_composite == 'After Generation': - image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - args, image = hybrid_composite(args, anim_args, frame_idx, image, depth_model, hybrid_comp_schedules, root) - image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) + temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, depth_model, hybrid_comp_schedules, root) + image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, so we do an extra generation to avoid the corruption introduced by the color match of first output - if frame_idx == 0 and (anim_args.color_coherence == 'Image' or ( - anim_args.color_coherence == 'Video Input' and hybrid_available)): - image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, - anim_args.color_coherence) - image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) - elif color_match_sample is not None and anim_args.color_coherence != 'None' and not anim_args.legacy_colormatch: - image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, - anim_args.color_coherence) - image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) + + if frame_idx == 0 and should_initialize_color_match(anim_args, hybrid_available, color_match_sample): + temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + temp_image = maintain_colors(temp_color, color_match_sample, anim_args.color_coherence) + image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) # intercept and override to grayscale if anim_args.color_force_grayscale: @@ -663,6 +648,26 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro cleanup(is_predicting_depths, is_keep_in_vram, depth_model, is_raft_active, raft_model) +def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): + """Determines whether to initialize color matching based on the given conditions.""" + has_video_input = anim_args.color_coherence == 'Video Input' and hybrid_available + has_image_color_coherence = anim_args.color_coherence == 'Image' + has_any_color_sample = color_match_sample is not None + has_coherent_non_legacy_color_match = anim_args.color_coherence != 'None' and not anim_args.legacy_colormatch + has_sample_and_match = has_any_color_sample and has_coherent_non_legacy_color_match + return has_video_input or has_image_color_coherence or has_sample_and_match + + +def setup_looper_arguments(loop_args, loop_schedules_and_data, i): + loop_args.imageStrength = loop_schedules_and_data.image_strength_schedule_series[i] + loop_args.blendFactorMax = loop_schedules_and_data.blendFactorMax_series[i] + loop_args.blendFactorSlope = loop_schedules_and_data.blendFactorSlope_series[i] + loop_args.tweeningFrameSchedule = loop_schedules_and_data.tweening_frames_schedule_series[i] + loop_args.colorCorrectionFactor = loop_schedules_and_data.color_correction_factor_series[i] + loop_args.use_looper = loop_schedules_and_data.use_looper + loop_args.imagesToKeyframe = loop_schedules_and_data.imagesToKeyframe + + def prepare_subtitle_file_if_active(outdir, timestring, fps): if opts.data.get("deforum_save_gen_info_as_srt", False): # create .srt file and set timeframe mechanism using FPS From 842c2b4d4693a3a2b80bace4c86b474595cfe945 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 14:50:20 +0200 Subject: [PATCH 006/132] Opts schedule extracted. --- scripts/deforum_helpers/render.py | 38 +++++++++++++++++++------------ 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index b7e3cdd36..0da48890d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -524,18 +524,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway - setup_looper_arguments(loop_args, loop_schedules_and_data, frame_idx); - if 'img2img_fix_steps' in opts.data and opts.data[ - "img2img_fix_steps"]: # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations - opts.data["img2img_fix_steps"] = False - if scheduled_clipskip is not None: - opts.data["CLIP_stop_at_last_layers"] = scheduled_clipskip - if scheduled_noise_multiplier is not None: - opts.data["initial_noise_multiplier"] = scheduled_noise_multiplier - if scheduled_ddim_eta is not None: - opts.data["eta_ddim"] = scheduled_ddim_eta - if scheduled_ancestral_eta is not None: - opts.data["eta_ancestral"] = scheduled_ancestral_eta + setup_looper_arguments(loop_args, loop_schedules_and_data, frame_idx) + setup_opts(opts, scheduled_clipskip, scheduled_noise_multiplier, scheduled_ddim_eta, scheduled_ancestral_eta) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): if is_predicting_depths: depth_model.to('cpu') @@ -593,8 +583,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, depth_model, hybrid_comp_schedules, root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) - # color matching on first frame is after generation, color match was collected earlier, so we do an extra generation to avoid the corruption introduced by the color match of first output - + # color matching on first frame is after generation, color match was collected earlier, + # so we do an extra generation to avoid the corruption introduced by the color match of first output if frame_idx == 0 and should_initialize_color_match(anim_args, hybrid_available, color_match_sample): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) temp_image = maintain_colors(temp_color, color_match_sample, anim_args.color_coherence) @@ -658,6 +648,26 @@ def should_initialize_color_match(anim_args, hybrid_available, color_match_sampl return has_video_input or has_image_color_coherence or has_sample_and_match +def has_img2img_fix_steps(opts): + return 'img2img_fix_steps' in opts.data and opts.data["img2img_fix_steps"] + + +def set_if_not_none(dictionary, key, value): + # TODO Helper method, move elsewhere? + if value is not None: + dictionary[key] = value + + +def setup_opts(opts, scheduled_clipskip, scheduled_noise_multiplier, scheduled_ddim_eta, scheduled_ancestral_eta): + if has_img2img_fix_steps(opts): + # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations + opts.data["img2img_fix_steps"] = False # TODO is this ever true? + set_if_not_none(opts.data, "CLIP_stop_at_last_layers", scheduled_clipskip) + set_if_not_none(opts.data, "initial_noise_multiplier", scheduled_noise_multiplier) + set_if_not_none(opts.data, "eta_ddim", scheduled_ddim_eta) + set_if_not_none(opts.data, "eta_ancestral", scheduled_ancestral_eta) + + def setup_looper_arguments(loop_args, loop_schedules_and_data, i): loop_args.imageStrength = loop_schedules_and_data.image_strength_schedule_series[i] loop_args.blendFactorMax = loop_schedules_and_data.blendFactorMax_series[i] From 6817c0118d97eb0806163639b02a0b5ea7e71253 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 16:10:25 +0200 Subject: [PATCH 007/132] Replaced sample tuple with new class and applied the values. --- scripts/deforum_helpers/render.py | 69 ++++++++++++++----------------- 1 file changed, 32 insertions(+), 37 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 0da48890d..a1bc4fb3c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -58,19 +58,22 @@ # TODO Temporary tuples to group some data. May be replaced later.. Srt = namedtuple('Srt', ['filename', 'frame_duration']) -Schedule = namedtuple('Schedule', [ - 'steps', - 'sampler', - 'clipskip', - 'noise_multiplier', - 'ddim_eta', - 'ancestral_eta', - 'mask', - 'noise_mask' -]) AnimMode = namedtuple('AnimMode', ['hybrid_frame_path', 'prev_flow']) +# TODO move elsewhere.. +class Schedule: + def __init__(self, steps, sampler_name, clipskip, noise_multiplier, ddim_eta, ancestral_eta, mask, noise_mask): + self.steps = steps + self.sampler_name = sampler_name + self.clipskip = clipskip + self.noise_multiplier = noise_multiplier + self.ddim_eta = ddim_eta + self.ancestral_eta = ancestral_eta + self.mask = mask + self.noise_mask = noise_mask + + def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) srt = prepare_subtitle_file_if_active(args.outdir, root.timestring, video_args.fps) @@ -223,19 +226,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), "flow_factor": keys.hybrid_flow_factor_schedule_series[frame_idx] } - scheduled_sampler_name = None - scheduled_clipskip = None - scheduled_noise_multiplier = None - scheduled_ddim_eta = None - scheduled_ancestral_eta = None - - mask_seq = None - noise_mask_seq = None schedule = apply_scheduling(keys, frame_idx, anim_args, args) #FIXME dead code?? if args.use_mask and not anim_args.use_noise_mask: - noise_mask_seq = mask_seq + noise_mask_seq = schedule.mask_seq depth = None @@ -521,11 +516,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro mask_vals['video_mask'] = temp_mask if args.use_mask: - args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, root.init_sample) \ + args.mask_image = compose_mask_with_check(root, args, schedule.mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway setup_looper_arguments(loop_args, loop_schedules_and_data, frame_idx) - setup_opts(opts, scheduled_clipskip, scheduled_noise_multiplier, scheduled_ddim_eta, scheduled_ancestral_eta) + setup_opts(opts, schedule) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): if is_predicting_depths: depth_model.to('cpu') @@ -543,7 +538,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, - frame_idx, sampler_name=scheduled_sampler_name) + frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) @@ -560,7 +555,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, - frame_idx, sampler_name=scheduled_sampler_name) + frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(anim_args.diffusion_redo): @@ -572,7 +567,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # generation image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, - sampler_name=scheduled_sampler_name) + sampler_name=schedule.sampler_name) if image is None: break @@ -658,14 +653,14 @@ def set_if_not_none(dictionary, key, value): dictionary[key] = value -def setup_opts(opts, scheduled_clipskip, scheduled_noise_multiplier, scheduled_ddim_eta, scheduled_ancestral_eta): +def setup_opts(opts, schedule): if has_img2img_fix_steps(opts): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations opts.data["img2img_fix_steps"] = False # TODO is this ever true? - set_if_not_none(opts.data, "CLIP_stop_at_last_layers", scheduled_clipskip) - set_if_not_none(opts.data, "initial_noise_multiplier", scheduled_noise_multiplier) - set_if_not_none(opts.data, "eta_ddim", scheduled_ddim_eta) - set_if_not_none(opts.data, "eta_ancestral", scheduled_ancestral_eta) + set_if_not_none(opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) + set_if_not_none(opts.data, "initial_noise_multiplier", schedule.noise_multiplier) + set_if_not_none(opts.data, "eta_ddim", schedule.ddim_eta) + set_if_not_none(opts.data, "eta_ancestral", schedule.ancestral_eta) def setup_looper_arguments(loop_args, loop_schedules_and_data, i): @@ -746,14 +741,14 @@ def schedule_noise_mask(keys, i, anim_args): def apply_scheduling(keys, i, anim_args, args): """Apply various scheduling settings based on the current frame index.""" - return Schedule(schedule_steps(keys, i, anim_args), - schedule_sampler(keys, i, anim_args), - schedule_clipskip(keys, i, anim_args), - schedule_noise_multiplier(keys, i, anim_args), - schedule_ddim_eta(keys, i, anim_args), - schedule_ancestral_eta(keys, i, anim_args), - schedule_mask(keys, i, args), #TODO for some reason use_mask is in args instead of anim_args - schedule_noise_mask(keys, i, anim_args)) + return Schedule(steps=schedule_steps(keys, i, anim_args), + sampler_name=schedule_sampler(keys, i, anim_args), + clipskip=schedule_clipskip(keys, i, anim_args), + noise_multiplier=schedule_noise_multiplier(keys, i, anim_args), + ddim_eta=schedule_ddim_eta(keys, i, anim_args), + ancestral_eta=schedule_ancestral_eta(keys, i, anim_args), + mask=schedule_mask(keys, i, args), #TODO for some reason use_mask is in args instead of anim_args + noise_mask=schedule_noise_mask(keys, i, anim_args)) def apply_animation_mode_settings(anim_args, args, loop_args, root): From fa05a2e5d3d63d8045c9043a025d971af84dbe78 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 17:23:01 +0200 Subject: [PATCH 008/132] Moved the new Schedule class into it's own module and prepared a new package. --- scripts/deforum_helpers/render.py | 24 +++----- .../deforum_helpers/render_tools/__init__.py | 1 + .../deforum_helpers/render_tools/schedule.py | 55 +++++++++++++++++++ 3 files changed, 63 insertions(+), 17 deletions(-) create mode 100644 scripts/deforum_helpers/render_tools/__init__.py create mode 100644 scripts/deforum_helpers/render_tools/schedule.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index a1bc4fb3c..3f393cc8a 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -56,24 +56,14 @@ from .video_audio_utilities import get_frame_name, get_next_frame, render_preview +from .render_tools import Schedule + + # TODO Temporary tuples to group some data. May be replaced later.. Srt = namedtuple('Srt', ['filename', 'frame_duration']) AnimMode = namedtuple('AnimMode', ['hybrid_frame_path', 'prev_flow']) -# TODO move elsewhere.. -class Schedule: - def __init__(self, steps, sampler_name, clipskip, noise_multiplier, ddim_eta, ancestral_eta, mask, noise_mask): - self.steps = steps - self.sampler_name = sampler_name - self.clipskip = clipskip - self.noise_multiplier = noise_multiplier - self.ddim_eta = ddim_eta - self.ancestral_eta = ancestral_eta - self.mask = mask - self.noise_mask = noise_mask - - def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) srt = prepare_subtitle_file_if_active(args.outdir, root.timestring, video_args.fps) @@ -659,8 +649,8 @@ def setup_opts(opts, schedule): opts.data["img2img_fix_steps"] = False # TODO is this ever true? set_if_not_none(opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) set_if_not_none(opts.data, "initial_noise_multiplier", schedule.noise_multiplier) - set_if_not_none(opts.data, "eta_ddim", schedule.ddim_eta) - set_if_not_none(opts.data, "eta_ancestral", schedule.ancestral_eta) + set_if_not_none(opts.data, "eta_ddim", schedule.eta_ddim) + set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) def setup_looper_arguments(loop_args, loop_schedules_and_data, i): @@ -745,8 +735,8 @@ def apply_scheduling(keys, i, anim_args, args): sampler_name=schedule_sampler(keys, i, anim_args), clipskip=schedule_clipskip(keys, i, anim_args), noise_multiplier=schedule_noise_multiplier(keys, i, anim_args), - ddim_eta=schedule_ddim_eta(keys, i, anim_args), - ancestral_eta=schedule_ancestral_eta(keys, i, anim_args), + eta_ddim=schedule_ddim_eta(keys, i, anim_args), + eta_ancestral=schedule_ancestral_eta(keys, i, anim_args), mask=schedule_mask(keys, i, args), #TODO for some reason use_mask is in args instead of anim_args noise_mask=schedule_noise_mask(keys, i, anim_args)) diff --git a/scripts/deforum_helpers/render_tools/__init__.py b/scripts/deforum_helpers/render_tools/__init__.py new file mode 100644 index 000000000..7aef82b28 --- /dev/null +++ b/scripts/deforum_helpers/render_tools/__init__.py @@ -0,0 +1 @@ +from .schedule import Schedule diff --git a/scripts/deforum_helpers/render_tools/schedule.py b/scripts/deforum_helpers/render_tools/schedule.py new file mode 100644 index 000000000..3eb7dc0a1 --- /dev/null +++ b/scripts/deforum_helpers/render_tools/schedule.py @@ -0,0 +1,55 @@ +from typing import Optional, Any + + +class Schedule: + def __init__(self, steps: int, sampler_name: str, clipskip: int, + noise_multiplier: float, eta_ddim: float, eta_ancestral: float, + mask: Optional[Any] = None, noise_mask: Optional[Any] = None): + self._steps = steps + self._sampler_name = sampler_name + self._clipskip = clipskip + self._noise_multiplier = noise_multiplier + self._eta_ddim = eta_ddim + self._eta_ancestral = eta_ancestral # TODO unify ddim- and a-eta to use one or the other, depending on sampler + self._mask = mask + self._noise_mask = noise_mask + + @property + def steps(self) -> int: + return self._steps + + @property + def sampler_name(self) -> str: + return self._sampler_name + + @property + def clipskip(self) -> int: + return self._clipskip + + @property + def noise_multiplier(self) -> float: + return self._noise_multiplier + + @property + def eta_ddim(self) -> float: + return self._eta_ddim + + @property + def eta_ancestral(self) -> float: + return self._eta_ancestral + + @property + def mask(self) -> Optional[Any]: + return self._mask + + @mask.setter + def mask(self, value): + self._mask = value + + @property + def noise_mask(self) -> Optional[Any]: + return self._noise_mask + + @noise_mask.setter + def noise_mask(self, value): + self._noise_mask = value From 49bce6fd6f6b928f48b8bf4d89dd156f12ad8016 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 18:08:24 +0200 Subject: [PATCH 009/132] Moved schedule related code to the new Schedule class and enforced proper initialization. --- scripts/deforum_helpers/render.py | 71 +--------------- .../deforum_helpers/render_tools/schedule.py | 81 ++++++++++++++++++- 2 files changed, 80 insertions(+), 72 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 3f393cc8a..34ac633a7 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -217,7 +217,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro "flow_factor": keys.hybrid_flow_factor_schedule_series[frame_idx] } - schedule = apply_scheduling(keys, frame_idx, anim_args, args) #FIXME dead code?? + schedule = Schedule.create(keys, frame_idx, anim_args, args) if args.use_mask and not anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -671,75 +671,6 @@ def prepare_subtitle_file_if_active(outdir, timestring, fps): return Srt(filename, frame_duration) -def has_schedule(keys, i): - return keys.steps_schedule_series[i] is not None - - -def has_mask_schedule(keys, i): - return keys.mask_schedule_series[i] is not None - - -def has_noise_mask_schedule(keys, i): - return keys.noise_mask_schedule_series[i] is not None - - -def use_on_cond_if_scheduled(keys, i, value, cond): - return value if cond and has_schedule(keys, i) else None - - -def schedule_steps(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, int(keys.steps_schedule_series[i]), - anim_args.enable_steps_scheduling) - - -def schedule_sampler(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, keys.sampler_schedule_series[i].casefold(), - anim_args.enable_sampler_scheduling) - - -def schedule_clipskip(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, int(keys.clipskip_schedule_series[i]), - anim_args.enable_clipskip_scheduling) - - -def schedule_noise_multiplier(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, float(keys.noise_multiplier_schedule_series[i]), - anim_args.enable_noise_multiplier_scheduling) - - -def schedule_ddim_eta(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, float(keys.ddim_eta_schedule_series[i]), - anim_args.enable_ddim_eta_scheduling) - - -def schedule_ancestral_eta(keys, i, anim_args): - return use_on_cond_if_scheduled(keys, i, float(keys.ancestral_eta_schedule_series[i]), - anim_args.enable_ancestral_eta_scheduling) - - -def schedule_mask(keys, i, args): - #TODO can we have a mask schedule without a normal schedule? if so check and optimize - return keys.mask_schedule_series[i] \ - if args.use_mask and has_mask_schedule(keys, i) else None - - -def schedule_noise_mask(keys, i, anim_args): - #TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize - return keys.noise_mask_schedule_series[i] \ - if anim_args.use_noise_mask and has_noise_mask_schedule(keys, i) else None - - -def apply_scheduling(keys, i, anim_args, args): - """Apply various scheduling settings based on the current frame index.""" - return Schedule(steps=schedule_steps(keys, i, anim_args), - sampler_name=schedule_sampler(keys, i, anim_args), - clipskip=schedule_clipskip(keys, i, anim_args), - noise_multiplier=schedule_noise_multiplier(keys, i, anim_args), - eta_ddim=schedule_ddim_eta(keys, i, anim_args), - eta_ancestral=schedule_ancestral_eta(keys, i, anim_args), - mask=schedule_mask(keys, i, args), #TODO for some reason use_mask is in args instead of anim_args - noise_mask=schedule_noise_mask(keys, i, anim_args)) - def apply_animation_mode_settings(anim_args, args, loop_args, root): hybrid_frame_path = None diff --git a/scripts/deforum_helpers/render_tools/schedule.py b/scripts/deforum_helpers/render_tools/schedule.py index 3eb7dc0a1..2f5606e1e 100644 --- a/scripts/deforum_helpers/render_tools/schedule.py +++ b/scripts/deforum_helpers/render_tools/schedule.py @@ -11,8 +11,85 @@ def __init__(self, steps: int, sampler_name: str, clipskip: int, self._noise_multiplier = noise_multiplier self._eta_ddim = eta_ddim self._eta_ancestral = eta_ancestral # TODO unify ddim- and a-eta to use one or the other, depending on sampler - self._mask = mask - self._noise_mask = noise_mask + self.mask = mask + self.noise_mask = noise_mask + + def __new__(cls, *args, **kwargs): # locks the normal constructor to enforce proper initialization + raise TypeError("Use Schedule.create() to create new instances.") + + @classmethod + def create(cls, keys, i, anim_args, args): + """Create a new Schedule instance based on the provided parameters.""" + steps = cls.schedule_steps(keys, i, anim_args) + sampler_name = cls.schedule_sampler(keys, i, anim_args) + clipskip = cls.schedule_clipskip(keys, i, anim_args) + noise_multiplier = cls.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = cls.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = cls.schedule_ancestral_eta(keys, i, anim_args) + mask = cls.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args + noise_mask = cls.schedule_noise_mask(keys, i, anim_args) + + instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. + instance.__init__(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) + return instance + + @classmethod + def _has_schedule(cls, keys, i): + return keys.steps_schedule_series[i] is not None + + @classmethod + def _has_mask_schedule(cls, keys, i): + return keys.mask_schedule_series[i] is not None + + @classmethod + def _has_noise_mask_schedule(cls, keys, i): + return keys.noise_mask_schedule_series[i] is not None + + @classmethod + def _use_on_cond_if_scheduled(cls, keys, i, value, cond): + return value if cond and cls._has_schedule(keys, i) else None + + @classmethod + def schedule_steps(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, int(keys.steps_schedule_series[i]), + anim_args.enable_steps_scheduling) + + @classmethod + def schedule_sampler(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, keys.sampler_schedule_series[i].casefold(), + anim_args.enable_sampler_scheduling) + + @classmethod + def schedule_clipskip(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, int(keys.clipskip_schedule_series[i]), + anim_args.enable_clipskip_scheduling) + + @classmethod + def schedule_noise_multiplier(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, float(keys.noise_multiplier_schedule_series[i]), + anim_args.enable_noise_multiplier_scheduling) + + @classmethod + def schedule_ddim_eta(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, float(keys.ddim_eta_schedule_series[i]), + anim_args.enable_ddim_eta_scheduling) + + @classmethod + def schedule_ancestral_eta(cls, keys, i, anim_args): + return cls._use_on_cond_if_scheduled(keys, i, float(keys.ancestral_eta_schedule_series[i]), + anim_args.enable_ancestral_eta_scheduling) + + @classmethod + def schedule_mask(cls, keys, i, args): + # TODO can we have a mask schedule without a normal schedule? if so check and optimize + return keys.mask_schedule_series[i] \ + if args.use_mask and cls._has_mask_schedule(keys, i) else None + + @classmethod + def schedule_noise_mask(cls, keys, i, anim_args): + #TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize + return keys.noise_mask_schedule_series[i] \ + if anim_args.use_noise_mask and cls._has_noise_mask_schedule(keys, i) else None @property def steps(self) -> int: From 4ef2432329640f58794f4315fb73f3d273c136e2 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 1 Jun 2024 19:41:14 +0200 Subject: [PATCH 010/132] Replaced tuples with proper classes. --- scripts/deforum_helpers/render.py | 23 +++-------------- .../deforum_helpers/render_tools/__init__.py | 2 ++ .../render_tools/animation_mode.py | 13 ++++++++++ scripts/deforum_helpers/render_tools/srt.py | 25 +++++++++++++++++++ 4 files changed, 44 insertions(+), 19 deletions(-) create mode 100644 scripts/deforum_helpers/render_tools/animation_mode.py create mode 100644 scripts/deforum_helpers/render_tools/srt.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 34ac633a7..dbb3eb9b8 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -18,7 +18,6 @@ import os import random import time -from collections import namedtuple import PIL import cv2 @@ -52,21 +51,16 @@ from .save_images import save_image from .seed import next_seed from .settings import save_settings_from_animation_run -from .subtitle_handler import init_srt_file, write_frame_subtitle, format_animation_params +from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -from .render_tools import Schedule - - -# TODO Temporary tuples to group some data. May be replaced later.. -Srt = namedtuple('Srt', ['filename', 'frame_duration']) -AnimMode = namedtuple('AnimMode', ['hybrid_frame_path', 'prev_flow']) +from .render_tools import AnimationMode, Schedule, Srt def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) - srt = prepare_subtitle_file_if_active(args.outdir, root.timestring, video_args.fps) + srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) anim_mode, args, anim_args, inputfiles = apply_animation_mode_settings(anim_args, args, loop_args, root) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) keys, loop_schedules_and_data = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) @@ -663,15 +657,6 @@ def setup_looper_arguments(loop_args, loop_schedules_and_data, i): loop_args.imagesToKeyframe = loop_schedules_and_data.imagesToKeyframe -def prepare_subtitle_file_if_active(outdir, timestring, fps): - if opts.data.get("deforum_save_gen_info_as_srt", False): - # create .srt file and set timeframe mechanism using FPS - filename = os.path.join(outdir, f"{timestring}.srt") - frame_duration = init_srt_file(filename, fps) - return Srt(filename, frame_duration) - - - def apply_animation_mode_settings(anim_args, args, loop_args, root): hybrid_frame_path = None prev_flow = None @@ -694,7 +679,7 @@ def apply_animation_mode_settings(anim_args, args, loop_args, root): args.seed_behavior = "schedule" if not isJson(loop_args.init_images): raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") - return AnimMode(hybrid_frame_path, prev_flow), args, anim_args, inputfiles + return AnimationMode(hybrid_frame_path, prev_flow), args, anim_args, inputfiles def initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args): diff --git a/scripts/deforum_helpers/render_tools/__init__.py b/scripts/deforum_helpers/render_tools/__init__.py index 7aef82b28..ddd762c76 100644 --- a/scripts/deforum_helpers/render_tools/__init__.py +++ b/scripts/deforum_helpers/render_tools/__init__.py @@ -1 +1,3 @@ +from .animation_mode import AnimationMode from .schedule import Schedule +from .srt import Srt diff --git a/scripts/deforum_helpers/render_tools/animation_mode.py b/scripts/deforum_helpers/render_tools/animation_mode.py new file mode 100644 index 000000000..2d7b5fedc --- /dev/null +++ b/scripts/deforum_helpers/render_tools/animation_mode.py @@ -0,0 +1,13 @@ + +class AnimationMode: + def __init__(self, filename, prev_flow): + self._filename = filename + self._prev_flow = prev_flow + + @property + def filename(self): + return self._filename + + @property + def prev_flow(self): + return self._prev_flow diff --git a/scripts/deforum_helpers/render_tools/srt.py b/scripts/deforum_helpers/render_tools/srt.py new file mode 100644 index 000000000..b24eda9cc --- /dev/null +++ b/scripts/deforum_helpers/render_tools/srt.py @@ -0,0 +1,25 @@ +import os + +from ..subtitle_handler import init_srt_file + + +class Srt: + def __init__(self, filename: str, frame_duration: int): + self._filename = filename + self._frame_duration = frame_duration + + def __new__(cls, *args, **kwargs): # locks the constructor to enforce proper initialization + raise TypeError("Use Srt.create_if_active() to create new instances.") + + @classmethod + def create_if_active(cls, opts_data, outdir: str, timestring: str, fps: float) -> 'Srt | None': + if not opts_data.get("deforum_save_gen_info_as_srt", False): + return None + else: + # create .srt file and set timeframe mechanism using FPS + filename = os.path.join(outdir, f"{timestring}.srt") + frame_duration = init_srt_file(filename, fps) + + instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. + instance.__init__(filename, frame_duration) + return instance From c40c8b593c551e52222467597b54fdb243503c71 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 00:42:16 +0200 Subject: [PATCH 011/132] Prepared new dataclass for key collections. --- .../deforum_helpers/animation_key_frames.py | 19 ++- scripts/deforum_helpers/render.py | 137 +++++++++--------- .../deforum_helpers/render_tools/schedule.py | 1 + 3 files changed, 89 insertions(+), 68 deletions(-) diff --git a/scripts/deforum_helpers/animation_key_frames.py b/scripts/deforum_helpers/animation_key_frames.py index ca4aa4310..7727df53e 100644 --- a/scripts/deforum_helpers/animation_key_frames.py +++ b/scripts/deforum_helpers/animation_key_frames.py @@ -11,7 +11,7 @@ # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . - +import dataclasses # Contact the authors: https://deforum.github.io/ import re @@ -98,6 +98,23 @@ def __init__(self, loop_args, anim_args, seed): self.tweening_frames_schedule_series = self.fi.parse_inbetweens(loop_args.tweening_frames_schedule, 'tweening_frames_schedule') self.color_correction_factor_series = self.fi.parse_inbetweens(loop_args.color_correction_factor, 'color_correction_factor') + +@dataclasses.dataclass +class AnimationKeys: + def __init__(self, deform_keys: DeformAnimKeys, looper_keys: LooperAnimKeys): + self.deform_keys = deform_keys # Not a typo. It's about deforming the frame. + self.looper_keys = looper_keys + + def update(self, i: int): + self.looper_keys.use_looper = self.looper_keys.use_looper + self.looper_keys.imagesToKeyframe = self.looper_keys.imagesToKeyframe + # TODO FIXME refactor index handling and remove new boilerplate + self.looper_keys.imageStrength = self.looper_keys.image_strength_schedule_series[i] + self.looper_keys.blendFactorMax = self.looper_keys.blendFactorMax_series[i] + self.looper_keys.blendFactorSlope = self.looper_keys.blendFactorSlope_series[i] + self.looper_keys.tweeningFramesSchedule = self.looper_keys.tweening_frames_schedule_series[i] + self.looper_keys.colorCorrectionFactor = self.looper_keys.color_correction_factor_series[i] + class FrameInterpolater(): def __init__(self, max_frames=0, seed=-1) -> None: self.max_frames = max_frames diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index dbb3eb9b8..7125eb3cf 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -31,7 +31,7 @@ from .RAFT import RAFT from .animation import anim_frame_warp -from .animation_key_frames import DeformAnimKeys, LooperAnimKeys +from .animation_key_frames import AnimationKeys, DeformAnimKeys, LooperAnimKeys from .colors import maintain_colors from .composable_masks import compose_mask_with_check from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled @@ -54,16 +54,21 @@ from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview - from .render_tools import AnimationMode, Schedule, Srt def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) anim_mode, args, anim_args, inputfiles = apply_animation_mode_settings(anim_args, args, loop_args, root) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - keys, loop_schedules_and_data = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) + + # TODO eventually try to init animation keys right after parseq adapter + # old code for temp reference: + # keys, loop_schedules_and_data = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) + animation_keys = expand_key_frame_strings_to_values(anim_args, loop_args, parseq_adapter, args.seed) + create_output_folder_for_the_batch(args) # save settings.txt file for the current run @@ -78,7 +83,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if parseq_adapter.use_parseq: anim_args.flip_2d_perspective = True - prompt_series = expand_prompts_out_to_per_frame(parseq_adapter, keys, anim_args, root) + if parseq_adapter.manages_prompts(): + prompt_series = animation_keys.deform_keys.prompts + else: + prompt_series = expand_prompts_out_to_per_frame(anim_args, root) # TODO bundle.. is_using_init_video = anim_args.animation_mode == 'Video Input' # check for video inits @@ -190,28 +198,29 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{anim_args.max_frames} ") - noise = keys.noise_schedule_series[frame_idx] - strength = keys.strength_schedule_series[frame_idx] - scale = keys.cfg_scale_schedule_series[frame_idx] - contrast = keys.contrast_schedule_series[frame_idx] - kernel = int(keys.kernel_schedule_series[frame_idx]) - sigma = keys.sigma_schedule_series[frame_idx] - amount = keys.amount_schedule_series[frame_idx] - threshold = keys.threshold_schedule_series[frame_idx] - cadence_flow_factor = keys.cadence_flow_factor_schedule_series[frame_idx] - redo_flow_factor = keys.redo_flow_factor_schedule_series[frame_idx] + # TODO move this to the new key collection + noise = animation_keys.deform_keys.noise_schedule_series[frame_idx] + strength = animation_keys.deform_keys.strength_schedule_series[frame_idx] + scale = animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] + contrast = animation_keys.deform_keys.contrast_schedule_series[frame_idx] + kernel = int(animation_keys.deform_keys.kernel_schedule_series[frame_idx]) + sigma = animation_keys.deform_keys.sigma_schedule_series[frame_idx] + amount = animation_keys.deform_keys.amount_schedule_series[frame_idx] + threshold = animation_keys.deform_keys.threshold_schedule_series[frame_idx] + cadence_flow_factor = animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] + redo_flow_factor = animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] hybrid_comp_schedules = { - "alpha": keys.hybrid_comp_alpha_schedule_series[frame_idx], - "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], - "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], + "alpha": animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], + "mask_blend_alpha": animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], + "mask_contrast": animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], "mask_auto_contrast_cutoff_low": int( - keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), + animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), "mask_auto_contrast_cutoff_high": int( - keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), - "flow_factor": keys.hybrid_flow_factor_schedule_series[frame_idx] + animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), + "flow_factor": animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] } - schedule = Schedule.create(keys, frame_idx, anim_args, args) + schedule = Schedule.create(animation_keys.deform_keys, frame_idx, anim_args, args) if args.use_mask and not anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -227,7 +236,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(keys, prompt_series, frame_idx, params_to_print) + params_string = format_animation_params(animation_keys.deform_keys, prompt_series, frame_idx, params_to_print) write_frame_subtitle(srt.filename, frame_idx, srt.frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") params_string = None @@ -247,7 +256,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # optical flow cadence setup before animation warping if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': - if keys.strength_schedule_series[tween_frame_start_idx] > 0: + if animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, anim_args.optical_flow_cadence, raft_model) / 2 @@ -255,7 +264,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(keys, prompt_series, tween_frame_idx, params_to_print) + params_string = format_animation_params(animation_keys.deform_keys, prompt_series, tween_frame_idx, params_to_print) write_frame_subtitle(srt.filename, tween_frame_idx, srt.frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None @@ -268,11 +277,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, keys, tween_frame_idx, + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, keys, tween_frame_idx, + turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -300,7 +309,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, - anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, + anim_mode.hybrid_frame_path, anim_mode.prev_flow, + prev_img, anim_args.hybrid_flow_method, raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, @@ -330,7 +340,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do optical flow cadence after animation warping if cadence_flow is not None: cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model, + cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, animation_keys.deform_keys, + tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.W, args.H) * tween @@ -384,7 +395,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, args, anim_args, keys, frame_idx, depth_model, depth=None, + prev_img, depth = anim_frame_warp(prev_img, args, anim_args, animation_keys.deform_keys, frame_idx, depth_model, depth=None, device=root.device, half_precision=root.half_precision) # do hybrid compositing before motion @@ -459,28 +470,28 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.scale = scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - args.pix2pix_img_cfg_scale = float(keys.pix2pix_img_cfg_scale_series[frame_idx]) + args.pix2pix_img_cfg_scale = float(animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) # grab prompt for current frame args.prompt = prompt_series[frame_idx] if args.seed_behavior == 'schedule' or parseq_adapter.manages_seed(): - args.seed = int(keys.seed_schedule_series[frame_idx]) + args.seed = int(animation_keys.deform_keys.seed_schedule_series[frame_idx]) if anim_args.enable_checkpoint_scheduling: - args.checkpoint = keys.checkpoint_schedule_series[frame_idx] + args.checkpoint = animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] else: args.checkpoint = None # SubSeed scheduling if anim_args.enable_subseed_scheduling: - root.subseed = int(keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = float(keys.subseed_strength_schedule_series[frame_idx]) + root.subseed = int(animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = float(animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) if parseq_adapter.manages_seed(): anim_args.enable_subseed_scheduling = True - root.subseed = int(keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = keys.subseed_strength_schedule_series[frame_idx] + root.subseed = int(animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) @@ -503,7 +514,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.mask_image = compose_mask_with_check(root, args, schedule.mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway - setup_looper_arguments(loop_args, loop_schedules_and_data, frame_idx) + animation_keys.update(frame_idx) setup_opts(opts, schedule) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): @@ -521,7 +532,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) @@ -538,7 +549,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro for n in range(0, int(anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -550,7 +561,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # generation - image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, + image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) if image is None: @@ -559,7 +570,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do hybrid video after generation if frame_idx > 0 and anim_args.hybrid_composite == 'After Generation': temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, depth_model, hybrid_comp_schedules, root) + args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, depth_model, + hybrid_comp_schedules, root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, @@ -647,16 +659,6 @@ def setup_opts(opts, schedule): set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) -def setup_looper_arguments(loop_args, loop_schedules_and_data, i): - loop_args.imageStrength = loop_schedules_and_data.image_strength_schedule_series[i] - loop_args.blendFactorMax = loop_schedules_and_data.blendFactorMax_series[i] - loop_args.blendFactorSlope = loop_schedules_and_data.blendFactorSlope_series[i] - loop_args.tweeningFrameSchedule = loop_schedules_and_data.tweening_frames_schedule_series[i] - loop_args.colorCorrectionFactor = loop_schedules_and_data.color_correction_factor_series[i] - loop_args.use_looper = loop_schedules_and_data.use_looper - loop_args.imagesToKeyframe = loop_schedules_and_data.imagesToKeyframe - - def apply_animation_mode_settings(anim_args, args, loop_args, root): hybrid_frame_path = None prev_flow = None @@ -700,11 +702,15 @@ def use_value_or_parseq_value(value, parseq_value, parseq_adapter): return value if not parseq_adapter.use_parseq else parseq_value -def expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args): - return (use_value_or_parseq_value(DeformAnimKeys(anim_args, args.seed), - parseq_adapter.anim_keys, parseq_adapter), - use_value_or_parseq_value(LooperAnimKeys(loop_args, anim_args, args.seed), - parseq_adapter.looper_keys, parseq_adapter)) +def expand_key_frame_strings_to_values(anim_args, loop_args, parseq_adapter, seed): + # TODO this should return an instance of AnimationKeys, but probably keep inheritance out of it, right? + # Parseq keys are decorated, see ParseqAnimKeysDecorator and ParseqLooperKeysDecorator + deform_keys: DeformAnimKeys = use_value_or_parseq_value(DeformAnimKeys(anim_args, seed), + parseq_adapter.anim_keys, parseq_adapter) + looper_keys: LooperAnimKeys = use_value_or_parseq_value(LooperAnimKeys(loop_args, anim_args, seed), + parseq_adapter.looper_keys, parseq_adapter) + return AnimationKeys(deform_keys, looper_keys) + def load_depth_model_for_3d(args, anim_args): @@ -714,17 +720,14 @@ def load_depth_model_for_3d(args, anim_args): return is_depth_used and not args.motion_preview_mode -def expand_prompts_out_to_per_frame(parseq_adapter, keys, anim_args, root): - if parseq_adapter.manages_prompts(): - return keys.prompts - else: - prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) - for i, prompt in root.animation_prompts.items(): - if str(i).isdigit(): - prompt_series[int(i)] = prompt - else: - prompt_series[int(numexpr.evaluate(i))] = prompt - return prompt_series.ffill().bfill() +def expand_prompts_out_to_per_frame(anim_args, root): + prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) + for i, prompt in root.animation_prompts.items(): + if str(i).isdigit(): + prompt_series[int(i)] = prompt + else: + prompt_series[int(numexpr.evaluate(i))] = prompt + return prompt_series.ffill().bfill() def is_composite_with_depth_mask(anim_args): diff --git a/scripts/deforum_helpers/render_tools/schedule.py b/scripts/deforum_helpers/render_tools/schedule.py index 2f5606e1e..209961bc2 100644 --- a/scripts/deforum_helpers/render_tools/schedule.py +++ b/scripts/deforum_helpers/render_tools/schedule.py @@ -19,6 +19,7 @@ def __new__(cls, *args, **kwargs): # locks the normal constructor to enforce pr @classmethod def create(cls, keys, i, anim_args, args): + # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" steps = cls.schedule_steps(keys, i, anim_args) sampler_name = cls.schedule_sampler(keys, i, anim_args) From b1eb3caef09b5027a2cec70f6fb6f6c9ce006486 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 00:56:55 +0200 Subject: [PATCH 012/132] Isolated agument expansion and parseq switch to static context. --- .../deforum_helpers/animation_key_frames.py | 14 ++++++++++++++ scripts/deforum_helpers/render.py | 19 +------------------ 2 files changed, 15 insertions(+), 18 deletions(-) diff --git a/scripts/deforum_helpers/animation_key_frames.py b/scripts/deforum_helpers/animation_key_frames.py index 7727df53e..d89c451ea 100644 --- a/scripts/deforum_helpers/animation_key_frames.py +++ b/scripts/deforum_helpers/animation_key_frames.py @@ -115,6 +115,20 @@ def update(self, i: int): self.looper_keys.tweeningFramesSchedule = self.looper_keys.tweening_frames_schedule_series[i] self.looper_keys.colorCorrectionFactor = self.looper_keys.color_correction_factor_series[i] + @staticmethod + def use_value_or_parseq_value(value, parseq_value, parseq_adapter): + return value if not parseq_adapter.use_parseq else parseq_value + + @staticmethod + def from_args(anim_args, loop_args, parseq_adapter, seed): + # Parseq keys are decorated, see ParseqAinmKeysDecorator and ParseqLooperKeysDecorator + new_deform_keys: DeformAnimKeys = AnimationKeys.use_value_or_parseq_value( + DeformAnimKeys(anim_args, seed), parseq_adapter.anim_keys, parseq_adapter) + new_looper_keys: LooperAnimKeys = AnimationKeys.use_value_or_parseq_value( + LooperAnimKeys(loop_args, anim_args, seed), parseq_adapter.looper_keys, parseq_adapter) + return AnimationKeys(new_deform_keys, new_looper_keys) + + class FrameInterpolater(): def __init__(self, max_frames=0, seed=-1) -> None: self.max_frames = max_frames diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 7125eb3cf..ac2734d36 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -65,9 +65,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) # TODO eventually try to init animation keys right after parseq adapter - # old code for temp reference: - # keys, loop_schedules_and_data = expand_key_frame_strings_to_values(anim_args, args, parseq_adapter, loop_args) - animation_keys = expand_key_frame_strings_to_values(anim_args, loop_args, parseq_adapter, args.seed) + animation_keys = AnimationKeys.from_args(anim_args, loop_args, parseq_adapter, args.seed) create_output_folder_for_the_batch(args) @@ -698,21 +696,6 @@ def create_output_folder_for_the_batch(args): print(f"Saving animation frames to:\n{args.outdir}") -def use_value_or_parseq_value(value, parseq_value, parseq_adapter): - return value if not parseq_adapter.use_parseq else parseq_value - - -def expand_key_frame_strings_to_values(anim_args, loop_args, parseq_adapter, seed): - # TODO this should return an instance of AnimationKeys, but probably keep inheritance out of it, right? - # Parseq keys are decorated, see ParseqAnimKeysDecorator and ParseqLooperKeysDecorator - deform_keys: DeformAnimKeys = use_value_or_parseq_value(DeformAnimKeys(anim_args, seed), - parseq_adapter.anim_keys, parseq_adapter) - looper_keys: LooperAnimKeys = use_value_or_parseq_value(LooperAnimKeys(loop_args, anim_args, seed), - parseq_adapter.looper_keys, parseq_adapter) - return AnimationKeys(deform_keys, looper_keys) - - - def load_depth_model_for_3d(args, anim_args): is_depth_warped_3d = anim_args.animation_mode == '3D' and anim_args.use_depth_warping is_composite_with_depth = anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] From 36b8dde2203cb28353cad8c43586b324fa2f8643 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 01:17:30 +0200 Subject: [PATCH 013/132] Reverted all changes in animation_key_frames and moved new code to a new module in dedicated package. --- .../deforum_helpers/animation_key_frames.py | 33 +------------------ scripts/deforum_helpers/render.py | 3 +- .../deforum_helpers/render_tools/__init__.py | 1 + .../render_tools/animation_keys.py | 33 +++++++++++++++++++ 4 files changed, 36 insertions(+), 34 deletions(-) create mode 100644 scripts/deforum_helpers/render_tools/animation_keys.py diff --git a/scripts/deforum_helpers/animation_key_frames.py b/scripts/deforum_helpers/animation_key_frames.py index d89c451ea..ca4aa4310 100644 --- a/scripts/deforum_helpers/animation_key_frames.py +++ b/scripts/deforum_helpers/animation_key_frames.py @@ -11,7 +11,7 @@ # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . -import dataclasses + # Contact the authors: https://deforum.github.io/ import re @@ -98,37 +98,6 @@ def __init__(self, loop_args, anim_args, seed): self.tweening_frames_schedule_series = self.fi.parse_inbetweens(loop_args.tweening_frames_schedule, 'tweening_frames_schedule') self.color_correction_factor_series = self.fi.parse_inbetweens(loop_args.color_correction_factor, 'color_correction_factor') - -@dataclasses.dataclass -class AnimationKeys: - def __init__(self, deform_keys: DeformAnimKeys, looper_keys: LooperAnimKeys): - self.deform_keys = deform_keys # Not a typo. It's about deforming the frame. - self.looper_keys = looper_keys - - def update(self, i: int): - self.looper_keys.use_looper = self.looper_keys.use_looper - self.looper_keys.imagesToKeyframe = self.looper_keys.imagesToKeyframe - # TODO FIXME refactor index handling and remove new boilerplate - self.looper_keys.imageStrength = self.looper_keys.image_strength_schedule_series[i] - self.looper_keys.blendFactorMax = self.looper_keys.blendFactorMax_series[i] - self.looper_keys.blendFactorSlope = self.looper_keys.blendFactorSlope_series[i] - self.looper_keys.tweeningFramesSchedule = self.looper_keys.tweening_frames_schedule_series[i] - self.looper_keys.colorCorrectionFactor = self.looper_keys.color_correction_factor_series[i] - - @staticmethod - def use_value_or_parseq_value(value, parseq_value, parseq_adapter): - return value if not parseq_adapter.use_parseq else parseq_value - - @staticmethod - def from_args(anim_args, loop_args, parseq_adapter, seed): - # Parseq keys are decorated, see ParseqAinmKeysDecorator and ParseqLooperKeysDecorator - new_deform_keys: DeformAnimKeys = AnimationKeys.use_value_or_parseq_value( - DeformAnimKeys(anim_args, seed), parseq_adapter.anim_keys, parseq_adapter) - new_looper_keys: LooperAnimKeys = AnimationKeys.use_value_or_parseq_value( - LooperAnimKeys(loop_args, anim_args, seed), parseq_adapter.looper_keys, parseq_adapter) - return AnimationKeys(new_deform_keys, new_looper_keys) - - class FrameInterpolater(): def __init__(self, max_frames=0, seed=-1) -> None: self.max_frames = max_frames diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index ac2734d36..7e74d0032 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -31,7 +31,6 @@ from .RAFT import RAFT from .animation import anim_frame_warp -from .animation_key_frames import AnimationKeys, DeformAnimKeys, LooperAnimKeys from .colors import maintain_colors from .composable_masks import compose_mask_with_check from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled @@ -54,7 +53,7 @@ from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -from .render_tools import AnimationMode, Schedule, Srt +from .render_tools import AnimationKeys, AnimationMode, Schedule, Srt def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): diff --git a/scripts/deforum_helpers/render_tools/__init__.py b/scripts/deforum_helpers/render_tools/__init__.py index ddd762c76..b0ae276db 100644 --- a/scripts/deforum_helpers/render_tools/__init__.py +++ b/scripts/deforum_helpers/render_tools/__init__.py @@ -1,3 +1,4 @@ +from .animation_keys import AnimationKeys from .animation_mode import AnimationMode from .schedule import Schedule from .srt import Srt diff --git a/scripts/deforum_helpers/render_tools/animation_keys.py b/scripts/deforum_helpers/render_tools/animation_keys.py new file mode 100644 index 000000000..16bfa4872 --- /dev/null +++ b/scripts/deforum_helpers/render_tools/animation_keys.py @@ -0,0 +1,33 @@ +import dataclasses + +from ..animation_key_frames import DeformAnimKeys, LooperAnimKeys + + +@dataclasses.dataclass +class AnimationKeys: + def __init__(self, deform_keys: DeformAnimKeys, looper_keys: LooperAnimKeys): + self.deform_keys = deform_keys # Not a typo. It's about deforming the frame. + self.looper_keys = looper_keys + + def update(self, i: int): + self.looper_keys.use_looper = self.looper_keys.use_looper + self.looper_keys.imagesToKeyframe = self.looper_keys.imagesToKeyframe + # TODO FIXME refactor index handling and remove new boilerplate + self.looper_keys.imageStrength = self.looper_keys.image_strength_schedule_series[i] + self.looper_keys.blendFactorMax = self.looper_keys.blendFactorMax_series[i] + self.looper_keys.blendFactorSlope = self.looper_keys.blendFactorSlope_series[i] + self.looper_keys.tweeningFramesSchedule = self.looper_keys.tweening_frames_schedule_series[i] + self.looper_keys.colorCorrectionFactor = self.looper_keys.color_correction_factor_series[i] + + @staticmethod + def choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): + return default_keys if not parseq_adapter.use_parseq else parseq_keys + + @staticmethod + def from_args(anim_args, loop_args, parseq_adapter, seed): + # Parseq keys are decorated, see ParseqAinmKeysDecorator and ParseqLooperKeysDecorator + return AnimationKeys( + AnimationKeys.choose_default_or_parseq_keys(DeformAnimKeys(anim_args, seed), + parseq_adapter.anim_keys, parseq_adapter), + AnimationKeys.choose_default_or_parseq_keys(LooperAnimKeys(loop_args, anim_args, seed), + parseq_adapter.looper_keys, parseq_adapter)) From 1af714fc5643990951b4a888ba5262db6544252f Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 01:33:26 +0200 Subject: [PATCH 014/132] Turned all new classes into data classes and renamed the render_tools package to render_data because, really it's all just data. --- scripts/deforum_helpers/render.py | 2 +- .../{render_tools => render_data}/__init__.py | 0 .../animation_keys.py | 5 +- .../render_data/animation_mode.py | 7 +++ .../{render_tools => render_data}/schedule.py | 61 +++---------------- .../{render_tools => render_data}/srt.py | 13 ++-- .../render_tools/animation_mode.py | 13 ---- 7 files changed, 27 insertions(+), 74 deletions(-) rename scripts/deforum_helpers/{render_tools => render_data}/__init__.py (100%) rename scripts/deforum_helpers/{render_tools => render_data}/animation_keys.py (88%) create mode 100644 scripts/deforum_helpers/render_data/animation_mode.py rename scripts/deforum_helpers/{render_tools => render_data}/schedule.py (73%) rename scripts/deforum_helpers/{render_tools => render_data}/srt.py (68%) delete mode 100644 scripts/deforum_helpers/render_tools/animation_mode.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 7e74d0032..4d417aca6 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -53,7 +53,7 @@ from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -from .render_tools import AnimationKeys, AnimationMode, Schedule, Srt +from .render_data import AnimationKeys, AnimationMode, Schedule, Srt def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): diff --git a/scripts/deforum_helpers/render_tools/__init__.py b/scripts/deforum_helpers/render_data/__init__.py similarity index 100% rename from scripts/deforum_helpers/render_tools/__init__.py rename to scripts/deforum_helpers/render_data/__init__.py diff --git a/scripts/deforum_helpers/render_tools/animation_keys.py b/scripts/deforum_helpers/render_data/animation_keys.py similarity index 88% rename from scripts/deforum_helpers/render_tools/animation_keys.py rename to scripts/deforum_helpers/render_data/animation_keys.py index 16bfa4872..2e63b0231 100644 --- a/scripts/deforum_helpers/render_tools/animation_keys.py +++ b/scripts/deforum_helpers/render_data/animation_keys.py @@ -5,9 +5,8 @@ @dataclasses.dataclass class AnimationKeys: - def __init__(self, deform_keys: DeformAnimKeys, looper_keys: LooperAnimKeys): - self.deform_keys = deform_keys # Not a typo. It's about deforming the frame. - self.looper_keys = looper_keys + deform_keys: DeformAnimKeys + looper_keys: LooperAnimKeys def update(self, i: int): self.looper_keys.use_looper = self.looper_keys.use_looper diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/animation_mode.py new file mode 100644 index 000000000..f7fd9de08 --- /dev/null +++ b/scripts/deforum_helpers/render_data/animation_mode.py @@ -0,0 +1,7 @@ +import dataclasses + + +@dataclasses.dataclass +class AnimationMode: + filename: str + prev_flow: str diff --git a/scripts/deforum_helpers/render_tools/schedule.py b/scripts/deforum_helpers/render_data/schedule.py similarity index 73% rename from scripts/deforum_helpers/render_tools/schedule.py rename to scripts/deforum_helpers/render_data/schedule.py index 209961bc2..90a6a7458 100644 --- a/scripts/deforum_helpers/render_tools/schedule.py +++ b/scripts/deforum_helpers/render_data/schedule.py @@ -1,18 +1,17 @@ +import dataclasses from typing import Optional, Any +@dataclasses.dataclass class Schedule: - def __init__(self, steps: int, sampler_name: str, clipskip: int, - noise_multiplier: float, eta_ddim: float, eta_ancestral: float, - mask: Optional[Any] = None, noise_mask: Optional[Any] = None): - self._steps = steps - self._sampler_name = sampler_name - self._clipskip = clipskip - self._noise_multiplier = noise_multiplier - self._eta_ddim = eta_ddim - self._eta_ancestral = eta_ancestral # TODO unify ddim- and a-eta to use one or the other, depending on sampler - self.mask = mask - self.noise_mask = noise_mask + steps: int + sampler_name: str + clipskip: int + noise_multiplier: float + eta_ddim: float + eta_ancestral: float # TODO unify ddim- and a-eta to use one or the other, depending on sampler + mask: Optional[Any] + noise_mask: Optional[Any] def __new__(cls, *args, **kwargs): # locks the normal constructor to enforce proper initialization raise TypeError("Use Schedule.create() to create new instances.") @@ -91,43 +90,3 @@ def schedule_noise_mask(cls, keys, i, anim_args): #TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize return keys.noise_mask_schedule_series[i] \ if anim_args.use_noise_mask and cls._has_noise_mask_schedule(keys, i) else None - - @property - def steps(self) -> int: - return self._steps - - @property - def sampler_name(self) -> str: - return self._sampler_name - - @property - def clipskip(self) -> int: - return self._clipskip - - @property - def noise_multiplier(self) -> float: - return self._noise_multiplier - - @property - def eta_ddim(self) -> float: - return self._eta_ddim - - @property - def eta_ancestral(self) -> float: - return self._eta_ancestral - - @property - def mask(self) -> Optional[Any]: - return self._mask - - @mask.setter - def mask(self, value): - self._mask = value - - @property - def noise_mask(self) -> Optional[Any]: - return self._noise_mask - - @noise_mask.setter - def noise_mask(self, value): - self._noise_mask = value diff --git a/scripts/deforum_helpers/render_tools/srt.py b/scripts/deforum_helpers/render_data/srt.py similarity index 68% rename from scripts/deforum_helpers/render_tools/srt.py rename to scripts/deforum_helpers/render_data/srt.py index b24eda9cc..2032f6fe3 100644 --- a/scripts/deforum_helpers/render_tools/srt.py +++ b/scripts/deforum_helpers/render_data/srt.py @@ -1,12 +1,13 @@ +import dataclasses import os from ..subtitle_handler import init_srt_file +@dataclasses.dataclass class Srt: - def __init__(self, filename: str, frame_duration: int): - self._filename = filename - self._frame_duration = frame_duration + filename: str + frame_duration: int def __new__(cls, *args, **kwargs): # locks the constructor to enforce proper initialization raise TypeError("Use Srt.create_if_active() to create new instances.") @@ -17,9 +18,9 @@ def create_if_active(cls, opts_data, outdir: str, timestring: str, fps: float) - return None else: # create .srt file and set timeframe mechanism using FPS - filename = os.path.join(outdir, f"{timestring}.srt") - frame_duration = init_srt_file(filename, fps) + intial_filename = os.path.join(outdir, f"{timestring}.srt") + intial_frame_duration = init_srt_file(intial_filename, fps) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(filename, frame_duration) + instance.__init__(intial_filename, intial_frame_duration) return instance diff --git a/scripts/deforum_helpers/render_tools/animation_mode.py b/scripts/deforum_helpers/render_tools/animation_mode.py deleted file mode 100644 index 2d7b5fedc..000000000 --- a/scripts/deforum_helpers/render_tools/animation_mode.py +++ /dev/null @@ -1,13 +0,0 @@ - -class AnimationMode: - def __init__(self, filename, prev_flow): - self._filename = filename - self._prev_flow = prev_flow - - @property - def filename(self): - return self._filename - - @property - def prev_flow(self): - return self._prev_flow From 4240f755d26b460d89b8fe21e84fa5731b2aa949 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 02:38:28 +0200 Subject: [PATCH 015/132] Moved animation mode initialization to static context, trying to reduce side effects and to limit scopes. --- scripts/deforum_helpers/render.py | 83 +++++++++---------- .../render_data/animation_mode.py | 36 +++++++- 2 files changed, 72 insertions(+), 47 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 4d417aca6..bbef07afb 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -46,6 +46,7 @@ from .noise import add_noise from .parseq_adapter import ParseqAdapter from .prompt import prepare_prompt +from .render_data import AnimationKeys, AnimationMode, Schedule, Srt from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed @@ -53,20 +54,16 @@ from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -from .render_data import AnimationKeys, AnimationMode, Schedule, Srt - def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) - - srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) - anim_mode, args, anim_args, inputfiles = apply_animation_mode_settings(anim_args, args, loop_args, root) - handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - - # TODO eventually try to init animation keys right after parseq adapter animation_keys = AnimationKeys.from_args(anim_args, loop_args, parseq_adapter, args.seed) + srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) + anim_mode = AnimationMode.from_args(anim_args, args, root) # possible side effects on anim_args and args - create_output_folder_for_the_batch(args) + init_looper_if_active(args, loop_args) + handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + create_output_directory_for_the_batch(args) # save settings.txt file for the current run save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) @@ -274,11 +271,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, animation_keys.deform_keys, tween_frame_idx, + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, + animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, animation_keys.deform_keys, tween_frame_idx, + turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, + animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -287,7 +286,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), - inputfiles, prev_img, anim_args.hybrid_motion) + anim_mode.hybrid_input_files, prev_img, + anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, anim_args.hybrid_motion) @@ -295,7 +295,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro turbo_next_image = image_transform_ransac(turbo_next_image, matrix, anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, + matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, @@ -305,7 +306,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, + flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, @@ -320,7 +322,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro hybrid_comp_schedules['flow_factor']) anim_mode.prev_flow = flow else: - flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, + flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_mode.hybrid_frame_path, anim_mode.prev_flow, anim_args.hybrid_flow_method, raft_model, anim_args.hybrid_flow_consistency, @@ -403,22 +406,26 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, prev_img, + matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, prev_img, anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, + matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, + flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) else: - flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, + flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), + anim_mode.hybrid_input_files, anim_mode.hybrid_frame_path, anim_mode.prev_flow, anim_args.hybrid_flow_method, raft_model, anim_args.hybrid_flow_consistency, @@ -656,41 +663,27 @@ def setup_opts(opts, schedule): set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) -def apply_animation_mode_settings(anim_args, args, loop_args, root): - hybrid_frame_path = None - prev_flow = None - inputfiles = None - if anim_args.animation_mode in ['2D', '3D']: - # handle hybrid video generation - if anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']: - args, anim_args, inputfiles = hybrid_generation(args, anim_args, root) - # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False - hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') - # initialize prev_flow - if anim_args.hybrid_motion == 'Optical Flow': - prev_flow = None #TODO prev_flow is always set to None in here, so probably remove this logic - - if loop_args.use_looper: - print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") - if args.strength == 0: - raise RuntimeError("Strength needs to be greater than 0 in Init tab") - args.strength_0_no_init = False - args.seed_behavior = "schedule" - if not isJson(loop_args.init_images): - raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") - return AnimationMode(hybrid_frame_path, prev_flow), args, anim_args, inputfiles - - def initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args): return ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) +def init_looper_if_active(args, loop_args): + if loop_args.use_looper: + print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") + if args.strength == 0: + raise RuntimeError("Strength needs to be greater than 0 in Init tab") + args.strength_0_no_init = False + args.seed_behavior = "schedule" + if not isJson(loop_args.init_images): + raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") + + def handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args): if is_controlnet_enabled(controlnet_args): unpack_controlnet_vids(args, anim_args, controlnet_args) -def create_output_folder_for_the_batch(args): +def create_output_directory_for_the_batch(args): os.makedirs(args.outdir, exist_ok=True) print(f"Saving animation frames to:\n{args.outdir}") diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/animation_mode.py index f7fd9de08..bf390c8c0 100644 --- a/scripts/deforum_helpers/render_data/animation_mode.py +++ b/scripts/deforum_helpers/render_data/animation_mode.py @@ -1,7 +1,39 @@ import dataclasses +import os + +from typing import Any +from ..generate import isJson +from ..hybrid_video import hybrid_generation @dataclasses.dataclass class AnimationMode: - filename: str - prev_flow: str + hybrid_input_files: Any = None + hybrid_frame_path: str = None + prev_flow: Any = None + + @staticmethod + def _is_2d_or_3d_mode(anim_args): + return anim_args.animation_mode in ['2D', '3D'] + + @staticmethod + def _is_using_hybris_frames(anim_args): + return (anim_args.hybrid_composite != 'None' + or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']) + + @staticmethod + def _is_needing_hybris_frames(anim_args): + return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) + + @staticmethod + def from_args(anim_args, args, root): + init_prev_flow = None + init_hybrid_input_files: Any = None + init_hybrid_frame_path = None + if AnimationMode._is_needing_hybris_frames(anim_args): + # handle hybrid video generation + # hybrid_generation may cause side effects on args and anim_args + _, __, init_hybrid_input_files = hybrid_generation(args, anim_args, root) + # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False + init_hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') + return AnimationMode(init_hybrid_input_files, init_hybrid_frame_path, init_prev_flow) From 81c3b94fa2b484b916fd0c575bc8b1adf06f8927 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 03:33:55 +0200 Subject: [PATCH 016/132] More code consolidation. --- scripts/deforum_helpers/render.py | 63 +++++++++---------- .../render_data/animation_mode.py | 25 ++++++-- 2 files changed, 49 insertions(+), 39 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index bbef07afb..14617a895 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -36,8 +36,7 @@ from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from .depth import DepthModel from .generate import generate, isJson -from .hybrid_video import ( - hybrid_generation, hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, +from .hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask @@ -60,34 +59,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro animation_keys = AnimationKeys.from_args(anim_args, loop_args, parseq_adapter, args.seed) srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) anim_mode = AnimationMode.from_args(anim_args, args, root) # possible side effects on anim_args and args - init_looper_if_active(args, loop_args) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) create_output_directory_for_the_batch(args) - - # save settings.txt file for the current run - save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - - # resume from timestring - if anim_args.resume_from_timestring: - root.timestring = anim_args.resume_timestring - - # Always enable pseudo-3d with parseq. No need for an extra toggle: - # Whether it's used or not in practice is defined by the schedules - if parseq_adapter.use_parseq: - anim_args.flip_2d_perspective = True - - if parseq_adapter.manages_prompts(): - prompt_series = animation_keys.deform_keys.prompts - else: - prompt_series = expand_prompts_out_to_per_frame(anim_args, root) + save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + root.timestring = maybe_resume_from_timestring(anim_args, root.timestring) + prompt_series = select_prompts(parseq_adapter, anim_args, animation_keys, root) # TODO bundle.. - is_using_init_video = anim_args.animation_mode == 'Video Input' # check for video inits - is_predicting_depths = load_depth_model_for_3d(args, anim_args) is_keep_in_vram = None - if is_predicting_depths: + if anim_mode.is_predicting_depths: is_keep_in_vram = opts.data.get("deforum_keep_3d_models_in_vram") device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) depth_model = DepthModel(root.models_path, device, root.half_precision, keep_in_vram=is_keep_in_vram, @@ -104,7 +86,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro is_raft_active, raft_model = load_raft(args, anim_args) # state for interpolating between diffusion steps - turbo_steps = 1 if is_using_init_video else int(anim_args.diffusion_cadence) + turbo_steps = 1 if anim_mode.has_video_input else int(anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -501,7 +483,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) # grab init image for current frame - if is_using_init_video: + if anim_mode.has_video_input: init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") args.init_image = init_frame @@ -601,7 +583,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not is_using_init_video: + if not anim_mode.has_video_input: prev_img = opencv_image if turbo_steps > 1: @@ -630,7 +612,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame) update_tracker(root, frame_idx, anim_args) - cleanup(is_predicting_depths, is_keep_in_vram, depth_model, is_raft_active, raft_model) + cleanup(anim_mode.is_predicting_depths, is_keep_in_vram, depth_model, is_raft_active, raft_model) def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): @@ -664,7 +646,12 @@ def setup_opts(opts, schedule): def initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args): - return ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + # Always enable pseudo-3d with parseq. No need for an extra toggle: + # Whether it's used or not in practice is defined by the schedules + adapter = ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + if adapter.use_parseq: + anim_args.flip_2d_perspective = True + return adapter def init_looper_if_active(args, loop_args): @@ -688,11 +675,17 @@ def create_output_directory_for_the_batch(args): print(f"Saving animation frames to:\n{args.outdir}") -def load_depth_model_for_3d(args, anim_args): - is_depth_warped_3d = anim_args.animation_mode == '3D' and anim_args.use_depth_warping - is_composite_with_depth = anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] - is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth - return is_depth_used and not args.motion_preview_mode +def save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): + save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + + +def maybe_resume_from_timestring(anim_args, current_value): + return anim_args.resume_timestring if anim_args.resume_from_timestring else current_value + + +def select_prompts(parseq_adapter, anim_args, animation_keys, root): + return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ + else expand_prompts_out_to_per_frame(anim_args, root) def expand_prompts_out_to_per_frame(anim_args, root): @@ -729,8 +722,8 @@ def update_tracker(root, frame_idx, anim_args): JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) -def cleanup(predict_depths, keep_in_vram, depth_model, is_load_raft, raft_model): - if predict_depths and not keep_in_vram: +def cleanup(is_predicting_depths, keep_in_vram, depth_model, is_load_raft, raft_model): + if is_predicting_depths and not keep_in_vram: depth_model.delete_model() # handles adabins too if is_load_raft: raft_model.delete_model() diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/animation_mode.py index bf390c8c0..bfa609a06 100644 --- a/scripts/deforum_helpers/render_data/animation_mode.py +++ b/scripts/deforum_helpers/render_data/animation_mode.py @@ -2,15 +2,22 @@ import os from typing import Any -from ..generate import isJson from ..hybrid_video import hybrid_generation @dataclasses.dataclass class AnimationMode: + has_video_input: bool = False hybrid_input_files: Any = None hybrid_frame_path: str = None prev_flow: Any = None + is_predicting_depths: Any = None + + + @staticmethod + def _has_video_input(anim_args) -> bool: + return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) + @staticmethod def _is_2d_or_3d_mode(anim_args): @@ -25,15 +32,25 @@ def _is_using_hybris_frames(anim_args): def _is_needing_hybris_frames(anim_args): return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) + @staticmethod + def _is_load_depth_model_for_3d(args, anim_args): + is_depth_warped_3d = anim_args.animation_mode == '3D' and anim_args.use_depth_warping + is_composite_with_depth = anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] + is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth + return is_depth_used and not args.motion_preview_mode + @staticmethod def from_args(anim_args, args, root): - init_prev_flow = None init_hybrid_input_files: Any = None - init_hybrid_frame_path = None if AnimationMode._is_needing_hybris_frames(anim_args): # handle hybrid video generation # hybrid_generation may cause side effects on args and anim_args _, __, init_hybrid_input_files = hybrid_generation(args, anim_args, root) # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False init_hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') - return AnimationMode(init_hybrid_input_files, init_hybrid_frame_path, init_prev_flow) + + return AnimationMode(AnimationMode._has_video_input(anim_args), + init_hybrid_input_files, + None, + None, + AnimationMode._is_load_depth_model_for_3d(args, anim_args)) From 021a3f4c786d57b4f8f2cfe5712069069e702634 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 04:31:29 +0200 Subject: [PATCH 017/132] Moved raft- and depth map initialization into AnimationMode. --- scripts/deforum_helpers/render.py | 76 ++++++++----------- .../render_data/animation_mode.py | 46 +++++++++-- 2 files changed, 70 insertions(+), 52 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 14617a895..c14c8f436 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -29,7 +29,6 @@ from modules import lowvram, devices, sd_hijack from modules.shared import opts, cmd_opts, state, sd_model -from .RAFT import RAFT from .animation import anim_frame_warp from .colors import maintain_colors from .composable_masks import compose_mask_with_check @@ -37,8 +36,9 @@ from .depth import DepthModel from .generate import generate, isJson from .hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, - get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, - image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, rel_flow_to_abs_flow) + get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, + image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, + rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask from .load_images import get_mask, load_img, load_image, get_mask_from_file from .masks import do_overlay_mask @@ -58,32 +58,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) animation_keys = AnimationKeys.from_args(anim_args, loop_args, parseq_adapter, args.seed) srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) - anim_mode = AnimationMode.from_args(anim_args, args, root) # possible side effects on anim_args and args + anim_mode = AnimationMode.from_args(anim_args, args, opts, root) # possible side effects on anim_args and args init_looper_if_active(args, loop_args) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) create_output_directory_for_the_batch(args) save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) root.timestring = maybe_resume_from_timestring(anim_args, root.timestring) prompt_series = select_prompts(parseq_adapter, anim_args, animation_keys, root) - - # TODO bundle.. - is_keep_in_vram = None - - if anim_mode.is_predicting_depths: - is_keep_in_vram = opts.data.get("deforum_keep_3d_models_in_vram") - device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) - depth_model = DepthModel(root.models_path, device, root.half_precision, keep_in_vram=is_keep_in_vram, - depth_algorithm=anim_args.depth_algorithm, Width=args.W, Height=args.H, - midas_weight=anim_args.midas_weight) - - # depth-based hybrid composite mask requires saved depth maps - if is_composite_with_depth_mask(anim_args): - anim_args.save_depth_maps = True - else: - depth_model = None - anim_args.save_depth_maps = False - - is_raft_active, raft_model = load_raft(args, anim_args) + depth_model = create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args) # state for interpolating between diffusion steps turbo_steps = 1 if anim_mode.has_video_input else int(anim_args.diffusion_cadence) @@ -212,7 +194,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(animation_keys.deform_keys, prompt_series, frame_idx, params_to_print) + params_string = format_animation_params(animation_keys.deform_keys, prompt_series, frame_idx, + params_to_print) write_frame_subtitle(srt.filename, frame_idx, srt.frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") params_string = None @@ -240,7 +223,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(animation_keys.deform_keys, prompt_series, tween_frame_idx, params_to_print) + params_string = format_animation_params(animation_keys.deform_keys, prompt_series, tween_frame_idx, + params_to_print) write_frame_subtitle(srt.filename, tween_frame_idx, srt.frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None @@ -377,7 +361,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, args, anim_args, animation_keys.deform_keys, frame_idx, depth_model, depth=None, + prev_img, depth = anim_frame_warp(prev_img, args, anim_args, animation_keys.deform_keys, frame_idx, + depth_model, depth=None, device=root.device, half_precision=root.half_precision) # do hybrid compositing before motion @@ -504,7 +489,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro setup_opts(opts, schedule) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): - if is_predicting_depths: depth_model.to('cpu') + if anim_mode.is_predicting_depths: depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) @@ -518,7 +503,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, + parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) @@ -535,7 +521,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro for n in range(0, int(anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + root, parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -547,7 +534,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # generation - image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, + image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, + frame_idx, sampler_name=schedule.sampler_name) if image is None: @@ -612,7 +600,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame) update_tracker(root, frame_idx, anim_args) - cleanup(anim_mode.is_predicting_depths, is_keep_in_vram, depth_model, is_raft_active, raft_model) + anim_mode.cleanup() def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): @@ -702,14 +690,18 @@ def is_composite_with_depth_mask(anim_args): return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' -def load_raft(args, anim_args): - is_cadenced_raft = anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1 - is_optical_flow_raft = anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT" - is_raft_redo = anim_args.optical_flow_redo_generation == "RAFT" - is_load_raft = (is_cadenced_raft or is_optical_flow_raft or is_raft_redo) and not args.motion_preview_mode - if is_load_raft: - print("Loading RAFT model...") - return is_load_raft, RAFT() if is_load_raft else None +def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args): + # depth-based hybrid composite mask requires saved depth maps + anim_args.save_depth_maps = anim_mode.is_predicting_depths and is_composite_with_depth_mask(anim_args) + if anim_mode.is_predicting_depths: + depth_device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) + return DepthModel(root.models_path, depth_device, root.half_precision, + keep_in_vram=anim_mode.is_keep_in_vram, + depth_algorithm=anim_args.depth_algorithm, + Width=args.W, Height=args.H, + midas_weight=anim_args.midas_weight) + else: + return None def progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame): @@ -721,9 +713,3 @@ def progress_and_make_preview(state, image, args, anim_args, video_args, root, f def update_tracker(root, frame_idx, anim_args): JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) - -def cleanup(is_predicting_depths, keep_in_vram, depth_model, is_load_raft, raft_model): - if is_predicting_depths and not keep_in_vram: - depth_model.delete_model() # handles adabins too - if is_load_raft: - raft_model.delete_model() diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/animation_mode.py index bfa609a06..470dc44a9 100644 --- a/scripts/deforum_helpers/render_data/animation_mode.py +++ b/scripts/deforum_helpers/render_data/animation_mode.py @@ -3,22 +3,36 @@ from typing import Any from ..hybrid_video import hybrid_generation +from ..RAFT import RAFT @dataclasses.dataclass class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None - hybrid_frame_path: str = None + hybrid_frame_path: str = "" prev_flow: Any = None - is_predicting_depths: Any = None + is_keep_in_vram: bool = False + depth_model: Any = None + raft_model: Any = None + + def is_predicting_depths(self) -> bool: + return self.depth_model is not None + + def is_raft_active(self) -> bool: + return self.raft_model is not None + + def cleanup(self): + if self.is_predicting_depths() and not self.is_keep_in_vram: + self.depth_model.delete_model() # handles adabins too + if self.is_raft_active(): + self.raft_model.delete_model() @staticmethod def _has_video_input(anim_args) -> bool: return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) - @staticmethod def _is_2d_or_3d_mode(anim_args): return anim_args.animation_mode in ['2D', '3D'] @@ -39,18 +53,36 @@ def _is_load_depth_model_for_3d(args, anim_args): is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth return is_depth_used and not args.motion_preview_mode + @staticmethod - def from_args(anim_args, args, root): + def _load_raft_if_active(anim_args, args): + is_cadenced_raft = anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1 + is_optical_flow_raft = anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT" + is_raft_redo = anim_args.optical_flow_redo_generation == "RAFT" + is_load_raft = (is_cadenced_raft or is_optical_flow_raft or is_raft_redo) and not args.motion_preview_mode + if is_load_raft: + print("Loading RAFT model...") + return RAFT() if is_load_raft else None + + @staticmethod + def _load_depth_model_if_active(args, anim_args, opts): + return AnimationMode._is_load_depth_model_for_3d(args, anim_args) \ + if opts.data.get("deforum_keep_3d_models_in_vram", False) else None + + @staticmethod + def from_args(anim_args, args, opts, root): init_hybrid_input_files: Any = None + init_hybrid_frame_path = "" if AnimationMode._is_needing_hybris_frames(anim_args): # handle hybrid video generation # hybrid_generation may cause side effects on args and anim_args _, __, init_hybrid_input_files = hybrid_generation(args, anim_args, root) # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False init_hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') - return AnimationMode(AnimationMode._has_video_input(anim_args), init_hybrid_input_files, + init_hybrid_frame_path, None, - None, - AnimationMode._is_load_depth_model_for_3d(args, anim_args)) + opts.data.get("deforum_keep_3d_models_in_vram", False), + AnimationMode._load_depth_model_if_active(args, anim_args, opts), + AnimationMode._load_raft_if_active(anim_args, args)) From bc8790aaf78c743368d6f46666fd69c43706a278 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 06:25:24 +0200 Subject: [PATCH 018/132] New Step class defined. --- scripts/deforum_helpers/render.py | 184 +++++++++--------- .../deforum_helpers/render_data/__init__.py | 1 + .../render_data/animation_keys.py | 6 +- .../render_data/animation_mode.py | 16 +- scripts/deforum_helpers/render_data/srt.py | 9 +- scripts/deforum_helpers/render_data/step.py | 54 +++++ 6 files changed, 161 insertions(+), 109 deletions(-) create mode 100644 scripts/deforum_helpers/render_data/step.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index c14c8f436..331113d76 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -11,7 +11,7 @@ # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . - +import dataclasses # Contact the authors: https://deforum.github.io/ import gc @@ -45,7 +45,7 @@ from .noise import add_noise from .parseq_adapter import ParseqAdapter from .prompt import prepare_prompt -from .render_data import AnimationKeys, AnimationMode, Schedule, Srt +from .render_data import AnimationKeys, AnimationMode, Schedule, Srt, Step from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed @@ -55,20 +55,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - parseq_adapter = initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) - animation_keys = AnimationKeys.from_args(anim_args, loop_args, parseq_adapter, args.seed) - srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) - anim_mode = AnimationMode.from_args(anim_args, args, opts, root) # possible side effects on anim_args and args + step = Step.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) init_looper_if_active(args, loop_args) handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) create_output_directory_for_the_batch(args) save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) root.timestring = maybe_resume_from_timestring(anim_args, root.timestring) - prompt_series = select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args) + prompt_series = select_prompts(step.parseq_adapter, anim_args, step.animation_keys, root) + depth_model = create_depth_model_and_enable_depth_map_saving_if_active(step.animation_mode, root, anim_args, args) # state for interpolating between diffusion steps - turbo_steps = 1 if anim_mode.has_video_input else int(anim_args.diffusion_cadence) + turbo_steps = 1 if step.animation_mode.has_video_input else int(anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -117,12 +114,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 if anim_args.use_mask_video: - - args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), - args) - root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), - args) - + args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) mask_vals['video_mask'] = get_mask_from_file( get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) noise_mask_vals['video_mask'] = get_mask_from_file( @@ -157,28 +150,28 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{anim_args.max_frames} ") # TODO move this to the new key collection - noise = animation_keys.deform_keys.noise_schedule_series[frame_idx] - strength = animation_keys.deform_keys.strength_schedule_series[frame_idx] - scale = animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] - contrast = animation_keys.deform_keys.contrast_schedule_series[frame_idx] - kernel = int(animation_keys.deform_keys.kernel_schedule_series[frame_idx]) - sigma = animation_keys.deform_keys.sigma_schedule_series[frame_idx] - amount = animation_keys.deform_keys.amount_schedule_series[frame_idx] - threshold = animation_keys.deform_keys.threshold_schedule_series[frame_idx] - cadence_flow_factor = animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] - redo_flow_factor = animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] + noise = step.animation_keys.deform_keys.noise_schedule_series[frame_idx] + strength = step.animation_keys.deform_keys.strength_schedule_series[frame_idx] + scale = step.animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] + contrast = step.animation_keys.deform_keys.contrast_schedule_series[frame_idx] + kernel = int(step.animation_keys.deform_keys.kernel_schedule_series[frame_idx]) + sigma = step.animation_keys.deform_keys.sigma_schedule_series[frame_idx] + amount = step.animation_keys.deform_keys.amount_schedule_series[frame_idx] + threshold = step.animation_keys.deform_keys.threshold_schedule_series[frame_idx] + cadence_flow_factor = step.animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] + redo_flow_factor = step.animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] hybrid_comp_schedules = { - "alpha": animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], - "mask_blend_alpha": animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], - "mask_contrast": animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], + "alpha": step.animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], + "mask_blend_alpha": step.animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], + "mask_contrast": step.animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], "mask_auto_contrast_cutoff_low": int( - animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), + step.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), "mask_auto_contrast_cutoff_high": int( - animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), - "flow_factor": animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] + step.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), + "flow_factor": step.animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] } - schedule = Schedule.create(animation_keys.deform_keys, frame_idx, anim_args, args) + schedule = Schedule.create(step.animation_keys.deform_keys, frame_idx, anim_args, args) if args.use_mask and not anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -190,13 +183,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - if is_predicting_depths: depth_model.to(root.device) + if step.animation_mode.is_predicting_depths: + step.animation_mode.depth_model.to(root.device) if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(animation_keys.deform_keys, prompt_series, frame_idx, + params_string = format_animation_params(step.animation_keys.deform_keys, prompt_series, frame_idx, params_to_print) - write_frame_subtitle(srt.filename, frame_idx, srt.frame_duration, + write_frame_subtitle(step.srt.filename, frame_idx, step.srt.frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") params_string = None @@ -215,17 +209,18 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # optical flow cadence setup before animation warping if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': - if animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: + if step.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, - anim_args.optical_flow_cadence, raft_model) / 2 + anim_args.optical_flow_cadence, + step.animation_mode.raft_model) / 2 turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(animation_keys.deform_keys, prompt_series, tween_frame_idx, + params_string = format_animation_params(step.animation_keys.deform_keys, prompt_series, tween_frame_idx, params_to_print) - write_frame_subtitle(srt.filename, tween_frame_idx, srt.frame_duration, + write_frame_subtitle(step.srt.filename, tween_frame_idx, step.srt.frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None @@ -238,12 +233,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_prev: turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, - animation_keys.deform_keys, tween_frame_idx, + step.animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, - animation_keys.deform_keys, tween_frame_idx, + step.animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -252,7 +247,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, prev_img, + step.animation_mode.hybrid_input_files, prev_img, anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, @@ -262,7 +257,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, + step.animation_mode.hybrid_input_files, anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, @@ -273,10 +268,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, - anim_mode.hybrid_frame_path, anim_mode.prev_flow, + step.animation_mode.hybrid_input_files, + step.animation_mode.hybrid_frame_path, + step.animation_mode.prev_flow, prev_img, - anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_method, + step.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) @@ -286,12 +283,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_next: turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - anim_mode.prev_flow = flow + step.animation_mode.prev_flow = flow else: flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, - anim_mode.hybrid_frame_path, anim_mode.prev_flow, - anim_args.hybrid_flow_method, raft_model, + step.animation_mode.hybrid_input_files, + step.animation_mode.hybrid_frame_path, + step.animation_mode.prev_flow, + anim_args.hybrid_flow_method, + step.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) @@ -301,12 +300,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_next: turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - anim_mode.prev_flow = flow + step.animation_mode.prev_flow = flow # do optical flow cadence after animation warping if cadence_flow is not None: cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, animation_keys.deform_keys, + cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, step.animation_keys.deform_keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -348,8 +347,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro depth) # get color match for video outside of prev_img conditional - hybrid_available = anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Optical Flow', 'Affine', - 'Perspective'] + hybrid_available = (step.step_args.anim_args.hybrid_composite != 'None' + or anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) if anim_args.color_coherence == 'Video Input' and hybrid_available: if int(frame_idx) % int(anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name( @@ -361,7 +360,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, args, anim_args, animation_keys.deform_keys, frame_idx, + prev_img, depth = anim_frame_warp(prev_img, args, anim_args, + step.animation_keys.deform_keys, frame_idx, depth_model, depth=None, device=root.device, half_precision=root.half_precision) @@ -374,32 +374,36 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, prev_img, + step.animation_mode.hybrid_input_files, prev_img, anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, + step.animation_mode.hybrid_input_files, anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, - anim_mode.hybrid_frame_path, anim_mode.prev_flow, prev_img, - anim_args.hybrid_flow_method, raft_model, + step.animation_mode.hybrid_input_files, + step.animation_mode.hybrid_frame_path, + step.animation_mode.prev_flow, prev_img, + anim_args.hybrid_flow_method, + step.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) else: flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), - anim_mode.hybrid_input_files, - anim_mode.hybrid_frame_path, - anim_mode.prev_flow, anim_args.hybrid_flow_method, raft_model, + step.animation_mode.hybrid_input_files, + step.animation_mode.hybrid_frame_path, + step.animation_mode.prev_flow, + anim_args.hybrid_flow_method, + step.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) - anim_mode.prev_flow = flow + step.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if anim_args.hybrid_composite == 'Normal': @@ -441,34 +445,34 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.scale = scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - args.pix2pix_img_cfg_scale = float(animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) + args.pix2pix_img_cfg_scale = float(step.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) # grab prompt for current frame args.prompt = prompt_series[frame_idx] - if args.seed_behavior == 'schedule' or parseq_adapter.manages_seed(): - args.seed = int(animation_keys.deform_keys.seed_schedule_series[frame_idx]) + if args.seed_behavior == 'schedule' or step.parseq_adapter.manages_seed(): + args.seed = int(step.animation_keys.deform_keys.seed_schedule_series[frame_idx]) if anim_args.enable_checkpoint_scheduling: - args.checkpoint = animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] + args.checkpoint = step.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] else: args.checkpoint = None # SubSeed scheduling if anim_args.enable_subseed_scheduling: - root.subseed = int(animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = float(animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) + root.subseed = int(step.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = float(step.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) - if parseq_adapter.manages_seed(): + if step.parseq_adapter.manages_seed(): anim_args.enable_subseed_scheduling = True - root.subseed = int(animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] + root.subseed = int(step.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = step.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) # grab init image for current frame - if anim_mode.has_video_input: + if step.animation_mode.has_video_input: init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") args.init_image = init_frame @@ -485,11 +489,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.mask_image = compose_mask_with_check(root, args, schedule.mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway - animation_keys.update(frame_idx) + step.animation_keys.update(frame_idx) setup_opts(opts, schedule) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): - if anim_mode.is_predicting_depths: depth_model.to('cpu') + if step.animation_mode.is_predicting_depths: depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) @@ -503,11 +507,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, - parseq_adapter, + disposable_image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + root, step.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) - disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) + disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, + step.animation_mode.raft_model) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) args.seed = stored_seed @@ -521,8 +526,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro for n in range(0, int(anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, - root, parseq_adapter, + disposable_image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, + controlnet_args, root, step.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -534,8 +539,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # generation - image = generate(args, animation_keys.deform_keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, - frame_idx, + image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + root, step.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) if image is None: @@ -571,7 +576,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not anim_mode.has_video_input: + if not step.animation_mode.has_video_input: prev_img = opencv_image if turbo_steps > 1: @@ -600,7 +605,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame) update_tracker(root, frame_idx, anim_args) - anim_mode.cleanup() + step.animation_mode.cleanup() def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): @@ -633,15 +638,6 @@ def setup_opts(opts, schedule): set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) -def initialize_parseq_adapter(parseq_args, anim_args, video_args, controlnet_args, loop_args): - # Always enable pseudo-3d with parseq. No need for an extra toggle: - # Whether it's used or not in practice is defined by the schedules - adapter = ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) - if adapter.use_parseq: - anim_args.flip_2d_perspective = True - return adapter - - def init_looper_if_active(args, loop_args): if loop_args.use_looper: print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") diff --git a/scripts/deforum_helpers/render_data/__init__.py b/scripts/deforum_helpers/render_data/__init__.py index b0ae276db..f53e57a35 100644 --- a/scripts/deforum_helpers/render_data/__init__.py +++ b/scripts/deforum_helpers/render_data/__init__.py @@ -2,3 +2,4 @@ from .animation_mode import AnimationMode from .schedule import Schedule from .srt import Srt +from .step import Step diff --git a/scripts/deforum_helpers/render_data/animation_keys.py b/scripts/deforum_helpers/render_data/animation_keys.py index 2e63b0231..74fee4bd6 100644 --- a/scripts/deforum_helpers/render_data/animation_keys.py +++ b/scripts/deforum_helpers/render_data/animation_keys.py @@ -23,10 +23,10 @@ def choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): return default_keys if not parseq_adapter.use_parseq else parseq_keys @staticmethod - def from_args(anim_args, loop_args, parseq_adapter, seed): + def from_args(step_args, parseq_adapter, seed): # Parseq keys are decorated, see ParseqAinmKeysDecorator and ParseqLooperKeysDecorator return AnimationKeys( - AnimationKeys.choose_default_or_parseq_keys(DeformAnimKeys(anim_args, seed), + AnimationKeys.choose_default_or_parseq_keys(DeformAnimKeys(step_args.anim_args, seed), parseq_adapter.anim_keys, parseq_adapter), - AnimationKeys.choose_default_or_parseq_keys(LooperAnimKeys(loop_args, anim_args, seed), + AnimationKeys.choose_default_or_parseq_keys(LooperAnimKeys(step_args.loop_args, step_args.anim_args, seed), parseq_adapter.looper_keys, parseq_adapter)) diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/animation_mode.py index 470dc44a9..df4fca612 100644 --- a/scripts/deforum_helpers/render_data/animation_mode.py +++ b/scripts/deforum_helpers/render_data/animation_mode.py @@ -70,19 +70,19 @@ def _load_depth_model_if_active(args, anim_args, opts): if opts.data.get("deforum_keep_3d_models_in_vram", False) else None @staticmethod - def from_args(anim_args, args, opts, root): + def from_args(step_args): init_hybrid_input_files: Any = None init_hybrid_frame_path = "" - if AnimationMode._is_needing_hybris_frames(anim_args): + if AnimationMode._is_needing_hybris_frames(step_args.anim_args): # handle hybrid video generation # hybrid_generation may cause side effects on args and anim_args - _, __, init_hybrid_input_files = hybrid_generation(args, anim_args, root) + _, __, init_hybrid_input_files = hybrid_generation(step_args.args, step_args.anim_args, step_args.root) # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False - init_hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') - return AnimationMode(AnimationMode._has_video_input(anim_args), + init_hybrid_frame_path = os.path.join(step_args.args.outdir, 'hybridframes') + return AnimationMode(AnimationMode._has_video_input(step_args.anim_args), init_hybrid_input_files, init_hybrid_frame_path, None, - opts.data.get("deforum_keep_3d_models_in_vram", False), - AnimationMode._load_depth_model_if_active(args, anim_args, opts), - AnimationMode._load_raft_if_active(anim_args, args)) + step_args.opts.data.get("deforum_keep_3d_models_in_vram", False), + AnimationMode._load_depth_model_if_active(step_args.args, step_args.anim_args, step_args.opts), + AnimationMode._load_raft_if_active(step_args.anim_args, step_args.args)) diff --git a/scripts/deforum_helpers/render_data/srt.py b/scripts/deforum_helpers/render_data/srt.py index 2032f6fe3..fd9673dfc 100644 --- a/scripts/deforum_helpers/render_data/srt.py +++ b/scripts/deforum_helpers/render_data/srt.py @@ -1,13 +1,14 @@ import dataclasses import os +from decimal import Decimal from ..subtitle_handler import init_srt_file @dataclasses.dataclass class Srt: filename: str - frame_duration: int + frame_duration: Decimal def __new__(cls, *args, **kwargs): # locks the constructor to enforce proper initialization raise TypeError("Use Srt.create_if_active() to create new instances.") @@ -18,9 +19,9 @@ def create_if_active(cls, opts_data, outdir: str, timestring: str, fps: float) - return None else: # create .srt file and set timeframe mechanism using FPS - intial_filename = os.path.join(outdir, f"{timestring}.srt") - intial_frame_duration = init_srt_file(intial_filename, fps) + init_filename = os.path.join(outdir, f"{timestring}.srt") + init_frame_duration = init_srt_file(init_filename, fps) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(intial_filename, intial_frame_duration) + instance.__init__(init_filename, init_frame_duration) return instance diff --git a/scripts/deforum_helpers/render_data/step.py b/scripts/deforum_helpers/render_data/step.py new file mode 100644 index 000000000..9d400663f --- /dev/null +++ b/scripts/deforum_helpers/render_data/step.py @@ -0,0 +1,54 @@ +import dataclasses + +from typing import Any +from . import AnimationKeys, AnimationMode, Srt +from ..parseq_adapter import ParseqAdapter + + +@dataclasses.dataclass +class StepArgs: + args: Any = None + parseq_args: Any = None + anim_args: Any = None + video_args: Any = None + controlnet_args: Any = None + loop_args: Any = None + opts: Any = None + root: Any = None + + @classmethod + def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): + return StepArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + + +@dataclasses.dataclass +class Step: + seed: int + step_args: StepArgs + parseq_adapter: Any + srt: Any + animation_keys: AnimationKeys + animation_mode: AnimationMode + + def __new__(cls, *args, **kwargs): + raise TypeError("Use Step.create() to create new instances.") + + @classmethod + def create_parseq_adapter(cls, args): + adapter = ParseqAdapter(args.parseq_args, args.anim_args, args.video_args, args.controlnet_args, args.loop_args) + # Always enable pseudo-3d with parseq. No need for an extra toggle: + # Whether it's used or not in practice is defined by the schedules + if adapter.use_parseq: + args.anim_args.flip_2d_perspective = True + return adapter + + @classmethod + def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'Step': + step_args = StepArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + parseq_adapter = Step.create_parseq_adapter(step_args) + instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. + instance.__init__(args.seed, step_args, parseq_adapter, + Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps), + AnimationKeys.from_args(step_args, parseq_adapter, args.seed), + AnimationMode.from_args(step_args)) # possible side effects on anim_args and args + return instance From 53e8ce22a2e835215acf18aa2c725d89bd0f04ed Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 16:03:29 +0200 Subject: [PATCH 019/132] Created initialization module to bundle render init code and tried to isolate and group side effects. More stuff moved to static context. --- scripts/deforum_helpers/render.py | 288 +++++++----------- .../deforum_helpers/render_data/__init__.py | 4 +- .../render_data/initialization.py | 159 ++++++++++ scripts/deforum_helpers/render_data/step.py | 54 ---- 4 files changed, 270 insertions(+), 235 deletions(-) create mode 100644 scripts/deforum_helpers/render_data/initialization.py delete mode 100644 scripts/deforum_helpers/render_data/step.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 331113d76..2193509d8 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -21,9 +21,7 @@ import PIL import cv2 -import numexpr import numpy as np -import pandas as pd from PIL import Image, ImageOps from deforum_api import JobStatusTracker from modules import lowvram, devices, sd_hijack @@ -32,9 +30,7 @@ from .animation import anim_frame_warp from .colors import maintain_colors from .composable_masks import compose_mask_with_check -from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled -from .depth import DepthModel -from .generate import generate, isJson +from .generate import generate from .hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, @@ -43,29 +39,20 @@ from .load_images import get_mask, load_img, load_image, get_mask_from_file from .masks import do_overlay_mask from .noise import add_noise -from .parseq_adapter import ParseqAdapter from .prompt import prepare_prompt -from .render_data import AnimationKeys, AnimationMode, Schedule, Srt, Step +from .render_data import Schedule, RenderInit from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed -from .settings import save_settings_from_animation_run from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - step = Step.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - init_looper_if_active(args, loop_args) - handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - create_output_directory_for_the_batch(args) - save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - root.timestring = maybe_resume_from_timestring(anim_args, root.timestring) - prompt_series = select_prompts(step.parseq_adapter, anim_args, step.animation_keys, root) - depth_model = create_depth_model_and_enable_depth_map_saving_if_active(step.animation_mode, root, anim_args, args) + init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) # state for interpolating between diffusion steps - turbo_steps = 1 if step.animation_mode.has_video_input else int(anim_args.diffusion_cadence) + turbo_steps = 1 if init.animation_mode.has_video_input else int(anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -150,28 +137,28 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{anim_args.max_frames} ") # TODO move this to the new key collection - noise = step.animation_keys.deform_keys.noise_schedule_series[frame_idx] - strength = step.animation_keys.deform_keys.strength_schedule_series[frame_idx] - scale = step.animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] - contrast = step.animation_keys.deform_keys.contrast_schedule_series[frame_idx] - kernel = int(step.animation_keys.deform_keys.kernel_schedule_series[frame_idx]) - sigma = step.animation_keys.deform_keys.sigma_schedule_series[frame_idx] - amount = step.animation_keys.deform_keys.amount_schedule_series[frame_idx] - threshold = step.animation_keys.deform_keys.threshold_schedule_series[frame_idx] - cadence_flow_factor = step.animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] - redo_flow_factor = step.animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] + noise = init.animation_keys.deform_keys.noise_schedule_series[frame_idx] + strength = init.animation_keys.deform_keys.strength_schedule_series[frame_idx] + scale = init.animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] + contrast = init.animation_keys.deform_keys.contrast_schedule_series[frame_idx] + kernel = int(init.animation_keys.deform_keys.kernel_schedule_series[frame_idx]) + sigma = init.animation_keys.deform_keys.sigma_schedule_series[frame_idx] + amount = init.animation_keys.deform_keys.amount_schedule_series[frame_idx] + threshold = init.animation_keys.deform_keys.threshold_schedule_series[frame_idx] + cadence_flow_factor = init.animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] + redo_flow_factor = init.animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] hybrid_comp_schedules = { - "alpha": step.animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], - "mask_blend_alpha": step.animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], - "mask_contrast": step.animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], + "alpha": init.animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], + "mask_blend_alpha": init.animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], + "mask_contrast": init.animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], "mask_auto_contrast_cutoff_low": int( - step.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), + init.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), "mask_auto_contrast_cutoff_high": int( - step.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), - "flow_factor": step.animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] + init.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), + "flow_factor": init.animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] } - schedule = Schedule.create(step.animation_keys.deform_keys, frame_idx, anim_args, args) + schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, anim_args, args) if args.use_mask and not anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -183,16 +170,16 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - if step.animation_mode.is_predicting_depths: - step.animation_mode.depth_model.to(root.device) + if init.animation_mode.is_predicting_depths: + init.animation_mode.depth_model.to(root.device) if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(step.animation_keys.deform_keys, prompt_series, frame_idx, + params_string = format_animation_params(init.animation_keys.deform_keys, init.prompt_series, frame_idx, params_to_print) - write_frame_subtitle(step.srt.filename, frame_idx, step.srt.frame_duration, + write_frame_subtitle(init.srt.filename, frame_idx, init.srt.frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") - params_string = None + params_string = None # FIXME ?? # emit in-between frames if turbo_steps > 1: @@ -209,37 +196,38 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # optical flow cadence setup before animation warping if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': - if step.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: + if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, anim_args.optical_flow_cadence, - step.animation_mode.raft_model) / 2 + init.animation_mode.raft_model) / 2 turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(step.animation_keys.deform_keys, prompt_series, tween_frame_idx, + params_string = format_animation_params(init.animation_keys.deform_keys, + init.prompt_series, + tween_frame_idx, params_to_print) - write_frame_subtitle(step.srt.filename, tween_frame_idx, step.srt.frame_duration, + write_frame_subtitle(init.srt.filename, tween_frame_idx, init.srt.frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None - print( - f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") + print(f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") - if depth_model is not None: + if init.depth_model is not None: assert (turbo_next_image is not None) - depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) + depth = init.depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) if advance_prev: turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, - step.animation_keys.deform_keys, tween_frame_idx, - depth_model, depth=depth, device=root.device, + init.animation_keys.deform_keys, tween_frame_idx, + init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, - step.animation_keys.deform_keys, tween_frame_idx, - depth_model, depth=depth, device=root.device, + init.animation_keys.deform_keys, tween_frame_idx, + init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion @@ -247,7 +235,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, prev_img, + init.animation_mode.hybrid_input_files, prev_img, anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, @@ -257,7 +245,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_input_files, anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, @@ -268,12 +256,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, - step.animation_mode.hybrid_frame_path, - step.animation_mode.prev_flow, + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, prev_img, anim_args.hybrid_flow_method, - step.animation_mode.raft_model, + init.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) @@ -283,14 +271,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_next: turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - step.animation_mode.prev_flow = flow + init.animation_mode.prev_flow = flow else: flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, - step.animation_mode.hybrid_frame_path, - step.animation_mode.prev_flow, + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, anim_args.hybrid_flow_method, - step.animation_mode.raft_model, + init.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) @@ -300,13 +288,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_next: turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - step.animation_mode.prev_flow = flow + init.animation_mode.prev_flow = flow # do optical flow cadence after animation warping if cadence_flow is not None: cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, step.animation_keys.deform_keys, - tween_frame_idx, depth_model, + cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, init.animation_keys.deform_keys, + tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.W, args.H) * tween @@ -342,17 +330,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # saving cadence frames filename = f"{root.timestring}_{tween_frame_idx:09}.png" cv2.imwrite(os.path.join(args.outdir, filename), img) - if anim_args.save_depth_maps: - depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), + if init.step_args.anim_args.save_depth_maps: + init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), depth) # get color match for video outside of prev_img conditional - hybrid_available = (step.step_args.anim_args.hybrid_composite != 'None' - or anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) - if anim_args.color_coherence == 'Video Input' and hybrid_available: - if int(frame_idx) % int(anim_args.color_coherence_video_every_N_frames) == 0: + hybrid_available = (init.step_args.anim_args.hybrid_composite != 'None' + or init.step_args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) + if init.step_args.anim_args.color_coherence == 'Video Input' and hybrid_available: + if int(frame_idx) % int(init.step_args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name( - anim_args.video_init_path) + f"{frame_idx:09}.jpg")) + init.step_args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) prev_vid_img = prev_vid_img.resize((args.W, args.H), PIL.Image.LANCZOS) color_match_sample = np.asarray(prev_vid_img) color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) @@ -360,54 +348,55 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, args, anim_args, - step.animation_keys.deform_keys, frame_idx, - depth_model, depth=None, + prev_img, depth = anim_frame_warp(prev_img, init.step_args.args, init.step_args.anim_args, + init.animation_keys.deform_keys, frame_idx, + init.depth_model, depth=None, device=root.device, half_precision=root.half_precision) # do hybrid compositing before motion - if anim_args.hybrid_composite == 'Before Motion': - args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, - hybrid_comp_schedules, root) + if init.step_args.anim_args.hybrid_composite == 'Before Motion': + init.step_args.args, prev_img = hybrid_composite(init.step_args.args, init.step_args.anim_args, + frame_idx, prev_img, init.depth_model, + hybrid_comp_schedules, init.step_args.root) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if anim_args.hybrid_motion in ['Affine', 'Perspective']: if anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, prev_img, + init.animation_mode.hybrid_input_files, prev_img, anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_input_files, anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) if anim_args.hybrid_motion in ['Optical Flow']: if anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, - step.animation_mode.hybrid_frame_path, - step.animation_mode.prev_flow, prev_img, + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, prev_img, anim_args.hybrid_flow_method, - step.animation_mode.raft_model, + init.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) else: flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), - step.animation_mode.hybrid_input_files, - step.animation_mode.hybrid_frame_path, - step.animation_mode.prev_flow, + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, anim_args.hybrid_flow_method, - step.animation_mode.raft_model, + init.animation_mode.raft_model, anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) - step.animation_mode.prev_flow = flow + init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if anim_args.hybrid_composite == 'Normal': - args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, + args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, init.depth_model, hybrid_comp_schedules, root) # apply color matching @@ -445,34 +434,34 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.scale = scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - args.pix2pix_img_cfg_scale = float(step.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) + args.pix2pix_img_cfg_scale = float(init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) # grab prompt for current frame - args.prompt = prompt_series[frame_idx] + args.prompt = init.prompt_series[frame_idx] - if args.seed_behavior == 'schedule' or step.parseq_adapter.manages_seed(): - args.seed = int(step.animation_keys.deform_keys.seed_schedule_series[frame_idx]) + if args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): + args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) if anim_args.enable_checkpoint_scheduling: - args.checkpoint = step.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] + args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] else: args.checkpoint = None # SubSeed scheduling if anim_args.enable_subseed_scheduling: - root.subseed = int(step.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = float(step.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) + root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = float(init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) - if step.parseq_adapter.manages_seed(): + if init.parseq_adapter.manages_seed(): anim_args.enable_subseed_scheduling = True - root.subseed = int(step.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = step.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] + root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) # grab init image for current frame - if step.animation_mode.has_video_input: + if init.animation_mode.has_video_input: init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") args.init_image = init_frame @@ -489,11 +478,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro args.mask_image = compose_mask_with_check(root, args, schedule.mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway - step.animation_keys.update(frame_idx) + init.animation_keys.update(frame_idx) setup_opts(opts, schedule) if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): - if step.animation_mode.is_predicting_depths: depth_model.to('cpu') + if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) @@ -507,12 +496,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, - root, step.parseq_adapter, + disposable_image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, - step.animation_mode.raft_model) + init.animation_mode.raft_model) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) args.seed = stored_seed @@ -526,8 +515,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro for n in range(0, int(anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, - controlnet_args, root, step.parseq_adapter, + disposable_image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, + controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -539,17 +528,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # generation - image = generate(args, step.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, - root, step.parseq_adapter, frame_idx, + image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) if image is None: break # do hybrid video after generation - if frame_idx > 0 and anim_args.hybrid_composite == 'After Generation': + if frame_idx > 0 and init.step_args.anim_args.hybrid_composite == 'After Generation': temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, depth_model, + args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, init.depth_model, hybrid_comp_schedules, root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) @@ -576,7 +565,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not step.animation_mode.has_video_input: + if not init.animation_mode.has_video_input: prev_img = opencv_image if turbo_steps > 1: @@ -588,15 +577,16 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro save_image(image, 'PIL', filename, args, video_args, root) if anim_args.save_depth_maps: + # TODO move all depth related stuff to new class. (also see RenderInit) if cmd_opts.lowvram or cmd_opts.medvram: lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - depth_model.to(root.device) - depth = depth_model.predict(opencv_image, anim_args.midas_weight, root.half_precision) - depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) + init.depth_model.to(root.device) + depth = init.depth_model.predict(opencv_image, anim_args.midas_weight, root.half_precision) + init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) if cmd_opts.lowvram or cmd_opts.medvram: - depth_model.to('cpu') + init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) @@ -605,7 +595,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame) update_tracker(root, frame_idx, anim_args) - step.animation_mode.cleanup() + init.animation_mode.cleanup() def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): @@ -638,68 +628,6 @@ def setup_opts(opts, schedule): set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) -def init_looper_if_active(args, loop_args): - if loop_args.use_looper: - print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") - if args.strength == 0: - raise RuntimeError("Strength needs to be greater than 0 in Init tab") - args.strength_0_no_init = False - args.seed_behavior = "schedule" - if not isJson(loop_args.init_images): - raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") - - -def handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args): - if is_controlnet_enabled(controlnet_args): - unpack_controlnet_vids(args, anim_args, controlnet_args) - - -def create_output_directory_for_the_batch(args): - os.makedirs(args.outdir, exist_ok=True) - print(f"Saving animation frames to:\n{args.outdir}") - - -def save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): - save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - - -def maybe_resume_from_timestring(anim_args, current_value): - return anim_args.resume_timestring if anim_args.resume_from_timestring else current_value - - -def select_prompts(parseq_adapter, anim_args, animation_keys, root): - return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ - else expand_prompts_out_to_per_frame(anim_args, root) - - -def expand_prompts_out_to_per_frame(anim_args, root): - prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) - for i, prompt in root.animation_prompts.items(): - if str(i).isdigit(): - prompt_series[int(i)] = prompt - else: - prompt_series[int(numexpr.evaluate(i))] = prompt - return prompt_series.ffill().bfill() - - -def is_composite_with_depth_mask(anim_args): - return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' - - -def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args): - # depth-based hybrid composite mask requires saved depth maps - anim_args.save_depth_maps = anim_mode.is_predicting_depths and is_composite_with_depth_mask(anim_args) - if anim_mode.is_predicting_depths: - depth_device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) - return DepthModel(root.models_path, depth_device, root.half_precision, - keep_in_vram=anim_mode.is_keep_in_vram, - depth_algorithm=anim_args.depth_algorithm, - Width=args.W, Height=args.H, - midas_weight=anim_args.midas_weight) - else: - return None - - def progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame): state.assign_current_image(image) args.seed = next_seed(args, root) diff --git a/scripts/deforum_helpers/render_data/__init__.py b/scripts/deforum_helpers/render_data/__init__.py index f53e57a35..53087cadb 100644 --- a/scripts/deforum_helpers/render_data/__init__.py +++ b/scripts/deforum_helpers/render_data/__init__.py @@ -2,4 +2,6 @@ from .animation_mode import AnimationMode from .schedule import Schedule from .srt import Srt -from .step import Step + +# initialization is initialized last because it refs to the other data objects. +from .initialization import RenderInit, RenderInitArgs diff --git a/scripts/deforum_helpers/render_data/initialization.py b/scripts/deforum_helpers/render_data/initialization.py new file mode 100644 index 000000000..e34b6eb8c --- /dev/null +++ b/scripts/deforum_helpers/render_data/initialization.py @@ -0,0 +1,159 @@ +import dataclasses +import numexpr +import numpy as np +import os +import pandas as pd + +from typing import Any +from modules.shared import cmd_opts # keep readonly + +from . import AnimationKeys, AnimationMode, Srt +from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled +from ..depth import DepthModel +from ..generate import isJson +from ..parseq_adapter import ParseqAdapter +from ..settings import save_settings_from_animation_run + + +@dataclasses.dataclass +class RenderInitArgs: + args: Any = None + parseq_args: Any = None + anim_args: Any = None + video_args: Any = None + controlnet_args: Any = None + loop_args: Any = None + opts: Any = None + root: Any = None + + @classmethod + def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): + return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + + +@dataclasses.dataclass +class RenderInit: + """The purpose of this class is to group and control all data used in a single iteration of render_animation""" + seed: int + step_args: RenderInitArgs + parseq_adapter: Any + srt: Any + animation_keys: AnimationKeys + animation_mode: AnimationMode + prompt_series: Any + depth_model: Any + + def __new__(cls, *args, **kwargs): + raise TypeError("Use RenderInit.create() to create new instances.") + + @classmethod + def create_parseq_adapter(cls, args): + adapter = ParseqAdapter(args.parseq_args, args.anim_args, args.video_args, args.controlnet_args, args.loop_args) + # Always enable pseudo-3d with parseq. No need for an extra toggle: + # Whether it's used or not in practice is defined by the schedules + if adapter.use_parseq: + args.anim_args.flip_2d_perspective = True + return adapter + + @classmethod + def init_looper_if_active(cls, args, loop_args): + if loop_args.use_looper: + print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") + if args.strength == 0: + raise RuntimeError("Strength needs to be greater than 0 in Init tab") + args.strength_0_no_init = False + args.seed_behavior = "schedule" + if not isJson(loop_args.init_images): + raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") + + @classmethod + def select_prompts(cls, parseq_adapter, anim_args, animation_keys, root): + return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ + else RenderInit.expand_prompts_out_to_per_frame(anim_args, root) + + + @classmethod + def is_composite_with_depth_mask(cls, anim_args): + return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' + + @classmethod + def is_low_or_med_vram(cls): + # TODO move methods like this to a new static helper class or something + return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly + + @classmethod + def select_depth_device(cls, root): + return 'cpu' if RenderInit.is_low_or_med_vram() else root.device + + @classmethod + def create_depth_model_and_enable_depth_map_saving_if_active(cls, anim_mode, root, anim_args, args): + # depth-based hybrid composite mask requires saved depth maps + # TODO avoid or isolate side effect: + anim_args.save_depth_maps = (anim_mode.is_predicting_depths + and RenderInit.is_composite_with_depth_mask(anim_args)) + return DepthModel(root.models_path, + RenderInit.select_depth_device(root), + root.half_precision, + keep_in_vram=anim_mode.is_keep_in_vram, + depth_algorithm=anim_args.depth_algorithm, + Width=args.W, Height=args.H, + midas_weight=anim_args.midas_weight) \ + if anim_mode.is_predicting_depths else None + + @classmethod + def expand_prompts_out_to_per_frame(cls, anim_args, root): + prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) + for i, prompt in root.animation_prompts.items(): + if str(i).isdigit(): + prompt_series[int(i)] = prompt + else: + prompt_series[int(numexpr.evaluate(i))] = prompt + return prompt_series.ffill().bfill() + + @classmethod + def handle_controlnet_video_input_frames_generation(cls, controlnet_args, args, anim_args): + if is_controlnet_enabled(controlnet_args): + unpack_controlnet_vids(args, anim_args, controlnet_args) + + @classmethod + def create_output_directory_for_the_batch(cls, args): + os.makedirs(args.outdir, exist_ok=True) + print(f"Saving animation frames to:\n{args.outdir}") + + @classmethod + def save_settings_txt(cls, args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): + save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + + @classmethod + def maybe_resume_from_timestring(cls, anim_args, root): + root.timestring = anim_args.resume_timestring if anim_args.resume_from_timestring else root.timestring + + @classmethod + def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): + # TODO all of those calls may cause a change in on of the passed args. + # Ideally it may be refactored so each one returns a new instance of the potentially changed args that are then + # attached as a property to this Step to be used for one render iteration. + RenderInit.init_looper_if_active(args, loop_args) + RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + RenderInit.create_output_directory_for_the_batch(args) + RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + RenderInit.maybe_resume_from_timestring(anim_args, root) + + @classmethod + def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': + step_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + parseq_adapter = RenderInit.create_parseq_adapter(step_args) + srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) + animation_keys = AnimationKeys.from_args(step_args, parseq_adapter, args.seed) + animation_mode = AnimationMode.from_args(step_args) + prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) + depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active(animation_mode, root, anim_args, args) + instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. + instance.__init__(args.seed, step_args, parseq_adapter, srt, + animation_keys, animation_mode, prompt_series, depth_model) + # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. + # It may be preferable to work on temporary copies within tight scope. + # TODO avoid or isolate more side effects + RenderInit.do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) + + return instance diff --git a/scripts/deforum_helpers/render_data/step.py b/scripts/deforum_helpers/render_data/step.py deleted file mode 100644 index 9d400663f..000000000 --- a/scripts/deforum_helpers/render_data/step.py +++ /dev/null @@ -1,54 +0,0 @@ -import dataclasses - -from typing import Any -from . import AnimationKeys, AnimationMode, Srt -from ..parseq_adapter import ParseqAdapter - - -@dataclasses.dataclass -class StepArgs: - args: Any = None - parseq_args: Any = None - anim_args: Any = None - video_args: Any = None - controlnet_args: Any = None - loop_args: Any = None - opts: Any = None - root: Any = None - - @classmethod - def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): - return StepArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - - -@dataclasses.dataclass -class Step: - seed: int - step_args: StepArgs - parseq_adapter: Any - srt: Any - animation_keys: AnimationKeys - animation_mode: AnimationMode - - def __new__(cls, *args, **kwargs): - raise TypeError("Use Step.create() to create new instances.") - - @classmethod - def create_parseq_adapter(cls, args): - adapter = ParseqAdapter(args.parseq_args, args.anim_args, args.video_args, args.controlnet_args, args.loop_args) - # Always enable pseudo-3d with parseq. No need for an extra toggle: - # Whether it's used or not in practice is defined by the schedules - if adapter.use_parseq: - args.anim_args.flip_2d_perspective = True - return adapter - - @classmethod - def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'Step': - step_args = StepArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - parseq_adapter = Step.create_parseq_adapter(step_args) - instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(args.seed, step_args, parseq_adapter, - Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps), - AnimationKeys.from_args(step_args, parseq_adapter, args.seed), - AnimationMode.from_args(step_args)) # possible side effects on anim_args and args - return instance From 843a91c16e4bae86eb4c3b6690294bce9ca7833c Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 16:26:20 +0200 Subject: [PATCH 020/132] Data package reorganized. --- scripts/deforum_helpers/render_data/__init__.py | 3 --- scripts/deforum_helpers/render_data/anim/__init__.py | 2 ++ .../deforum_helpers/render_data/{ => anim}/animation_keys.py | 2 +- .../deforum_helpers/render_data/{ => anim}/animation_mode.py | 4 ++-- scripts/deforum_helpers/render_data/initialization.py | 5 +++-- scripts/deforum_helpers/render_data/subtitle/__init__.py | 1 + scripts/deforum_helpers/render_data/{ => subtitle}/srt.py | 2 +- 7 files changed, 10 insertions(+), 9 deletions(-) create mode 100644 scripts/deforum_helpers/render_data/anim/__init__.py rename scripts/deforum_helpers/render_data/{ => anim}/animation_keys.py (96%) rename scripts/deforum_helpers/render_data/{ => anim}/animation_mode.py (98%) create mode 100644 scripts/deforum_helpers/render_data/subtitle/__init__.py rename scripts/deforum_helpers/render_data/{ => subtitle}/srt.py (95%) diff --git a/scripts/deforum_helpers/render_data/__init__.py b/scripts/deforum_helpers/render_data/__init__.py index 53087cadb..27a781291 100644 --- a/scripts/deforum_helpers/render_data/__init__.py +++ b/scripts/deforum_helpers/render_data/__init__.py @@ -1,7 +1,4 @@ -from .animation_keys import AnimationKeys -from .animation_mode import AnimationMode from .schedule import Schedule -from .srt import Srt # initialization is initialized last because it refs to the other data objects. from .initialization import RenderInit, RenderInitArgs diff --git a/scripts/deforum_helpers/render_data/anim/__init__.py b/scripts/deforum_helpers/render_data/anim/__init__.py new file mode 100644 index 000000000..24b69f2a0 --- /dev/null +++ b/scripts/deforum_helpers/render_data/anim/__init__.py @@ -0,0 +1,2 @@ +from .animation_keys import AnimationKeys +from .animation_mode import AnimationMode diff --git a/scripts/deforum_helpers/render_data/animation_keys.py b/scripts/deforum_helpers/render_data/anim/animation_keys.py similarity index 96% rename from scripts/deforum_helpers/render_data/animation_keys.py rename to scripts/deforum_helpers/render_data/anim/animation_keys.py index 74fee4bd6..be3df666e 100644 --- a/scripts/deforum_helpers/render_data/animation_keys.py +++ b/scripts/deforum_helpers/render_data/anim/animation_keys.py @@ -1,6 +1,6 @@ import dataclasses -from ..animation_key_frames import DeformAnimKeys, LooperAnimKeys +from ...animation_key_frames import DeformAnimKeys, LooperAnimKeys @dataclasses.dataclass diff --git a/scripts/deforum_helpers/render_data/animation_mode.py b/scripts/deforum_helpers/render_data/anim/animation_mode.py similarity index 98% rename from scripts/deforum_helpers/render_data/animation_mode.py rename to scripts/deforum_helpers/render_data/anim/animation_mode.py index df4fca612..76afedcf4 100644 --- a/scripts/deforum_helpers/render_data/animation_mode.py +++ b/scripts/deforum_helpers/render_data/anim/animation_mode.py @@ -2,8 +2,8 @@ import os from typing import Any -from ..hybrid_video import hybrid_generation -from ..RAFT import RAFT +from ...hybrid_video import hybrid_generation +from ...RAFT import RAFT @dataclasses.dataclass diff --git a/scripts/deforum_helpers/render_data/initialization.py b/scripts/deforum_helpers/render_data/initialization.py index e34b6eb8c..7109b71c8 100644 --- a/scripts/deforum_helpers/render_data/initialization.py +++ b/scripts/deforum_helpers/render_data/initialization.py @@ -7,7 +7,8 @@ from typing import Any from modules.shared import cmd_opts # keep readonly -from . import AnimationKeys, AnimationMode, Srt +from .anim import AnimationKeys, AnimationMode +from .subtitle import Srt from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel from ..generate import isJson @@ -132,7 +133,7 @@ def maybe_resume_from_timestring(cls, anim_args, root): def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): # TODO all of those calls may cause a change in on of the passed args. # Ideally it may be refactored so each one returns a new instance of the potentially changed args that are then - # attached as a property to this Step to be used for one render iteration. + # attached as a property to this class to be used for one single render only. RenderInit.init_looper_if_active(args, loop_args) RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) RenderInit.create_output_directory_for_the_batch(args) diff --git a/scripts/deforum_helpers/render_data/subtitle/__init__.py b/scripts/deforum_helpers/render_data/subtitle/__init__.py new file mode 100644 index 000000000..236ac2b47 --- /dev/null +++ b/scripts/deforum_helpers/render_data/subtitle/__init__.py @@ -0,0 +1 @@ +from .srt import Srt diff --git a/scripts/deforum_helpers/render_data/srt.py b/scripts/deforum_helpers/render_data/subtitle/srt.py similarity index 95% rename from scripts/deforum_helpers/render_data/srt.py rename to scripts/deforum_helpers/render_data/subtitle/srt.py index fd9673dfc..cabfa982e 100644 --- a/scripts/deforum_helpers/render_data/srt.py +++ b/scripts/deforum_helpers/render_data/subtitle/srt.py @@ -2,7 +2,7 @@ import os from decimal import Decimal -from ..subtitle_handler import init_srt_file +from ...subtitle_handler import init_srt_file @dataclasses.dataclass From 10bc067a8e88840e36465276c1182cbc09b0c92b Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 18:18:18 +0200 Subject: [PATCH 021/132] Prepared new util package for modules with static helper functions and renamed the render package to avoid conflicts and confusion with the render module. --- scripts/deforum_helpers/render.py | 15 +++++++++------ .../deforum_helpers/render_data/__init__.py | 4 ---- scripts/deforum_helpers/rendering/__init__.py | 1 + .../rendering/data/__init__.py | 1 + .../data}/anim/__init__.py | 2 +- .../data}/anim/animation_keys.py | 2 +- .../data}/anim/animation_mode.py | 4 ++-- .../data}/schedule.py | 1 + .../data}/subtitle/__init__.py | 0 .../data}/subtitle/srt.py | 2 +- .../initialization.py | 19 ++++--------------- .../rendering/util/__init__.py | 3 +++ .../rendering/util/memory_utils.py | 13 +++++++++++++ 13 files changed, 37 insertions(+), 30 deletions(-) delete mode 100644 scripts/deforum_helpers/render_data/__init__.py create mode 100644 scripts/deforum_helpers/rendering/__init__.py create mode 100644 scripts/deforum_helpers/rendering/data/__init__.py rename scripts/deforum_helpers/{render_data => rendering/data}/anim/__init__.py (50%) rename scripts/deforum_helpers/{render_data => rendering/data}/anim/animation_keys.py (96%) rename scripts/deforum_helpers/{render_data => rendering/data}/anim/animation_mode.py (98%) rename scripts/deforum_helpers/{render_data => rendering/data}/schedule.py (99%) rename scripts/deforum_helpers/{render_data => rendering/data}/subtitle/__init__.py (100%) rename scripts/deforum_helpers/{render_data => rendering/data}/subtitle/srt.py (95%) rename scripts/deforum_helpers/{render_data => rendering}/initialization.py (92%) create mode 100644 scripts/deforum_helpers/rendering/util/__init__.py create mode 100644 scripts/deforum_helpers/rendering/util/memory_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 2193509d8..6a10f0466 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -11,7 +11,6 @@ # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . -import dataclasses # Contact the authors: https://deforum.github.io/ import gc @@ -40,13 +39,17 @@ from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt -from .render_data import Schedule, RenderInit from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview +# TODO reorg when done.. +from .rendering.initialization import RenderInit +from .rendering.data.schedule import Schedule +from .rendering.util.memory_utils import MemoryUtils + def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) @@ -165,7 +168,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro depth = None - if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): + if anim_args.animation_mode == '3D' and MemoryUtils.is_low_or_med_vram(): # Unload the main checkpoint and load the depth model lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) @@ -481,7 +484,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.animation_keys.update(frame_idx) setup_opts(opts, schedule) - if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): + if anim_args.animation_mode == '3D' and MemoryUtils.is_low_or_med_vram(): if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) @@ -578,14 +581,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if anim_args.save_depth_maps: # TODO move all depth related stuff to new class. (also see RenderInit) - if cmd_opts.lowvram or cmd_opts.medvram: + if MemoryUtils.is_low_or_med_vram(): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() init.depth_model.to(root.device) depth = init.depth_model.predict(opencv_image, anim_args.midas_weight, root.half_precision) init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) - if cmd_opts.lowvram or cmd_opts.medvram: + if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) diff --git a/scripts/deforum_helpers/render_data/__init__.py b/scripts/deforum_helpers/render_data/__init__.py deleted file mode 100644 index 27a781291..000000000 --- a/scripts/deforum_helpers/render_data/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -from .schedule import Schedule - -# initialization is initialized last because it refs to the other data objects. -from .initialization import RenderInit, RenderInitArgs diff --git a/scripts/deforum_helpers/rendering/__init__.py b/scripts/deforum_helpers/rendering/__init__.py new file mode 100644 index 000000000..11abfc6c2 --- /dev/null +++ b/scripts/deforum_helpers/rendering/__init__.py @@ -0,0 +1 @@ +from .initialization import RenderInit, RenderInitArgs diff --git a/scripts/deforum_helpers/rendering/data/__init__.py b/scripts/deforum_helpers/rendering/data/__init__.py new file mode 100644 index 000000000..7aef82b28 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/__init__.py @@ -0,0 +1 @@ +from .schedule import Schedule diff --git a/scripts/deforum_helpers/render_data/anim/__init__.py b/scripts/deforum_helpers/rendering/data/anim/__init__.py similarity index 50% rename from scripts/deforum_helpers/render_data/anim/__init__.py rename to scripts/deforum_helpers/rendering/data/anim/__init__.py index 24b69f2a0..19ab35d05 100644 --- a/scripts/deforum_helpers/render_data/anim/__init__.py +++ b/scripts/deforum_helpers/rendering/data/anim/__init__.py @@ -1,2 +1,2 @@ from .animation_keys import AnimationKeys -from .animation_mode import AnimationMode +from .animation_mode import AnimationMode \ No newline at end of file diff --git a/scripts/deforum_helpers/render_data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py similarity index 96% rename from scripts/deforum_helpers/render_data/anim/animation_keys.py rename to scripts/deforum_helpers/rendering/data/anim/animation_keys.py index be3df666e..9fac734ed 100644 --- a/scripts/deforum_helpers/render_data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -1,6 +1,6 @@ import dataclasses -from ...animation_key_frames import DeformAnimKeys, LooperAnimKeys +from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys @dataclasses.dataclass diff --git a/scripts/deforum_helpers/render_data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py similarity index 98% rename from scripts/deforum_helpers/render_data/anim/animation_mode.py rename to scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 76afedcf4..2a4b99f61 100644 --- a/scripts/deforum_helpers/render_data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -2,8 +2,8 @@ import os from typing import Any -from ...hybrid_video import hybrid_generation -from ...RAFT import RAFT +from ....hybrid_video import hybrid_generation +from ....RAFT import RAFT @dataclasses.dataclass diff --git a/scripts/deforum_helpers/render_data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py similarity index 99% rename from scripts/deforum_helpers/render_data/schedule.py rename to scripts/deforum_helpers/rendering/data/schedule.py index 90a6a7458..9b3b0dd3c 100644 --- a/scripts/deforum_helpers/render_data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,4 +1,5 @@ import dataclasses + from typing import Optional, Any diff --git a/scripts/deforum_helpers/render_data/subtitle/__init__.py b/scripts/deforum_helpers/rendering/data/subtitle/__init__.py similarity index 100% rename from scripts/deforum_helpers/render_data/subtitle/__init__.py rename to scripts/deforum_helpers/rendering/data/subtitle/__init__.py diff --git a/scripts/deforum_helpers/render_data/subtitle/srt.py b/scripts/deforum_helpers/rendering/data/subtitle/srt.py similarity index 95% rename from scripts/deforum_helpers/render_data/subtitle/srt.py rename to scripts/deforum_helpers/rendering/data/subtitle/srt.py index cabfa982e..8e905119b 100644 --- a/scripts/deforum_helpers/render_data/subtitle/srt.py +++ b/scripts/deforum_helpers/rendering/data/subtitle/srt.py @@ -2,7 +2,7 @@ import os from decimal import Decimal -from ...subtitle_handler import init_srt_file +from ....subtitle_handler import init_srt_file @dataclasses.dataclass diff --git a/scripts/deforum_helpers/render_data/initialization.py b/scripts/deforum_helpers/rendering/initialization.py similarity index 92% rename from scripts/deforum_helpers/render_data/initialization.py rename to scripts/deforum_helpers/rendering/initialization.py index 7109b71c8..4ae6a1e90 100644 --- a/scripts/deforum_helpers/render_data/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -5,10 +5,9 @@ import pandas as pd from typing import Any -from modules.shared import cmd_opts # keep readonly - -from .anim import AnimationKeys, AnimationMode -from .subtitle import Srt +from .data.anim import AnimationKeys, AnimationMode +from .data.subtitle import Srt +from .util import MemoryUtils from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel from ..generate import isJson @@ -72,20 +71,10 @@ def select_prompts(cls, parseq_adapter, anim_args, animation_keys, root): return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ else RenderInit.expand_prompts_out_to_per_frame(anim_args, root) - @classmethod def is_composite_with_depth_mask(cls, anim_args): return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' - @classmethod - def is_low_or_med_vram(cls): - # TODO move methods like this to a new static helper class or something - return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly - - @classmethod - def select_depth_device(cls, root): - return 'cpu' if RenderInit.is_low_or_med_vram() else root.device - @classmethod def create_depth_model_and_enable_depth_map_saving_if_active(cls, anim_mode, root, anim_args, args): # depth-based hybrid composite mask requires saved depth maps @@ -93,7 +82,7 @@ def create_depth_model_and_enable_depth_map_saving_if_active(cls, anim_mode, roo anim_args.save_depth_maps = (anim_mode.is_predicting_depths and RenderInit.is_composite_with_depth_mask(anim_args)) return DepthModel(root.models_path, - RenderInit.select_depth_device(root), + MemoryUtils.select_depth_device(root), root.half_precision, keep_in_vram=anim_mode.is_keep_in_vram, depth_algorithm=anim_args.depth_algorithm, diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py new file mode 100644 index 000000000..da035e878 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -0,0 +1,3 @@ +# All modules in this package are intended to not hold or change any state. +# Only use static class methods or loose defs that don't change anything in the arguments passed. +from .memory_utils import MemoryUtils diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py new file mode 100644 index 000000000..133595be3 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -0,0 +1,13 @@ +from modules.shared import cmd_opts # keep readonly + + +class MemoryUtils(): + # Don't put any variables here, it's meant for static methods only. + + @staticmethod + def is_low_or_med_vram(): + return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly + + @staticmethod + def select_depth_device(root): + return 'cpu' if MemoryUtils.is_low_or_med_vram() else root.device From 1fff071dbf1fc997ba1b5b0b5f2af710ad0e1c36 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 19:23:09 +0200 Subject: [PATCH 022/132] Started switching some args to init.args and did some cleanup. Renamed stuff and added more TODOs. --- scripts/deforum_helpers/render.py | 258 ++++++++++-------- .../rendering/initialization.py | 31 ++- .../rendering/util/memory_utils.py | 2 + 3 files changed, 159 insertions(+), 132 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 6a10f0466..d35bac376 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -11,6 +11,7 @@ # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . + # Contact the authors: https://deforum.github.io/ import gc @@ -39,23 +40,23 @@ from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt +from .rendering.data.schedule import Schedule +from .rendering.initialization import RenderInit +from .rendering.util.memory_utils import MemoryUtils from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed from .subtitle_handler import write_frame_subtitle, format_animation_params from .video_audio_utilities import get_frame_name, get_next_frame, render_preview -# TODO reorg when done.. -from .rendering.initialization import RenderInit -from .rendering.data.schedule import Schedule -from .rendering.util.memory_utils import MemoryUtils - def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + # TODO make sure that from here on, no args are accessed directly and instead readonly use init.args for now. + # state for interpolating between diffusion steps - turbo_steps = 1 if init.animation_mode.has_video_input else int(anim_args.diffusion_cadence) + turbo_steps = 1 if init.animation_mode.has_video_input else int(init.args.anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -65,13 +66,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro start_frame = 0 # resume animation (requires at least two frames - see function) - if anim_args.resume_from_timestring: + if init.args.anim_args.resume_from_timestring: # determine last frame and frame to start on prev_frame, next_frame, prev_img, next_img = get_resume_vars( folder=args.outdir, - timestring=anim_args.resume_timestring, - cadence=turbo_steps - ) + timestring=init.args.anim_args.resume_timestring, + cadence=turbo_steps) # set up turbo step vars if turbo_steps > 1: @@ -103,31 +103,40 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # Grab the first frame masks since they wont be provided until next frame # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 - if anim_args.use_mask_video: - args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) - root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + if init.args.anim_args.use_mask_video: + args.mask_file = get_mask_from_file( + get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + args) + root.noise_mask = get_mask_from_file( + get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + args) mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), args) noise_mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), args) elif mask_image is None and args.use_mask: mask_vals['video_mask'] = get_mask(args) noise_mask_vals['video_mask'] = get_mask(args) # TODO?: add a different default noisc mask # get color match for 'Image' color coherence only once, before loop - if anim_args.color_coherence == 'Image': - color_match_sample = load_image(anim_args.color_coherence_image_path, None) + if init.args.anim_args.color_coherence == 'Image': + color_match_sample = load_image(init.args.anim_args.color_coherence_image_path, None) color_match_sample = color_match_sample.resize((args.W, args.H), PIL.Image.LANCZOS) color_match_sample = cv2.cvtColor(np.array(color_match_sample), cv2.COLOR_RGB2BGR) # Webui - state.job_count = anim_args.max_frames + state.job_count = init.args.anim_args.max_frames last_preview_frame = 0 + # TODO create a Step class in rendering.data with all the iteration specific info, + # then eventually try to replace this while loop with functions that: + # - 1. Create a collection of Steps with all the required info that is already known before we enter the iteration. + # - 2. Transform and process the steps however needed (i.e. space out or reassign turbo frames etc.) + # - 3. Actually do the render by foreaching over the steps in sequence while frame_idx < anim_args.max_frames: # Webui - state.job = f"frame {frame_idx + 1}/{anim_args.max_frames}" + state.job = f"frame {frame_idx + 1}/{init.args.anim_args.max_frames}" state.job_no = frame_idx + 1 if state.skipped: @@ -137,7 +146,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro time.sleep(0.1) print("** RESUMING **") - print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{anim_args.max_frames} ") + print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{init.args.anim_args.max_frames} ") # TODO move this to the new key collection noise = init.animation_keys.deform_keys.noise_schedule_series[frame_idx] @@ -161,14 +170,15 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro "flow_factor": init.animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] } - schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, anim_args, args) + # TODO eventually move schedule into new Step class + schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, args) - if args.use_mask and not anim_args.use_noise_mask: + if args.use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq depth = None - if anim_args.animation_mode == '3D' and MemoryUtils.is_low_or_med_vram(): + if init.is_3d_with_med_or_low_vram(): # Unload the main checkpoint and load the depth model lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) @@ -190,7 +200,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro cadence_flow = None for tween_frame_idx in range(tween_frame_start_idx, frame_idx): # update progress during cadence - state.job = f"frame {tween_frame_idx + 1}/{anim_args.max_frames}" + state.job = f"frame {tween_frame_idx + 1}/{init.args.anim_args.max_frames}" state.job_no = tween_frame_idx + 1 # cadence vars tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx) @@ -198,11 +208,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro advance_next = tween_frame_idx > turbo_next_frame_idx # optical flow cadence setup before animation warping - if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': + if init.args.anim_args.animation_mode in ['2D', + '3D'] and init.args.anim_args.optical_flow_cadence != 'None': if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, - anim_args.optical_flow_cadence, + init.args.anim_args.optical_flow_cadence, init.animation_mode.raft_model) / 2 turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) @@ -216,58 +227,60 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") params_string = None - print(f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") + print( + f"Creating in-between {'' if cadence_flow is None else init.args.anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") if init.depth_model is not None: assert (turbo_next_image is not None) - depth = init.depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) + depth = init.depth_model.predict(turbo_next_image, init.args.anim_args.midas_weight, + root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, + turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion if tween_frame_idx > 0: - if anim_args.hybrid_motion in ['Affine', 'Perspective']: - if anim_args.hybrid_motion_use_prev_img: + if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: + if init.args.anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, prev_img, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) if advance_next: turbo_next_image = image_transform_ransac(turbo_next_image, matrix, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) if advance_next: turbo_next_image = image_transform_ransac(turbo_next_image, matrix, - anim_args.hybrid_motion) - if anim_args.hybrid_motion in ['Optical Flow']: - if anim_args.hybrid_motion_use_prev_img: + init.args.anim_args.hybrid_motion) + if init.args.anim_args.hybrid_motion in ['Optical Flow']: + if init.args.anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, prev_img, - anim_args.hybrid_flow_method, + init.args.anim_args.hybrid_flow_method, init.animation_mode.raft_model, - anim_args.hybrid_flow_consistency, - anim_args.hybrid_consistency_blur, - anim_args.hybrid_comp_save_extra_frames) + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) if advance_prev: turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) @@ -280,11 +293,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, - anim_args.hybrid_flow_method, + init.args.anim_args.hybrid_flow_method, init.animation_mode.raft_model, - anim_args.hybrid_flow_consistency, - anim_args.hybrid_consistency_blur, - anim_args.hybrid_comp_save_extra_frames) + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) if advance_prev: turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) @@ -296,7 +309,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do optical flow cadence after animation warping if cadence_flow is not None: cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, init.animation_keys.deform_keys, + cadence_flow, _ = anim_frame_warp(cadence_flow, args, init.args.anim_args, + init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -316,13 +330,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro img = turbo_next_image # intercept and override to grayscale - if anim_args.color_force_grayscale: + if init.args.anim_args.color_force_grayscale: img = cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY) img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) # overlay mask - if args.overlay_mask and (anim_args.use_mask_video or args.use_mask): - img = do_overlay_mask(args, anim_args, img, tween_frame_idx, True) + if args.overlay_mask and (init.args.anim_args.use_mask_video or args.use_mask): + img = do_overlay_mask(args, init.args.anim_args, img, tween_frame_idx, True) # get prev_img during cadence prev_img = img @@ -333,17 +347,17 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # saving cadence frames filename = f"{root.timestring}_{tween_frame_idx:09}.png" cv2.imwrite(os.path.join(args.outdir, filename), img) - if init.step_args.anim_args.save_depth_maps: - init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), - depth) + if init.args.anim_args.save_depth_maps: + init.depth_model.save( + os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), depth) # get color match for video outside of prev_img conditional - hybrid_available = (init.step_args.anim_args.hybrid_composite != 'None' - or init.step_args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) - if init.step_args.anim_args.color_coherence == 'Video Input' and hybrid_available: - if int(frame_idx) % int(init.step_args.anim_args.color_coherence_video_every_N_frames) == 0: + hybrid_available = (init.args.anim_args.hybrid_composite != 'None' + or init.args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) + if init.args.anim_args.color_coherence == 'Video Input' and hybrid_available: + if int(frame_idx) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name( - init.step_args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) + init.args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) prev_vid_img = prev_vid_img.resize((args.W, args.H), PIL.Image.LANCZOS) color_match_sample = np.asarray(prev_vid_img) color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) @@ -351,66 +365,66 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, init.step_args.args, init.step_args.anim_args, + prev_img, depth = anim_frame_warp(prev_img, init.args.args, init.args.anim_args, init.animation_keys.deform_keys, frame_idx, init.depth_model, depth=None, device=root.device, half_precision=root.half_precision) # do hybrid compositing before motion - if init.step_args.anim_args.hybrid_composite == 'Before Motion': - init.step_args.args, prev_img = hybrid_composite(init.step_args.args, init.step_args.anim_args, - frame_idx, prev_img, init.depth_model, - hybrid_comp_schedules, init.step_args.root) + if init.args.anim_args.hybrid_composite == 'Before Motion': + init.args.args, prev_img = hybrid_composite(init.args.args, init.args.anim_args, + frame_idx, prev_img, init.depth_model, + hybrid_comp_schedules, init.args.root) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing - if anim_args.hybrid_motion in ['Affine', 'Perspective']: - if anim_args.hybrid_motion_use_prev_img: + if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: + if init.args.anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, prev_img, - anim_args.hybrid_motion) + init.args.anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, - anim_args.hybrid_motion) - prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) - if anim_args.hybrid_motion in ['Optical Flow']: - if anim_args.hybrid_motion_use_prev_img: + init.args.anim_args.hybrid_motion) + prev_img = image_transform_ransac(prev_img, matrix, init.args.anim_args.hybrid_motion) + if init.args.anim_args.hybrid_motion in ['Optical Flow']: + if init.args.anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, prev_img, - anim_args.hybrid_flow_method, + init.args.anim_args.hybrid_flow_method, init.animation_mode.raft_model, - anim_args.hybrid_flow_consistency, - anim_args.hybrid_consistency_blur, - anim_args.hybrid_comp_save_extra_frames) + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) else: flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, - anim_args.hybrid_flow_method, + init.args.anim_args.hybrid_flow_method, init.animation_mode.raft_model, - anim_args.hybrid_flow_consistency, - anim_args.hybrid_consistency_blur, - anim_args.hybrid_comp_save_extra_frames) + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) - if anim_args.hybrid_composite == 'Normal': - args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, init.depth_model, + if init.args.anim_args.hybrid_composite == 'Normal': + args, prev_img = hybrid_composite(args, init.args.anim_args, frame_idx, prev_img, init.depth_model, hybrid_comp_schedules, root) # apply color matching - if anim_args.color_coherence != 'None': + if init.args.anim_args.color_coherence != 'None': if color_match_sample is None: color_match_sample = prev_img.copy() else: - prev_img = maintain_colors(prev_img, color_match_sample, anim_args.color_coherence) + prev_img = maintain_colors(prev_img, color_match_sample, init.args.anim_args.color_coherence) # intercept and override to grayscale - if anim_args.color_force_grayscale: + if init.args.anim_args.color_force_grayscale: prev_img = cv2.cvtColor(prev_img, cv2.COLOR_BGR2GRAY) prev_img = cv2.cvtColor(prev_img, cv2.COLOR_GRAY2BGR) @@ -421,12 +435,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, mask_image if args.use_mask else None) # apply frame noising - if args.use_mask or anim_args.use_noise_mask: + if args.use_mask or init.args.anim_args.use_noise_mask: root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq, noise_mask_vals, Image.fromarray( cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) - noised_image = add_noise(contrast_image, noise, args.seed, anim_args.noise_type, - (anim_args.perlin_w, anim_args.perlin_h, anim_args.perlin_octaves, - anim_args.perlin_persistence), + noised_image = add_noise(contrast_image, noise, args.seed, init.args.anim_args.noise_type, + (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, + init.args.anim_args.perlin_octaves, + init.args.anim_args.perlin_persistence), root.noise_mask, args.invert_mask) # use transformed previous frame as init for current @@ -445,33 +460,33 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) - if anim_args.enable_checkpoint_scheduling: + if init.args.anim_args.enable_checkpoint_scheduling: args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] else: args.checkpoint = None # SubSeed scheduling - if anim_args.enable_subseed_scheduling: + if init.args.anim_args.enable_subseed_scheduling: root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) root.subseed_strength = float(init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) if init.parseq_adapter.manages_seed(): - anim_args.enable_subseed_scheduling = True + init.args.anim_args.enable_subseed_scheduling = True root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed - args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) + args.prompt = prepare_prompt(args.prompt, init.args.anim_args.max_frames, args.seed, frame_idx) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) + init_frame = get_next_frame(args.outdir, init.args.anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") args.init_image = init_frame args.init_image_box = None # init_image_box not used in this case args.strength = max(0.0, min(1.0, strength)) - if anim_args.use_mask_video: - mask_init_frame = get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True) + if init.args.anim_args.use_mask_video: + mask_init_frame = get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True) temp_mask = get_mask_from_file(mask_init_frame, args) args.mask_file = temp_mask root.noise_mask = temp_mask @@ -484,13 +499,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.animation_keys.update(frame_idx) setup_opts(opts, schedule) - if anim_args.animation_mode == '3D' and MemoryUtils.is_low_or_med_vram(): + if init.is_3d_with_med_or_low_vram(): if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - optical_flow_redo_generation = anim_args.optical_flow_redo_generation if not args.motion_preview_mode else 'None' + optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation if not args.motion_preview_mode else 'None' # optical flow redo before generation if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: @@ -499,7 +514,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") - disposable_image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + disposable_image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, + controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) @@ -513,25 +529,26 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro gc.collect() # diffusion redo - if int(anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not args.motion_preview_mode: + if int(init.args.anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not args.motion_preview_mode: stored_seed = args.seed - for n in range(0, int(anim_args.diffusion_redo)): - print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") + for n in range(0, int(init.args.anim_args.diffusion_redo)): + print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, + disposable_image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only - if n == int(anim_args.diffusion_redo): - disposable_image = maintain_colors(prev_img, color_match_sample, anim_args.color_coherence) + if n == int(init.args.anim_args.diffusion_redo): + disposable_image = maintain_colors(prev_img, color_match_sample, + init.args.anim_args.color_coherence) args.seed = stored_seed root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) del (disposable_image, stored_seed) gc.collect() # generation - image = generate(args, init.animation_keys.deform_keys, anim_args, loop_args, controlnet_args, + image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) @@ -539,32 +556,32 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro break # do hybrid video after generation - if frame_idx > 0 and init.step_args.anim_args.hybrid_composite == 'After Generation': + if frame_idx > 0 and init.args.anim_args.hybrid_composite == 'After Generation': temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - args, temp_image_2 = hybrid_composite(args, anim_args, frame_idx, temp_image, init.depth_model, + args, temp_image_2 = hybrid_composite(args, init.args.anim_args, frame_idx, temp_image, init.depth_model, hybrid_comp_schedules, root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if frame_idx == 0 and should_initialize_color_match(anim_args, hybrid_available, color_match_sample): + if frame_idx == 0 and should_initialize_color_match(init.args.anim_args, hybrid_available, color_match_sample): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - temp_image = maintain_colors(temp_color, color_match_sample, anim_args.color_coherence) + temp_image = maintain_colors(temp_color, color_match_sample, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) # intercept and override to grayscale - if anim_args.color_force_grayscale: + if init.args.anim_args.color_force_grayscale: image = ImageOps.grayscale(image) image = ImageOps.colorize(image, black="black", white="white") # overlay mask - if args.overlay_mask and (anim_args.use_mask_video or args.use_mask): - image = do_overlay_mask(args, anim_args, image, frame_idx) + if args.overlay_mask and (init.args.anim_args.use_mask_video or args.use_mask): + image = do_overlay_mask(args, init.args.anim_args, image, frame_idx) # on strength 0, set color match to generation - if ((not anim_args.legacy_colormatch and not args.use_init) or ( - anim_args.legacy_colormatch and strength == 0)) and not anim_args.color_coherence in ['Image', - 'Video Input']: + if (((not init.args.anim_args.legacy_colormatch and not args.use_init) + or (init.args.anim_args.legacy_colormatch and strength == 0)) + and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) @@ -579,14 +596,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro filename = f"{root.timestring}_{frame_idx:09}.png" save_image(image, 'PIL', filename, args, video_args, root) - if anim_args.save_depth_maps: + if init.args.anim_args.save_depth_maps: # TODO move all depth related stuff to new class. (also see RenderInit) if MemoryUtils.is_low_or_med_vram(): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() init.depth_model.to(root.device) - depth = init.depth_model.predict(opencv_image, anim_args.midas_weight, root.half_precision) + depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, root.half_precision) init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') @@ -595,9 +612,9 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro sd_hijack.model_hijack.hijack(sd_model) frame_idx += 1 - last_preview_frame = progress_and_make_preview(state, image, args, anim_args, video_args, + last_preview_frame = progress_and_make_preview(state, image, args, init.args.anim_args, video_args, root, frame_idx, last_preview_frame) - update_tracker(root, frame_idx, anim_args) + update_tracker(root, frame_idx, init.args.anim_args) init.animation_mode.cleanup() @@ -639,4 +656,3 @@ def progress_and_make_preview(state, image, args, anim_args, video_args, root, f def update_tracker(root, frame_idx, anim_args): JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) - diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 4ae6a1e90..9e92cea57 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -33,9 +33,9 @@ def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_ @dataclasses.dataclass class RenderInit: - """The purpose of this class is to group and control all data used in a single iteration of render_animation""" + """The purpose of this class is to group and control all data used in render_animation""" seed: int - step_args: RenderInitArgs + args: RenderInitArgs parseq_adapter: Any srt: Any animation_keys: AnimationKeys @@ -46,6 +46,13 @@ class RenderInit: def __new__(cls, *args, **kwargs): raise TypeError("Use RenderInit.create() to create new instances.") + def is_3d(self): + return self.args.anim_args.animation_mode == '3D' + + def is_3d_with_med_or_low_vram(self): + return self.is_3d() and MemoryUtils.is_low_or_med_vram() + + @classmethod def create_parseq_adapter(cls, args): adapter = ParseqAdapter(args.parseq_args, args.anim_args, args.video_args, args.controlnet_args, args.loop_args) @@ -130,20 +137,22 @@ def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, RenderInit.maybe_resume_from_timestring(anim_args, root) @classmethod - def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': - step_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - parseq_adapter = RenderInit.create_parseq_adapter(step_args) - srt = Srt.create_if_active(opts.data, args.outdir, root.timestring, video_args.fps) - animation_keys = AnimationKeys.from_args(step_args, parseq_adapter, args.seed) - animation_mode = AnimationMode.from_args(step_args) + def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_args, + loop_args, opts, root) -> 'RenderInit': + args = RenderInitArgs(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + parseq_adapter = RenderInit.create_parseq_adapter(args) + srt = Srt.create_if_active(opts.data, args_argument.outdir, root.timestring, video_args.fps) + animation_keys = AnimationKeys.from_args(args, parseq_adapter, args_argument.seed) + animation_mode = AnimationMode.from_args(args) prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active(animation_mode, root, anim_args, args) + depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( + animation_mode, root, anim_args, args_argument) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(args.seed, step_args, parseq_adapter, srt, + instance.__init__(args_argument.seed, args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model) # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. # It may be preferable to work on temporary copies within tight scope. # TODO avoid or isolate more side effects - RenderInit.do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) + RenderInit.do_void_inits(args_argument, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) return instance diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 133595be3..34895e982 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -6,6 +6,8 @@ class MemoryUtils(): @staticmethod def is_low_or_med_vram(): + # TODO Ideally this should only be called once at the beginning of the render. + # Perhaps add a constant bool to RenderInit. return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly @staticmethod From 1b11a8a406935eefcdd2a9d932b3a1678de1e131 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 19:34:22 +0200 Subject: [PATCH 023/132] All new dataclasses frozen and configured down to the required essentials only. --- scripts/deforum_helpers/rendering/data/anim/animation_keys.py | 2 +- scripts/deforum_helpers/rendering/data/anim/animation_mode.py | 2 +- scripts/deforum_helpers/rendering/data/schedule.py | 2 +- scripts/deforum_helpers/rendering/data/subtitle/srt.py | 2 +- scripts/deforum_helpers/rendering/initialization.py | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index 9fac734ed..9ad69e482 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -3,7 +3,7 @@ from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class AnimationKeys: deform_keys: DeformAnimKeys looper_keys: LooperAnimKeys diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 2a4b99f61..61e7e73d1 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -6,7 +6,7 @@ from ....RAFT import RAFT -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index 9b3b0dd3c..639291d91 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -3,7 +3,7 @@ from typing import Optional, Any -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: steps: int sampler_name: str diff --git a/scripts/deforum_helpers/rendering/data/subtitle/srt.py b/scripts/deforum_helpers/rendering/data/subtitle/srt.py index 8e905119b..2f5bc0d90 100644 --- a/scripts/deforum_helpers/rendering/data/subtitle/srt.py +++ b/scripts/deforum_helpers/rendering/data/subtitle/srt.py @@ -5,7 +5,7 @@ from ....subtitle_handler import init_srt_file -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class Srt: filename: str frame_duration: Decimal diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 9e92cea57..585e7ea49 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -15,7 +15,7 @@ from ..settings import save_settings_from_animation_run -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInitArgs: args: Any = None parseq_args: Any = None @@ -31,7 +31,7 @@ def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_ return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -@dataclasses.dataclass +@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" seed: int From 9184cf64b8a2fc639623f0b5e55b15f639769780 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 21:01:47 +0200 Subject: [PATCH 024/132] User-independent metadata for PyCharm users. --- .idea/.gitignore | 3 +++ .idea/deforum.iml | 12 ++++++++++++ .idea/inspectionProfiles/profiles_settings.xml | 6 ++++++ .idea/misc.xml | 7 +++++++ .idea/modules.xml | 8 ++++++++ .idea/vcs.xml | 6 ++++++ 6 files changed, 42 insertions(+) create mode 100644 .idea/.gitignore create mode 100644 .idea/deforum.iml create mode 100644 .idea/inspectionProfiles/profiles_settings.xml create mode 100644 .idea/misc.xml create mode 100644 .idea/modules.xml create mode 100644 .idea/vcs.xml diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 000000000..26d33521a --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,3 @@ +# Default ignored files +/shelf/ +/workspace.xml diff --git a/.idea/deforum.iml b/.idea/deforum.iml new file mode 100644 index 000000000..8b8c39547 --- /dev/null +++ b/.idea/deforum.iml @@ -0,0 +1,12 @@ + + + + + + + + + + \ No newline at end of file diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml new file mode 100644 index 000000000..105ce2da2 --- /dev/null +++ b/.idea/inspectionProfiles/profiles_settings.xml @@ -0,0 +1,6 @@ + + + + \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml new file mode 100644 index 000000000..b3fbd492f --- /dev/null +++ b/.idea/misc.xml @@ -0,0 +1,7 @@ + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 000000000..1b7de0730 --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 000000000..35eb1ddfb --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file From 04f1edb231c91c66a8a33e525bcaf489e3afcd5c Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 21:49:54 +0200 Subject: [PATCH 025/132] Added setup to easily run tests and start the webui with API enabled on a button-click in PyCharm (Interpreter path pointing to standard Windows powershell dir by default, all other paths provided relative). --- .idea/runConfigurations/Deforum_Tests.xml | 18 ++++++++++++++++++ .idea/runConfigurations/Postprocess_Tests.xml | 18 ++++++++++++++++++ .idea/runConfigurations/WebUI_with_API.xml | 17 +++++++++++++++++ 3 files changed, 53 insertions(+) create mode 100644 .idea/runConfigurations/Deforum_Tests.xml create mode 100644 .idea/runConfigurations/Postprocess_Tests.xml create mode 100644 .idea/runConfigurations/WebUI_with_API.xml diff --git a/.idea/runConfigurations/Deforum_Tests.xml b/.idea/runConfigurations/Deforum_Tests.xml new file mode 100644 index 000000000..4ecd59c57 --- /dev/null +++ b/.idea/runConfigurations/Deforum_Tests.xml @@ -0,0 +1,18 @@ + + + + + \ No newline at end of file diff --git a/.idea/runConfigurations/Postprocess_Tests.xml b/.idea/runConfigurations/Postprocess_Tests.xml new file mode 100644 index 000000000..d2a930043 --- /dev/null +++ b/.idea/runConfigurations/Postprocess_Tests.xml @@ -0,0 +1,18 @@ + + + + + \ No newline at end of file diff --git a/.idea/runConfigurations/WebUI_with_API.xml b/.idea/runConfigurations/WebUI_with_API.xml new file mode 100644 index 000000000..3fe93568e --- /dev/null +++ b/.idea/runConfigurations/WebUI_with_API.xml @@ -0,0 +1,17 @@ + + + + \ No newline at end of file From ac6abce4aab377d799c553c67ba5ae6ee1b93658 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 22:45:30 +0200 Subject: [PATCH 026/132] Replaced all aruments with the ones provided by init.args in order to prepare for reduction of complexities in method calls. render_animation method torn appart to ensure tightly scoped args and also unfroze the AnimationMode class again to allow for the late initialization of prev_flow for now. --- scripts/deforum_helpers/render.py | 182 ++++++++++-------- .../rendering/data/anim/animation_mode.py | 5 +- .../rendering/initialization.py | 2 + 3 files changed, 107 insertions(+), 82 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d35bac376..92e39f39c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -51,10 +51,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + render_init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + run_render_animation(render_init) - # TODO make sure that from here on, no args are accessed directly and instead readonly use init.args for now. +def run_render_animation(init): # state for interpolating between diffusion steps turbo_steps = 1 if init.animation_mode.has_video_input else int(init.args.anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 @@ -69,7 +70,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if init.args.anim_args.resume_from_timestring: # determine last frame and frame to start on prev_frame, next_frame, prev_img, next_img = get_resume_vars( - folder=args.outdir, + folder=init.args.args.outdir, timestring=init.args.anim_args.resume_timestring, cadence=turbo_steps) @@ -87,12 +88,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro mask_vals = {} noise_mask_vals = {} - mask_vals['everywhere'] = Image.new('1', (args.W, args.H), 1) - noise_mask_vals['everywhere'] = Image.new('1', (args.W, args.H), 1) + mask_vals['everywhere'] = Image.new('1', (init.args.args.W, init.args.args.H), 1) + noise_mask_vals['everywhere'] = Image.new('1', (init.args.args.W, init.args.args.H), 1) mask_image = None - if args.use_init and ((args.init_image != None and args.init_image != '') or args.init_image_box != None): + if init.args.args.use_init and ((init.args.args.init_image != None and init.args.args.init_image != '') + or init.args.args.init_image_box != None): _, mask_image = load_img(args.init_image, args.init_image_box, shape=(args.W, args.H), @@ -104,24 +106,26 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 if init.args.anim_args.use_mask_video: - args.mask_file = get_mask_from_file( - get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), - args) + init.args.args.mask_file = get_mask_from_file( + get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + init.args.args) root.noise_mask = get_mask_from_file( - get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), - args) + get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + init.args.args) mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), args) + get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + init.args.args) noise_mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), args) + get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + init.args.args) elif mask_image is None and args.use_mask: - mask_vals['video_mask'] = get_mask(args) - noise_mask_vals['video_mask'] = get_mask(args) # TODO?: add a different default noisc mask + mask_vals['video_mask'] = get_mask(init.args.args) + noise_mask_vals['video_mask'] = get_mask(init.args.args) # TODO?: add a different default noisc mask # get color match for 'Image' color coherence only once, before loop if init.args.anim_args.color_coherence == 'Image': color_match_sample = load_image(init.args.anim_args.color_coherence_image_path, None) - color_match_sample = color_match_sample.resize((args.W, args.H), PIL.Image.LANCZOS) + color_match_sample = color_match_sample.resize((init.args.args.W, init.args.args.H), PIL.Image.LANCZOS) color_match_sample = cv2.cvtColor(np.array(color_match_sample), cv2.COLOR_RGB2BGR) # Webui @@ -133,7 +137,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # - 1. Create a collection of Steps with all the required info that is already known before we enter the iteration. # - 2. Transform and process the steps however needed (i.e. space out or reassign turbo frames etc.) # - 3. Actually do the render by foreaching over the steps in sequence - while frame_idx < anim_args.max_frames: + while frame_idx < init.args.anim_args.max_frames: # Webui state.job = f"frame {frame_idx + 1}/{init.args.anim_args.max_frames}" @@ -173,7 +177,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # TODO eventually move schedule into new Step class schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, args) - if args.use_mask and not init.args.anim_args.use_noise_mask: + if init.args.args.use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq depth = None @@ -191,7 +195,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro params_string = format_animation_params(init.animation_keys.deform_keys, init.prompt_series, frame_idx, params_to_print) write_frame_subtitle(init.srt.filename, frame_idx, init.srt.frame_duration, - f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") + f"F#: {frame_idx}; Cadence: false; Seed: {init.args.args.seed}; {params_string}") params_string = None # FIXME ?? # emit in-between frames @@ -224,7 +228,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro tween_frame_idx, params_to_print) write_frame_subtitle(init.srt.filename, tween_frame_idx, init.srt.frame_duration, - f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") + f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {init.args.args.seed}; {params_string}") params_string = None print( @@ -236,12 +240,12 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, init.args.anim_args, + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, init.args.args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, init.args.anim_args, + turbo_next_image, _ = anim_frame_warp(turbo_next_image, init.args.args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, depth=depth, device=root.device, half_precision=root.half_precision) @@ -250,7 +254,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if tween_frame_idx > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), + matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, prev_img, init.args.anim_args.hybrid_motion) if advance_prev: @@ -260,7 +265,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro turbo_next_image = image_transform_ransac(turbo_next_image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), + matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, init.args.anim_args.hybrid_motion) if advance_prev: @@ -271,7 +277,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), + flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, @@ -287,7 +294,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if advance_next: turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) - init.animation_mode.prev_flow = flow + init.animation_mode.prev_flow = flow # FIXME shouldn't else: flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), init.animation_mode.hybrid_input_files, @@ -372,24 +379,28 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do hybrid compositing before motion if init.args.anim_args.hybrid_composite == 'Before Motion': - init.args.args, prev_img = hybrid_composite(init.args.args, init.args.anim_args, - frame_idx, prev_img, init.depth_model, - hybrid_comp_schedules, init.args.root) + # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) + _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, + frame_idx, prev_img, init.depth_model, + hybrid_comp_schedules, init.args.root) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), + matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, prev_img, init.args.anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), + matrix = get_matrix_for_hybrid_motion(frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, init.args.anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), + flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, prev_img, @@ -399,7 +410,8 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.args.anim_args.hybrid_consistency_blur, init.args.anim_args.hybrid_comp_save_extra_frames) else: - flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), + flow = get_flow_for_hybrid_motion(frame_idx - 1, + (init.args.args.W, init.args.args.H), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, @@ -413,8 +425,9 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do hybrid compositing after motion (normal) if init.args.anim_args.hybrid_composite == 'Normal': - args, prev_img = hybrid_composite(args, init.args.anim_args, frame_idx, prev_img, init.depth_model, - hybrid_comp_schedules, root) + # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) + _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, prev_img, + init.depth_model, hybrid_comp_schedules, init.args.root) # apply color matching if init.args.anim_args.color_coherence != 'None': @@ -433,37 +446,40 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # anti-blur if amount > 0: contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, - mask_image if args.use_mask else None) + mask_image if init.args.args.use_mask else None) # apply frame noising - if args.use_mask or init.args.anim_args.use_noise_mask: - root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq, noise_mask_vals, Image.fromarray( - cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) - noised_image = add_noise(contrast_image, noise, args.seed, init.args.anim_args.noise_type, + if init.args.args.use_mask or init.args.anim_args.use_noise_mask: + root.noise_mask = compose_mask_with_check(root, + init.args.args, noise_mask_seq, + noise_mask_vals, + Image.fromarray( + cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) + noised_image = add_noise(contrast_image, noise, init.args.args.seed, init.args.anim_args.noise_type, (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, init.args.anim_args.perlin_octaves, init.args.anim_args.perlin_persistence), - root.noise_mask, args.invert_mask) + root.noise_mask, init.args.args.invert_mask) # use transformed previous frame as init for current - args.use_init = True + init.args.args.use_init = True root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) - args.strength = max(0.0, min(1.0, strength)) + init.args.args.strength = max(0.0, min(1.0, strength)) - args.scale = scale + init.args.args.scale = scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - args.pix2pix_img_cfg_scale = float(init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) + init.args.args.pix2pix_img_cfg_scale = float(init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) # grab prompt for current frame - args.prompt = init.prompt_series[frame_idx] + init.args.args.prompt = init.prompt_series[frame_idx] if args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) if init.args.anim_args.enable_checkpoint_scheduling: - args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] + init.args.args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] else: - args.checkpoint = None + init.args.args.checkpoint = None # SubSeed scheduling if init.args.anim_args.enable_subseed_scheduling: @@ -476,24 +492,28 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed - args.prompt = prepare_prompt(args.prompt, init.args.anim_args.max_frames, args.seed, frame_idx) + init.args.args.prompt = prepare_prompt(args.prompt, init.args.anim_args.max_frames, + init.args.args.seed, frame_idx) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = get_next_frame(args.outdir, init.args.anim_args.video_init_path, frame_idx, False) + init_frame = get_next_frame(init.args.args.outdir, init.args.anim_args.video_init_path, + frame_idx, False) print(f"Using video init frame {init_frame}") - args.init_image = init_frame - args.init_image_box = None # init_image_box not used in this case - args.strength = max(0.0, min(1.0, strength)) + init.args.args.init_image = init_frame + init.args.args.init_image_box = None # init_image_box not used in this case + init.args.args.strength = max(0.0, min(1.0, strength)) if init.args.anim_args.use_mask_video: - mask_init_frame = get_next_frame(args.outdir, init.args.anim_args.video_mask_path, frame_idx, True) - temp_mask = get_mask_from_file(mask_init_frame, args) - args.mask_file = temp_mask + mask_init_frame = get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, + frame_idx, True) + temp_mask = get_mask_from_file(mask_init_frame, init.args.args) + init.args.args.mask_file = temp_mask root.noise_mask = temp_mask mask_vals['video_mask'] = temp_mask - if args.use_mask: - args.mask_image = compose_mask_with_check(root, args, schedule.mask_seq, mask_vals, root.init_sample) \ + if init.args.args.use_mask: + init.args.args.mask_image = compose_mask_with_check(root, init.args.args, + schedule.mask_seq, mask_vals, root.init_sample) \ if root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) @@ -505,16 +525,15 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation if not args.motion_preview_mode else 'None' + optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation if not init.args.args.motion_preview_mode else 'None' # optical flow redo before generation if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: - stored_seed = args.seed - args.seed = random.randint(0, 2 ** 32 - 1) - print( - f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") + stored_seed = init.args.args.seed + init.args.args.seed = random.randint(0, 2 ** 32 - 1) + print(f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") - disposable_image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, + disposable_image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) @@ -523,18 +542,18 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro init.animation_mode.raft_model) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) - args.seed = stored_seed + init.args.args.seed = stored_seed root.init_sample = Image.fromarray(disposable_image) del (disposable_image, disposable_flow, stored_seed) gc.collect() # diffusion redo - if int(init.args.anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not args.motion_preview_mode: - stored_seed = args.seed + if int(init.args.anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not init.args.args.motion_preview_mode: + stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") - args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, + init.args.args.seed = random.randint(0, 2 ** 32 - 1) + disposable_image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, controlnet_args, root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) @@ -542,14 +561,15 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro if n == int(init.args.anim_args.diffusion_redo): disposable_image = maintain_colors(prev_img, color_match_sample, init.args.anim_args.color_coherence) - args.seed = stored_seed + init.args.args.seed = stored_seed root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) del (disposable_image, stored_seed) gc.collect() # generation - image = generate(args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, controlnet_args, - root, init.parseq_adapter, frame_idx, + image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, + init.args.loop_args, init.args.controlnet_args, + init.args.root, init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) if image is None: @@ -558,8 +578,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # do hybrid video after generation if frame_idx > 0 and init.args.anim_args.hybrid_composite == 'After Generation': temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - args, temp_image_2 = hybrid_composite(args, init.args.anim_args, frame_idx, temp_image, init.depth_model, - hybrid_comp_schedules, root) + # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) + _, temp_image_2 = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, temp_image, + init.depth_model, + hybrid_comp_schedules, root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, @@ -575,11 +597,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro image = ImageOps.colorize(image, black="black", white="white") # overlay mask - if args.overlay_mask and (init.args.anim_args.use_mask_video or args.use_mask): - image = do_overlay_mask(args, init.args.anim_args, image, frame_idx) + if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): + image = do_overlay_mask(init.args.args, init.args.anim_args, image, frame_idx) # on strength 0, set color match to generation - if (((not init.args.anim_args.legacy_colormatch and not args.use_init) + if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) or (init.args.anim_args.legacy_colormatch and strength == 0)) and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) @@ -594,7 +616,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro frame_idx += turbo_steps else: filename = f"{root.timestring}_{frame_idx:09}.png" - save_image(image, 'PIL', filename, args, video_args, root) + save_image(image, 'PIL', filename, init.args.args, video_args, root) if init.args.anim_args.save_depth_maps: # TODO move all depth related stuff to new class. (also see RenderInit) @@ -612,7 +634,9 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro sd_hijack.model_hijack.hijack(sd_model) frame_idx += 1 - last_preview_frame = progress_and_make_preview(state, image, args, init.args.anim_args, video_args, + # TODO isolate more state.... + last_preview_frame = progress_and_make_preview(state, image, init.args.args, + init.args.anim_args, init.args.video_args, root, frame_idx, last_preview_frame) update_tracker(root, frame_idx, init.args.anim_args) init.animation_mode.cleanup() diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 61e7e73d1..0df36de56 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -6,7 +6,8 @@ from ....RAFT import RAFT -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +# TODO FIXME find a way to assign prev_flow right away, then set frozen=true again, otherwise move prev_flow elsewhere. +@dataclasses.dataclass(init=True, frozen=False, repr=False, eq=False) class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None @@ -28,7 +29,6 @@ def cleanup(self): if self.is_raft_active(): self.raft_model.delete_model() - @staticmethod def _has_video_input(anim_args) -> bool: return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) @@ -53,7 +53,6 @@ def _is_load_depth_model_for_3d(args, anim_args): is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth return is_depth_used and not args.motion_preview_mode - @staticmethod def _load_raft_if_active(anim_args, args): is_cadenced_raft = anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1 diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 585e7ea49..23349e61a 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -17,6 +17,8 @@ @dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInitArgs: + # TODO eventually this should only keep the information required to run render_animation once + # for now it's just a direct reference or a copy of the actual args provided to the render_animation call. args: Any = None parseq_args: Any = None anim_args: Any = None From bb06999d55b56bc3a1a3e9dbe2a75ca89b37fac0 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 2 Jun 2024 23:41:19 +0200 Subject: [PATCH 027/132] Started delegating args trough new methods in init and fixed some refs that broke in the last commit. All original arguments have been taken out of scope for now. More TODOs added. --- scripts/deforum_helpers/render.py | 206 +++++++++++------- .../rendering/initialization.py | 9 + 2 files changed, 133 insertions(+), 82 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 92e39f39c..71a58d08d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -52,10 +52,19 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): render_init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + # TODO method is temporarily torn apart to remove args from direct access in larger execution scope. run_render_animation(render_init) def run_render_animation(init): + # TODO refactor to try and avoid all reassignments to "init.args.args". + # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. + # TODO cleanup and find potential side effects in 31 refs to init.args.root + # TODO isolate the relevant data in +250 refs to init.args, + # move that stuff to init and eventually try to drop init.args + # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. + + # state for interpolating between diffusion steps turbo_steps = 1 if init.animation_mode.has_video_input else int(init.args.anim_args.diffusion_cadence) turbo_prev_image, turbo_prev_frame_idx = None, 0 @@ -88,17 +97,17 @@ def run_render_animation(init): mask_vals = {} noise_mask_vals = {} - mask_vals['everywhere'] = Image.new('1', (init.args.args.W, init.args.args.H), 1) - noise_mask_vals['everywhere'] = Image.new('1', (init.args.args.W, init.args.args.H), 1) + mask_vals['everywhere'] = Image.new('1', init.dimensions(), 1) + noise_mask_vals['everywhere'] = Image.new('1', init.dimensions(), 1) mask_image = None if init.args.args.use_init and ((init.args.args.init_image != None and init.args.args.init_image != '') or init.args.args.init_image_box != None): - _, mask_image = load_img(args.init_image, - args.init_image_box, - shape=(args.W, args.H), - use_alpha_as_mask=args.use_alpha_as_mask) + _, mask_image = load_img(init.args.args.init_image, + init.args.args.init_image_box, + shape=init.dimensions(), + use_alpha_as_mask=init.args.use_alpha_as_mask) mask_vals['video_mask'] = mask_image noise_mask_vals['video_mask'] = mask_image @@ -109,7 +118,7 @@ def run_render_animation(init): init.args.args.mask_file = get_mask_from_file( get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) - root.noise_mask = get_mask_from_file( + init.args.root.noise_mask = get_mask_from_file( get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) mask_vals['video_mask'] = get_mask_from_file( @@ -118,14 +127,14 @@ def run_render_animation(init): noise_mask_vals['video_mask'] = get_mask_from_file( get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) - elif mask_image is None and args.use_mask: + elif mask_image is None and init.args.args.use_mask: mask_vals['video_mask'] = get_mask(init.args.args) noise_mask_vals['video_mask'] = get_mask(init.args.args) # TODO?: add a different default noisc mask # get color match for 'Image' color coherence only once, before loop if init.args.anim_args.color_coherence == 'Image': color_match_sample = load_image(init.args.anim_args.color_coherence_image_path, None) - color_match_sample = color_match_sample.resize((init.args.args.W, init.args.args.H), PIL.Image.LANCZOS) + color_match_sample = color_match_sample.resize(init.dimensions(), PIL.Image.LANCZOS) color_match_sample = cv2.cvtColor(np.array(color_match_sample), cv2.COLOR_RGB2BGR) # Webui @@ -175,7 +184,7 @@ def run_render_animation(init): } # TODO eventually move schedule into new Step class - schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, args) + schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, init.args.args) if init.args.args.use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -188,7 +197,7 @@ def run_render_animation(init): sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(root.device) + init.animation_mode.depth_model.to(init.args.root.device) if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) @@ -212,8 +221,8 @@ def run_render_animation(init): advance_next = tween_frame_idx > turbo_next_frame_idx # optical flow cadence setup before animation warping - if init.args.anim_args.animation_mode in ['2D', - '3D'] and init.args.anim_args.optical_flow_cadence != 'None': + if (init.args.anim_args.animation_mode in ['2D', '3D'] + and init.args.anim_args.optical_flow_cadence != 'None'): if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, @@ -237,25 +246,25 @@ def run_render_animation(init): if init.depth_model is not None: assert (turbo_next_image is not None) depth = init.depth_model.predict(turbo_next_image, init.args.anim_args.midas_weight, - root.half_precision) + init.args.root.half_precision) if advance_prev: turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, init.args.args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, - init.depth_model, depth=depth, device=root.device, - half_precision=root.half_precision) + init.depth_model, depth=depth, device=init.args.root.device, + half_precision=init.args.root.half_precision) if advance_next: turbo_next_image, _ = anim_frame_warp(turbo_next_image, init.args.args, init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, - init.depth_model, depth=depth, device=root.device, - half_precision=root.half_precision) + init.depth_model, depth=depth, device=init.args.root.device, + half_precision=init.args.root.half_precision) # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion if tween_frame_idx > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, prev_img, init.args.anim_args.hybrid_motion) if advance_prev: @@ -266,7 +275,7 @@ def run_render_animation(init): init.args.anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, init.args.anim_args.hybrid_motion) if advance_prev: @@ -278,7 +287,7 @@ def run_render_animation(init): if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, @@ -296,7 +305,7 @@ def run_render_animation(init): hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow # FIXME shouldn't else: - flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), + flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, init.dimensions(), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, @@ -315,13 +324,19 @@ def run_render_animation(init): # do optical flow cadence after animation warping if cadence_flow is not None: - cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) - cadence_flow, _ = anim_frame_warp(cadence_flow, args, init.args.anim_args, + cadence_flow = abs_flow_to_rel_flow(cadence_flow, + init.width(), + init.height()) + cadence_flow, _ = anim_frame_warp(cadence_flow, + init.args.args, + init.args.anim_args, init.animation_keys.deform_keys, tween_frame_idx, init.depth_model, - depth=depth, device=root.device, - half_precision=root.half_precision) - cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.W, args.H) * tween + depth=depth, device=init.args.root.device, + half_precision=init.args.root.half_precision) + cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, + init.dimensions().width(), + init.dimensions().height()) * tween if advance_prev: turbo_prev_image = image_transform_optical_flow(turbo_prev_image, cadence_flow_inc, cadence_flow_factor) @@ -342,8 +357,8 @@ def run_render_animation(init): img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) # overlay mask - if args.overlay_mask and (init.args.anim_args.use_mask_video or args.use_mask): - img = do_overlay_mask(args, init.args.anim_args, img, tween_frame_idx, True) + if args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): + img = do_overlay_mask(init.args.args, init.args.anim_args, img, tween_frame_idx, True) # get prev_img during cadence prev_img = img @@ -352,55 +367,64 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{root.timestring}_{tween_frame_idx:09}.png" - cv2.imwrite(os.path.join(args.outdir, filename), img) + filename = f"{init.args.root.timestring}_{tween_frame_idx:09}.png" + cv2.imwrite(os.path.join(init.args.args.outdir, filename), img) if init.args.anim_args.save_depth_maps: init.depth_model.save( - os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), depth) + os.path.join(init.args.args.outdir, f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png"), depth) # get color match for video outside of prev_img conditional hybrid_available = (init.args.anim_args.hybrid_composite != 'None' or init.args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) if init.args.anim_args.color_coherence == 'Video Input' and hybrid_available: if int(frame_idx) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name( + prev_vid_img = Image.open(os.path.join(init.args.args.outdir, 'inputframes', get_frame_name( init.args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) - prev_vid_img = prev_vid_img.resize((args.W, args.H), PIL.Image.LANCZOS) + prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) color_match_sample = np.asarray(prev_vid_img) color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, init.args.args, init.args.anim_args, - init.animation_keys.deform_keys, frame_idx, - init.depth_model, depth=None, - device=root.device, half_precision=root.half_precision) + prev_img, depth = anim_frame_warp(prev_img, + init.args.args, + init.args.anim_args, + init.animation_keys.deform_keys, + frame_idx, + init.depth_model, + depth=None, + device=init.args.root.device, + half_precision=init.args.root.half_precision) # do hybrid compositing before motion if init.args.anim_args.hybrid_composite == 'Before Motion': # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, - frame_idx, prev_img, init.depth_model, - hybrid_comp_schedules, init.args.root) + _, prev_img = hybrid_composite(init.args.args, + init.args.anim_args, + frame_idx, + prev_img, + init.depth_model, + hybrid_comp_schedules, + init.args.root) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, prev_img, init.args.anim_args.hybrid_motion) else: matrix = get_matrix_for_hybrid_motion(frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, init.args.anim_args.hybrid_motion) prev_img = image_transform_ransac(prev_img, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, prev_img, @@ -411,7 +435,7 @@ def run_render_animation(init): init.args.anim_args.hybrid_comp_save_extra_frames) else: flow = get_flow_for_hybrid_motion(frame_idx - 1, - (init.args.args.W, init.args.args.H), + init.dimensions(), init.animation_mode.hybrid_input_files, init.animation_mode.hybrid_frame_path, init.animation_mode.prev_flow, @@ -449,20 +473,21 @@ def run_render_animation(init): mask_image if init.args.args.use_mask else None) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: - root.noise_mask = compose_mask_with_check(root, - init.args.args, noise_mask_seq, + init.args.root.noise_mask = compose_mask_with_check(init.args.root, + init.args.args, + noise_mask_seq, noise_mask_vals, - Image.fromarray( - cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) + Image.fromarray(cv2.cvtColor(contrast_image, + cv2.COLOR_BGR2RGB))) noised_image = add_noise(contrast_image, noise, init.args.args.seed, init.args.anim_args.noise_type, (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, init.args.anim_args.perlin_octaves, init.args.anim_args.perlin_persistence), - root.noise_mask, init.args.args.invert_mask) + init.args.root.noise_mask, init.args.args.invert_mask) # use transformed previous frame as init for current init.args.args.use_init = True - root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) + init.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) init.args.args.strength = max(0.0, min(1.0, strength)) init.args.args.scale = scale @@ -473,8 +498,8 @@ def run_render_animation(init): # grab prompt for current frame init.args.args.prompt = init.prompt_series[frame_idx] - if args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) + if init.args.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): + init.args.args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) if init.args.anim_args.enable_checkpoint_scheduling: init.args.args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] @@ -483,16 +508,16 @@ def run_render_animation(init): # SubSeed scheduling if init.args.anim_args.enable_subseed_scheduling: - root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = float(init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) + init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + init.args.root.subseed_strength = float(init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] + init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + init.args.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed - init.args.args.prompt = prepare_prompt(args.prompt, init.args.anim_args.max_frames, + init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, init.args.args.seed, frame_idx) # grab init image for current frame @@ -508,13 +533,14 @@ def run_render_animation(init): frame_idx, True) temp_mask = get_mask_from_file(mask_init_frame, init.args.args) init.args.args.mask_file = temp_mask - root.noise_mask = temp_mask + init.args.root.noise_mask = temp_mask mask_vals['video_mask'] = temp_mask if init.args.args.use_mask: - init.args.args.mask_image = compose_mask_with_check(root, init.args.args, - schedule.mask_seq, mask_vals, root.init_sample) \ - if root.init_sample is not None else None # we need it only after the first frame anyway + init.args.args.mask_image = compose_mask_with_check(init.args.root, init.args.args, + schedule.mask_seq, mask_vals, + init.args.root.init_sample) \ + if init.args.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) setup_opts(opts, schedule) @@ -525,7 +551,8 @@ def run_render_animation(init): lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation if not init.args.args.motion_preview_mode else 'None' + optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ + if not init.args.args.motion_preview_mode else 'None' # optical flow redo before generation if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: @@ -533,9 +560,13 @@ def run_render_animation(init): init.args.args.seed = random.randint(0, 2 ** 32 - 1) print(f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") - disposable_image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, - controlnet_args, - root, init.parseq_adapter, + disposable_image = generate(init.args.args, + init.animation_keys.deform_keys, + init.args.anim_args, + init.args.loop_args, + init.args.controlnet_args, + init.args.root, + init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, @@ -543,18 +574,25 @@ def run_render_animation(init): disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) init.args.args.seed = stored_seed - root.init_sample = Image.fromarray(disposable_image) + init.args.root.init_sample = Image.fromarray(disposable_image) del (disposable_image, disposable_flow, stored_seed) gc.collect() # diffusion redo - if int(init.args.anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not init.args.args.motion_preview_mode: + if (int(init.args.anim_args.diffusion_redo) > 0 + and prev_img is not None and strength > 0 + and not init.args.args.motion_preview_mode): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") init.args.args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, loop_args, - controlnet_args, root, init.parseq_adapter, + disposable_image = generate(init.args.args, + init.animation_keys.deform_keys, + init.args.anim_args, + init.args.loop_args, + init.args.controlnet_args, + init.args.root, + init.parseq_adapter, frame_idx, sampler_name=schedule.sampler_name) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -562,7 +600,7 @@ def run_render_animation(init): disposable_image = maintain_colors(prev_img, color_match_sample, init.args.anim_args.color_coherence) init.args.args.seed = stored_seed - root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) + init.args.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) del (disposable_image, stored_seed) gc.collect() @@ -580,8 +618,7 @@ def run_render_animation(init): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, temp_image_2 = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, temp_image, - init.depth_model, - hybrid_comp_schedules, root) + init.depth_model, hybrid_comp_schedules, init.args.root) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, @@ -615,8 +652,8 @@ def run_render_animation(init): turbo_next_image, turbo_next_frame_idx = opencv_image, frame_idx frame_idx += turbo_steps else: - filename = f"{root.timestring}_{frame_idx:09}.png" - save_image(image, 'PIL', filename, init.args.args, video_args, root) + filename = f"{init.args.root.timestring}_{frame_idx:09}.png" + save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) if init.args.anim_args.save_depth_maps: # TODO move all depth related stuff to new class. (also see RenderInit) @@ -624,9 +661,9 @@ def run_render_animation(init): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - init.depth_model.to(root.device) - depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, root.half_precision) - init.depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) + init.depth_model.to(init.args.root.device) + depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.args.root.half_precision) + init.depth_model.save(os.path.join(args.outdir, f"{init.args.root.timestring}_depth_{frame_idx:09}.png"), depth) if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') devices.torch_gc() @@ -635,10 +672,15 @@ def run_render_animation(init): frame_idx += 1 # TODO isolate more state.... - last_preview_frame = progress_and_make_preview(state, image, init.args.args, - init.args.anim_args, init.args.video_args, - root, frame_idx, last_preview_frame) - update_tracker(root, frame_idx, init.args.anim_args) + last_preview_frame = progress_and_make_preview(state, + image, + init.args.args, + init.args.anim_args, + init.args.video_args, + init.args.root, + frame_idx, + last_preview_frame) + update_tracker(init.args.root, frame_idx, init.args.anim_args) init.animation_mode.cleanup() diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 23349e61a..7f7d5de5f 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -54,6 +54,15 @@ def is_3d(self): def is_3d_with_med_or_low_vram(self): return self.is_3d() and MemoryUtils.is_low_or_med_vram() + def width(self) -> int: + return self.args.args.W + + def height(self) -> int: + return self.args.args.H + + def dimensions(self) -> tuple[int, int]: + # TODO should ideally only be called once each render + return self.width(), self.height() @classmethod def create_parseq_adapter(cls, args): From de2a40789935bd86de35223053744c6cb43f70d4 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 3 Jun 2024 03:25:20 +0200 Subject: [PATCH 028/132] Created helper to group arguments for calling anim_frame_warp. Also juggled some bools. --- scripts/deforum_helpers/render.py | 163 +++++++----------- .../rendering/initialization.py | 67 ++++++- .../rendering/util/__init__.py | 1 + .../deforum_helpers/rendering/util/utils.py | 19 ++ 4 files changed, 144 insertions(+), 106 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 71a58d08d..582cdb280 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -27,7 +27,6 @@ from modules import lowvram, devices, sd_hijack from modules.shared import opts, cmd_opts, state, sd_model -from .animation import anim_frame_warp from .colors import maintain_colors from .composable_masks import compose_mask_with_check from .generate import generate @@ -42,6 +41,7 @@ from .prompt import prepare_prompt from .rendering.data.schedule import Schedule from .rendering.initialization import RenderInit +from .rendering.util import put_if_present, call_anim_frame_warp from .rendering.util.memory_utils import MemoryUtils from .resume import get_resume_vars from .save_images import save_image @@ -57,16 +57,16 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(init): - # TODO refactor to try and avoid all reassignments to "init.args.args". + # TODO refactor to try and avoid all usage and reassignments to "init.args.args". # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. + # TODO isolate "depth" with other moving parts # TODO cleanup and find potential side effects in 31 refs to init.args.root # TODO isolate the relevant data in +250 refs to init.args, # move that stuff to init and eventually try to drop init.args # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. - # state for interpolating between diffusion steps - turbo_steps = 1 if init.animation_mode.has_video_input else int(init.args.anim_args.diffusion_cadence) + turbo_steps = 1 if init.has_video_input() else init.cadence() turbo_prev_image, turbo_prev_frame_idx = None, 0 turbo_next_image, turbo_next_frame_idx = None, 0 @@ -76,7 +76,7 @@ def run_render_animation(init): start_frame = 0 # resume animation (requires at least two frames - see function) - if init.args.anim_args.resume_from_timestring: + if init.is_resuming_from_timestring(): # determine last frame and frame to start on prev_frame, next_frame, prev_img, next_img = get_resume_vars( folder=init.args.args.outdir, @@ -102,12 +102,14 @@ def run_render_animation(init): mask_image = None - if init.args.args.use_init and ((init.args.args.init_image != None and init.args.args.init_image != '') - or init.args.args.init_image_box != None): + if (init.args.args.use_init + and ((init.args.args.init_image != None + and init.args.args.init_image != '') + or init.args.args.init_image_box != None)): _, mask_image = load_img(init.args.args.init_image, init.args.args.init_image_box, shape=init.dimensions(), - use_alpha_as_mask=init.args.use_alpha_as_mask) + use_alpha_as_mask=init.args.args.use_alpha_as_mask) mask_vals['video_mask'] = mask_image noise_mask_vals['video_mask'] = mask_image @@ -116,18 +118,18 @@ def run_render_animation(init): # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 if init.args.anim_args.use_mask_video: init.args.args.mask_file = get_mask_from_file( - get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) init.args.root.noise_mask = get_mask_from_file( - get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) noise_mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, frame_idx, True), + get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), init.args.args) - elif mask_image is None and init.args.args.use_mask: + elif mask_image is None and init.is_use_mask: mask_vals['video_mask'] = get_mask(init.args.args) noise_mask_vals['video_mask'] = get_mask(init.args.args) # TODO?: add a different default noisc mask @@ -186,7 +188,7 @@ def run_render_animation(init): # TODO eventually move schedule into new Step class schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, init.args.args) - if init.args.args.use_mask and not init.args.anim_args.use_noise_mask: + if init.is_use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq depth = None @@ -249,15 +251,10 @@ def run_render_animation(init): init.args.root.half_precision) if advance_prev: - turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, init.args.args, init.args.anim_args, - init.animation_keys.deform_keys, tween_frame_idx, - init.depth_model, depth=depth, device=init.args.root.device, - half_precision=init.args.root.half_precision) + turbo_prev_image, _ = call_anim_frame_warp(init, turbo_prev_image, tween_frame_idx, depth) if advance_next: - turbo_next_image, _ = anim_frame_warp(turbo_next_image, init.args.args, init.args.anim_args, - init.animation_keys.deform_keys, tween_frame_idx, - init.depth_model, depth=depth, device=init.args.root.device, - half_precision=init.args.root.half_precision) + turbo_next_image, _ = call_anim_frame_warp(init, turbo_next_image, tween_frame_idx, depth) + # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion if tween_frame_idx > 0: @@ -327,13 +324,7 @@ def run_render_animation(init): cadence_flow = abs_flow_to_rel_flow(cadence_flow, init.width(), init.height()) - cadence_flow, _ = anim_frame_warp(cadence_flow, - init.args.args, - init.args.anim_args, - init.animation_keys.deform_keys, - tween_frame_idx, init.depth_model, - depth=depth, device=init.args.root.device, - half_precision=init.args.root.half_precision) + cadence_flow, _ = call_anim_frame_warp(init, cadence_flow, tween_frame_idx, depth) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, init.dimensions().width(), init.dimensions().height()) * tween @@ -356,8 +347,8 @@ def run_render_animation(init): img = cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY) img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) - # overlay mask - if args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): + # overlay mask + if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): img = do_overlay_mask(init.args.args, init.args.anim_args, img, tween_frame_idx, True) # get prev_img during cadence @@ -368,17 +359,17 @@ def run_render_animation(init): # saving cadence frames filename = f"{init.args.root.timestring}_{tween_frame_idx:09}.png" - cv2.imwrite(os.path.join(init.args.args.outdir, filename), img) + save_path = os.path.join(init.args.args.outdir, filename) + cv2.imwrite(save_path, img) + if init.args.anim_args.save_depth_maps: - init.depth_model.save( - os.path.join(init.args.args.outdir, f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png"), depth) + dm_save_path = os.path.join(init.output_directory, f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png") + init.depth_model.save(dm_save_path, depth) # get color match for video outside of prev_img conditional - hybrid_available = (init.args.anim_args.hybrid_composite != 'None' - or init.args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective']) - if init.args.anim_args.color_coherence == 'Video Input' and hybrid_available: + if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): if int(frame_idx) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(os.path.join(init.args.args.outdir, 'inputframes', get_frame_name( + prev_vid_img = Image.open(os.path.join(init.output_directory, 'inputframes', get_frame_name( init.args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) color_match_sample = np.asarray(prev_vid_img) @@ -387,18 +378,10 @@ def run_render_animation(init): # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = anim_frame_warp(prev_img, - init.args.args, - init.args.anim_args, - init.animation_keys.deform_keys, - frame_idx, - init.depth_model, - depth=None, - device=init.args.root.device, - half_precision=init.args.root.half_precision) + prev_img, depth = call_anim_frame_warp(init, prev_img, frame_idx, None) # do hybrid compositing before motion - if init.args.anim_args.hybrid_composite == 'Before Motion': + if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, @@ -448,13 +431,13 @@ def run_render_animation(init): init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) - if init.args.anim_args.hybrid_composite == 'Normal': + if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, prev_img, - init.depth_model, hybrid_comp_schedules, init.args.root) + init.depth_model, hybrid_comp_schedules, init.args.root) # apply color matching - if init.args.anim_args.color_coherence != 'None': + if init.has_color_coherence(): if color_match_sample is None: color_match_sample = prev_img.copy() else: @@ -474,11 +457,11 @@ def run_render_animation(init): # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: init.args.root.noise_mask = compose_mask_with_check(init.args.root, - init.args.args, - noise_mask_seq, - noise_mask_vals, - Image.fromarray(cv2.cvtColor(contrast_image, - cv2.COLOR_BGR2RGB))) + init.args.args, + noise_mask_seq, # FIXME might be ref'd b4 assignment + noise_mask_vals, + Image.fromarray(cv2.cvtColor(contrast_image, + cv2.COLOR_BGR2RGB))) noised_image = add_noise(contrast_image, noise, init.args.args.seed, init.args.anim_args.noise_type, (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, init.args.anim_args.perlin_octaves, @@ -493,7 +476,8 @@ def run_render_animation(init): init.args.args.scale = scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - init.args.args.pix2pix_img_cfg_scale = float(init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) + init.args.args.pix2pix_img_cfg_scale = float( + init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) # grab prompt for current frame init.args.args.prompt = init.prompt_series[frame_idx] @@ -509,12 +493,14 @@ def run_render_animation(init): # SubSeed scheduling if init.args.anim_args.enable_subseed_scheduling: init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.args.root.subseed_strength = float(init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) + init.args.root.subseed_strength = float( + init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.args.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] + init.args.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[ + frame_idx] # set value back into the prompt - prepare and report prompt and seed init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, @@ -522,14 +508,14 @@ def run_render_animation(init): # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = get_next_frame(init.args.args.outdir, init.args.anim_args.video_init_path, + init_frame = get_next_frame(init.output_directory, init.args.anim_args.video_init_path, frame_idx, False) print(f"Using video init frame {init_frame}") init.args.args.init_image = init_frame init.args.args.init_image_box = None # init_image_box not used in this case init.args.args.strength = max(0.0, min(1.0, strength)) if init.args.anim_args.use_mask_video: - mask_init_frame = get_next_frame(init.args.args.outdir, init.args.anim_args.video_mask_path, + mask_init_frame = get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True) temp_mask = get_mask_from_file(mask_init_frame, init.args.args) init.args.args.mask_file = temp_mask @@ -543,7 +529,7 @@ def run_render_animation(init): if init.args.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) - setup_opts(opts, schedule) + setup_opts(init, schedule) if init.is_3d_with_med_or_low_vram(): if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') @@ -558,7 +544,8 @@ def run_render_animation(init): if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: stored_seed = init.args.args.seed init.args.args.seed = random.randint(0, 2 ** 32 - 1) - print(f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") + print( + f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") disposable_image = generate(init.args.args, init.animation_keys.deform_keys, @@ -601,8 +588,8 @@ def run_render_animation(init): init.args.anim_args.color_coherence) init.args.args.seed = stored_seed init.args.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) - del (disposable_image, stored_seed) - gc.collect() + del (disposable_image, stored_seed) # FIXME disposable_image might be referenced before assignment. + gc.collect() # TODO try to eventually kick the gc only once at the end of every generation or iteration. # generation image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, @@ -614,7 +601,7 @@ def run_render_animation(init): break # do hybrid video after generation - if frame_idx > 0 and init.args.anim_args.hybrid_composite == 'After Generation': + if frame_idx > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, temp_image_2 = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, temp_image, @@ -623,7 +610,7 @@ def run_render_animation(init): # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if frame_idx == 0 and should_initialize_color_match(init.args.anim_args, hybrid_available, color_match_sample): + if frame_idx == 0 and init.is_color_match_to_be_initialized(color_match_sample): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) temp_image = maintain_colors(temp_color, color_match_sample, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) @@ -634,7 +621,7 @@ def run_render_animation(init): image = ImageOps.colorize(image, black="black", white="white") # overlay mask - if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): + if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): image = do_overlay_mask(init.args.args, init.args.anim_args, image, frame_idx) # on strength 0, set color match to generation @@ -662,8 +649,10 @@ def run_render_animation(init): sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() init.depth_model.to(init.args.root.device) - depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.args.root.half_precision) - init.depth_model.save(os.path.join(args.outdir, f"{init.args.root.timestring}_depth_{frame_idx:09}.png"), depth) + depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, + init.args.root.half_precision) + init.depth_model.save( + os.path.join(init.output_directory, f"{init.args.root.timestring}_depth_{frame_idx:09}.png"), depth) if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') devices.torch_gc() @@ -684,34 +673,14 @@ def run_render_animation(init): init.animation_mode.cleanup() -def should_initialize_color_match(anim_args, hybrid_available, color_match_sample): - """Determines whether to initialize color matching based on the given conditions.""" - has_video_input = anim_args.color_coherence == 'Video Input' and hybrid_available - has_image_color_coherence = anim_args.color_coherence == 'Image' - has_any_color_sample = color_match_sample is not None - has_coherent_non_legacy_color_match = anim_args.color_coherence != 'None' and not anim_args.legacy_colormatch - has_sample_and_match = has_any_color_sample and has_coherent_non_legacy_color_match - return has_video_input or has_image_color_coherence or has_sample_and_match - - -def has_img2img_fix_steps(opts): - return 'img2img_fix_steps' in opts.data and opts.data["img2img_fix_steps"] - - -def set_if_not_none(dictionary, key, value): - # TODO Helper method, move elsewhere? - if value is not None: - dictionary[key] = value - - -def setup_opts(opts, schedule): - if has_img2img_fix_steps(opts): +def setup_opts(init, schedule): + if init.has_img2img_fix_steps(): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations - opts.data["img2img_fix_steps"] = False # TODO is this ever true? - set_if_not_none(opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) - set_if_not_none(opts.data, "initial_noise_multiplier", schedule.noise_multiplier) - set_if_not_none(opts.data, "eta_ddim", schedule.eta_ddim) - set_if_not_none(opts.data, "eta_ancestral", schedule.eta_ancestral) + init.opts.data["img2img_fix_steps"] = False + put_if_present(init.args.opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) + put_if_present(init.args.opts.data, "initial_noise_multiplier", schedule.noise_multiplier) + put_if_present(init.args.opts.data, "eta_ddim", schedule.eta_ddim) + put_if_present(init.args.opts.data, "eta_ancestral", schedule.eta_ancestral) def progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame): diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 7f7d5de5f..90d45a602 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -44,6 +44,8 @@ class RenderInit: animation_mode: AnimationMode prompt_series: Any depth_model: Any + output_directory: str + is_use_mask: bool def __new__(cls, *args, **kwargs): raise TypeError("Use RenderInit.create() to create new instances.") @@ -62,7 +64,56 @@ def height(self) -> int: def dimensions(self) -> tuple[int, int]: # TODO should ideally only be called once each render - return self.width(), self.height() + return (self.width(), self.height()) + + # TODO group hybrid stuff elsewhere + def is_hybrid_composite(self) -> bool: + return self.args.anim_args.hybrid_composite != 'None' + + def is_normal_hybrid_composite(self) -> bool: + return self.args.anim_args.hybrid_composite == 'Normal' + + def has_hybrid_motion(self) -> bool: + return self.args.anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective'] + + def is_hybrid_available(self) -> bool: + return self.is_hybrid_composite() or self.has_hybrid_motion() + + def is_hybrid_composite_before_motion(self) -> bool: + return self.args.anim_args.hybrid_composite == 'Before Motion' + + def is_hybrid_composite_after_generation(self) -> bool: + return self.args.anim_args.hybrid_composite == 'After Generation' + + def is_color_match_to_be_initialized(self, color_match_sample): + """Determines whether to initialize color matching based on the given conditions.""" + has_video_input = self.args.anim_args.color_coherence == 'Video Input' and self.is_hybrid_available() + has_image_color_coherence = self.args.anim_args.color_coherence == 'Image' + has_any_color_sample = color_match_sample is not None # TODO extract to own method? + has_coherent_non_legacy_color_match = (self.args.anim_args.color_coherence != 'None' + and not self.args.anim_args.legacy_colormatch) + has_sample_and_match = has_any_color_sample and has_coherent_non_legacy_color_match + return has_video_input or has_image_color_coherence or has_sample_and_match + + def has_color_coherence(self): + return self.args.anim_args.color_coherence != 'None' + + def is_resuming_from_timestring(self): + return self.args.anim_args.resume_from_timestring + + def has_video_input(self): + return self.animation_mode.has_video_input + + def has_img2img_fix_steps(self): + return 'img2img_fix_steps' in self.args.opts.data and self.args.opts.data["img2img_fix_steps"] + + def cadence(self) -> int: + return int(self.args.anim_args.diffusion_cadence) + + @classmethod + def create_output_directory_for_the_batch(self, dir): + os.makedirs(dir, exist_ok=True) + print(f"Saving animation frames to:\n{dir}") @classmethod def create_parseq_adapter(cls, args): @@ -123,11 +174,6 @@ def handle_controlnet_video_input_frames_generation(cls, controlnet_args, args, if is_controlnet_enabled(controlnet_args): unpack_controlnet_vids(args, anim_args, controlnet_args) - @classmethod - def create_output_directory_for_the_batch(cls, args): - os.makedirs(args.outdir, exist_ok=True) - print(f"Saving animation frames to:\n{args.outdir}") - @classmethod def save_settings_txt(cls, args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) @@ -143,16 +189,19 @@ def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, # attached as a property to this class to be used for one single render only. RenderInit.init_looper_if_active(args, loop_args) RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - RenderInit.create_output_directory_for_the_batch(args) + RenderInit.create_output_directory_for_the_batch(args.outdir) RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) RenderInit.maybe_resume_from_timestring(anim_args, root) @classmethod def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': + # TODO deepcopy args? args = RenderInitArgs(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + output_directory = args_argument.outdir + is_use_mask = args_argument.use_mask parseq_adapter = RenderInit.create_parseq_adapter(args) - srt = Srt.create_if_active(opts.data, args_argument.outdir, root.timestring, video_args.fps) + srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) animation_keys = AnimationKeys.from_args(args, parseq_adapter, args_argument.seed) animation_mode = AnimationMode.from_args(args) prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) @@ -160,7 +209,7 @@ def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_ar animation_mode, root, anim_args, args_argument) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. instance.__init__(args_argument.seed, args, parseq_adapter, srt, - animation_keys, animation_mode, prompt_series, depth_model) + animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. # It may be preferable to work on temporary copies within tight scope. # TODO avoid or isolate more side effects diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index da035e878..c078b74c4 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,3 +1,4 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. from .memory_utils import MemoryUtils +from .utils import put_if_present, call_anim_frame_warp diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py new file mode 100644 index 000000000..aacd4533e --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -0,0 +1,19 @@ +from ...animation import anim_frame_warp + + +def put_if_present(dictionary, key, value): + # FIXME does this even make sense? (..reading py docs) + if value is not None: + dictionary[key] = value + + +def call_anim_frame_warp(init, image, frame_index, depth): + return anim_frame_warp(image, + init.args.args, + init.args.anim_args, + init.animation_keys.deform_keys, + frame_index, + init.depth_model, + depth=depth, + device=init.args.root.device, + half_precision=init.args.root.half_precision) From bfa7f04619208fd342f82a73190c526e7ba00d39 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 3 Jun 2024 20:49:27 +0200 Subject: [PATCH 029/132] Created call_utils to funnel some calls and regroup their arguments. --- scripts/deforum_helpers/render.py | 103 ++++++------------ .../rendering/data/anim/animation_mode.py | 10 +- .../rendering/util/__init__.py | 4 +- .../rendering/util/call_utils.py | 69 ++++++++++++ .../deforum_helpers/rendering/util/utils.py | 13 --- 5 files changed, 112 insertions(+), 87 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/call_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 582cdb280..eb8137c60 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -29,10 +29,9 @@ from .colors import maintain_colors from .composable_masks import compose_mask_with_check -from .generate import generate -from .hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, +from .hybrid_video import (get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, - image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, + image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask from .load_images import get_mask, load_img, load_image, get_mask_from_file @@ -42,12 +41,14 @@ from .rendering.data.schedule import Schedule from .rendering.initialization import RenderInit from .rendering.util import put_if_present, call_anim_frame_warp +from .rendering.util.call_utils import call_get_flow_from_images, call_generate, call_render_preview, \ + call_hybrid_composite from .rendering.util.memory_utils import MemoryUtils from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed from .subtitle_handler import write_frame_subtitle, format_animation_params -from .video_audio_utilities import get_frame_name, get_next_frame, render_preview +from .video_audio_utilities import get_frame_name, get_next_frame def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -145,8 +146,10 @@ def run_render_animation(init): # TODO create a Step class in rendering.data with all the iteration specific info, # then eventually try to replace this while loop with functions that: - # - 1. Create a collection of Steps with all the required info that is already known before we enter the iteration. - # - 2. Transform and process the steps however needed (i.e. space out or reassign turbo frames etc.) + # - 1. Create a collection of Steps with all the required info that is already known or can be calculated + # before we enter the iteration. + # - 2. Transform and reprocess the steps however needed (i.e. space out or reassign turbo frames etc.) + # TODO cadence framing and logic that is currently working off-index may eventually be moved into a 2nd pass. # - 3. Actually do the render by foreaching over the steps in sequence while frame_idx < init.args.anim_args.max_frames: # Webui @@ -227,9 +230,8 @@ def run_render_animation(init): and init.args.anim_args.optical_flow_cadence != 'None'): if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: - cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, - init.args.anim_args.optical_flow_cadence, - init.animation_mode.raft_model) / 2 + cadence_flow = call_get_flow_from_images(init, turbo_prev_image, turbo_next_image, + init.args.anim_args.optical_flow_cadence) / 2 turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): @@ -251,10 +253,9 @@ def run_render_animation(init): init.args.root.half_precision) if advance_prev: - turbo_prev_image, _ = call_anim_frame_warp(init, turbo_prev_image, tween_frame_idx, depth) + turbo_prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo_prev_image, depth) if advance_next: - turbo_next_image, _ = call_anim_frame_warp(init, turbo_next_image, tween_frame_idx, depth) - + turbo_prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo_next_image, depth) # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion if tween_frame_idx > 0: @@ -319,20 +320,19 @@ def run_render_animation(init): hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow + # TODO cadence related transforms to be decoupled and handled in a 2nd pass # do optical flow cadence after animation warping if cadence_flow is not None: - cadence_flow = abs_flow_to_rel_flow(cadence_flow, - init.width(), - init.height()) - cadence_flow, _ = call_anim_frame_warp(init, cadence_flow, tween_frame_idx, depth) - cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, - init.dimensions().width(), - init.dimensions().height()) * tween + cadence_flow = abs_flow_to_rel_flow(cadence_flow, init.width(), init.height()) + cadence_flow, _ = call_anim_frame_warp(init, tween_frame_idx, cadence_flow, depth) + cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, init.width(), init.height()) * tween if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, cadence_flow_inc, + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, + cadence_flow_inc, cadence_flow_factor) if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, cadence_flow_inc, + turbo_next_image = image_transform_optical_flow(turbo_next_image, + cadence_flow_inc, cadence_flow_factor) turbo_prev_frame_idx = turbo_next_frame_idx = tween_frame_idx @@ -378,18 +378,12 @@ def run_render_animation(init): # after 1st frame, prev_img exists if prev_img is not None: # apply transforms to previous frame - prev_img, depth = call_anim_frame_warp(init, prev_img, frame_idx, None) + prev_img, depth = call_anim_frame_warp(init, frame_idx, prev_img, None) # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = hybrid_composite(init.args.args, - init.args.anim_args, - frame_idx, - prev_img, - init.depth_model, - hybrid_comp_schedules, - init.args.root) + _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, hybrid_comp_schedules) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: @@ -433,9 +427,7 @@ def run_render_animation(init): # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, prev_img, - init.depth_model, hybrid_comp_schedules, init.args.root) - + _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): if color_match_sample is None: @@ -537,6 +529,7 @@ def run_render_animation(init): lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) + # TODO try init early, also see "call_get_flow_from_images" optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ if not init.args.args.motion_preview_mode else 'None' @@ -547,17 +540,10 @@ def run_render_animation(init): print( f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") - disposable_image = generate(init.args.args, - init.animation_keys.deform_keys, - init.args.anim_args, - init.args.loop_args, - init.args.controlnet_args, - init.args.root, - init.parseq_adapter, - frame_idx, sampler_name=schedule.sampler_name) + disposable_image = call_generate(init, frame_idx, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) - disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, - init.animation_mode.raft_model) + disposable_flow = call_get_flow_from_images(init, prev_img, disposable_image, + optical_flow_redo_generation) / 2 disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) init.args.args.seed = stored_seed @@ -573,14 +559,7 @@ def run_render_animation(init): for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") init.args.args.seed = random.randint(0, 2 ** 32 - 1) - disposable_image = generate(init.args.args, - init.animation_keys.deform_keys, - init.args.anim_args, - init.args.loop_args, - init.args.controlnet_args, - init.args.root, - init.parseq_adapter, - frame_idx, sampler_name=schedule.sampler_name) + disposable_image = call_generate(init, frame_idx, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(init.args.anim_args.diffusion_redo): @@ -592,10 +571,7 @@ def run_render_animation(init): gc.collect() # TODO try to eventually kick the gc only once at the end of every generation or iteration. # generation - image = generate(init.args.args, init.animation_keys.deform_keys, init.args.anim_args, - init.args.loop_args, init.args.controlnet_args, - init.args.root, init.parseq_adapter, frame_idx, - sampler_name=schedule.sampler_name) + image = call_generate(init, frame_idx, schedule) if image is None: break @@ -604,8 +580,7 @@ def run_render_animation(init): if frame_idx > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = hybrid_composite(init.args.args, init.args.anim_args, frame_idx, temp_image, - init.depth_model, hybrid_comp_schedules, init.args.root) + _, temp_image_2 = call_hybrid_composite(init, frame_idx, temp_image, hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, @@ -660,15 +635,7 @@ def run_render_animation(init): sd_hijack.model_hijack.hijack(sd_model) frame_idx += 1 - # TODO isolate more state.... - last_preview_frame = progress_and_make_preview(state, - image, - init.args.args, - init.args.anim_args, - init.args.video_args, - init.args.root, - frame_idx, - last_preview_frame) + last_preview_frame = progress_and_make_preview(init, image, frame_idx, state, last_preview_frame) update_tracker(init.args.root, frame_idx, init.args.anim_args) init.animation_mode.cleanup() @@ -683,10 +650,10 @@ def setup_opts(init, schedule): put_if_present(init.args.opts.data, "eta_ancestral", schedule.eta_ancestral) -def progress_and_make_preview(state, image, args, anim_args, video_args, root, frame_idx, last_preview_frame): +def progress_and_make_preview(init, image, frame_idx, state, last_preview_frame): state.assign_current_image(image) - args.seed = next_seed(args, root) - return render_preview(args, anim_args, video_args, root, frame_idx, last_preview_frame) + init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO refactor assignment + return call_render_preview(init, frame_idx, last_preview_frame) def update_tracker(root, frame_idx, anim_args): diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 0df36de56..99de5b96f 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -31,20 +31,20 @@ def cleanup(self): @staticmethod def _has_video_input(anim_args) -> bool: - return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) + return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod def _is_2d_or_3d_mode(anim_args): return anim_args.animation_mode in ['2D', '3D'] @staticmethod - def _is_using_hybris_frames(anim_args): + def _is_using_hybrid_frames(anim_args): return (anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']) @staticmethod - def _is_needing_hybris_frames(anim_args): - return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybris_frames(anim_args) + def _is_needing_hybrid_frames(anim_args): + return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod def _is_load_depth_model_for_3d(args, anim_args): @@ -72,7 +72,7 @@ def _load_depth_model_if_active(args, anim_args, opts): def from_args(step_args): init_hybrid_input_files: Any = None init_hybrid_frame_path = "" - if AnimationMode._is_needing_hybris_frames(step_args.anim_args): + if AnimationMode._is_needing_hybrid_frames(step_args.anim_args): # handle hybrid video generation # hybrid_generation may cause side effects on args and anim_args _, __, init_hybrid_input_files = hybrid_generation(step_args.args, step_args.anim_args, step_args.root) diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index c078b74c4..0ea5a3d80 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,4 +1,6 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. +from .call_utils import (call_anim_frame_warp, call_generate, call_get_flow_from_images, + call_hybrid_composite, call_render_preview) from .memory_utils import MemoryUtils -from .utils import put_if_present, call_anim_frame_warp +from .utils import put_if_present diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py new file mode 100644 index 000000000..bc240a1bf --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call_utils.py @@ -0,0 +1,69 @@ +from ...animation import anim_frame_warp +from ...generate import generate +from ...hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, + get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, + image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, + rel_flow_to_abs_flow) +from ...video_audio_utilities import get_frame_name, get_next_frame, render_preview + + +# Purpose: +# This module mostly exists for refactoring and reducing the complexity of render.py without touching any other modules. +# Currently useful to reduce complexity in calls with many arguments, that have been or will be regrouped into "init". +# Alternatively all methods may be overloaded in the original modules, but doing so would propagate the Init class to +# namespaces where it doesn't really belong, which is rather undesirable. +# +# Form: +# The following functions shouldn't contain any logic and directly return with the call to the actual method. +# - Naming starts with "call_". +# - "init" to be passed as 1st argument. +# - pass frame_idx or twin_frame_idx or other indices as 2nd argument "i" where applicable. + +# Animation: +def call_anim_frame_warp(init, i, image, depth): + return anim_frame_warp(image, + init.args.args, + init.args.anim_args, + init.animation_keys.deform_keys, + i, + init.depth_model, + depth=depth, + device=init.args.root.device, + half_precision=init.args.root.half_precision) + + +# Generation: +def call_generate(init, i, schedule): + return generate(init.args.args, + init.animation_keys.deform_keys, + init.args.anim_args, + init.args.loop_args, + init.args.controlnet_args, + init.args.root, + init.parseq_adapter, + i, sampler_name=schedule.sampler_name) + + +# Hybrid Video +def call_get_flow_from_images(init, prev_image, next_image, cadence): + # cadence is currently either "optical_flow_redo_generation" or "init.args.anim_args.optical_flow_cadence" + # TODO try to init "optical_flow_redo_generation" early, then remove the "cadence" arg again + return get_flow_from_images(prev_image, next_image, cadence, init.animation_mode.raft_model) + + +def call_hybrid_composite(init, i, image, hybrid_comp_schedules): + return hybrid_composite(init.args.args, + init.args.anim_args, + i, image, + init.depth_model, + hybrid_comp_schedules, + init.args.root) + + +# Video & Audio +def call_render_preview(init, i, last_preview_frame): + return render_preview(init.args.args, + init.args.anim_args, + init.args.video_args, + init.args.root, + i, last_preview_frame) diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index aacd4533e..4b48d44ca 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,19 +1,6 @@ -from ...animation import anim_frame_warp def put_if_present(dictionary, key, value): - # FIXME does this even make sense? (..reading py docs) if value is not None: dictionary[key] = value - -def call_anim_frame_warp(init, image, frame_index, depth): - return anim_frame_warp(image, - init.args.args, - init.args.anim_args, - init.animation_keys.deform_keys, - frame_index, - init.depth_model, - depth=depth, - device=init.args.root.device, - half_precision=init.args.root.half_precision) From a4107a0d0d26e87cf310d8e9c4d0037933bef2f6 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 3 Jun 2024 22:07:56 +0200 Subject: [PATCH 030/132] More calls delegated. --- scripts/deforum_helpers/render.py | 134 +++++------------- .../rendering/util/__init__.py | 6 +- .../rendering/util/call_utils.py | 73 +++++++++- 3 files changed, 109 insertions(+), 104 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index eb8137c60..991324d06 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -29,26 +29,27 @@ from .colors import maintain_colors from .composable_masks import compose_mask_with_check -from .hybrid_video import (get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, - get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, - image_transform_optical_flow, abs_flow_to_rel_flow, - rel_flow_to_abs_flow) +from .hybrid_video import (image_transform_ransac, image_transform_optical_flow, + abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask -from .load_images import get_mask, load_img, load_image, get_mask_from_file +from .load_images import get_mask, load_img, load_image from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt from .rendering.data.schedule import Schedule from .rendering.initialization import RenderInit from .rendering.util import put_if_present, call_anim_frame_warp -from .rendering.util.call_utils import call_get_flow_from_images, call_generate, call_render_preview, \ - call_hybrid_composite +from .rendering.util.call_utils import (call_get_flow_from_images, call_generate, call_render_preview, + call_hybrid_composite, call_format_animation_params, + call_write_frame_subtitle, call_get_mask_from_file_with_frame, + call_get_mask_from_file, call_get_next_frame, + call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, + call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev) from .rendering.util.memory_utils import MemoryUtils from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed -from .subtitle_handler import write_frame_subtitle, format_animation_params -from .video_audio_utilities import get_frame_name, get_next_frame +from .video_audio_utilities import get_frame_name def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -105,8 +106,8 @@ def run_render_animation(init): if (init.args.args.use_init and ((init.args.args.init_image != None - and init.args.args.init_image != '') - or init.args.args.init_image_box != None)): + and init.args.args.init_image != '') + or init.args.args.init_image_box != None)): _, mask_image = load_img(init.args.args.init_image, init.args.args.init_image_box, shape=init.dimensions(), @@ -118,18 +119,12 @@ def run_render_animation(init): # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 if init.args.anim_args.use_mask_video: - init.args.args.mask_file = get_mask_from_file( - get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), - init.args.args) - init.args.root.noise_mask = get_mask_from_file( - get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), - init.args.args) - mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), - init.args.args) - noise_mask_vals['video_mask'] = get_mask_from_file( - get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, frame_idx, True), - init.args.args) + temp_mask = call_get_mask_from_file(init, frame_idx, True) + init.args.args.mask_file = temp_mask + init.args.args.mask_file = temp_mask + init.args.root.noise_mask = temp_mask + mask_vals['video_mask'] = temp_mask + noise_mask_vals['video_mask'] = temp_mask elif mask_image is None and init.is_use_mask: mask_vals['video_mask'] = get_mask(init.args.args) noise_mask_vals['video_mask'] = get_mask(init.args.args) # TODO?: add a different default noisc mask @@ -206,10 +201,8 @@ def run_render_animation(init): if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(init.animation_keys.deform_keys, init.prompt_series, frame_idx, - params_to_print) - write_frame_subtitle(init.srt.filename, frame_idx, init.srt.frame_duration, - f"F#: {frame_idx}; Cadence: false; Seed: {init.args.args.seed}; {params_string}") + params_string = call_format_animation_params(init, frame_idx, params_to_print) + call_write_frame_subtitle(init, frame_idx, params_string) params_string = None # FIXME ?? # emit in-between frames @@ -236,12 +229,8 @@ def run_render_animation(init): if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = format_animation_params(init.animation_keys.deform_keys, - init.prompt_series, - tween_frame_idx, - params_to_print) - write_frame_subtitle(init.srt.filename, tween_frame_idx, init.srt.frame_duration, - f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {init.args.args.seed}; {params_string}") + params_string = call_format_animation_params(init, tween_frame_idx, params_to_print) + call_write_frame_subtitle(init, tween_frame_idx, params_string, tween < 1.0) params_string = None print( @@ -261,10 +250,7 @@ def run_render_animation(init): if tween_frame_idx > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, prev_img, - init.args.anim_args.hybrid_motion) + matrix = call_get_matrix_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, init.args.anim_args.hybrid_motion) @@ -272,10 +258,7 @@ def run_render_animation(init): turbo_next_image = image_transform_ransac(turbo_next_image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, - init.args.anim_args.hybrid_motion) + matrix = call_get_matrix_for_hybrid_motion(init, tween_frame_idx - 1) if advance_prev: turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, init.args.anim_args.hybrid_motion) @@ -284,17 +267,7 @@ def run_render_animation(init): init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, - prev_img, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + flow = call_get_flow_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) if advance_prev: turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) @@ -303,15 +276,7 @@ def run_render_animation(init): hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow # FIXME shouldn't else: - flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + flow = call_get_flow_for_hybrid_motion(init, tween_frame_idx - 1) if advance_prev: turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) @@ -363,7 +328,8 @@ def run_render_animation(init): cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(init.output_directory, f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png") + dm_save_path = os.path.join(init.output_directory, + f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png") init.depth_model.save(dm_save_path, depth) # get color match for video outside of prev_img conditional @@ -388,39 +354,15 @@ def run_render_animation(init): # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, prev_img, - init.args.anim_args.hybrid_motion) + matrix = call_get_matrix_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) else: - matrix = get_matrix_for_hybrid_motion(frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, - init.args.anim_args.hybrid_motion) + matrix = call_get_matrix_for_hybrid_motion(init, frame_idx - 1) prev_img = image_transform_ransac(prev_img, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, prev_img, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + flow = call_get_flow_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) else: - flow = get_flow_for_hybrid_motion(frame_idx - 1, - init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + flow = call_get_flow_for_hybrid_motion(init, frame_idx - 1) prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow @@ -450,7 +392,8 @@ def run_render_animation(init): if init.args.args.use_mask or init.args.anim_args.use_noise_mask: init.args.root.noise_mask = compose_mask_with_check(init.args.root, init.args.args, - noise_mask_seq, # FIXME might be ref'd b4 assignment + noise_mask_seq, + # FIXME might be ref'd b4 assignment noise_mask_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) @@ -500,16 +443,15 @@ def run_render_animation(init): # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = get_next_frame(init.output_directory, init.args.anim_args.video_init_path, - frame_idx, False) + init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_init_path) print(f"Using video init frame {init_frame}") init.args.args.init_image = init_frame init.args.args.init_image_box = None # init_image_box not used in this case init.args.args.strength = max(0.0, min(1.0, strength)) if init.args.anim_args.use_mask_video: - mask_init_frame = get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, - frame_idx, True) - temp_mask = get_mask_from_file(mask_init_frame, init.args.args) + mask_init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_mask_path, True) + temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) + init.args.args.mask_file = temp_mask init.args.root.noise_mask = temp_mask mask_vals['video_mask'] = temp_mask diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 0ea5a3d80..5b6862201 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,6 +1,8 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .call_utils import (call_anim_frame_warp, call_generate, call_get_flow_from_images, - call_hybrid_composite, call_render_preview) +from .call_utils import (call_anim_frame_warp, call_generate, call_get_flow_from_images, call_hybrid_composite, + call_render_preview, call_format_animation_params, call_get_next_frame, + call_get_matrix_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, + call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion) from .memory_utils import MemoryUtils from .utils import put_if_present diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py index bc240a1bf..fa718c9fb 100644 --- a/scripts/deforum_helpers/rendering/util/call_utils.py +++ b/scripts/deforum_helpers/rendering/util/call_utils.py @@ -1,10 +1,11 @@ from ...animation import anim_frame_warp from ...generate import generate -from ...hybrid_video import (hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, - get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, - image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, - rel_flow_to_abs_flow) -from ...video_audio_utilities import get_frame_name, get_next_frame, render_preview +from ...hybrid_video import (hybrid_composite, get_flow_from_images, + get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, + get_flow_for_hybrid_motion_prev, get_flow_for_hybrid_motion) +from ...load_images import get_mask_from_file +from ...subtitle_handler import format_animation_params, write_frame_subtitle +from ...video_audio_utilities import get_next_frame, render_preview # Purpose: @@ -14,7 +15,8 @@ # namespaces where it doesn't really belong, which is rather undesirable. # # Form: -# The following functions shouldn't contain any logic and directly return with the call to the actual method. +# The following functions shouldn't contain any complex logic besides perhaps some formatting +# and aim to directly return with the call to the actual method. # - Naming starts with "call_". # - "init" to be passed as 1st argument. # - pass frame_idx or twin_frame_idx or other indices as 2nd argument "i" where applicable. @@ -51,6 +53,41 @@ def call_get_flow_from_images(init, prev_image, next_image, cadence): return get_flow_from_images(prev_image, next_image, cadence, init.animation_mode.raft_model) +def call_get_flow_for_hybrid_motion_prev(init, i, image): + return get_flow_for_hybrid_motion_prev(i, init.dimensions(), + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, + image, + init.args.anim_args.hybrid_flow_method, + init.animation_mode.raft_model, + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) + + +def call_get_flow_for_hybrid_motion(init, i): + return get_flow_for_hybrid_motion(i, init.dimensions(), + init.animation_mode.hybrid_input_files, + init.animation_mode.hybrid_frame_path, + init.animation_mode.prev_flow, + init.args.anim_args.hybrid_flow_method, + init.animation_mode.raft_model, + init.args.anim_args.hybrid_flow_consistency, + init.args.anim_args.hybrid_consistency_blur, + init.args.anim_args.hybrid_comp_save_extra_frames) + + +def call_get_matrix_for_hybrid_motion_prev(init, i, image): + return get_matrix_for_hybrid_motion_prev(i, init.dimensions(), init.animation_mode.hybrid_input_files, + image, init.args.anim_args.hybrid_motion) + + +def call_get_matrix_for_hybrid_motion(init, i): + return get_matrix_for_hybrid_motion(i, init.dimensions(), init.animation_mode.hybrid_input_files, + init.args.anim_args.hybrid_motion) + + def call_hybrid_composite(init, i, image, hybrid_comp_schedules): return hybrid_composite(init.args.args, init.args.anim_args, @@ -60,6 +97,26 @@ def call_hybrid_composite(init, i, image, hybrid_comp_schedules): init.args.root) +# Load Images +def call_get_mask_from_file(init, i, is_mask: bool = False): + next_frame = get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, i, is_mask) + return get_mask_from_file(next_frame, init.args.args) + + +def call_get_mask_from_file_with_frame(init, frame): + return get_mask_from_file(frame, init.args.args) + + +# Subtitle +def call_format_animation_params(init, i, params_to_print): + return format_animation_params(init.animation_keys.deform_keys, init.prompt_series, i, params_to_print) + + +def call_write_frame_subtitle(init, i, params_string, is_cadence: bool = False) -> None: + text = f"F#: {i}; Cadence: {is_cadence}; Seed: {init.args.args.seed}; {params_string}" + write_frame_subtitle(init.srt.filename, i, init.srt.frame_duration, text) + + # Video & Audio def call_render_preview(init, i, last_preview_frame): return render_preview(init.args.args, @@ -67,3 +124,7 @@ def call_render_preview(init, i, last_preview_frame): init.args.video_args, init.args.root, i, last_preview_frame) + + +def call_get_next_frame(init, i, video_path, is_mask: bool = False): + return get_next_frame(init.output_directory, video_path, i, is_mask) From 7249563dc8fe55adc51711485db49762ea49e51a Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 3 Jun 2024 22:13:44 +0200 Subject: [PATCH 031/132] Unnecessary divisor removed. --- scripts/deforum_helpers/render.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 991324d06..051fc33e3 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -484,8 +484,7 @@ def run_render_animation(init): disposable_image = call_generate(init, frame_idx, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) - disposable_flow = call_get_flow_from_images(init, prev_img, disposable_image, - optical_flow_redo_generation) / 2 + disposable_flow = call_get_flow_from_images(init, prev_img, disposable_image, optical_flow_redo_generation) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) init.args.args.seed = stored_seed From d62d98bbacb4dda621e362abc538ab5af9308bc9 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 4 Jun 2024 00:32:13 +0200 Subject: [PATCH 032/132] New put_all helper and more cleanup. --- scripts/deforum_helpers/render.py | 41 ++++--- .../rendering/data/schedule.py | 110 +++++++++--------- .../rendering/initialization.py | 18 ++- .../deforum_helpers/rendering/util/utils.py | 4 + 4 files changed, 94 insertions(+), 79 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 051fc33e3..9564ea9ba 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -46,6 +46,7 @@ call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev) from .rendering.util.memory_utils import MemoryUtils +from .rendering.util.utils import put_all from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed @@ -99,35 +100,19 @@ def run_render_animation(init): mask_vals = {} noise_mask_vals = {} - mask_vals['everywhere'] = Image.new('1', init.dimensions(), 1) - noise_mask_vals['everywhere'] = Image.new('1', init.dimensions(), 1) + put_all([mask_vals, noise_mask_vals], 'everywhere', + lambda: Image.new('1', init.dimensions(), 1)) mask_image = None - if (init.args.args.use_init - and ((init.args.args.init_image != None - and init.args.args.init_image != '') - or init.args.args.init_image_box != None)): + if init.is_using_init_image_or_box(): _, mask_image = load_img(init.args.args.init_image, init.args.args.init_image_box, shape=init.dimensions(), use_alpha_as_mask=init.args.args.use_alpha_as_mask) - mask_vals['video_mask'] = mask_image - noise_mask_vals['video_mask'] = mask_image + put_all([mask_vals, noise_mask_vals], 'video_mask', mask_image) - # Grab the first frame masks since they wont be provided until next frame - # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set - # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 - if init.args.anim_args.use_mask_video: - temp_mask = call_get_mask_from_file(init, frame_idx, True) - init.args.args.mask_file = temp_mask - init.args.args.mask_file = temp_mask - init.args.root.noise_mask = temp_mask - mask_vals['video_mask'] = temp_mask - noise_mask_vals['video_mask'] = temp_mask - elif mask_image is None and init.is_use_mask: - mask_vals['video_mask'] = get_mask(init.args.args) - noise_mask_vals['video_mask'] = get_mask(init.args.args) # TODO?: add a different default noisc mask + assign_masks(init, frame_idx, mask_image, [mask_vals, noise_mask_vals]) # get color match for 'Image' color coherence only once, before loop if init.args.anim_args.color_coherence == 'Image': @@ -581,6 +566,20 @@ def run_render_animation(init): init.animation_mode.cleanup() +def assign_masks(init, i, is_mask_image, dicts): + # Grab the first frame masks since they wont be provided until next frame + # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set + # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 + key = 'video_mask' + if init.args.anim_args.use_mask_video: + mask = call_get_mask_from_file(init, i, True) + init.args.args.mask_file = mask + init.args.root.noise_mask = mask + put_all(dicts, key, mask) + elif is_mask_image is None and init.is_use_mask: + put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask + + def setup_opts(init, schedule): if init.has_img2img_fix_steps(): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index 639291d91..9b77bd1b6 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -21,73 +21,73 @@ def __new__(cls, *args, **kwargs): # locks the normal constructor to enforce pr def create(cls, keys, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" - steps = cls.schedule_steps(keys, i, anim_args) - sampler_name = cls.schedule_sampler(keys, i, anim_args) - clipskip = cls.schedule_clipskip(keys, i, anim_args) - noise_multiplier = cls.schedule_noise_multiplier(keys, i, anim_args) - eta_ddim = cls.schedule_ddim_eta(keys, i, anim_args) - eta_ancestral = cls.schedule_ancestral_eta(keys, i, anim_args) - mask = cls.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args - noise_mask = cls.schedule_noise_mask(keys, i, anim_args) + steps = Schedule.schedule_steps(keys, i, anim_args) + sampler_name = Schedule.schedule_sampler(keys, i, anim_args) + clipskip = Schedule.schedule_clipskip(keys, i, anim_args) + noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) + mask = Schedule.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args + noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. instance.__init__(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) return instance - @classmethod - def _has_schedule(cls, keys, i): + @staticmethod + def _has_schedule(keys, i): return keys.steps_schedule_series[i] is not None - @classmethod - def _has_mask_schedule(cls, keys, i): + @staticmethod + def _has_mask_schedule(keys, i): return keys.mask_schedule_series[i] is not None - @classmethod - def _has_noise_mask_schedule(cls, keys, i): + @staticmethod + def _has_noise_mask_schedule(keys, i): return keys.noise_mask_schedule_series[i] is not None - @classmethod - def _use_on_cond_if_scheduled(cls, keys, i, value, cond): - return value if cond and cls._has_schedule(keys, i) else None - - @classmethod - def schedule_steps(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, int(keys.steps_schedule_series[i]), - anim_args.enable_steps_scheduling) - - @classmethod - def schedule_sampler(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, keys.sampler_schedule_series[i].casefold(), - anim_args.enable_sampler_scheduling) - - @classmethod - def schedule_clipskip(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, int(keys.clipskip_schedule_series[i]), - anim_args.enable_clipskip_scheduling) - - @classmethod - def schedule_noise_multiplier(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, float(keys.noise_multiplier_schedule_series[i]), - anim_args.enable_noise_multiplier_scheduling) - - @classmethod - def schedule_ddim_eta(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, float(keys.ddim_eta_schedule_series[i]), - anim_args.enable_ddim_eta_scheduling) - - @classmethod - def schedule_ancestral_eta(cls, keys, i, anim_args): - return cls._use_on_cond_if_scheduled(keys, i, float(keys.ancestral_eta_schedule_series[i]), - anim_args.enable_ancestral_eta_scheduling) - - @classmethod - def schedule_mask(cls, keys, i, args): + @staticmethod + def _use_on_cond_if_scheduled(keys, i, value, cond): + return value if cond and Schedule._has_schedule(keys, i) else None + + @staticmethod + def schedule_steps(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, int(keys.steps_schedule_series[i]), + anim_args.enable_steps_scheduling) + + @staticmethod + def schedule_sampler(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, keys.sampler_schedule_series[i].casefold(), + anim_args.enable_sampler_scheduling) + + @staticmethod + def schedule_clipskip(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, int(keys.clipskip_schedule_series[i]), + anim_args.enable_clipskip_scheduling) + + @staticmethod + def schedule_noise_multiplier(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, float(keys.noise_multiplier_schedule_series[i]), + anim_args.enable_noise_multiplier_scheduling) + + @staticmethod + def schedule_ddim_eta(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, float(keys.ddim_eta_schedule_series[i]), + anim_args.enable_ddim_eta_scheduling) + + @staticmethod + def schedule_ancestral_eta(keys, i, anim_args): + return Schedule._use_on_cond_if_scheduled(keys, i, float(keys.ancestral_eta_schedule_series[i]), + anim_args.enable_ancestral_eta_scheduling) + + @staticmethod + def schedule_mask(keys, i, args): # TODO can we have a mask schedule without a normal schedule? if so check and optimize return keys.mask_schedule_series[i] \ - if args.use_mask and cls._has_mask_schedule(keys, i) else None + if args.use_mask and Schedule._has_mask_schedule(keys, i) else None - @classmethod - def schedule_noise_mask(cls, keys, i, anim_args): - #TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize + @staticmethod + def schedule_noise_mask(keys, i, anim_args): + # TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize return keys.noise_mask_schedule_series[i] \ - if anim_args.use_noise_mask and cls._has_noise_mask_schedule(keys, i) else None + if anim_args.use_noise_mask and Schedule._has_noise_mask_schedule(keys, i) else None diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 90d45a602..85ece2b4b 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -110,10 +110,22 @@ def has_img2img_fix_steps(self): def cadence(self) -> int: return int(self.args.anim_args.diffusion_cadence) + def _has_init_image(self) -> bool: + return self.args.args.init_image is not None and self.args.args.init_image != '' + + def _has_init_box(self) -> bool: + return self.args.args.init_image_box is not None + + def _has_init_image_or_box(self) -> bool: + return self._has_init_image() or self._has_init_box() + + def is_using_init_image_or_box(self) -> bool: + return self.args.args.use_init and self._has_init_image_or_box() + @classmethod - def create_output_directory_for_the_batch(self, dir): - os.makedirs(dir, exist_ok=True) - print(f"Saving animation frames to:\n{dir}") + def create_output_directory_for_the_batch(cls, directory): + os.makedirs(directory, exist_ok=True) + print(f"Saving animation frames to:\n{directory}") @classmethod def create_parseq_adapter(cls, args): diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index 4b48d44ca..bd86a9eed 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -4,3 +4,7 @@ def put_if_present(dictionary, key, value): if value is not None: dictionary[key] = value + +def put_all(dictionaries, key, callable_or_value): + for dictionary in dictionaries: + dictionary[key] = callable_or_value() if callable(callable_or_value) else callable_or_value From 692f53e048a141352b91c99d095304a66879e9b2 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 4 Jun 2024 00:43:21 +0200 Subject: [PATCH 033/132] Invalid opts ref fixed. --- scripts/deforum_helpers/render.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 9564ea9ba..3633b1294 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -581,13 +581,14 @@ def assign_masks(init, i, is_mask_image, dicts): def setup_opts(init, schedule): + data = init.args.opts.data if init.has_img2img_fix_steps(): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations - init.opts.data["img2img_fix_steps"] = False - put_if_present(init.args.opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) - put_if_present(init.args.opts.data, "initial_noise_multiplier", schedule.noise_multiplier) - put_if_present(init.args.opts.data, "eta_ddim", schedule.eta_ddim) - put_if_present(init.args.opts.data, "eta_ancestral", schedule.eta_ancestral) + data["img2img_fix_steps"] = False + put_if_present(data, "CLIP_stop_at_last_layers", schedule.clipskip) + put_if_present(data, "initial_noise_multiplier", schedule.noise_multiplier) + put_if_present(data, "eta_ddim", schedule.eta_ddim) + put_if_present(data, "eta_ancestral", schedule.eta_ancestral) def progress_and_make_preview(init, image, frame_idx, state, last_preview_frame): From 53b10da5569cd0c1fefbdba847f73e1f9bb8d7f3 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 4 Jun 2024 02:11:13 +0200 Subject: [PATCH 034/132] Started isolating turbo frame generation related data and did some cleanup. --- scripts/deforum_helpers/render.py | 102 ++++++++---------- .../rendering/data/__init__.py | 1 + .../rendering/data/anim/animation_keys.py | 5 +- .../rendering/data/anim/animation_mode.py | 4 +- .../rendering/data/schedule.py | 17 +-- .../rendering/data/subtitle/srt.py | 24 ++--- .../deforum_helpers/rendering/data/turbo.py | 42 ++++++++ .../rendering/initialization.py | 11 +- .../rendering/util/memory_utils.py | 2 +- 9 files changed, 117 insertions(+), 91 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/turbo.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 3633b1294..19878a318 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -36,6 +36,7 @@ from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt +from .rendering.data import Turbo from .rendering.data.schedule import Schedule from .rendering.initialization import RenderInit from .rendering.util import put_if_present, call_anim_frame_warp @@ -68,10 +69,7 @@ def run_render_animation(init): # move that stuff to init and eventually try to drop init.args # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. - # state for interpolating between diffusion steps - turbo_steps = 1 if init.has_video_input() else init.cadence() - turbo_prev_image, turbo_prev_frame_idx = None, 0 - turbo_next_image, turbo_next_frame_idx = None, 0 + turbo = Turbo.create(init) # state for interpolating between diffusion steps # initialize vars prev_img = None @@ -84,12 +82,9 @@ def run_render_animation(init): prev_frame, next_frame, prev_img, next_img = get_resume_vars( folder=init.args.args.outdir, timestring=init.args.anim_args.resume_timestring, - cadence=turbo_steps) + cadence=turbo.steps) - # set up turbo step vars - if turbo_steps > 1: - turbo_prev_image, turbo_prev_frame_idx = prev_img, prev_frame - turbo_next_image, turbo_next_frame_idx = next_img, next_frame + turbo.set_up_step_vars(prev_img, prev_frame, next_img, next_frame) # advance start_frame to next frame start_frame = next_frame + 1 @@ -184,15 +179,14 @@ def run_render_animation(init): if init.animation_mode.is_predicting_depths: init.animation_mode.depth_model.to(init.args.root.device) - if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): + if turbo.is_first_step_with_subtitles(init): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) params_string = call_format_animation_params(init, frame_idx, params_to_print) call_write_frame_subtitle(init, frame_idx, params_string) params_string = None # FIXME ?? - # emit in-between frames - if turbo_steps > 1: - tween_frame_start_idx = max(start_frame, frame_idx - turbo_steps) + if turbo.is_emit_in_between_frames(): + tween_frame_start_idx = max(start_frame, frame_idx - turbo.steps) cadence_flow = None for tween_frame_idx in range(tween_frame_start_idx, frame_idx): # update progress during cadence @@ -200,17 +194,15 @@ def run_render_animation(init): state.job_no = tween_frame_idx + 1 # cadence vars tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx) - advance_prev = turbo_prev_image is not None and tween_frame_idx > turbo_prev_frame_idx - advance_next = tween_frame_idx > turbo_next_frame_idx # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: - if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: - cadence_flow = call_get_flow_from_images(init, turbo_prev_image, turbo_next_image, + if cadence_flow is None and turbo.prev_image is not None and turbo.next_image is not None: + cadence_flow = call_get_flow_from_images(init, turbo.prev_image, turbo.next_image, init.args.anim_args.optical_flow_cadence) / 2 - turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) + turbo.next_image = image_transform_optical_flow(turbo.next_image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) @@ -222,51 +214,52 @@ def run_render_animation(init): f"Creating in-between {'' if cadence_flow is None else init.args.anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") if init.depth_model is not None: - assert (turbo_next_image is not None) - depth = init.depth_model.predict(turbo_next_image, init.args.anim_args.midas_weight, + assert (turbo.next_image is not None) + depth = init.depth_model.predict(turbo.next_image, init.args.anim_args.midas_weight, init.args.root.half_precision) - if advance_prev: - turbo_prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo_prev_image, depth) - if advance_next: - turbo_prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo_next_image, depth) + # TODO collect images + if turbo.is_advance_prev(tween_frame_idx): + turbo.prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo.prev_image, depth) + if turbo.is_advance_next(tween_frame_idx): + turbo.prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo.next_image, depth) - # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion + # hybrid video motion - warps turbo.prev_image or turbo.next_image to match motion if tween_frame_idx > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: matrix = call_get_matrix_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) - if advance_prev: - turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, + if turbo.is_advance_prev(tween_frame_idx): + turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, init.args.anim_args.hybrid_motion) - if advance_next: - turbo_next_image = image_transform_ransac(turbo_next_image, matrix, + if turbo.is_advance_next(tween_frame_idx): + turbo.next_image = image_transform_ransac(turbo.next_image, matrix, init.args.anim_args.hybrid_motion) else: matrix = call_get_matrix_for_hybrid_motion(init, tween_frame_idx - 1) - if advance_prev: - turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, + if turbo.is_advance_prev(tween_frame_idx): + turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, init.args.anim_args.hybrid_motion) - if advance_next: - turbo_next_image = image_transform_ransac(turbo_next_image, matrix, + if turbo.is_advance_next(tween_frame_idx): + turbo.next_image = image_transform_ransac(turbo.next_image, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: flow = call_get_flow_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) - if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, + if turbo.is_advance_prev(tween_frame_idx): + turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, hybrid_comp_schedules['flow_factor']) - if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, + if turbo.is_advance_next(tween_frame_idx): + turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, hybrid_comp_schedules['flow_factor']) - init.animation_mode.prev_flow = flow # FIXME shouldn't + init.animation_mode.prev_flow = flow else: flow = call_get_flow_for_hybrid_motion(init, tween_frame_idx - 1) - if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, + if turbo.is_advance_prev(tween_frame_idx): + turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, hybrid_comp_schedules['flow_factor']) - if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, + if turbo.is_advance_next(tween_frame_idx): + turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, hybrid_comp_schedules['flow_factor']) init.animation_mode.prev_flow = flow @@ -276,21 +269,21 @@ def run_render_animation(init): cadence_flow = abs_flow_to_rel_flow(cadence_flow, init.width(), init.height()) cadence_flow, _ = call_anim_frame_warp(init, tween_frame_idx, cadence_flow, depth) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, init.width(), init.height()) * tween - if advance_prev: - turbo_prev_image = image_transform_optical_flow(turbo_prev_image, + if is_advance_prev: + turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, cadence_flow_factor) - if advance_next: - turbo_next_image = image_transform_optical_flow(turbo_next_image, + if is_advance_next: + turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, cadence_flow_factor) - turbo_prev_frame_idx = turbo_next_frame_idx = tween_frame_idx + turbo.prev_frame_idx = turbo.next_frame_idx = tween_frame_idx - if turbo_prev_image is not None and tween < 1.0: - img = turbo_prev_image * (1.0 - tween) + turbo_next_image * tween + if turbo.prev_image is not None and tween < 1.0: + img = turbo.prev_image * (1.0 - tween) + turbo.next_image * tween else: - img = turbo_next_image + img = turbo.next_image # intercept and override to grayscale if init.args.anim_args.color_force_grayscale: @@ -436,7 +429,6 @@ def run_render_animation(init): if init.args.anim_args.use_mask_video: mask_init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) - init.args.args.mask_file = temp_mask init.args.root.noise_mask = temp_mask mask_vals['video_mask'] = temp_mask @@ -535,10 +527,10 @@ def run_render_animation(init): if not init.animation_mode.has_video_input: prev_img = opencv_image - if turbo_steps > 1: - turbo_prev_image, turbo_prev_frame_idx = turbo_next_image, turbo_next_frame_idx - turbo_next_image, turbo_next_frame_idx = opencv_image, frame_idx - frame_idx += turbo_steps + if turbo.steps > 1: + turbo.prev_image, turbo.prev_frame_idx = turbo.next_image, turbo.next_frame_idx + turbo.next_image, turbo.next_frame_idx = opencv_image, frame_idx + frame_idx += turbo.steps else: filename = f"{init.args.root.timestring}_{frame_idx:09}.png" save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) diff --git a/scripts/deforum_helpers/rendering/data/__init__.py b/scripts/deforum_helpers/rendering/data/__init__.py index 7aef82b28..dfe07eb23 100644 --- a/scripts/deforum_helpers/rendering/data/__init__.py +++ b/scripts/deforum_helpers/rendering/data/__init__.py @@ -1 +1,2 @@ from .schedule import Schedule +from .turbo import Turbo diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index 9ad69e482..a0adeb3f1 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -1,9 +1,8 @@ -import dataclasses - +from dataclasses import dataclass from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class AnimationKeys: deform_keys: DeformAnimKeys looper_keys: LooperAnimKeys diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 99de5b96f..1fa35e0fa 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -1,13 +1,13 @@ -import dataclasses import os +from dataclasses import dataclass from typing import Any from ....hybrid_video import hybrid_generation from ....RAFT import RAFT # TODO FIXME find a way to assign prev_flow right away, then set frozen=true again, otherwise move prev_flow elsewhere. -@dataclasses.dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=False) class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index 9b77bd1b6..d398d9f92 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,9 +1,8 @@ -import dataclasses - +from dataclasses import dataclass from typing import Optional, Any -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: steps: int sampler_name: str @@ -14,11 +13,8 @@ class Schedule: mask: Optional[Any] noise_mask: Optional[Any] - def __new__(cls, *args, **kwargs): # locks the normal constructor to enforce proper initialization - raise TypeError("Use Schedule.create() to create new instances.") - - @classmethod - def create(cls, keys, i, anim_args, args): + @staticmethod + def create(keys, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" steps = Schedule.schedule_steps(keys, i, anim_args) @@ -29,10 +25,7 @@ def create(cls, keys, i, anim_args, args): eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) mask = Schedule.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) - - instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) - return instance + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/data/subtitle/srt.py b/scripts/deforum_helpers/rendering/data/subtitle/srt.py index 2f5bc0d90..cb90c0e1e 100644 --- a/scripts/deforum_helpers/rendering/data/subtitle/srt.py +++ b/scripts/deforum_helpers/rendering/data/subtitle/srt.py @@ -1,27 +1,25 @@ -import dataclasses import os +from dataclasses import dataclass from decimal import Decimal from ....subtitle_handler import init_srt_file -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class Srt: filename: str frame_duration: Decimal - def __new__(cls, *args, **kwargs): # locks the constructor to enforce proper initialization - raise TypeError("Use Srt.create_if_active() to create new instances.") + @staticmethod + def is_subtitle_generation_active(opts_data): + return opts_data.get("deforum_save_gen_info_as_srt", False) - @classmethod - def create_if_active(cls, opts_data, outdir: str, timestring: str, fps: float) -> 'Srt | None': - if not opts_data.get("deforum_save_gen_info_as_srt", False): + @staticmethod + def create_if_active(opts_data, out_dir: str, timestring: str, fps: float) -> 'Srt | None': + if not Srt.is_subtitle_generation_active(opts_data): return None else: # create .srt file and set timeframe mechanism using FPS - init_filename = os.path.join(outdir, f"{timestring}.srt") - init_frame_duration = init_srt_file(init_filename, fps) - - instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(init_filename, init_frame_duration) - return instance + filename = os.path.join(out_dir, f"{timestring}.srt") + frame_duration = init_srt_file(filename, fps) + return Srt(filename, frame_duration) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py new file mode 100644 index 000000000..ceb2eb644 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -0,0 +1,42 @@ +from dataclasses import dataclass +from typing import Any + +from .subtitle import Srt + + +# TODO freeze.. +@dataclass(frozen=False) +class Turbo: + steps: int + prev_image: Any + prev_frame_idx: int + next_image: Any + next_frame_idx: int + + @staticmethod + def create(init): + steps = 1 if init.has_video_input() else init.cadence() + return Turbo(steps, None, 0, None, 0) + + def set_up_step_vars(self, prev_img, prev_frame, next_img, next_frame): + if self.steps > 1: + self.prev_image, self.prev_frame_idx = prev_img, prev_frame + self.next_image, self.next_frame_idx = next_img, next_frame + + def _has_prev_image(self): + return self.prev_image is not None + + def is_advance_prev(self, i) -> bool: + return self._has_prev_image() and i > self.prev_frame_idx + + def is_advance_next(self, i) -> bool: + return i > self.next_frame_idx + + def is_first_step(self) -> bool: + return self.steps == 1 + + def is_first_step_with_subtitles(self, init) -> bool: + return self.is_first_step() and Srt.is_subtitle_generation_active(init.args.opts.data) + + def is_emit_in_between_frames(self) -> bool: + return self.steps > 1 diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 85ece2b4b..195a25b9e 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -1,9 +1,9 @@ -import dataclasses import numexpr import numpy as np import os import pandas as pd +from dataclasses import dataclass from typing import Any from .data.anim import AnimationKeys, AnimationMode from .data.subtitle import Srt @@ -15,7 +15,7 @@ from ..settings import save_settings_from_animation_run -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInitArgs: # TODO eventually this should only keep the information required to run render_animation once # for now it's just a direct reference or a copy of the actual args provided to the render_animation call. @@ -33,7 +33,7 @@ def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_ return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -@dataclasses.dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" seed: int @@ -64,7 +64,7 @@ def height(self) -> int: def dimensions(self) -> tuple[int, int]: # TODO should ideally only be called once each render - return (self.width(), self.height()) + return self.width(), self.height() # TODO group hybrid stuff elsewhere def is_hybrid_composite(self) -> bool: @@ -221,7 +221,8 @@ def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_ar animation_mode, root, anim_args, args_argument) instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. instance.__init__(args_argument.seed, args, parseq_adapter, srt, - animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) + animation_keys, animation_mode, prompt_series, + depth_model, output_directory, is_use_mask) # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. # It may be preferable to work on temporary copies within tight scope. # TODO avoid or isolate more side effects diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 34895e982..4d9d01028 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -1,7 +1,7 @@ from modules.shared import cmd_opts # keep readonly -class MemoryUtils(): +class MemoryUtils: # Don't put any variables here, it's meant for static methods only. @staticmethod From 7435cf1cb966cadf7f6436658f09fe471538de71 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 4 Jun 2024 03:07:45 +0200 Subject: [PATCH 035/132] Step initialization. --- scripts/deforum_helpers/render.py | 84 ++++++------- .../rendering/initialization.py | 113 ++++++++++++------ 2 files changed, 112 insertions(+), 85 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 19878a318..e5538779d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -38,7 +38,7 @@ from .prompt import prepare_prompt from .rendering.data import Turbo from .rendering.data.schedule import Schedule -from .rendering.initialization import RenderInit +from .rendering.initialization import RenderInit, StepInit from .rendering.util import put_if_present, call_anim_frame_warp from .rendering.util.call_utils import (call_get_flow_from_images, call_generate, call_render_preview, call_hybrid_composite, call_format_animation_params, @@ -141,27 +141,8 @@ def run_render_animation(init): print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{init.args.anim_args.max_frames} ") - # TODO move this to the new key collection - noise = init.animation_keys.deform_keys.noise_schedule_series[frame_idx] - strength = init.animation_keys.deform_keys.strength_schedule_series[frame_idx] - scale = init.animation_keys.deform_keys.cfg_scale_schedule_series[frame_idx] - contrast = init.animation_keys.deform_keys.contrast_schedule_series[frame_idx] - kernel = int(init.animation_keys.deform_keys.kernel_schedule_series[frame_idx]) - sigma = init.animation_keys.deform_keys.sigma_schedule_series[frame_idx] - amount = init.animation_keys.deform_keys.amount_schedule_series[frame_idx] - threshold = init.animation_keys.deform_keys.threshold_schedule_series[frame_idx] - cadence_flow_factor = init.animation_keys.deform_keys.cadence_flow_factor_schedule_series[frame_idx] - redo_flow_factor = init.animation_keys.deform_keys.redo_flow_factor_schedule_series[frame_idx] - hybrid_comp_schedules = { - "alpha": init.animation_keys.deform_keys.hybrid_comp_alpha_schedule_series[frame_idx], - "mask_blend_alpha": init.animation_keys.deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], - "mask_contrast": init.animation_keys.deform_keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], - "mask_auto_contrast_cutoff_low": int( - init.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), - "mask_auto_contrast_cutoff_high": int( - init.animation_keys.deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), - "flow_factor": init.animation_keys.deform_keys.hybrid_flow_factor_schedule_series[frame_idx] - } + step_init = StepInit.create(init.animation_keys.deform_keys, frame_idx) + # TODO eventually move schedule into new Step class schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, init.args.args) @@ -248,19 +229,19 @@ def run_render_animation(init): flow = call_get_flow_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) if turbo.is_advance_prev(tween_frame_idx): turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, - hybrid_comp_schedules['flow_factor']) + step_init.flow_factor()) if turbo.is_advance_next(tween_frame_idx): turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, - hybrid_comp_schedules['flow_factor']) + step_init.flow_factor()) init.animation_mode.prev_flow = flow else: flow = call_get_flow_for_hybrid_motion(init, tween_frame_idx - 1) if turbo.is_advance_prev(tween_frame_idx): turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, - hybrid_comp_schedules['flow_factor']) + step_init.flow_factor()) if turbo.is_advance_next(tween_frame_idx): turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, - hybrid_comp_schedules['flow_factor']) + step_init.flow_factor()) init.animation_mode.prev_flow = flow # TODO cadence related transforms to be decoupled and handled in a 2nd pass @@ -269,14 +250,12 @@ def run_render_animation(init): cadence_flow = abs_flow_to_rel_flow(cadence_flow, init.width(), init.height()) cadence_flow, _ = call_anim_frame_warp(init, tween_frame_idx, cadence_flow, depth) cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, init.width(), init.height()) * tween - if is_advance_prev: - turbo.prev_image = image_transform_optical_flow(turbo.prev_image, - cadence_flow_inc, - cadence_flow_factor) - if is_advance_next: - turbo.next_image = image_transform_optical_flow(turbo.next_image, - cadence_flow_inc, - cadence_flow_factor) + if turbo.is_advance_prev(): + turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, + step_init.cadence_flow_factor) + if turbo.is_advance_next(): + turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, + step_init.cadence_flow_factor) turbo.prev_frame_idx = turbo.next_frame_idx = tween_frame_idx @@ -327,7 +306,7 @@ def run_render_animation(init): # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, hybrid_comp_schedules) + _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, step_init.hybrid_comp_schedules) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: @@ -341,13 +320,13 @@ def run_render_animation(init): flow = call_get_flow_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) else: flow = call_get_flow_for_hybrid_motion(init, frame_idx - 1) - prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) + prev_img = image_transform_optical_flow(prev_img, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, hybrid_comp_schedules) + _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, step_init.hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): if color_match_sample is None: @@ -361,10 +340,15 @@ def run_render_animation(init): prev_img = cv2.cvtColor(prev_img, cv2.COLOR_GRAY2BGR) # apply scaling - contrast_image = (prev_img * contrast).round().astype(np.uint8) + contrast_image = (prev_img * step_init.contrast).round().astype(np.uint8) # anti-blur - if amount > 0: - contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, + if step_init.amount > 0: + step_init.kernel_size() + contrast_image = unsharp_mask(contrast_image, + (step_init.kernel, step_init.kernel), + step_init.sigma, + step_init.amount, + step_init.threshold, mask_image if init.args.args.use_mask else None) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: @@ -375,7 +359,8 @@ def run_render_animation(init): noise_mask_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) - noised_image = add_noise(contrast_image, noise, init.args.args.seed, init.args.anim_args.noise_type, + noised_image = add_noise(contrast_image, step_init.noise, init.args.args.seed, + init.args.anim_args.noise_type, (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, init.args.anim_args.perlin_octaves, init.args.anim_args.perlin_persistence), @@ -384,9 +369,9 @@ def run_render_animation(init): # use transformed previous frame as init for current init.args.args.use_init = True init.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) - init.args.args.strength = max(0.0, min(1.0, strength)) + init.args.args.strength = max(0.0, min(1.0, step_init.strength)) - init.args.args.scale = scale + init.args.args.scale = step_init.scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints init.args.args.pix2pix_img_cfg_scale = float( @@ -425,7 +410,7 @@ def run_render_animation(init): print(f"Using video init frame {init_frame}") init.args.args.init_image = init_frame init.args.args.init_image_box = None # init_image_box not used in this case - init.args.args.strength = max(0.0, min(1.0, strength)) + init.args.args.strength = max(0.0, min(1.0, step_init.strength)) if init.args.anim_args.use_mask_video: mask_init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) @@ -453,7 +438,7 @@ def run_render_animation(init): if not init.args.args.motion_preview_mode else 'None' # optical flow redo before generation - if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: + if optical_flow_redo_generation != 'None' and prev_img is not None and step_init.strength > 0: stored_seed = init.args.args.seed init.args.args.seed = random.randint(0, 2 ** 32 - 1) print( @@ -463,7 +448,8 @@ def run_render_animation(init): disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) disposable_flow = call_get_flow_from_images(init, prev_img, disposable_image, optical_flow_redo_generation) disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) - disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) + disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, + step_init.redo_flow_factor) init.args.args.seed = stored_seed init.args.root.init_sample = Image.fromarray(disposable_image) del (disposable_image, disposable_flow, stored_seed) @@ -471,7 +457,7 @@ def run_render_animation(init): # diffusion redo if (int(init.args.anim_args.diffusion_redo) > 0 - and prev_img is not None and strength > 0 + and prev_img is not None and step_init.strength > 0 and not init.args.args.motion_preview_mode): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): @@ -498,7 +484,7 @@ def run_render_animation(init): if frame_idx > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = call_hybrid_composite(init, frame_idx, temp_image, hybrid_comp_schedules) + _, temp_image_2 = call_hybrid_composite(init, frame_idx, temp_image, step_init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, @@ -519,7 +505,7 @@ def run_render_animation(init): # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) - or (init.args.anim_args.legacy_colormatch and strength == 0)) + or (init.args.anim_args.legacy_colormatch and step_init.strength == 0)) and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 195a25b9e..aaf75d6fb 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -1,10 +1,11 @@ +import os +from dataclasses import dataclass +from typing import Any + import numexpr import numpy as np -import os import pandas as pd -from dataclasses import dataclass -from typing import Any from .data.anim import AnimationKeys, AnimationMode from .data.subtitle import Srt from .util import MemoryUtils @@ -28,11 +29,55 @@ class RenderInitArgs: opts: Any = None root: Any = None - @classmethod - def create(cls, args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): + @staticmethod + def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) +# TODO move elsewhere +@dataclass(init=True, frozen=True, repr=False, eq=False) +class StepInit: + noise: Any = None + strength: Any = None + scale: Any = None + contrast: Any = None + kernel: int = 0 + sigma: Any = None + amount: Any = None + threshold: Any = None + cadence_flow_factor: Any = None + redo_flow_factor: Any = None + hybrid_comp_schedules: Any = None + + def kernel_size(self) -> tuple[int, int]: + return self.kernel, self.kernel + + def flow_factor(self): + return self.hybrid_comp_schedules['flow_factor'] + + @staticmethod + def create(deform_keys, i): + return StepInit( + deform_keys.noise_schedule_series[i], + deform_keys.strength_schedule_series[i], + deform_keys.cfg_scale_schedule_series[i], + deform_keys.contrast_schedule_series[i], + int(deform_keys.kernel_schedule_series[i]), + deform_keys.sigma_schedule_series[i], + deform_keys.amount_schedule_series[i], + deform_keys.threshold_schedule_series[i], + deform_keys.cadence_flow_factor_schedule_series[i], + deform_keys.redo_flow_factor_schedule_series[i], { + "alpha": deform_keys.hybrid_comp_alpha_schedule_series[i], + "mask_blend_alpha": deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[i], + "mask_contrast": deform_keys.hybrid_comp_mask_contrast_schedule_series[i], + "mask_auto_contrast_cutoff_low": + int(deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), + "mask_auto_contrast_cutoff_high": + int(deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), + "flow_factor": deform_keys.hybrid_flow_factor_schedule_series[i]}) + + @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" @@ -47,9 +92,6 @@ class RenderInit: output_directory: str is_use_mask: bool - def __new__(cls, *args, **kwargs): - raise TypeError("Use RenderInit.create() to create new instances.") - def is_3d(self): return self.args.anim_args.animation_mode == '3D' @@ -122,13 +164,13 @@ def _has_init_image_or_box(self) -> bool: def is_using_init_image_or_box(self) -> bool: return self.args.args.use_init and self._has_init_image_or_box() - @classmethod - def create_output_directory_for_the_batch(cls, directory): + @staticmethod + def create_output_directory_for_the_batch(directory): os.makedirs(directory, exist_ok=True) print(f"Saving animation frames to:\n{directory}") - @classmethod - def create_parseq_adapter(cls, args): + @staticmethod + def create_parseq_adapter(args): adapter = ParseqAdapter(args.parseq_args, args.anim_args, args.video_args, args.controlnet_args, args.loop_args) # Always enable pseudo-3d with parseq. No need for an extra toggle: # Whether it's used or not in practice is defined by the schedules @@ -136,8 +178,8 @@ def create_parseq_adapter(cls, args): args.anim_args.flip_2d_perspective = True return adapter - @classmethod - def init_looper_if_active(cls, args, loop_args): + @staticmethod + def init_looper_if_active(args, loop_args): if loop_args.use_looper: print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") if args.strength == 0: @@ -147,17 +189,17 @@ def init_looper_if_active(cls, args, loop_args): if not isJson(loop_args.init_images): raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") - @classmethod - def select_prompts(cls, parseq_adapter, anim_args, animation_keys, root): + @staticmethod + def select_prompts(parseq_adapter, anim_args, animation_keys, root): return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ else RenderInit.expand_prompts_out_to_per_frame(anim_args, root) - @classmethod - def is_composite_with_depth_mask(cls, anim_args): + @staticmethod + def is_composite_with_depth_mask(anim_args): return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' - @classmethod - def create_depth_model_and_enable_depth_map_saving_if_active(cls, anim_mode, root, anim_args, args): + @staticmethod + def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args): # depth-based hybrid composite mask requires saved depth maps # TODO avoid or isolate side effect: anim_args.save_depth_maps = (anim_mode.is_predicting_depths @@ -171,8 +213,8 @@ def create_depth_model_and_enable_depth_map_saving_if_active(cls, anim_mode, roo midas_weight=anim_args.midas_weight) \ if anim_mode.is_predicting_depths else None - @classmethod - def expand_prompts_out_to_per_frame(cls, anim_args, root): + @staticmethod + def expand_prompts_out_to_per_frame(anim_args, root): prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) for i, prompt in root.animation_prompts.items(): if str(i).isdigit(): @@ -181,21 +223,21 @@ def expand_prompts_out_to_per_frame(cls, anim_args, root): prompt_series[int(numexpr.evaluate(i))] = prompt return prompt_series.ffill().bfill() - @classmethod - def handle_controlnet_video_input_frames_generation(cls, controlnet_args, args, anim_args): + @staticmethod + def handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args): if is_controlnet_enabled(controlnet_args): unpack_controlnet_vids(args, anim_args, controlnet_args) - @classmethod - def save_settings_txt(cls, args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): + @staticmethod + def save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root): save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - @classmethod - def maybe_resume_from_timestring(cls, anim_args, root): + @staticmethod + def maybe_resume_from_timestring(anim_args, root): root.timestring = anim_args.resume_timestring if anim_args.resume_from_timestring else root.timestring - @classmethod - def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): + @staticmethod + def do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): # TODO all of those calls may cause a change in on of the passed args. # Ideally it may be refactored so each one returns a new instance of the potentially changed args that are then # attached as a property to this class to be used for one single render only. @@ -205,8 +247,8 @@ def do_void_inits(cls, args, loop_args, controlnet_args, anim_args, parseq_args, RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) RenderInit.maybe_resume_from_timestring(anim_args, root) - @classmethod - def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_args, + @staticmethod + def create(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': # TODO deepcopy args? args = RenderInitArgs(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) @@ -219,10 +261,9 @@ def create(cls, args_argument, parseq_args, anim_args, video_args, controlnet_ar prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args_argument) - instance = object.__new__(cls) # creating the instance without raising the type error defined in __new__. - instance.__init__(args_argument.seed, args, parseq_adapter, srt, - animation_keys, animation_mode, prompt_series, - depth_model, output_directory, is_use_mask) + + instance = RenderInit(args_argument.seed, args, parseq_adapter, srt, animation_keys, + animation_mode, prompt_series, depth_model, output_directory, is_use_mask) # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. # It may be preferable to work on temporary copies within tight scope. # TODO avoid or isolate more side effects From 680031c628fc9c4e349f53f88f3c03e2f6dab2ec Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 5 Jun 2024 01:52:23 +0200 Subject: [PATCH 036/132] Root data proxy prepared. --- scripts/deforum_helpers/render.py | 108 ++++++++------ .../rendering/data/proxy/__init__.py | 1 + .../rendering/data/proxy/root_data_proxy.py | 133 ++++++++++++++++++ .../rendering/initialization.py | 13 +- .../rendering/util/__init__.py | 2 +- .../rendering/util/call_utils.py | 51 +++++-- .../deforum_helpers/rendering/util/utils.py | 10 +- 7 files changed, 250 insertions(+), 68 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/proxy/__init__.py create mode 100644 scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index e5538779d..e286809ec 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -39,15 +39,32 @@ from .rendering.data import Turbo from .rendering.data.schedule import Schedule from .rendering.initialization import RenderInit, StepInit -from .rendering.util import put_if_present, call_anim_frame_warp -from .rendering.util.call_utils import (call_get_flow_from_images, call_generate, call_render_preview, - call_hybrid_composite, call_format_animation_params, - call_write_frame_subtitle, call_get_mask_from_file_with_frame, - call_get_mask_from_file, call_get_next_frame, - call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, - call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev) +from .rendering.util import put_if_present, put_all +from .rendering.util.call_utils import ( + # Animation Functions + call_anim_frame_warp, + call_format_animation_params, + call_get_next_frame, + call_write_frame_subtitle, + + # Generation and Rendering + call_generate, + call_render_preview, + + # Hybrid Motion Functions + call_get_flow_for_hybrid_motion, + call_get_flow_for_hybrid_motion_prev, + call_get_matrix_for_hybrid_motion, + call_get_matrix_for_hybrid_motion_prev, + call_hybrid_composite, + + # Mask Functions + call_get_mask_from_file, + call_get_mask_from_file_with_frame, + + # Flow Functions + call_get_flow_from_images) from .rendering.util.memory_utils import MemoryUtils -from .rendering.util.utils import put_all from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed @@ -56,15 +73,19 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): render_init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + # print("Debug: " + str(render_init.root.proxy)) # TODO method is temporarily torn apart to remove args from direct access in larger execution scope. run_render_animation(render_init) +def run_render_animation_controlled(init): + raise NotImplementedError("not implemented.") + + def run_render_animation(init): # TODO refactor to try and avoid all usage and reassignments to "init.args.args". # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. # TODO isolate "depth" with other moving parts - # TODO cleanup and find potential side effects in 31 refs to init.args.root # TODO isolate the relevant data in +250 refs to init.args, # move that stuff to init and eventually try to drop init.args # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. @@ -143,7 +164,6 @@ def run_render_animation(init): step_init = StepInit.create(init.animation_keys.deform_keys, frame_idx) - # TODO eventually move schedule into new Step class schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, init.args.args) @@ -158,7 +178,7 @@ def run_render_animation(init): sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(init.args.root.device) + init.animation_mode.depth_model.to(init.root.device) if turbo.is_first_step_with_subtitles(init): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) @@ -197,7 +217,7 @@ def run_render_animation(init): if init.depth_model is not None: assert (turbo.next_image is not None) depth = init.depth_model.predict(turbo.next_image, init.args.anim_args.midas_weight, - init.args.root.half_precision) + init.root.half_precision) # TODO collect images if turbo.is_advance_prev(tween_frame_idx): @@ -280,13 +300,13 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{init.args.root.timestring}_{tween_frame_idx:09}.png" + filename = f"{init.root.timestring}_{tween_frame_idx:09}.png" save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, - f"{init.args.root.timestring}_depth_{tween_frame_idx:09}.png") + f"{init.root.timestring}_depth_{tween_frame_idx:09}.png") init.depth_model.save(dm_save_path, depth) # get color match for video outside of prev_img conditional @@ -352,23 +372,23 @@ def run_render_animation(init): mask_image if init.args.args.use_mask else None) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: - init.args.root.noise_mask = compose_mask_with_check(init.args.root, - init.args.args, - noise_mask_seq, - # FIXME might be ref'd b4 assignment - noise_mask_vals, - Image.fromarray(cv2.cvtColor(contrast_image, - cv2.COLOR_BGR2RGB))) + init.root.noise_mask = compose_mask_with_check(init.root, + init.args.args, + noise_mask_seq, + # FIXME might be ref'd b4 assignment + noise_mask_vals, + Image.fromarray(cv2.cvtColor(contrast_image, + cv2.COLOR_BGR2RGB))) noised_image = add_noise(contrast_image, step_init.noise, init.args.args.seed, init.args.anim_args.noise_type, (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, init.args.anim_args.perlin_octaves, init.args.anim_args.perlin_persistence), - init.args.root.noise_mask, init.args.args.invert_mask) + init.root.noise_mask, init.args.args.invert_mask) # use transformed previous frame as init for current init.args.args.use_init = True - init.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) + init.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) init.args.args.strength = max(0.0, min(1.0, step_init.strength)) init.args.args.scale = step_init.scale @@ -390,15 +410,14 @@ def run_render_animation(init): # SubSeed scheduling if init.args.anim_args.enable_subseed_scheduling: - init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.args.root.subseed_strength = float( + init.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + init.root.subseed_strength = float( init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - init.args.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.args.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[ - frame_idx] + init.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) + init.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] # set value back into the prompt - prepare and report prompt and seed init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, @@ -415,14 +434,13 @@ def run_render_animation(init): mask_init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) init.args.args.mask_file = temp_mask - init.args.root.noise_mask = temp_mask + init.root.noise_mask = temp_mask mask_vals['video_mask'] = temp_mask if init.args.args.use_mask: - init.args.args.mask_image = compose_mask_with_check(init.args.root, init.args.args, - schedule.mask_seq, mask_vals, - init.args.root.init_sample) \ - if init.args.root.init_sample is not None else None # we need it only after the first frame anyway + init.args.args.mask_image = compose_mask_with_check(init.root, init.args.args, schedule.mask_seq, + mask_vals, init.root.init_sample) \ + if init.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) setup_opts(init, schedule) @@ -451,7 +469,7 @@ def run_render_animation(init): disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, step_init.redo_flow_factor) init.args.args.seed = stored_seed - init.args.root.init_sample = Image.fromarray(disposable_image) + init.root.init_sample = Image.fromarray(disposable_image) del (disposable_image, disposable_flow, stored_seed) gc.collect() @@ -470,7 +488,7 @@ def run_render_animation(init): disposable_image = maintain_colors(prev_img, color_match_sample, init.args.anim_args.color_coherence) init.args.args.seed = stored_seed - init.args.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) + init.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) del (disposable_image, stored_seed) # FIXME disposable_image might be referenced before assignment. gc.collect() # TODO try to eventually kick the gc only once at the end of every generation or iteration. @@ -518,8 +536,8 @@ def run_render_animation(init): turbo.next_image, turbo.next_frame_idx = opencv_image, frame_idx frame_idx += turbo.steps else: - filename = f"{init.args.root.timestring}_{frame_idx:09}.png" - save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) + filename = f"{init.root.timestring}_{frame_idx:09}.png" + save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) if init.args.anim_args.save_depth_maps: # TODO move all depth related stuff to new class. (also see RenderInit) @@ -527,11 +545,11 @@ def run_render_animation(init): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - init.depth_model.to(init.args.root.device) + init.depth_model.to(init.root.device) depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, - init.args.root.half_precision) + init.root.half_precision) init.depth_model.save( - os.path.join(init.output_directory, f"{init.args.root.timestring}_depth_{frame_idx:09}.png"), depth) + os.path.join(init.output_directory, f"{init.root.timestring}_depth_{frame_idx:09}.png"), depth) if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') devices.torch_gc() @@ -540,7 +558,7 @@ def run_render_animation(init): frame_idx += 1 last_preview_frame = progress_and_make_preview(init, image, frame_idx, state, last_preview_frame) - update_tracker(init.args.root, frame_idx, init.args.anim_args) + update_tracker(init.root, frame_idx, init.args.anim_args) init.animation_mode.cleanup() @@ -552,7 +570,7 @@ def assign_masks(init, i, is_mask_image, dicts): if init.args.anim_args.use_mask_video: mask = call_get_mask_from_file(init, i, True) init.args.args.mask_file = mask - init.args.root.noise_mask = mask + init.root.noise_mask = mask put_all(dicts, key, mask) elif is_mask_image is None and init.is_use_mask: put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask @@ -571,7 +589,11 @@ def setup_opts(init, schedule): def progress_and_make_preview(init, image, frame_idx, state, last_preview_frame): state.assign_current_image(image) - init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO refactor assignment + + # may reassign init.args.args and/or root.seed_internal # FIXME? + init.args.args.seed = next_seed(init.args.args, init.root) # TODO refactor assignment + # init.seed = init.args.args.seed # TODO group all seeds and sub-seeds + return call_render_preview(init, frame_idx, last_preview_frame) diff --git a/scripts/deforum_helpers/rendering/data/proxy/__init__.py b/scripts/deforum_helpers/rendering/data/proxy/__init__.py new file mode 100644 index 000000000..f558745ee --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/proxy/__init__.py @@ -0,0 +1 @@ +from .root_data_proxy import RootDataProxyWrapper diff --git a/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py b/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py new file mode 100644 index 000000000..74f105ad7 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py @@ -0,0 +1,133 @@ +from dataclasses import dataclass, replace +from typing import Any + + +@dataclass(frozen=True, kw_only=True) +class RootDataProxy: + """ + An immutable data container for holding data required by "render_animation". + + This class is designed to be used with `RootDataProxyWrapper` which provides + a mutable interface through "fake" setters. These setters do not modify the + original `RootDataProxy` instance, but instead create new instances with the + updated data. This approach ensures that the original data remains immutable + while still allowing for changes to be made. + """ + + # TODO improve typehints + # Read-only attributes, accessed through the wrapper + device: Any + half_precision: Any + timestring: Any + mask_preset_names: Any + frames_cache: Any + job_id: Any + animation_prompts: Any + + # Attributes that can be replaced by the wrapper + noise_mask: Any + init_sample: Any + subseed: Any + subseed_strength: Any + clipseg_model: Any + seed_internal: Any + + @staticmethod + def create(root: Any) -> 'RootDataProxy': + return RootDataProxy( + device=root.device, + half_precision=root.half_precision, + timestring=root.timestring, + mask_preset_names=root.mask_preset_names, + frames_cache=root.frames_cache, + job_id=root.job_id, + animation_prompts=root.animation_prompts, + noise_mask=root.noise_mask, + init_sample=root.init_sample, + subseed=root.subseed, + subseed_strength=root.subseed_strength, + clipseg_model=root.clipseg_model, + seed_internal=root.seed_internal) + + +class RootDataProxyWrapper: + """ + Provides a mutable interface to the `RootDataProxy` by replacing the + entire proxy instance when a setter is called. + """ + + def __init__(self, root_data_proxy: RootDataProxy): + self._proxy = root_data_proxy + + @staticmethod + def create(root: Any) -> 'RootDataProxyWrapper': + return RootDataProxyWrapper(RootDataProxy.create(root)) + + def device(self): + return self._proxy.device + + def half_precision(self): + return self._proxy.half_precision + + def timestring(self): + return self._proxy.timestring + + def mask_preset_names(self): + return self._proxy.mask_preset_names + + def frames_cache(self): + return self._proxy.frames_cache + + def job_id(self): + return self._proxy.job_id + + def animation_prompts(self): + return self._proxy.animation_prompts + + @property + def noise_mask(self): + return self._proxy.noise_mask + + @noise_mask.setter + def noise_mask(self, value): + self._proxy = replace(self._proxy, noise_mask=value) + + @property + def init_sample(self): + return self._proxy.init_sample + + @init_sample.setter + def init_sample(self, value): + self._proxy = replace(self._proxy, init_sample=value) + + @property + def subseed(self): + return self._proxy.subseed + + @subseed.setter + def subseed(self, value): + self._proxy = replace(self._proxy, subseed=value) + + @property + def subseed_strength(self): + return self._proxy.subseed_strength + + @subseed_strength.setter + def subseed_strength(self, value): + self._proxy = replace(self._proxy, subseed_strength=value) + + @property + def clipseg_model(self): + return self._proxy.clipseg_model + + @clipseg_model.setter + def clipseg_model(self, value): + self._proxy = replace(self._proxy, clipseg_model=value) + + @property + def seed_internal(self): + return self._proxy.seed_internal + + @seed_internal.setter + def seed_internal(self, value): + self._proxy = replace(self._proxy, seed_internal=value) diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index aaf75d6fb..14719e627 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -7,6 +7,7 @@ import pandas as pd from .data.anim import AnimationKeys, AnimationMode +from .data.proxy.root_data_proxy import RootDataProxyWrapper from .data.subtitle import Srt from .util import MemoryUtils from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled @@ -81,6 +82,7 @@ def create(deform_keys, i): @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" + root: RootDataProxyWrapper seed: int args: RenderInitArgs parseq_adapter: Any @@ -250,7 +252,6 @@ def do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, vide @staticmethod def create(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': - # TODO deepcopy args? args = RenderInitArgs(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) output_directory = args_argument.outdir is_use_mask = args_argument.use_mask @@ -262,10 +263,14 @@ def create(args_argument, parseq_args, anim_args, video_args, controlnet_args, depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args_argument) - instance = RenderInit(args_argument.seed, args, parseq_adapter, srt, animation_keys, + # TODO proxy other args like this. + # FIXME, somethings are still missing or accessors are not behaving as they should in the wrapper. + # root_proxy = RootDataProxyWrapper.create(root) + root_proxy = root + + instance = RenderInit(root_proxy, args_argument.seed, args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - # Ideally, a call to render_animation in render.py shouldn't cause changes in any of the args passed there. - # It may be preferable to work on temporary copies within tight scope. + # TODO avoid or isolate more side effects RenderInit.do_void_inits(args_argument, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 5b6862201..6e8cf76df 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -5,4 +5,4 @@ call_get_matrix_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion) from .memory_utils import MemoryUtils -from .utils import put_if_present +from .utils import put_all, put_if_present diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py index fa718c9fb..41dac86e3 100644 --- a/scripts/deforum_helpers/rendering/util/call_utils.py +++ b/scripts/deforum_helpers/rendering/util/call_utils.py @@ -1,25 +1,46 @@ from ...animation import anim_frame_warp from ...generate import generate -from ...hybrid_video import (hybrid_composite, get_flow_from_images, - get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, - get_flow_for_hybrid_motion_prev, get_flow_for_hybrid_motion) +from ...hybrid_video import ( + # Functions related to flow calculation + get_flow_from_images, + get_flow_for_hybrid_motion, + get_flow_for_hybrid_motion_prev, + + # Functions related to matrix calculation + get_matrix_for_hybrid_motion, + get_matrix_for_hybrid_motion_prev, + + # Other hybrid functions + hybrid_composite) from ...load_images import get_mask_from_file from ...subtitle_handler import format_animation_params, write_frame_subtitle from ...video_audio_utilities import get_next_frame, render_preview +""" +This module provides utility functions for simplifying calls to other modules within the `render.py` module. + +**Purpose:** +- **Reduce Argument Complexity:** Provides a way to call functions in other modules without directly handling + a large number of complex arguments. This simplifies code within `render.py` by encapsulating argument management. +- **Minimize Namespace Pollution:** Provides an alternative to overloading methods in the original modules, + which would introduce the `RenderInit` class into namespaces where it's not inherently needed. + +**Structure:** +- **Simple Call Forwarding:** Functions in this module primarily act as wrappers. They perform minimal logic, + often just formatting or passing arguments, and directly call the corresponding method. +- **Naming Convention:** + - Function names begin with "call_", followed by the name of the actual method to call. + - The `init` object is always passed as the first argument. + - Frame indices (e.g., `frame_idx`, `twin_frame_idx`) are passed as the second argument "i", when relevant. + +**Example:** +```python +# Example function in this module +def call_some_function(init, i, ...): + return some_module.some_function(init.arg77, init.arg.arg.whatever, i, ...) +``` +""" -# Purpose: -# This module mostly exists for refactoring and reducing the complexity of render.py without touching any other modules. -# Currently useful to reduce complexity in calls with many arguments, that have been or will be regrouped into "init". -# Alternatively all methods may be overloaded in the original modules, but doing so would propagate the Init class to -# namespaces where it doesn't really belong, which is rather undesirable. -# -# Form: -# The following functions shouldn't contain any complex logic besides perhaps some formatting -# and aim to directly return with the call to the actual method. -# - Naming starts with "call_". -# - "init" to be passed as 1st argument. -# - pass frame_idx or twin_frame_idx or other indices as 2nd argument "i" where applicable. # Animation: def call_anim_frame_warp(init, i, image, depth): diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index bd86a9eed..c5c294d63 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,10 +1,10 @@ -def put_if_present(dictionary, key, value): - if value is not None: - dictionary[key] = value - - def put_all(dictionaries, key, callable_or_value): for dictionary in dictionaries: dictionary[key] = callable_or_value() if callable(callable_or_value) else callable_or_value + + +def put_if_present(dictionary, key, value): + if value is not None: + dictionary[key] = value From 4b327a4fce6e91769b01836346ddaebba3159e62 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 7 Jun 2024 22:55:58 +0200 Subject: [PATCH 037/132] Moved proxy and wrapper stuff to a different branch. --- scripts/deforum_helpers/render.py | 2 - .../rendering/data/proxy/__init__.py | 1 - .../rendering/data/proxy/root_data_proxy.py | 133 ------------------ .../rendering/initialization.py | 21 +-- 4 files changed, 3 insertions(+), 154 deletions(-) delete mode 100644 scripts/deforum_helpers/rendering/data/proxy/__init__.py delete mode 100644 scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index e286809ec..5fe3323ad 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -73,7 +73,6 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): render_init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - # print("Debug: " + str(render_init.root.proxy)) # TODO method is temporarily torn apart to remove args from direct access in larger execution scope. run_render_animation(render_init) @@ -83,7 +82,6 @@ def run_render_animation_controlled(init): def run_render_animation(init): - # TODO refactor to try and avoid all usage and reassignments to "init.args.args". # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. # TODO isolate "depth" with other moving parts # TODO isolate the relevant data in +250 refs to init.args, diff --git a/scripts/deforum_helpers/rendering/data/proxy/__init__.py b/scripts/deforum_helpers/rendering/data/proxy/__init__.py deleted file mode 100644 index f558745ee..000000000 --- a/scripts/deforum_helpers/rendering/data/proxy/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from .root_data_proxy import RootDataProxyWrapper diff --git a/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py b/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py deleted file mode 100644 index 74f105ad7..000000000 --- a/scripts/deforum_helpers/rendering/data/proxy/root_data_proxy.py +++ /dev/null @@ -1,133 +0,0 @@ -from dataclasses import dataclass, replace -from typing import Any - - -@dataclass(frozen=True, kw_only=True) -class RootDataProxy: - """ - An immutable data container for holding data required by "render_animation". - - This class is designed to be used with `RootDataProxyWrapper` which provides - a mutable interface through "fake" setters. These setters do not modify the - original `RootDataProxy` instance, but instead create new instances with the - updated data. This approach ensures that the original data remains immutable - while still allowing for changes to be made. - """ - - # TODO improve typehints - # Read-only attributes, accessed through the wrapper - device: Any - half_precision: Any - timestring: Any - mask_preset_names: Any - frames_cache: Any - job_id: Any - animation_prompts: Any - - # Attributes that can be replaced by the wrapper - noise_mask: Any - init_sample: Any - subseed: Any - subseed_strength: Any - clipseg_model: Any - seed_internal: Any - - @staticmethod - def create(root: Any) -> 'RootDataProxy': - return RootDataProxy( - device=root.device, - half_precision=root.half_precision, - timestring=root.timestring, - mask_preset_names=root.mask_preset_names, - frames_cache=root.frames_cache, - job_id=root.job_id, - animation_prompts=root.animation_prompts, - noise_mask=root.noise_mask, - init_sample=root.init_sample, - subseed=root.subseed, - subseed_strength=root.subseed_strength, - clipseg_model=root.clipseg_model, - seed_internal=root.seed_internal) - - -class RootDataProxyWrapper: - """ - Provides a mutable interface to the `RootDataProxy` by replacing the - entire proxy instance when a setter is called. - """ - - def __init__(self, root_data_proxy: RootDataProxy): - self._proxy = root_data_proxy - - @staticmethod - def create(root: Any) -> 'RootDataProxyWrapper': - return RootDataProxyWrapper(RootDataProxy.create(root)) - - def device(self): - return self._proxy.device - - def half_precision(self): - return self._proxy.half_precision - - def timestring(self): - return self._proxy.timestring - - def mask_preset_names(self): - return self._proxy.mask_preset_names - - def frames_cache(self): - return self._proxy.frames_cache - - def job_id(self): - return self._proxy.job_id - - def animation_prompts(self): - return self._proxy.animation_prompts - - @property - def noise_mask(self): - return self._proxy.noise_mask - - @noise_mask.setter - def noise_mask(self, value): - self._proxy = replace(self._proxy, noise_mask=value) - - @property - def init_sample(self): - return self._proxy.init_sample - - @init_sample.setter - def init_sample(self, value): - self._proxy = replace(self._proxy, init_sample=value) - - @property - def subseed(self): - return self._proxy.subseed - - @subseed.setter - def subseed(self, value): - self._proxy = replace(self._proxy, subseed=value) - - @property - def subseed_strength(self): - return self._proxy.subseed_strength - - @subseed_strength.setter - def subseed_strength(self, value): - self._proxy = replace(self._proxy, subseed_strength=value) - - @property - def clipseg_model(self): - return self._proxy.clipseg_model - - @clipseg_model.setter - def clipseg_model(self, value): - self._proxy = replace(self._proxy, clipseg_model=value) - - @property - def seed_internal(self): - return self._proxy.seed_internal - - @seed_internal.setter - def seed_internal(self, value): - self._proxy = replace(self._proxy, seed_internal=value) diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 14719e627..8f7227a51 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -7,9 +7,9 @@ import pandas as pd from .data.anim import AnimationKeys, AnimationMode -from .data.proxy.root_data_proxy import RootDataProxyWrapper from .data.subtitle import Srt from .util import MemoryUtils +from ..args import RootArgs from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel from ..generate import isJson @@ -19,8 +19,6 @@ @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInitArgs: - # TODO eventually this should only keep the information required to run render_animation once - # for now it's just a direct reference or a copy of the actual args provided to the render_animation call. args: Any = None parseq_args: Any = None anim_args: Any = None @@ -35,7 +33,6 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -# TODO move elsewhere @dataclass(init=True, frozen=True, repr=False, eq=False) class StepInit: noise: Any = None @@ -82,7 +79,7 @@ def create(deform_keys, i): @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" - root: RootDataProxyWrapper + root: RootArgs seed: int args: RenderInitArgs parseq_adapter: Any @@ -240,9 +237,6 @@ def maybe_resume_from_timestring(anim_args, root): @staticmethod def do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): - # TODO all of those calls may cause a change in on of the passed args. - # Ideally it may be refactored so each one returns a new instance of the potentially changed args that are then - # attached as a property to this class to be used for one single render only. RenderInit.init_looper_if_active(args, loop_args) RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) RenderInit.create_output_directory_for_the_batch(args.outdir) @@ -262,16 +256,7 @@ def create(args_argument, parseq_args, anim_args, video_args, controlnet_args, prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args_argument) - - # TODO proxy other args like this. - # FIXME, somethings are still missing or accessors are not behaving as they should in the wrapper. - # root_proxy = RootDataProxyWrapper.create(root) - root_proxy = root - - instance = RenderInit(root_proxy, args_argument.seed, args, parseq_adapter, srt, animation_keys, + instance = RenderInit(root, args_argument.seed, args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - - # TODO avoid or isolate more side effects RenderInit.do_void_inits(args_argument, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) - return instance From dc03db8869cf37830f631ad9b7793602559a6a32 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 05:28:37 +0200 Subject: [PATCH 038/132] Context managenemt. --- scripts/deforum_helpers/render.py | 174 +++++++++--------- .../rendering/data/anim/animation_keys.py | 18 +- .../rendering/data/schedule.py | 21 ++- .../rendering/initialization.py | 86 ++++----- .../rendering/util/call_utils.py | 76 +++----- .../deforum_helpers/rendering/util/utils.py | 6 + 6 files changed, 187 insertions(+), 194 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 5fe3323ad..ecef425f5 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -65,6 +65,7 @@ # Flow Functions call_get_flow_from_images) from .rendering.util.memory_utils import MemoryUtils +from .rendering.util.utils import context from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed @@ -264,16 +265,17 @@ def run_render_animation(init): # TODO cadence related transforms to be decoupled and handled in a 2nd pass # do optical flow cadence after animation warping - if cadence_flow is not None: - cadence_flow = abs_flow_to_rel_flow(cadence_flow, init.width(), init.height()) - cadence_flow, _ = call_anim_frame_warp(init, tween_frame_idx, cadence_flow, depth) - cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, init.width(), init.height()) * tween - if turbo.is_advance_prev(): - turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, - step_init.cadence_flow_factor) - if turbo.is_advance_next(): - turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, - step_init.cadence_flow_factor) + with context(cadence_flow) as cf: + if cf is not None: + cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) + cf, _ = call_anim_frame_warp(init, tween_frame_idx, cf, depth) + cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * tween + if turbo.is_advance_prev(): + turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, + step_init.cadence_flow_factor) + if turbo.is_advance_next(): + turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, + step_init.cadence_flow_factor) turbo.prev_frame_idx = turbo.next_frame_idx = tween_frame_idx @@ -327,19 +329,20 @@ def run_render_animation(init): _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, step_init.hybrid_comp_schedules) # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing - if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: - if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) - else: - matrix = call_get_matrix_for_hybrid_motion(init, frame_idx - 1) - prev_img = image_transform_ransac(prev_img, matrix, init.args.anim_args.hybrid_motion) - if init.args.anim_args.hybrid_motion in ['Optical Flow']: - if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) - else: - flow = call_get_flow_for_hybrid_motion(init, frame_idx - 1) - prev_img = image_transform_optical_flow(prev_img, flow, step_init.flow_factor()) - init.animation_mode.prev_flow = flow + with context(init.args.anim_args) as aa: + if aa.hybrid_motion in ['Affine', 'Perspective']: + if aa.hybrid_motion_use_prev_img: + matrix = call_get_matrix_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) + else: + matrix = call_get_matrix_for_hybrid_motion(init, frame_idx - 1) + prev_img = image_transform_ransac(prev_img, matrix, aa.hybrid_motion) + if aa.hybrid_motion in ['Optical Flow']: + if aa.hybrid_motion_use_prev_img: + flow = call_get_flow_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) + else: + flow = call_get_flow_for_hybrid_motion(init, frame_idx - 1) + prev_img = image_transform_optical_flow(prev_img, flow, step_init.flow_factor()) + init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): @@ -377,12 +380,11 @@ def run_render_animation(init): noise_mask_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) - noised_image = add_noise(contrast_image, step_init.noise, init.args.args.seed, - init.args.anim_args.noise_type, - (init.args.anim_args.perlin_w, init.args.anim_args.perlin_h, - init.args.anim_args.perlin_octaves, - init.args.anim_args.perlin_persistence), - init.root.noise_mask, init.args.args.invert_mask) + + with context(init.args.anim_args) as aa: + noised_image = add_noise(contrast_image, step_init.noise, init.args.args.seed, aa.noise_type, + (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence), + init.root.noise_mask, init.args.args.invert_mask) # use transformed previous frame as init for current init.args.args.use_init = True @@ -398,47 +400,44 @@ def run_render_animation(init): # grab prompt for current frame init.args.args.prompt = init.prompt_series[frame_idx] - if init.args.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - init.args.args.seed = int(init.animation_keys.deform_keys.seed_schedule_series[frame_idx]) - - if init.args.anim_args.enable_checkpoint_scheduling: - init.args.args.checkpoint = init.animation_keys.deform_keys.checkpoint_schedule_series[frame_idx] - else: - init.args.args.checkpoint = None - - # SubSeed scheduling - if init.args.anim_args.enable_subseed_scheduling: - init.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.root.subseed_strength = float( - init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx]) - - if init.parseq_adapter.manages_seed(): - init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(init.animation_keys.deform_keys.subseed_schedule_series[frame_idx]) - init.root.subseed_strength = init.animation_keys.deform_keys.subseed_strength_schedule_series[frame_idx] - - # set value back into the prompt - prepare and report prompt and seed - init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, - init.args.args.seed, frame_idx) - - # grab init image for current frame - if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_init_path) - print(f"Using video init frame {init_frame}") - init.args.args.init_image = init_frame - init.args.args.init_image_box = None # init_image_box not used in this case - init.args.args.strength = max(0.0, min(1.0, step_init.strength)) - if init.args.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, frame_idx, init.args.anim_args.video_mask_path, True) - temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) - init.args.args.mask_file = temp_mask - init.root.noise_mask = temp_mask - mask_vals['video_mask'] = temp_mask - - if init.args.args.use_mask: - init.args.args.mask_image = compose_mask_with_check(init.root, init.args.args, schedule.mask_seq, - mask_vals, init.root.init_sample) \ - if init.root.init_sample is not None else None # we need it only after the first frame anyway + with context(init.args) as ia: + with context(init.animation_keys.deform_keys) as keys: + if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): + ia.args.seed = int(keys.seed_schedule_series[frame_idx]) + if ia.anim_args.enable_checkpoint_scheduling: + ia.args.checkpoint = keys.checkpoint_schedule_series[frame_idx] + else: + ia.args.checkpoint = None + + # SubSeed scheduling + if ia.anim_args.enable_subseed_scheduling: + init.root.subseed = int(keys.subseed_schedule_series[frame_idx]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[frame_idx]) + if init.parseq_adapter.manages_seed(): + init.args.anim_args.enable_subseed_scheduling = True + init.root.subseed = int(keys.subseed_schedule_series[frame_idx]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[frame_idx] + + # set value back into the prompt - prepare and report prompt and seed + ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, frame_idx) + # grab init image for current frame + if init.animation_mode.has_video_input: + init_frame = call_get_next_frame(init, frame_idx, ia.anim_args.video_init_path) + print(f"Using video init frame {init_frame}") + ia.args.init_image = init_frame + ia.args.init_image_box = None # init_image_box not used in this case + ia.args.strength = max(0.0, min(1.0, step_init.strength)) + if ia.anim_args.use_mask_video: + mask_init_frame = call_get_next_frame(init, frame_idx, ia.anim_args.video_mask_path, True) + temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) + ia.args.mask_file = temp_mask + init.root.noise_mask = temp_mask + mask_vals['video_mask'] = temp_mask + + if ia.args.use_mask: + ia.args.mask_image = compose_mask_with_check(init.root, ia.args, schedule.mask_seq, + mask_vals, init.root.init_sample) \ + if init.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) setup_opts(init, schedule) @@ -456,20 +455,21 @@ def run_render_animation(init): # optical flow redo before generation if optical_flow_redo_generation != 'None' and prev_img is not None and step_init.strength > 0: stored_seed = init.args.args.seed - init.args.args.seed = random.randint(0, 2 ** 32 - 1) - print( - f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {init.args.args.seed} optical flow before generation.") - - disposable_image = call_generate(init, frame_idx, schedule) - disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) - disposable_flow = call_get_flow_from_images(init, prev_img, disposable_image, optical_flow_redo_generation) - disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) - disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, - step_init.redo_flow_factor) - init.args.args.seed = stored_seed - init.root.init_sample = Image.fromarray(disposable_image) - del (disposable_image, disposable_flow, stored_seed) - gc.collect() + init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere + msg_start = "Optical flow redo is diffusing and warping using" + msg_end = "optical flow before generation." + print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") + + with context(call_generate(init, frame_idx, schedule)) as img: + img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) + disposable_flow = call_get_flow_from_images(init, prev_img, img, optical_flow_redo_generation) + img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) + img = image_transform_optical_flow(img, disposable_flow, step_init.redo_flow_factor) + init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds + init.root.init_sample = Image.fromarray(img) + disposable_image = img # TODO refactor + del (img, disposable_flow, stored_seed) + gc.collect() # diffusion redo if (int(init.args.anim_args.diffusion_redo) > 0 @@ -478,7 +478,7 @@ def run_render_animation(init): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") - init.args.args.seed = random.randint(0, 2 ** 32 - 1) + init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere disposable_image = call_generate(init, frame_idx, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -487,8 +487,8 @@ def run_render_animation(init): init.args.anim_args.color_coherence) init.args.args.seed = stored_seed init.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) - del (disposable_image, stored_seed) # FIXME disposable_image might be referenced before assignment. - gc.collect() # TODO try to eventually kick the gc only once at the end of every generation or iteration. + del (disposable_image, stored_seed) + gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation image = call_generate(init, frame_idx, schedule) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index a0adeb3f1..e34f54f3a 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -1,4 +1,6 @@ from dataclasses import dataclass + +from ...util.utils import context from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys @@ -8,14 +10,14 @@ class AnimationKeys: looper_keys: LooperAnimKeys def update(self, i: int): - self.looper_keys.use_looper = self.looper_keys.use_looper - self.looper_keys.imagesToKeyframe = self.looper_keys.imagesToKeyframe - # TODO FIXME refactor index handling and remove new boilerplate - self.looper_keys.imageStrength = self.looper_keys.image_strength_schedule_series[i] - self.looper_keys.blendFactorMax = self.looper_keys.blendFactorMax_series[i] - self.looper_keys.blendFactorSlope = self.looper_keys.blendFactorSlope_series[i] - self.looper_keys.tweeningFramesSchedule = self.looper_keys.tweening_frames_schedule_series[i] - self.looper_keys.colorCorrectionFactor = self.looper_keys.color_correction_factor_series[i] + with context(self.looper_keys) as keys: + keys.use_looper = keys.use_looper + keys.imagesToKeyframe = keys.imagesToKeyframe + keys.imageStrength = keys.image_strength_schedule_series[i] + keys.blendFactorMax = keys.blendFactorMax_series[i] + keys.blendFactorSlope = keys.blendFactorSlope_series[i] + keys.tweeningFramesSchedule = keys.tweening_frames_schedule_series[i] + keys.colorCorrectionFactor = keys.color_correction_factor_series[i] @staticmethod def choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index d398d9f92..f05368096 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,6 +1,8 @@ from dataclasses import dataclass from typing import Optional, Any +from ..util.utils import context + @dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: @@ -17,15 +19,16 @@ class Schedule: def create(keys, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" - steps = Schedule.schedule_steps(keys, i, anim_args) - sampler_name = Schedule.schedule_sampler(keys, i, anim_args) - clipskip = Schedule.schedule_clipskip(keys, i, anim_args) - noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) - eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) - eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) - mask = Schedule.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args - noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) - return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) + with context(Schedule) as S: + steps = S.schedule_steps(keys, i, anim_args) + sampler_name = S.schedule_sampler(keys, i, anim_args) + clipskip = S.schedule_clipskip(keys, i, anim_args) + noise_multiplier = S.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = S.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = S.schedule_ancestral_eta(keys, i, anim_args) + mask = S.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args + noise_mask = S.schedule_noise_mask(keys, i, anim_args) + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index 8f7227a51..fa223bd2b 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -9,6 +9,7 @@ from .data.anim import AnimationKeys, AnimationMode from .data.subtitle import Srt from .util import MemoryUtils +from .util.utils import context from ..args import RootArgs from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel @@ -55,25 +56,28 @@ def flow_factor(self): @staticmethod def create(deform_keys, i): - return StepInit( - deform_keys.noise_schedule_series[i], - deform_keys.strength_schedule_series[i], - deform_keys.cfg_scale_schedule_series[i], - deform_keys.contrast_schedule_series[i], - int(deform_keys.kernel_schedule_series[i]), - deform_keys.sigma_schedule_series[i], - deform_keys.amount_schedule_series[i], - deform_keys.threshold_schedule_series[i], - deform_keys.cadence_flow_factor_schedule_series[i], - deform_keys.redo_flow_factor_schedule_series[i], { - "alpha": deform_keys.hybrid_comp_alpha_schedule_series[i], - "mask_blend_alpha": deform_keys.hybrid_comp_mask_blend_alpha_schedule_series[i], - "mask_contrast": deform_keys.hybrid_comp_mask_contrast_schedule_series[i], - "mask_auto_contrast_cutoff_low": - int(deform_keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), - "mask_auto_contrast_cutoff_high": - int(deform_keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), - "flow_factor": deform_keys.hybrid_flow_factor_schedule_series[i]}) + with context(deform_keys) as keys: + return StepInit(keys.noise_schedule_series[i], + keys.strength_schedule_series[i], + keys.cfg_scale_schedule_series[i], + keys.contrast_schedule_series[i], + int(keys.kernel_schedule_series[i]), + keys.sigma_schedule_series[i], + keys.amount_schedule_series[i], + keys.threshold_schedule_series[i], + keys.cadence_flow_factor_schedule_series[i], + keys.redo_flow_factor_schedule_series[i], + StepInit._hybrid_comp_args(keys, i)) + + @staticmethod + def _hybrid_comp_args(keys, i): + return { + "alpha": keys.hybrid_comp_alpha_schedule_series[i], + "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[i], + "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[i], + "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), + "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), + "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} @dataclass(init=True, frozen=True, repr=False, eq=False) @@ -104,7 +108,6 @@ def height(self) -> int: return self.args.args.H def dimensions(self) -> tuple[int, int]: - # TODO should ideally only be called once each render return self.width(), self.height() # TODO group hybrid stuff elsewhere @@ -236,27 +239,24 @@ def maybe_resume_from_timestring(anim_args, root): root.timestring = anim_args.resume_timestring if anim_args.resume_from_timestring else root.timestring @staticmethod - def do_void_inits(args, loop_args, controlnet_args, anim_args, parseq_args, video_args, root): - RenderInit.init_looper_if_active(args, loop_args) - RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - RenderInit.create_output_directory_for_the_batch(args.outdir) - RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - RenderInit.maybe_resume_from_timestring(anim_args, root) - - @staticmethod - def create(args_argument, parseq_args, anim_args, video_args, controlnet_args, + def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': - args = RenderInitArgs(args_argument, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - output_directory = args_argument.outdir - is_use_mask = args_argument.use_mask - parseq_adapter = RenderInit.create_parseq_adapter(args) - srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) - animation_keys = AnimationKeys.from_args(args, parseq_adapter, args_argument.seed) - animation_mode = AnimationMode.from_args(args) - prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( - animation_mode, root, anim_args, args_argument) - instance = RenderInit(root, args_argument.seed, args, parseq_adapter, srt, animation_keys, - animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - RenderInit.do_void_inits(args_argument, loop_args, controlnet_args, anim_args, parseq_args, video_args, root) - return instance + ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + output_directory = args.outdir + is_use_mask = args.use_mask + with context(RenderInit) as RI: + parseq_adapter = RI.create_parseq_adapter(ri_args) + srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) + animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) + animation_mode = AnimationMode.from_args(ri_args) + prompt_series = RI.select_prompts(parseq_adapter, anim_args, animation_keys, root) + depth_model = RI.create_depth_model_and_enable_depth_map_saving_if_active( + animation_mode, root, anim_args, args) + instance = RenderInit(root, args.seed, ri_args, parseq_adapter, srt, animation_keys, + animation_mode, prompt_series, depth_model, output_directory, is_use_mask) + RI.init_looper_if_active(args, loop_args) + RI.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + RI.create_output_directory_for_the_batch(args.outdir) + RI.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + RI.maybe_resume_from_timestring(anim_args, root) + return instance diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py index 41dac86e3..d08157c86 100644 --- a/scripts/deforum_helpers/rendering/util/call_utils.py +++ b/scripts/deforum_helpers/rendering/util/call_utils.py @@ -1,3 +1,4 @@ +from .utils import context from ...animation import anim_frame_warp from ...generate import generate from ...hybrid_video import ( @@ -44,27 +45,16 @@ def call_some_function(init, i, ...): # Animation: def call_anim_frame_warp(init, i, image, depth): - return anim_frame_warp(image, - init.args.args, - init.args.anim_args, - init.animation_keys.deform_keys, - i, - init.depth_model, - depth=depth, - device=init.args.root.device, - half_precision=init.args.root.half_precision) + with context(init.args) as ia: + return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, + depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) # Generation: def call_generate(init, i, schedule): - return generate(init.args.args, - init.animation_keys.deform_keys, - init.args.anim_args, - init.args.loop_args, - init.args.controlnet_args, - init.args.root, - init.parseq_adapter, - i, sampler_name=schedule.sampler_name) + with context(init.args) as ia: + return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, + ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) # Hybrid Video @@ -75,28 +65,26 @@ def call_get_flow_from_images(init, prev_image, next_image, cadence): def call_get_flow_for_hybrid_motion_prev(init, i, image): - return get_flow_for_hybrid_motion_prev(i, init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, - image, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + with context(init.animation_mode) as mode: + with context(init.args.anim_args) as aa: + return get_flow_for_hybrid_motion_prev(i, init.dimensions(), + mode.hybrid_input_files, + mode.hybrid_frame_path, + mode.prev_flow, + image, + aa.hybrid_flow_method, + mode.raft_model, + aa.hybrid_flow_consistency, + aa.hybrid_consistency_blur, + aa.hybrid_comp_save_extra_frames) def call_get_flow_for_hybrid_motion(init, i): - return get_flow_for_hybrid_motion(i, init.dimensions(), - init.animation_mode.hybrid_input_files, - init.animation_mode.hybrid_frame_path, - init.animation_mode.prev_flow, - init.args.anim_args.hybrid_flow_method, - init.animation_mode.raft_model, - init.args.anim_args.hybrid_flow_consistency, - init.args.anim_args.hybrid_consistency_blur, - init.args.anim_args.hybrid_comp_save_extra_frames) + with context(init.animation_mode) as mode: + with context(init.args.anim_args) as args: + return get_flow_for_hybrid_motion(i, init.dimensions(), mode.hybrid_input_files, mode.hybrid_frame_path, + mode.prev_flow, args.hybrid_flow_method, mode.raft_model, + args.hybrid_flow_consistency, args.hybrid_consistency_blur, args) def call_get_matrix_for_hybrid_motion_prev(init, i, image): @@ -110,12 +98,9 @@ def call_get_matrix_for_hybrid_motion(init, i): def call_hybrid_composite(init, i, image, hybrid_comp_schedules): - return hybrid_composite(init.args.args, - init.args.anim_args, - i, image, - init.depth_model, - hybrid_comp_schedules, - init.args.root) + with context(init.args) as ia: + return hybrid_composite(ia.args, ia.anim_args, i, image, init.depth_model, + hybrid_comp_schedules, init.args.root) # Load Images @@ -140,11 +125,8 @@ def call_write_frame_subtitle(init, i, params_string, is_cadence: bool = False) # Video & Audio def call_render_preview(init, i, last_preview_frame): - return render_preview(init.args.args, - init.args.anim_args, - init.args.video_args, - init.args.root, - i, last_preview_frame) + with context(init.args) as ia: + return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) def call_get_next_frame(init, i, video_path, is_mask: bool = False): diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index c5c294d63..8999b6dfe 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,3 +1,4 @@ +from contextlib import contextmanager def put_all(dictionaries, key, callable_or_value): @@ -8,3 +9,8 @@ def put_all(dictionaries, key, callable_or_value): def put_if_present(dictionary, key, value): if value is not None: dictionary[key] = value + + +@contextmanager +def context(cls_or_instance): + yield cls_or_instance From 986e7812202a30543a7d3bb7af537b0ef292af2d Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 07:35:22 +0200 Subject: [PATCH 039/132] Mask init and resume logic extracted. --- scripts/deforum_helpers/generate.py | 2 +- scripts/deforum_helpers/masks.py | 2 +- scripts/deforum_helpers/render.py | 64 +++-------------- .../rendering/data/__init__.py | 1 + .../deforum_helpers/rendering/data/mask.py | 71 +++++++++++++++++++ .../deforum_helpers/rendering/data/turbo.py | 14 +++- .../rendering/initialization.py | 3 +- .../rendering/util/__init__.py | 2 +- .../deforum_helpers/rendering/util/utils.py | 18 ++++- 9 files changed, 114 insertions(+), 63 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/mask.py diff --git a/scripts/deforum_helpers/generate.py b/scripts/deforum_helpers/generate.py index 64fc6135f..b352067c1 100644 --- a/scripts/deforum_helpers/generate.py +++ b/scripts/deforum_helpers/generate.py @@ -256,7 +256,7 @@ def generate_inner(args, keys, anim_args, loop_args, controlnet_args, root, pars if processed is None: # Mask functions if args.use_mask: - mask_image = args.mask_image + mask_image = args.image mask = prepare_mask(args.mask_file if mask_image is None else mask_image, (args.W, args.H), args.mask_contrast_adjust, diff --git a/scripts/deforum_helpers/masks.py b/scripts/deforum_helpers/masks.py index af3d8bfe0..2f3bd7f63 100644 --- a/scripts/deforum_helpers/masks.py +++ b/scripts/deforum_helpers/masks.py @@ -31,7 +31,7 @@ def do_overlay_mask(args, anim_args, img, frame_idx, is_bgr_array=False): current_mask = Image.open(os.path.join(args.outdir, 'maskframes', get_frame_name(anim_args.video_mask_path) + f"{frame_idx:09}.jpg")) current_frame = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name(anim_args.video_init_path) + f"{frame_idx:09}.jpg")) elif args.use_mask: - current_mask = args.mask_image if args.mask_image is not None else load_image(args.mask_file, None) + current_mask = args.image if args.image is not None else load_image(args.mask_file, None) if args.init_image is None and args.init_image_box is None: current_frame = img else: diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index ecef425f5..1d6602e02 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -32,12 +32,13 @@ from .hybrid_video import (image_transform_ransac, image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask -from .load_images import get_mask, load_img, load_image +from .load_images import load_image from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt from .rendering.data import Turbo from .rendering.data.schedule import Schedule +from .rendering.data.mask import Mask from .rendering.initialization import RenderInit, StepInit from .rendering.util import put_if_present, put_all from .rendering.util.call_utils import ( @@ -66,7 +67,6 @@ call_get_flow_from_images) from .rendering.util.memory_utils import MemoryUtils from .rendering.util.utils import context -from .resume import get_resume_vars from .save_images import save_image from .seed import next_seed from .video_audio_utilities import get_frame_name @@ -90,44 +90,11 @@ def run_render_animation(init): # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. turbo = Turbo.create(init) # state for interpolating between diffusion steps - - # initialize vars prev_img = None color_match_sample = None - start_frame = 0 - - # resume animation (requires at least two frames - see function) - if init.is_resuming_from_timestring(): - # determine last frame and frame to start on - prev_frame, next_frame, prev_img, next_img = get_resume_vars( - folder=init.args.args.outdir, - timestring=init.args.anim_args.resume_timestring, - cadence=turbo.steps) - - turbo.set_up_step_vars(prev_img, prev_frame, next_img, next_frame) - - # advance start_frame to next frame - start_frame = next_frame + 1 - + start_frame = turbo.find_start(init, turbo) frame_idx = start_frame - - # reset the mask vals as they are overwritten in the compose_mask algorithm - mask_vals = {} - noise_mask_vals = {} - - put_all([mask_vals, noise_mask_vals], 'everywhere', - lambda: Image.new('1', init.dimensions(), 1)) - - mask_image = None - - if init.is_using_init_image_or_box(): - _, mask_image = load_img(init.args.args.init_image, - init.args.args.init_image_box, - shape=init.dimensions(), - use_alpha_as_mask=init.args.args.use_alpha_as_mask) - put_all([mask_vals, noise_mask_vals], 'video_mask', mask_image) - - assign_masks(init, frame_idx, mask_image, [mask_vals, noise_mask_vals]) + mask = Mask.create(init, frame_idx) # reset the mask vals as they are overwritten in the compose_mask algorithm # get color match for 'Image' color coherence only once, before loop if init.args.anim_args.color_coherence == 'Image': @@ -370,14 +337,14 @@ def run_render_animation(init): step_init.sigma, step_init.amount, step_init.threshold, - mask_image if init.args.args.use_mask else None) + mask.image if init.args.args.use_mask else None) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: init.root.noise_mask = compose_mask_with_check(init.root, init.args.args, noise_mask_seq, # FIXME might be ref'd b4 assignment - noise_mask_vals, + mask.noise_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) @@ -432,11 +399,12 @@ def run_render_animation(init): temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) ia.args.mask_file = temp_mask init.root.noise_mask = temp_mask - mask_vals['video_mask'] = temp_mask + mask.vals['video_mask'] = temp_mask if ia.args.use_mask: + # TODO figure why this is different from mask.image ia.args.mask_image = compose_mask_with_check(init.root, ia.args, schedule.mask_seq, - mask_vals, init.root.init_sample) \ + mask.vals, init.root.init_sample) \ if init.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(frame_idx) @@ -560,20 +528,6 @@ def run_render_animation(init): init.animation_mode.cleanup() -def assign_masks(init, i, is_mask_image, dicts): - # Grab the first frame masks since they wont be provided until next frame - # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set - # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 - key = 'video_mask' - if init.args.anim_args.use_mask_video: - mask = call_get_mask_from_file(init, i, True) - init.args.args.mask_file = mask - init.root.noise_mask = mask - put_all(dicts, key, mask) - elif is_mask_image is None and init.is_use_mask: - put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask - - def setup_opts(init, schedule): data = init.args.opts.data if init.has_img2img_fix_steps(): diff --git a/scripts/deforum_helpers/rendering/data/__init__.py b/scripts/deforum_helpers/rendering/data/__init__.py index dfe07eb23..aeb459403 100644 --- a/scripts/deforum_helpers/rendering/data/__init__.py +++ b/scripts/deforum_helpers/rendering/data/__init__.py @@ -1,2 +1,3 @@ +from .mask import Mask from .schedule import Schedule from .turbo import Turbo diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py new file mode 100644 index 000000000..ae2909276 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -0,0 +1,71 @@ +from dataclasses import dataclass +from typing import Any + +from PIL import Image + +from ..util import put_all +from ..util.utils import create_img, call_or_use_on_cond, context +from ...load_images import get_mask, load_img +from ...rendering.util.call_utils import call_get_mask_from_file + + +# TODO freeze? +@dataclass(init=True, frozen=False, repr=False, eq=False) +class Mask: + image: Image + vals: Any + noise_vals: Any + + def has_mask_image(self): + return self.image is not None + + @staticmethod + def assign_masks(init, i, is_mask_image, dicts): + # Grab the first frame masks since they wont be provided until next frame + # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set + # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 + key = 'video_mask' + if init.args.anim_args.use_mask_video: + mask = call_get_mask_from_file(init, i, True) + init.args.args.mask_file = mask + init.root.noise_mask = mask + put_all(dicts, key, mask) + elif is_mask_image is None and init.is_use_mask: + put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask + + @staticmethod + def _create_vals(count, dimensions): + return list(map(lambda _: {'everywhere': create_img(dimensions)}, range(count))) + + @staticmethod + def _assign(init, i, is_mask_image, dicts): + # Grab the first frame masks since they wont be provided until next frame + # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set + # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 + key = 'video_mask' + if init.args.anim_args.use_mask_video: + mask = call_get_mask_from_file(init, i, True) + init.args.args.mask_file = mask + init.root.noise_mask = mask + put_all(dicts, key, mask) + elif is_mask_image is None and init.is_use_mask: + put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask + + @staticmethod + def _create_mask_image(init): + with context(init.args.args) as args: + return call_or_use_on_cond(init.is_using_init_image_or_box(), + lambda: load_img(args.init_image, args.init_image_box, shape=init.dimensions(), + use_alpha_as_mask=args.use_alpha_as_mask)[1]) + + @staticmethod + def _create(init, i, mask_image): + mask_and_noise_mask = Mask._create_vals(2, init.dimensions()) + put_all(mask_and_noise_mask, 'video_mask', mask_image) + Mask._assign(init, i, mask_image, mask_and_noise_mask) + return Mask(mask_image, mask_and_noise_mask[0], mask_and_noise_mask[1]) + + @staticmethod + def create(init, i): + mask_image = Mask._create_mask_image(init) + return call_or_use_on_cond(mask_image is not None, Mask._create(init, i, mask_image)) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index ceb2eb644..3548d5aaf 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -2,6 +2,7 @@ from typing import Any from .subtitle import Srt +from ...resume import get_resume_vars # TODO freeze.. @@ -18,10 +19,21 @@ def create(init): steps = 1 if init.has_video_input() else init.cadence() return Turbo(steps, None, 0, None, 0) - def set_up_step_vars(self, prev_img, prev_frame, next_img, next_frame): + def _set_up_step_vars(self, init, turbo): + # determine last frame and frame to start on + prev_frame, next_frame, prev_img, next_img = get_resume_vars( + folder=init.args.args.outdir, + timestring=init.args.anim_args.resume_timestring, + cadence=turbo.steps) if self.steps > 1: self.prev_image, self.prev_frame_idx = prev_img, prev_frame self.next_image, self.next_frame_idx = next_img, next_frame + return next_frame + + def find_start(self, init, turbo): + """Maybe resume animation (requires at least two frames - see function).""" + # set start_frame to next frame + return self._set_up_step_vars(init, turbo) + 1 if init.is_resuming_from_timestring() else 0 def _has_prev_image(self): return self.prev_image is not None diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index fa223bd2b..e1a6f6e4a 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -5,11 +5,12 @@ import numexpr import numpy as np import pandas as pd +from PIL import Image from .data.anim import AnimationKeys, AnimationMode from .data.subtitle import Srt from .util import MemoryUtils -from .util.utils import context +from .util.utils import context, put_all, create_img from ..args import RootArgs from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 6e8cf76df..6708474e9 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -5,4 +5,4 @@ call_get_matrix_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion) from .memory_utils import MemoryUtils -from .utils import put_all, put_if_present +from .utils import put_all, put_if_present, call_or_use_on_cond diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index 8999b6dfe..11150598d 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,9 +1,9 @@ from contextlib import contextmanager +from PIL import Image -def put_all(dictionaries, key, callable_or_value): - for dictionary in dictionaries: - dictionary[key] = callable_or_value() if callable(callable_or_value) else callable_or_value +def put_all(dictionaries, key, value): + return list(map(lambda d: {**d, key: value}, dictionaries)) def put_if_present(dictionary, key, value): @@ -11,6 +11,18 @@ def put_if_present(dictionary, key, value): dictionary[key] = value +def _call_or_use(): + return callable_or_value() if callable(callable_or_value) else callable_or_value + + +def call_or_use_on_cond(condition, callable_or_value): + return _call_or_use() if condition else None + + @contextmanager def context(cls_or_instance): yield cls_or_instance + + +def create_img(dimensions): + return Image.new('1', dimensions, 1) From 4c0b4a627f44c2ba2aff5340a17b4c67fb78c227 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 15:17:45 +0200 Subject: [PATCH 040/132] Juggled some logic and moved indices and images into new dataclasses. --- .../inspectionProfiles/profiles_settings.xml | 1 + scripts/deforum_helpers/render.py | 290 +++++++++--------- .../rendering/data/__init__.py | 2 + .../deforum_helpers/rendering/data/images.py | 22 ++ .../deforum_helpers/rendering/data/indexes.py | 14 + 5 files changed, 181 insertions(+), 148 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/images.py create mode 100644 scripts/deforum_helpers/rendering/data/indexes.py diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml index 105ce2da2..dd4c951ef 100644 --- a/.idea/inspectionProfiles/profiles_settings.xml +++ b/.idea/inspectionProfiles/profiles_settings.xml @@ -1,5 +1,6 @@ + diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 1d6602e02..57a4367b3 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -18,7 +18,6 @@ import os import random import time - import PIL import cv2 import numpy as np @@ -26,21 +25,21 @@ from deforum_api import JobStatusTracker from modules import lowvram, devices, sd_hijack from modules.shared import opts, cmd_opts, state, sd_model - from .colors import maintain_colors from .composable_masks import compose_mask_with_check from .hybrid_video import (image_transform_ransac, image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask -from .load_images import load_image from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt from .rendering.data import Turbo from .rendering.data.schedule import Schedule +from .rendering.data.images import Images +from .rendering.data.indexes import Indexes from .rendering.data.mask import Mask from .rendering.initialization import RenderInit, StepInit -from .rendering.util import put_if_present, put_all +from .rendering.util import put_if_present from .rendering.util.call_utils import ( # Animation Functions call_anim_frame_warp, @@ -79,59 +78,35 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation_controlled(init): - raise NotImplementedError("not implemented.") - - -def run_render_animation(init): - # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. - # TODO isolate "depth" with other moving parts - # TODO isolate the relevant data in +250 refs to init.args, - # move that stuff to init and eventually try to drop init.args - # see dimensions() in RenderInit for an example of delegating the relevant stuff from args. - - turbo = Turbo.create(init) # state for interpolating between diffusion steps - prev_img = None - color_match_sample = None - start_frame = turbo.find_start(init, turbo) - frame_idx = start_frame - mask = Mask.create(init, frame_idx) # reset the mask vals as they are overwritten in the compose_mask algorithm - - # get color match for 'Image' color coherence only once, before loop - if init.args.anim_args.color_coherence == 'Image': - color_match_sample = load_image(init.args.anim_args.color_coherence_image_path, None) - color_match_sample = color_match_sample.resize(init.dimensions(), PIL.Image.LANCZOS) - color_match_sample = cv2.cvtColor(np.array(color_match_sample), cv2.COLOR_RGB2BGR) - - # Webui - state.job_count = init.args.anim_args.max_frames - last_preview_frame = 0 - # TODO create a Step class in rendering.data with all the iteration specific info, - # then eventually try to replace this while loop with functions that: + # then eventually try to replace the main while loop in `run_render_animation` with functions that: # - 1. Create a collection of Steps with all the required info that is already known or can be calculated # before we enter the iteration. # - 2. Transform and reprocess the steps however needed (i.e. space out or reassign turbo frames etc.) # TODO cadence framing and logic that is currently working off-index may eventually be moved into a 2nd pass. # - 3. Actually do the render by foreaching over the steps in sequence - while frame_idx < init.args.anim_args.max_frames: - # Webui + # TODO also create a SubStep class for the inner for-loop in `run_render_animation` (maybe do that 1st). + raise NotImplementedError("not implemented.") - state.job = f"frame {frame_idx + 1}/{init.args.anim_args.max_frames}" - state.job_no = frame_idx + 1 - if state.skipped: - print("\n** PAUSED **") - state.skipped = False - while not state.skipped: - time.sleep(0.1) - print("** RESUMING **") +def run_render_animation(init): + # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. + # TODO isolate "depth" with other moving parts - print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{init.args.anim_args.max_frames} ") + images = Images.create(init) + turbo = Turbo.create(init) # state for interpolating between diffusion steps + indexes = Indexes.create(init, turbo) # TODO reconsider index set... + mask = Mask.create(init, indexes.frame) # reset the mask vals as they are overwritten in the compose_mask algorithm - step_init = StepInit.create(init.animation_keys.deform_keys, frame_idx) + _init_web_ui_job(init) + last_preview_frame = 0 + while indexes.frame < init.args.anim_args.max_frames: + _update_web_ui_job(init, indexes) + print(f"\033[36mAnimation frame: \033[0m{indexes.frame}/{init.args.anim_args.max_frames} ") - # TODO eventually move schedule into new Step class - schedule = Schedule.create(init.animation_keys.deform_keys, frame_idx, init.args.anim_args, init.args.args) + # TODO Stuff to be moved into new Step class in `run_render_animation_controlled`: + step_init = StepInit.create(init.animation_keys.deform_keys, indexes.frame) + schedule = Schedule.create(init.animation_keys.deform_keys, indexes.frame, init.args.anim_args, init.args.args) if init.is_use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq @@ -148,24 +123,26 @@ def run_render_animation(init): if turbo.is_first_step_with_subtitles(init): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, frame_idx, params_to_print) - call_write_frame_subtitle(init, frame_idx, params_string) + params_string = call_format_animation_params(init, indexes.frame, params_to_print) + call_write_frame_subtitle(init, indexes.frame, params_string) params_string = None # FIXME ?? if turbo.is_emit_in_between_frames(): - tween_frame_start_idx = max(start_frame, frame_idx - turbo.steps) + indexes.tween_frame_start = max(indexes.start_frame, indexes.frame - turbo.steps) cadence_flow = None - for tween_frame_idx in range(tween_frame_start_idx, frame_idx): + # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` + for indexes.tween_frame in range(indexes.tween_frame_start, indexes.frame): # update progress during cadence - state.job = f"frame {tween_frame_idx + 1}/{init.args.anim_args.max_frames}" - state.job_no = tween_frame_idx + 1 + state.job = f"frame {indexes.tween_frame + 1}/{init.args.anim_args.max_frames}" + state.job_no = indexes.tween_frame + 1 # cadence vars - tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx) + tween = (float(indexes.tween_frame - indexes.tween_frame_start + 1) / + float(indexes.frame - indexes.tween_frame_start)) # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): - if init.animation_keys.deform_keys.strength_schedule_series[tween_frame_start_idx] > 0: + if init.animation_keys.deform_keys.strength_schedule_series[indexes.tween_frame_start] > 0: if cadence_flow is None and turbo.prev_image is not None and turbo.next_image is not None: cadence_flow = call_get_flow_from_images(init, turbo.prev_image, turbo.next_image, init.args.anim_args.optical_flow_cadence) / 2 @@ -173,12 +150,14 @@ def run_render_animation(init): if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, tween_frame_idx, params_to_print) - call_write_frame_subtitle(init, tween_frame_idx, params_string, tween < 1.0) + params_string = call_format_animation_params(init, indexes.tween_frame, params_to_print) + call_write_frame_subtitle(init, indexes.tween_frame, params_string, tween < 1.0) params_string = None - print( - f"Creating in-between {'' if cadence_flow is None else init.args.anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") + msg_flow_name = '' if cadence_flow is None \ + else init.args.anim_args.optical_flow_cadence + ' optical flow ' + msg_frame_info = f"cadence frame: {indexes.tween_frame}; tween:{tween:0.2f};" + print(f"Creating in-between {msg_flow_name}{msg_frame_info}") if init.depth_model is not None: assert (turbo.next_image is not None) @@ -186,46 +165,46 @@ def run_render_animation(init): init.root.half_precision) # TODO collect images - if turbo.is_advance_prev(tween_frame_idx): - turbo.prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo.prev_image, depth) - if turbo.is_advance_next(tween_frame_idx): - turbo.prev_image, _ = call_anim_frame_warp(init, tween_frame_idx, turbo.next_image, depth) + if turbo.is_advance_prev(indexes.tween_frame): + turbo.prev_image, _ = call_anim_frame_warp(init, indexes.tween_frame, turbo.prev_image, depth) + if turbo.is_advance_next(indexes.tween_frame): + turbo.prev_image, _ = call_anim_frame_warp(init, indexes.tween_frame, turbo.next_image, depth) # hybrid video motion - warps turbo.prev_image or turbo.next_image to match motion - if tween_frame_idx > 0: + if indexes.tween_frame > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) - if turbo.is_advance_prev(tween_frame_idx): + matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween_frame - 1, images.previous) + if turbo.is_advance_prev(indexes.tween_frame): turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(tween_frame_idx): + if turbo.is_advance_next(indexes.tween_frame): turbo.next_image = image_transform_ransac(turbo.next_image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = call_get_matrix_for_hybrid_motion(init, tween_frame_idx - 1) - if turbo.is_advance_prev(tween_frame_idx): + matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween_frame - 1) + if turbo.is_advance_prev(indexes.tween_frame): turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(tween_frame_idx): + if turbo.is_advance_next(indexes.tween_frame): turbo.next_image = image_transform_ransac(turbo.next_image, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, tween_frame_idx - 1, prev_img) - if turbo.is_advance_prev(tween_frame_idx): + flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween_frame - 1, images.previous) + if turbo.is_advance_prev(indexes.tween_frame): turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, step_init.flow_factor()) - if turbo.is_advance_next(tween_frame_idx): + if turbo.is_advance_next(indexes.tween_frame): turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow else: - flow = call_get_flow_for_hybrid_motion(init, tween_frame_idx - 1) - if turbo.is_advance_prev(tween_frame_idx): + flow = call_get_flow_for_hybrid_motion(init, indexes.tween_frame - 1) + if turbo.is_advance_prev(indexes.tween_frame): turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, step_init.flow_factor()) - if turbo.is_advance_next(tween_frame_idx): + if turbo.is_advance_next(indexes.tween_frame): turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow @@ -235,7 +214,7 @@ def run_render_animation(init): with context(cadence_flow) as cf: if cf is not None: cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) - cf, _ = call_anim_frame_warp(init, tween_frame_idx, cf, depth) + cf, _ = call_anim_frame_warp(init, indexes.tween_frame, cf, depth) cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * tween if turbo.is_advance_prev(): turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, @@ -244,7 +223,7 @@ def run_render_animation(init): turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, step_init.cadence_flow_factor) - turbo.prev_frame_idx = turbo.next_frame_idx = tween_frame_idx + turbo.prev_frame_idx = turbo.next_frame_idx = indexes.tween_frame if turbo.prev_image is not None and tween < 1.0: img = turbo.prev_image * (1.0 - tween) + turbo.next_image * tween @@ -258,77 +237,77 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): - img = do_overlay_mask(init.args.args, init.args.anim_args, img, tween_frame_idx, True) + img = do_overlay_mask(init.args.args, init.args.anim_args, img, indexes.tween_frame, True) - # get prev_img during cadence - prev_img = img + # get images.previous during cadence + images.previous = img # current image update for cadence frames (left commented because it doesn't currently update the preview) # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{init.root.timestring}_{tween_frame_idx:09}.png" + filename = f"{init.root.timestring}_{indexes.tween_frame:09}.png" save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, - f"{init.root.timestring}_depth_{tween_frame_idx:09}.png") + f"{init.root.timestring}_depth_{indexes.tween_frame:09}.png") init.depth_model.save(dm_save_path, depth) - # get color match for video outside of prev_img conditional + # get color match for video outside of images.previous conditional if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(frame_idx) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + if int(indexes.frame) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(init.output_directory, 'inputframes', get_frame_name( - init.args.anim_args.video_init_path) + f"{frame_idx:09}.jpg")) + init.args.anim_args.video_init_path) + f"{indexes.frame:09}.jpg")) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) - color_match_sample = np.asarray(prev_vid_img) - color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) + images.color_match = np.asarray(prev_vid_img) + images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) - # after 1st frame, prev_img exists - if prev_img is not None: + # after 1st frame, images.previous exists + if images.previous is not None: # apply transforms to previous frame - prev_img, depth = call_anim_frame_warp(init, frame_idx, prev_img, None) + images.previous, depth = call_anim_frame_warp(init, indexes.frame, images.previous, None) # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, indexes.frame, images.previous, step_init.hybrid_comp_schedules) - # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing + # hybrid video motion - warps images.previous to match motion, usually to prepare for compositing with context(init.args.anim_args) as aa: if aa.hybrid_motion in ['Affine', 'Perspective']: if aa.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) + matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.frame - 1, images.previous) else: - matrix = call_get_matrix_for_hybrid_motion(init, frame_idx - 1) - prev_img = image_transform_ransac(prev_img, matrix, aa.hybrid_motion) + matrix = call_get_matrix_for_hybrid_motion(init, indexes.frame - 1) + images.previous = image_transform_ransac(images.previous, matrix, aa.hybrid_motion) if aa.hybrid_motion in ['Optical Flow']: if aa.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, frame_idx - 1, prev_img) + flow = call_get_flow_for_hybrid_motion_prev(init, indexes.frame - 1, images.previous) else: - flow = call_get_flow_for_hybrid_motion(init, frame_idx - 1) - prev_img = image_transform_optical_flow(prev_img, flow, step_init.flow_factor()) + flow = call_get_flow_for_hybrid_motion(init, indexes.frame - 1) + images.previous = image_transform_optical_flow(images.previous, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, prev_img = call_hybrid_composite(init, frame_idx, prev_img, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, indexes.frame, images.previous, step_init.hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): - if color_match_sample is None: - color_match_sample = prev_img.copy() + if images.color_match is None: + images.color_match = images.previous.copy() else: - prev_img = maintain_colors(prev_img, color_match_sample, init.args.anim_args.color_coherence) + images.previous = maintain_colors(images.previous, images.color_match, init.args.anim_args.color_coherence) # intercept and override to grayscale if init.args.anim_args.color_force_grayscale: - prev_img = cv2.cvtColor(prev_img, cv2.COLOR_BGR2GRAY) - prev_img = cv2.cvtColor(prev_img, cv2.COLOR_GRAY2BGR) + images.previous = cv2.cvtColor(images.previous, cv2.COLOR_BGR2GRAY) + images.previous = cv2.cvtColor(images.previous, cv2.COLOR_GRAY2BGR) # apply scaling - contrast_image = (prev_img * step_init.contrast).round().astype(np.uint8) + contrast_image = (images.previous * step_init.contrast).round().astype(np.uint8) # anti-blur if step_init.amount > 0: step_init.kernel_size() @@ -362,40 +341,41 @@ def run_render_animation(init): # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints init.args.args.pix2pix_img_cfg_scale = float( - init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[frame_idx]) + init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[indexes.frame]) # grab prompt for current frame - init.args.args.prompt = init.prompt_series[frame_idx] + init.args.args.prompt = init.prompt_series[indexes.frame] with context(init.args) as ia: with context(init.animation_keys.deform_keys) as keys: + # FIXME? check ia.args.seed_behavior if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - ia.args.seed = int(keys.seed_schedule_series[frame_idx]) + ia.args.seed = int(keys.seed_schedule_series[indexes.frame]) if ia.anim_args.enable_checkpoint_scheduling: - ia.args.checkpoint = keys.checkpoint_schedule_series[frame_idx] + ia.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame] else: ia.args.checkpoint = None # SubSeed scheduling if ia.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[frame_idx]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[frame_idx]) + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[frame_idx]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[frame_idx] + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame] # set value back into the prompt - prepare and report prompt and seed - ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, frame_idx) + ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, indexes.frame) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, frame_idx, ia.anim_args.video_init_path) + init_frame = call_get_next_frame(init, indexes.frame, ia.anim_args.video_init_path) print(f"Using video init frame {init_frame}") ia.args.init_image = init_frame ia.args.init_image_box = None # init_image_box not used in this case ia.args.strength = max(0.0, min(1.0, step_init.strength)) if ia.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, frame_idx, ia.anim_args.video_mask_path, True) + mask_init_frame = call_get_next_frame(init, indexes.frame, ia.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) ia.args.mask_file = temp_mask init.root.noise_mask = temp_mask @@ -407,8 +387,8 @@ def run_render_animation(init): mask.vals, init.root.init_sample) \ if init.root.init_sample is not None else None # we need it only after the first frame anyway - init.animation_keys.update(frame_idx) - setup_opts(init, schedule) + init.animation_keys.update(indexes.frame) + _setup_opts(init, schedule) if init.is_3d_with_med_or_low_vram(): if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') @@ -421,16 +401,16 @@ def run_render_animation(init): if not init.args.args.motion_preview_mode else 'None' # optical flow redo before generation - if optical_flow_redo_generation != 'None' and prev_img is not None and step_init.strength > 0: + if optical_flow_redo_generation != 'None' and images.previous is not None and step_init.strength > 0: stored_seed = init.args.args.seed init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere msg_start = "Optical flow redo is diffusing and warping using" msg_end = "optical flow before generation." print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") - with context(call_generate(init, frame_idx, schedule)) as img: + with context(call_generate(init, indexes.frame, schedule)) as img: img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - disposable_flow = call_get_flow_from_images(init, prev_img, img, optical_flow_redo_generation) + disposable_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) img = image_transform_optical_flow(img, disposable_flow, step_init.redo_flow_factor) init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds @@ -441,17 +421,17 @@ def run_render_animation(init): # diffusion redo if (int(init.args.anim_args.diffusion_redo) > 0 - and prev_img is not None and step_init.strength > 0 + and images.previous is not None and step_init.strength > 0 and not init.args.args.motion_preview_mode): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere - disposable_image = call_generate(init, frame_idx, schedule) + disposable_image = call_generate(init, indexes.frame, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(init.args.anim_args.diffusion_redo): - disposable_image = maintain_colors(prev_img, color_match_sample, + disposable_image = maintain_colors(images.previous, images.color_match, init.args.anim_args.color_coherence) init.args.args.seed = stored_seed init.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) @@ -459,23 +439,23 @@ def run_render_animation(init): gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation - image = call_generate(init, frame_idx, schedule) + image = call_generate(init, indexes.frame, schedule) if image is None: break # do hybrid video after generation - if frame_idx > 0 and init.is_hybrid_composite_after_generation(): + if indexes.frame > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = call_hybrid_composite(init, frame_idx, temp_image, step_init.hybrid_comp_schedules) + _, temp_image_2 = call_hybrid_composite(init, indexes.frame, temp_image, step_init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if frame_idx == 0 and init.is_color_match_to_be_initialized(color_match_sample): + if indexes.frame == 0 and init.is_color_match_to_be_initialized(images.color_match): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - temp_image = maintain_colors(temp_color, color_match_sample, init.args.anim_args.color_coherence) + temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) # intercept and override to grayscale @@ -485,24 +465,24 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): - image = do_overlay_mask(init.args.args, init.args.anim_args, image, frame_idx) + image = do_overlay_mask(init.args.args, init.args.anim_args, image, indexes.frame) # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) or (init.args.anim_args.legacy_colormatch and step_init.strength == 0)) and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): - color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) + images.color_match = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) if not init.animation_mode.has_video_input: - prev_img = opencv_image + images.previous = opencv_image if turbo.steps > 1: turbo.prev_image, turbo.prev_frame_idx = turbo.next_image, turbo.next_frame_idx - turbo.next_image, turbo.next_frame_idx = opencv_image, frame_idx - frame_idx += turbo.steps + turbo.next_image, turbo.next_frame_idx = opencv_image, indexes.frame + indexes.frame += turbo.steps else: - filename = f"{init.root.timestring}_{frame_idx:09}.png" + filename = f"{init.root.timestring}_{indexes.frame:09}.png" save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) if init.args.anim_args.save_depth_maps: @@ -515,20 +495,24 @@ def run_render_animation(init): depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.root.half_precision) init.depth_model.save( - os.path.join(init.output_directory, f"{init.root.timestring}_depth_{frame_idx:09}.png"), depth) + os.path.join(init.output_directory, f"{init.root.timestring}_depth_{indexes.frame:09}.png"), depth) if MemoryUtils.is_low_or_med_vram(): init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - frame_idx += 1 - - last_preview_frame = progress_and_make_preview(init, image, frame_idx, state, last_preview_frame) - update_tracker(init.root, frame_idx, init.args.anim_args) + indexes.frame += 1 + + # TODO consider state is a global import, so wouldn't really need passing here + state.assign_current_image(image) + # may reassign init.args.args and/or root.seed_internal # FIXME? + init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds + last_preview_frame = call_render_preview(init, indexes.frame, last_preview_frame) + _update_status_tracker(init.root, indexes.frame, init.args.anim_args) init.animation_mode.cleanup() -def setup_opts(init, schedule): +def _setup_opts(init, schedule): data = init.args.opts.data if init.has_img2img_fix_steps(): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations @@ -539,15 +523,25 @@ def setup_opts(init, schedule): put_if_present(data, "eta_ancestral", schedule.eta_ancestral) -def progress_and_make_preview(init, image, frame_idx, state, last_preview_frame): - state.assign_current_image(image) +WEB_UI_SLEEP_DELAY = 0.1 # TODO make a WebUi class? + + +def _init_web_ui_job(init): + state.job_count = init.args.anim_args.max_frames - # may reassign init.args.args and/or root.seed_internal # FIXME? - init.args.args.seed = next_seed(init.args.args, init.root) # TODO refactor assignment - # init.seed = init.args.args.seed # TODO group all seeds and sub-seeds - return call_render_preview(init, frame_idx, last_preview_frame) +def _update_web_ui_job(init, indexes): + frame = indexes.frame + 1 + max_frames = init.args.anim_args.max_frames + state.job = f"frame {frame}/{max_frames}" + state.job_no = frame + 1 + if state.skipped: + print("\n** PAUSED **") + state.skipped = False + while not state.skipped: + time.sleep(WEB_UI_SLEEP_DELAY) + print("** RESUMING **") -def update_tracker(root, frame_idx, anim_args): +def _update_status_tracker(root, frame_idx, anim_args): JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) diff --git a/scripts/deforum_helpers/rendering/data/__init__.py b/scripts/deforum_helpers/rendering/data/__init__.py index aeb459403..82ae199ac 100644 --- a/scripts/deforum_helpers/rendering/data/__init__.py +++ b/scripts/deforum_helpers/rendering/data/__init__.py @@ -1,3 +1,5 @@ +from .images import Images +from .indexes import Indexes from .mask import Mask from .schedule import Schedule from .turbo import Turbo diff --git a/scripts/deforum_helpers/rendering/data/images.py b/scripts/deforum_helpers/rendering/data/images.py new file mode 100644 index 000000000..9cec751dc --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/images.py @@ -0,0 +1,22 @@ +from dataclasses import dataclass + +import cv2 +from cv2.typing import MatLike + + +@dataclass(init=True, frozen=False, repr=False, eq=True) +class Images: + previous: MatLike = None + color_match: MatLike = None + + @staticmethod + def _load_color_match_sample(init) -> MatLike: + """get color match for 'Image' color coherence only once, before loop""" + if init.args.anim_args.color_coherence == 'Image': + raw_image = load_image(init.args.anim_args.color_coherence_image_path, None) + resized = raw_image.resize(init.dimensions(), PIL.Image.LANCZOS) + return cv2.cvtColor(np.array(resized), cv2.COLOR_RGB2BGR) + + @staticmethod + def create(init): + return Images(None, Images._load_color_match_sample(init)) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py new file mode 100644 index 000000000..896916342 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -0,0 +1,14 @@ +from dataclasses import dataclass + + +@dataclass(init=True, frozen=False, repr=False, eq=False) +class Indexes: + frame: int = 0 + start_frame: int = 0 + tween_frame: int = 0 + tween_frame_start: int = 0 + + @staticmethod + def create(init, turbo): + # TODO try to init everything right away... + return Indexes(0, turbo.find_start(init, turbo), 0, 0) From f5c4348f3956bd552ce7ce0d3f75f4dc80ba638a Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 15:26:55 +0200 Subject: [PATCH 041/132] Accidental change reverted. --- scripts/deforum_helpers/generate.py | 2 +- scripts/deforum_helpers/masks.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/deforum_helpers/generate.py b/scripts/deforum_helpers/generate.py index b352067c1..64fc6135f 100644 --- a/scripts/deforum_helpers/generate.py +++ b/scripts/deforum_helpers/generate.py @@ -256,7 +256,7 @@ def generate_inner(args, keys, anim_args, loop_args, controlnet_args, root, pars if processed is None: # Mask functions if args.use_mask: - mask_image = args.image + mask_image = args.mask_image mask = prepare_mask(args.mask_file if mask_image is None else mask_image, (args.W, args.H), args.mask_contrast_adjust, diff --git a/scripts/deforum_helpers/masks.py b/scripts/deforum_helpers/masks.py index 2f3bd7f63..af3d8bfe0 100644 --- a/scripts/deforum_helpers/masks.py +++ b/scripts/deforum_helpers/masks.py @@ -31,7 +31,7 @@ def do_overlay_mask(args, anim_args, img, frame_idx, is_bgr_array=False): current_mask = Image.open(os.path.join(args.outdir, 'maskframes', get_frame_name(anim_args.video_mask_path) + f"{frame_idx:09}.jpg")) current_frame = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name(anim_args.video_init_path) + f"{frame_idx:09}.jpg")) elif args.use_mask: - current_mask = args.image if args.image is not None else load_image(args.mask_file, None) + current_mask = args.mask_image if args.mask_image is not None else load_image(args.mask_file, None) if args.init_image is None and args.init_image_box is None: current_frame = img else: From 52b40dd1e365e3b7f0d3dbc8e8ab36bc757e6c97 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 17:38:20 +0200 Subject: [PATCH 042/132] Separated web-UI and memory handling and started extracting progression step logic. --- scripts/deforum_helpers/render.py | 278 +++++++----------- .../rendering/data/anim/animation_mode.py | 2 +- .../deforum_helpers/rendering/data/turbo.py | 53 ++-- .../rendering/util/call_utils.py | 8 + .../rendering/util/memory_utils.py | 36 +++ .../deforum_helpers/rendering/util/web_ui.py | 33 +++ 6 files changed, 227 insertions(+), 183 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/web_ui.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 57a4367b3..3a4fe7578 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -17,13 +17,10 @@ import gc import os import random -import time import PIL import cv2 import numpy as np from PIL import Image, ImageOps -from deforum_api import JobStatusTracker -from modules import lowvram, devices, sd_hijack from modules.shared import opts, cmd_opts, state, sd_model from .colors import maintain_colors from .composable_masks import compose_mask_with_check @@ -33,11 +30,7 @@ from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt -from .rendering.data import Turbo -from .rendering.data.schedule import Schedule -from .rendering.data.images import Images -from .rendering.data.indexes import Indexes -from .rendering.data.mask import Mask +from .rendering.data import (Turbo, Schedule, Images, Indexes, Mask) from .rendering.initialization import RenderInit, StepInit from .rendering.util import put_if_present from .rendering.util.call_utils import ( @@ -59,13 +52,13 @@ call_hybrid_composite, # Mask Functions - call_get_mask_from_file, call_get_mask_from_file_with_frame, # Flow Functions call_get_flow_from_images) from .rendering.util.memory_utils import MemoryUtils from .rendering.util.utils import context +from .rendering.util.web_ui import WebUi from .save_images import save_image from .seed import next_seed from .video_audio_utilities import get_frame_name @@ -95,117 +88,105 @@ def run_render_animation(init): images = Images.create(init) turbo = Turbo.create(init) # state for interpolating between diffusion steps - indexes = Indexes.create(init, turbo) # TODO reconsider index set... - mask = Mask.create(init, indexes.frame) # reset the mask vals as they are overwritten in the compose_mask algorithm + idx = Indexes.create(init, turbo) + mask = Mask.create(init, idx.frame) # reset the mask vals as they are overwritten in the compose_mask algorithm - _init_web_ui_job(init) + WebUi.init_job(init) last_preview_frame = 0 - while indexes.frame < init.args.anim_args.max_frames: - _update_web_ui_job(init, indexes) - print(f"\033[36mAnimation frame: \033[0m{indexes.frame}/{init.args.anim_args.max_frames} ") + while idx.frame < init.args.anim_args.max_frames: + WebUi.update_job(init, idx) + print(f"\033[36mAnimation frame: \033[0m{idx.frame}/{init.args.anim_args.max_frames} ") # TODO Stuff to be moved into new Step class in `run_render_animation_controlled`: - step_init = StepInit.create(init.animation_keys.deform_keys, indexes.frame) - schedule = Schedule.create(init.animation_keys.deform_keys, indexes.frame, init.args.anim_args, init.args.args) + step_init = StepInit.create(init.animation_keys.deform_keys, idx.frame) + schedule = Schedule.create(init.animation_keys.deform_keys, idx.frame, init.args.anim_args, init.args.args) if init.is_use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq depth = None - - if init.is_3d_with_med_or_low_vram(): - # Unload the main checkpoint and load the depth model - lowvram.send_everything_to_cpu() - sd_hijack.model_hijack.undo_hijack(sd_model) - devices.torch_gc() - if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(init.root.device) + MemoryUtils.handle_med_or_low_vram_before_step(init) if turbo.is_first_step_with_subtitles(init): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, indexes.frame, params_to_print) - call_write_frame_subtitle(init, indexes.frame, params_string) + params_string = call_format_animation_params(init, idx.frame, params_to_print) + call_write_frame_subtitle(init, idx.frame, params_string) params_string = None # FIXME ?? if turbo.is_emit_in_between_frames(): - indexes.tween_frame_start = max(indexes.start_frame, indexes.frame - turbo.steps) + idx.tween_frame_start = max(idx.start_frame, idx.frame - turbo.steps) cadence_flow = None # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` - for indexes.tween_frame in range(indexes.tween_frame_start, indexes.frame): - # update progress during cadence - state.job = f"frame {indexes.tween_frame + 1}/{init.args.anim_args.max_frames}" - state.job_no = indexes.tween_frame + 1 + for idx.tween_frame in range(idx.tween_frame_start, idx.frame): + WebUi.update_progress_during_cadence(init, idx) # cadence vars - tween = (float(indexes.tween_frame - indexes.tween_frame_start + 1) / - float(indexes.frame - indexes.tween_frame_start)) + tween = (float(idx.tween_frame - idx.tween_frame_start + 1) / + float(idx.frame - idx.tween_frame_start)) # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): - if init.animation_keys.deform_keys.strength_schedule_series[indexes.tween_frame_start] > 0: - if cadence_flow is None and turbo.prev_image is not None and turbo.next_image is not None: - cadence_flow = call_get_flow_from_images(init, turbo.prev_image, turbo.next_image, + if init.animation_keys.deform_keys.strength_schedule_series[idx.tween_frame_start] > 0: + if cadence_flow is None and turbo.prev.image is not None and turbo.next.image is not None: + cadence_flow = call_get_flow_from_images(init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2 - turbo.next_image = image_transform_optical_flow(turbo.next_image, -cadence_flow, 1) + turbo.next.image = image_transform_optical_flow(turbo.next.image, -cadence_flow, 1) if opts.data.get("deforum_save_gen_info_as_srt"): params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, indexes.tween_frame, params_to_print) - call_write_frame_subtitle(init, indexes.tween_frame, params_string, tween < 1.0) + params_string = call_format_animation_params(init, idx.tween_frame, params_to_print) + call_write_frame_subtitle(init, idx.tween_frame, params_string, tween < 1.0) params_string = None msg_flow_name = '' if cadence_flow is None \ else init.args.anim_args.optical_flow_cadence + ' optical flow ' - msg_frame_info = f"cadence frame: {indexes.tween_frame}; tween:{tween:0.2f};" + msg_frame_info = f"cadence frame: {idx.tween_frame}; tween:{tween:0.2f};" print(f"Creating in-between {msg_flow_name}{msg_frame_info}") if init.depth_model is not None: - assert (turbo.next_image is not None) - depth = init.depth_model.predict(turbo.next_image, init.args.anim_args.midas_weight, + assert (turbo.next.image is not None) + depth = init.depth_model.predict(turbo.next.image, + init.args.anim_args.midas_weight, init.root.half_precision) - # TODO collect images - if turbo.is_advance_prev(indexes.tween_frame): - turbo.prev_image, _ = call_anim_frame_warp(init, indexes.tween_frame, turbo.prev_image, depth) - if turbo.is_advance_next(indexes.tween_frame): - turbo.prev_image, _ = call_anim_frame_warp(init, indexes.tween_frame, turbo.next_image, depth) + turbo.advance(init, idx, depth) - # hybrid video motion - warps turbo.prev_image or turbo.next_image to match motion - if indexes.tween_frame > 0: + # hybrid video motion - warps turbo.prev.image or turbo.next.image to match motion + if idx.tween_frame > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween_frame - 1, images.previous) - if turbo.is_advance_prev(indexes.tween_frame): - turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, + matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.tween_frame - 1, images.previous) + if turbo.is_advance_prev(idx.tween_frame): + turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(indexes.tween_frame): - turbo.next_image = image_transform_ransac(turbo.next_image, matrix, + if turbo.is_advance_next(idx.tween_frame): + turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween_frame - 1) - if turbo.is_advance_prev(indexes.tween_frame): - turbo.prev_image = image_transform_ransac(turbo.prev_image, matrix, + matrix = call_get_matrix_for_hybrid_motion(init, idx.tween_frame - 1) + if turbo.is_advance_prev(idx.tween_frame): + turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(indexes.tween_frame): - turbo.next_image = image_transform_ransac(turbo.next_image, matrix, + if turbo.is_advance_next(idx.tween_frame): + turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween_frame - 1, images.previous) - if turbo.is_advance_prev(indexes.tween_frame): - turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, + flow = call_get_flow_for_hybrid_motion_prev(init, idx.tween_frame - 1, images.previous) + if turbo.is_advance_prev(idx.tween_frame): + turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, step_init.flow_factor()) - if turbo.is_advance_next(indexes.tween_frame): - turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, + if turbo.is_advance_next(idx.tween_frame): + turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow else: - flow = call_get_flow_for_hybrid_motion(init, indexes.tween_frame - 1) - if turbo.is_advance_prev(indexes.tween_frame): - turbo.prev_image = image_transform_optical_flow(turbo.prev_image, flow, + flow = call_get_flow_for_hybrid_motion(init, idx.tween_frame - 1) + if turbo.is_advance_prev(idx.tween_frame): + turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, step_init.flow_factor()) - if turbo.is_advance_next(indexes.tween_frame): - turbo.next_image = image_transform_optical_flow(turbo.next_image, flow, + if turbo.is_advance_next(idx.tween_frame): + turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow @@ -214,21 +195,21 @@ def run_render_animation(init): with context(cadence_flow) as cf: if cf is not None: cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) - cf, _ = call_anim_frame_warp(init, indexes.tween_frame, cf, depth) + cf, _ = call_anim_frame_warp(init, idx.tween_frame, cf, depth) cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * tween if turbo.is_advance_prev(): - turbo.prev_image = image_transform_optical_flow(turbo.prev_image, cadence_flow_inc, + turbo.prev.image = image_transform_optical_flow(turbo.prev.image, cadence_flow_inc, step_init.cadence_flow_factor) if turbo.is_advance_next(): - turbo.next_image = image_transform_optical_flow(turbo.next_image, cadence_flow_inc, + turbo.next.image = image_transform_optical_flow(turbo.next.image, cadence_flow_inc, step_init.cadence_flow_factor) - turbo.prev_frame_idx = turbo.next_frame_idx = indexes.tween_frame + turbo.prev.frame_index = turbo.next.frame_idx = idx.tween_frame - if turbo.prev_image is not None and tween < 1.0: - img = turbo.prev_image * (1.0 - tween) + turbo.next_image * tween + if turbo.prev.image is not None and tween < 1.0: + img = turbo.prev.image * (1.0 - tween) + turbo.next.image * tween else: - img = turbo.next_image + img = turbo.next.image # intercept and override to grayscale if init.args.anim_args.color_force_grayscale: @@ -237,7 +218,7 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): - img = do_overlay_mask(init.args.args, init.args.anim_args, img, indexes.tween_frame, True) + img = do_overlay_mask(init.args.args, init.args.anim_args, img, idx.tween_frame, True) # get images.previous during cadence images.previous = img @@ -246,20 +227,20 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{init.root.timestring}_{indexes.tween_frame:09}.png" + filename = f"{init.root.timestring}_{idx.tween_frame:09}.png" save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, - f"{init.root.timestring}_depth_{indexes.tween_frame:09}.png") + f"{init.root.timestring}_depth_{idx.tween_frame:09}.png") init.depth_model.save(dm_save_path, depth) # get color match for video outside of images.previous conditional if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(indexes.frame) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + if int(idx.frame) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(init.output_directory, 'inputframes', get_frame_name( - init.args.anim_args.video_init_path) + f"{indexes.frame:09}.jpg")) + init.args.anim_args.video_init_path) + f"{idx.frame:09}.jpg")) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) images.color_match = np.asarray(prev_vid_img) images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) @@ -267,33 +248,33 @@ def run_render_animation(init): # after 1st frame, images.previous exists if images.previous is not None: # apply transforms to previous frame - images.previous, depth = call_anim_frame_warp(init, indexes.frame, images.previous, None) + images.previous, depth = call_anim_frame_warp(init, idx.frame, images.previous, None) # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, indexes.frame, images.previous, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, idx.frame, images.previous, step_init.hybrid_comp_schedules) # hybrid video motion - warps images.previous to match motion, usually to prepare for compositing with context(init.args.anim_args) as aa: if aa.hybrid_motion in ['Affine', 'Perspective']: if aa.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.frame - 1, images.previous) + matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.frame - 1, images.previous) else: - matrix = call_get_matrix_for_hybrid_motion(init, indexes.frame - 1) + matrix = call_get_matrix_for_hybrid_motion(init, idx.frame - 1) images.previous = image_transform_ransac(images.previous, matrix, aa.hybrid_motion) if aa.hybrid_motion in ['Optical Flow']: if aa.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, indexes.frame - 1, images.previous) + flow = call_get_flow_for_hybrid_motion_prev(init, idx.frame - 1, images.previous) else: - flow = call_get_flow_for_hybrid_motion(init, indexes.frame - 1) + flow = call_get_flow_for_hybrid_motion(init, idx.frame - 1) images.previous = image_transform_optical_flow(images.previous, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, indexes.frame, images.previous, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, idx.frame, images.previous, step_init.hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): if images.color_match is None: @@ -341,41 +322,41 @@ def run_render_animation(init): # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints init.args.args.pix2pix_img_cfg_scale = float( - init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[indexes.frame]) + init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[idx.frame]) # grab prompt for current frame - init.args.args.prompt = init.prompt_series[indexes.frame] + init.args.args.prompt = init.prompt_series[idx.frame] with context(init.args) as ia: with context(init.animation_keys.deform_keys) as keys: # FIXME? check ia.args.seed_behavior if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - ia.args.seed = int(keys.seed_schedule_series[indexes.frame]) + ia.args.seed = int(keys.seed_schedule_series[idx.frame]) if ia.anim_args.enable_checkpoint_scheduling: - ia.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame] + ia.args.checkpoint = keys.checkpoint_schedule_series[idx.frame] else: ia.args.checkpoint = None # SubSeed scheduling if ia.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame]) + init.root.subseed = int(keys.subseed_schedule_series[idx.frame]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[idx.frame]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame] + init.root.subseed = int(keys.subseed_schedule_series[idx.frame]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[idx.frame] # set value back into the prompt - prepare and report prompt and seed - ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, indexes.frame) + ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, idx.frame) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, indexes.frame, ia.anim_args.video_init_path) + init_frame = call_get_next_frame(init, idx.frame, ia.anim_args.video_init_path) print(f"Using video init frame {init_frame}") ia.args.init_image = init_frame ia.args.init_image_box = None # init_image_box not used in this case ia.args.strength = max(0.0, min(1.0, step_init.strength)) if ia.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, indexes.frame, ia.anim_args.video_mask_path, True) + mask_init_frame = call_get_next_frame(init, idx.frame, ia.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) ia.args.mask_file = temp_mask init.root.noise_mask = temp_mask @@ -387,14 +368,10 @@ def run_render_animation(init): mask.vals, init.root.init_sample) \ if init.root.init_sample is not None else None # we need it only after the first frame anyway - init.animation_keys.update(indexes.frame) + init.animation_keys.update(idx.frame) _setup_opts(init, schedule) - if init.is_3d_with_med_or_low_vram(): - if init.animation_mode.is_predicting_depths: init.depth_model.to('cpu') - devices.torch_gc() - lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) - sd_hijack.model_hijack.hijack(sd_model) + MemoryUtils.handle_vram_if_depth_is_predicted(init) # TODO try init early, also see "call_get_flow_from_images" optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ @@ -407,8 +384,7 @@ def run_render_animation(init): msg_start = "Optical flow redo is diffusing and warping using" msg_end = "optical flow before generation." print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") - - with context(call_generate(init, indexes.frame, schedule)) as img: + with context(call_generate(init, idx.frame, schedule)) as img: img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) disposable_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) @@ -427,7 +403,7 @@ def run_render_animation(init): for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere - disposable_image = call_generate(init, indexes.frame, schedule) + disposable_image = call_generate(init, idx.frame, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(init.args.anim_args.diffusion_redo): @@ -439,21 +415,21 @@ def run_render_animation(init): gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation - image = call_generate(init, indexes.frame, schedule) + image = call_generate(init, idx.frame, schedule) if image is None: break # do hybrid video after generation - if indexes.frame > 0 and init.is_hybrid_composite_after_generation(): + if idx.frame > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = call_hybrid_composite(init, indexes.frame, temp_image, step_init.hybrid_comp_schedules) + _, temp_image_2 = call_hybrid_composite(init, idx.frame, temp_image, step_init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if indexes.frame == 0 and init.is_color_match_to_be_initialized(images.color_match): + if idx.frame == 0 and init.is_color_match_to_be_initialized(images.color_match): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) @@ -465,7 +441,7 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): - image = do_overlay_mask(init.args.args, init.args.anim_args, image, indexes.frame) + image = do_overlay_mask(init.args.args, init.args.anim_args, image, idx.frame) # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) @@ -477,39 +453,15 @@ def run_render_animation(init): if not init.animation_mode.has_video_input: images.previous = opencv_image - if turbo.steps > 1: - turbo.prev_image, turbo.prev_frame_idx = turbo.next_image, turbo.next_frame_idx - turbo.next_image, turbo.next_frame_idx = opencv_image, indexes.frame - indexes.frame += turbo.steps - else: - filename = f"{init.root.timestring}_{indexes.frame:09}.png" - save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) - - if init.args.anim_args.save_depth_maps: - # TODO move all depth related stuff to new class. (also see RenderInit) - if MemoryUtils.is_low_or_med_vram(): - lowvram.send_everything_to_cpu() - sd_hijack.model_hijack.undo_hijack(sd_model) - devices.torch_gc() - init.depth_model.to(init.root.device) - depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, - init.root.half_precision) - init.depth_model.save( - os.path.join(init.output_directory, f"{init.root.timestring}_depth_{indexes.frame:09}.png"), depth) - if MemoryUtils.is_low_or_med_vram(): - init.depth_model.to('cpu') - devices.torch_gc() - lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) - sd_hijack.model_hijack.hijack(sd_model) - indexes.frame += 1 + idx.frame, depth = progress_step(init, idx, turbo, opencv_image, image, depth) # TODO consider state is a global import, so wouldn't really need passing here state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal # FIXME? init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(init, indexes.frame, last_preview_frame) - _update_status_tracker(init.root, indexes.frame, init.args.anim_args) - init.animation_mode.cleanup() + last_preview_frame = call_render_preview(init, idx.frame, last_preview_frame) + WebUi.update_status_tracker(init, idx) + init.animation_mode.unload_raft_and_depth_model() def _setup_opts(init, schedule): @@ -523,25 +475,23 @@ def _setup_opts(init, schedule): put_if_present(data, "eta_ancestral", schedule.eta_ancestral) -WEB_UI_SLEEP_DELAY = 0.1 # TODO make a WebUi class? - - -def _init_web_ui_job(init): - state.job_count = init.args.anim_args.max_frames - - -def _update_web_ui_job(init, indexes): - frame = indexes.frame + 1 - max_frames = init.args.anim_args.max_frames - state.job = f"frame {frame}/{max_frames}" - state.job_no = frame + 1 - if state.skipped: - print("\n** PAUSED **") - state.skipped = False - while not state.skipped: - time.sleep(WEB_UI_SLEEP_DELAY) - print("** RESUMING **") - - -def _update_status_tracker(root, frame_idx, anim_args): - JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx / anim_args.max_frames) +def generate_depth_maps_if_active(init): + # TODO move all depth related stuff to new class. + if init.args.anim_args.save_depth_maps: + MemoryUtils.handle_vram_before_depth_map_generation(init) + depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.root.half_precision) + depth_filename = f"{init.root.timestring}_depth_{idx.frame:09}.png" + init.depth_model.save(os.path.join(init.output_directory, depth_filename), depth) + MemoryUtils.handle_vram_after_depth_map_generation(init) + return depth + + +def progress_step(init, idx, turbo, opencv_image, image, depth): + """Will progress frame or turbo-frame step and return next index and `depth`.""" + if turbo.has_steps(): + return idx.frame + turbo.progress_step(idx, opencv_image), depth + else: + filename = f"{init.root.timestring}_{idx.frame:09}.png" + save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) + depth = generate_depth_maps_if_active(init) + return idx.frame + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 1fa35e0fa..6dd8b6209 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -23,7 +23,7 @@ def is_predicting_depths(self) -> bool: def is_raft_active(self) -> bool: return self.raft_model is not None - def cleanup(self): + def unload_raft_and_depth_model(self): if self.is_predicting_depths() and not self.is_keep_in_vram: self.depth_model.delete_model() # handles adabins too if self.is_raft_active(): diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 3548d5aaf..dc31a9676 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -1,48 +1,65 @@ from dataclasses import dataclass -from typing import Any +from cv2.typing import MatLike from .subtitle import Srt -from ...resume import get_resume_vars +from ..util.call_utils import call_anim_frame_warp, call_get_resume_vars +from ..util.utils import context + + +@dataclass(init=True, frozen=False, repr=False, eq=False) +class ImageFrame: + image: MatLike + frame_index: int # TODO freeze.. @dataclass(frozen=False) class Turbo: steps: int - prev_image: Any - prev_frame_idx: int - next_image: Any - next_frame_idx: int + prev: ImageFrame + next: ImageFrame @staticmethod def create(init): steps = 1 if init.has_video_input() else init.cadence() - return Turbo(steps, None, 0, None, 0) + return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) + + def advance(self, init, indexes, depth): + with context(indexes.tween_frame) as i: + if self.is_advance_prev(i): + self.prev.image, _ = call_anim_frame_warp(init, i, self.prev.image, depth) + if self.is_advance_next(i): + self.next.image, _ = call_anim_frame_warp(init, i, self.next.image, depth) + + def progress_step(self, indexes, opencv_image): + self.prev.image, self.prev.frame_index = self.next.image, self.next.frame_index + self.next.image, self.next.frame_index = opencv_image, indexes.frame + return self.steps def _set_up_step_vars(self, init, turbo): # determine last frame and frame to start on - prev_frame, next_frame, prev_img, next_img = get_resume_vars( - folder=init.args.args.outdir, - timestring=init.args.anim_args.resume_timestring, - cadence=turbo.steps) + prev_frame, next_frame, prev_img, next_img = call_get_resume_vars(init, turbo) if self.steps > 1: - self.prev_image, self.prev_frame_idx = prev_img, prev_frame - self.next_image, self.next_frame_idx = next_img, next_frame - return next_frame + self.prev.image, self.prev.frame_index = prev_img, prev_frame + self.next.image, self.next.frame_index = next_img, next_frame def find_start(self, init, turbo): """Maybe resume animation (requires at least two frames - see function).""" # set start_frame to next frame - return self._set_up_step_vars(init, turbo) + 1 if init.is_resuming_from_timestring() else 0 + self._set_up_step_vars(init, turbo) + return self.next.frame_index + 1 if init.is_resuming_from_timestring() else 0 + + def has_steps(self): + return self.steps > 1 def _has_prev_image(self): - return self.prev_image is not None + return self.prev.image is not None def is_advance_prev(self, i) -> bool: - return self._has_prev_image() and i > self.prev_frame_idx + return self._has_prev_image() and i > self.prev.frame_index def is_advance_next(self, i) -> bool: - return i > self.next_frame_idx + return i > self.next.frame_index def is_first_step(self) -> bool: return self.steps == 1 diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py index d08157c86..43bd1de17 100644 --- a/scripts/deforum_helpers/rendering/util/call_utils.py +++ b/scripts/deforum_helpers/rendering/util/call_utils.py @@ -14,6 +14,7 @@ # Other hybrid functions hybrid_composite) from ...load_images import get_mask_from_file +from ...resume import get_resume_vars from ...subtitle_handler import format_animation_params, write_frame_subtitle from ...video_audio_utilities import get_next_frame, render_preview @@ -113,6 +114,13 @@ def call_get_mask_from_file_with_frame(init, frame): return get_mask_from_file(frame, init.args.args) +# Resume +def call_get_resume_vars(init, turbo): + return get_resume_vars(folder=init.args.args.outdir, + timestring=init.args.anim_args.resume_timestring, + cadence=turbo.steps) + + # Subtitle def call_format_animation_params(init, i, params_to_print): return format_animation_params(init.animation_keys.deform_keys, init.prompt_series, i, params_to_print) diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 4d9d01028..2139e7074 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -1,3 +1,4 @@ +from modules import lowvram, devices, sd_hijack from modules.shared import cmd_opts # keep readonly @@ -10,6 +11,41 @@ def is_low_or_med_vram(): # Perhaps add a constant bool to RenderInit. return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly + @staticmethod + def handle_med_or_low_vram_before_step(init): + if init.is_3d_with_med_or_low_vram(): + # Unload the main checkpoint and load the depth model + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() + if init.animation_mode.is_predicting_depths: + init.animation_mode.depth_model.to(init.root.device) + + @staticmethod + def handle_vram_if_depth_is_predicted(init): + if init.animation_mode.is_predicting_depths: + if init.is_3d_with_med_or_low_vram(): + init.depth_model.to('cpu') + devices.torch_gc() + lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) + sd_hijack.model_hijack.hijack(sd_model) + + @staticmethod + def handle_vram_before_depth_map_generation(init): + if MemoryUtils.is_low_or_med_vram(): + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() + init.depth_model.to(init.root.device) + + @staticmethod + def handle_vram_after_depth_map_generation(init): + if MemoryUtils.is_low_or_med_vram(): + init.depth_model.to('cpu') + devices.torch_gc() + lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) + sd_hijack.model_hijack.hijack(sd_model) + @staticmethod def select_depth_device(root): return 'cpu' if MemoryUtils.is_low_or_med_vram() else root.device diff --git a/scripts/deforum_helpers/rendering/util/web_ui.py b/scripts/deforum_helpers/rendering/util/web_ui.py new file mode 100644 index 000000000..45f7ed3a0 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/web_ui.py @@ -0,0 +1,33 @@ +from deforum_api import JobStatusTracker +from modules.shared import state + + +class WebUi: + WEB_UI_SLEEP_DELAY = 0.1 + + @staticmethod + def init_job(init): + state.job_count = init.args.anim_args.max_frames + + @staticmethod + def update_job(init, indexes): + frame = indexes.frame + 1 + max_frames = init.args.anim_args.max_frames + state.job = f"frame {frame}/{max_frames}" + state.job_no = frame + 1 + if state.skipped: + print("\n** PAUSED **") + state.skipped = False + while not state.skipped: + time.sleep(WEB_UI_SLEEP_DELAY) + print("** RESUMING **") + + @staticmethod + def update_status_tracker(init, indexes): + progress = indexes.frame / init.args.anim_args.max_frames + JobStatusTracker().update_phase(init.root.job_id, phase="GENERATING", progress=progress) + + @staticmethod + def update_progress_during_cadence(init, indexes): + state.job = f"frame {indexes.tween_frame + 1}/{init.args.anim_args.max_frames}" + state.job_no = indexes.tween_frame + 1 From 2a048a0387da75fffba49c3eccf64ffea2a72d76 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 8 Jun 2024 23:40:33 +0200 Subject: [PATCH 043/132] Index recontextualization. Calling concern separation. Utilities untied from static context and opt_utils added. --- scripts/deforum_helpers/render.py | 232 ++++++++---------- .../deforum_helpers/rendering/data/indexes.py | 17 +- .../deforum_helpers/rendering/data/mask.py | 4 +- .../rendering/data/schedule.py | 21 +- .../deforum_helpers/rendering/data/turbo.py | 40 +-- .../rendering/initialization.py | 9 +- .../rendering/util/__init__.py | 10 +- .../rendering/util/call/__init__.py | 24 ++ .../rendering/util/call/anim.py | 8 + .../rendering/util/call/gen.py | 8 + .../rendering/util/call/hybrid.py | 58 +++++ .../rendering/util/call/images.py | 10 + .../rendering/util/call/resume.py | 7 + .../rendering/util/call/subtitle.py | 10 + .../rendering/util/call/video_and_audio.py | 11 + .../rendering/util/call_utils.py | 141 ----------- .../rendering/util/memory_utils.py | 74 +++--- .../rendering/util/opt_utils.py | 20 ++ .../deforum_helpers/rendering/util/web_ui.py | 33 --- .../rendering/util/web_ui_utils.py | 34 +++ 20 files changed, 380 insertions(+), 391 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/call/__init__.py create mode 100644 scripts/deforum_helpers/rendering/util/call/anim.py create mode 100644 scripts/deforum_helpers/rendering/util/call/gen.py create mode 100644 scripts/deforum_helpers/rendering/util/call/hybrid.py create mode 100644 scripts/deforum_helpers/rendering/util/call/images.py create mode 100644 scripts/deforum_helpers/rendering/util/call/resume.py create mode 100644 scripts/deforum_helpers/rendering/util/call/subtitle.py create mode 100644 scripts/deforum_helpers/rendering/util/call/video_and_audio.py delete mode 100644 scripts/deforum_helpers/rendering/util/call_utils.py create mode 100644 scripts/deforum_helpers/rendering/util/opt_utils.py delete mode 100644 scripts/deforum_helpers/rendering/util/web_ui.py create mode 100644 scripts/deforum_helpers/rendering/util/web_ui_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 3a4fe7578..d8b61cff8 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -17,11 +17,14 @@ import gc import os import random + import PIL import cv2 import numpy as np from PIL import Image, ImageOps -from modules.shared import opts, cmd_opts, state, sd_model +# noinspection PyUnresolvedReferences +from modules.shared import opts, state + from .colors import maintain_colors from .composable_masks import compose_mask_with_check from .hybrid_video import (image_transform_ransac, image_transform_optical_flow, @@ -30,44 +33,27 @@ from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt -from .rendering.data import (Turbo, Schedule, Images, Indexes, Mask) +from .rendering.data import Turbo, Schedule, Images, Indexes, Mask from .rendering.initialization import RenderInit, StepInit -from .rendering.util import put_if_present -from .rendering.util.call_utils import ( - # Animation Functions - call_anim_frame_warp, - call_format_animation_params, - call_get_next_frame, - call_write_frame_subtitle, - - # Generation and Rendering - call_generate, - call_render_preview, - - # Hybrid Motion Functions - call_get_flow_for_hybrid_motion, - call_get_flow_for_hybrid_motion_prev, - call_get_matrix_for_hybrid_motion, - call_get_matrix_for_hybrid_motion_prev, - call_hybrid_composite, - - # Mask Functions - call_get_mask_from_file_with_frame, - - # Flow Functions - call_get_flow_from_images) -from .rendering.util.memory_utils import MemoryUtils +from .rendering.util import opt_utils, web_ui_utils, memory_utils +from .rendering.util.call.anim import call_anim_frame_warp +from .rendering.util.call.gen import call_generate +from .rendering.util.call.hybrid import ( + call_get_flow_from_images, call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, + call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) +from .rendering.util.call.images import call_get_mask_from_file_with_frame +from .rendering.util.call.subtitle import call_format_animation_params, call_write_frame_subtitle +from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame from .rendering.util.utils import context -from .rendering.util.web_ui import WebUi from .save_images import save_image from .seed import next_seed from .video_audio_utilities import get_frame_name def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - render_init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) # TODO method is temporarily torn apart to remove args from direct access in larger execution scope. - run_render_animation(render_init) + run_render_animation(init) def run_render_animation_controlled(init): @@ -89,58 +75,56 @@ def run_render_animation(init): images = Images.create(init) turbo = Turbo.create(init) # state for interpolating between diffusion steps idx = Indexes.create(init, turbo) - mask = Mask.create(init, idx.frame) # reset the mask vals as they are overwritten in the compose_mask algorithm + mask = Mask.create(init, idx.frame.i) # reset the mask vals as they are overwritten in the compose_mask algorithm - WebUi.init_job(init) + web_ui_utils.init_job(init) last_preview_frame = 0 - while idx.frame < init.args.anim_args.max_frames: - WebUi.update_job(init, idx) - print(f"\033[36mAnimation frame: \033[0m{idx.frame}/{init.args.anim_args.max_frames} ") + while idx.frame.i < init.args.anim_args.max_frames: + web_ui_utils.update_job(init, idx) + print(f"\033[36mAnimation frame: \033[0m{idx.frame.i}/{init.args.anim_args.max_frames} ") # TODO Stuff to be moved into new Step class in `run_render_animation_controlled`: - step_init = StepInit.create(init.animation_keys.deform_keys, idx.frame) - schedule = Schedule.create(init.animation_keys.deform_keys, idx.frame, init.args.anim_args, init.args.args) + step_init = StepInit.create(init.animation_keys.deform_keys, idx.frame.i) + schedule = Schedule.create(init.animation_keys.deform_keys, idx.frame.i, init.args.anim_args, init.args.args) if init.is_use_mask and not init.args.anim_args.use_noise_mask: noise_mask_seq = schedule.mask_seq depth = None - MemoryUtils.handle_med_or_low_vram_before_step(init) + memory_utils.handle_med_or_low_vram_before_step(init) if turbo.is_first_step_with_subtitles(init): - params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, idx.frame, params_to_print) - call_write_frame_subtitle(init, idx.frame, params_string) - params_string = None # FIXME ?? + params_to_print = opt_utils.generation_info_for_subtitles(init) + params_string = call_format_animation_params(init, idx.frame.i, params_to_print) + call_write_frame_subtitle(init, idx.frame.i, params_string) if turbo.is_emit_in_between_frames(): - idx.tween_frame_start = max(idx.start_frame, idx.frame - turbo.steps) + idx.tween.start = max(idx.frame.start, idx.frame.i - turbo.steps) cadence_flow = None # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` - for idx.tween_frame in range(idx.tween_frame_start, idx.frame): - WebUi.update_progress_during_cadence(init, idx) + for idx.tween.i in range(idx.tween.start, idx.frame.i): + web_ui_utils.update_progress_during_cadence(init, idx) # cadence vars - tween = (float(idx.tween_frame - idx.tween_frame_start + 1) / - float(idx.frame - idx.tween_frame_start)) + tween = (float(idx.tween.i - idx.tween.start + 1) / + float(idx.frame.i - idx.tween.start)) # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): - if init.animation_keys.deform_keys.strength_schedule_series[idx.tween_frame_start] > 0: + if init.animation_keys.deform_keys.strength_schedule_series[idx.tween.start.i] > 0: if cadence_flow is None and turbo.prev.image is not None and turbo.next.image is not None: cadence_flow = call_get_flow_from_images(init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2 turbo.next.image = image_transform_optical_flow(turbo.next.image, -cadence_flow, 1) - if opts.data.get("deforum_save_gen_info_as_srt"): - params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) - params_string = call_format_animation_params(init, idx.tween_frame, params_to_print) - call_write_frame_subtitle(init, idx.tween_frame, params_string, tween < 1.0) - params_string = None + if opt_utils.is_generate_subtitles(init): + params_to_print = opt_utils.generation_info_for_subtitles(init) + params_string = call_format_animation_params(init, idx.tween.i, params_to_print) + call_write_frame_subtitle(init, idx.tween.i, params_string, tween < 1.0) msg_flow_name = '' if cadence_flow is None \ else init.args.anim_args.optical_flow_cadence + ' optical flow ' - msg_frame_info = f"cadence frame: {idx.tween_frame}; tween:{tween:0.2f};" + msg_frame_info = f"cadence frame: {idx.tween.i}; tween:{tween:0.2f};" print(f"Creating in-between {msg_flow_name}{msg_frame_info}") if init.depth_model is not None: @@ -149,43 +133,43 @@ def run_render_animation(init): init.args.anim_args.midas_weight, init.root.half_precision) - turbo.advance(init, idx, depth) + turbo.advance(init, idx.tween.i, depth) # hybrid video motion - warps turbo.prev.image or turbo.next.image to match motion - if idx.tween_frame > 0: + if idx.tween.i > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.tween_frame - 1, images.previous) - if turbo.is_advance_prev(idx.tween_frame): + matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.tween.i - 1, images.previous) + if turbo.is_advance_prev(idx.tween.i): turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(idx.tween_frame): + if turbo.is_advance_next(idx.tween.i): turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = call_get_matrix_for_hybrid_motion(init, idx.tween_frame - 1) - if turbo.is_advance_prev(idx.tween_frame): + matrix = call_get_matrix_for_hybrid_motion(init, idx.tween.i - 1) + if turbo.is_advance_prev(idx.tween.i): turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(idx.tween_frame): + if turbo.is_advance_next(idx.tween.i): turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, idx.tween_frame - 1, images.previous) - if turbo.is_advance_prev(idx.tween_frame): + flow = call_get_flow_for_hybrid_motion_prev(init, idx.tween.i - 1, images.previous) + if turbo.is_advance_prev(idx.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, step_init.flow_factor()) - if turbo.is_advance_next(idx.tween_frame): + if turbo.is_advance_next(idx.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow else: - flow = call_get_flow_for_hybrid_motion(init, idx.tween_frame - 1) - if turbo.is_advance_prev(idx.tween_frame): + flow = call_get_flow_for_hybrid_motion(init, idx.tween.i - 1) + if turbo.is_advance_prev(idx.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, step_init.flow_factor()) - if turbo.is_advance_next(idx.tween_frame): + if turbo.is_advance_next(idx.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow @@ -195,16 +179,16 @@ def run_render_animation(init): with context(cadence_flow) as cf: if cf is not None: cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) - cf, _ = call_anim_frame_warp(init, idx.tween_frame, cf, depth) + cf, _ = call_anim_frame_warp(init, idx.tween.i, cf, depth) cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * tween - if turbo.is_advance_prev(): + if turbo.is_advance_prev(idx.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, cadence_flow_inc, step_init.cadence_flow_factor) - if turbo.is_advance_next(): + if turbo.is_advance_next(idx.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, cadence_flow_inc, step_init.cadence_flow_factor) - turbo.prev.frame_index = turbo.next.frame_idx = idx.tween_frame + turbo.prev.index = turbo.next.frame_idx = idx.tween.i if turbo.prev.image is not None and tween < 1.0: img = turbo.prev.image * (1.0 - tween) + turbo.next.image * tween @@ -218,7 +202,7 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): - img = do_overlay_mask(init.args.args, init.args.anim_args, img, idx.tween_frame, True) + img = do_overlay_mask(init.args.args, init.args.anim_args, img, idx.tween.i, True) # get images.previous during cadence images.previous = img @@ -227,20 +211,20 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{init.root.timestring}_{idx.tween_frame:09}.png" + filename = f"{init.root.timestring}_{idx.tween.i:09}.png" save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, - f"{init.root.timestring}_depth_{idx.tween_frame:09}.png") + f"{init.root.timestring}_depth_{idx.tween.i:09}.png") init.depth_model.save(dm_save_path, depth) # get color match for video outside of images.previous conditional if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(idx.frame) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + if int(idx.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(os.path.join(init.output_directory, 'inputframes', get_frame_name( - init.args.anim_args.video_init_path) + f"{idx.frame:09}.jpg")) + init.args.anim_args.video_init_path) + f"{idx.frame.i:09}.jpg")) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) images.color_match = np.asarray(prev_vid_img) images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) @@ -248,39 +232,42 @@ def run_render_animation(init): # after 1st frame, images.previous exists if images.previous is not None: # apply transforms to previous frame - images.previous, depth = call_anim_frame_warp(init, idx.frame, images.previous, None) + images.previous, depth = call_anim_frame_warp(init, idx.frame.i, images.previous, None) # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, idx.frame, images.previous, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, idx.frame.i, images.previous, + step_init.hybrid_comp_schedules) # hybrid video motion - warps images.previous to match motion, usually to prepare for compositing with context(init.args.anim_args) as aa: if aa.hybrid_motion in ['Affine', 'Perspective']: if aa.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.frame - 1, images.previous) + matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.frame.i - 1, images.previous) else: - matrix = call_get_matrix_for_hybrid_motion(init, idx.frame - 1) + matrix = call_get_matrix_for_hybrid_motion(init, idx.frame.i - 1) images.previous = image_transform_ransac(images.previous, matrix, aa.hybrid_motion) if aa.hybrid_motion in ['Optical Flow']: if aa.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, idx.frame - 1, images.previous) + flow = call_get_flow_for_hybrid_motion_prev(init, idx.frame.i - 1, images.previous) else: - flow = call_get_flow_for_hybrid_motion(init, idx.frame - 1) + flow = call_get_flow_for_hybrid_motion(init, idx.frame.i - 1) images.previous = image_transform_optical_flow(images.previous, flow, step_init.flow_factor()) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, idx.frame, images.previous, step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, idx.frame.i, images.previous, + step_init.hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): if images.color_match is None: images.color_match = images.previous.copy() else: - images.previous = maintain_colors(images.previous, images.color_match, init.args.anim_args.color_coherence) + images.previous = maintain_colors(images.previous, images.color_match, + init.args.anim_args.color_coherence) # intercept and override to grayscale if init.args.anim_args.color_force_grayscale: @@ -322,41 +309,41 @@ def run_render_animation(init): # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints init.args.args.pix2pix_img_cfg_scale = float( - init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[idx.frame]) + init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[idx.frame.i]) # grab prompt for current frame - init.args.args.prompt = init.prompt_series[idx.frame] + init.args.args.prompt = init.prompt_series[idx.frame.i] with context(init.args) as ia: with context(init.animation_keys.deform_keys) as keys: # FIXME? check ia.args.seed_behavior if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - ia.args.seed = int(keys.seed_schedule_series[idx.frame]) + ia.args.seed = int(keys.seed_schedule_series[idx.frame.i]) # TODO recontextualize frame index if ia.anim_args.enable_checkpoint_scheduling: - ia.args.checkpoint = keys.checkpoint_schedule_series[idx.frame] + ia.args.checkpoint = keys.checkpoint_schedule_series[idx.frame.i] else: ia.args.checkpoint = None # SubSeed scheduling if ia.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[idx.frame]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[idx.frame]) + init.root.subseed = int(keys.subseed_schedule_series[idx.frame.i]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[idx.frame.i]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[idx.frame]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[idx.frame] + init.root.subseed = int(keys.subseed_schedule_series[idx.frame.i]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[idx.frame.i] # set value back into the prompt - prepare and report prompt and seed - ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, idx.frame) + ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, idx.frame.i) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, idx.frame, ia.anim_args.video_init_path) + init_frame = call_get_next_frame(init, idx.frame.i, ia.anim_args.video_init_path) print(f"Using video init frame {init_frame}") ia.args.init_image = init_frame ia.args.init_image_box = None # init_image_box not used in this case ia.args.strength = max(0.0, min(1.0, step_init.strength)) if ia.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, idx.frame, ia.anim_args.video_mask_path, True) + mask_init_frame = call_get_next_frame(init, idx.frame.i, ia.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) ia.args.mask_file = temp_mask init.root.noise_mask = temp_mask @@ -368,10 +355,10 @@ def run_render_animation(init): mask.vals, init.root.init_sample) \ if init.root.init_sample is not None else None # we need it only after the first frame anyway - init.animation_keys.update(idx.frame) - _setup_opts(init, schedule) + init.animation_keys.update(idx.frame.i) + opt_utils.setup(init, schedule) - MemoryUtils.handle_vram_if_depth_is_predicted(init) + memory_utils.handle_vram_if_depth_is_predicted(init) # TODO try init early, also see "call_get_flow_from_images" optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ @@ -384,7 +371,7 @@ def run_render_animation(init): msg_start = "Optical flow redo is diffusing and warping using" msg_end = "optical flow before generation." print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") - with context(call_generate(init, idx.frame, schedule)) as img: + with context(call_generate(init, idx.frame.i, schedule)) as img: img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) disposable_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) @@ -403,7 +390,7 @@ def run_render_animation(init): for n in range(0, int(init.args.anim_args.diffusion_redo)): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere - disposable_image = call_generate(init, idx.frame, schedule) + disposable_image = call_generate(init, idx.frame.i, schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(init.args.anim_args.diffusion_redo): @@ -415,21 +402,21 @@ def run_render_animation(init): gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation - image = call_generate(init, idx.frame, schedule) + image = call_generate(init, idx.frame.i, schedule) if image is None: break # do hybrid video after generation - if idx.frame > 0 and init.is_hybrid_composite_after_generation(): + if idx.frame.i > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = call_hybrid_composite(init, idx.frame, temp_image, step_init.hybrid_comp_schedules) + _, temp_image_2 = call_hybrid_composite(init, idx.frame.i, temp_image, step_init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if idx.frame == 0 and init.is_color_match_to_be_initialized(images.color_match): + if idx.frame.i == 0 and init.is_color_match_to_be_initialized(images.color_match): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) @@ -441,7 +428,7 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): - image = do_overlay_mask(init.args.args, init.args.anim_args, image, idx.frame) + image = do_overlay_mask(init.args.args, init.args.anim_args, image, idx.frame.i) # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) @@ -453,45 +440,32 @@ def run_render_animation(init): if not init.animation_mode.has_video_input: images.previous = opencv_image - idx.frame, depth = progress_step(init, idx, turbo, opencv_image, image, depth) - - # TODO consider state is a global import, so wouldn't really need passing here + idx.frame.i, depth = progress_step(init, idx, turbo, opencv_image, image, depth) state.assign_current_image(image) - # may reassign init.args.args and/or root.seed_internal # FIXME? + # may reassign init.args.args and/or root.seed_internal init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(init, idx.frame, last_preview_frame) - WebUi.update_status_tracker(init, idx) + last_preview_frame = call_render_preview(init, idx.frame.i, last_preview_frame) + web_ui_utils.update_status_tracker(init, idx) init.animation_mode.unload_raft_and_depth_model() -def _setup_opts(init, schedule): - data = init.args.opts.data - if init.has_img2img_fix_steps(): - # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations - data["img2img_fix_steps"] = False - put_if_present(data, "CLIP_stop_at_last_layers", schedule.clipskip) - put_if_present(data, "initial_noise_multiplier", schedule.noise_multiplier) - put_if_present(data, "eta_ddim", schedule.eta_ddim) - put_if_present(data, "eta_ancestral", schedule.eta_ancestral) - - def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: - MemoryUtils.handle_vram_before_depth_map_generation(init) + memory_utils.handle_vram_before_depth_map_generation(init) depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.root.half_precision) - depth_filename = f"{init.root.timestring}_depth_{idx.frame:09}.png" + depth_filename = f"{init.root.timestring}_depth_{idx.frame.i:09}.png" init.depth_model.save(os.path.join(init.output_directory, depth_filename), depth) - MemoryUtils.handle_vram_after_depth_map_generation(init) + memory_utils.handle_vram_after_depth_map_generation(init) return depth def progress_step(init, idx, turbo, opencv_image, image, depth): """Will progress frame or turbo-frame step and return next index and `depth`.""" if turbo.has_steps(): - return idx.frame + turbo.progress_step(idx, opencv_image), depth + return idx.frame.i + turbo.progress_step(idx, opencv_image), depth else: - filename = f"{init.root.timestring}_{idx.frame:09}.png" + filename = f"{init.root.timestring}_{idx.frame.i:09}.png" save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) depth = generate_depth_maps_if_active(init) - return idx.frame + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. + return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index 896916342..b97210842 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -1,14 +1,19 @@ from dataclasses import dataclass +@dataclass(init=True, frozen=False, repr=True, eq=True) +class IndexWithStart: + start: int = 0 + i: int = 0 + + @dataclass(init=True, frozen=False, repr=False, eq=False) class Indexes: - frame: int = 0 - start_frame: int = 0 - tween_frame: int = 0 - tween_frame_start: int = 0 + frame: IndexWithStart = None + tween: IndexWithStart = None @staticmethod def create(init, turbo): - # TODO try to init everything right away... - return Indexes(0, turbo.find_start(init, turbo), 0, 0) + frame = IndexWithStart(turbo.find_start(init, turbo), 0) + tween = IndexWithStart(0, 0) + return Indexes(frame, tween) diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index ae2909276..62e578f1a 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -6,7 +6,7 @@ from ..util import put_all from ..util.utils import create_img, call_or_use_on_cond, context from ...load_images import get_mask, load_img -from ...rendering.util.call_utils import call_get_mask_from_file +from ...rendering.util.call.images import call_get_mask_from_file # TODO freeze? @@ -49,7 +49,7 @@ def _assign(init, i, is_mask_image, dicts): init.root.noise_mask = mask put_all(dicts, key, mask) elif is_mask_image is None and init.is_use_mask: - put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask + put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noise mask @staticmethod def _create_mask_image(init): diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index f05368096..d398d9f92 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,8 +1,6 @@ from dataclasses import dataclass from typing import Optional, Any -from ..util.utils import context - @dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: @@ -19,16 +17,15 @@ class Schedule: def create(keys, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" - with context(Schedule) as S: - steps = S.schedule_steps(keys, i, anim_args) - sampler_name = S.schedule_sampler(keys, i, anim_args) - clipskip = S.schedule_clipskip(keys, i, anim_args) - noise_multiplier = S.schedule_noise_multiplier(keys, i, anim_args) - eta_ddim = S.schedule_ddim_eta(keys, i, anim_args) - eta_ancestral = S.schedule_ancestral_eta(keys, i, anim_args) - mask = S.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args - noise_mask = S.schedule_noise_mask(keys, i, anim_args) - return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) + steps = Schedule.schedule_steps(keys, i, anim_args) + sampler_name = Schedule.schedule_sampler(keys, i, anim_args) + clipskip = Schedule.schedule_clipskip(keys, i, anim_args) + noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) + mask = Schedule.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args + noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index dc31a9676..34627f8e5 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -1,15 +1,16 @@ from dataclasses import dataclass from cv2.typing import MatLike + from .subtitle import Srt -from ..util.call_utils import call_anim_frame_warp, call_get_resume_vars -from ..util.utils import context +from ..util.call.anim import call_anim_frame_warp +from ..util.call.resume import call_get_resume_vars -@dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=True) class ImageFrame: image: MatLike - frame_index: int + index: int # TODO freeze.. @@ -24,30 +25,29 @@ def create(init): steps = 1 if init.has_video_input() else init.cadence() return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) - def advance(self, init, indexes, depth): - with context(indexes.tween_frame) as i: - if self.is_advance_prev(i): - self.prev.image, _ = call_anim_frame_warp(init, i, self.prev.image, depth) - if self.is_advance_next(i): - self.next.image, _ = call_anim_frame_warp(init, i, self.next.image, depth) + def advance(self, init, i: int, depth): + if self.is_advance_prev(i): + self.prev.image, _ = call_anim_frame_warp(init, i, self.prev.image, depth) + if self.is_advance_next(i): + self.next.image, _ = call_anim_frame_warp(init, i, self.next.image, depth) def progress_step(self, indexes, opencv_image): - self.prev.image, self.prev.frame_index = self.next.image, self.next.frame_index - self.next.image, self.next.frame_index = opencv_image, indexes.frame + self.prev.image, self.prev.index = self.next.image, self.next.index + self.next.image, self.next.index = opencv_image, indexes.frame.i return self.steps def _set_up_step_vars(self, init, turbo): # determine last frame and frame to start on prev_frame, next_frame, prev_img, next_img = call_get_resume_vars(init, turbo) if self.steps > 1: - self.prev.image, self.prev.frame_index = prev_img, prev_frame - self.next.image, self.next.frame_index = next_img, next_frame + self.prev.image, self.prev.index = prev_img, prev_frame + self.next.image, self.next.index = next_img, next_frame - def find_start(self, init, turbo): + def find_start(self, init, turbo) -> int: """Maybe resume animation (requires at least two frames - see function).""" # set start_frame to next frame self._set_up_step_vars(init, turbo) - return self.next.frame_index + 1 if init.is_resuming_from_timestring() else 0 + return self.next.index + 1 if init.is_resuming_from_timestring() else 0 def has_steps(self): return self.steps > 1 @@ -55,11 +55,11 @@ def has_steps(self): def _has_prev_image(self): return self.prev.image is not None - def is_advance_prev(self, i) -> bool: - return self._has_prev_image() and i > self.prev.frame_index + def is_advance_prev(self, i: int) -> bool: + return self._has_prev_image() and i > self.prev.index - def is_advance_next(self, i) -> bool: - return i > self.next.frame_index + def is_advance_next(self, i: int) -> bool: + return i > self.next.index def is_first_step(self) -> bool: return self.steps == 1 diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/initialization.py index e1a6f6e4a..88251dbf8 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/initialization.py @@ -5,12 +5,11 @@ import numexpr import numpy as np import pandas as pd -from PIL import Image from .data.anim import AnimationKeys, AnimationMode from .data.subtitle import Srt -from .util import MemoryUtils -from .util.utils import context, put_all, create_img +from .util import memory_utils +from .util.utils import context from ..args import RootArgs from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ..depth import DepthModel @@ -100,7 +99,7 @@ def is_3d(self): return self.args.anim_args.animation_mode == '3D' def is_3d_with_med_or_low_vram(self): - return self.is_3d() and MemoryUtils.is_low_or_med_vram() + return self.is_3d() and memory_utils.is_low_or_med_vram() def width(self) -> int: return self.args.args.W @@ -208,7 +207,7 @@ def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, an anim_args.save_depth_maps = (anim_mode.is_predicting_depths and RenderInit.is_composite_with_depth_mask(anim_args)) return DepthModel(root.models_path, - MemoryUtils.select_depth_device(root), + memory_utils.select_depth_device(root), root.half_precision, keep_in_vram=anim_mode.is_keep_in_vram, depth_algorithm=anim_args.depth_algorithm, diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 6708474e9..76b06411d 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,8 +1,8 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .call_utils import (call_anim_frame_warp, call_generate, call_get_flow_from_images, call_hybrid_composite, - call_render_preview, call_format_animation_params, call_get_next_frame, - call_get_matrix_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, - call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion) -from .memory_utils import MemoryUtils +from .memory_utils import (is_low_or_med_vram, handle_med_or_low_vram_before_step, handle_vram_if_depth_is_predicted, + handle_vram_before_depth_map_generation, handle_vram_after_depth_map_generation, + select_depth_device) +from .opt_utils import setup, generation_info_for_subtitles, is_generate_subtitles from .utils import put_all, put_if_present, call_or_use_on_cond +from .web_ui_utils import init_job, update_job, update_status_tracker, update_progress_during_cadence diff --git a/scripts/deforum_helpers/rendering/util/call/__init__.py b/scripts/deforum_helpers/rendering/util/call/__init__.py new file mode 100644 index 000000000..0f3ca1733 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/__init__.py @@ -0,0 +1,24 @@ +""" +This module provides utility functions for simplifying calls to other modules within the `render.py` module. + +**Purpose:** +- **Reduce Argument Complexity:** Provides a way to call functions in other modules without directly handling + a large number of complex arguments. This simplifies code within `render.py` by encapsulating argument management. +- **Minimize Namespace Pollution:** Provides an alternative to overloading methods in the original modules, + which would introduce the `RenderInit` class into namespaces where it's not inherently needed. + +**Structure:** +- **Simple Call Forwarding:** Functions in this module primarily act as wrappers. They perform minimal logic, + often just formatting or passing arguments, and directly call the corresponding method. +- **Naming Convention:** + - Function names begin with "call_", followed by the name of the actual method to call. + - The `init` object is always passed as the first argument. + - Frame indices (e.g., `frame_idx`, `twin_frame_idx`) are passed as the second argument "i", when relevant. + +**Example:** +```python +# Example function in this module +def call_some_function(init, i, ...): + return some_module.some_function(init.arg77, init.arg.arg.whatever, i, ...) +``` +""" diff --git a/scripts/deforum_helpers/rendering/util/call/anim.py b/scripts/deforum_helpers/rendering/util/call/anim.py new file mode 100644 index 000000000..42fcb7e21 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/anim.py @@ -0,0 +1,8 @@ +from ..utils import context +from ....animation import anim_frame_warp + + +def call_anim_frame_warp(init, i, image, depth): + with context(init.args) as ia: + return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, + depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) diff --git a/scripts/deforum_helpers/rendering/util/call/gen.py b/scripts/deforum_helpers/rendering/util/call/gen.py new file mode 100644 index 000000000..721f41e7a --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/gen.py @@ -0,0 +1,8 @@ +from ..utils import context +from ....generate import generate + + +def call_generate(init, i, schedule): + with context(init.args) as ia: + return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, + ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) diff --git a/scripts/deforum_helpers/rendering/util/call/hybrid.py b/scripts/deforum_helpers/rendering/util/call/hybrid.py new file mode 100644 index 000000000..57f2bbcbb --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/hybrid.py @@ -0,0 +1,58 @@ +from ..utils import context +from ....hybrid_video import ( + # Functions related to flow calculation + get_flow_from_images, + get_flow_for_hybrid_motion, + get_flow_for_hybrid_motion_prev, + + # Functions related to matrix calculation + get_matrix_for_hybrid_motion, + get_matrix_for_hybrid_motion_prev, + + # Other hybrid functions + hybrid_composite) + + +def call_get_flow_from_images(init, prev_image, next_image, cadence): + # cadence is currently either "optical_flow_redo_generation" or "init.args.anim_args.optical_flow_cadence" + # TODO try to init "optical_flow_redo_generation" early, then remove the "cadence" arg again + return get_flow_from_images(prev_image, next_image, cadence, init.animation_mode.raft_model) + + +def call_get_flow_for_hybrid_motion_prev(init, i, image): + with context(init.animation_mode) as mode: + with context(init.args.anim_args) as aa: + return get_flow_for_hybrid_motion_prev(i, init.dimensions(), + mode.hybrid_input_files, + mode.hybrid_frame_path, + mode.prev_flow, + image, + aa.hybrid_flow_method, + mode.raft_model, + aa.hybrid_flow_consistency, + aa.hybrid_consistency_blur, + aa.hybrid_comp_save_extra_frames) + + +def call_get_flow_for_hybrid_motion(init, i): + with context(init.animation_mode) as mode: + with context(init.args.anim_args) as args: + return get_flow_for_hybrid_motion(i, init.dimensions(), mode.hybrid_input_files, mode.hybrid_frame_path, + mode.prev_flow, args.hybrid_flow_method, mode.raft_model, + args.hybrid_flow_consistency, args.hybrid_consistency_blur, args) + + +def call_get_matrix_for_hybrid_motion_prev(init, i, image): + return get_matrix_for_hybrid_motion_prev(i, init.dimensions(), init.animation_mode.hybrid_input_files, + image, init.args.anim_args.hybrid_motion) + + +def call_get_matrix_for_hybrid_motion(init, i): + return get_matrix_for_hybrid_motion(i, init.dimensions(), init.animation_mode.hybrid_input_files, + init.args.anim_args.hybrid_motion) + + +def call_hybrid_composite(init, i, image, hybrid_comp_schedules): + with context(init.args) as ia: + return hybrid_composite(ia.args, ia.anim_args, i, image, init.depth_model, + hybrid_comp_schedules, init.args.root) diff --git a/scripts/deforum_helpers/rendering/util/call/images.py b/scripts/deforum_helpers/rendering/util/call/images.py new file mode 100644 index 000000000..ab3dd60da --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/images.py @@ -0,0 +1,10 @@ +from ....load_images import get_mask_from_file + + +def call_get_mask_from_file(init, i, is_mask: bool = False): + next_frame = get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, i, is_mask) + return get_mask_from_file(next_frame, init.args.args) + + +def call_get_mask_from_file_with_frame(init, frame): + return get_mask_from_file(frame, init.args.args) diff --git a/scripts/deforum_helpers/rendering/util/call/resume.py b/scripts/deforum_helpers/rendering/util/call/resume.py new file mode 100644 index 000000000..c3cc3cbd1 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/resume.py @@ -0,0 +1,7 @@ +from ....resume import get_resume_vars + + +def call_get_resume_vars(init, turbo): + return get_resume_vars(folder=init.args.args.outdir, + timestring=init.args.anim_args.resume_timestring, + cadence=turbo.steps) diff --git a/scripts/deforum_helpers/rendering/util/call/subtitle.py b/scripts/deforum_helpers/rendering/util/call/subtitle.py new file mode 100644 index 000000000..876fabfa4 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/subtitle.py @@ -0,0 +1,10 @@ +from ....subtitle_handler import format_animation_params, write_frame_subtitle + + +def call_format_animation_params(init, i, params_to_print): + return format_animation_params(init.animation_keys.deform_keys, init.prompt_series, i, params_to_print) + + +def call_write_frame_subtitle(init, i, params_string, is_cadence: bool = False) -> None: + text = f"F#: {i}; Cadence: {is_cadence}; Seed: {init.args.args.seed}; {params_string}" + write_frame_subtitle(init.srt.filename, i, init.srt.frame_duration, text) diff --git a/scripts/deforum_helpers/rendering/util/call/video_and_audio.py b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py new file mode 100644 index 000000000..0cd505de9 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py @@ -0,0 +1,11 @@ +from ..utils import context +from ....video_audio_utilities import get_next_frame, render_preview + + +def call_render_preview(init, i, last_preview_frame): + with context(init.args) as ia: + return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) + + +def call_get_next_frame(init, i, video_path, is_mask: bool = False): + return get_next_frame(init.output_directory, video_path, i, is_mask) diff --git a/scripts/deforum_helpers/rendering/util/call_utils.py b/scripts/deforum_helpers/rendering/util/call_utils.py deleted file mode 100644 index 43bd1de17..000000000 --- a/scripts/deforum_helpers/rendering/util/call_utils.py +++ /dev/null @@ -1,141 +0,0 @@ -from .utils import context -from ...animation import anim_frame_warp -from ...generate import generate -from ...hybrid_video import ( - # Functions related to flow calculation - get_flow_from_images, - get_flow_for_hybrid_motion, - get_flow_for_hybrid_motion_prev, - - # Functions related to matrix calculation - get_matrix_for_hybrid_motion, - get_matrix_for_hybrid_motion_prev, - - # Other hybrid functions - hybrid_composite) -from ...load_images import get_mask_from_file -from ...resume import get_resume_vars -from ...subtitle_handler import format_animation_params, write_frame_subtitle -from ...video_audio_utilities import get_next_frame, render_preview - -""" -This module provides utility functions for simplifying calls to other modules within the `render.py` module. - -**Purpose:** -- **Reduce Argument Complexity:** Provides a way to call functions in other modules without directly handling - a large number of complex arguments. This simplifies code within `render.py` by encapsulating argument management. -- **Minimize Namespace Pollution:** Provides an alternative to overloading methods in the original modules, - which would introduce the `RenderInit` class into namespaces where it's not inherently needed. - -**Structure:** -- **Simple Call Forwarding:** Functions in this module primarily act as wrappers. They perform minimal logic, - often just formatting or passing arguments, and directly call the corresponding method. -- **Naming Convention:** - - Function names begin with "call_", followed by the name of the actual method to call. - - The `init` object is always passed as the first argument. - - Frame indices (e.g., `frame_idx`, `twin_frame_idx`) are passed as the second argument "i", when relevant. - -**Example:** -```python -# Example function in this module -def call_some_function(init, i, ...): - return some_module.some_function(init.arg77, init.arg.arg.whatever, i, ...) -``` -""" - - -# Animation: -def call_anim_frame_warp(init, i, image, depth): - with context(init.args) as ia: - return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, - depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) - - -# Generation: -def call_generate(init, i, schedule): - with context(init.args) as ia: - return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, - ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) - - -# Hybrid Video -def call_get_flow_from_images(init, prev_image, next_image, cadence): - # cadence is currently either "optical_flow_redo_generation" or "init.args.anim_args.optical_flow_cadence" - # TODO try to init "optical_flow_redo_generation" early, then remove the "cadence" arg again - return get_flow_from_images(prev_image, next_image, cadence, init.animation_mode.raft_model) - - -def call_get_flow_for_hybrid_motion_prev(init, i, image): - with context(init.animation_mode) as mode: - with context(init.args.anim_args) as aa: - return get_flow_for_hybrid_motion_prev(i, init.dimensions(), - mode.hybrid_input_files, - mode.hybrid_frame_path, - mode.prev_flow, - image, - aa.hybrid_flow_method, - mode.raft_model, - aa.hybrid_flow_consistency, - aa.hybrid_consistency_blur, - aa.hybrid_comp_save_extra_frames) - - -def call_get_flow_for_hybrid_motion(init, i): - with context(init.animation_mode) as mode: - with context(init.args.anim_args) as args: - return get_flow_for_hybrid_motion(i, init.dimensions(), mode.hybrid_input_files, mode.hybrid_frame_path, - mode.prev_flow, args.hybrid_flow_method, mode.raft_model, - args.hybrid_flow_consistency, args.hybrid_consistency_blur, args) - - -def call_get_matrix_for_hybrid_motion_prev(init, i, image): - return get_matrix_for_hybrid_motion_prev(i, init.dimensions(), init.animation_mode.hybrid_input_files, - image, init.args.anim_args.hybrid_motion) - - -def call_get_matrix_for_hybrid_motion(init, i): - return get_matrix_for_hybrid_motion(i, init.dimensions(), init.animation_mode.hybrid_input_files, - init.args.anim_args.hybrid_motion) - - -def call_hybrid_composite(init, i, image, hybrid_comp_schedules): - with context(init.args) as ia: - return hybrid_composite(ia.args, ia.anim_args, i, image, init.depth_model, - hybrid_comp_schedules, init.args.root) - - -# Load Images -def call_get_mask_from_file(init, i, is_mask: bool = False): - next_frame = get_next_frame(init.output_directory, init.args.anim_args.video_mask_path, i, is_mask) - return get_mask_from_file(next_frame, init.args.args) - - -def call_get_mask_from_file_with_frame(init, frame): - return get_mask_from_file(frame, init.args.args) - - -# Resume -def call_get_resume_vars(init, turbo): - return get_resume_vars(folder=init.args.args.outdir, - timestring=init.args.anim_args.resume_timestring, - cadence=turbo.steps) - - -# Subtitle -def call_format_animation_params(init, i, params_to_print): - return format_animation_params(init.animation_keys.deform_keys, init.prompt_series, i, params_to_print) - - -def call_write_frame_subtitle(init, i, params_string, is_cadence: bool = False) -> None: - text = f"F#: {i}; Cadence: {is_cadence}; Seed: {init.args.args.seed}; {params_string}" - write_frame_subtitle(init.srt.filename, i, init.srt.frame_duration, text) - - -# Video & Audio -def call_render_preview(init, i, last_preview_frame): - with context(init.args) as ia: - return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) - - -def call_get_next_frame(init, i, video_path, is_mask: bool = False): - return get_next_frame(init.output_directory, video_path, i, is_mask) diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 2139e7074..4cb7699a3 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -1,51 +1,49 @@ +# noinspection PyUnresolvedReferences from modules import lowvram, devices, sd_hijack +# noinspection PyUnresolvedReferences from modules.shared import cmd_opts # keep readonly -class MemoryUtils: - # Don't put any variables here, it's meant for static methods only. +def is_low_or_med_vram(): + # TODO Ideally this should only be called once at the beginning of the render. + # Perhaps add a constant bool to RenderInit. + return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly - @staticmethod - def is_low_or_med_vram(): - # TODO Ideally this should only be called once at the beginning of the render. - # Perhaps add a constant bool to RenderInit. - return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly - @staticmethod - def handle_med_or_low_vram_before_step(init): - if init.is_3d_with_med_or_low_vram(): - # Unload the main checkpoint and load the depth model - lowvram.send_everything_to_cpu() - sd_hijack.model_hijack.undo_hijack(sd_model) - devices.torch_gc() - if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(init.root.device) - - @staticmethod - def handle_vram_if_depth_is_predicted(init): +def handle_med_or_low_vram_before_step(init): + if init.is_3d_with_med_or_low_vram(): + # Unload the main checkpoint and load the depth model + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() if init.animation_mode.is_predicting_depths: - if init.is_3d_with_med_or_low_vram(): - init.depth_model.to('cpu') - devices.torch_gc() - lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) - sd_hijack.model_hijack.hijack(sd_model) - - @staticmethod - def handle_vram_before_depth_map_generation(init): - if MemoryUtils.is_low_or_med_vram(): - lowvram.send_everything_to_cpu() - sd_hijack.model_hijack.undo_hijack(sd_model) - devices.torch_gc() - init.depth_model.to(init.root.device) + init.animation_mode.depth_model.to(init.root.device) + - @staticmethod - def handle_vram_after_depth_map_generation(init): - if MemoryUtils.is_low_or_med_vram(): +def handle_vram_if_depth_is_predicted(init): + if init.animation_mode.is_predicting_depths: + if init.is_3d_with_med_or_low_vram(): init.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) - @staticmethod - def select_depth_device(root): - return 'cpu' if MemoryUtils.is_low_or_med_vram() else root.device + +def handle_vram_before_depth_map_generation(init): + if is_low_or_med_vram(): + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() + init.depth_model.to(init.root.device) + + +def handle_vram_after_depth_map_generation(init): + if is_low_or_med_vram(): + init.depth_model.to('cpu') + devices.torch_gc() + lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) + sd_hijack.model_hijack.hijack(sd_model) + + +def select_depth_device(root): + return 'cpu' if is_low_or_med_vram() else root.device diff --git a/scripts/deforum_helpers/rendering/util/opt_utils.py b/scripts/deforum_helpers/rendering/util/opt_utils.py new file mode 100644 index 000000000..54403b2d6 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/opt_utils.py @@ -0,0 +1,20 @@ +from .utils import put_if_present + + +def setup(init, schedule): + data = init.args.opts.data + if init.has_img2img_fix_steps(): + # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations + data["img2img_fix_steps"] = False + put_if_present(data, "CLIP_stop_at_last_layers", schedule.clipskip) + put_if_present(data, "initial_noise_multiplier", schedule.noise_multiplier) + put_if_present(data, "eta_ddim", schedule.eta_ddim) + put_if_present(data, "eta_ancestral", schedule.eta_ancestral) + + +def generation_info_for_subtitles(init): + return init.args.opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) + + +def is_generate_subtitles(init): + return init.args.opts.data.get("deforum_save_gen_info_as_srt") diff --git a/scripts/deforum_helpers/rendering/util/web_ui.py b/scripts/deforum_helpers/rendering/util/web_ui.py deleted file mode 100644 index 45f7ed3a0..000000000 --- a/scripts/deforum_helpers/rendering/util/web_ui.py +++ /dev/null @@ -1,33 +0,0 @@ -from deforum_api import JobStatusTracker -from modules.shared import state - - -class WebUi: - WEB_UI_SLEEP_DELAY = 0.1 - - @staticmethod - def init_job(init): - state.job_count = init.args.anim_args.max_frames - - @staticmethod - def update_job(init, indexes): - frame = indexes.frame + 1 - max_frames = init.args.anim_args.max_frames - state.job = f"frame {frame}/{max_frames}" - state.job_no = frame + 1 - if state.skipped: - print("\n** PAUSED **") - state.skipped = False - while not state.skipped: - time.sleep(WEB_UI_SLEEP_DELAY) - print("** RESUMING **") - - @staticmethod - def update_status_tracker(init, indexes): - progress = indexes.frame / init.args.anim_args.max_frames - JobStatusTracker().update_phase(init.root.job_id, phase="GENERATING", progress=progress) - - @staticmethod - def update_progress_during_cadence(init, indexes): - state.job = f"frame {indexes.tween_frame + 1}/{init.args.anim_args.max_frames}" - state.job_no = indexes.tween_frame + 1 diff --git a/scripts/deforum_helpers/rendering/util/web_ui_utils.py b/scripts/deforum_helpers/rendering/util/web_ui_utils.py new file mode 100644 index 000000000..8e0b1314e --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/web_ui_utils.py @@ -0,0 +1,34 @@ +# noinspection PyUnresolvedReferences +from deforum_api import JobStatusTracker +# noinspection PyUnresolvedReferences +from modules.shared import state + + +WEB_UI_SLEEP_DELAY = 0.1 + + +def init_job(init): + state.job_count = init.args.anim_args.max_frames + + +def update_job(init, indexes): + frame = indexes.frame.i + 1 + max_frames = init.args.anim_args.max_frames + state.job = f"frame {frame}/{max_frames}" + state.job_no = frame + 1 + if state.skipped: + print("\n** PAUSED **") + state.skipped = False + while not state.skipped: + time.sleep(WEB_UI_SLEEP_DELAY) + print("** RESUMING **") + + +def update_status_tracker(init, indexes): + progress = indexes.frame.i / init.args.anim_args.max_frames + JobStatusTracker().update_phase(init.root.job_id, phase="GENERATING", progress=progress) + + +def update_progress_during_cadence(init, indexes): + state.job = f"frame {indexes.tween.i + 1}/{init.args.anim_args.max_frames}" + state.job_no = indexes.tween.i + 1 From 4a1ca7e9e92686b15f8097ca7a49338e3c8272bb Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 01:50:41 +0200 Subject: [PATCH 044/132] Namespace cleanup and new utility module for file naming. --- scripts/deforum_helpers/render.py | 16 +++----- scripts/deforum_helpers/rendering/__init__.py | 1 - .../rendering/data/anim/__init__.py | 2 +- .../rendering/util/__init__.py | 8 +--- .../rendering/util/filename_utils.py | 40 +++++++++++++++++++ .../deforum_helpers/rendering/util/utils.py | 38 ++++++++++++++---- 6 files changed, 80 insertions(+), 25 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/filename_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d8b61cff8..1d7174fbc 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -16,7 +16,6 @@ import gc import os -import random import PIL import cv2 @@ -35,7 +34,7 @@ from .prompt import prepare_prompt from .rendering.data import Turbo, Schedule, Images, Indexes, Mask from .rendering.initialization import RenderInit, StepInit -from .rendering.util import opt_utils, web_ui_utils, memory_utils +from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.anim import call_anim_frame_warp from .rendering.util.call.gen import call_generate from .rendering.util.call.hybrid import ( @@ -47,7 +46,6 @@ from .rendering.util.utils import context from .save_images import save_image from .seed import next_seed -from .video_audio_utilities import get_frame_name def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -211,20 +209,18 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = f"{init.root.timestring}_{idx.tween.i:09}.png" + filename = filename_utils.tween_frame(init, idx) save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(init.output_directory, - f"{init.root.timestring}_depth_{idx.tween.i:09}.png") + dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, idx)) init.depth_model.save(dm_save_path, depth) # get color match for video outside of images.previous conditional if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): if int(idx.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(os.path.join(init.output_directory, 'inputframes', get_frame_name( - init.args.anim_args.video_init_path) + f"{idx.frame.i:09}.jpg")) + prev_vid_img = Image.open(preview_video_image_path(init, idx)) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) images.color_match = np.asarray(prev_vid_img) images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) @@ -454,7 +450,7 @@ def generate_depth_maps_if_active(init): if init.args.anim_args.save_depth_maps: memory_utils.handle_vram_before_depth_map_generation(init) depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.root.half_precision) - depth_filename = f"{init.root.timestring}_depth_{idx.frame.i:09}.png" + depth_filename = filename_utils.depth_frame(init, idx) init.depth_model.save(os.path.join(init.output_directory, depth_filename), depth) memory_utils.handle_vram_after_depth_map_generation(init) return depth @@ -465,7 +461,7 @@ def progress_step(init, idx, turbo, opencv_image, image, depth): if turbo.has_steps(): return idx.frame.i + turbo.progress_step(idx, opencv_image), depth else: - filename = f"{init.root.timestring}_{idx.frame.i:09}.png" + filename = filename_utils.frame(init, idx) save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) depth = generate_depth_maps_if_active(init) return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. diff --git a/scripts/deforum_helpers/rendering/__init__.py b/scripts/deforum_helpers/rendering/__init__.py index 11abfc6c2..e69de29bb 100644 --- a/scripts/deforum_helpers/rendering/__init__.py +++ b/scripts/deforum_helpers/rendering/__init__.py @@ -1 +0,0 @@ -from .initialization import RenderInit, RenderInitArgs diff --git a/scripts/deforum_helpers/rendering/data/anim/__init__.py b/scripts/deforum_helpers/rendering/data/anim/__init__.py index 19ab35d05..24b69f2a0 100644 --- a/scripts/deforum_helpers/rendering/data/anim/__init__.py +++ b/scripts/deforum_helpers/rendering/data/anim/__init__.py @@ -1,2 +1,2 @@ from .animation_keys import AnimationKeys -from .animation_mode import AnimationMode \ No newline at end of file +from .animation_mode import AnimationMode diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 76b06411d..e7bc19969 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,8 +1,4 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .memory_utils import (is_low_or_med_vram, handle_med_or_low_vram_before_step, handle_vram_if_depth_is_predicted, - handle_vram_before_depth_map_generation, handle_vram_after_depth_map_generation, - select_depth_device) -from .opt_utils import setup, generation_info_for_subtitles, is_generate_subtitles -from .utils import put_all, put_if_present, call_or_use_on_cond -from .web_ui_utils import init_job, update_job, update_status_tracker, update_progress_during_cadence +from .filename_utils import frame, depth_frame, tween_frame, preview_video_image_path +from .utils import context, put_all, put_if_present, call_or_use_on_cond diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py new file mode 100644 index 000000000..4c990eaa6 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -0,0 +1,40 @@ +from enum import Enum +from ..initialization import StepInit +from ..data import Indexes +from ...video_audio_utilities import get_frame_name +from pathlib import Path + + +class FileFormat(Enum): + JPG = "jpg" + PNG = "png" + + +def _frame_filename_index(i: int, file_format: FileFormat) -> str: + return f"{i:09}.{file_format.value}" + + +def _frame_filename(init: StepInit, i: int, is_depth: bool = False, file_format: FileFormat = FileFormat.PNG) -> str: + infix = "_depth_" if is_depth else "_" + return f"{init.root.timestring}{infix}{_frame_filename_index(i, file_format)}" + + +def frame(init: StepInit, indexes: Indexes) -> str: + return _frame_filename(init, indexes.frame.i) + + +def depth_frame(init: StepInit, indexes: Indexes) -> str: + return _frame_filename(init, indexes.frame.i, True) + + +def tween_frame(init: StepInit, indexes: Indexes) -> str: + return _frame_filename(init, indexes.tween.i) + + +def tween_depth_frame(init: StepInit, indexes: Indexes) -> str: + return _frame_filename(init, indexes.tween.i,True) + + +def preview_video_image_path(init: StepInit, indexes: Indexes) -> Path: + frame_name = get_frame_name(init.args.anim_args.video_init_path) + return Path(init.output_directory) / "inputframes" / (frame_name + _frame_filename_index(indexes, FileFormat.JPG)) diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index 11150598d..33ea193e9 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -2,6 +2,35 @@ from PIL import Image +@contextmanager +def context(resource): + """ + Context manager for aliasing and managing objects or class references within a `with` statement block. + The function can help keeping code compact, less repetitive or more structured by providing a dedicated scope + for resource access, at the tradeoff of introducing an additional level of nesting. + It can be particularly useful when extracting logic to a separate function is undesirable. + + **Example Usage:** + ```python + with context(annoyingly_long_reference.to_some_expression.or_to.MyClass().or_something) as res: + res.do_something() + do_something_with_the_thing(res) + res.do_something_else() + # more lines using res + ``` + + **Arguments:** + - `resource (object or class reference)`: The object or class reference to be managed within the context. + + **Yields:** + - The provided `resource` argument. + + **Note:** + The context manager is responsible for any necessary resource cleanup upon exiting the `with` block (if applicable). + """ + yield resource + + def put_all(dictionaries, key, value): return list(map(lambda d: {**d, key: value}, dictionaries)) @@ -11,17 +40,12 @@ def put_if_present(dictionary, key, value): dictionary[key] = value -def _call_or_use(): +def _call_or_use(callable_or_value): return callable_or_value() if callable(callable_or_value) else callable_or_value def call_or_use_on_cond(condition, callable_or_value): - return _call_or_use() if condition else None - - -@contextmanager -def context(cls_or_instance): - yield cls_or_instance + return _call_or_use(callable_or_value) if condition else None def create_img(dimensions): From bb66790637ed7fc3f265df440d4e676d38586911 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 04:14:41 +0200 Subject: [PATCH 045/132] Optimized some code and defined simple combo_context. --- .../rendering/data/anim/animation_keys.py | 16 +++--- .../rendering/data/anim/animation_mode.py | 53 +++++++++++-------- .../rendering/util/filename_utils.py | 15 ++++-- .../rendering/util/memory_utils.py | 4 ++ .../deforum_helpers/rendering/util/utils.py | 5 ++ 5 files changed, 61 insertions(+), 32 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index e34f54f3a..ed2416e21 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -20,14 +20,16 @@ def update(self, i: int): keys.colorCorrectionFactor = keys.color_correction_factor_series[i] @staticmethod - def choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): + def _choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): return default_keys if not parseq_adapter.use_parseq else parseq_keys @staticmethod def from_args(step_args, parseq_adapter, seed): - # Parseq keys are decorated, see ParseqAinmKeysDecorator and ParseqLooperKeysDecorator - return AnimationKeys( - AnimationKeys.choose_default_or_parseq_keys(DeformAnimKeys(step_args.anim_args, seed), - parseq_adapter.anim_keys, parseq_adapter), - AnimationKeys.choose_default_or_parseq_keys(LooperAnimKeys(step_args.loop_args, step_args.anim_args, seed), - parseq_adapter.looper_keys, parseq_adapter)) + ada = parseq_adapter + + def _choose(default_keys): + return AnimationKeys._choose_default_or_parseq_keys(default_keys, ada.anim_keys, ada) + + # Parseq keys are decorated, see ParseqAnimKeysDecorator and ParseqLooperKeysDecorator + return AnimationKeys(_choose(DeformAnimKeys(step_args.anim_args, seed)), + _choose(LooperAnimKeys(step_args.loop_args, step_args.anim_args, seed))) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 6dd8b6209..39dbba4bb 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -1,9 +1,12 @@ import os - from dataclasses import dataclass +from pathlib import Path from typing import Any -from ....hybrid_video import hybrid_generation + +from ...util.memory_utils import keep_3d_models_in_vram +from ...util.utils import combo_context from ....RAFT import RAFT +from ....hybrid_video import hybrid_generation # TODO FIXME find a way to assign prev_flow right away, then set frozen=true again, otherwise move prev_flow elsewhere. @@ -30,7 +33,7 @@ def unload_raft_and_depth_model(self): self.raft_model.delete_model() @staticmethod - def _has_video_input(anim_args) -> bool: + def has_video_input(anim_args) -> bool: return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod @@ -43,18 +46,19 @@ def _is_using_hybrid_frames(anim_args): or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']) @staticmethod - def _is_needing_hybrid_frames(anim_args): + def _is_requiring_hybrid_frames(anim_args): return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod def _is_load_depth_model_for_3d(args, anim_args): is_depth_warped_3d = anim_args.animation_mode == '3D' and anim_args.use_depth_warping - is_composite_with_depth = anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] - is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth + has_depth_or_depth_video_mask = anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'] + is_composite_with_depth_mask = anim_args.hybrid_composite and has_depth_or_depth_video_mask + is_depth_used = is_depth_warped_3d or anim_args.save_depth_maps or is_composite_with_depth_mask return is_depth_used and not args.motion_preview_mode @staticmethod - def _load_raft_if_active(anim_args, args): + def load_raft_if_active(anim_args, args): is_cadenced_raft = anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1 is_optical_flow_raft = anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT" is_raft_redo = anim_args.optical_flow_redo_generation == "RAFT" @@ -64,24 +68,29 @@ def _load_raft_if_active(anim_args, args): return RAFT() if is_load_raft else None @staticmethod - def _load_depth_model_if_active(args, anim_args, opts): + def load_depth_model_if_active(args, anim_args, opts): return AnimationMode._is_load_depth_model_for_3d(args, anim_args) \ if opts.data.get("deforum_keep_3d_models_in_vram", False) else None + @staticmethod + def initial_hybrid_files(sa) -> list[Path]: + """ + Returns a list of initial hybrid input files if required, otherwise an empty list. + """ + if AnimationMode._is_requiring_hybrid_frames(sa.anim_args): + # may cause side effects on args and anim_args. + _, __, init_hybrid_input_files = hybrid_generation(sa.args, sa.anim_args, sa.root) + return init_hybrid_input_files + else: + return [] + @staticmethod def from_args(step_args): - init_hybrid_input_files: Any = None - init_hybrid_frame_path = "" - if AnimationMode._is_needing_hybrid_frames(step_args.anim_args): - # handle hybrid video generation - # hybrid_generation may cause side effects on args and anim_args - _, __, init_hybrid_input_files = hybrid_generation(step_args.args, step_args.anim_args, step_args.root) + with combo_context(step_args, AnimationMode) as (sa, AM): # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False - init_hybrid_frame_path = os.path.join(step_args.args.outdir, 'hybridframes') - return AnimationMode(AnimationMode._has_video_input(step_args.anim_args), - init_hybrid_input_files, - init_hybrid_frame_path, - None, - step_args.opts.data.get("deforum_keep_3d_models_in_vram", False), - AnimationMode._load_depth_model_if_active(step_args.args, step_args.anim_args, step_args.opts), - AnimationMode._load_raft_if_active(step_args.anim_args, step_args.args)) + hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') + return AnimationMode( + AM.has_video_input(sa.anim_args), AM.initial_hybrid_files(sa), + hybrid_input_files, None, keep_3d_models_in_vram(sa), + AM.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), + AM.load_raft_if_active(sa.anim_args, sa.args)) diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 4c990eaa6..674a1e96d 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -9,12 +9,20 @@ class FileFormat(Enum): JPG = "jpg" PNG = "png" + @staticmethod + def frame_format(): + return FileFormat.PNG + + @staticmethod + def video_frame_format(): + return FileFormat.JPG + def _frame_filename_index(i: int, file_format: FileFormat) -> str: return f"{i:09}.{file_format.value}" -def _frame_filename(init: StepInit, i: int, is_depth: bool = False, file_format: FileFormat = FileFormat.PNG) -> str: +def _frame_filename(init: StepInit, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" return f"{init.root.timestring}{infix}{_frame_filename_index(i, file_format)}" @@ -32,9 +40,10 @@ def tween_frame(init: StepInit, indexes: Indexes) -> str: def tween_depth_frame(init: StepInit, indexes: Indexes) -> str: - return _frame_filename(init, indexes.tween.i,True) + return _frame_filename(init, indexes.tween.i, True) def preview_video_image_path(init: StepInit, indexes: Indexes) -> Path: frame_name = get_frame_name(init.args.anim_args.video_init_path) - return Path(init.output_directory) / "inputframes" / (frame_name + _frame_filename_index(indexes, FileFormat.JPG)) + index = _frame_filename_index(indexes.frame.i, FileFormat.video_frame_format()) + return Path(init.output_directory) / "inputframes" / (frame_name + index) diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 4cb7699a3..aee826c53 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -47,3 +47,7 @@ def handle_vram_after_depth_map_generation(init): def select_depth_device(root): return 'cpu' if is_low_or_med_vram() else root.device + + +def keep_3d_models_in_vram(step_args): + return step_args.opts.data.get("deforum_keep_3d_models_in_vram", False) diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index 33ea193e9..f3b44ca47 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -31,6 +31,11 @@ def context(resource): yield resource +@contextmanager +def combo_context(*resources): + yield resources + + def put_all(dictionaries, key, value): return list(map(lambda d: {**d, key: value}, dictionaries)) From 1326a3f12d061d8fede73b2853efda1c5eb53e08 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 06:16:43 +0200 Subject: [PATCH 046/132] Step and SubStep preparation. Info printing separation. --- scripts/deforum_helpers/render.py | 233 +++++++++--------- .../rendering/{ => data}/initialization.py | 66 +---- .../rendering/data/schedule.py | 27 +- .../rendering/data/step/__init__.py | 2 + .../rendering/data/step/step.py | 64 +++++ .../rendering/data/step/sub_step.py | 12 + .../rendering/util/__init__.py | 2 +- .../rendering/util/filename_utils.py | 5 +- .../rendering/util/log_utils.py | 25 ++ 9 files changed, 246 insertions(+), 190 deletions(-) rename scripts/deforum_helpers/rendering/{ => data}/initialization.py (79%) create mode 100644 scripts/deforum_helpers/rendering/data/step/__init__.py create mode 100644 scripts/deforum_helpers/rendering/data/step/step.py create mode 100644 scripts/deforum_helpers/rendering/data/step/sub_step.py create mode 100644 scripts/deforum_helpers/rendering/util/log_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 1d7174fbc..d56ac4f66 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -26,14 +26,15 @@ from .colors import maintain_colors from .composable_masks import compose_mask_with_check -from .hybrid_video import (image_transform_ransac, image_transform_optical_flow, - abs_flow_to_rel_flow, rel_flow_to_abs_flow) +from .hybrid_video import ( + image_transform_ransac, image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) from .image_sharpening import unsharp_mask from .masks import do_overlay_mask from .noise import add_noise from .prompt import prepare_prompt -from .rendering.data import Turbo, Schedule, Images, Indexes, Mask -from .rendering.initialization import RenderInit, StepInit +from .rendering.data import Turbo, Images, Indexes, Mask +from .rendering.data.step import StepInit, Step, SubStep +from .rendering.data.initialization import RenderInit from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.anim import call_anim_frame_warp from .rendering.util.call.gen import call_generate @@ -43,6 +44,9 @@ from .rendering.util.call.images import call_get_mask_from_file_with_frame from .rendering.util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame +from .rendering.util.log_utils import ( + print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, + print_redo_generation_info) from .rendering.util.utils import context from .save_images import save_image from .seed import next_seed @@ -69,47 +73,40 @@ def run_render_animation_controlled(init): def run_render_animation(init): # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. # TODO isolate "depth" with other moving parts - images = Images.create(init) turbo = Turbo.create(init) # state for interpolating between diffusion steps - idx = Indexes.create(init, turbo) - mask = Mask.create(init, idx.frame.i) # reset the mask vals as they are overwritten in the compose_mask algorithm + indexes = Indexes.create(init, turbo) + mask = Mask.create(init, indexes.frame.i) # reset mask vals as they are overwritten in the compose_mask algorithm web_ui_utils.init_job(init) last_preview_frame = 0 - while idx.frame.i < init.args.anim_args.max_frames: - web_ui_utils.update_job(init, idx) - print(f"\033[36mAnimation frame: \033[0m{idx.frame.i}/{init.args.anim_args.max_frames} ") + while indexes.frame.i < init.args.anim_args.max_frames: + web_ui_utils.update_job(init, indexes) + print_animation_frame_info(init, indexes) # TODO Stuff to be moved into new Step class in `run_render_animation_controlled`: - step_init = StepInit.create(init.animation_keys.deform_keys, idx.frame.i) - schedule = Schedule.create(init.animation_keys.deform_keys, idx.frame.i, init.args.anim_args, init.args.args) - - if init.is_use_mask and not init.args.anim_args.use_noise_mask: - noise_mask_seq = schedule.mask_seq + step = Step.create(init, indexes) - depth = None memory_utils.handle_med_or_low_vram_before_step(init) if turbo.is_first_step_with_subtitles(init): params_to_print = opt_utils.generation_info_for_subtitles(init) - params_string = call_format_animation_params(init, idx.frame.i, params_to_print) - call_write_frame_subtitle(init, idx.frame.i, params_string) + params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) + call_write_frame_subtitle(init, indexes.frame.i, params_string) if turbo.is_emit_in_between_frames(): - idx.tween.start = max(idx.frame.start, idx.frame.i - turbo.steps) + indexes.tween.start = max(indexes.frame.start, indexes.frame.i - turbo.steps) cadence_flow = None # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` - for idx.tween.i in range(idx.tween.start, idx.frame.i): - web_ui_utils.update_progress_during_cadence(init, idx) - # cadence vars - tween = (float(idx.tween.i - idx.tween.start + 1) / - float(idx.frame.i - idx.tween.start)) + for indexes.tween.i in range(indexes.tween.start, indexes.frame.i): + web_ui_utils.update_progress_during_cadence(init, indexes) + + sub_step = SubStep.create(init, indexes) # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): - if init.animation_keys.deform_keys.strength_schedule_series[idx.tween.start.i] > 0: + if init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0: if cadence_flow is None and turbo.prev.image is not None and turbo.next.image is not None: cadence_flow = call_get_flow_from_images(init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2 @@ -117,59 +114,56 @@ def run_render_animation(init): if opt_utils.is_generate_subtitles(init): params_to_print = opt_utils.generation_info_for_subtitles(init) - params_string = call_format_animation_params(init, idx.tween.i, params_to_print) - call_write_frame_subtitle(init, idx.tween.i, params_string, tween < 1.0) + params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) + call_write_frame_subtitle(init, indexes.tween.i, params_string, sub_step.tween < 1.0) - msg_flow_name = '' if cadence_flow is None \ - else init.args.anim_args.optical_flow_cadence + ' optical flow ' - msg_frame_info = f"cadence frame: {idx.tween.i}; tween:{tween:0.2f};" - print(f"Creating in-between {msg_flow_name}{msg_frame_info}") + print_tween_frame_info(init, indexes, cadence_flow, sub_step.tween) if init.depth_model is not None: assert (turbo.next.image is not None) - depth = init.depth_model.predict(turbo.next.image, + step.depth = init.depth_model.predict(turbo.next.image, init.args.anim_args.midas_weight, init.root.half_precision) - turbo.advance(init, idx.tween.i, depth) + turbo.advance(init, indexes.tween.i, step.depth) # hybrid video motion - warps turbo.prev.image or turbo.next.image to match motion - if idx.tween.i > 0: + if indexes.tween.i > 0: if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.tween.i - 1, images.previous) - if turbo.is_advance_prev(idx.tween.i): + matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween.i - 1, images.previous) + if turbo.is_advance_prev(indexes.tween.i): turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(idx.tween.i): + if turbo.is_advance_next(indexes.tween.i): turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) else: - matrix = call_get_matrix_for_hybrid_motion(init, idx.tween.i - 1) - if turbo.is_advance_prev(idx.tween.i): + matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween.i - 1) + if turbo.is_advance_prev(indexes.tween.i): turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(idx.tween.i): + if turbo.is_advance_next(indexes.tween.i): turbo.next.image = image_transform_ransac(turbo.next.image, matrix, init.args.anim_args.hybrid_motion) if init.args.anim_args.hybrid_motion in ['Optical Flow']: if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, idx.tween.i - 1, images.previous) - if turbo.is_advance_prev(idx.tween.i): + flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween.i - 1, images.previous) + if turbo.is_advance_prev(indexes.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, - step_init.flow_factor()) - if turbo.is_advance_next(idx.tween.i): + step.init.flow_factor()) + if turbo.is_advance_next(indexes.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, - step_init.flow_factor()) + step.init.flow_factor()) init.animation_mode.prev_flow = flow else: - flow = call_get_flow_for_hybrid_motion(init, idx.tween.i - 1) - if turbo.is_advance_prev(idx.tween.i): + flow = call_get_flow_for_hybrid_motion(init, indexes.tween.i - 1) + if turbo.is_advance_prev(indexes.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, - step_init.flow_factor()) - if turbo.is_advance_next(idx.tween.i): + step.init.flow_factor()) + if turbo.is_advance_next(indexes.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, - step_init.flow_factor()) + step.init.flow_factor()) init.animation_mode.prev_flow = flow # TODO cadence related transforms to be decoupled and handled in a 2nd pass @@ -177,19 +171,19 @@ def run_render_animation(init): with context(cadence_flow) as cf: if cf is not None: cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) - cf, _ = call_anim_frame_warp(init, idx.tween.i, cf, depth) - cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * tween - if turbo.is_advance_prev(idx.tween.i): + cf, _ = call_anim_frame_warp(init, indexes.tween.i, cf, step.depth) + cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * sub_step.tween + if turbo.is_advance_prev(indexes.tween.i): turbo.prev.image = image_transform_optical_flow(turbo.prev.image, cadence_flow_inc, - step_init.cadence_flow_factor) - if turbo.is_advance_next(idx.tween.i): + step.init.cadence_flow_factor) + if turbo.is_advance_next(indexes.tween.i): turbo.next.image = image_transform_optical_flow(turbo.next.image, cadence_flow_inc, - step_init.cadence_flow_factor) + step.init.cadence_flow_factor) - turbo.prev.index = turbo.next.frame_idx = idx.tween.i + turbo.prev.index = turbo.next.frame_idx = indexes.tween.i - if turbo.prev.image is not None and tween < 1.0: - img = turbo.prev.image * (1.0 - tween) + turbo.next.image * tween + if turbo.prev.image is not None and sub_step.tween < 1.0: + img = turbo.prev.image * (1.0 - sub_step.tween) + turbo.next.image * sub_step.tween else: img = turbo.next.image @@ -200,7 +194,7 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): - img = do_overlay_mask(init.args.args, init.args.anim_args, img, idx.tween.i, True) + img = do_overlay_mask(init.args.args, init.args.anim_args, img, indexes.tween.i, True) # get images.previous during cadence images.previous = img @@ -209,18 +203,18 @@ def run_render_animation(init): # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) # saving cadence frames - filename = filename_utils.tween_frame(init, idx) + filename = filename_utils.tween_frame(init, indexes) save_path = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, img) if init.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, idx)) - init.depth_model.save(dm_save_path, depth) + dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) + init.depth_model.save(dm_save_path, step.depth) # get color match for video outside of images.previous conditional if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(idx.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(preview_video_image_path(init, idx)) + if int(indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + prev_vid_img = Image.open(preview_video_image_path(init, indexes)) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) images.color_match = np.asarray(prev_vid_img) images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) @@ -228,35 +222,35 @@ def run_render_animation(init): # after 1st frame, images.previous exists if images.previous is not None: # apply transforms to previous frame - images.previous, depth = call_anim_frame_warp(init, idx.frame.i, images.previous, None) + images.previous, step.depth = call_anim_frame_warp(init, indexes.frame.i, images.previous, None) # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, idx.frame.i, images.previous, - step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, + step.init.hybrid_comp_schedules) # hybrid video motion - warps images.previous to match motion, usually to prepare for compositing with context(init.args.anim_args) as aa: if aa.hybrid_motion in ['Affine', 'Perspective']: if aa.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, idx.frame.i - 1, images.previous) + matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.frame.i - 1, images.previous) else: - matrix = call_get_matrix_for_hybrid_motion(init, idx.frame.i - 1) + matrix = call_get_matrix_for_hybrid_motion(init, indexes.frame.i - 1) images.previous = image_transform_ransac(images.previous, matrix, aa.hybrid_motion) if aa.hybrid_motion in ['Optical Flow']: if aa.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, idx.frame.i - 1, images.previous) + flow = call_get_flow_for_hybrid_motion_prev(init, indexes.frame.i - 1, images.previous) else: - flow = call_get_flow_for_hybrid_motion(init, idx.frame.i - 1) - images.previous = image_transform_optical_flow(images.previous, flow, step_init.flow_factor()) + flow = call_get_flow_for_hybrid_motion(init, indexes.frame.i - 1) + images.previous = image_transform_optical_flow(images.previous, flow, step.init.flow_factor()) init.animation_mode.prev_flow = flow # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, images.previous = call_hybrid_composite(init, idx.frame.i, images.previous, - step_init.hybrid_comp_schedules) + _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, + step.init.hybrid_comp_schedules) # apply color matching if init.has_color_coherence(): if images.color_match is None: @@ -271,75 +265,74 @@ def run_render_animation(init): images.previous = cv2.cvtColor(images.previous, cv2.COLOR_GRAY2BGR) # apply scaling - contrast_image = (images.previous * step_init.contrast).round().astype(np.uint8) + contrast_image = (images.previous * step.init.contrast).round().astype(np.uint8) # anti-blur - if step_init.amount > 0: - step_init.kernel_size() + if step.init.amount > 0: + step.init.kernel_size() contrast_image = unsharp_mask(contrast_image, - (step_init.kernel, step_init.kernel), - step_init.sigma, - step_init.amount, - step_init.threshold, + (step.init.kernel, step.init.kernel), + step.init.sigma, + step.init.amount, + step.init.threshold, mask.image if init.args.args.use_mask else None) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: init.root.noise_mask = compose_mask_with_check(init.root, init.args.args, - noise_mask_seq, - # FIXME might be ref'd b4 assignment + step.schedule.noise_mask_seq, mask.noise_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) with context(init.args.anim_args) as aa: - noised_image = add_noise(contrast_image, step_init.noise, init.args.args.seed, aa.noise_type, + noised_image = add_noise(contrast_image, step.init.noise, init.args.args.seed, aa.noise_type, (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence), init.root.noise_mask, init.args.args.invert_mask) # use transformed previous frame as init for current init.args.args.use_init = True init.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) - init.args.args.strength = max(0.0, min(1.0, step_init.strength)) + init.args.args.strength = max(0.0, min(1.0, step.init.strength)) - init.args.args.scale = step_init.scale + init.args.args.scale = step.init.scale # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints init.args.args.pix2pix_img_cfg_scale = float( - init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[idx.frame.i]) + init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[indexes.frame.i]) # grab prompt for current frame - init.args.args.prompt = init.prompt_series[idx.frame.i] + init.args.args.prompt = init.prompt_series[indexes.frame.i] with context(init.args) as ia: with context(init.animation_keys.deform_keys) as keys: # FIXME? check ia.args.seed_behavior if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - ia.args.seed = int(keys.seed_schedule_series[idx.frame.i]) # TODO recontextualize frame index + ia.args.seed = int(keys.seed_schedule_series[indexes.frame.i]) # TODO recontextualize frame index if ia.anim_args.enable_checkpoint_scheduling: - ia.args.checkpoint = keys.checkpoint_schedule_series[idx.frame.i] + ia.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame.i] else: ia.args.checkpoint = None # SubSeed scheduling if ia.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[idx.frame.i]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[idx.frame.i]) + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame.i]) if init.parseq_adapter.manages_seed(): init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[idx.frame.i]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[idx.frame.i] + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame.i] # set value back into the prompt - prepare and report prompt and seed - ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, idx.frame.i) + ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, indexes.frame.i) # grab init image for current frame if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, idx.frame.i, ia.anim_args.video_init_path) - print(f"Using video init frame {init_frame}") + init_frame = call_get_next_frame(init, indexes.frame.i, ia.anim_args.video_init_path) + print_init_frame_info(init_frame) ia.args.init_image = init_frame ia.args.init_image_box = None # init_image_box not used in this case - ia.args.strength = max(0.0, min(1.0, step_init.strength)) + ia.args.strength = max(0.0, min(1.0, step.init.strength)) if ia.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, idx.frame.i, ia.anim_args.video_mask_path, True) + mask_init_frame = call_get_next_frame(init, indexes.frame.i, ia.anim_args.video_mask_path, True) temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) ia.args.mask_file = temp_mask init.root.noise_mask = temp_mask @@ -347,12 +340,12 @@ def run_render_animation(init): if ia.args.use_mask: # TODO figure why this is different from mask.image - ia.args.mask_image = compose_mask_with_check(init.root, ia.args, schedule.mask_seq, + ia.args.mask_image = compose_mask_with_check(init.root, ia.args, step.schedule.mask_seq, mask.vals, init.root.init_sample) \ if init.root.init_sample is not None else None # we need it only after the first frame anyway - init.animation_keys.update(idx.frame.i) - opt_utils.setup(init, schedule) + init.animation_keys.update(indexes.frame.i) + opt_utils.setup(init, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(init) @@ -361,17 +354,15 @@ def run_render_animation(init): if not init.args.args.motion_preview_mode else 'None' # optical flow redo before generation - if optical_flow_redo_generation != 'None' and images.previous is not None and step_init.strength > 0: + if optical_flow_redo_generation != 'None' and images.previous is not None and step.init.strength > 0: stored_seed = init.args.args.seed init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere - msg_start = "Optical flow redo is diffusing and warping using" - msg_end = "optical flow before generation." - print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") - with context(call_generate(init, idx.frame.i, schedule)) as img: + print_optical_flow_info(init, optical_flow_redo_generation) + with context(call_generate(init, indexes.frame.i, step.schedule)) as img: img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) disposable_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) - img = image_transform_optical_flow(img, disposable_flow, step_init.redo_flow_factor) + img = image_transform_optical_flow(img, disposable_flow, step.init.redo_flow_factor) init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds init.root.init_sample = Image.fromarray(img) disposable_image = img # TODO refactor @@ -380,13 +371,13 @@ def run_render_animation(init): # diffusion redo if (int(init.args.anim_args.diffusion_redo) > 0 - and images.previous is not None and step_init.strength > 0 + and images.previous is not None and step.init.strength > 0 and not init.args.args.motion_preview_mode): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): - print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") + print_redo_generation_info(init, n) init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere - disposable_image = call_generate(init, idx.frame.i, schedule) + disposable_image = call_generate(init, indexes.frame.i, step.schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only if n == int(init.args.anim_args.diffusion_redo): @@ -398,21 +389,21 @@ def run_render_animation(init): gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation - image = call_generate(init, idx.frame.i, schedule) + image = call_generate(init, indexes.frame.i, step.schedule) if image is None: break # do hybrid video after generation - if idx.frame.i > 0 and init.is_hybrid_composite_after_generation(): + if indexes.frame.i > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) - _, temp_image_2 = call_hybrid_composite(init, idx.frame.i, temp_image, step_init.hybrid_comp_schedules) + _, temp_image_2 = call_hybrid_composite(init, indexes.frame.i, temp_image, step.init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if idx.frame.i == 0 and init.is_color_match_to_be_initialized(images.color_match): + if indexes.frame.i == 0 and init.is_color_match_to_be_initialized(images.color_match): temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) @@ -424,11 +415,11 @@ def run_render_animation(init): # overlay mask if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): - image = do_overlay_mask(init.args.args, init.args.anim_args, image, idx.frame.i) + image = do_overlay_mask(init.args.args, init.args.anim_args, image, indexes.frame.i) # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) - or (init.args.anim_args.legacy_colormatch and step_init.strength == 0)) + or (init.args.anim_args.legacy_colormatch and step.init.strength == 0)) and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): images.color_match = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) @@ -436,12 +427,12 @@ def run_render_animation(init): if not init.animation_mode.has_video_input: images.previous = opencv_image - idx.frame.i, depth = progress_step(init, idx, turbo, opencv_image, image, depth) + indexes.frame.i, step.depth = progress_step(init, indexes, turbo, opencv_image, image, step.depth) state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(init, idx.frame.i, last_preview_frame) - web_ui_utils.update_status_tracker(init, idx) + last_preview_frame = call_render_preview(init, indexes.frame.i, last_preview_frame) + web_ui_utils.update_status_tracker(init, indexes) init.animation_mode.unload_raft_and_depth_model() diff --git a/scripts/deforum_helpers/rendering/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py similarity index 79% rename from scripts/deforum_helpers/rendering/initialization.py rename to scripts/deforum_helpers/rendering/data/initialization.py index 88251dbf8..f5aff4445 100644 --- a/scripts/deforum_helpers/rendering/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -6,16 +6,16 @@ import numpy as np import pandas as pd -from .data.anim import AnimationKeys, AnimationMode -from .data.subtitle import Srt -from .util import memory_utils -from .util.utils import context -from ..args import RootArgs -from ..deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled -from ..depth import DepthModel -from ..generate import isJson -from ..parseq_adapter import ParseqAdapter -from ..settings import save_settings_from_animation_run +from .anim import AnimationKeys, AnimationMode +from .subtitle import Srt +from ..util import memory_utils +from ..util.utils import context +from ...args import RootArgs +from ...deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled +from ...depth import DepthModel +from ...generate import (isJson) +from ...parseq_adapter import ParseqAdapter +from ...settings import save_settings_from_animation_run @dataclass(init=True, frozen=True, repr=False, eq=False) @@ -34,52 +34,6 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -@dataclass(init=True, frozen=True, repr=False, eq=False) -class StepInit: - noise: Any = None - strength: Any = None - scale: Any = None - contrast: Any = None - kernel: int = 0 - sigma: Any = None - amount: Any = None - threshold: Any = None - cadence_flow_factor: Any = None - redo_flow_factor: Any = None - hybrid_comp_schedules: Any = None - - def kernel_size(self) -> tuple[int, int]: - return self.kernel, self.kernel - - def flow_factor(self): - return self.hybrid_comp_schedules['flow_factor'] - - @staticmethod - def create(deform_keys, i): - with context(deform_keys) as keys: - return StepInit(keys.noise_schedule_series[i], - keys.strength_schedule_series[i], - keys.cfg_scale_schedule_series[i], - keys.contrast_schedule_series[i], - int(keys.kernel_schedule_series[i]), - keys.sigma_schedule_series[i], - keys.amount_schedule_series[i], - keys.threshold_schedule_series[i], - keys.cadence_flow_factor_schedule_series[i], - keys.redo_flow_factor_schedule_series[i], - StepInit._hybrid_comp_args(keys, i)) - - @staticmethod - def _hybrid_comp_args(keys, i): - return { - "alpha": keys.hybrid_comp_alpha_schedule_series[i], - "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[i], - "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[i], - "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), - "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), - "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} - - @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index d398d9f92..0c7bf72a4 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,6 +1,8 @@ from dataclasses import dataclass from typing import Optional, Any +from ..util import context + @dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: @@ -12,20 +14,25 @@ class Schedule: eta_ancestral: float # TODO unify ddim- and a-eta to use one or the other, depending on sampler mask: Optional[Any] noise_mask: Optional[Any] + noise_mask_seq: Any @staticmethod - def create(keys, i, anim_args, args): + def create(init, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" - steps = Schedule.schedule_steps(keys, i, anim_args) - sampler_name = Schedule.schedule_sampler(keys, i, anim_args) - clipskip = Schedule.schedule_clipskip(keys, i, anim_args) - noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) - eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) - eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) - mask = Schedule.schedule_mask(keys, i, args) # TODO for some reason use_mask is in args instead of anim_args - noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) - return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) + is_use_mask_without_noise = init.is_use_mask and not init.args.anim_args.use_noise_mask + with context(init.animation_keys.deform_keys) as keys: + steps = Schedule.schedule_steps(keys, i, anim_args) + sampler_name = Schedule.schedule_sampler(keys, i, anim_args) + clipskip = Schedule.schedule_clipskip(keys, i, anim_args) + noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) + mask = Schedule.schedule_mask(keys, i, args) + noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) + noise_mask_seq = schedule.mask_seq if is_use_mask_without_noise else None + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, + noise_mask, noise_mask_seq) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py new file mode 100644 index 000000000..1f33bbe1e --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -0,0 +1,2 @@ +from .step import StepInit, Step +from .sub_step import SubStep diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py new file mode 100644 index 000000000..b35c7cc5b --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -0,0 +1,64 @@ +from dataclasses import dataclass +from typing import Any + +from ..schedule import Schedule +from ...util.utils import context + + +@dataclass(init=True, frozen=True, repr=False, eq=False) +class StepInit: + noise: Any = None + strength: Any = None + scale: Any = None + contrast: Any = None + kernel: int = 0 + sigma: Any = None + amount: Any = None + threshold: Any = None + cadence_flow_factor: Any = None + redo_flow_factor: Any = None + hybrid_comp_schedules: Any = None + + def kernel_size(self) -> tuple[int, int]: + return self.kernel, self.kernel + + def flow_factor(self): + return self.hybrid_comp_schedules['flow_factor'] + + @staticmethod + def create(deform_keys, i): + with context(deform_keys) as keys: + return StepInit(keys.noise_schedule_series[i], + keys.strength_schedule_series[i], + keys.cfg_scale_schedule_series[i], + keys.contrast_schedule_series[i], + int(keys.kernel_schedule_series[i]), + keys.sigma_schedule_series[i], + keys.amount_schedule_series[i], + keys.threshold_schedule_series[i], + keys.cadence_flow_factor_schedule_series[i], + keys.redo_flow_factor_schedule_series[i], + StepInit._hybrid_comp_args(keys, i)) + + @staticmethod + def _hybrid_comp_args(keys, i): + return { + "alpha": keys.hybrid_comp_alpha_schedule_series[i], + "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[i], + "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[i], + "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), + "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), + "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} + + +@dataclass(init=True, frozen=False, repr=False, eq=False) +class Step: + init: StepInit + schedule: Schedule + depth: Any # TODO try to init early, then freeze class + + @staticmethod + def create(init, indexes): + step_init = StepInit.create(init.animation_keys.deform_keys, indexes.frame.i) + schedule = Schedule.create(init, indexes.frame.i, init.args.anim_args, init.args.args) + return Step(step_init, schedule, None) diff --git a/scripts/deforum_helpers/rendering/data/step/sub_step.py b/scripts/deforum_helpers/rendering/data/step/sub_step.py new file mode 100644 index 000000000..257e8cd02 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/step/sub_step.py @@ -0,0 +1,12 @@ +from dataclasses import dataclass +from typing import Any + + +@dataclass(init=True, frozen=True, repr=False, eq=False) +class SubStep: + tween: Any # cadence vars + + @staticmethod + def create(init, indexes): + tween = (float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start)) + return SubStep(tween) diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index e7bc19969..84a4f0f1e 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,4 +1,4 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .filename_utils import frame, depth_frame, tween_frame, preview_video_image_path from .utils import context, put_all, put_if_present, call_or_use_on_cond +from .filename_utils import frame, depth_frame, tween_frame, preview_video_image_path diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 674a1e96d..f7f3657d9 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -1,8 +1,9 @@ from enum import Enum -from ..initialization import StepInit +from pathlib import Path + from ..data import Indexes +from ..data.step import StepInit from ...video_audio_utilities import get_frame_name -from pathlib import Path class FileFormat(Enum): diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py new file mode 100644 index 000000000..2a82d8a65 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -0,0 +1,25 @@ + + +def print_animation_frame_info(init, indexes): + print(f"\033[36mAnimation frame: \033[0m{indexes.frame.i}/{init.args.anim_args.max_frames}") + + +def print_tween_frame_info(init, indexes, cadence_flow, tween): + msg_flow_name = '' if cadence_flow is None \ + else init.args.anim_args.optical_flow_cadence + ' optical flow ' + msg_frame_info = f"cadence frame: {indexes.tween.i}; tween:{tween:0.2f};" + print(f"Creating in-between {msg_flow_name}{msg_frame_info}") + + +def print_init_frame_info(init_frame): + print(f"Using video init frame {init_frame}") + + +def print_optical_flow_info(init, optical_flow_redo_generation): + msg_start = "Optical flow redo is diffusing and warping using" + msg_end = "optical flow before generation." + print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") + + +def print_redo_generation_info(init, n): + print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") From 5cf6c08154d89284145d7f2fd90baf4fb3aafe2e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 07:57:38 +0200 Subject: [PATCH 047/132] Subtitle writing moved to Step. Cadence flow moved to TweenStep. --- scripts/deforum_helpers/render.py | 80 ++++++++----------- .../deforum_helpers/rendering/data/indexes.py | 18 ++++- .../rendering/data/step/__init__.py | 2 +- .../rendering/data/step/step.py | 17 +++- .../rendering/data/step/sub_step.py | 12 --- .../rendering/data/step/tween_step.py | 15 ++++ .../deforum_helpers/rendering/data/turbo.py | 1 + 7 files changed, 82 insertions(+), 63 deletions(-) delete mode 100644 scripts/deforum_helpers/rendering/data/step/sub_step.py create mode 100644 scripts/deforum_helpers/rendering/data/step/tween_step.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d56ac4f66..8712f355d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -33,8 +33,8 @@ from .noise import add_noise from .prompt import prepare_prompt from .rendering.data import Turbo, Images, Indexes, Mask -from .rendering.data.step import StepInit, Step, SubStep from .rendering.data.initialization import RenderInit +from .rendering.data.step import Step, TweenStep from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.anim import call_anim_frame_warp from .rendering.util.call.gen import call_generate @@ -42,7 +42,6 @@ call_get_flow_from_images, call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) from .rendering.util.call.images import call_get_mask_from_file_with_frame -from .rendering.util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame from .rendering.util.log_utils import ( print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, @@ -81,49 +80,37 @@ def run_render_animation(init): web_ui_utils.init_job(init) last_preview_frame = 0 while indexes.frame.i < init.args.anim_args.max_frames: - web_ui_utils.update_job(init, indexes) + memory_utils.handle_med_or_low_vram_before_step(init) print_animation_frame_info(init, indexes) - - # TODO Stuff to be moved into new Step class in `run_render_animation_controlled`: + web_ui_utils.update_job(init, indexes) step = Step.create(init, indexes) - - memory_utils.handle_med_or_low_vram_before_step(init) - - if turbo.is_first_step_with_subtitles(init): - params_to_print = opt_utils.generation_info_for_subtitles(init) - params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) - call_write_frame_subtitle(init, indexes.frame.i, params_string) - + step.write_frame_subtitle(init, indexes, turbo) if turbo.is_emit_in_between_frames(): - indexes.tween.start = max(indexes.frame.start, indexes.frame.i - turbo.steps) - cadence_flow = None + indexes.update_tween_start(turbo) # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` - for indexes.tween.i in range(indexes.tween.start, indexes.frame.i): + for tween_index in range(indexes.tween.start, indexes.frame.i): + indexes.update_tween(tween_index) web_ui_utils.update_progress_during_cadence(init, indexes) - - sub_step = SubStep.create(init, indexes) + tween_step = TweenStep.create(init, indexes) # optical flow cadence setup before animation warping if (init.args.anim_args.animation_mode in ['2D', '3D'] and init.args.anim_args.optical_flow_cadence != 'None'): if init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0: - if cadence_flow is None and turbo.prev.image is not None and turbo.next.image is not None: - cadence_flow = call_get_flow_from_images(init, turbo.prev.image, turbo.next.image, - init.args.anim_args.optical_flow_cadence) / 2 - turbo.next.image = image_transform_optical_flow(turbo.next.image, -cadence_flow, 1) - - if opt_utils.is_generate_subtitles(init): - params_to_print = opt_utils.generation_info_for_subtitles(init) - params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) - call_write_frame_subtitle(init, indexes.tween.i, params_string, sub_step.tween < 1.0) - - print_tween_frame_info(init, indexes, cadence_flow, sub_step.tween) + if (tween_step.cadence_flow is None + and turbo.prev.image is not None + and turbo.next.image is not None): + tween_step.cadence_flow = (call_get_flow_from_images( + init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2) + turbo.next.image = image_transform_optical_flow(turbo.next.image, -tween_step.cadence_flow, 1) + step.write_frame_subtitle_if_active(init, indexes, opt_utils) + print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) if init.depth_model is not None: assert (turbo.next.image is not None) step.depth = init.depth_model.predict(turbo.next.image, - init.args.anim_args.midas_weight, - init.root.half_precision) + init.args.anim_args.midas_weight, + init.root.half_precision) turbo.advance(init, indexes.tween.i, step.depth) @@ -168,22 +155,23 @@ def run_render_animation(init): # TODO cadence related transforms to be decoupled and handled in a 2nd pass # do optical flow cadence after animation warping - with context(cadence_flow) as cf: - if cf is not None: - cf = abs_flow_to_rel_flow(cf, init.width(), init.height()) - cf, _ = call_anim_frame_warp(init, indexes.tween.i, cf, step.depth) - cadence_flow_inc = rel_flow_to_abs_flow(cf, init.width(), init.height()) * sub_step.tween - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_optical_flow(turbo.prev.image, cadence_flow_inc, - step.init.cadence_flow_factor) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_optical_flow(turbo.next.image, cadence_flow_inc, - step.init.cadence_flow_factor) + if tween_step.cadence_flow is not None: + tween_step.cadence_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, init.width(), init.height()) + tween_step.cadence_flow, _ = call_anim_frame_warp( + init, indexes.tween.i, tween_step.cadence_flow, step.depth) + tween_step.cadence_flow_inc = rel_flow_to_abs_flow( + tween_step.cadence_flow, init.width(), init.height()) * tween_step.tween + if turbo.is_advance_prev(indexes.tween.i): + turbo.prev.image = image_transform_optical_flow( + turbo.prev.image, tween_step.cadence_flow_inc, step.init.sub_step.cadence_flow_factor) + if turbo.is_advance_next(indexes.tween.i): + turbo.next.image = image_transform_optical_flow( + turbo.next.image, tween_step.cadence_flow_inc, step.init.sub_step.cadence_flow_factor) turbo.prev.index = turbo.next.frame_idx = indexes.tween.i - if turbo.prev.image is not None and sub_step.tween < 1.0: - img = turbo.prev.image * (1.0 - sub_step.tween) + turbo.next.image * sub_step.tween + if turbo.prev.image is not None and tween_step.tween < 1.0: + img = turbo.prev.image * (1.0 - tween_step.tween) + turbo.next.image * tween_step.tween else: img = turbo.next.image @@ -427,7 +415,9 @@ def run_render_animation(init): if not init.animation_mode.has_video_input: images.previous = opencv_image - indexes.frame.i, step.depth = progress_step(init, indexes, turbo, opencv_image, image, step.depth) + next_frame, step.depth = progress_step(init, indexes, turbo, opencv_image, image, step.depth) + indexes.update_frame(next_frame) + state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index b97210842..5fb8aad9c 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -1,7 +1,7 @@ from dataclasses import dataclass -@dataclass(init=True, frozen=False, repr=True, eq=True) +@dataclass(init=True, frozen=True, repr=True, eq=True) class IndexWithStart: start: int = 0 i: int = 0 @@ -14,6 +14,16 @@ class Indexes: @staticmethod def create(init, turbo): - frame = IndexWithStart(turbo.find_start(init, turbo), 0) - tween = IndexWithStart(0, 0) - return Indexes(frame, tween) + frame_start = turbo.find_start(init, turbo) + tween_start = 0 + return Indexes(IndexWithStart(frame_start, 0), IndexWithStart(tween_start, 0)) + + def update_tween(self, i: int): + self.tween = IndexWithStart(self.tween.start, i) + + def update_tween_start(self, turbo): + tween_start = max(self.frame.start, self.frame.i - turbo.steps) + self.tween = IndexWithStart(tween_start, self.tween.i) + + def update_frame(self, i: int): + self.frame = IndexWithStart(self.frame.start, i) diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py index 1f33bbe1e..47868a343 100644 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -1,2 +1,2 @@ from .step import StepInit, Step -from .sub_step import SubStep +from .tween_step import TweenStep diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index b35c7cc5b..c9b709dd2 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -2,6 +2,7 @@ from typing import Any from ..schedule import Schedule +from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.utils import context @@ -56,9 +57,23 @@ class Step: init: StepInit schedule: Schedule depth: Any # TODO try to init early, then freeze class + subtitle_params_to_print: Any + subtitle_params_string: str @staticmethod def create(init, indexes): step_init = StepInit.create(init.animation_keys.deform_keys, indexes.frame.i) schedule = Schedule.create(init, indexes.frame.i, init.args.anim_args, init.args.args) - return Step(step_init, schedule, None) + return Step(step_init, schedule, None, None, "") + + def write_frame_subtitle(self, init, indexes, turbo): + if turbo.is_first_step_with_subtitles(init): + self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) + self.subtitle_params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) + call_write_frame_subtitle(init, indexes.frame.i, params_string) + + def write_frame_subtitle_if_active(self, init, indexes, opt_utils): + if opt_utils.is_generate_subtitles(init): + self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) + self.subtitle_params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) + call_write_frame_subtitle(init, indexes.tween.i, params_string, sub_step.tween < 1.0) diff --git a/scripts/deforum_helpers/rendering/data/step/sub_step.py b/scripts/deforum_helpers/rendering/data/step/sub_step.py deleted file mode 100644 index 257e8cd02..000000000 --- a/scripts/deforum_helpers/rendering/data/step/sub_step.py +++ /dev/null @@ -1,12 +0,0 @@ -from dataclasses import dataclass -from typing import Any - - -@dataclass(init=True, frozen=True, repr=False, eq=False) -class SubStep: - tween: Any # cadence vars - - @staticmethod - def create(init, indexes): - tween = (float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start)) - return SubStep(tween) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py new file mode 100644 index 000000000..a98990bc2 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -0,0 +1,15 @@ +from dataclasses import dataclass +from typing import Any + + +@dataclass(init=True, frozen=False, repr=False, eq=False) +class TweenStep: + """cadence vars""" + tween: float + cadence_flow: Any + cadence_flow_inc: Any + + @staticmethod + def create(init, indexes): + tween = float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start) + return TweenStep(tween, None, None) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 34627f8e5..a9f87f308 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -1,4 +1,5 @@ from dataclasses import dataclass +from typing import Any from cv2.typing import MatLike From 3317b1bf51af57f28e4a17eec48ce5aceb0d9cc0 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 15:59:16 +0200 Subject: [PATCH 048/132] Adjusted some calls related to noise and image masking. --- scripts/deforum_helpers/render.py | 110 ++++++++---------- .../rendering/util/call/images.py | 12 ++ .../rendering/util/call/mask.py | 13 +++ 3 files changed, 74 insertions(+), 61 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/call/mask.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 8712f355d..d4aca6c90 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -25,12 +25,9 @@ from modules.shared import opts, state from .colors import maintain_colors -from .composable_masks import compose_mask_with_check from .hybrid_video import ( image_transform_ransac, image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) -from .image_sharpening import unsharp_mask from .masks import do_overlay_mask -from .noise import add_noise from .prompt import prepare_prompt from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit @@ -41,7 +38,8 @@ from .rendering.util.call.hybrid import ( call_get_flow_from_images, call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) -from .rendering.util.call.images import call_get_mask_from_file_with_frame +from .rendering.util.call.images import call_add_noise, call_get_mask_from_file_with_frame +from .rendering.util.call.mask import call_compose_mask_with_check, call_unsharp_mask from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame from .rendering.util.log_utils import ( print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, @@ -102,7 +100,8 @@ def run_render_animation(init): and turbo.next.image is not None): tween_step.cadence_flow = (call_get_flow_from_images( init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2) - turbo.next.image = image_transform_optical_flow(turbo.next.image, -tween_step.cadence_flow, 1) + turbo.next.image = image_transform_optical_flow(turbo.next.image, -tween_step.cadence_flow, + 1) step.write_frame_subtitle_if_active(init, indexes, opt_utils) print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) @@ -257,25 +256,13 @@ def run_render_animation(init): # anti-blur if step.init.amount > 0: step.init.kernel_size() - contrast_image = unsharp_mask(contrast_image, - (step.init.kernel, step.init.kernel), - step.init.sigma, - step.init.amount, - step.init.threshold, - mask.image if init.args.args.use_mask else None) + contrast_image = call_unsharp_mask(init, step, contrast_image, mask) # apply frame noising if init.args.args.use_mask or init.args.anim_args.use_noise_mask: - init.root.noise_mask = compose_mask_with_check(init.root, - init.args.args, - step.schedule.noise_mask_seq, - mask.noise_vals, - Image.fromarray(cv2.cvtColor(contrast_image, - cv2.COLOR_BGR2RGB))) + init.root.noise_mask = call_compose_mask_with_check( + init, step.schedule.noise_mask_seq, mask.noise_vals, contrast_image) - with context(init.args.anim_args) as aa: - noised_image = add_noise(contrast_image, step.init.noise, init.args.args.seed, aa.noise_type, - (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence), - init.root.noise_mask, init.args.args.invert_mask) + noised_image = call_add_noise(init, step, contrast_image) # use transformed previous frame as init for current init.args.args.use_init = True @@ -291,46 +278,47 @@ def run_render_animation(init): # grab prompt for current frame init.args.args.prompt = init.prompt_series[indexes.frame.i] - with context(init.args) as ia: - with context(init.animation_keys.deform_keys) as keys: - # FIXME? check ia.args.seed_behavior - if ia.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - ia.args.seed = int(keys.seed_schedule_series[indexes.frame.i]) # TODO recontextualize frame index - if ia.anim_args.enable_checkpoint_scheduling: - ia.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame.i] - else: - ia.args.checkpoint = None - - # SubSeed scheduling - if ia.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame.i]) - if init.parseq_adapter.manages_seed(): - init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame.i] - - # set value back into the prompt - prepare and report prompt and seed - ia.args.prompt = prepare_prompt(ia.args.prompt, ia.anim_args.max_frames, ia.args.seed, indexes.frame.i) - # grab init image for current frame - if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, indexes.frame.i, ia.anim_args.video_init_path) - print_init_frame_info(init_frame) - ia.args.init_image = init_frame - ia.args.init_image_box = None # init_image_box not used in this case - ia.args.strength = max(0.0, min(1.0, step.init.strength)) - if ia.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, indexes.frame.i, ia.anim_args.video_mask_path, True) - temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) - ia.args.mask_file = temp_mask - init.root.noise_mask = temp_mask - mask.vals['video_mask'] = temp_mask - - if ia.args.use_mask: - # TODO figure why this is different from mask.image - ia.args.mask_image = compose_mask_with_check(init.root, ia.args, step.schedule.mask_seq, - mask.vals, init.root.init_sample) \ - if init.root.init_sample is not None else None # we need it only after the first frame anyway + with context(init.animation_keys.deform_keys) as keys: + if init.args.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): + init.args.args.seed = int(keys.seed_schedule_series[indexes.frame.i]) + if init.args.anim_args.enable_checkpoint_scheduling: + init.args.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame.i] + else: + init.args.args.checkpoint = None + + # SubSeed scheduling + if init.args.anim_args.enable_subseed_scheduling: + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) + init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame.i]) + if init.parseq_adapter.manages_seed(): + init.args.anim_args.enable_subseed_scheduling = True + init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) + init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame.i] + + # set value back into the prompt - prepare and report prompt and seed + init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, + init.args.args.seed, indexes.frame.i) + # grab init image for current frame + if init.animation_mode.has_video_input: + init_frame = call_get_next_frame(init, indexes.frame.i, init.args.anim_args.video_init_path) + print_init_frame_info(init_frame) + init.args.args.init_image = init_frame + init.args.args.init_image_box = None # init_image_box not used in this case + init.args.args.strength = max(0.0, min(1.0, step.init.strength)) + if init.args.anim_args.use_mask_video: + mask_init_frame = call_get_next_frame(init, indexes.frame.i, init.args.anim_args.video_mask_path, True) + temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) + init.args.args.mask_file = temp_mask + init.root.noise_mask = temp_mask + mask.vals['video_mask'] = temp_mask + + if init.args.args.use_mask: + # TODO figure why this is different from mask.image + init.args.args.mask_image = call_compose_mask_with_check( + init, step.schedule.mask_seq, mask.vals, init.root.init_sample) + init.args.args.mask_image = compose_mask_with_check(init.root, init.args.args, step.schedule.mask_seq, + mask.vals, init.root.init_sample) \ + if init.root.init_sample is not None else None # we need it only after the first frame anyway init.animation_keys.update(indexes.frame.i) opt_utils.setup(init, step.schedule) diff --git a/scripts/deforum_helpers/rendering/util/call/images.py b/scripts/deforum_helpers/rendering/util/call/images.py index ab3dd60da..38781b9e1 100644 --- a/scripts/deforum_helpers/rendering/util/call/images.py +++ b/scripts/deforum_helpers/rendering/util/call/images.py @@ -1,4 +1,16 @@ from ....load_images import get_mask_from_file +from ....noise import add_noise + + +def call_add_noise(init, step, image): + aa = init.args.anim_args + amount: float = step.init.noise + seed: int = init.args.args.seed + n_type: str = aa.noise_type + perlin_arguments = (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence) + mask = init.root.noise_mask + is_do_maks_invert = init.args.args.invert_mask + return add_noise(image, amount, seed, n_type, perlin_arguments, mask, is_do_maks_invert) def call_get_mask_from_file(init, i, is_mask: bool = False): diff --git a/scripts/deforum_helpers/rendering/util/call/mask.py b/scripts/deforum_helpers/rendering/util/call/mask.py new file mode 100644 index 000000000..5b0223b72 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/call/mask.py @@ -0,0 +1,13 @@ +from ....composable_masks import compose_mask_with_check +from ....image_sharpening import unsharp_mask + + +def call_compose_mask_with_check(init, mask_seq, val_masks, image): + return compose_mask_with_check(init.root, init.args.args, mask_seq, val_masks, + Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))) + + +def call_unsharp_mask(init, step, image, mask): + kernel_size = (step.init.kernel, step.init.kernel) + mask_image = mask.image if init.args.args.use_mask else None + return unsharp_mask(image, kernel_size, step.init.sigma, step.init.amount, step.init.threshold, mask_image) From 887f10f85462f517eb3c069c1b9cd8f810023caa Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 17:52:00 +0200 Subject: [PATCH 049/132] Extracted depth prediction and started extracting hybrid motion logic. Solved and obsolete TODOs removed. --- scripts/deforum_helpers/render.py | 104 ++---------------- .../rendering/data/initialization.py | 6 + .../rendering/data/step/step.py | 11 ++ .../deforum_helpers/rendering/data/turbo.py | 88 ++++++++++++++- 4 files changed, 114 insertions(+), 95 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d4aca6c90..046c47a5c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -51,25 +51,14 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - # TODO method is temporarily torn apart to remove args from direct access in larger execution scope. run_render_animation(init) def run_render_animation_controlled(init): - # TODO create a Step class in rendering.data with all the iteration specific info, - # then eventually try to replace the main while loop in `run_render_animation` with functions that: - # - 1. Create a collection of Steps with all the required info that is already known or can be calculated - # before we enter the iteration. - # - 2. Transform and reprocess the steps however needed (i.e. space out or reassign turbo frames etc.) - # TODO cadence framing and logic that is currently working off-index may eventually be moved into a 2nd pass. - # - 3. Actually do the render by foreaching over the steps in sequence - # TODO also create a SubStep class for the inner for-loop in `run_render_animation` (maybe do that 1st). raise NotImplementedError("not implemented.") def run_render_animation(init): - # TODO try to avoid late init of "prev_flow" or isolate it together with all other moving parts. - # TODO isolate "depth" with other moving parts images = Images.create(init) turbo = Turbo.create(init) # state for interpolating between diffusion steps indexes = Indexes.create(init, turbo) @@ -83,96 +72,25 @@ def run_render_animation(init): web_ui_utils.update_job(init, indexes) step = Step.create(init, indexes) step.write_frame_subtitle(init, indexes, turbo) - if turbo.is_emit_in_between_frames(): + if turbo.is_emit_in_between_frames(): # TODO extract tween frame emition indexes.update_tween_start(turbo) - # TODO stuff for new SubStep class to be used in `run_render_animation_controlled` for tween_index in range(indexes.tween.start, indexes.frame.i): indexes.update_tween(tween_index) web_ui_utils.update_progress_during_cadence(init, indexes) tween_step = TweenStep.create(init, indexes) - # optical flow cadence setup before animation warping - if (init.args.anim_args.animation_mode in ['2D', '3D'] - and init.args.anim_args.optical_flow_cadence != 'None'): - if init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0: - if (tween_step.cadence_flow is None - and turbo.prev.image is not None - and turbo.next.image is not None): - tween_step.cadence_flow = (call_get_flow_from_images( - init, turbo.prev.image, turbo.next.image, init.args.anim_args.optical_flow_cadence) / 2) - turbo.next.image = image_transform_optical_flow(turbo.next.image, -tween_step.cadence_flow, - 1) + turbo.do_optical_flow_cadence_setup_before_animation_warping(init, tween_step) + step.write_frame_subtitle_if_active(init, indexes, opt_utils) print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) - if init.depth_model is not None: - assert (turbo.next.image is not None) - step.depth = init.depth_model.predict(turbo.next.image, - init.args.anim_args.midas_weight, - init.root.half_precision) - + step.update_depth_prediction(init, turbo) turbo.advance(init, indexes.tween.i, step.depth) - # hybrid video motion - warps turbo.prev.image or turbo.next.image to match motion - if indexes.tween.i > 0: - if init.args.anim_args.hybrid_motion in ['Affine', 'Perspective']: - if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween.i - 1, images.previous) - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, - init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_ransac(turbo.next.image, matrix, - init.args.anim_args.hybrid_motion) - else: - matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween.i - 1) - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_ransac(turbo.prev.image, matrix, - init.args.anim_args.hybrid_motion) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_ransac(turbo.next.image, matrix, - init.args.anim_args.hybrid_motion) - if init.args.anim_args.hybrid_motion in ['Optical Flow']: - if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween.i - 1, images.previous) - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, - step.init.flow_factor()) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, - step.init.flow_factor()) - init.animation_mode.prev_flow = flow - else: - flow = call_get_flow_for_hybrid_motion(init, indexes.tween.i - 1) - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_optical_flow(turbo.prev.image, flow, - step.init.flow_factor()) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_optical_flow(turbo.next.image, flow, - step.init.flow_factor()) - init.animation_mode.prev_flow = flow + turbo.do_hybrid_video_motion(init, indexes, images) # TODO cadence related transforms to be decoupled and handled in a 2nd pass - # do optical flow cadence after animation warping - if tween_step.cadence_flow is not None: - tween_step.cadence_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, init.width(), init.height()) - tween_step.cadence_flow, _ = call_anim_frame_warp( - init, indexes.tween.i, tween_step.cadence_flow, step.depth) - tween_step.cadence_flow_inc = rel_flow_to_abs_flow( - tween_step.cadence_flow, init.width(), init.height()) * tween_step.tween - if turbo.is_advance_prev(indexes.tween.i): - turbo.prev.image = image_transform_optical_flow( - turbo.prev.image, tween_step.cadence_flow_inc, step.init.sub_step.cadence_flow_factor) - if turbo.is_advance_next(indexes.tween.i): - turbo.next.image = image_transform_optical_flow( - turbo.next.image, tween_step.cadence_flow_inc, step.init.sub_step.cadence_flow_factor) - - turbo.prev.index = turbo.next.frame_idx = indexes.tween.i - - if turbo.prev.image is not None and tween_step.tween < 1.0: - img = turbo.prev.image * (1.0 - tween_step.tween) + turbo.next.image * tween_step.tween - else: - img = turbo.next.image + img = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, tween_step) # intercept and override to grayscale if init.args.anim_args.color_force_grayscale: @@ -213,7 +131,6 @@ def run_render_animation(init): # do hybrid compositing before motion if init.is_hybrid_composite_before_motion(): - # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, step.init.hybrid_comp_schedules) @@ -235,7 +152,6 @@ def run_render_animation(init): # do hybrid compositing after motion (normal) if init.is_normal_hybrid_composite(): - # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, step.init.hybrid_comp_schedules) # apply color matching @@ -313,7 +229,6 @@ def run_render_animation(init): mask.vals['video_mask'] = temp_mask if init.args.args.use_mask: - # TODO figure why this is different from mask.image init.args.args.mask_image = call_compose_mask_with_check( init, step.schedule.mask_seq, mask.vals, init.root.init_sample) init.args.args.mask_image = compose_mask_with_check(init.root, init.args.args, step.schedule.mask_seq, @@ -324,8 +239,6 @@ def run_render_animation(init): opt_utils.setup(init, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(init) - - # TODO try init early, also see "call_get_flow_from_images" optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ if not init.args.args.motion_preview_mode else 'None' @@ -373,7 +286,6 @@ def run_render_animation(init): # do hybrid video after generation if indexes.frame.i > 0 and init.is_hybrid_composite_after_generation(): temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - # TODO test, returned args seem unchanged, so might as well be ignored here (renamed to _) _, temp_image_2 = call_hybrid_composite(init, indexes.frame.i, temp_image, step.init.hybrid_comp_schedules) image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) @@ -414,6 +326,10 @@ def run_render_animation(init): init.animation_mode.unload_raft_and_depth_model() +def emit_in_between_frames(): + raise NotImplemented("") + + def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index f5aff4445..13b2a7019 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -52,6 +52,12 @@ class RenderInit: def is_3d(self): return self.args.anim_args.animation_mode == '3D' + def is_3d_or_2d(self): + return self.args.anim_args.animation_mode in ['2D', '3D'] + + def has_optical_flow_cadence(self): + return self.args.anim_args.optical_flow_cadence != 'None' + def is_3d_with_med_or_low_vram(self): return self.is_3d() and memory_utils.is_low_or_med_vram() diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index c9b709dd2..d4717c05f 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -1,7 +1,9 @@ from dataclasses import dataclass from typing import Any +from ..initialization import RenderInit from ..schedule import Schedule +from ..turbo import Turbo from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.utils import context @@ -66,6 +68,15 @@ def create(init, indexes): schedule = Schedule.create(init, indexes.frame.i, init.args.anim_args, init.args.args) return Step(step_init, schedule, None, None, "") + def update_depth_prediction(self, init: RenderInit, turbo: Turbo): + has_depth = init.depth_model is not None + has_next = turbo.next.image is not None + if has_depth and has_next: + image = turbo.next.image + weight = init.args.anim_args.midas_weight + precision = init.root.half_precision + self.depth = init.depth_model.predict(image, weight, precision) + def write_frame_subtitle(self, init, indexes, turbo): if turbo.is_first_step_with_subtitles(init): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index a9f87f308..583df2400 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -1,7 +1,7 @@ from dataclasses import dataclass -from typing import Any from cv2.typing import MatLike +from ...hybrid_video import image_transform_ransac, image_transform_optical_flow from .subtitle import Srt from ..util.call.anim import call_anim_frame_warp @@ -32,6 +32,92 @@ def advance(self, init, i: int, depth): if self.is_advance_next(i): self.next.image, _ = call_anim_frame_warp(init, i, self.next.image, depth) + def do_hybrid_video_motion(self, init, indexes, reference_images): + """Warps the previous and/or the next to match the motion of the provided reference images.""" + motion = init.args.anim_args.hybrid_motion + + def _is_do_motion(motions): + return indexes.tween.i > 0 and motion in motions + + if _is_do_motion(['Affine', 'Perspective']): + self.advance_hybrid_motion_ransac_trasform(init, indexes, reference_images) + if _is_do_motion(['Optical Flow']): + self.advance_hybrid_motion_optical_tween_flow(init, indexes, reference_images, step) + + def advance_optical_flow(self, tween_step, flow_factor: int = 1): + flow = tween_step.cadence_flow * -1 + self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) + + def advance_optical_tween_flow(self, step, flow): + ff = step.init.flow_factor() + i = indexes.tween.i + if self.is_advance_prev(i): + self.prev.image = image_transform_optical_flow(self.prev.image, flow, ff) + if self.is_advance_next(i): + self.next.image = image_transform_optical_flow(self.next.image, flow, ff) + + def advance_hybrid_motion_optical_tween_flow(self, init, indexes, reference_images, step): + if init.args.anim_args.hybrid_motion_use_prev_img: + flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween.i - 1, reference_images.previous) + turbo.advance_optical_tween_flow(self, step, flow) + init.animation_mode.prev_flow = flow + else: + flow = call_get_flow_for_hybrid_motion(init, indexes.tween.i - 1) + turbo.advance_optical_tween_flow(self, step, flow) + init.animation_mode.prev_flow = flow + + def advance_cadence_flow(self, tween_step): + ff = step.init.sub_step.cadence_flow_factor + i = indexes.tween.i + inc = tween_step.cadence_flow_inc + if self.is_advance_prev(i): + self.prev.image = image_transform_optical_flow(self.prev.image, inc, ff) + if self.is_advance_next(i): + self.next.image = image_transform_optical_flow(self.next.image, inc, ff) + + def advance_ransac_trasform(self, init, matrix): + i = indexes.tween.i + motion = init.args.anim_args.hybrid_motion + if self.is_advance_prev(i): + self.prev.image = image_transform_ransac(self.prev.image, matrix, motion) + if self.is_advance_next(i): + self.next.image = image_transform_ransac(self.next.image, matrix, motion) + + def advance_hybrid_motion_ransac_trasform(self, init, indexes, reference_images): + if init.args.anim_args.hybrid_motion_use_prev_img: + matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween.i - 1, reference_images.previous) + turbo.advance_ransac_trasform(init, matrix) + else: + matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween.i - 1) + turbo.advance_ransac_trasform(init, matrix) + + def do_optical_flow_cadence_setup_before_animation_warping(self, init, tween_step): + if init.is_3d_or_2d() and init.has_optical_flow_cadence(): + has_tween_schedule = init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0 + has_images = self.prev.image is not None and self.next.image is not None + has_step_and_images = tween_step.cadence_flow is None and has_images + if has_tween_schedule and has_step_and_images: + cadence = init.args.anim_args.optical_flow_cadence + flow = call_get_flow_from_images(init, self.prev.image, self.next.image, cadence) + tween_step.cadence_flow = (flow / 2) + self.next.image = advance_optical_flow + self.advance_optical_flow(tween_step) + self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow) + + def do_optical_flow_cadence_after_animation_warping(self, init, indexes, step, tween_step): + if tween_step.cadence_flow is not None: + temp_flow = abs_flow_to_rel_flow(tween_step.flow, init.width(), init.height()) + new_flow, _ = call_anim_frame_warp(init, indexes.tween.i, temp_flow, step.depth) + tween_step.cadence_flow = new_flow + abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, init.width(), init.height()) + tween_step.cadence_flow_inc = abs_flow * tween_step.tween + self.advance_cadence_flow(tween_step) + self.prev.index = self.next.frame_idx = indexes.tween.i + if self.prev.image is not None and tween_step.tween < 1.0: + return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween + else: + return self.next.image + def progress_step(self, indexes, opencv_image): self.prev.image, self.prev.index = self.next.image, self.next.index self.next.image, self.next.index = opencv_image, indexes.frame.i From bce30e7efc00e9fc88c32f65712fae466958d481 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 20:21:17 +0200 Subject: [PATCH 050/132] Image transformation step extraction. --- scripts/deforum_helpers/render.py | 41 ++++--------------- .../rendering/data/step/tween_step.py | 15 ++++++- .../rendering/util/__init__.py | 3 +- .../rendering/util/filename_utils.py | 2 +- .../rendering/util/image_utils.py | 39 ++++++++++++++++++ 5 files changed, 63 insertions(+), 37 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/image_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 046c47a5c..db7e62f02 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -20,14 +20,12 @@ import PIL import cv2 import numpy as np -from PIL import Image, ImageOps +from PIL import Image # noinspection PyUnresolvedReferences from modules.shared import opts, state from .colors import maintain_colors -from .hybrid_video import ( - image_transform_ransac, image_transform_optical_flow, abs_flow_to_rel_flow, rel_flow_to_abs_flow) -from .masks import do_overlay_mask +from .hybrid_video import image_transform_ransac, image_transform_optical_flow from .prompt import prepare_prompt from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit @@ -41,6 +39,7 @@ from .rendering.util.call.images import call_add_noise, call_get_mask_from_file_with_frame from .rendering.util.call.mask import call_compose_mask_with_check, call_unsharp_mask from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame +from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame from .rendering.util.log_utils import ( print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, print_redo_generation_info) @@ -63,7 +62,6 @@ def run_render_animation(init): turbo = Turbo.create(init) # state for interpolating between diffusion steps indexes = Indexes.create(init, turbo) mask = Mask.create(init, indexes.frame.i) # reset mask vals as they are overwritten in the compose_mask algorithm - web_ui_utils.init_job(init) last_preview_frame = 0 while indexes.frame.i < init.args.anim_args.max_frames: @@ -77,7 +75,7 @@ def run_render_animation(init): for tween_index in range(indexes.tween.start, indexes.frame.i): indexes.update_tween(tween_index) web_ui_utils.update_progress_during_cadence(init, indexes) - tween_step = TweenStep.create(init, indexes) + tween_step = TweenStep.create(indexes) turbo.do_optical_flow_cadence_setup_before_animation_warping(init, tween_step) @@ -89,28 +87,13 @@ def run_render_animation(init): turbo.do_hybrid_video_motion(init, indexes, images) - # TODO cadence related transforms to be decoupled and handled in a 2nd pass - img = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, tween_step) - - # intercept and override to grayscale - if init.args.anim_args.color_force_grayscale: - img = cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY) - img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) - - # overlay mask - if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.args.args.use_mask): - img = do_overlay_mask(init.args.args, init.args.anim_args, img, indexes.tween.i, True) - - # get images.previous during cadence - images.previous = img + img = tween_step.generate_tween_image(init, indexes, step, turbo) + images.previous = img # get images.previous during cadence # current image update for cadence frames (left commented because it doesn't currently update the preview) # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) - # saving cadence frames - filename = filename_utils.tween_frame(init, indexes) - save_path = os.path.join(init.args.args.outdir, filename) - cv2.imwrite(save_path, img) + save_cadence_frame(init, indexes, img) if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) @@ -296,14 +279,8 @@ def run_render_animation(init): temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) - # intercept and override to grayscale - if init.args.anim_args.color_force_grayscale: - image = ImageOps.grayscale(image) - image = ImageOps.colorize(image, black="black", white="white") - - # overlay mask - if init.args.args.overlay_mask and (init.args.anim_args.use_mask_video or init.is_use_mask): - image = do_overlay_mask(init.args.args, init.args.anim_args, image, indexes.frame.i) + image = force_to_grayscale_if_required(init, image) + image = add_overlay_mask_if_active(init, image) # on strength 0, set color match to generation if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index a98990bc2..0cd4f9412 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -1,6 +1,8 @@ from dataclasses import dataclass from typing import Any +from ...util.image_utils import add_overlay_mask_if_active, force_tween_to_grayscale_if_required + @dataclass(init=True, frozen=False, repr=False, eq=False) class TweenStep: @@ -10,6 +12,15 @@ class TweenStep: cadence_flow_inc: Any @staticmethod - def create(init, indexes): - tween = float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start) + def create(indexes): + from_i = indexes.frame.i - indexes.tween.start + to_i = indexes.tween.i - indexes.tween.start + 1 + tween = float(to_i) / float(from_i) return TweenStep(tween, None, None) + + def generate_tween_image(self, init, indexes, step, turbo): + is_tween = True + warped = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, self) + recolored = force_tween_to_grayscale_if_required(init, warped) + masked = add_overlay_mask_if_active(init, recolored, is_tween) + return masked diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 84a4f0f1e..9f91a462e 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,4 +1,3 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .utils import context, put_all, put_if_present, call_or_use_on_cond -from .filename_utils import frame, depth_frame, tween_frame, preview_video_image_path +from .utils import call_or_use_on_cond, combo_context, context, put_all, put_if_present diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index f7f3657d9..ae26c7abb 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -36,7 +36,7 @@ def depth_frame(init: StepInit, indexes: Indexes) -> str: return _frame_filename(init, indexes.frame.i, True) -def tween_frame(init: StepInit, indexes: Indexes) -> str: +def tween_frame_name(init: StepInit, indexes: Indexes) -> str: return _frame_filename(init, indexes.tween.i) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py new file mode 100644 index 000000000..95c09c251 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -0,0 +1,39 @@ +import os + +import cv2 + +from .filename_utils import tween_frame_name +from ...masks import do_overlay_mask + + +def force_tween_to_grayscale_if_required(init, image): + if init.args.anim_args.color_force_grayscale: + gray_image = cv2.cvtColor(image.astype(np.uint8), cv2.COLOR_BGR2GRAY) + return cv2.cvtColor(gray_image, cv2.COLOR_GRAY2BGR) + else: + return image + + +def force_to_grayscale_if_required(init, image): + if init.args.anim_args.color_force_grayscale: + gray_image = ImageOps.grayscale(image) + return ImageOps.colorize(gray_image, black="black", white="white") + else: + return image + + +def add_overlay_mask_if_active(init, image, is_tween: bool = False): + is_use_overlay = init.args.args.overlay_mask + is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask + if is_use_overlay and is_use_mask: + index = indexes.tween.i if is_tween else indexes.frame.i + is_bgr_array = True + return do_overlay_mask(init.args.args, init.args.anim_args, image, index, is_bgr_array) + else: + return image + + +def save_cadence_frame(init, indexes, image): + filename = tween_frame_name(init, indexes) + save_path: str = os.path.join(init.args.args.outdir, filename) + cv2.imwrite(save_path, image) From 4fbe9d2d3582cf340a1e9a1d2c4606606489849d Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 21:34:25 +0200 Subject: [PATCH 051/132] Image transformation step isolation. --- scripts/deforum_helpers/render.py | 66 +++------------ .../rendering/data/step/step.py | 84 +++++++++++++++++++ 2 files changed, 96 insertions(+), 54 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index db7e62f02..2f6fa9289 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -25,21 +25,18 @@ from modules.shared import opts, state from .colors import maintain_colors -from .hybrid_video import image_transform_ransac, image_transform_optical_flow from .prompt import prepare_prompt from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils -from .rendering.util.call.anim import call_anim_frame_warp from .rendering.util.call.gen import call_generate -from .rendering.util.call.hybrid import ( - call_get_flow_from_images, call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, - call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) +from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .rendering.util.call.images import call_add_noise, call_get_mask_from_file_with_frame from .rendering.util.call.mask import call_compose_mask_with_check, call_unsharp_mask from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame -from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame +from .rendering.util.image_utils import ( + add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame) from .rendering.util.log_utils import ( print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, print_redo_generation_info) @@ -73,6 +70,7 @@ def run_render_animation(init): if turbo.is_emit_in_between_frames(): # TODO extract tween frame emition indexes.update_tween_start(turbo) for tween_index in range(indexes.tween.start, indexes.frame.i): + indexes.update_tween(tween_index) web_ui_utils.update_progress_during_cadence(init, indexes) tween_step = TweenStep.create(indexes) @@ -99,56 +97,16 @@ def run_render_animation(init): dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) init.depth_model.save(dm_save_path, step.depth) - # get color match for video outside of images.previous conditional - if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(preview_video_image_path(init, indexes)) - prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) - images.color_match = np.asarray(prev_vid_img) - images.color_match = cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) - + images.color_match = Step.create_color_match_for_video(init, indexes) # after 1st frame, images.previous exists if images.previous is not None: - # apply transforms to previous frame - images.previous, step.depth = call_anim_frame_warp(init, indexes.frame.i, images.previous, None) - - # do hybrid compositing before motion - if init.is_hybrid_composite_before_motion(): - _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, - step.init.hybrid_comp_schedules) - - # hybrid video motion - warps images.previous to match motion, usually to prepare for compositing - with context(init.args.anim_args) as aa: - if aa.hybrid_motion in ['Affine', 'Perspective']: - if aa.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.frame.i - 1, images.previous) - else: - matrix = call_get_matrix_for_hybrid_motion(init, indexes.frame.i - 1) - images.previous = image_transform_ransac(images.previous, matrix, aa.hybrid_motion) - if aa.hybrid_motion in ['Optical Flow']: - if aa.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, indexes.frame.i - 1, images.previous) - else: - flow = call_get_flow_for_hybrid_motion(init, indexes.frame.i - 1) - images.previous = image_transform_optical_flow(images.previous, flow, step.init.flow_factor()) - init.animation_mode.prev_flow = flow - - # do hybrid compositing after motion (normal) - if init.is_normal_hybrid_composite(): - _, images.previous = call_hybrid_composite(init, indexes.frame.i, images.previous, - step.init.hybrid_comp_schedules) - # apply color matching - if init.has_color_coherence(): - if images.color_match is None: - images.color_match = images.previous.copy() - else: - images.previous = maintain_colors(images.previous, images.color_match, - init.args.anim_args.color_coherence) - - # intercept and override to grayscale - if init.args.anim_args.color_force_grayscale: - images.previous = cv2.cvtColor(images.previous, cv2.COLOR_BGR2GRAY) - images.previous = cv2.cvtColor(images.previous, cv2.COLOR_GRAY2BGR) + images.previous = step.apply_frame_warp_transform(init, indexes, images.previous) + images.previous = step.do_hybrid_compositing_before_motion(init, indexes, images.previous) + images.previous = Step.apply_hybrid_motion_ransac_transform(init, indexes, images, images.previous) + images.previous = Step.apply_hybrid_motion_optical_flow(init, indexes, images, images.previous) + images.previous = step.do_normal_hybrid_compositing_after_motion(init, indexes, images.previous) + images.previous = Step.apply_color_matching(init, images, images.previous) + images.previous = Step.transform_to_grayscale_if_active(init, images, images.previous) # apply scaling contrast_image = (images.previous * step.init.contrast).round().astype(np.uint8) diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index d4717c05f..e3eca24e6 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -1,11 +1,19 @@ from dataclasses import dataclass from typing import Any +import cv2 + from ..initialization import RenderInit from ..schedule import Schedule from ..turbo import Turbo +from ...util.call.anim import call_anim_frame_warp +from ...util.call.hybrid import ( + call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, + call_get_matrix_for_hybrid_motion, + call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.utils import context +from ....hybrid_video import image_transform_ransac, image_transform_optical_flow @dataclass(init=True, frozen=True, repr=False, eq=False) @@ -88,3 +96,79 @@ def write_frame_subtitle_if_active(self, init, indexes, opt_utils): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) self.subtitle_params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) call_write_frame_subtitle(init, indexes.tween.i, params_string, sub_step.tween < 1.0) + + def apply_frame_warp_transform(self, init, indexes, image): + previous, self.depth = call_anim_frame_warp(init, indexes.frame.i, image, None) + return previous + + def _do_hybrid_compositing_on_cond(self, init, indexes, image, condition): + i = indexes.frame.i + schedules = self.init.hybrid_comp_schedules + if condition: + _, composed = call_hybrid_composite(init, i, image, schedules) + return composed + else: + return image + + def do_hybrid_compositing_before_motion(self, init, indexes, image): + condition = init.is_hybrid_composite_before_motion() + return self._do_hybrid_compositing_on_cond(init, indexes, image, condition) + + def do_normal_hybrid_compositing_after_motion(self, init, indexes, image): + condition = init.is_normal_hybrid_composite() + return self._do_hybrid_compositing_on_cond(init, indexes, image, condition) + + @staticmethod + def apply_color_matching(init, images, image): + if init.has_color_coherence(): + if images.color_match is None: + # TODO questionable + # initialize color_match for next iteration with current image, but don't do anything yet. + images.color_match = image.copy() + else: + return maintain_colors(image, images.color_match, init.args.anim_args.color_coherence) + return image + + @staticmethod + def transform_to_grayscale_if_active(init, images, image): + if init.args.anim_args.color_force_grayscale: + grayscale = cv2.cvtColor(images.previous, cv2.COLOR_BGR2GRAY) + return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) + else: + return image + + @staticmethod + def apply_hybrid_motion_ransac_transform(init, indexes, reference_images, image): + """hybrid video motion - warps images.previous to match motion, usually to prepare for compositing""" + motion = init.args.anim_args.hybrid_motion + if motion in ['Affine', 'Perspective']: + last_i = indexes.frame.i - 1 + matrix = call_get_matrix_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ + if init.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_matrix_for_hybrid_motion(init, last_i) + return image_transform_ransac(image, matrix, init.args.anim_args.hybrid_motion) + return image + + @staticmethod + def apply_hybrid_motion_optical_flow(init, indexes, reference_images, image): + motion = init.args.anim_args.hybrid_motion + if motion in ['Optical Flow']: + last_i = indexes.frame.i - 1 + flow = call_get_flow_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ + if init.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_flow_for_hybrid_motion(init, last_i) + transformed = image_transform_optical_flow(images.previous, flow, step.init.flow_factor()) + init.animation_mode.prev_flow = flow # side effect + return transformed + else: + return image + + @staticmethod + def create_color_match_for_video(init, indexes): + if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): + if int(indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + prev_vid_img = Image.open(preview_video_image_path(init, indexes)) + prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) + images.color_match = np.asarray(prev_vid_img) + return cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) + return None From 0bf186f38189d66ad6bcd98dd6ae67e649e41673 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 9 Jun 2024 22:45:51 +0200 Subject: [PATCH 052/132] Image transformation pipe. --- scripts/deforum_helpers/render.py | 21 ++++++++++++------- .../rendering/util/fun_utils.py | 17 +++++++++++++++ 2 files changed, 30 insertions(+), 8 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/util/fun_utils.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 2f6fa9289..bb7536462 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -17,7 +17,6 @@ import gc import os -import PIL import cv2 import numpy as np from PIL import Image @@ -35,6 +34,7 @@ from .rendering.util.call.images import call_add_noise, call_get_mask_from_file_with_frame from .rendering.util.call.mask import call_compose_mask_with_check, call_unsharp_mask from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame +from .rendering.util.fun_utils import pipe from .rendering.util.image_utils import ( add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame) from .rendering.util.log_utils import ( @@ -100,13 +100,7 @@ def run_render_animation(init): images.color_match = Step.create_color_match_for_video(init, indexes) # after 1st frame, images.previous exists if images.previous is not None: - images.previous = step.apply_frame_warp_transform(init, indexes, images.previous) - images.previous = step.do_hybrid_compositing_before_motion(init, indexes, images.previous) - images.previous = Step.apply_hybrid_motion_ransac_transform(init, indexes, images, images.previous) - images.previous = Step.apply_hybrid_motion_optical_flow(init, indexes, images, images.previous) - images.previous = step.do_normal_hybrid_compositing_after_motion(init, indexes, images.previous) - images.previous = Step.apply_color_matching(init, images, images.previous) - images.previous = Step.transform_to_grayscale_if_active(init, images, images.previous) + images.previous = image_transformation_pipe(init, indexes, step, images)(images.previous) # apply scaling contrast_image = (images.previous * step.init.contrast).round().astype(np.uint8) @@ -265,6 +259,17 @@ def emit_in_between_frames(): raise NotImplemented("") +def image_transformation_pipe(init, indexes, step, images): + # make sure `im` stays the last argument in each call. + return pipe(lambda im: step.apply_frame_warp_transform(init, indexes, im), + lambda im: step.do_hybrid_compositing_before_motion(init, indexes, im), + lambda im: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, im), + lambda im: Step.apply_hybrid_motion_optical_flow(init, indexes, images, im), + lambda im: step.do_normal_hybrid_compositing_after_motion(init, indexes, im), + lambda im: Step.apply_color_matching(init, images, im), + lambda im: Step.transform_to_grayscale_if_active(init, images, im)) + + def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: diff --git a/scripts/deforum_helpers/rendering/util/fun_utils.py b/scripts/deforum_helpers/rendering/util/fun_utils.py new file mode 100644 index 000000000..ab9c944bf --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/fun_utils.py @@ -0,0 +1,17 @@ +import collections.abc +from functools import reduce +from itertools import chain + + +def flat_map(func, iterable): + """Applies a function to each element in an iterable and flattens the results.""" + mapped_iterable = map(func, iterable) + if any(isinstance(item, collections.abc.Iterable) for item in mapped_iterable): + return chain.from_iterable(mapped_iterable) + else: + return mapped_iterable + + +def pipe(*funcs): + """Pipes a value through a sequence of functions""" + return lambda value: reduce(lambda x, f: f(x), funcs, value) From 3d2d9cbd1f81d615d95a0984c48ae709d7d9390e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 10 Jun 2024 00:31:06 +0200 Subject: [PATCH 053/132] SubSeed scheduling and transformation pipes for contrast- and noise images. --- scripts/deforum_helpers/render.py | 68 ++++++------------- .../rendering/data/initialization.py | 40 +++++++++++ .../rendering/data/step/step.py | 20 ++++++ 3 files changed, 80 insertions(+), 48 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index bb7536462..f6f2bd880 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -31,8 +31,8 @@ from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite -from .rendering.util.call.images import call_add_noise, call_get_mask_from_file_with_frame -from .rendering.util.call.mask import call_compose_mask_with_check, call_unsharp_mask +from .rendering.util.call.images import call_get_mask_from_file_with_frame +from .rendering.util.call.mask import call_compose_mask_with_check from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame from .rendering.util.fun_utils import pipe from .rendering.util.image_utils import ( @@ -98,53 +98,16 @@ def run_render_animation(init): init.depth_model.save(dm_save_path, step.depth) images.color_match = Step.create_color_match_for_video(init, indexes) - # after 1st frame, images.previous exists - if images.previous is not None: - images.previous = image_transformation_pipe(init, indexes, step, images)(images.previous) - - # apply scaling - contrast_image = (images.previous * step.init.contrast).round().astype(np.uint8) - # anti-blur - if step.init.amount > 0: - step.init.kernel_size() - contrast_image = call_unsharp_mask(init, step, contrast_image, mask) - # apply frame noising - if init.args.args.use_mask or init.args.anim_args.use_noise_mask: - init.root.noise_mask = call_compose_mask_with_check( - init, step.schedule.noise_mask_seq, mask.noise_vals, contrast_image) - - noised_image = call_add_noise(init, step, contrast_image) - - # use transformed previous frame as init for current - init.args.args.use_init = True - init.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) - init.args.args.strength = max(0.0, min(1.0, step.init.strength)) - - init.args.args.scale = step.init.scale - - # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints - init.args.args.pix2pix_img_cfg_scale = float( - init.animation_keys.deform_keys.pix2pix_img_cfg_scale_series[indexes.frame.i]) - - # grab prompt for current frame - init.args.args.prompt = init.prompt_series[indexes.frame.i] - with context(init.animation_keys.deform_keys) as keys: - if init.args.args.seed_behavior == 'schedule' or init.parseq_adapter.manages_seed(): - init.args.args.seed = int(keys.seed_schedule_series[indexes.frame.i]) - if init.args.anim_args.enable_checkpoint_scheduling: - init.args.args.checkpoint = keys.checkpoint_schedule_series[indexes.frame.i] - else: - init.args.args.checkpoint = None + if images.previous is not None: # skipping 1st iteration + images.previous = frame_image_transformation_pipe(init, indexes, step, images)(images.previous) + contrast_image = contrast_image_transformation_pipe(init, step, mask)(images.previous) + noised_image = noise_image_transformation_pipe(init, step)(contrast_image) + init.update_some_args_for_current_progression_step(step, noised_image) - # SubSeed scheduling - if init.args.anim_args.enable_subseed_scheduling: - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) - init.root.subseed_strength = float(keys.subseed_strength_schedule_series[indexes.frame.i]) - if init.parseq_adapter.manages_seed(): - init.args.anim_args.enable_subseed_scheduling = True - init.root.subseed = int(keys.subseed_schedule_series[indexes.frame.i]) - init.root.subseed_strength = keys.subseed_strength_schedule_series[indexes.frame.i] + init.update_some_args_for_current_step(indexes, step) + init.update_seed_and_checkpoint_for_current_step(indexes) + init.update_sub_seed_schedule_for_current_step(indexes) # set value back into the prompt - prepare and report prompt and seed init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, @@ -259,7 +222,7 @@ def emit_in_between_frames(): raise NotImplemented("") -def image_transformation_pipe(init, indexes, step, images): +def frame_image_transformation_pipe(init, indexes, step, images): # make sure `im` stays the last argument in each call. return pipe(lambda im: step.apply_frame_warp_transform(init, indexes, im), lambda im: step.do_hybrid_compositing_before_motion(init, indexes, im), @@ -270,6 +233,15 @@ def image_transformation_pipe(init, indexes, step, images): lambda im: Step.transform_to_grayscale_if_active(init, images, im)) +def contrast_image_transformation_pipe(init, step, mask): + return pipe(lambda im: step.apply_scaling(im), + lambda im: step.apply_anti_blur(init, mask, im)) + + +def noise_image_transformation_pipe(init, step): + return pipe(lambda im: step.apply_frame_noising(init, step, im)) + + def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index 13b2a7019..cecb19bf8 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -2,9 +2,11 @@ from dataclasses import dataclass from typing import Any +import cv2 import numexpr import numpy as np import pandas as pd +from PIL import Image from .anim import AnimationKeys, AnimationMode from .subtitle import Srt @@ -126,6 +128,44 @@ def _has_init_image_or_box(self) -> bool: def is_using_init_image_or_box(self) -> bool: return self.args.args.use_init and self._has_init_image_or_box() + def update_some_args_for_current_progression_step(self, step, noised_image): + # use transformed previous frame as init for current + self.args.args.use_init = True + self.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) + self.args.args.strength = max(0.0, min(1.0, step.init.strength)) + + def update_some_args_for_current_step(self, indexes, step): + i = indexes.frame.i + keys = self.animation_keys.deform_keys + # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints + self.args.args.pix2pix_img_cfg_scale = float(keys.pix2pix_img_cfg_scale_series[i]) + self.args.args.prompt = self.prompt_series[i] # grab prompt for current frame + self.args.args.scale = step.init.scale + + def update_seed_and_checkpoint_for_current_step(self, indexes): + i = indexes.frame.i + keys = self.animation_keys.deform_keys + is_seed_scheduled = self.args.args.seed_behavior == 'schedule' + is_seed_managed = self.parseq_adapter.manages_seed() + is_seed_scheduled_or_managed = is_seed_scheduled or is_seed_managed + if is_seed_scheduled_or_managed: + self.args.args.seed = int(keys.seed_schedule_series[i]) + self.args.args.checkpoint = keys.checkpoint_schedule_series[i] \ + if self.args.anim_args.enable_checkpoint_scheduling else None + + def update_sub_seed_schedule_for_current_step(self, indexes): + i = indexes.frame.i + keys = self.animation_keys.deform_keys + is_subseed_scheduling_enabled = self.args.anim_args.enable_subseed_scheduling + is_seed_managed_by_parseq = self.parseq_adapter.manages_seed() + if is_subseed_scheduling_enabled or is_seed_managed_by_parseq: + self.root.subseed = int(keys.subseed_schedule_series[i]) + if is_subseed_scheduling_enabled and not is_seed_managed_by_parseq: + self.root.subseed_strength = float(keys.subseed_strength_schedule_series[i]) + if is_seed_managed_by_parseq: + self.root.subseed_strength = keys.subseed_strength_schedule_series[i] # TODO not sure why not type-coerced. + self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. + @staticmethod def create_output_directory_for_the_batch(directory): os.makedirs(directory, exist_ok=True) diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index e3eca24e6..66f20eaaf 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -2,6 +2,7 @@ from typing import Any import cv2 +import numpy as np from ..initialization import RenderInit from ..schedule import Schedule @@ -11,6 +12,8 @@ call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) +from ...util.call.images import call_add_noise +from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.utils import context from ....hybrid_video import image_transform_ransac, image_transform_optical_flow @@ -118,6 +121,23 @@ def do_normal_hybrid_compositing_after_motion(self, init, indexes, image): condition = init.is_normal_hybrid_composite() return self._do_hybrid_compositing_on_cond(init, indexes, image, condition) + def apply_scaling(self, image): + return (image * self.init.contrast).round().astype(np.uint8) + + def apply_anti_blur(self, init, mask, image): + if self.init.amount > 0: + return call_unsharp_mask(init, self, image, mask) + else: + return image + + def apply_frame_noising(self, init, mask, image): + is_use_any_mask = init.args.args.use_mask or init.args.anim_args.use_noise_mask + if is_use_any_mask: + seq = self.schedule.noise_mask_seq + vals = mask.noise_vals + init.root.noise_mask = call_compose_mask_with_check(init, seq, vals, contrast_image) + return call_add_noise(init, self, image) + @staticmethod def apply_color_matching(init, images, image): if init.has_color_coherence(): From d06419bf0f82a0d1b053e157be69f5c98a496493 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 10 Jun 2024 01:10:08 +0200 Subject: [PATCH 054/132] Step preparation. --- scripts/deforum_helpers/render.py | 30 ++---------- .../rendering/data/initialization.py | 48 ++++++++++++++++++- 2 files changed, 51 insertions(+), 27 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index f6f2bd880..bd9feb15e 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -103,36 +103,14 @@ def run_render_animation(init): images.previous = frame_image_transformation_pipe(init, indexes, step, images)(images.previous) contrast_image = contrast_image_transformation_pipe(init, step, mask)(images.previous) noised_image = noise_image_transformation_pipe(init, step)(contrast_image) - init.update_some_args_for_current_progression_step(step, noised_image) + init.update_sample_and_args_for_current_progression_step(step, noised_image) init.update_some_args_for_current_step(indexes, step) init.update_seed_and_checkpoint_for_current_step(indexes) init.update_sub_seed_schedule_for_current_step(indexes) - - # set value back into the prompt - prepare and report prompt and seed - init.args.args.prompt = prepare_prompt(init.args.args.prompt, init.args.anim_args.max_frames, - init.args.args.seed, indexes.frame.i) - # grab init image for current frame - if init.animation_mode.has_video_input: - init_frame = call_get_next_frame(init, indexes.frame.i, init.args.anim_args.video_init_path) - print_init_frame_info(init_frame) - init.args.args.init_image = init_frame - init.args.args.init_image_box = None # init_image_box not used in this case - init.args.args.strength = max(0.0, min(1.0, step.init.strength)) - if init.args.anim_args.use_mask_video: - mask_init_frame = call_get_next_frame(init, indexes.frame.i, init.args.anim_args.video_mask_path, True) - temp_mask = call_get_mask_from_file_with_frame(init, mask_init_frame) - init.args.args.mask_file = temp_mask - init.root.noise_mask = temp_mask - mask.vals['video_mask'] = temp_mask - - if init.args.args.use_mask: - init.args.args.mask_image = call_compose_mask_with_check( - init, step.schedule.mask_seq, mask.vals, init.root.init_sample) - init.args.args.mask_image = compose_mask_with_check(init.root, init.args.args, step.schedule.mask_seq, - mask.vals, init.root.init_sample) \ - if init.root.init_sample is not None else None # we need it only after the first frame anyway - + init.prompt_for_current_step(indexes) + init.update_video_data_for_current_frame(indexes, step) + init.update_mask_image(step, mask) init.animation_keys.update(indexes.frame.i) opt_utils.setup(init, step.schedule) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index cecb19bf8..eedd82fb7 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -11,12 +11,15 @@ from .anim import AnimationKeys, AnimationMode from .subtitle import Srt from ..util import memory_utils +from ..util.call.mask import call_compose_mask_with_check +from ..util.call.video_and_audio import call_get_next_frame from ..util.utils import context from ...args import RootArgs from ...deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ...depth import DepthModel from ...generate import (isJson) from ...parseq_adapter import ParseqAdapter +from ...prompt import prepare_prompt from ...settings import save_settings_from_animation_run @@ -128,7 +131,7 @@ def _has_init_image_or_box(self) -> bool: def is_using_init_image_or_box(self) -> bool: return self.args.args.use_init and self._has_init_image_or_box() - def update_some_args_for_current_progression_step(self, step, noised_image): + def update_sample_and_args_for_current_progression_step(self, step, noised_image): # use transformed previous frame as init for current self.args.args.use_init = True self.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) @@ -166,6 +169,49 @@ def update_sub_seed_schedule_for_current_step(self, indexes): self.root.subseed_strength = keys.subseed_strength_schedule_series[i] # TODO not sure why not type-coerced. self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. + def prompt_for_current_step(self, indexes): + """returns value to be set back into the prompt""" + prompt = self.args.args.prompt + max_frames = self.args.anim_args.max_frames + seed = self.args.args.seed + i = indexes.frame.i + return prepare_prompt(prompt, max_frames, seed, i) + + def _update_video_input_for_current_frame(self, i, step): + video_init_path = self.args.anim_args.video_init_path + init_frame = call_get_next_frame(init, i, video_init_path) + print_init_frame_info(init_frame) + self.args.args.init_image = init_frame + self.args.args.init_image_box = None # init_image_box not used in this case + self.args.args.strength = max(0.0, min(1.0, step.init.strength)) + + def _update_video_mask_for_current_frame(self, i): + video_mask_path = self.args.anim_args.video_mask_path + is_mask = True + mask_init_frame = call_get_next_frame(self, i, video_mask_path, is_mask) + new_mask = call_get_mask_from_file_with_frame(self, mask_init_frame) + self.args.args.mask_file = new_mask + self.root.noise_mask = new_mask + mask.vals['video_mask'] = new_mask + + def update_video_data_for_current_frame(self, indexes, step): + i = indexes.frame.i + if self.animation_mode.has_video_input: + self._update_video_input_for_current_frame(i, step) + if self.args.anim_args.use_mask_video: + self._update_video_mask_for_current_frame(i) + + def update_mask_image(self, step, mask): + is_use_mask = self.args.args.use_mask + if is_use_mask: + has_sample = self.root.init_sample is not None + if has_sample: + mask_seq = step.schedule.mask_seq + sample = init.root.init_sample + self.args.args.mask_image = call_compose_mask_with_check(self, mask_seq, mask.vals, sample) + else: + self.args.args.mask_image = None # we need it only after the first frame anyway + @staticmethod def create_output_directory_for_the_batch(directory): os.makedirs(directory, exist_ok=True) From ae820289288d887da28db3cc9e8b30a51bc75c48 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 10 Jun 2024 03:06:45 +0200 Subject: [PATCH 055/132] Generated update for readme. --- README.md | 35 ++++++++++++++++++++++++------- scripts/deforum_helpers/render.py | 7 ++----- 2 files changed, 29 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index e8e9b13b7..a78488cfa 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,33 @@ -# Experimental -Experimental fork of the Deforum A1111 extension with the intention of making "turbo-frames" controllable directly. -Will require an experimental version of Parseq to supply new "is-turbo-frame" values, fork here: https://github.com/Tok/sd-parseq +## Experimental Deforum Fork -This is an experiment in progress and there's no good reason to install this as of now. -Check out the original project instead: https://github.com/deforum-art/sd-webui-deforum +This is an experimental fork of the Deforum A1111 extension with the goal of refactoring the core and allow for direct control of "turbo-frames". +It will eventually also require an experimental version of Parseq to supply new "is_turbo_frame" values, future fork here: https://github.com/Tok/sd-parseq +**Current Status:** +This is a work in progress, and installation is not recommended yet. +Please refer to the original project for a stable version: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) +## Neo-Core -## License +This section details the changes made to the render core in this fork. -This program is distributed under the terms of the GNU Affero Public License v3.0, copyright (c) 2023 Deforum LLC. +For easy integration, this fork isolates changes to the `render.py` module and introduces the `rendering` package. +Existing code in all other Deforum modules remains untouched. -Some of its sublicensed integrated 3rd party components may have other licenses, see LICENSE for usage terms. +* **Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. +* **Key Improvements:** + * Reduced cyclomatic complexity of the `render_animation` method. + * Improved separation of concerns (e.g., dedicated module for printing). + * Reduced argument complexity and improved scope control and variable organization using dataclasses. + * Enhanced code clarity with improved naming and removal of obsolete comments. + * But preserving domain specific lingo. + * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. + +**New Rendering Modules:** +* `rendering/data`: Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. +* `rendering/util`: Contains stateless utility functions for data transformation and manipulation used in `render.py`. +* `rendering/util/call`: Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. + +**Implementation Details:** +* **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. +* **Style and Standards:** The code adheres to PEP 8 style guidelines and to other such practices. diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index bd9feb15e..09d372edf 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -24,21 +24,18 @@ from modules.shared import opts, state from .colors import maintain_colors -from .prompt import prepare_prompt from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite -from .rendering.util.call.images import call_get_mask_from_file_with_frame -from .rendering.util.call.mask import call_compose_mask_with_check -from .rendering.util.call.video_and_audio import call_render_preview, call_get_next_frame +from .rendering.util.call.video_and_audio import call_render_preview from .rendering.util.fun_utils import pipe from .rendering.util.image_utils import ( add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame) from .rendering.util.log_utils import ( - print_animation_frame_info, print_tween_frame_info, print_init_frame_info, print_optical_flow_info, + print_animation_frame_info, print_tween_frame_info, print_optical_flow_info, print_redo_generation_info) from .rendering.util.utils import context from .save_images import save_image From fd52bffece4b97230ac9c77dc335c1788f81c711 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 11 Jun 2024 01:30:56 +0200 Subject: [PATCH 056/132] Extracted tween frame emission and moved random seed generation to utility module. --- scripts/deforum_helpers/render.py | 70 +++++++++---------- .../rendering/data/step/step.py | 3 +- .../rendering/data/step/tween_step.py | 39 +++++++++-- .../deforum_helpers/rendering/data/turbo.py | 5 +- .../rendering/util/__init__.py | 2 +- .../rendering/util/image_utils.py | 7 ++ .../rendering/util/log_utils.py | 5 +- .../deforum_helpers/rendering/util/utils.py | 6 ++ 8 files changed, 86 insertions(+), 51 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 09d372edf..9aeb11e32 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -27,16 +27,13 @@ from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep -from .rendering.util import memory_utils, filename_utils, opt_utils, web_ui_utils +from .rendering.util import generate_random_seed, memory_utils, filename_utils, opt_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .rendering.util.call.video_and_audio import call_render_preview from .rendering.util.fun_utils import pipe -from .rendering.util.image_utils import ( - add_overlay_mask_if_active, force_to_grayscale_if_required, save_cadence_frame) -from .rendering.util.log_utils import ( - print_animation_frame_info, print_tween_frame_info, print_optical_flow_info, - print_redo_generation_info) +from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required +from .rendering.util.log_utils import print_animation_frame_info, print_optical_flow_info, print_redo_generation_info from .rendering.util.utils import context from .save_images import save_image from .seed import next_seed @@ -64,35 +61,9 @@ def run_render_animation(init): web_ui_utils.update_job(init, indexes) step = Step.create(init, indexes) step.write_frame_subtitle(init, indexes, turbo) - if turbo.is_emit_in_between_frames(): # TODO extract tween frame emition - indexes.update_tween_start(turbo) - for tween_index in range(indexes.tween.start, indexes.frame.i): - indexes.update_tween(tween_index) - web_ui_utils.update_progress_during_cadence(init, indexes) - tween_step = TweenStep.create(indexes) - - turbo.do_optical_flow_cadence_setup_before_animation_warping(init, tween_step) - - step.write_frame_subtitle_if_active(init, indexes, opt_utils) - print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) - - step.update_depth_prediction(init, turbo) - turbo.advance(init, indexes.tween.i, step.depth) - - turbo.do_hybrid_video_motion(init, indexes, images) - - img = tween_step.generate_tween_image(init, indexes, step, turbo) - images.previous = img # get images.previous during cadence - - # current image update for cadence frames (left commented because it doesn't currently update the preview) - # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) - - save_cadence_frame(init, indexes, img) - - if init.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) - init.depth_model.save(dm_save_path, step.depth) + if turbo.is_emit_in_between_frames(): + emit_in_between_frames(init, indexes, step, turbo, images) images.color_match = Step.create_color_match_for_video(init, indexes) @@ -118,7 +89,7 @@ def run_render_animation(init): # optical flow redo before generation if optical_flow_redo_generation != 'None' and images.previous is not None and step.init.strength > 0: stored_seed = init.args.args.seed - init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere + init.args.args.seed = generate_random_seed() print_optical_flow_info(init, optical_flow_redo_generation) with context(call_generate(init, indexes.frame.i, step.schedule)) as img: img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) @@ -138,7 +109,7 @@ def run_render_animation(init): stored_seed = init.args.args.seed for n in range(0, int(init.args.anim_args.diffusion_redo)): print_redo_generation_info(init, n) - init.args.args.seed = random.randint(0, 2 ** 32 - 1) # TODO move elsewhere + init.args.args.seed = generate_random_seed() disposable_image = call_generate(init, indexes.frame.i, step.schedule) disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) # color match on last one only @@ -193,8 +164,31 @@ def run_render_animation(init): init.animation_mode.unload_raft_and_depth_model() -def emit_in_between_frames(): - raise NotImplemented("") +def emit_in_between_frames(init, indexes, step, turbo, images): + tween_frame_start_i = max(indexes.frame.start, indexes.frame.i - turbo.steps) + emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, indexes.frame.i) + + +def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, frame_i): + """Emits tween frames (also known as turbo- or cadence-frames) between the provided indicis.""" + # TODO refactor until this works with just 2 args: RenderInit and a collection of immutable TweenStep objects. + indexes.update_tween_start(turbo) + + tween_range = range(tween_frame_start_i, frame_i) + + # TODO Instead of indexes, pass a set of TweenStep objects to be processed instead of creating them in the loop. + for tween_index in tween_range: + # TODO tween index shouldn't really be updated and passed around like this here. + # ideally provide index data within an immutable TweenStep instance. + indexes.update_tween(tween_index) # TODO Nope + + tween_step = TweenStep.create(indexes) + + TweenStep.handle_synchronous_status_concerns(init, indexes, step, tween_step) + TweenStep.process(init, indexes, step, turbo, images, tween_step) + new_image = TweenStep.generate_and_save_frame(init, indexes, step, turbo, tween_step) + + images.previous = new_image # TODO shouldn't def frame_image_transformation_pipe(init, indexes, step, images): diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 66f20eaaf..f520e9e29 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -7,6 +7,7 @@ from ..initialization import RenderInit from ..schedule import Schedule from ..turbo import Turbo +from ...util import opt_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.hybrid import ( call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, @@ -94,7 +95,7 @@ def write_frame_subtitle(self, init, indexes, turbo): self.subtitle_params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) call_write_frame_subtitle(init, indexes.frame.i, params_string) - def write_frame_subtitle_if_active(self, init, indexes, opt_utils): + def write_frame_subtitle_if_active(self, init, indexes): if opt_utils.is_generate_subtitles(init): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) self.subtitle_params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 0cd4f9412..e50a29d2d 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -1,7 +1,7 @@ from dataclasses import dataclass from typing import Any -from ...util.image_utils import add_overlay_mask_if_active, force_tween_to_grayscale_if_required +from ...util import image_utils, log_utils, web_ui_utils @dataclass(init=True, frozen=False, repr=False, eq=False) @@ -11,16 +11,43 @@ class TweenStep: cadence_flow: Any cadence_flow_inc: Any + @staticmethod + def _calculate_tween_from_indices(frame_difference, last_step) -> float: + return min(0.0, max(1.0, float(last_step) / float(frame_difference))) + @staticmethod def create(indexes): - from_i = indexes.frame.i - indexes.tween.start - to_i = indexes.tween.i - indexes.tween.start + 1 - tween = float(to_i) / float(from_i) + tween = float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start) + return TweenStep(tween, None, None) + + @staticmethod + def create_directly(from_index, to_index): + tween = TweenStep._calculate_tween_from_indices(from_index, to_index) return TweenStep(tween, None, None) def generate_tween_image(self, init, indexes, step, turbo): is_tween = True warped = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, self) - recolored = force_tween_to_grayscale_if_required(init, warped) - masked = add_overlay_mask_if_active(init, recolored, is_tween) + recolored = image_utils.force_tween_to_grayscale_if_required(init, warped) + masked = image_utils.add_overlay_mask_if_active(init, recolored, is_tween) return masked + + @staticmethod + def process(init, indexes, step, turbo, images, tween_step): + turbo.advance_optical_flow_cadence_before_animation_warping(init, tween_step) + step.update_depth_prediction(init, turbo) + turbo.advance(init, indexes.tween.i, step.depth) + turbo.do_hybrid_video_motion(init, indexes, images) + + @staticmethod + def generate_and_save_frame(init, indexes, step, turbo, tween_step, is_do_save: bool = True): + new_image = tween_step.generate_tween_image(init, indexes, step, turbo) + if is_do_save: + image_utils.save_cadence_frame_and_depth_map_if_active(init, indexes, new_image) + return new_image + + @staticmethod + def handle_synchronous_status_concerns(init, indexes, step, tween_step): + step.write_frame_subtitle_if_active(init, indexes) # TODO decouple from execution and calc all in advance? + log_utils.print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) + web_ui_utils.update_progress_during_cadence(init, indexes) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 583df2400..d01a1828b 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -91,7 +91,7 @@ def advance_hybrid_motion_ransac_trasform(self, init, indexes, reference_images) matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween.i - 1) turbo.advance_ransac_trasform(init, matrix) - def do_optical_flow_cadence_setup_before_animation_warping(self, init, tween_step): + def advance_optical_flow_cadence_before_animation_warping(self, init, tween_step): if init.is_3d_or_2d() and init.has_optical_flow_cadence(): has_tween_schedule = init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0 has_images = self.prev.image is not None and self.next.image is not None @@ -106,7 +106,8 @@ def do_optical_flow_cadence_setup_before_animation_warping(self, init, tween_ste def do_optical_flow_cadence_after_animation_warping(self, init, indexes, step, tween_step): if tween_step.cadence_flow is not None: - temp_flow = abs_flow_to_rel_flow(tween_step.flow, init.width(), init.height()) + # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). + temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, init.width(), init.height()) new_flow, _ = call_anim_frame_warp(init, indexes.tween.i, temp_flow, step.depth) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, init.width(), init.height()) diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 9f91a462e..fda732d10 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,3 +1,3 @@ # All modules in this package are intended to not hold or change any state. # Only use static class methods or loose defs that don't change anything in the arguments passed. -from .utils import call_or_use_on_cond, combo_context, context, put_all, put_if_present +from .utils import call_or_use_on_cond, combo_context, context, generate_random_seed, put_all, put_if_present diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index 95c09c251..3279f73c8 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -37,3 +37,10 @@ def save_cadence_frame(init, indexes, image): filename = tween_frame_name(init, indexes) save_path: str = os.path.join(init.args.args.outdir, filename) cv2.imwrite(save_path, image) + + +def save_cadence_frame_and_depth_map_if_active(init, indexes, image): + save_cadence_frame(init, indexes, image) + if init.args.anim_args.save_depth_maps: + dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) + init.depth_model.save(dm_save_path, step.depth) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 2a82d8a65..2375604ed 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -5,9 +5,8 @@ def print_animation_frame_info(init, indexes): def print_tween_frame_info(init, indexes, cadence_flow, tween): - msg_flow_name = '' if cadence_flow is None \ - else init.args.anim_args.optical_flow_cadence + ' optical flow ' - msg_frame_info = f"cadence frame: {indexes.tween.i}; tween:{tween:0.2f};" + msg_flow_name = '' if cadence_flow is None else init.args.anim_args.optical_flow_cadence + ' optical flow ' + msg_frame_info = f"cadence frame: {indexes.tween.i}; tween: {tween:0.2f};" print(f"Creating in-between {msg_flow_name}{msg_frame_info}") diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index f3b44ca47..4c3f628cd 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,4 +1,6 @@ +import random from contextlib import contextmanager + from PIL import Image @@ -55,3 +57,7 @@ def call_or_use_on_cond(condition, callable_or_value): def create_img(dimensions): return Image.new('1', dimensions, 1) + + +def generate_random_seed(): + return random.randint(0, 2 ** 32 - 1) From c939f2d900794eab9d4b17c6f86a31b408cf6ed6 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 11 Jun 2024 22:46:51 +0200 Subject: [PATCH 057/132] Prepared conditions. Removed direct root from step and added more typehints. Renamed "pipe" function to "tube" so it's not causing any confusion with pytorch pipelines. --- scripts/deforum_helpers/render.py | 32 +++--- .../deforum_helpers/rendering/data/images.py | 3 + .../rendering/data/initialization.py | 102 +++++++++++------- .../deforum_helpers/rendering/data/mask.py | 4 +- .../rendering/data/step/step.py | 7 +- .../rendering/util/call/images.py | 2 +- .../rendering/util/call/mask.py | 2 +- .../rendering/util/filename_utils.py | 2 +- .../rendering/util/fun_utils.py | 4 +- .../rendering/util/memory_utils.py | 4 +- .../rendering/util/web_ui_utils.py | 2 +- 11 files changed, 100 insertions(+), 64 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 9aeb11e32..c7d64f330 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -31,7 +31,7 @@ from .rendering.util.call.gen import call_generate from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .rendering.util.call.video_and_audio import call_render_preview -from .rendering.util.fun_utils import pipe +from .rendering.util.fun_utils import tube from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required from .rendering.util.log_utils import print_animation_frame_info, print_optical_flow_info, print_redo_generation_info from .rendering.util.utils import context @@ -68,9 +68,9 @@ def run_render_animation(init): images.color_match = Step.create_color_match_for_video(init, indexes) if images.previous is not None: # skipping 1st iteration - images.previous = frame_image_transformation_pipe(init, indexes, step, images)(images.previous) - contrast_image = contrast_image_transformation_pipe(init, step, mask)(images.previous) - noised_image = noise_image_transformation_pipe(init, step)(contrast_image) + images.previous = frame_image_transformation_tube(init, indexes, step, images)(images.previous) + contrast_image = contrast_image_transformation_tube(init, step, mask)(images.previous) + noised_image = noise_image_transformation_tube(init, step)(contrast_image) init.update_sample_and_args_for_current_progression_step(step, noised_image) init.update_some_args_for_current_step(indexes, step) @@ -97,7 +97,7 @@ def run_render_animation(init): img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) img = image_transform_optical_flow(img, disposable_flow, step.init.redo_flow_factor) init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds - init.root.init_sample = Image.fromarray(img) + init.args.root.init_sample = Image.fromarray(img) disposable_image = img # TODO refactor del (img, disposable_flow, stored_seed) gc.collect() @@ -117,14 +117,14 @@ def run_render_animation(init): disposable_image = maintain_colors(images.previous, images.color_match, init.args.anim_args.color_coherence) init.args.args.seed = stored_seed - init.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) + init.args.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) del (disposable_image, stored_seed) gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. # generation image = call_generate(init, indexes.frame.i, step.schedule) - if image is None: + if image is None: # TODO throw error or log warning or something break # do hybrid video after generation @@ -158,7 +158,7 @@ def run_render_animation(init): state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal - init.args.args.seed = next_seed(init.args.args, init.root) # TODO group all seeds and sub-seeds + init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO group all seeds and sub-seeds last_preview_frame = call_render_preview(init, indexes.frame.i, last_preview_frame) web_ui_utils.update_status_tracker(init, indexes) init.animation_mode.unload_raft_and_depth_model() @@ -191,9 +191,9 @@ def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_fra images.previous = new_image # TODO shouldn't -def frame_image_transformation_pipe(init, indexes, step, images): +def frame_image_transformation_tube(init, indexes, step, images): # make sure `im` stays the last argument in each call. - return pipe(lambda im: step.apply_frame_warp_transform(init, indexes, im), + return tube(lambda im: step.apply_frame_warp_transform(init, indexes, im), lambda im: step.do_hybrid_compositing_before_motion(init, indexes, im), lambda im: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, im), lambda im: Step.apply_hybrid_motion_optical_flow(init, indexes, images, im), @@ -202,20 +202,20 @@ def frame_image_transformation_pipe(init, indexes, step, images): lambda im: Step.transform_to_grayscale_if_active(init, images, im)) -def contrast_image_transformation_pipe(init, step, mask): - return pipe(lambda im: step.apply_scaling(im), +def contrast_image_transformation_tube(init, step, mask): + return tube(lambda im: step.apply_scaling(im), lambda im: step.apply_anti_blur(init, mask, im)) -def noise_image_transformation_pipe(init, step): - return pipe(lambda im: step.apply_frame_noising(init, step, im)) +def noise_image_transformation_tube(init, step): + return tube(lambda im: step.apply_frame_noising(init, step, im)) def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: memory_utils.handle_vram_before_depth_map_generation(init) - depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.root.half_precision) + depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.args.root.half_precision) depth_filename = filename_utils.depth_frame(init, idx) init.depth_model.save(os.path.join(init.output_directory, depth_filename), depth) memory_utils.handle_vram_after_depth_map_generation(init) @@ -228,6 +228,6 @@ def progress_step(init, idx, turbo, opencv_image, image, depth): return idx.frame.i + turbo.progress_step(idx, opencv_image), depth else: filename = filename_utils.frame(init, idx) - save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.root) + save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) depth = generate_depth_maps_if_active(init) return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. diff --git a/scripts/deforum_helpers/rendering/data/images.py b/scripts/deforum_helpers/rendering/data/images.py index 9cec751dc..4471b5b3f 100644 --- a/scripts/deforum_helpers/rendering/data/images.py +++ b/scripts/deforum_helpers/rendering/data/images.py @@ -9,6 +9,9 @@ class Images: previous: MatLike = None color_match: MatLike = None + def has_previous(self): + return self.previous is not None + @staticmethod def _load_color_match_sample(init) -> MatLike: """get color match for 'Image' color coherence only once, before loop""" diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index eedd82fb7..a2b5954f8 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -14,7 +14,7 @@ from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame from ..util.utils import context -from ...args import RootArgs +from ...args import DeforumArgs, DeforumAnimArgs, LoopArgs, ParseqArgs, RootArgs from ...deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ...depth import DepthModel from ...generate import (isJson) @@ -25,14 +25,14 @@ @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInitArgs: - args: Any = None - parseq_args: Any = None - anim_args: Any = None + args: DeforumArgs = None + parseq_args: ParseqArgs = None + anim_args: DeforumAnimArgs = None video_args: Any = None controlnet_args: Any = None - loop_args: Any = None + loop_args: LoopArgs = None opts: Any = None - root: Any = None + root: RootArgs = None @staticmethod def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): @@ -42,7 +42,6 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" - root: RootArgs seed: int args: RenderInitArgs parseq_adapter: Any @@ -54,6 +53,29 @@ class RenderInit: output_directory: str is_use_mask: bool + @staticmethod + def create(args, parseq_args, anim_args, video_args, controlnet_args, + loop_args, opts, root) -> 'RenderInit': + ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + output_directory = args.outdir + is_use_mask = args.use_mask + with context(RenderInit) as RI: + parseq_adapter = RI.create_parseq_adapter(ri_args) + srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) + animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) + animation_mode = AnimationMode.from_args(ri_args) + prompt_series = RI.select_prompts(parseq_adapter, anim_args, animation_keys, root) + depth_model = RI.create_depth_model_and_enable_depth_map_saving_if_active( + animation_mode, root, anim_args, args) + instance = RenderInit(args.seed, ri_args, parseq_adapter, srt, animation_keys, + animation_mode, prompt_series, depth_model, output_directory, is_use_mask) + RI.init_looper_if_active(args, loop_args) + RI.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + RI.create_output_directory_for_the_batch(args.outdir) + RI.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + RI.maybe_resume_from_timestring(anim_args, root) + return instance + def is_3d(self): return self.args.anim_args.animation_mode == '3D' @@ -107,6 +129,9 @@ def is_color_match_to_be_initialized(self, color_match_sample): def has_color_coherence(self): return self.args.anim_args.color_coherence != 'None' + def has_non_video_or_image_color_coherence(self): + return self.args.anim_args.color_coherence not in ['Image', 'Video Input'] + def is_resuming_from_timestring(self): return self.args.anim_args.resume_from_timestring @@ -131,10 +156,37 @@ def _has_init_image_or_box(self) -> bool: def is_using_init_image_or_box(self) -> bool: return self.args.args.use_init and self._has_init_image_or_box() + def is_not_in_motion_preview_mode(self): + return not self.args.args.motion_preview_mode + + def color_coherence_mode(self): + return self.args.anim_args.color_coherence + + def diffusion_redo(self): + return self.args.anim_args.diffusion_redo + + def diffusion_redo_as_int(self): + return int(self.diffusion_redo()) + + def optical_flow_redo_generation(self): + return self.args.anim_args.optical_flow_redo_generation + + def optical_flow_redo_generation_steps_if_not_in_preview_mode(self): + is_not_preview = self.is_not_in_motion_preview_mode() + return self.optical_flow_redo_generation() if is_not_preview else None + + def is_do_color_match_conversion(self, step): + is_legacy_cm = self.args.anim_args.legacy_colormatch + is_use_init = self.args.args.use_init + is_not_legacy_with_use_init = not is_legacy_cm and not is_use_init + is_legacy_cm_without_strength = is_legacy_cm and step.init.strength == 0 + is_maybe_special_legacy = is_not_legacy_with_use_init or is_legacy_cm_without_strength + return is_maybe_special_legacy and self.has_non_video_or_image_color_coherence() + def update_sample_and_args_for_current_progression_step(self, step, noised_image): # use transformed previous frame as init for current self.args.args.use_init = True - self.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) + self.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) self.args.args.strength = max(0.0, min(1.0, step.init.strength)) def update_some_args_for_current_step(self, indexes, step): @@ -162,11 +214,11 @@ def update_sub_seed_schedule_for_current_step(self, indexes): is_subseed_scheduling_enabled = self.args.anim_args.enable_subseed_scheduling is_seed_managed_by_parseq = self.parseq_adapter.manages_seed() if is_subseed_scheduling_enabled or is_seed_managed_by_parseq: - self.root.subseed = int(keys.subseed_schedule_series[i]) + self.args.root.subseed = int(keys.subseed_schedule_series[i]) if is_subseed_scheduling_enabled and not is_seed_managed_by_parseq: - self.root.subseed_strength = float(keys.subseed_strength_schedule_series[i]) + self.args.root.subseed_strength = float(keys.subseed_strength_schedule_series[i]) if is_seed_managed_by_parseq: - self.root.subseed_strength = keys.subseed_strength_schedule_series[i] # TODO not sure why not type-coerced. + self.args.root.subseed_strength = keys.subseed_strength_schedule_series[i] # TODO not sure why not type-coerced. self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. def prompt_for_current_step(self, indexes): @@ -191,7 +243,7 @@ def _update_video_mask_for_current_frame(self, i): mask_init_frame = call_get_next_frame(self, i, video_mask_path, is_mask) new_mask = call_get_mask_from_file_with_frame(self, mask_init_frame) self.args.args.mask_file = new_mask - self.root.noise_mask = new_mask + self.args.root.noise_mask = new_mask mask.vals['video_mask'] = new_mask def update_video_data_for_current_frame(self, indexes, step): @@ -204,10 +256,10 @@ def update_video_data_for_current_frame(self, indexes, step): def update_mask_image(self, step, mask): is_use_mask = self.args.args.use_mask if is_use_mask: - has_sample = self.root.init_sample is not None + has_sample = self.args.root.init_sample is not None if has_sample: mask_seq = step.schedule.mask_seq - sample = init.root.init_sample + sample = init.args.root.init_sample self.args.args.mask_image = call_compose_mask_with_check(self, mask_seq, mask.vals, sample) else: self.args.args.mask_image = None # we need it only after the first frame anyway @@ -284,25 +336,3 @@ def save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, def maybe_resume_from_timestring(anim_args, root): root.timestring = anim_args.resume_timestring if anim_args.resume_from_timestring else root.timestring - @staticmethod - def create(args, parseq_args, anim_args, video_args, controlnet_args, - loop_args, opts, root) -> 'RenderInit': - ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - output_directory = args.outdir - is_use_mask = args.use_mask - with context(RenderInit) as RI: - parseq_adapter = RI.create_parseq_adapter(ri_args) - srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) - animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) - animation_mode = AnimationMode.from_args(ri_args) - prompt_series = RI.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RI.create_depth_model_and_enable_depth_map_saving_if_active( - animation_mode, root, anim_args, args) - instance = RenderInit(root, args.seed, ri_args, parseq_adapter, srt, animation_keys, - animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - RI.init_looper_if_active(args, loop_args) - RI.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - RI.create_output_directory_for_the_batch(args.outdir) - RI.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - RI.maybe_resume_from_timestring(anim_args, root) - return instance diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index 62e578f1a..9e27a1062 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -28,7 +28,7 @@ def assign_masks(init, i, is_mask_image, dicts): if init.args.anim_args.use_mask_video: mask = call_get_mask_from_file(init, i, True) init.args.args.mask_file = mask - init.root.noise_mask = mask + init.args.root.noise_mask = mask put_all(dicts, key, mask) elif is_mask_image is None and init.is_use_mask: put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask @@ -46,7 +46,7 @@ def _assign(init, i, is_mask_image, dicts): if init.args.anim_args.use_mask_video: mask = call_get_mask_from_file(init, i, True) init.args.args.mask_file = mask - init.root.noise_mask = mask + init.args.root.noise_mask = mask put_all(dicts, key, mask) elif is_mask_image is None and init.is_use_mask: put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noise mask diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index f520e9e29..7866a26ed 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -40,6 +40,9 @@ def kernel_size(self) -> tuple[int, int]: def flow_factor(self): return self.hybrid_comp_schedules['flow_factor'] + def has_strength(self): + return self.strength > 0 + @staticmethod def create(deform_keys, i): with context(deform_keys) as keys: @@ -86,7 +89,7 @@ def update_depth_prediction(self, init: RenderInit, turbo: Turbo): if has_depth and has_next: image = turbo.next.image weight = init.args.anim_args.midas_weight - precision = init.root.half_precision + precision = init.args.root.half_precision self.depth = init.depth_model.predict(image, weight, precision) def write_frame_subtitle(self, init, indexes, turbo): @@ -136,7 +139,7 @@ def apply_frame_noising(self, init, mask, image): if is_use_any_mask: seq = self.schedule.noise_mask_seq vals = mask.noise_vals - init.root.noise_mask = call_compose_mask_with_check(init, seq, vals, contrast_image) + init.args.root.noise_mask = call_compose_mask_with_check(init, seq, vals, contrast_image) return call_add_noise(init, self, image) @staticmethod diff --git a/scripts/deforum_helpers/rendering/util/call/images.py b/scripts/deforum_helpers/rendering/util/call/images.py index 38781b9e1..91ad86ae5 100644 --- a/scripts/deforum_helpers/rendering/util/call/images.py +++ b/scripts/deforum_helpers/rendering/util/call/images.py @@ -8,7 +8,7 @@ def call_add_noise(init, step, image): seed: int = init.args.args.seed n_type: str = aa.noise_type perlin_arguments = (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence) - mask = init.root.noise_mask + mask = init.args.root.noise_mask is_do_maks_invert = init.args.args.invert_mask return add_noise(image, amount, seed, n_type, perlin_arguments, mask, is_do_maks_invert) diff --git a/scripts/deforum_helpers/rendering/util/call/mask.py b/scripts/deforum_helpers/rendering/util/call/mask.py index 5b0223b72..1b0d0de00 100644 --- a/scripts/deforum_helpers/rendering/util/call/mask.py +++ b/scripts/deforum_helpers/rendering/util/call/mask.py @@ -3,7 +3,7 @@ def call_compose_mask_with_check(init, mask_seq, val_masks, image): - return compose_mask_with_check(init.root, init.args.args, mask_seq, val_masks, + return compose_mask_with_check(init.args.root, init.args.args, mask_seq, val_masks, Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))) diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index ae26c7abb..67cbc2f87 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -25,7 +25,7 @@ def _frame_filename_index(i: int, file_format: FileFormat) -> str: def _frame_filename(init: StepInit, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" - return f"{init.root.timestring}{infix}{_frame_filename_index(i, file_format)}" + return f"{init.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" def frame(init: StepInit, indexes: Indexes) -> str: diff --git a/scripts/deforum_helpers/rendering/util/fun_utils.py b/scripts/deforum_helpers/rendering/util/fun_utils.py index ab9c944bf..fbed5a2ce 100644 --- a/scripts/deforum_helpers/rendering/util/fun_utils.py +++ b/scripts/deforum_helpers/rendering/util/fun_utils.py @@ -12,6 +12,6 @@ def flat_map(func, iterable): return mapped_iterable -def pipe(*funcs): - """Pipes a value through a sequence of functions""" +def tube(*funcs): + """Tubes a value through a sequence of functions""" return lambda value: reduce(lambda x, f: f(x), funcs, value) diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index aee826c53..76ef6e4ea 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -17,7 +17,7 @@ def handle_med_or_low_vram_before_step(init): sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(init.root.device) + init.animation_mode.depth_model.to(init.args.root.device) def handle_vram_if_depth_is_predicted(init): @@ -34,7 +34,7 @@ def handle_vram_before_depth_map_generation(init): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - init.depth_model.to(init.root.device) + init.depth_model.to(init.args.root.device) def handle_vram_after_depth_map_generation(init): diff --git a/scripts/deforum_helpers/rendering/util/web_ui_utils.py b/scripts/deforum_helpers/rendering/util/web_ui_utils.py index 8e0b1314e..953b1c2d1 100644 --- a/scripts/deforum_helpers/rendering/util/web_ui_utils.py +++ b/scripts/deforum_helpers/rendering/util/web_ui_utils.py @@ -26,7 +26,7 @@ def update_job(init, indexes): def update_status_tracker(init, indexes): progress = indexes.frame.i / init.args.anim_args.max_frames - JobStatusTracker().update_phase(init.root.job_id, phase="GENERATING", progress=progress) + JobStatusTracker().update_phase(init.args.root.job_id, phase="GENERATING", progress=progress) def update_progress_during_cadence(init, indexes): From 95ad88f70c3eb6baef7f435cfac3275b6c75ca91 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 12 Jun 2024 00:34:39 +0200 Subject: [PATCH 058/132] Uncurried image transformation tubes into separate calls for creation and execution and sorted out some complex conditions. Removed disposables and context manager from the main render loop because scoping is now mostly under control. --- scripts/deforum_helpers/render.py | 83 +++++++++++-------- .../rendering/data/initialization.py | 10 ++- .../rendering/data/step/step.py | 4 + 3 files changed, 59 insertions(+), 38 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index c7d64f330..d1f7ac98d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -34,7 +34,6 @@ from .rendering.util.fun_utils import tube from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required from .rendering.util.log_utils import print_animation_frame_info, print_optical_flow_info, print_redo_generation_info -from .rendering.util.utils import context from .save_images import save_image from .seed import next_seed @@ -67,10 +66,10 @@ def run_render_animation(init): images.color_match = Step.create_color_match_for_video(init, indexes) - if images.previous is not None: # skipping 1st iteration - images.previous = frame_image_transformation_tube(init, indexes, step, images)(images.previous) - contrast_image = contrast_image_transformation_tube(init, step, mask)(images.previous) - noised_image = noise_image_transformation_tube(init, step)(contrast_image) + # transform image + if images.has_previous(): # skipping 1st iteration + frame_tube, contrast_tube, noise_tube = image_transformation_tubes(init, indexes, step, images, mask) + images.previous, noised_image = run_tubes(frame_tube, contrast_tube, noise_tube, images.previous) init.update_sample_and_args_for_current_progression_step(step, noised_image) init.update_some_args_for_current_step(indexes, step) @@ -83,43 +82,45 @@ def run_render_animation(init): opt_utils.setup(init, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(init) - optical_flow_redo_generation = init.args.anim_args.optical_flow_redo_generation \ - if not init.args.args.motion_preview_mode else 'None' - # optical flow redo before generation - if optical_flow_redo_generation != 'None' and images.previous is not None and step.init.strength > 0: + # optical flow + optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() + if step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images): + # TODO create and move to optical_flow_redo_before_generation? stored_seed = init.args.args.seed init.args.args.seed = generate_random_seed() print_optical_flow_info(init, optical_flow_redo_generation) - with context(call_generate(init, indexes.frame.i, step.schedule)) as img: - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - disposable_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) - img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) - img = image_transform_optical_flow(img, disposable_flow, step.init.redo_flow_factor) - init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds - init.args.root.init_sample = Image.fromarray(img) - disposable_image = img # TODO refactor - del (img, disposable_flow, stored_seed) - gc.collect() + + # TODO create and move to optical_flow_redo_tube + img = call_generate(init, indexes.frame.i, step.schedule) + img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) + optical_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) + img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) + img = image_transform_optical_flow(img, optical_flow, step.init.redo_flow_factor) + + init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds + init.args.root.init_sample = Image.fromarray(img) + gc.collect() # diffusion redo - if (int(init.args.anim_args.diffusion_redo) > 0 - and images.previous is not None and step.init.strength > 0 - and not init.args.args.motion_preview_mode): + is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() + is_not_preview = init.is_not_in_motion_preview_mode() + if is_diffusion_redo and is_not_preview: stored_seed = init.args.args.seed - for n in range(0, int(init.args.anim_args.diffusion_redo)): + last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) + for n in range(0, last_diffusion_redo_index): print_redo_generation_info(init, n) init.args.args.seed = generate_random_seed() - disposable_image = call_generate(init, indexes.frame.i, step.schedule) - disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) + diffusion_redo_image = call_generate(init, indexes.frame.i, step.schedule) + diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) # color match on last one only - if n == int(init.args.anim_args.diffusion_redo): - disposable_image = maintain_colors(images.previous, images.color_match, - init.args.anim_args.color_coherence) + is_last_iteration = n == last_diffusion_redo_index + if is_last_iteration: + mode = init.args.anim_args.color_coherence + diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) init.args.args.seed = stored_seed - init.args.root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) - del (disposable_image, stored_seed) - gc.collect() # TODO try to eventually kick the gc only once at the end of every generation, iteration. + init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) + gc.collect() # generation image = call_generate(init, indexes.frame.i, step.schedule) @@ -191,7 +192,21 @@ def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_fra images.previous = new_image # TODO shouldn't -def frame_image_transformation_tube(init, indexes, step, images): +def image_transformation_tubes(init, indexes, step, images, mask): + return (frame_transformation_tube(init, indexes, step, images), + contrast_transformation_tube(init, step, mask), + noise_transformation_tube(init, step)) + + +def run_tubes(frame_tube, contrast_tube, noise_tube, original_image): + transformed_image = frame_tube(original_image) + contrasted_image = contrast_tube(transformed_image) + noised_image = noise_tube(contrasted_image) + return transformed_image, noised_image + + +# Image transformation tubes +def frame_transformation_tube(init, indexes, step, images): # make sure `im` stays the last argument in each call. return tube(lambda im: step.apply_frame_warp_transform(init, indexes, im), lambda im: step.do_hybrid_compositing_before_motion(init, indexes, im), @@ -202,12 +217,12 @@ def frame_image_transformation_tube(init, indexes, step, images): lambda im: Step.transform_to_grayscale_if_active(init, images, im)) -def contrast_image_transformation_tube(init, step, mask): +def contrast_transformation_tube(init, step, mask): return tube(lambda im: step.apply_scaling(im), lambda im: step.apply_anti_blur(init, mask, im)) -def noise_image_transformation_tube(init, step): +def noise_transformation_tube(init, step): return tube(lambda im: step.apply_frame_noising(init, step, im)) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index a2b5954f8..e7b544466 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -168,12 +168,15 @@ def diffusion_redo(self): def diffusion_redo_as_int(self): return int(self.diffusion_redo()) + def has_positive_diffusion_redo(self): + return self.diffusion_redo_as_int() > 0 + def optical_flow_redo_generation(self): return self.args.anim_args.optical_flow_redo_generation - def optical_flow_redo_generation_steps_if_not_in_preview_mode(self): + def optical_flow_redo_generation_if_not_in_preview_mode(self): is_not_preview = self.is_not_in_motion_preview_mode() - return self.optical_flow_redo_generation() if is_not_preview else None + return self.optical_flow_redo_generation() if is_not_preview else 'None' # don't replace with None value def is_do_color_match_conversion(self, step): is_legacy_cm = self.args.anim_args.legacy_colormatch @@ -218,7 +221,7 @@ def update_sub_seed_schedule_for_current_step(self, indexes): if is_subseed_scheduling_enabled and not is_seed_managed_by_parseq: self.args.root.subseed_strength = float(keys.subseed_strength_schedule_series[i]) if is_seed_managed_by_parseq: - self.args.root.subseed_strength = keys.subseed_strength_schedule_series[i] # TODO not sure why not type-coerced. + self.args.root.subseed_strength = keys.subseed_strength_schedule_series[i] self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. def prompt_for_current_step(self, indexes): @@ -335,4 +338,3 @@ def save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, @staticmethod def maybe_resume_from_timestring(anim_args, root): root.timestring = anim_args.resume_timestring if anim_args.resume_from_timestring else root.timestring - diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 7866a26ed..e389f2a26 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -83,6 +83,10 @@ def create(init, indexes): schedule = Schedule.create(init, indexes.frame.i, init.args.anim_args, init.args.args) return Step(step_init, schedule, None, None, "") + def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): + has_flow_redo = optical_flow_redo_generation != 'None' + return has_flow_redo and images.has_previous() and self.init.has_strength() + def update_depth_prediction(self, init: RenderInit, turbo: Turbo): has_depth = init.depth_model is not None has_next = turbo.next.image is not None From 744eadfd83f66fc1b0d1263f4237d6bb842ed244 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 12 Jun 2024 01:25:53 +0200 Subject: [PATCH 059/132] Optical flow tube. --- scripts/deforum_helpers/render.py | 62 +++++++++++++++++-------------- 1 file changed, 35 insertions(+), 27 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d1f7ac98d..8e016d3ec 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -83,24 +83,10 @@ def run_render_animation(init): memory_utils.handle_vram_if_depth_is_predicted(init) - # optical flow + # optical flow redo before generation optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() if step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images): - # TODO create and move to optical_flow_redo_before_generation? - stored_seed = init.args.args.seed - init.args.args.seed = generate_random_seed() - print_optical_flow_info(init, optical_flow_redo_generation) - - # TODO create and move to optical_flow_redo_tube - img = call_generate(init, indexes.frame.i, step.schedule) - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - optical_flow = call_get_flow_from_images(init, images.previous, img, optical_flow_redo_generation) - img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) - img = image_transform_optical_flow(img, optical_flow, step.init.redo_flow_factor) - - init.args.args.seed = stored_seed # TODO check if (or make) unnecessary and group seeds - init.args.root.init_sample = Image.fromarray(img) - gc.collect() + optical_flow_redo_before_generation(init, indexes, step, images) # diffusion redo is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() @@ -207,23 +193,31 @@ def run_tubes(frame_tube, contrast_tube, noise_tube, original_image): # Image transformation tubes def frame_transformation_tube(init, indexes, step, images): - # make sure `im` stays the last argument in each call. - return tube(lambda im: step.apply_frame_warp_transform(init, indexes, im), - lambda im: step.do_hybrid_compositing_before_motion(init, indexes, im), - lambda im: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, im), - lambda im: Step.apply_hybrid_motion_optical_flow(init, indexes, images, im), - lambda im: step.do_normal_hybrid_compositing_after_motion(init, indexes, im), - lambda im: Step.apply_color_matching(init, images, im), - lambda im: Step.transform_to_grayscale_if_active(init, images, im)) + # make sure `img` stays the last argument in each call. + return tube(lambda img: step.apply_frame_warp_transform(init, indexes, img), + lambda img: step.do_hybrid_compositing_before_motion(init, indexes, img), + lambda img: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, img), + lambda img: Step.apply_hybrid_motion_optical_flow(init, indexes, images, img), + lambda img: step.do_normal_hybrid_compositing_after_motion(init, indexes, img), + lambda img: Step.apply_color_matching(init, images, img), + lambda img: Step.transform_to_grayscale_if_active(init, images, img)) def contrast_transformation_tube(init, step, mask): - return tube(lambda im: step.apply_scaling(im), - lambda im: step.apply_anti_blur(init, mask, im)) + return tube(lambda img: step.apply_scaling(img), + lambda img: step.apply_anti_blur(init, mask, img)) def noise_transformation_tube(init, step): - return tube(lambda im: step.apply_frame_noising(init, step, im)) + return tube(lambda img: step.apply_frame_noising(init, step, img)) + + +def optical_flow_redo_tube(init, optical_flow, images): + return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), + lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), + lambda img: image_transform_optical_flow( # TODO extract get_flow + img, call_get_flow_from_images(init, images.previous, img, optical_flow), + step.init.redo_flow_factor)) def generate_depth_maps_if_active(init): @@ -246,3 +240,17 @@ def progress_step(init, idx, turbo, opencv_image, image, depth): save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) depth = generate_depth_maps_if_active(init) return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. + + +def optical_flow_redo_before_generation(init, indexes, step, images): + stored_seed = init.args.args.seed # keep original to reset it after executing the optical flow + init.args.args.seed = generate_random_seed() # set a new random seed + print_optical_flow_info(init, optical_flow_redo_generation) # TODO output temp seed? + + sample_image = call_generate(init, indexes.frame.i, step.schedule) + optical_tube = optical_flow_redo_tube(init, optical_flow_redo_generation, images) + transformed_sample_image = optical_tube(sample_image) + + init.args.args.seed = stored_seed # restore stored seed + init.args.root.init_sample = Image.fromarray(transformed_sample_image) + gc.collect() From 8459d6d4110d17bc1beb461b4577347e4474ccf8 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 12 Jun 2024 02:12:56 +0200 Subject: [PATCH 060/132] Cleanup. --- scripts/deforum_helpers/render.py | 5 ++--- scripts/deforum_helpers/rendering/data/initialization.py | 6 +----- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 8e016d3ec..84894421d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -94,6 +94,7 @@ def run_render_animation(init): if is_diffusion_redo and is_not_preview: stored_seed = init.args.args.seed last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) + # TODO extract for n in range(0, last_diffusion_redo_index): print_redo_generation_info(init, n) init.args.args.seed = generate_random_seed() @@ -131,9 +132,7 @@ def run_render_animation(init): image = add_overlay_mask_if_active(init, image) # on strength 0, set color match to generation - if (((not init.args.anim_args.legacy_colormatch and not init.args.args.use_init) - or (init.args.anim_args.legacy_colormatch and step.init.strength == 0)) - and init.args.anim_args.color_coherence not in ['Image', 'Video Input']): + if init.is_do_color_match_conversion(step): images.color_match = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index e7b544466..629e06244 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -31,13 +31,9 @@ class RenderInitArgs: video_args: Any = None controlnet_args: Any = None loop_args: LoopArgs = None - opts: Any = None + opts: Any = None # opts were passed from modules.shared. # TODO import and access directly? root: RootArgs = None - @staticmethod - def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root): - return RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: From eba86d3e735a39642c9668584da1b99241aaeff9 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 14 Jun 2024 21:59:03 +0200 Subject: [PATCH 061/132] Extracted diffusion redo and generation state update. Image to image tube isolation. --- scripts/deforum_helpers/render.py | 104 ++++++------------ .../deforum_helpers/rendering/data/indexes.py | 6 + .../rendering/data/initialization.py | 13 ++- .../rendering/img_2_img_tubes.py | 55 +++++++++ .../rendering/util/__init__.py | 3 +- 5 files changed, 108 insertions(+), 73 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/img_2_img_tubes.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 84894421d..ed7bc2df8 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -27,11 +27,12 @@ from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep -from .rendering.util import generate_random_seed, memory_utils, filename_utils, opt_utils, web_ui_utils +from .rendering.img_2_img_tubes import ( + frame_transformation_tube, contrast_transformation_tube, noise_transformation_tube, + optical_flow_redo_tube, hybrid_video_after_generation_tube, color_match_tube) +from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate -from .rendering.util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .rendering.util.call.video_and_audio import call_render_preview -from .rendering.util.fun_utils import tube from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required from .rendering.util.log_utils import print_animation_frame_info, print_optical_flow_info, print_redo_generation_info from .save_images import save_image @@ -72,41 +73,21 @@ def run_render_animation(init): images.previous, noised_image = run_tubes(frame_tube, contrast_tube, noise_tube, images.previous) init.update_sample_and_args_for_current_progression_step(step, noised_image) - init.update_some_args_for_current_step(indexes, step) - init.update_seed_and_checkpoint_for_current_step(indexes) - init.update_sub_seed_schedule_for_current_step(indexes) - init.prompt_for_current_step(indexes) - init.update_video_data_for_current_frame(indexes, step) - init.update_mask_image(step, mask) - init.animation_keys.update(indexes.frame.i) - opt_utils.setup(init, step.schedule) - + init.prepare_generation(indexes, step, mask) memory_utils.handle_vram_if_depth_is_predicted(init) # optical flow redo before generation optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() - if step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images): - optical_flow_redo_before_generation(init, indexes, step, images) + is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) + if is_redo_optical_flow: + init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) + gc.collect() # diffusion redo is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() is_not_preview = init.is_not_in_motion_preview_mode() if is_diffusion_redo and is_not_preview: - stored_seed = init.args.args.seed - last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) - # TODO extract - for n in range(0, last_diffusion_redo_index): - print_redo_generation_info(init, n) - init.args.args.seed = generate_random_seed() - diffusion_redo_image = call_generate(init, indexes.frame.i, step.schedule) - diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) - # color match on last one only - is_last_iteration = n == last_diffusion_redo_index - if is_last_iteration: - mode = init.args.anim_args.color_coherence - diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) - init.args.args.seed = stored_seed - init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) + do_diffusion_redo(init, indexes, step, images) gc.collect() # generation @@ -116,17 +97,13 @@ def run_render_animation(init): break # do hybrid video after generation - if indexes.frame.i > 0 and init.is_hybrid_composite_after_generation(): - temp_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - _, temp_image_2 = call_hybrid_composite(init, indexes.frame.i, temp_image, step.init.hybrid_comp_schedules) - image = Image.fromarray(cv2.cvtColor(temp_image_2, cv2.COLOR_BGR2RGB)) + if indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation(): + image = hybrid_video_after_generation_tube(init, indexes, step)(image) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if indexes.frame.i == 0 and init.is_color_match_to_be_initialized(images.color_match): - temp_color = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - temp_image = maintain_colors(temp_color, images.color_match, init.args.anim_args.color_coherence) - image = Image.fromarray(cv2.cvtColor(temp_image, cv2.COLOR_BGR2RGB)) + if indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match): + image = color_match_tube(init, images)(image) image = force_to_grayscale_if_required(init, image) image = add_overlay_mask_if_active(init, image) @@ -187,36 +164,7 @@ def run_tubes(frame_tube, contrast_tube, noise_tube, original_image): transformed_image = frame_tube(original_image) contrasted_image = contrast_tube(transformed_image) noised_image = noise_tube(contrasted_image) - return transformed_image, noised_image - - -# Image transformation tubes -def frame_transformation_tube(init, indexes, step, images): - # make sure `img` stays the last argument in each call. - return tube(lambda img: step.apply_frame_warp_transform(init, indexes, img), - lambda img: step.do_hybrid_compositing_before_motion(init, indexes, img), - lambda img: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, img), - lambda img: Step.apply_hybrid_motion_optical_flow(init, indexes, images, img), - lambda img: step.do_normal_hybrid_compositing_after_motion(init, indexes, img), - lambda img: Step.apply_color_matching(init, images, img), - lambda img: Step.transform_to_grayscale_if_active(init, images, img)) - - -def contrast_transformation_tube(init, step, mask): - return tube(lambda img: step.apply_scaling(img), - lambda img: step.apply_anti_blur(init, mask, img)) - - -def noise_transformation_tube(init, step): - return tube(lambda img: step.apply_frame_noising(init, step, img)) - - -def optical_flow_redo_tube(init, optical_flow, images): - return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), - lambda img: image_transform_optical_flow( # TODO extract get_flow - img, call_get_flow_from_images(init, images.previous, img, optical_flow), - step.init.redo_flow_factor)) + return transformed_image, noised_image # TODO separate def generate_depth_maps_if_active(init): @@ -241,7 +189,7 @@ def progress_step(init, idx, turbo, opencv_image, image, depth): return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. -def optical_flow_redo_before_generation(init, indexes, step, images): +def do_optical_flow_redo_before_generation(init, indexes, step, images): stored_seed = init.args.args.seed # keep original to reset it after executing the optical flow init.args.args.seed = generate_random_seed() # set a new random seed print_optical_flow_info(init, optical_flow_redo_generation) # TODO output temp seed? @@ -251,5 +199,21 @@ def optical_flow_redo_before_generation(init, indexes, step, images): transformed_sample_image = optical_tube(sample_image) init.args.args.seed = stored_seed # restore stored seed - init.args.root.init_sample = Image.fromarray(transformed_sample_image) - gc.collect() + return Image.fromarray(transformed_sample_image) + + +def do_diffusion_redo(init, indexes, step, images): + stored_seed = init.args.args.seed + last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) + for n in range(0, last_diffusion_redo_index): + print_redo_generation_info(init, n) + init.args.args.seed = generate_random_seed() + diffusion_redo_image = call_generate(init, indexes.frame.i, step.schedule) + diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) + # color match on last one only + is_last_iteration = n == last_diffusion_redo_index + if is_last_iteration: + mode = init.args.anim_args.color_coherence + diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) + init.args.args.seed = stored_seed + init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index 5fb8aad9c..b72d1392d 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -27,3 +27,9 @@ def update_tween_start(self, turbo): def update_frame(self, i: int): self.frame = IndexWithStart(self.frame.start, i) + + def is_not_first_frame(self): + return self.frame.i > 0 + + def is_first_frame(self): + return self.frame.i == 0 diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index 629e06244..a507cd8d8 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -10,7 +10,7 @@ from .anim import AnimationKeys, AnimationMode from .subtitle import Srt -from ..util import memory_utils +from ..util import memory_utils, opt_utils from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame from ..util.utils import context @@ -263,6 +263,17 @@ def update_mask_image(self, step, mask): else: self.args.args.mask_image = None # we need it only after the first frame anyway + def prepare_generation(self, indexes, step, mask): + # TODO move all of this to Step? + self.update_some_args_for_current_step(indexes, step) + self.update_seed_and_checkpoint_for_current_step(indexes) + self.update_sub_seed_schedule_for_current_step(indexes) + self.prompt_for_current_step(indexes) + self.update_video_data_for_current_frame(indexes, step) + self.update_mask_image(step, mask) + self.animation_keys.update(indexes.frame.i) + opt_utils.setup(self, step.schedule) + @staticmethod def create_output_directory_for_the_batch(directory): os.makedirs(directory, exist_ok=True) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py new file mode 100644 index 000000000..4cf1669a6 --- /dev/null +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -0,0 +1,55 @@ +from .data.step.step import Step +from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite +from .util.fun_utils import tube + +""" +This module provides functions for processing images through various transformations. +The `tube` function allows chaining these transformations together to create flexible image processing pipelines. +Easily experiment by changing, or changing the order of function calls in the tube. + +All functions within the tube take and return an image (`img` argument). They may (and must) pass through +the original image unchanged if a specific transformation is disabled or not required. + +Example: +transformed_image = my_tube(arguments)(original_image) +""" + + +def frame_transformation_tube(init, indexes, step, images): + # make sure `img` stays the last argument in each call. + return tube(lambda img: step.apply_frame_warp_transform(init, indexes, img), + lambda img: step.do_hybrid_compositing_before_motion(init, indexes, img), + lambda img: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, img), + lambda img: Step.apply_hybrid_motion_optical_flow(init, indexes, images, img), + lambda img: step.do_normal_hybrid_compositing_after_motion(init, indexes, img), + lambda img: Step.apply_color_matching(init, images, img), + lambda img: Step.transform_to_grayscale_if_active(init, images, img)) + + +def contrast_transformation_tube(init, step, mask): + return tube(lambda img: step.apply_scaling(img), + lambda img: step.apply_anti_blur(init, mask, img)) + + +def noise_transformation_tube(init, step): + return tube(lambda img: step.apply_frame_noising(init, step, img)) + + +def optical_flow_redo_tube(init, optical_flow, images): + return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), + lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), + lambda img: image_transform_optical_flow( # TODO create img.get_flow + img, call_get_flow_from_images(init, images.previous, img, optical_flow), + step.init.redo_flow_factor)) + + +def hybrid_video_after_generation_tube(init, indexes, step): + return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), + lambda img: call_hybrid_composite(init, indexes.frame.i, img, step.init.hybrid_comp_schedules), + lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))) + + +def color_match_tube(init, images): + return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), + lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), + lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))) diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index fda732d10..3ee9c1aa3 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,3 +1,2 @@ -# All modules in this package are intended to not hold or change any state. -# Only use static class methods or loose defs that don't change anything in the arguments passed. +# All modules in this package are intended to not hold any state. from .utils import call_or_use_on_cond, combo_context, context, generate_random_seed, put_all, put_if_present From 2877398b09b1748b4163d6fb777a26fb928c4bd3 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 14 Jun 2024 22:29:24 +0200 Subject: [PATCH 062/132] Removed code that abused contextmanagers to control scope because it's no longer useful. --- .../rendering/data/anim/animation_keys.py | 17 +++--- .../rendering/data/anim/animation_mode.py | 17 +++--- .../rendering/data/initialization.py | 35 ++++++------ .../deforum_helpers/rendering/data/mask.py | 13 +++-- .../rendering/data/schedule.py | 26 +++++---- .../rendering/data/step/step.py | 25 +++++---- .../rendering/util/__init__.py | 2 +- .../rendering/util/call/anim.py | 7 ++- .../rendering/util/call/gen.py | 7 ++- .../rendering/util/call/hybrid.py | 54 ++++++++++--------- .../rendering/util/call/video_and_audio.py | 5 +- .../deforum_helpers/rendering/util/utils.py | 35 ------------ 12 files changed, 102 insertions(+), 141 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index ed2416e21..5d585240e 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -1,6 +1,5 @@ from dataclasses import dataclass -from ...util.utils import context from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys @@ -10,14 +9,14 @@ class AnimationKeys: looper_keys: LooperAnimKeys def update(self, i: int): - with context(self.looper_keys) as keys: - keys.use_looper = keys.use_looper - keys.imagesToKeyframe = keys.imagesToKeyframe - keys.imageStrength = keys.image_strength_schedule_series[i] - keys.blendFactorMax = keys.blendFactorMax_series[i] - keys.blendFactorSlope = keys.blendFactorSlope_series[i] - keys.tweeningFramesSchedule = keys.tweening_frames_schedule_series[i] - keys.colorCorrectionFactor = keys.color_correction_factor_series[i] + keys = self.looper_keys + keys.use_looper = keys.use_looper + keys.imagesToKeyframe = keys.imagesToKeyframe + keys.imageStrength = keys.image_strength_schedule_series[i] + keys.blendFactorMax = keys.blendFactorMax_series[i] + keys.blendFactorSlope = keys.blendFactorSlope_series[i] + keys.tweeningFramesSchedule = keys.tweening_frames_schedule_series[i] + keys.colorCorrectionFactor = keys.color_correction_factor_series[i] @staticmethod def _choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 39dbba4bb..bc11bc17a 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -4,7 +4,6 @@ from typing import Any from ...util.memory_utils import keep_3d_models_in_vram -from ...util.utils import combo_context from ....RAFT import RAFT from ....hybrid_video import hybrid_generation @@ -86,11 +85,11 @@ def initial_hybrid_files(sa) -> list[Path]: @staticmethod def from_args(step_args): - with combo_context(step_args, AnimationMode) as (sa, AM): - # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False - hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') - return AnimationMode( - AM.has_video_input(sa.anim_args), AM.initial_hybrid_files(sa), - hybrid_input_files, None, keep_3d_models_in_vram(sa), - AM.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), - AM.load_raft_if_active(sa.anim_args, sa.args)) + sa = step_args + # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False + hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') + return AnimationMode( + AnimationMode.has_video_input(sa.anim_args), AnimationMode.initial_hybrid_files(sa), + hybrid_input_files, None, keep_3d_models_in_vram(sa), + AnimationMode.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), + AnimationMode.load_raft_if_active(sa.anim_args, sa.args)) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index a507cd8d8..1a4fcdadf 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -13,7 +13,6 @@ from ..util import memory_utils, opt_utils from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame -from ..util.utils import context from ...args import DeforumArgs, DeforumAnimArgs, LoopArgs, ParseqArgs, RootArgs from ...deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled from ...depth import DepthModel @@ -50,27 +49,25 @@ class RenderInit: is_use_mask: bool @staticmethod - def create(args, parseq_args, anim_args, video_args, controlnet_args, - loop_args, opts, root) -> 'RenderInit': + def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) output_directory = args.outdir is_use_mask = args.use_mask - with context(RenderInit) as RI: - parseq_adapter = RI.create_parseq_adapter(ri_args) - srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) - animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) - animation_mode = AnimationMode.from_args(ri_args) - prompt_series = RI.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RI.create_depth_model_and_enable_depth_map_saving_if_active( - animation_mode, root, anim_args, args) - instance = RenderInit(args.seed, ri_args, parseq_adapter, srt, animation_keys, - animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - RI.init_looper_if_active(args, loop_args) - RI.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - RI.create_output_directory_for_the_batch(args.outdir) - RI.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - RI.maybe_resume_from_timestring(anim_args, root) - return instance + parseq_adapter = RenderInit.create_parseq_adapter(ri_args) + srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) + animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) + animation_mode = AnimationMode.from_args(ri_args) + prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) + depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( + animation_mode, root, anim_args, args) + instance = RenderInit(args.seed, ri_args, parseq_adapter, srt, animation_keys, + animation_mode, prompt_series, depth_model, output_directory, is_use_mask) + RenderInit.init_looper_if_active(args, loop_args) + RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + RenderInit.create_output_directory_for_the_batch(args.outdir) + RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + RenderInit.maybe_resume_from_timestring(anim_args, root) + return instance def is_3d(self): return self.args.anim_args.animation_mode == '3D' diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index 9e27a1062..f095aaccb 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -4,7 +4,7 @@ from PIL import Image from ..util import put_all -from ..util.utils import create_img, call_or_use_on_cond, context +from ..util.utils import create_img, call_or_use_on_cond from ...load_images import get_mask, load_img from ...rendering.util.call.images import call_get_mask_from_file @@ -51,12 +51,15 @@ def _assign(init, i, is_mask_image, dicts): elif is_mask_image is None and init.is_use_mask: put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noise mask + @staticmethod + def _load_mask(init, args): + return load_img(args.init_image, args.init_image_box, shape=init.dimensions(), + use_alpha_as_mask=args.use_alpha_as_mask)[1] + @staticmethod def _create_mask_image(init): - with context(init.args.args) as args: - return call_or_use_on_cond(init.is_using_init_image_or_box(), - lambda: load_img(args.init_image, args.init_image_box, shape=init.dimensions(), - use_alpha_as_mask=args.use_alpha_as_mask)[1]) + args = init.args.args + return call_or_use_on_cond(init.is_using_init_image_or_box(), lambda: _load_mask(init, args)) @staticmethod def _create(init, i, mask_image): diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index 0c7bf72a4..51ec35864 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,8 +1,6 @@ from dataclasses import dataclass from typing import Optional, Any -from ..util import context - @dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: @@ -21,18 +19,18 @@ def create(init, i, anim_args, args): # TODO typecheck keys as DeformAnimKeys or provide key collection or something """Create a new Schedule instance based on the provided parameters.""" is_use_mask_without_noise = init.is_use_mask and not init.args.anim_args.use_noise_mask - with context(init.animation_keys.deform_keys) as keys: - steps = Schedule.schedule_steps(keys, i, anim_args) - sampler_name = Schedule.schedule_sampler(keys, i, anim_args) - clipskip = Schedule.schedule_clipskip(keys, i, anim_args) - noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) - eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) - eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) - mask = Schedule.schedule_mask(keys, i, args) - noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) - noise_mask_seq = schedule.mask_seq if is_use_mask_without_noise else None - return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, - noise_mask, noise_mask_seq) + keys = init.animation_keys.deform_keys + steps = Schedule.schedule_steps(keys, i, anim_args) + sampler_name = Schedule.schedule_sampler(keys, i, anim_args) + clipskip = Schedule.schedule_clipskip(keys, i, anim_args) + noise_multiplier = Schedule.schedule_noise_multiplier(keys, i, anim_args) + eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) + eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) + mask = Schedule.schedule_mask(keys, i, args) + noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) + noise_mask_seq = schedule.mask_seq if is_use_mask_without_noise else None + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, + noise_mask, noise_mask_seq) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index e389f2a26..8f447c5c1 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -16,7 +16,6 @@ from ...util.call.images import call_add_noise from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle -from ...util.utils import context from ....hybrid_video import image_transform_ransac, image_transform_optical_flow @@ -45,18 +44,18 @@ def has_strength(self): @staticmethod def create(deform_keys, i): - with context(deform_keys) as keys: - return StepInit(keys.noise_schedule_series[i], - keys.strength_schedule_series[i], - keys.cfg_scale_schedule_series[i], - keys.contrast_schedule_series[i], - int(keys.kernel_schedule_series[i]), - keys.sigma_schedule_series[i], - keys.amount_schedule_series[i], - keys.threshold_schedule_series[i], - keys.cadence_flow_factor_schedule_series[i], - keys.redo_flow_factor_schedule_series[i], - StepInit._hybrid_comp_args(keys, i)) + keys = deform_keys + return StepInit(keys.noise_schedule_series[i], + keys.strength_schedule_series[i], + keys.cfg_scale_schedule_series[i], + keys.contrast_schedule_series[i], + int(keys.kernel_schedule_series[i]), + keys.sigma_schedule_series[i], + keys.amount_schedule_series[i], + keys.threshold_schedule_series[i], + keys.cadence_flow_factor_schedule_series[i], + keys.redo_flow_factor_schedule_series[i], + StepInit._hybrid_comp_args(keys, i)) @staticmethod def _hybrid_comp_args(keys, i): diff --git a/scripts/deforum_helpers/rendering/util/__init__.py b/scripts/deforum_helpers/rendering/util/__init__.py index 3ee9c1aa3..ea37bcc42 100644 --- a/scripts/deforum_helpers/rendering/util/__init__.py +++ b/scripts/deforum_helpers/rendering/util/__init__.py @@ -1,2 +1,2 @@ # All modules in this package are intended to not hold any state. -from .utils import call_or_use_on_cond, combo_context, context, generate_random_seed, put_all, put_if_present +from .utils import call_or_use_on_cond, generate_random_seed, put_all, put_if_present diff --git a/scripts/deforum_helpers/rendering/util/call/anim.py b/scripts/deforum_helpers/rendering/util/call/anim.py index 42fcb7e21..5c871499d 100644 --- a/scripts/deforum_helpers/rendering/util/call/anim.py +++ b/scripts/deforum_helpers/rendering/util/call/anim.py @@ -1,8 +1,7 @@ -from ..utils import context from ....animation import anim_frame_warp def call_anim_frame_warp(init, i, image, depth): - with context(init.args) as ia: - return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, - depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) + ia = init.args + return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, + depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) diff --git a/scripts/deforum_helpers/rendering/util/call/gen.py b/scripts/deforum_helpers/rendering/util/call/gen.py index 721f41e7a..1a63ff5c8 100644 --- a/scripts/deforum_helpers/rendering/util/call/gen.py +++ b/scripts/deforum_helpers/rendering/util/call/gen.py @@ -1,8 +1,7 @@ -from ..utils import context from ....generate import generate def call_generate(init, i, schedule): - with context(init.args) as ia: - return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, - ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) + ia = init.args + return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, + ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) diff --git a/scripts/deforum_helpers/rendering/util/call/hybrid.py b/scripts/deforum_helpers/rendering/util/call/hybrid.py index 57f2bbcbb..b05f7c2fc 100644 --- a/scripts/deforum_helpers/rendering/util/call/hybrid.py +++ b/scripts/deforum_helpers/rendering/util/call/hybrid.py @@ -1,4 +1,3 @@ -from ..utils import context from ....hybrid_video import ( # Functions related to flow calculation get_flow_from_images, @@ -20,39 +19,44 @@ def call_get_flow_from_images(init, prev_image, next_image, cadence): def call_get_flow_for_hybrid_motion_prev(init, i, image): - with context(init.animation_mode) as mode: - with context(init.args.anim_args) as aa: - return get_flow_for_hybrid_motion_prev(i, init.dimensions(), - mode.hybrid_input_files, - mode.hybrid_frame_path, - mode.prev_flow, - image, - aa.hybrid_flow_method, - mode.raft_model, - aa.hybrid_flow_consistency, - aa.hybrid_consistency_blur, - aa.hybrid_comp_save_extra_frames) + mode = init.animation_mode + aa = init.args.anim_args + return get_flow_for_hybrid_motion_prev( + i, init.dimensions(), + mode.hybrid_input_files, + mode.hybrid_frame_path, + mode.prev_flow, + image, + aa.hybrid_flow_method, + mode.raft_model, + aa.hybrid_flow_consistency, + aa.hybrid_consistency_blur, + aa.hybrid_comp_save_extra_frames) def call_get_flow_for_hybrid_motion(init, i): - with context(init.animation_mode) as mode: - with context(init.args.anim_args) as args: - return get_flow_for_hybrid_motion(i, init.dimensions(), mode.hybrid_input_files, mode.hybrid_frame_path, - mode.prev_flow, args.hybrid_flow_method, mode.raft_model, - args.hybrid_flow_consistency, args.hybrid_consistency_blur, args) + mode = init.animation_mode + args = init.args.anim_args + return get_flow_for_hybrid_motion( + i, init.dimensions(), mode.hybrid_input_files, mode.hybrid_frame_path, + mode.prev_flow, args.hybrid_flow_method, mode.raft_model, + args.hybrid_flow_consistency, args.hybrid_consistency_blur, args) def call_get_matrix_for_hybrid_motion_prev(init, i, image): - return get_matrix_for_hybrid_motion_prev(i, init.dimensions(), init.animation_mode.hybrid_input_files, - image, init.args.anim_args.hybrid_motion) + return get_matrix_for_hybrid_motion_prev( + i, init.dimensions(), init.animation_mode.hybrid_input_files, + image, init.args.anim_args.hybrid_motion) def call_get_matrix_for_hybrid_motion(init, i): - return get_matrix_for_hybrid_motion(i, init.dimensions(), init.animation_mode.hybrid_input_files, - init.args.anim_args.hybrid_motion) + return get_matrix_for_hybrid_motion( + i, init.dimensions(), init.animation_mode.hybrid_input_files, + init.args.anim_args.hybrid_motion) def call_hybrid_composite(init, i, image, hybrid_comp_schedules): - with context(init.args) as ia: - return hybrid_composite(ia.args, ia.anim_args, i, image, init.depth_model, - hybrid_comp_schedules, init.args.root) + ia = init.args + return hybrid_composite( + ia.args, ia.anim_args, i, image, + init.depth_model, hybrid_comp_schedules, init.args.root) diff --git a/scripts/deforum_helpers/rendering/util/call/video_and_audio.py b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py index 0cd505de9..1cdb73f22 100644 --- a/scripts/deforum_helpers/rendering/util/call/video_and_audio.py +++ b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py @@ -1,10 +1,9 @@ -from ..utils import context from ....video_audio_utilities import get_next_frame, render_preview def call_render_preview(init, i, last_preview_frame): - with context(init.args) as ia: - return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) + ia = init.args + return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) def call_get_next_frame(init, i, video_path, is_mask: bool = False): diff --git a/scripts/deforum_helpers/rendering/util/utils.py b/scripts/deforum_helpers/rendering/util/utils.py index 4c3f628cd..80fa1be59 100644 --- a/scripts/deforum_helpers/rendering/util/utils.py +++ b/scripts/deforum_helpers/rendering/util/utils.py @@ -1,43 +1,8 @@ import random -from contextlib import contextmanager from PIL import Image -@contextmanager -def context(resource): - """ - Context manager for aliasing and managing objects or class references within a `with` statement block. - The function can help keeping code compact, less repetitive or more structured by providing a dedicated scope - for resource access, at the tradeoff of introducing an additional level of nesting. - It can be particularly useful when extracting logic to a separate function is undesirable. - - **Example Usage:** - ```python - with context(annoyingly_long_reference.to_some_expression.or_to.MyClass().or_something) as res: - res.do_something() - do_something_with_the_thing(res) - res.do_something_else() - # more lines using res - ``` - - **Arguments:** - - `resource (object or class reference)`: The object or class reference to be managed within the context. - - **Yields:** - - The provided `resource` argument. - - **Note:** - The context manager is responsible for any necessary resource cleanup upon exiting the `with` block (if applicable). - """ - yield resource - - -@contextmanager -def combo_context(*resources): - yield resources - - def put_all(dictionaries, key, value): return list(map(lambda d: {**d, key: value}, dictionaries)) From 9e65e7a4de55451e69928f10562d58f107551fdf Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 14 Jun 2024 23:05:36 +0200 Subject: [PATCH 063/132] Added processing predicate to make img2img tubes conditional. --- scripts/deforum_helpers/render.py | 12 +++++------- .../deforum_helpers/rendering/img_2_img_tubes.py | 15 ++++++++++----- .../deforum_helpers/rendering/util/fun_utils.py | 6 +++--- 3 files changed, 18 insertions(+), 15 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index ed7bc2df8..291272379 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -28,8 +28,8 @@ from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep from .rendering.img_2_img_tubes import ( - frame_transformation_tube, contrast_transformation_tube, noise_transformation_tube, - optical_flow_redo_tube, hybrid_video_after_generation_tube, color_match_tube) + frame_transformation_tube, conditional_color_match_tube , conditional_hybrid_video_after_generation_tube, + contrast_transformation_tube, noise_transformation_tube, optical_flow_redo_tube) from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview @@ -97,15 +97,13 @@ def run_render_animation(init): break # do hybrid video after generation - if indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation(): - image = hybrid_video_after_generation_tube(init, indexes, step)(image) + image = conditional_hybrid_video_after_generation_tube(init, indexes, step)(image) # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - if indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match): - image = color_match_tube(init, images)(image) + image = conditional_color_match_tube(init, indexes, images)(image) - image = force_to_grayscale_if_required(init, image) + image = force_to_grayscale_if_required(init, image) # TODO move to tubes? image = add_overlay_mask_if_active(init, image) # on strength 0, set color match to generation diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 4cf1669a6..873ba5cc7 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -43,13 +43,18 @@ def optical_flow_redo_tube(init, optical_flow, images): step.init.redo_flow_factor)) -def hybrid_video_after_generation_tube(init, indexes, step): +def conditional_hybrid_video_after_generation_tube(init, indexes, step): return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: call_hybrid_composite(init, indexes.frame.i, img, step.init.hybrid_comp_schedules), - lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))) + lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), + is_do_process= + lambda: indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) -def color_match_tube(init, images): - return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), +def conditional_color_match_tube(init, indexes, images): + return tube(lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), + lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), - lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))) + lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), + is_do_process= + lambda: indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match)) diff --git a/scripts/deforum_helpers/rendering/util/fun_utils.py b/scripts/deforum_helpers/rendering/util/fun_utils.py index fbed5a2ce..b7f1a1084 100644 --- a/scripts/deforum_helpers/rendering/util/fun_utils.py +++ b/scripts/deforum_helpers/rendering/util/fun_utils.py @@ -12,6 +12,6 @@ def flat_map(func, iterable): return mapped_iterable -def tube(*funcs): - """Tubes a value through a sequence of functions""" - return lambda value: reduce(lambda x, f: f(x), funcs, value) +def tube(*funcs, is_do_process=lambda: True): + """Tubes a value through a sequence of functions with a predicate `is_do_process` for skipping.""" + return lambda value: reduce(lambda x, f: f(x) if is_do_process() else x, funcs, value) From a49fd53664eeb36fc976f0b2fc60e8023abe135a Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 01:02:09 +0200 Subject: [PATCH 064/132] Noise and contrast tubes combined and conditional redoes extracted. --- scripts/deforum_helpers/render.py | 91 ++++++++++--------- .../rendering/img_2_img_tubes.py | 19 +++- 2 files changed, 66 insertions(+), 44 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 291272379..b897cf612 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -28,8 +28,8 @@ from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep from .rendering.img_2_img_tubes import ( - frame_transformation_tube, conditional_color_match_tube , conditional_hybrid_video_after_generation_tube, - contrast_transformation_tube, noise_transformation_tube, optical_flow_redo_tube) + conditional_color_match_tube, conditional_hybrid_video_after_generation_tube, conditional_extra_color_match_tube, + contrasted_noise_transformation_tube, optical_flow_redo_tube, frame_transformation_tube) from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview @@ -41,13 +41,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + # gc.set_debug(True) # TODO remove run_render_animation(init) -def run_render_animation_controlled(init): - raise NotImplementedError("not implemented.") - - def run_render_animation(init): images = Images.create(init) turbo = Turbo.create(init) # state for interpolating between diffusion steps @@ -62,33 +59,21 @@ def run_render_animation(init): step = Step.create(init, indexes) step.write_frame_subtitle(init, indexes, turbo) - if turbo.is_emit_in_between_frames(): - emit_in_between_frames(init, indexes, step, turbo, images) + maybe_emit_in_between_frames(init, indexes, step, turbo, images) images.color_match = Step.create_color_match_for_video(init, indexes) # transform image - if images.has_previous(): # skipping 1st iteration - frame_tube, contrast_tube, noise_tube = image_transformation_tubes(init, indexes, step, images, mask) - images.previous, noised_image = run_tubes(frame_tube, contrast_tube, noise_tube, images.previous) - init.update_sample_and_args_for_current_progression_step(step, noised_image) + images.previous = transform_and_update_noised_sample(init, indexes, step, images, mask) init.prepare_generation(indexes, step, mask) memory_utils.handle_vram_if_depth_is_predicted(init) # optical flow redo before generation - optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() - is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) - if is_redo_optical_flow: - init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) - gc.collect() + maybe_redo_optical_flow(init, indexes, step, images) # diffusion redo - is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() - is_not_preview = init.is_not_in_motion_preview_mode() - if is_diffusion_redo and is_not_preview: - do_diffusion_redo(init, indexes, step, images) - gc.collect() + maybe_redo_diffusion(init, indexes, step, images) # generation image = call_generate(init, indexes.frame.i, step.schedule) @@ -101,14 +86,13 @@ def run_render_animation(init): # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - image = conditional_color_match_tube(init, indexes, images)(image) + image = conditional_extra_color_match_tube(init, indexes, images)(image) - image = force_to_grayscale_if_required(init, image) # TODO move to tubes? + image = force_to_grayscale_if_required(init, image) image = add_overlay_mask_if_active(init, image) # on strength 0, set color match to generation - if init.is_do_color_match_conversion(step): - images.color_match = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) + images.color_match = conditional_color_match_tube(init, step)(image) opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) if not init.animation_mode.has_video_input: @@ -125,13 +109,14 @@ def run_render_animation(init): init.animation_mode.unload_raft_and_depth_model() -def emit_in_between_frames(init, indexes, step, turbo, images): - tween_frame_start_i = max(indexes.frame.start, indexes.frame.i - turbo.steps) - emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, indexes.frame.i) +def maybe_emit_in_between_frames(init, indexes, step, turbo, images): + if turbo.is_emit_in_between_frames(): + tween_frame_start_i = max(indexes.frame.start, indexes.frame.i - turbo.steps) + emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, indexes.frame.i) def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, frame_i): - """Emits tween frames (also known as turbo- or cadence-frames) between the provided indicis.""" + """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" # TODO refactor until this works with just 2 args: RenderInit and a collection of immutable TweenStep objects. indexes.update_tween_start(turbo) @@ -152,19 +137,6 @@ def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_fra images.previous = new_image # TODO shouldn't -def image_transformation_tubes(init, indexes, step, images, mask): - return (frame_transformation_tube(init, indexes, step, images), - contrast_transformation_tube(init, step, mask), - noise_transformation_tube(init, step)) - - -def run_tubes(frame_tube, contrast_tube, noise_tube, original_image): - transformed_image = frame_tube(original_image) - contrasted_image = contrast_tube(transformed_image) - noised_image = noise_tube(contrasted_image) - return transformed_image, noised_image # TODO separate - - def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: @@ -215,3 +187,36 @@ def do_diffusion_redo(init, indexes, step, images): diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) init.args.args.seed = stored_seed init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) + + +def transform_and_update_noised_sample(init, indexes, step, images, mask): + if images.has_previous(): # skipping 1st iteration + transformed_image = frame_transformation_tube(init, indexes, step, images)(images.previous) + # TODO separate + noised_image = contrasted_noise_transformation_tube(init, step, mask)(transformed_image) + init.update_sample_and_args_for_current_progression_step(step, noised_image) + return transformed_image + else: + return None + + +# Conditional Redoes +IS_KICK_GC_AFTER_REDO = True + + +def maybe_redo_optical_flow(init, indexes, step, images, is_collect_garbage: bool = IS_KICK_GC_AFTER_REDO): + optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() + is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) + if is_redo_optical_flow: + init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) + if is_collect_garbage: + gc.collect() + + +def maybe_redo_diffusion(init, indexes, step, images, is_collect_garbage: bool = IS_KICK_GC_AFTER_REDO): + is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() + is_not_preview = init.is_not_in_motion_preview_mode() + if is_diffusion_redo and is_not_preview: + do_diffusion_redo(init, indexes, step, images) + if is_collect_garbage: + gc.collect() diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 873ba5cc7..38f9b21df 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -1,3 +1,6 @@ +import cv2 +import numpy as np + from .data.step.step import Step from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube @@ -51,10 +54,24 @@ def conditional_hybrid_video_after_generation_tube(init, indexes, step): lambda: indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) -def conditional_color_match_tube(init, indexes, images): +def conditional_extra_color_match_tube(init, indexes, images): return tube(lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= lambda: indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match)) + + +def conditional_color_match_tube(init, step): + return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), + is_do_process= + lambda: init.is_do_color_match_conversion(step)) + + +# Composite Tubes +def contrasted_noise_transformation_tube(init, step, mask): + """Combines contrast and noise transformation tubes.""" + contrast_tube = contrast_transformation_tube(init, step, mask) + noise_tube = noise_transformation_tube(init, step) + return tube(lambda img: noise_tube(contrast_tube(img))) From 185cd69d1eb35a80b4f6bef9b5a5502086895b37 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 01:14:18 +0200 Subject: [PATCH 065/132] Removed manual garbage collection because memory profile is changing. --- scripts/deforum_helpers/render.py | 61 +++++++++++++------------------ 1 file changed, 26 insertions(+), 35 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index b897cf612..9419efbf7 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -14,7 +14,6 @@ # Contact the authors: https://deforum.github.io/ -import gc import os import cv2 @@ -41,7 +40,6 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - # gc.set_debug(True) # TODO remove run_render_animation(init) @@ -159,6 +157,32 @@ def progress_step(init, idx, turbo, opencv_image, image, depth): return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. +def transform_and_update_noised_sample(init, indexes, step, images, mask): + if images.has_previous(): # skipping 1st iteration + transformed_image = frame_transformation_tube(init, indexes, step, images)(images.previous) + # TODO separate + noised_image = contrasted_noise_transformation_tube(init, step, mask)(transformed_image) + init.update_sample_and_args_for_current_progression_step(step, noised_image) + return transformed_image + else: + return None + + +# Conditional Redoes +def maybe_redo_optical_flow(init, indexes, step, images): + optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() + is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) + if is_redo_optical_flow: + init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) + + +def maybe_redo_diffusion(init, indexes, step, images): + is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() + is_not_preview = init.is_not_in_motion_preview_mode() + if is_diffusion_redo and is_not_preview: + do_diffusion_redo(init, indexes, step, images) + + def do_optical_flow_redo_before_generation(init, indexes, step, images): stored_seed = init.args.args.seed # keep original to reset it after executing the optical flow init.args.args.seed = generate_random_seed() # set a new random seed @@ -187,36 +211,3 @@ def do_diffusion_redo(init, indexes, step, images): diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) init.args.args.seed = stored_seed init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) - - -def transform_and_update_noised_sample(init, indexes, step, images, mask): - if images.has_previous(): # skipping 1st iteration - transformed_image = frame_transformation_tube(init, indexes, step, images)(images.previous) - # TODO separate - noised_image = contrasted_noise_transformation_tube(init, step, mask)(transformed_image) - init.update_sample_and_args_for_current_progression_step(step, noised_image) - return transformed_image - else: - return None - - -# Conditional Redoes -IS_KICK_GC_AFTER_REDO = True - - -def maybe_redo_optical_flow(init, indexes, step, images, is_collect_garbage: bool = IS_KICK_GC_AFTER_REDO): - optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() - is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) - if is_redo_optical_flow: - init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) - if is_collect_garbage: - gc.collect() - - -def maybe_redo_diffusion(init, indexes, step, images, is_collect_garbage: bool = IS_KICK_GC_AFTER_REDO): - is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() - is_not_preview = init.is_not_in_motion_preview_mode() - if is_diffusion_redo and is_not_preview: - do_diffusion_redo(init, indexes, step, images) - if is_collect_garbage: - gc.collect() From 48648438426d5749f73c1dd8e8aafc3091968d49 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 02:47:28 +0200 Subject: [PATCH 066/132] Warning when iteration is skipped after the call to "generate" returns no image. General code cleanup. Moved comments out of the main render loop and removed some obsolete or redundant comments. Created new tubes for utility functions that take and return an image. --- scripts/deforum_helpers/render.py | 52 ++++++------------- .../rendering/data/initialization.py | 3 +- .../rendering/data/step/tween_step.py | 5 +- .../rendering/img_2_img_tubes.py | 38 +++++++++++++- .../rendering/util/image_utils.py | 28 ---------- .../rendering/util/log_utils.py | 9 +++- 6 files changed, 64 insertions(+), 71 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 9419efbf7..ca8ca78e8 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -26,14 +26,14 @@ from .rendering.data import Turbo, Images, Indexes, Mask from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep -from .rendering.img_2_img_tubes import ( - conditional_color_match_tube, conditional_hybrid_video_after_generation_tube, conditional_extra_color_match_tube, - contrasted_noise_transformation_tube, optical_flow_redo_tube, frame_transformation_tube) +from .rendering.img_2_img_tubes import (conditional_color_match_tube, conditional_frame_transformation_tube, + contrasted_noise_transformation_tube, optical_flow_redo_tube, + frame_transformation_tube) from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview -from .rendering.util.image_utils import add_overlay_mask_if_active, force_to_grayscale_if_required -from .rendering.util.log_utils import print_animation_frame_info, print_optical_flow_info, print_redo_generation_info +from .rendering.util.log_utils import (print_animation_frame_info, print_optical_flow_info, + print_redo_generation_info, print_warning_generate_returned_no_image) from .save_images import save_image from .seed import next_seed @@ -45,7 +45,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(init): images = Images.create(init) - turbo = Turbo.create(init) # state for interpolating between diffusion steps + turbo = Turbo.create(init) indexes = Indexes.create(init, turbo) mask = Mask.create(init, indexes.frame.i) # reset mask vals as they are overwritten in the compose_mask algorithm web_ui_utils.init_job(init) @@ -60,45 +60,20 @@ def run_render_animation(init): maybe_emit_in_between_frames(init, indexes, step, turbo, images) images.color_match = Step.create_color_match_for_video(init, indexes) - - # transform image images.previous = transform_and_update_noised_sample(init, indexes, step, images, mask) - - init.prepare_generation(indexes, step, mask) - memory_utils.handle_vram_if_depth_is_predicted(init) - - # optical flow redo before generation + init.prepare_generation(init, indexes, step, mask) maybe_redo_optical_flow(init, indexes, step, images) - - # diffusion redo maybe_redo_diffusion(init, indexes, step, images) - # generation image = call_generate(init, indexes.frame.i, step.schedule) - - if image is None: # TODO throw error or log warning or something + if image is None: + print_warning_generate_returned_no_image() break - # do hybrid video after generation - image = conditional_hybrid_video_after_generation_tube(init, indexes, step)(image) - - # color matching on first frame is after generation, color match was collected earlier, - # so we do an extra generation to avoid the corruption introduced by the color match of first output - image = conditional_extra_color_match_tube(init, indexes, images)(image) - - image = force_to_grayscale_if_required(init, image) - image = add_overlay_mask_if_active(init, image) - - # on strength 0, set color match to generation + image = conditional_frame_transformation_tube(init, indexes, step, images)(image) images.color_match = conditional_color_match_tube(init, step)(image) - - opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not init.animation_mode.has_video_input: - images.previous = opencv_image - - next_frame, step.depth = progress_step(init, indexes, turbo, opencv_image, image, step.depth) + next_frame, step.depth = progress_step(init, indexes, turbo, images, image, step.depth) indexes.update_frame(next_frame) - state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO group all seeds and sub-seeds @@ -146,8 +121,11 @@ def generate_depth_maps_if_active(init): return depth -def progress_step(init, idx, turbo, opencv_image, image, depth): +def progress_step(init, idx, turbo, images, image, depth): """Will progress frame or turbo-frame step and return next index and `depth`.""" + opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + if not init.animation_mode.has_video_input: + images.previous = opencv_image if turbo.has_steps(): return idx.frame.i + turbo.progress_step(idx, opencv_image), depth else: diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index 1a4fcdadf..4172e5dac 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -260,7 +260,7 @@ def update_mask_image(self, step, mask): else: self.args.args.mask_image = None # we need it only after the first frame anyway - def prepare_generation(self, indexes, step, mask): + def prepare_generation(self, init, indexes, step, mask): # TODO move all of this to Step? self.update_some_args_for_current_step(indexes, step) self.update_seed_and_checkpoint_for_current_step(indexes) @@ -270,6 +270,7 @@ def prepare_generation(self, indexes, step, mask): self.update_mask_image(step, mask) self.animation_keys.update(indexes.frame.i) opt_utils.setup(self, step.schedule) + memory_utils.handle_vram_if_depth_is_predicted(init) @staticmethod def create_output_directory_for_the_batch(directory): diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index e50a29d2d..a977af79e 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -2,6 +2,7 @@ from typing import Any from ...util import image_utils, log_utils, web_ui_utils +from ...img_2_img_tubes import conditional_force_tween_to_grayscale_tube, conditional_add_overlay_mask_tube @dataclass(init=True, frozen=False, repr=False, eq=False) @@ -28,8 +29,8 @@ def create_directly(from_index, to_index): def generate_tween_image(self, init, indexes, step, turbo): is_tween = True warped = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, self) - recolored = image_utils.force_tween_to_grayscale_if_required(init, warped) - masked = image_utils.add_overlay_mask_if_active(init, recolored, is_tween) + recolored = conditional_force_tween_to_grayscale_tube(init)(warped) + masked = conditional_add_overlay_mask_tube(init, indexes, is_tween)(recolored) return masked @staticmethod diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 38f9b21df..4ba181dec 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -4,6 +4,7 @@ from .data.step.step import Step from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube +from ..masks import do_overlay_mask """ This module provides functions for processing images through various transformations. @@ -46,6 +47,7 @@ def optical_flow_redo_tube(init, optical_flow, images): step.init.redo_flow_factor)) +# Conditional Tubes def conditional_hybrid_video_after_generation_tube(init, indexes, step): return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: call_hybrid_composite(init, indexes.frame.i, img, step.init.hybrid_comp_schedules), @@ -55,6 +57,8 @@ def conditional_hybrid_video_after_generation_tube(init, indexes, step): def conditional_extra_color_match_tube(init, indexes, images): + # color matching on first frame is after generation, color match was collected earlier, + # so we do an extra generation to avoid the corruption introduced by the color match of first output return tube(lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), @@ -64,9 +68,31 @@ def conditional_extra_color_match_tube(init, indexes, images): def conditional_color_match_tube(init, step): + # on strength 0, set color match to generation return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), - is_do_process= - lambda: init.is_do_color_match_conversion(step)) + is_do_process=lambda: init.is_do_color_match_conversion(step)) + + +def conditional_force_to_grayscale_tube(init): + return tube(lambda img: ImageOps.grayscale(img), + lambda img: ImageOps.colorize(img, black="black", white="white"), + is_do_process=lambda: init.args.anim_args.color_force_grayscale) + + +def conditional_add_overlay_mask_tube(init, indexes, is_tween): + is_use_overlay = init.args.args.overlay_mask + is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask + index = indexes.tween.i if is_tween else indexes.frame.i + is_bgr_array = True + return tube(lambda img: ImageOps.grayscale(img), + lambda img: do_overlay_mask(init.args.args, init.args.anim_args, img, index, is_bgr_array), + is_do_process=lambda: is_use_overlay and is_use_mask) + + +def conditional_force_tween_to_grayscale_tube(init): + return tube(lambda img: cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY), + lambda img: cv2.cvtColor(img, cv2.COLOR_GRAY2BGR), + is_do_process=lambda: init.args.anim_args.color_force_grayscale) # Composite Tubes @@ -75,3 +101,11 @@ def contrasted_noise_transformation_tube(init, step, mask): contrast_tube = contrast_transformation_tube(init, step, mask) noise_tube = noise_transformation_tube(init, step) return tube(lambda img: noise_tube(contrast_tube(img))) + + +def conditional_frame_transformation_tube(init, indexes, step, images, is_tween: bool = False): + hybrid_tube = conditional_hybrid_video_after_generation_tube(init, indexes, step) + extra_tube = conditional_extra_color_match_tube(init, indexes, images) + gray_tube = conditional_force_to_grayscale_tube(init) + mask_tube = conditional_add_overlay_mask_tube(init, indexes, is_tween) + return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index 3279f73c8..cb2328483 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -3,34 +3,6 @@ import cv2 from .filename_utils import tween_frame_name -from ...masks import do_overlay_mask - - -def force_tween_to_grayscale_if_required(init, image): - if init.args.anim_args.color_force_grayscale: - gray_image = cv2.cvtColor(image.astype(np.uint8), cv2.COLOR_BGR2GRAY) - return cv2.cvtColor(gray_image, cv2.COLOR_GRAY2BGR) - else: - return image - - -def force_to_grayscale_if_required(init, image): - if init.args.anim_args.color_force_grayscale: - gray_image = ImageOps.grayscale(image) - return ImageOps.colorize(gray_image, black="black", white="white") - else: - return image - - -def add_overlay_mask_if_active(init, image, is_tween: bool = False): - is_use_overlay = init.args.args.overlay_mask - is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask - if is_use_overlay and is_use_mask: - index = indexes.tween.i if is_tween else indexes.frame.i - is_bgr_array = True - return do_overlay_mask(init.args.args, init.args.anim_args, image, index, is_bgr_array) - else: - return image def save_cadence_frame(init, indexes, image): diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 2375604ed..a98463da6 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,7 +1,10 @@ +CYAN = "\033[36m" +YELLOW = "\033[33m" +RESET = "\033[0m" def print_animation_frame_info(init, indexes): - print(f"\033[36mAnimation frame: \033[0m{indexes.frame.i}/{init.args.anim_args.max_frames}") + print(f"{CYAN}Animation frame: {RESET}{indexes.frame.i}/{init.args.anim_args.max_frames}") def print_tween_frame_info(init, indexes, cadence_flow, tween): @@ -22,3 +25,7 @@ def print_optical_flow_info(init, optical_flow_redo_generation): def print_redo_generation_info(init, n): print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") + + +def print_warning_generate_returned_no_image(): + print(f"{YELLOW}Warning: {RESET}Generate returned no image. Skipping to next iteration.") From e17cadd778d0616afe6e22646171249e4a6f21d6 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 12:47:15 +0200 Subject: [PATCH 067/132] Created typealias for tubes and added some typehints and comments to the tubes module. --- .../rendering/img_2_img_tubes.py | 51 +++++++++++-------- .../rendering/util/fun_utils.py | 6 ++- 2 files changed, 34 insertions(+), 23 deletions(-) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 4ba181dec..37b6dabed 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -1,5 +1,8 @@ +from typing import Callable + import cv2 import numpy as np +from cv2.typing import MatLike from .data.step.step import Step from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite @@ -7,9 +10,10 @@ from ..masks import do_overlay_mask """ -This module provides functions for processing images through various transformations. +This module provides functions for conditionally processing images through various transformations. The `tube` function allows chaining these transformations together to create flexible image processing pipelines. -Easily experiment by changing, or changing the order of function calls in the tube. +Easily experiment by changing, or changing the order of function calls in the tube without having to worry +about the larger context and without having to invent unnecessary names for intermediary processing results. All functions within the tube take and return an image (`img` argument). They may (and must) pass through the original image unchanged if a specific transformation is disabled or not required. @@ -18,8 +22,11 @@ transformed_image = my_tube(arguments)(original_image) """ +# ImageTubes are functions that take a MatLike image and return a newly processed (or the same unchanged) MatLike image. +ImageTube = Callable[[MatLike], MatLike] + -def frame_transformation_tube(init, indexes, step, images): +def frame_transformation_tube(init, indexes, step, images) -> ImageTube: # make sure `img` stays the last argument in each call. return tube(lambda img: step.apply_frame_warp_transform(init, indexes, img), lambda img: step.do_hybrid_compositing_before_motion(init, indexes, img), @@ -30,16 +37,16 @@ def frame_transformation_tube(init, indexes, step, images): lambda img: Step.transform_to_grayscale_if_active(init, images, img)) -def contrast_transformation_tube(init, step, mask): +def contrast_transformation_tube(init, step, mask) -> ImageTube: return tube(lambda img: step.apply_scaling(img), lambda img: step.apply_anti_blur(init, mask, img)) -def noise_transformation_tube(init, step): +def noise_transformation_tube(init, step) -> ImageTube: return tube(lambda img: step.apply_frame_noising(init, step, img)) -def optical_flow_redo_tube(init, optical_flow, images): +def optical_flow_redo_tube(init, optical_flow, images) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( # TODO create img.get_flow @@ -47,8 +54,8 @@ def optical_flow_redo_tube(init, optical_flow, images): step.init.redo_flow_factor)) -# Conditional Tubes -def conditional_hybrid_video_after_generation_tube(init, indexes, step): +# Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). +def conditional_hybrid_video_after_generation_tube(init, indexes, step) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: call_hybrid_composite(init, indexes.frame.i, img, step.init.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), @@ -56,7 +63,7 @@ def conditional_hybrid_video_after_generation_tube(init, indexes, step): lambda: indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) -def conditional_extra_color_match_tube(init, indexes, images): +def conditional_extra_color_match_tube(init, indexes, images) -> ImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output return tube(lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), @@ -67,19 +74,19 @@ def conditional_extra_color_match_tube(init, indexes, images): lambda: indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match)) -def conditional_color_match_tube(init, step): +def conditional_color_match_tube(init, step) -> ImageTube: # on strength 0, set color match to generation return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), is_do_process=lambda: init.is_do_color_match_conversion(step)) -def conditional_force_to_grayscale_tube(init): +def conditional_force_to_grayscale_tube(init) -> ImageTube: return tube(lambda img: ImageOps.grayscale(img), lambda img: ImageOps.colorize(img, black="black", white="white"), is_do_process=lambda: init.args.anim_args.color_force_grayscale) -def conditional_add_overlay_mask_tube(init, indexes, is_tween): +def conditional_add_overlay_mask_tube(init, indexes, is_tween) -> ImageTube: is_use_overlay = init.args.args.overlay_mask is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask index = indexes.tween.i if is_tween else indexes.frame.i @@ -89,23 +96,23 @@ def conditional_add_overlay_mask_tube(init, indexes, is_tween): is_do_process=lambda: is_use_overlay and is_use_mask) -def conditional_force_tween_to_grayscale_tube(init): +def conditional_force_tween_to_grayscale_tube(init) -> ImageTube: return tube(lambda img: cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY), lambda img: cv2.cvtColor(img, cv2.COLOR_GRAY2BGR), is_do_process=lambda: init.args.anim_args.color_force_grayscale) -# Composite Tubes -def contrasted_noise_transformation_tube(init, step, mask): +# Composite Tubes, made from other Tubes. +def contrasted_noise_transformation_tube(init, step, mask) -> ImageTube: """Combines contrast and noise transformation tubes.""" - contrast_tube = contrast_transformation_tube(init, step, mask) - noise_tube = noise_transformation_tube(init, step) + contrast_tube: Tube = contrast_transformation_tube(init, step, mask) + noise_tube: Tube = noise_transformation_tube(init, step) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(init, indexes, step, images, is_tween: bool = False): - hybrid_tube = conditional_hybrid_video_after_generation_tube(init, indexes, step) - extra_tube = conditional_extra_color_match_tube(init, indexes, images) - gray_tube = conditional_force_to_grayscale_tube(init) - mask_tube = conditional_add_overlay_mask_tube(init, indexes, is_tween) +def conditional_frame_transformation_tube(init, indexes, step, images, is_tween: bool = False) -> ImageTube: + hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(init, indexes, step) + extra_tube: Tube = conditional_extra_color_match_tube(init, indexes, images) + gray_tube: Tube = conditional_force_to_grayscale_tube(init) + mask_tube: Tube = conditional_add_overlay_mask_tube(init, indexes, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) diff --git a/scripts/deforum_helpers/rendering/util/fun_utils.py b/scripts/deforum_helpers/rendering/util/fun_utils.py index b7f1a1084..48899341d 100644 --- a/scripts/deforum_helpers/rendering/util/fun_utils.py +++ b/scripts/deforum_helpers/rendering/util/fun_utils.py @@ -1,6 +1,7 @@ import collections.abc from functools import reduce from itertools import chain +from typing import Callable, TypeVar def flat_map(func, iterable): @@ -12,6 +13,9 @@ def flat_map(func, iterable): return mapped_iterable -def tube(*funcs, is_do_process=lambda: True): +T = TypeVar('T') + + +def tube(*funcs: Callable[[T], T], is_do_process=lambda: True) -> Callable[[T], T]: """Tubes a value through a sequence of functions with a predicate `is_do_process` for skipping.""" return lambda value: reduce(lambda x, f: f(x) if is_do_process() else x, funcs, value) From 0f0f14668846bf4f8d45d60b081b1619d0a46bde Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 12:58:34 +0200 Subject: [PATCH 068/132] Removing .idea directory from main branches. --- .gitignore | 3 +++ .idea/.gitignore | 3 --- .idea/deforum.iml | 12 ------------ .idea/inspectionProfiles/profiles_settings.xml | 7 ------- .idea/misc.xml | 7 ------- .idea/modules.xml | 8 -------- .idea/runConfigurations/Deforum_Tests.xml | 18 ------------------ .idea/runConfigurations/Postprocess_Tests.xml | 18 ------------------ .idea/runConfigurations/WebUI_with_API.xml | 17 ----------------- .idea/vcs.xml | 6 ------ 10 files changed, 3 insertions(+), 96 deletions(-) delete mode 100644 .idea/.gitignore delete mode 100644 .idea/deforum.iml delete mode 100644 .idea/inspectionProfiles/profiles_settings.xml delete mode 100644 .idea/misc.xml delete mode 100644 .idea/modules.xml delete mode 100644 .idea/runConfigurations/Deforum_Tests.xml delete mode 100644 .idea/runConfigurations/Postprocess_Tests.xml delete mode 100644 .idea/runConfigurations/WebUI_with_API.xml delete mode 100644 .idea/vcs.xml diff --git a/.gitignore b/.gitignore index 1b0eb546e..8fcf14a14 100644 --- a/.gitignore +++ b/.gitignore @@ -32,3 +32,6 @@ htmlcov tests/results.xml .coverage* serverlog.txt + +# PyCharm config +.idea diff --git a/.idea/.gitignore b/.idea/.gitignore deleted file mode 100644 index 26d33521a..000000000 --- a/.idea/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -# Default ignored files -/shelf/ -/workspace.xml diff --git a/.idea/deforum.iml b/.idea/deforum.iml deleted file mode 100644 index 8b8c39547..000000000 --- a/.idea/deforum.iml +++ /dev/null @@ -1,12 +0,0 @@ - - - - - - - - - - \ No newline at end of file diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml deleted file mode 100644 index dd4c951ef..000000000 --- a/.idea/inspectionProfiles/profiles_settings.xml +++ /dev/null @@ -1,7 +0,0 @@ - - - - \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml deleted file mode 100644 index b3fbd492f..000000000 --- a/.idea/misc.xml +++ /dev/null @@ -1,7 +0,0 @@ - - - - - - \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml deleted file mode 100644 index 1b7de0730..000000000 --- a/.idea/modules.xml +++ /dev/null @@ -1,8 +0,0 @@ - - - - - - - - \ No newline at end of file diff --git a/.idea/runConfigurations/Deforum_Tests.xml b/.idea/runConfigurations/Deforum_Tests.xml deleted file mode 100644 index 4ecd59c57..000000000 --- a/.idea/runConfigurations/Deforum_Tests.xml +++ /dev/null @@ -1,18 +0,0 @@ - - - - - \ No newline at end of file diff --git a/.idea/runConfigurations/Postprocess_Tests.xml b/.idea/runConfigurations/Postprocess_Tests.xml deleted file mode 100644 index d2a930043..000000000 --- a/.idea/runConfigurations/Postprocess_Tests.xml +++ /dev/null @@ -1,18 +0,0 @@ - - - - - \ No newline at end of file diff --git a/.idea/runConfigurations/WebUI_with_API.xml b/.idea/runConfigurations/WebUI_with_API.xml deleted file mode 100644 index 3fe93568e..000000000 --- a/.idea/runConfigurations/WebUI_with_API.xml +++ /dev/null @@ -1,17 +0,0 @@ - - - - \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml deleted file mode 100644 index 35eb1ddfb..000000000 --- a/.idea/vcs.xml +++ /dev/null @@ -1,6 +0,0 @@ - - - - - - \ No newline at end of file From 4c88c8cda0c915f77b96a1814c947539ef8fb687 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 13:20:38 +0200 Subject: [PATCH 069/132] Links in README. --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index a78488cfa..45d59db86 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,12 @@ Please refer to the original project for a stable version: [https://github.com/d This section details the changes made to the render core in this fork. -For easy integration, this fork isolates changes to the `render.py` module and introduces the `rendering` package. +For easy integration, this fork isolates changes to the [~`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [~`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. Existing code in all other Deforum modules remains untouched. * **Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. * **Key Improvements:** - * Reduced cyclomatic complexity of the `render_animation` method. + * Reduced cyclomatic complexity of the [~`render_animation`](~https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. * Improved separation of concerns (e.g., dedicated module for printing). * Reduced argument complexity and improved scope control and variable organization using dataclasses. * Enhanced code clarity with improved naming and removal of obsolete comments. @@ -24,9 +24,10 @@ Existing code in all other Deforum modules remains untouched. * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. **New Rendering Modules:** -* `rendering/data`: Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. -* `rendering/util`: Contains stateless utility functions for data transformation and manipulation used in `render.py`. -* `rendering/util/call`: Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. +* [~`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. +* [~`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. +* [~`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. +* [~`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. **Implementation Details:** * **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. From 458e16465b0c30d5432e7a76bc8c26b7b40c158a Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 15:35:54 +0200 Subject: [PATCH 070/132] Tween step generation refactored. --- README.md | 12 ++-- scripts/deforum_helpers/render.py | 36 ++++++------ .../deforum_helpers/rendering/data/indexes.py | 9 ++- .../rendering/data/step/step.py | 6 +- .../rendering/data/step/tween_step.py | 56 +++++++++++-------- .../rendering/util/image_utils.py | 5 ++ 6 files changed, 71 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index 45d59db86..bb8cef004 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,12 @@ Please refer to the original project for a stable version: [https://github.com/d This section details the changes made to the render core in this fork. -For easy integration, this fork isolates changes to the [~`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [~`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. +For easy integration, this fork isolates changes to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. Existing code in all other Deforum modules remains untouched. * **Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. * **Key Improvements:** - * Reduced cyclomatic complexity of the [~`render_animation`](~https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. + * Reduced cyclomatic complexity of the [`render_animation`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. * Improved separation of concerns (e.g., dedicated module for printing). * Reduced argument complexity and improved scope control and variable organization using dataclasses. * Enhanced code clarity with improved naming and removal of obsolete comments. @@ -24,10 +24,10 @@ Existing code in all other Deforum modules remains untouched. * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. **New Rendering Modules:** -* [~`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. -* [~`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. -* [~`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. -* [~`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. +* [`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. +* [`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. +* [`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. +* [`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. **Implementation Details:** * **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index ca8ca78e8..8e22d4b53 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -32,6 +32,7 @@ from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview +from .rendering.util.image_utils import save_and_return_frame from .rendering.util.log_utils import (print_animation_frame_info, print_optical_flow_info, print_redo_generation_info, print_warning_generate_returned_no_image) from .save_images import save_image @@ -88,26 +89,25 @@ def maybe_emit_in_between_frames(init, indexes, step, turbo, images): emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, indexes.frame.i) -def emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, frame_i): +def emit_frames_between_index_pair(init, indexes, last_step, turbo, images, tween_frame_start_i, frame_i): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" - # TODO refactor until this works with just 2 args: RenderInit and a collection of immutable TweenStep objects. - indexes.update_tween_start(turbo) - tween_range = range(tween_frame_start_i, frame_i) - - # TODO Instead of indexes, pass a set of TweenStep objects to be processed instead of creating them in the loop. - for tween_index in tween_range: - # TODO tween index shouldn't really be updated and passed around like this here. - # ideally provide index data within an immutable TweenStep instance. - indexes.update_tween(tween_index) # TODO Nope - - tween_step = TweenStep.create(indexes) - - TweenStep.handle_synchronous_status_concerns(init, indexes, step, tween_step) - TweenStep.process(init, indexes, step, turbo, images, tween_step) - new_image = TweenStep.generate_and_save_frame(init, indexes, step, turbo, tween_step) - - images.previous = new_image # TODO shouldn't + tween_indexes_list: List[Indexes] = TweenStep.create_indexes(indexes, tween_range) + tween_steps: List[TweenStep] = TweenStep.create_steps(last_step, tween_indexes_list) + indexes.update_tween_start(turbo) # TODO... + emit_tween_frames(init, tween_steps, turbo, images) + + +def emit_tween_frames(init, tween_steps, turbo, images): + """Emits a tween frame for each provided tween_step.""" + for tween_step in tween_steps: + tween_step.handle_synchronous_status_concerns(init) + tween_step.process(init, turbo, images) # side effects on turbo and on step + + new_image = tween_step.generate(init, turbo) + # TODO pass depth instead of tween_step.indexes + new_image = save_and_return_frame(init, tween_step.indexes, new_image) + images.previous = new_image # updating reference images to calculate hybrid motions in next iteration def generate_depth_maps_if_active(init): diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index b72d1392d..c2072145e 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -18,8 +18,13 @@ def create(init, turbo): tween_start = 0 return Indexes(IndexWithStart(frame_start, 0), IndexWithStart(tween_start, 0)) - def update_tween(self, i: int): - self.tween = IndexWithStart(self.tween.start, i) + @staticmethod + def create_from_last(last_indexes, i: int): + """Creates a new `Indexes` object based on the last one, but updates the tween start index.""" + return Indexes(last_indexes.frame, IndexWithStart(last_indexes.tween.start, i)) + + def create_next(self): + return Indexes(self.frame, IndexWithStart(self.tween.start, self.tween.i + 1)) def update_tween_start(self, turbo): tween_start = max(self.frame.start, self.frame.i - turbo.steps) diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 8f447c5c1..f2ac7e4a7 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -101,11 +101,11 @@ def write_frame_subtitle(self, init, indexes, turbo): self.subtitle_params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) call_write_frame_subtitle(init, indexes.frame.i, params_string) - def write_frame_subtitle_if_active(self, init, indexes): + def write_frame_subtitle_if_active(self, init): if opt_utils.is_generate_subtitles(init): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) - self.subtitle_params_string = call_format_animation_params(init, indexes.tween.i, params_to_print) - call_write_frame_subtitle(init, indexes.tween.i, params_string, sub_step.tween < 1.0) + self.subtitle_params_string = call_format_animation_params(init, self.indexes.tween.i, params_to_print) + call_write_frame_subtitle(init, self.indexes.tween.i, params_string, sub_step.tween < 1.0) def apply_frame_warp_transform(self, init, indexes, image): previous, self.depth = call_anim_frame_warp(init, indexes.frame.i, image, None) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index a977af79e..8bc0326b6 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -1,13 +1,19 @@ from dataclasses import dataclass -from typing import Any +from itertools import chain +from typing import Any, Iterable -from ...util import image_utils, log_utils, web_ui_utils +from ...data.indexes import Indexes +from ...data.step import Step +from ...data.turbo import Turbo from ...img_2_img_tubes import conditional_force_tween_to_grayscale_tube, conditional_add_overlay_mask_tube +from ...util import log_utils, web_ui_utils @dataclass(init=True, frozen=False, repr=False, eq=False) class TweenStep: """cadence vars""" + indexes: Indexes + last_step: Step tween: float cadence_flow: Any cadence_flow_inc: Any @@ -17,38 +23,40 @@ def _calculate_tween_from_indices(frame_difference, last_step) -> float: return min(0.0, max(1.0, float(last_step) / float(frame_difference))) @staticmethod - def create(indexes): + def create(indexes, last_step): tween = float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start) - return TweenStep(tween, None, None) + return TweenStep(indexes, last_step, tween, None, None) + + @staticmethod + def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[Indexes]: + return list(chain.from_iterable([Indexes.create_from_last(base_indexes, i)] for i in frame_range)) + + @staticmethod + def create_steps(last_step, tween_indexes_list: list[Indexes]) -> list['TweenStep']: + return [TweenStep.create(i.create_next(), last_step) for i in tween_indexes_list] @staticmethod def create_directly(from_index, to_index): tween = TweenStep._calculate_tween_from_indices(from_index, to_index) return TweenStep(tween, None, None) - def generate_tween_image(self, init, indexes, step, turbo): + def generate_tween_image(self, init, turbo): is_tween = True - warped = turbo.do_optical_flow_cadence_after_animation_warping(init, indexes, step, self) + warped = turbo.do_optical_flow_cadence_after_animation_warping(init, self.indexes, self.last_step, self) recolored = conditional_force_tween_to_grayscale_tube(init)(warped) - masked = conditional_add_overlay_mask_tube(init, indexes, is_tween)(recolored) + masked = conditional_add_overlay_mask_tube(init, self.indexes, is_tween)(recolored) return masked - @staticmethod - def process(init, indexes, step, turbo, images, tween_step): - turbo.advance_optical_flow_cadence_before_animation_warping(init, tween_step) - step.update_depth_prediction(init, turbo) - turbo.advance(init, indexes.tween.i, step.depth) - turbo.do_hybrid_video_motion(init, indexes, images) + def generate(self, init, turbo): + return self.generate_tween_image(init, turbo) - @staticmethod - def generate_and_save_frame(init, indexes, step, turbo, tween_step, is_do_save: bool = True): - new_image = tween_step.generate_tween_image(init, indexes, step, turbo) - if is_do_save: - image_utils.save_cadence_frame_and_depth_map_if_active(init, indexes, new_image) - return new_image + def process(self, init, turbo, images): + turbo.advance_optical_flow_cadence_before_animation_warping(init, self) + self.last_step.update_depth_prediction(init, turbo) + turbo.advance(init, self.indexes.tween.i, self.last_step.depth) + turbo.do_hybrid_video_motion(init, self.indexes, images) - @staticmethod - def handle_synchronous_status_concerns(init, indexes, step, tween_step): - step.write_frame_subtitle_if_active(init, indexes) # TODO decouple from execution and calc all in advance? - log_utils.print_tween_frame_info(init, indexes, tween_step.cadence_flow, tween_step.tween) - web_ui_utils.update_progress_during_cadence(init, indexes) + def handle_synchronous_status_concerns(self, init): + self.last_step.write_frame_subtitle_if_active(init) # TODO decouple from execution and calc all in advance? + log_utils.print_tween_frame_info(init, self.indexes, self.cadence_flow, self.tween) + web_ui_utils.update_progress_during_cadence(init, self.indexes) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index cb2328483..02a4624c5 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -16,3 +16,8 @@ def save_cadence_frame_and_depth_map_if_active(init, indexes, image): if init.args.anim_args.save_depth_maps: dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) init.depth_model.save(dm_save_path, step.depth) + + +def save_and_return_frame(init, indexes, image): + save_cadence_frame_and_depth_map_if_active(init, indexes, image) + return image From 6a56672f11280ddfdb9ffe1f1710347a7fa73930 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 17:02:59 +0200 Subject: [PATCH 071/132] Reduced argument complexity by moving state objects to RenderInit. --- scripts/deforum_helpers/render.py | 119 +++++++++--------- .../rendering/data/initialization.py | 39 ++++-- .../rendering/data/step/step.py | 77 +++++++----- .../rendering/data/step/tween_step.py | 20 +-- .../rendering/img_2_img_tubes.py | 54 ++++---- .../rendering/util/call/gen.py | 4 +- .../rendering/util/call/video_and_audio.py | 3 +- .../rendering/util/log_utils.py | 4 +- .../rendering/util/web_ui_utils.py | 8 +- 9 files changed, 176 insertions(+), 152 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 8e22d4b53..6e0ca232b 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -23,7 +23,7 @@ from modules.shared import opts, state from .colors import maintain_colors -from .rendering.data import Turbo, Images, Indexes, Mask +from .rendering.data import Indexes from .rendering.data.initialization import RenderInit from .rendering.data.step import Step, TweenStep from .rendering.img_2_img_tubes import (conditional_color_match_tube, conditional_frame_transformation_tube, @@ -33,8 +33,8 @@ from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview from .rendering.util.image_utils import save_and_return_frame -from .rendering.util.log_utils import (print_animation_frame_info, print_optical_flow_info, - print_redo_generation_info, print_warning_generate_returned_no_image) +from .rendering.util.log_utils import (print_optical_flow_info, print_redo_generation_info, + print_warning_generate_returned_no_image) from .save_images import save_image from .seed import next_seed @@ -45,69 +45,62 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(init): - images = Images.create(init) - turbo = Turbo.create(init) - indexes = Indexes.create(init, turbo) - mask = Mask.create(init, indexes.frame.i) # reset mask vals as they are overwritten in the compose_mask algorithm web_ui_utils.init_job(init) last_preview_frame = 0 - while indexes.frame.i < init.args.anim_args.max_frames: - memory_utils.handle_med_or_low_vram_before_step(init) - print_animation_frame_info(init, indexes) - web_ui_utils.update_job(init, indexes) - step = Step.create(init, indexes) - step.write_frame_subtitle(init, indexes, turbo) - - maybe_emit_in_between_frames(init, indexes, step, turbo, images) - - images.color_match = Step.create_color_match_for_video(init, indexes) - images.previous = transform_and_update_noised_sample(init, indexes, step, images, mask) - init.prepare_generation(init, indexes, step, mask) - maybe_redo_optical_flow(init, indexes, step, images) - maybe_redo_diffusion(init, indexes, step, images) - - image = call_generate(init, indexes.frame.i, step.schedule) + while init.indexes.frame.i < init.args.anim_args.max_frames: + step = Step.do_start_and_create(init) # TODO step should have an immutable init? + step.write_frame_subtitle(init) # TODO move step concerns from init to step.. + + maybe_emit_in_between_frames(init, step) + + init.images.color_match = Step.create_color_match_for_video(init) # TODO move to step? + init.images.previous = transform_and_update_noised_sample(init, step) # TODO move to step? + init.prepare_generation(init, step) # TODO move to step? + maybe_redo_optical_flow(init, step) # TODO move to step? + maybe_redo_diffusion(init, step) # TODO move to step? + + image = call_generate(init, step) # TODO move to step? if image is None: print_warning_generate_returned_no_image() break - image = conditional_frame_transformation_tube(init, indexes, step, images)(image) - images.color_match = conditional_color_match_tube(init, step)(image) - next_frame, step.depth = progress_step(init, indexes, turbo, images, image, step.depth) - indexes.update_frame(next_frame) + image = conditional_frame_transformation_tube(init, step)(image) # TODO move to step? + init.images.color_match = conditional_color_match_tube(init, step)(image) # TODO move to step? + next_frame, step.depth = progress_step(init, image, step.depth) # TODO move to step? + init.indexes.update_frame(next_frame) state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(init, indexes.frame.i, last_preview_frame) - web_ui_utils.update_status_tracker(init, indexes) + last_preview_frame = call_render_preview(init, last_preview_frame) + web_ui_utils.update_status_tracker(init) init.animation_mode.unload_raft_and_depth_model() -def maybe_emit_in_between_frames(init, indexes, step, turbo, images): - if turbo.is_emit_in_between_frames(): - tween_frame_start_i = max(indexes.frame.start, indexes.frame.i - turbo.steps) - emit_frames_between_index_pair(init, indexes, step, turbo, images, tween_frame_start_i, indexes.frame.i) +def maybe_emit_in_between_frames(init, step): + if init.turbo.is_emit_in_between_frames(): + tween_frame_start_i = max(init.indexes.frame.start, init.indexes.frame.i - init.turbo.steps) + emit_frames_between_index_pair(init, step, tween_frame_start_i, init.indexes.frame.i) -def emit_frames_between_index_pair(init, indexes, last_step, turbo, images, tween_frame_start_i, frame_i): +def emit_frames_between_index_pair(init, last_step, tween_frame_start_i, frame_i): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" tween_range = range(tween_frame_start_i, frame_i) - tween_indexes_list: List[Indexes] = TweenStep.create_indexes(indexes, tween_range) + tween_indexes_list: List[Indexes] = TweenStep.create_indexes(init.indexes, tween_range) tween_steps: List[TweenStep] = TweenStep.create_steps(last_step, tween_indexes_list) - indexes.update_tween_start(turbo) # TODO... - emit_tween_frames(init, tween_steps, turbo, images) + init.indexes.update_tween_start(init.turbo) # TODO... + emit_tween_frames(init, tween_steps) -def emit_tween_frames(init, tween_steps, turbo, images): +def emit_tween_frames(init, tween_steps): """Emits a tween frame for each provided tween_step.""" for tween_step in tween_steps: tween_step.handle_synchronous_status_concerns(init) - tween_step.process(init, turbo, images) # side effects on turbo and on step + tween_step.process(init) # side effects on turbo and on step - new_image = tween_step.generate(init, turbo) + new_image = tween_step.generate(init) # TODO pass depth instead of tween_step.indexes new_image = save_and_return_frame(init, tween_step.indexes, new_image) - images.previous = new_image # updating reference images to calculate hybrid motions in next iteration + init.images.previous = new_image # updating reference images to calculate hybrid motions in next iteration def generate_depth_maps_if_active(init): @@ -121,25 +114,25 @@ def generate_depth_maps_if_active(init): return depth -def progress_step(init, idx, turbo, images, image, depth): +def progress_step(init, image, depth): """Will progress frame or turbo-frame step and return next index and `depth`.""" opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) if not init.animation_mode.has_video_input: - images.previous = opencv_image - if turbo.has_steps(): - return idx.frame.i + turbo.progress_step(idx, opencv_image), depth + init.images.previous = opencv_image + if init.turbo.has_steps(): + return init.indexes.frame.i + init.turbo.progress_step(init.indexes, opencv_image), depth else: - filename = filename_utils.frame(init, idx) + filename = filename_utils.frame(init, init.indexes) save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) depth = generate_depth_maps_if_active(init) - return idx.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. + return init.indexes.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. -def transform_and_update_noised_sample(init, indexes, step, images, mask): - if images.has_previous(): # skipping 1st iteration - transformed_image = frame_transformation_tube(init, indexes, step, images)(images.previous) +def transform_and_update_noised_sample(init, step): + if init.images.has_previous(): # skipping 1st iteration + transformed_image = frame_transformation_tube(init, step)(init.images.previous) # TODO separate - noised_image = contrasted_noise_transformation_tube(init, step, mask)(transformed_image) + noised_image = contrasted_noise_transformation_tube(init, step)(transformed_image) init.update_sample_and_args_for_current_progression_step(step, noised_image) return transformed_image else: @@ -147,45 +140,45 @@ def transform_and_update_noised_sample(init, indexes, step, images, mask): # Conditional Redoes -def maybe_redo_optical_flow(init, indexes, step, images): +def maybe_redo_optical_flow(init, step): optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() - is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, images) + is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, init.images) if is_redo_optical_flow: - init.args.root.init_sample = do_optical_flow_redo_before_generation(init, indexes, step, images) + init.args.root.init_sample = do_optical_flow_redo_before_generation(init, step) -def maybe_redo_diffusion(init, indexes, step, images): - is_diffusion_redo = init.has_positive_diffusion_redo and images.has_previous() and step.init.has_strength() +def maybe_redo_diffusion(init, step): + is_diffusion_redo = init.has_positive_diffusion_redo and init.images.has_previous() and step.init.has_strength() is_not_preview = init.is_not_in_motion_preview_mode() if is_diffusion_redo and is_not_preview: - do_diffusion_redo(init, indexes, step, images) + do_diffusion_redo(init, step) -def do_optical_flow_redo_before_generation(init, indexes, step, images): +def do_optical_flow_redo_before_generation(init, step): stored_seed = init.args.args.seed # keep original to reset it after executing the optical flow init.args.args.seed = generate_random_seed() # set a new random seed print_optical_flow_info(init, optical_flow_redo_generation) # TODO output temp seed? - sample_image = call_generate(init, indexes.frame.i, step.schedule) - optical_tube = optical_flow_redo_tube(init, optical_flow_redo_generation, images) + sample_image = call_generate(init, init.indexes.frame.i, step.schedule) + optical_tube = optical_flow_redo_tube(init, optical_flow_redo_generation) transformed_sample_image = optical_tube(sample_image) init.args.args.seed = stored_seed # restore stored seed return Image.fromarray(transformed_sample_image) -def do_diffusion_redo(init, indexes, step, images): +def do_diffusion_redo(init, step): stored_seed = init.args.args.seed last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) for n in range(0, last_diffusion_redo_index): print_redo_generation_info(init, n) init.args.args.seed = generate_random_seed() - diffusion_redo_image = call_generate(init, indexes.frame.i, step.schedule) + diffusion_redo_image = call_generate(init, step) diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) # color match on last one only is_last_iteration = n == last_diffusion_redo_index if is_last_iteration: mode = init.args.anim_args.color_coherence - diffusion_redo_image = maintain_colors(images.previous, images.color_match, mode) + diffusion_redo_image = maintain_colors(init.images.previous, init.images.color_match, mode) init.args.args.seed = stored_seed init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/initialization.py index 4172e5dac..66747a21c 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/initialization.py @@ -9,7 +9,11 @@ from PIL import Image from .anim import AnimationKeys, AnimationMode +from .images import Images +from .indexes import Indexes +from .mask import Mask from .subtitle import Srt +from .turbo import Turbo from ..util import memory_utils, opt_utils from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame @@ -37,6 +41,10 @@ class RenderInitArgs: @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderInit: """The purpose of this class is to group and control all data used in render_animation""" + images: Images | None + turbo: Turbo | None + indexes: Indexes | None + mask: Mask | None seed: int args: RenderInitArgs parseq_adapter: Any @@ -51,6 +59,7 @@ class RenderInit: @staticmethod def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + output_directory = args.outdir is_use_mask = args.use_mask parseq_adapter = RenderInit.create_parseq_adapter(ri_args) @@ -60,7 +69,17 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args) - instance = RenderInit(args.seed, ri_args, parseq_adapter, srt, animation_keys, + + # Temporary instance only exists for using it to easily create other objects required by the actual instance. + # Feels slightly awkward, but it's probably not worth optimizing since only 1st and gc can take care of it fine. + incomplete_init = RenderInit(None, None, None, None, args.seed, ri_args, parseq_adapter, srt, animation_keys, + animation_mode, prompt_series, depth_model, output_directory, is_use_mask) + images = Images.create(incomplete_init) + turbo = Turbo.create(incomplete_init) + indexes = Indexes.create(incomplete_init, turbo) + mask = Mask.create(incomplete_init, indexes.frame.i) + + instance = RenderInit(images, turbo, indexes, mask, args.seed, ri_args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) RenderInit.init_looper_if_active(args, loop_args) RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) @@ -109,7 +128,7 @@ def is_hybrid_composite_before_motion(self) -> bool: def is_hybrid_composite_after_generation(self) -> bool: return self.args.anim_args.hybrid_composite == 'After Generation' - def is_color_match_to_be_initialized(self, color_match_sample): + def is_initialize_color_match(self, color_match_sample): """Determines whether to initialize color matching based on the given conditions.""" has_video_input = self.args.anim_args.color_coherence == 'Video Input' and self.is_hybrid_available() has_image_color_coherence = self.args.anim_args.color_coherence == 'Image' @@ -260,15 +279,15 @@ def update_mask_image(self, step, mask): else: self.args.args.mask_image = None # we need it only after the first frame anyway - def prepare_generation(self, init, indexes, step, mask): + def prepare_generation(self, init, step): # TODO move all of this to Step? - self.update_some_args_for_current_step(indexes, step) - self.update_seed_and_checkpoint_for_current_step(indexes) - self.update_sub_seed_schedule_for_current_step(indexes) - self.prompt_for_current_step(indexes) - self.update_video_data_for_current_frame(indexes, step) - self.update_mask_image(step, mask) - self.animation_keys.update(indexes.frame.i) + self.update_some_args_for_current_step(init.indexes, step) + self.update_seed_and_checkpoint_for_current_step(init.indexes) + self.update_sub_seed_schedule_for_current_step(init.indexes) + self.prompt_for_current_step(init.indexes) + self.update_video_data_for_current_frame(init.indexes, step) + self.update_mask_image(step, init.mask) + self.animation_keys.update(init.indexes.frame.i) opt_utils.setup(self, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(init) diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index f2ac7e4a7..4c943ce51 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -7,7 +7,10 @@ from ..initialization import RenderInit from ..schedule import Schedule from ..turbo import Turbo -from ...util import opt_utils +from ..images import Images +from ..indexes import Indexes +from ..mask import Mask +from ...util import memory_utils, opt_utils, web_ui_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.hybrid import ( call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, @@ -17,6 +20,7 @@ from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ....hybrid_video import image_transform_ransac, image_transform_optical_flow +from ...util.log_utils import print_animation_frame_info @dataclass(init=True, frozen=True, repr=False, eq=False) @@ -77,9 +81,14 @@ class Step: subtitle_params_string: str @staticmethod - def create(init, indexes): - step_init = StepInit.create(init.animation_keys.deform_keys, indexes.frame.i) - schedule = Schedule.create(init, indexes.frame.i, init.args.anim_args, init.args.args) + def do_start_and_create(init): + # Perform the necessary side effects + memory_utils.handle_med_or_low_vram_before_step(init) + print_animation_frame_info(init) + web_ui_utils.update_job(init) + # Actual create + step_init = StepInit.create(init.animation_keys.deform_keys, init.indexes.frame.i) + schedule = Schedule.create(init, init.indexes.frame.i, init.args.anim_args, init.args.args) return Step(step_init, schedule, None, None, "") def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): @@ -95,11 +104,11 @@ def update_depth_prediction(self, init: RenderInit, turbo: Turbo): precision = init.args.root.half_precision self.depth = init.depth_model.predict(image, weight, precision) - def write_frame_subtitle(self, init, indexes, turbo): - if turbo.is_first_step_with_subtitles(init): + def write_frame_subtitle(self, init): + if init.turbo.is_first_step_with_subtitles(init): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) - self.subtitle_params_string = call_format_animation_params(init, indexes.frame.i, params_to_print) - call_write_frame_subtitle(init, indexes.frame.i, params_string) + self.subtitle_params_string = call_format_animation_params(init, init.indexes.frame.i, params_to_print) + call_write_frame_subtitle(init, init.indexes.frame.i, params_string) def write_frame_subtitle_if_active(self, init): if opt_utils.is_generate_subtitles(init): @@ -107,12 +116,12 @@ def write_frame_subtitle_if_active(self, init): self.subtitle_params_string = call_format_animation_params(init, self.indexes.tween.i, params_to_print) call_write_frame_subtitle(init, self.indexes.tween.i, params_string, sub_step.tween < 1.0) - def apply_frame_warp_transform(self, init, indexes, image): - previous, self.depth = call_anim_frame_warp(init, indexes.frame.i, image, None) + def apply_frame_warp_transform(self, init, image): + previous, self.depth = call_anim_frame_warp(init, init.indexes.frame.i, image, None) return previous - def _do_hybrid_compositing_on_cond(self, init, indexes, image, condition): - i = indexes.frame.i + def _do_hybrid_compositing_on_cond(self, init, image, condition): + i = init.indexes.frame.i schedules = self.init.hybrid_comp_schedules if condition: _, composed = call_hybrid_composite(init, i, image, schedules) @@ -120,20 +129,20 @@ def _do_hybrid_compositing_on_cond(self, init, indexes, image, condition): else: return image - def do_hybrid_compositing_before_motion(self, init, indexes, image): + def do_hybrid_compositing_before_motion(self, init, image): condition = init.is_hybrid_composite_before_motion() - return self._do_hybrid_compositing_on_cond(init, indexes, image, condition) + return self._do_hybrid_compositing_on_cond(init, image, condition) - def do_normal_hybrid_compositing_after_motion(self, init, indexes, image): + def do_normal_hybrid_compositing_after_motion(self, init, image): condition = init.is_normal_hybrid_composite() - return self._do_hybrid_compositing_on_cond(init, indexes, image, condition) + return self._do_hybrid_compositing_on_cond(init, image, condition) def apply_scaling(self, image): return (image * self.init.contrast).round().astype(np.uint8) - def apply_anti_blur(self, init, mask, image): + def apply_anti_blur(self, init, image): if self.init.amount > 0: - return call_unsharp_mask(init, self, image, mask) + return call_unsharp_mask(init, self, image, init.mask) else: return image @@ -146,30 +155,31 @@ def apply_frame_noising(self, init, mask, image): return call_add_noise(init, self, image) @staticmethod - def apply_color_matching(init, images, image): + def apply_color_matching(init, image): if init.has_color_coherence(): - if images.color_match is None: + if init.images.color_match is None: # TODO questionable # initialize color_match for next iteration with current image, but don't do anything yet. - images.color_match = image.copy() + init.images.color_match = image.copy() else: - return maintain_colors(image, images.color_match, init.args.anim_args.color_coherence) + return maintain_colors(image, init.images.color_match, init.args.anim_args.color_coherence) return image @staticmethod - def transform_to_grayscale_if_active(init, images, image): + def transform_to_grayscale_if_active(init, image): if init.args.anim_args.color_force_grayscale: - grayscale = cv2.cvtColor(images.previous, cv2.COLOR_BGR2GRAY) + grayscale = cv2.cvtColor(init.images.previous, cv2.COLOR_BGR2GRAY) return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) else: return image @staticmethod - def apply_hybrid_motion_ransac_transform(init, indexes, reference_images, image): + def apply_hybrid_motion_ransac_transform(init, image): """hybrid video motion - warps images.previous to match motion, usually to prepare for compositing""" motion = init.args.anim_args.hybrid_motion if motion in ['Affine', 'Perspective']: - last_i = indexes.frame.i - 1 + last_i = init.indexes.frame.i - 1 + reference_images = init.images matrix = call_get_matrix_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ if init.args.anim_args.hybrid_motion_use_prev_img \ else call_get_matrix_for_hybrid_motion(init, last_i) @@ -177,10 +187,11 @@ def apply_hybrid_motion_ransac_transform(init, indexes, reference_images, image) return image @staticmethod - def apply_hybrid_motion_optical_flow(init, indexes, reference_images, image): + def apply_hybrid_motion_optical_flow(init, image): motion = init.args.anim_args.hybrid_motion if motion in ['Optical Flow']: - last_i = indexes.frame.i - 1 + last_i = init.indexes.frame.i - 1 + reference_images = init.images flow = call_get_flow_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ if init.args.anim_args.hybrid_motion_use_prev_img \ else call_get_flow_for_hybrid_motion(init, last_i) @@ -191,11 +202,11 @@ def apply_hybrid_motion_optical_flow(init, indexes, reference_images, image): return image @staticmethod - def create_color_match_for_video(init, indexes): + def create_color_match_for_video(init): if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(preview_video_image_path(init, indexes)) + if int(init.indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: + prev_vid_img = Image.open(preview_video_image_path(init, init.indexes)) prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) - images.color_match = np.asarray(prev_vid_img) - return cv2.cvtColor(images.color_match, cv2.COLOR_RGB2BGR) + init.images.color_match = np.asarray(prev_vid_img) + return cv2.cvtColor(init.images.color_match, cv2.COLOR_RGB2BGR) return None diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 8bc0326b6..72f301d23 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -40,21 +40,21 @@ def create_directly(from_index, to_index): tween = TweenStep._calculate_tween_from_indices(from_index, to_index) return TweenStep(tween, None, None) - def generate_tween_image(self, init, turbo): + def generate_tween_image(self, init): is_tween = True - warped = turbo.do_optical_flow_cadence_after_animation_warping(init, self.indexes, self.last_step, self) + warped = init.turbo.do_optical_flow_cadence_after_animation_warping(init, self.indexes, self.last_step, self) recolored = conditional_force_tween_to_grayscale_tube(init)(warped) - masked = conditional_add_overlay_mask_tube(init, self.indexes, is_tween)(recolored) + masked = conditional_add_overlay_mask_tube(init, is_tween)(recolored) return masked - def generate(self, init, turbo): - return self.generate_tween_image(init, turbo) + def generate(self, init): + return self.generate_tween_image(init) - def process(self, init, turbo, images): - turbo.advance_optical_flow_cadence_before_animation_warping(init, self) - self.last_step.update_depth_prediction(init, turbo) - turbo.advance(init, self.indexes.tween.i, self.last_step.depth) - turbo.do_hybrid_video_motion(init, self.indexes, images) + def process(self, init): + init.turbo.advance_optical_flow_cadence_before_animation_warping(init, self) + self.last_step.update_depth_prediction(init, init.turbo) + init.turbo.advance(init, self.indexes.tween.i, self.last_step.depth) + init.turbo.do_hybrid_video_motion(init, self.indexes, init.images) # TODO remove self.indexes or init.indexes def handle_synchronous_status_concerns(self, init): self.last_step.write_frame_subtitle_if_active(init) # TODO decouple from execution and calc all in advance? diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 37b6dabed..61b370268 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -26,52 +26,52 @@ ImageTube = Callable[[MatLike], MatLike] -def frame_transformation_tube(init, indexes, step, images) -> ImageTube: +def frame_transformation_tube(init, step) -> ImageTube: # make sure `img` stays the last argument in each call. - return tube(lambda img: step.apply_frame_warp_transform(init, indexes, img), - lambda img: step.do_hybrid_compositing_before_motion(init, indexes, img), - lambda img: Step.apply_hybrid_motion_ransac_transform(init, indexes, images, img), - lambda img: Step.apply_hybrid_motion_optical_flow(init, indexes, images, img), - lambda img: step.do_normal_hybrid_compositing_after_motion(init, indexes, img), - lambda img: Step.apply_color_matching(init, images, img), - lambda img: Step.transform_to_grayscale_if_active(init, images, img)) + return tube(lambda img: step.apply_frame_warp_transform(init, img), + lambda img: step.do_hybrid_compositing_before_motion(init, img), + lambda img: Step.apply_hybrid_motion_ransac_transform(init, img), + lambda img: Step.apply_hybrid_motion_optical_flow(init, img), + lambda img: step.do_normal_hybrid_compositing_after_motion(init, img), + lambda img: Step.apply_color_matching(init, img), + lambda img: Step.transform_to_grayscale_if_active(init, img)) -def contrast_transformation_tube(init, step, mask) -> ImageTube: +def contrast_transformation_tube(init, step) -> ImageTube: return tube(lambda img: step.apply_scaling(img), - lambda img: step.apply_anti_blur(init, mask, img)) + lambda img: step.apply_anti_blur(init, img)) def noise_transformation_tube(init, step) -> ImageTube: return tube(lambda img: step.apply_frame_noising(init, step, img)) -def optical_flow_redo_tube(init, optical_flow, images) -> ImageTube: +def optical_flow_redo_tube(init, optical_flow) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( # TODO create img.get_flow - img, call_get_flow_from_images(init, images.previous, img, optical_flow), + img, call_get_flow_from_images(init, init.images.previous, img, optical_flow), step.init.redo_flow_factor)) # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). -def conditional_hybrid_video_after_generation_tube(init, indexes, step) -> ImageTube: +def conditional_hybrid_video_after_generation_tube(init, step) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: call_hybrid_composite(init, indexes.frame.i, img, step.init.hybrid_comp_schedules), + lambda img: call_hybrid_composite(init, init.indexes.frame.i, img, step.init.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= - lambda: indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) + lambda: init.indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) -def conditional_extra_color_match_tube(init, indexes, images) -> ImageTube: +def conditional_extra_color_match_tube(init) -> ImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - return tube(lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), + return tube(lambda img: maintain_colors(img, init.images.color_match, init.args.anim_args.color_coherence), lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: maintain_colors(img, images.color_match, init.args.anim_args.color_coherence), + lambda img: maintain_colors(img, init.images.color_match, init.args.anim_args.color_coherence), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= - lambda: indexes.is_first_frame() and init.is_color_match_to_be_initialized(images.color_match)) + lambda: init.indexes.is_first_frame() and init.is_initialize_color_match(init.images.color_match)) def conditional_color_match_tube(init, step) -> ImageTube: @@ -86,10 +86,10 @@ def conditional_force_to_grayscale_tube(init) -> ImageTube: is_do_process=lambda: init.args.anim_args.color_force_grayscale) -def conditional_add_overlay_mask_tube(init, indexes, is_tween) -> ImageTube: +def conditional_add_overlay_mask_tube(init, is_tween) -> ImageTube: is_use_overlay = init.args.args.overlay_mask is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask - index = indexes.tween.i if is_tween else indexes.frame.i + index = init.indexes.tween.i if is_tween else init.indexes.frame.i is_bgr_array = True return tube(lambda img: ImageOps.grayscale(img), lambda img: do_overlay_mask(init.args.args, init.args.anim_args, img, index, is_bgr_array), @@ -103,16 +103,16 @@ def conditional_force_tween_to_grayscale_tube(init) -> ImageTube: # Composite Tubes, made from other Tubes. -def contrasted_noise_transformation_tube(init, step, mask) -> ImageTube: +def contrasted_noise_transformation_tube(init, step) -> ImageTube: """Combines contrast and noise transformation tubes.""" - contrast_tube: Tube = contrast_transformation_tube(init, step, mask) + contrast_tube: Tube = contrast_transformation_tube(init, step) noise_tube: Tube = noise_transformation_tube(init, step) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(init, indexes, step, images, is_tween: bool = False) -> ImageTube: - hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(init, indexes, step) - extra_tube: Tube = conditional_extra_color_match_tube(init, indexes, images) +def conditional_frame_transformation_tube(init, step, is_tween: bool = False) -> ImageTube: + hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(init, step) + extra_tube: Tube = conditional_extra_color_match_tube(init) gray_tube: Tube = conditional_force_to_grayscale_tube(init) - mask_tube: Tube = conditional_add_overlay_mask_tube(init, indexes, is_tween) + mask_tube: Tube = conditional_add_overlay_mask_tube(init, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) diff --git a/scripts/deforum_helpers/rendering/util/call/gen.py b/scripts/deforum_helpers/rendering/util/call/gen.py index 1a63ff5c8..54f6a0c91 100644 --- a/scripts/deforum_helpers/rendering/util/call/gen.py +++ b/scripts/deforum_helpers/rendering/util/call/gen.py @@ -1,7 +1,7 @@ from ....generate import generate -def call_generate(init, i, schedule): +def call_generate(init, step): ia = init.args return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, - ia.root, init.parseq_adapter, i, sampler_name=schedule.sampler_name) + ia.root, init.parseq_adapter, init.indexes.frame.i, sampler_name=step.schedule.sampler_name) diff --git a/scripts/deforum_helpers/rendering/util/call/video_and_audio.py b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py index 1cdb73f22..6a8a6812f 100644 --- a/scripts/deforum_helpers/rendering/util/call/video_and_audio.py +++ b/scripts/deforum_helpers/rendering/util/call/video_and_audio.py @@ -1,8 +1,9 @@ from ....video_audio_utilities import get_next_frame, render_preview -def call_render_preview(init, i, last_preview_frame): +def call_render_preview(init, last_preview_frame): ia = init.args + i = init.indexes.frame.i return render_preview(ia.args, ia.anim_args, ia.video_args, ia.root, i, last_preview_frame) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index a98463da6..3b4ff9fc8 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -3,8 +3,8 @@ RESET = "\033[0m" -def print_animation_frame_info(init, indexes): - print(f"{CYAN}Animation frame: {RESET}{indexes.frame.i}/{init.args.anim_args.max_frames}") +def print_animation_frame_info(init): + print(f"{CYAN}Animation frame: {RESET}{init.indexes.frame.i}/{init.args.anim_args.max_frames}") def print_tween_frame_info(init, indexes, cadence_flow, tween): diff --git a/scripts/deforum_helpers/rendering/util/web_ui_utils.py b/scripts/deforum_helpers/rendering/util/web_ui_utils.py index 953b1c2d1..d75a5612d 100644 --- a/scripts/deforum_helpers/rendering/util/web_ui_utils.py +++ b/scripts/deforum_helpers/rendering/util/web_ui_utils.py @@ -11,8 +11,8 @@ def init_job(init): state.job_count = init.args.anim_args.max_frames -def update_job(init, indexes): - frame = indexes.frame.i + 1 +def update_job(init): + frame = init.indexes.frame.i + 1 max_frames = init.args.anim_args.max_frames state.job = f"frame {frame}/{max_frames}" state.job_no = frame + 1 @@ -24,8 +24,8 @@ def update_job(init, indexes): print("** RESUMING **") -def update_status_tracker(init, indexes): - progress = indexes.frame.i / init.args.anim_args.max_frames +def update_status_tracker(init): + progress = init.indexes.frame.i / init.args.anim_args.max_frames JobStatusTracker().update_phase(init.args.root.job_id, phase="GENERATING", progress=progress) From 235b61f1c14cb5bbccc3def2564f786ead4a7f12 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 18:00:59 +0200 Subject: [PATCH 072/132] RenderInit renamed to RenderData and StepInit renamed to StepData. --- scripts/deforum_helpers/render.py | 48 ++--- .../rendering/data/anim/animation_mode.py | 4 +- .../{initialization.py => render_data.py} | 36 ++-- .../rendering/data/step/__init__.py | 2 +- .../rendering/data/step/step.py | 170 +++++++++--------- .../rendering/data/step/tween_step.py | 1 - .../deforum_helpers/rendering/data/turbo.py | 10 +- .../rendering/img_2_img_tubes.py | 4 +- .../rendering/util/call/images.py | 2 +- .../rendering/util/call/mask.py | 4 +- .../rendering/util/call/subtitle.py | 11 +- .../rendering/util/filename_utils.py | 28 +-- .../rendering/util/opt_utils.py | 9 +- 13 files changed, 163 insertions(+), 166 deletions(-) rename scripts/deforum_helpers/rendering/data/{initialization.py => render_data.py} (93%) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 6e0ca232b..01d57fe3c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -24,7 +24,7 @@ from .colors import maintain_colors from .rendering.data import Indexes -from .rendering.data.initialization import RenderInit +from .rendering.data.render_data import RenderData from .rendering.data.step import Step, TweenStep from .rendering.img_2_img_tubes import (conditional_color_match_tube, conditional_frame_transformation_tube, contrasted_noise_transformation_tube, optical_flow_redo_tube, @@ -40,40 +40,40 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - init = RenderInit.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) - run_render_animation(init) + render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + run_render_animation(render_data) -def run_render_animation(init): - web_ui_utils.init_job(init) +def run_render_animation(data: RenderData): + web_ui_utils.init_job(data) last_preview_frame = 0 - while init.indexes.frame.i < init.args.anim_args.max_frames: - step = Step.do_start_and_create(init) # TODO step should have an immutable init? - step.write_frame_subtitle(init) # TODO move step concerns from init to step.. + while data.indexes.frame.i < data.args.anim_args.max_frames: + step = Step.do_start_and_create(data) # TODO step should have an immutable init? + step.write_frame_subtitle(data) # TODO move step concerns from init to step.. - maybe_emit_in_between_frames(init, step) + maybe_emit_in_between_frames(data, step) - init.images.color_match = Step.create_color_match_for_video(init) # TODO move to step? - init.images.previous = transform_and_update_noised_sample(init, step) # TODO move to step? - init.prepare_generation(init, step) # TODO move to step? - maybe_redo_optical_flow(init, step) # TODO move to step? - maybe_redo_diffusion(init, step) # TODO move to step? + data.images.color_match = Step.create_color_match_for_video(data) # TODO move to step? + data.images.previous = transform_and_update_noised_sample(data, step) # TODO move to step? + data.prepare_generation(data, step) # TODO move to step? + maybe_redo_optical_flow(data, step) # TODO move to step? + maybe_redo_diffusion(data, step) # TODO move to step? - image = call_generate(init, step) # TODO move to step? + image = call_generate(data, step) # TODO move to step? if image is None: print_warning_generate_returned_no_image() break - image = conditional_frame_transformation_tube(init, step)(image) # TODO move to step? - init.images.color_match = conditional_color_match_tube(init, step)(image) # TODO move to step? - next_frame, step.depth = progress_step(init, image, step.depth) # TODO move to step? - init.indexes.update_frame(next_frame) + image = conditional_frame_transformation_tube(data, step)(image) # TODO move to step? + data.images.color_match = conditional_color_match_tube(data, step)(image) # TODO move to step? + next_frame, step.depth = progress_step(data, image, step.depth) # TODO move to step? + data.indexes.update_frame(next_frame) state.assign_current_image(image) # may reassign init.args.args and/or root.seed_internal - init.args.args.seed = next_seed(init.args.args, init.args.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(init, last_preview_frame) - web_ui_utils.update_status_tracker(init) - init.animation_mode.unload_raft_and_depth_model() + data.args.args.seed = next_seed(data.args.args, data.args.root) # TODO group all seeds and sub-seeds + last_preview_frame = call_render_preview(data, last_preview_frame) + web_ui_utils.update_status_tracker(data) + data.animation_mode.unload_raft_and_depth_model() def maybe_emit_in_between_frames(init, step): @@ -148,7 +148,7 @@ def maybe_redo_optical_flow(init, step): def maybe_redo_diffusion(init, step): - is_diffusion_redo = init.has_positive_diffusion_redo and init.images.has_previous() and step.init.has_strength() + is_diffusion_redo = init.has_positive_diffusion_redo and init.images.has_previous() and step.step_data.has_strength() is_not_preview = init.is_not_in_motion_preview_mode() if is_diffusion_redo and is_not_preview: do_diffusion_redo(init, step) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index bc11bc17a..c7a0c1dcc 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -73,9 +73,7 @@ def load_depth_model_if_active(args, anim_args, opts): @staticmethod def initial_hybrid_files(sa) -> list[Path]: - """ - Returns a list of initial hybrid input files if required, otherwise an empty list. - """ + """Returns a list of initial hybrid input files if required, otherwise an empty list.""" if AnimationMode._is_requiring_hybrid_frames(sa.anim_args): # may cause side effects on args and anim_args. _, __, init_hybrid_input_files = hybrid_generation(sa.args, sa.anim_args, sa.root) diff --git a/scripts/deforum_helpers/rendering/data/initialization.py b/scripts/deforum_helpers/rendering/data/render_data.py similarity index 93% rename from scripts/deforum_helpers/rendering/data/initialization.py rename to scripts/deforum_helpers/rendering/data/render_data.py index 66747a21c..104f676f0 100644 --- a/scripts/deforum_helpers/rendering/data/initialization.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -39,7 +39,7 @@ class RenderInitArgs: @dataclass(init=True, frozen=True, repr=False, eq=False) -class RenderInit: +class RenderData: """The purpose of this class is to group and control all data used in render_animation""" images: Images | None turbo: Turbo | None @@ -57,35 +57,35 @@ class RenderInit: is_use_mask: bool @staticmethod - def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderInit': + def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderData': ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) output_directory = args.outdir is_use_mask = args.use_mask - parseq_adapter = RenderInit.create_parseq_adapter(ri_args) + parseq_adapter = RenderData.create_parseq_adapter(ri_args) srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) animation_mode = AnimationMode.from_args(ri_args) - prompt_series = RenderInit.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RenderInit.create_depth_model_and_enable_depth_map_saving_if_active( + prompt_series = RenderData.select_prompts(parseq_adapter, anim_args, animation_keys, root) + depth_model = RenderData.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args) # Temporary instance only exists for using it to easily create other objects required by the actual instance. # Feels slightly awkward, but it's probably not worth optimizing since only 1st and gc can take care of it fine. - incomplete_init = RenderInit(None, None, None, None, args.seed, ri_args, parseq_adapter, srt, animation_keys, + incomplete_init = RenderData(None, None, None, None, args.seed, ri_args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) images = Images.create(incomplete_init) turbo = Turbo.create(incomplete_init) indexes = Indexes.create(incomplete_init, turbo) mask = Mask.create(incomplete_init, indexes.frame.i) - instance = RenderInit(images, turbo, indexes, mask, args.seed, ri_args, parseq_adapter, srt, animation_keys, + instance = RenderData(images, turbo, indexes, mask, args.seed, ri_args, parseq_adapter, srt, animation_keys, animation_mode, prompt_series, depth_model, output_directory, is_use_mask) - RenderInit.init_looper_if_active(args, loop_args) - RenderInit.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) - RenderInit.create_output_directory_for_the_batch(args.outdir) - RenderInit.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) - RenderInit.maybe_resume_from_timestring(anim_args, root) + RenderData.init_looper_if_active(args, loop_args) + RenderData.handle_controlnet_video_input_frames_generation(controlnet_args, args, anim_args) + RenderData.create_output_directory_for_the_batch(args.outdir) + RenderData.save_settings_txt(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + RenderData.maybe_resume_from_timestring(anim_args, root) return instance def is_3d(self): @@ -194,7 +194,7 @@ def is_do_color_match_conversion(self, step): is_legacy_cm = self.args.anim_args.legacy_colormatch is_use_init = self.args.args.use_init is_not_legacy_with_use_init = not is_legacy_cm and not is_use_init - is_legacy_cm_without_strength = is_legacy_cm and step.init.strength == 0 + is_legacy_cm_without_strength = is_legacy_cm and step.step_data.strength == 0 is_maybe_special_legacy = is_not_legacy_with_use_init or is_legacy_cm_without_strength return is_maybe_special_legacy and self.has_non_video_or_image_color_coherence() @@ -202,7 +202,7 @@ def update_sample_and_args_for_current_progression_step(self, step, noised_image # use transformed previous frame as init for current self.args.args.use_init = True self.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) - self.args.args.strength = max(0.0, min(1.0, step.init.strength)) + self.args.args.strength = max(0.0, min(1.0, step.step_data.strength)) def update_some_args_for_current_step(self, indexes, step): i = indexes.frame.i @@ -210,7 +210,7 @@ def update_some_args_for_current_step(self, indexes, step): # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints self.args.args.pix2pix_img_cfg_scale = float(keys.pix2pix_img_cfg_scale_series[i]) self.args.args.prompt = self.prompt_series[i] # grab prompt for current frame - self.args.args.scale = step.init.scale + self.args.args.scale = step.step_data.scale def update_seed_and_checkpoint_for_current_step(self, indexes): i = indexes.frame.i @@ -250,7 +250,7 @@ def _update_video_input_for_current_frame(self, i, step): print_init_frame_info(init_frame) self.args.args.init_image = init_frame self.args.args.init_image_box = None # init_image_box not used in this case - self.args.args.strength = max(0.0, min(1.0, step.init.strength)) + self.args.args.strength = max(0.0, min(1.0, step.step_data.strength)) def _update_video_mask_for_current_frame(self, i): video_mask_path = self.args.anim_args.video_mask_path @@ -319,7 +319,7 @@ def init_looper_if_active(args, loop_args): @staticmethod def select_prompts(parseq_adapter, anim_args, animation_keys, root): return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ - else RenderInit.expand_prompts_out_to_per_frame(anim_args, root) + else RenderData.expand_prompts_out_to_per_frame(anim_args, root) @staticmethod def is_composite_with_depth_mask(anim_args): @@ -330,7 +330,7 @@ def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, an # depth-based hybrid composite mask requires saved depth maps # TODO avoid or isolate side effect: anim_args.save_depth_maps = (anim_mode.is_predicting_depths - and RenderInit.is_composite_with_depth_mask(anim_args)) + and RenderData.is_composite_with_depth_mask(anim_args)) return DepthModel(root.models_path, memory_utils.select_depth_device(root), root.half_precision, diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py index 47868a343..10dfb1e84 100644 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -1,2 +1,2 @@ -from .step import StepInit, Step +from .step import StepData, Step from .tween_step import TweenStep diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 4c943ce51..89ad77360 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -4,12 +4,9 @@ import cv2 import numpy as np -from ..initialization import RenderInit +from ..render_data import RenderData from ..schedule import Schedule from ..turbo import Turbo -from ..images import Images -from ..indexes import Indexes -from ..mask import Mask from ...util import memory_utils, opt_utils, web_ui_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.hybrid import ( @@ -19,12 +16,12 @@ from ...util.call.images import call_add_noise from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle -from ....hybrid_video import image_transform_ransac, image_transform_optical_flow from ...util.log_utils import print_animation_frame_info +from ....hybrid_video import image_transform_ransac, image_transform_optical_flow @dataclass(init=True, frozen=True, repr=False, eq=False) -class StepInit: +class StepData: noise: Any = None strength: Any = None scale: Any = None @@ -49,7 +46,7 @@ def has_strength(self): @staticmethod def create(deform_keys, i): keys = deform_keys - return StepInit(keys.noise_schedule_series[i], + return StepData(keys.noise_schedule_series[i], keys.strength_schedule_series[i], keys.cfg_scale_schedule_series[i], keys.contrast_schedule_series[i], @@ -59,7 +56,7 @@ def create(deform_keys, i): keys.threshold_schedule_series[i], keys.cadence_flow_factor_schedule_series[i], keys.redo_flow_factor_schedule_series[i], - StepInit._hybrid_comp_args(keys, i)) + StepData._hybrid_comp_args(keys, i)) @staticmethod def _hybrid_comp_args(keys, i): @@ -74,139 +71,140 @@ def _hybrid_comp_args(keys, i): @dataclass(init=True, frozen=False, repr=False, eq=False) class Step: - init: StepInit + step_data: StepData schedule: Schedule depth: Any # TODO try to init early, then freeze class subtitle_params_to_print: Any subtitle_params_string: str @staticmethod - def do_start_and_create(init): + def do_start_and_create(data: RenderData): # Perform the necessary side effects - memory_utils.handle_med_or_low_vram_before_step(init) - print_animation_frame_info(init) - web_ui_utils.update_job(init) + memory_utils.handle_med_or_low_vram_before_step(data) + print_animation_frame_info(data) + web_ui_utils.update_job(data) # Actual create - step_init = StepInit.create(init.animation_keys.deform_keys, init.indexes.frame.i) - schedule = Schedule.create(init, init.indexes.frame.i, init.args.anim_args, init.args.args) + step_init = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) + schedule = Schedule.create(data, data.indexes.frame.i, + data.args.anim_args, data.args.args) return Step(step_init, schedule, None, None, "") def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' - return has_flow_redo and images.has_previous() and self.init.has_strength() + return has_flow_redo and images.has_previous() and self.step_data.has_strength() - def update_depth_prediction(self, init: RenderInit, turbo: Turbo): - has_depth = init.depth_model is not None + def update_depth_prediction(self, data: RenderData, turbo: Turbo): + has_depth = data.depth_model is not None has_next = turbo.next.image is not None if has_depth and has_next: image = turbo.next.image - weight = init.args.anim_args.midas_weight - precision = init.args.root.half_precision - self.depth = init.depth_model.predict(image, weight, precision) - - def write_frame_subtitle(self, init): - if init.turbo.is_first_step_with_subtitles(init): - self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) - self.subtitle_params_string = call_format_animation_params(init, init.indexes.frame.i, params_to_print) - call_write_frame_subtitle(init, init.indexes.frame.i, params_string) - - def write_frame_subtitle_if_active(self, init): - if opt_utils.is_generate_subtitles(init): - self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(init) - self.subtitle_params_string = call_format_animation_params(init, self.indexes.tween.i, params_to_print) - call_write_frame_subtitle(init, self.indexes.tween.i, params_string, sub_step.tween < 1.0) - - def apply_frame_warp_transform(self, init, image): - previous, self.depth = call_anim_frame_warp(init, init.indexes.frame.i, image, None) + weight = data.args.anim_args.midas_weight + precision = data.args.root.half_precision + self.depth = data.depth_model.predict(image, weight, precision) + + def write_frame_subtitle(self, data: RenderData): + if data.turbo.is_first_step_with_subtitles(data): + self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(data) + self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_to_print) + call_write_frame_subtitle(data, data.indexes.frame.i, params_string) + + def write_frame_subtitle_if_active(self, data: RenderData): + if opt_utils.is_generate_subtitles(data): + self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(data) + self.subtitle_params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) + call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) + + def apply_frame_warp_transform(self, data: RenderData, image): + previous, self.depth = call_anim_frame_warp(data, data.indexes.frame.i, image, None) return previous - def _do_hybrid_compositing_on_cond(self, init, image, condition): - i = init.indexes.frame.i - schedules = self.init.hybrid_comp_schedules + def _do_hybrid_compositing_on_cond(self, data: RenderData, image, condition): + i = data.indexes.frame.i + schedules = self.step_data.hybrid_comp_schedules if condition: - _, composed = call_hybrid_composite(init, i, image, schedules) + _, composed = call_hybrid_composite(data, i, image, schedules) return composed else: return image - def do_hybrid_compositing_before_motion(self, init, image): - condition = init.is_hybrid_composite_before_motion() - return self._do_hybrid_compositing_on_cond(init, image, condition) + def do_hybrid_compositing_before_motion(self, data: RenderData, image): + condition = data.is_hybrid_composite_before_motion() + return self._do_hybrid_compositing_on_cond(data, image, condition) - def do_normal_hybrid_compositing_after_motion(self, init, image): - condition = init.is_normal_hybrid_composite() - return self._do_hybrid_compositing_on_cond(init, image, condition) + def do_normal_hybrid_compositing_after_motion(self, data: RenderData, image): + condition = data.is_normal_hybrid_composite() + return self._do_hybrid_compositing_on_cond(data, image, condition) def apply_scaling(self, image): - return (image * self.init.contrast).round().astype(np.uint8) + return (image * self.step_data.contrast).round().astype(np.uint8) - def apply_anti_blur(self, init, image): - if self.init.amount > 0: - return call_unsharp_mask(init, self, image, init.mask) + def apply_anti_blur(self, data: RenderData, image): + if self.step_data.amount > 0: + return call_unsharp_mask(data, self, image, data.mask) else: return image - def apply_frame_noising(self, init, mask, image): - is_use_any_mask = init.args.args.use_mask or init.args.anim_args.use_noise_mask + def apply_frame_noising(self, data: RenderData, mask, image): + is_use_any_mask = data.args.args.use_mask or data.args.anim_args.use_noise_mask if is_use_any_mask: seq = self.schedule.noise_mask_seq vals = mask.noise_vals - init.args.root.noise_mask = call_compose_mask_with_check(init, seq, vals, contrast_image) - return call_add_noise(init, self, image) + data.args.root.noise_mask = call_compose_mask_with_check(data, seq, vals, contrast_image) + return call_add_noise(data, self, image) @staticmethod - def apply_color_matching(init, image): - if init.has_color_coherence(): - if init.images.color_match is None: + def apply_color_matching(data: RenderData, image): + if data.has_color_coherence(): + if data.images.color_match is None: # TODO questionable # initialize color_match for next iteration with current image, but don't do anything yet. - init.images.color_match = image.copy() + data.images.color_match = image.copy() else: - return maintain_colors(image, init.images.color_match, init.args.anim_args.color_coherence) + return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) return image @staticmethod - def transform_to_grayscale_if_active(init, image): - if init.args.anim_args.color_force_grayscale: - grayscale = cv2.cvtColor(init.images.previous, cv2.COLOR_BGR2GRAY) + def transform_to_grayscale_if_active(data: RenderData, image): + if data.args.anim_args.color_force_grayscale: + grayscale = cv2.cvtColor(data.images.previous, cv2.COLOR_BGR2GRAY) return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) else: return image @staticmethod - def apply_hybrid_motion_ransac_transform(init, image): + def apply_hybrid_motion_ransac_transform(data: RenderData, image): """hybrid video motion - warps images.previous to match motion, usually to prepare for compositing""" - motion = init.args.anim_args.hybrid_motion + motion = data.args.anim_args.hybrid_motion if motion in ['Affine', 'Perspective']: - last_i = init.indexes.frame.i - 1 - reference_images = init.images - matrix = call_get_matrix_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ - if init.args.anim_args.hybrid_motion_use_prev_img \ - else call_get_matrix_for_hybrid_motion(init, last_i) - return image_transform_ransac(image, matrix, init.args.anim_args.hybrid_motion) + last_i = data.indexes.frame.i - 1 + reference_images = data.images + matrix = call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ + if data.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_matrix_for_hybrid_motion(data, last_i) + return image_transform_ransac(image, matrix, data.args.anim_args.hybrid_motion) return image @staticmethod - def apply_hybrid_motion_optical_flow(init, image): - motion = init.args.anim_args.hybrid_motion + def apply_hybrid_motion_optical_flow(data: RenderData, image): + motion = data.args.anim_args.hybrid_motion if motion in ['Optical Flow']: - last_i = init.indexes.frame.i - 1 - reference_images = init.images - flow = call_get_flow_for_hybrid_motion_prev(init, last_i, reference_images.previous) \ - if init.args.anim_args.hybrid_motion_use_prev_img \ - else call_get_flow_for_hybrid_motion(init, last_i) - transformed = image_transform_optical_flow(images.previous, flow, step.init.flow_factor()) - init.animation_mode.prev_flow = flow # side effect + last_i = data.indexes.frame.i - 1 + reference_images = data.images + flow = call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ + if data.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_flow_for_hybrid_motion(data, last_i) + transformed = image_transform_optical_flow(images.previous, flow, step.step_data.flow_factor()) + data.animation_mode.prev_flow = flow # side effect return transformed else: return image @staticmethod - def create_color_match_for_video(init): - if init.args.anim_args.color_coherence == 'Video Input' and init.is_hybrid_available(): - if int(init.indexes.frame.i) % int(init.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(preview_video_image_path(init, init.indexes)) - prev_vid_img = prev_vid_img.resize(init.dimensions(), PIL.Image.LANCZOS) - init.images.color_match = np.asarray(prev_vid_img) - return cv2.cvtColor(init.images.color_match, cv2.COLOR_RGB2BGR) + def create_color_match_for_video(data: RenderData): + if data.args.anim_args.color_coherence == 'Video Input' and data.is_hybrid_available(): + if int(data.indexes.frame.i) % int(data.args.anim_args.color_coherence_video_every_N_frames) == 0: + prev_vid_img = Image.open(preview_video_image_path(data, data.indexes)) + prev_vid_img = prev_vid_img.resize(data.dimensions(), PIL.Image.LANCZOS) + data.images.color_match = np.asarray(prev_vid_img) + return cv2.cvtColor(data.images.color_match, cv2.COLOR_RGB2BGR) return None diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 72f301d23..71a239389 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -4,7 +4,6 @@ from ...data.indexes import Indexes from ...data.step import Step -from ...data.turbo import Turbo from ...img_2_img_tubes import conditional_force_tween_to_grayscale_tube, conditional_add_overlay_mask_tube from ...util import log_utils, web_ui_utils diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index d01a1828b..52c166b15 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -1,11 +1,11 @@ from dataclasses import dataclass from cv2.typing import MatLike -from ...hybrid_video import image_transform_ransac, image_transform_optical_flow from .subtitle import Srt from ..util.call.anim import call_anim_frame_warp from ..util.call.resume import call_get_resume_vars +from ...hybrid_video import image_transform_ransac, image_transform_optical_flow @dataclass(init=True, frozen=False, repr=False, eq=True) @@ -49,7 +49,7 @@ def advance_optical_flow(self, tween_step, flow_factor: int = 1): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) def advance_optical_tween_flow(self, step, flow): - ff = step.init.flow_factor() + ff = step.step_data.flow_factor() i = indexes.tween.i if self.is_advance_prev(i): self.prev.image = image_transform_optical_flow(self.prev.image, flow, ff) @@ -67,7 +67,7 @@ def advance_hybrid_motion_optical_tween_flow(self, init, indexes, reference_imag init.animation_mode.prev_flow = flow def advance_cadence_flow(self, tween_step): - ff = step.init.sub_step.cadence_flow_factor + ff = step.step_data.sub_step.cadence_flow_factor i = indexes.tween.i inc = tween_step.cadence_flow_inc if self.is_advance_prev(i): @@ -152,8 +152,8 @@ def is_advance_next(self, i: int) -> bool: def is_first_step(self) -> bool: return self.steps == 1 - def is_first_step_with_subtitles(self, init) -> bool: - return self.is_first_step() and Srt.is_subtitle_generation_active(init.args.opts.data) + def is_first_step_with_subtitles(self, render_data) -> bool: + return self.is_first_step() and Srt.is_subtitle_generation_active(render_data.args.opts.data) def is_emit_in_between_frames(self) -> bool: return self.steps > 1 diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 61b370268..b58ebebf2 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -51,13 +51,13 @@ def optical_flow_redo_tube(init, optical_flow) -> ImageTube: lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( # TODO create img.get_flow img, call_get_flow_from_images(init, init.images.previous, img, optical_flow), - step.init.redo_flow_factor)) + step.step_data.redo_flow_factor)) # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). def conditional_hybrid_video_after_generation_tube(init, step) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: call_hybrid_composite(init, init.indexes.frame.i, img, step.init.hybrid_comp_schedules), + lambda img: call_hybrid_composite(init, init.indexes.frame.i, img, step.step_data.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= lambda: init.indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) diff --git a/scripts/deforum_helpers/rendering/util/call/images.py b/scripts/deforum_helpers/rendering/util/call/images.py index 91ad86ae5..4b3f225c1 100644 --- a/scripts/deforum_helpers/rendering/util/call/images.py +++ b/scripts/deforum_helpers/rendering/util/call/images.py @@ -4,7 +4,7 @@ def call_add_noise(init, step, image): aa = init.args.anim_args - amount: float = step.init.noise + amount: float = step.step_data.noise seed: int = init.args.args.seed n_type: str = aa.noise_type perlin_arguments = (aa.perlin_w, aa.perlin_h, aa.perlin_octaves, aa.perlin_persistence) diff --git a/scripts/deforum_helpers/rendering/util/call/mask.py b/scripts/deforum_helpers/rendering/util/call/mask.py index 1b0d0de00..b31e51223 100644 --- a/scripts/deforum_helpers/rendering/util/call/mask.py +++ b/scripts/deforum_helpers/rendering/util/call/mask.py @@ -8,6 +8,6 @@ def call_compose_mask_with_check(init, mask_seq, val_masks, image): def call_unsharp_mask(init, step, image, mask): - kernel_size = (step.init.kernel, step.init.kernel) + kernel_size = (step.step_data.kernel, step.step_data.kernel) mask_image = mask.image if init.args.args.use_mask else None - return unsharp_mask(image, kernel_size, step.init.sigma, step.init.amount, step.init.threshold, mask_image) + return unsharp_mask(image, kernel_size, step.step_data.sigma, step.step_data.amount, step.step_data.threshold, mask_image) diff --git a/scripts/deforum_helpers/rendering/util/call/subtitle.py b/scripts/deforum_helpers/rendering/util/call/subtitle.py index 876fabfa4..a694bbdfc 100644 --- a/scripts/deforum_helpers/rendering/util/call/subtitle.py +++ b/scripts/deforum_helpers/rendering/util/call/subtitle.py @@ -1,10 +1,11 @@ from ....subtitle_handler import format_animation_params, write_frame_subtitle -def call_format_animation_params(init, i, params_to_print): - return format_animation_params(init.animation_keys.deform_keys, init.prompt_series, i, params_to_print) +def call_format_animation_params(render_data, i, params_to_print): + return format_animation_params(render_data.animation_keys.deform_keys, + render_data.prompt_series, i, params_to_print) -def call_write_frame_subtitle(init, i, params_string, is_cadence: bool = False) -> None: - text = f"F#: {i}; Cadence: {is_cadence}; Seed: {init.args.args.seed}; {params_string}" - write_frame_subtitle(init.srt.filename, i, init.srt.frame_duration, text) +def call_write_frame_subtitle(render_data, i, params_string, is_cadence: bool = False) -> None: + text = f"F#: {i}; Cadence: {is_cadence}; Seed: {render_data.args.args.seed}; {params_string}" + write_frame_subtitle(render_data.srt.filename, i, render_data.srt.frame_duration, text) diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 67cbc2f87..1ece174d6 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -2,7 +2,7 @@ from pathlib import Path from ..data import Indexes -from ..data.step import StepInit +from ..data.step import StepData from ...video_audio_utilities import get_frame_name @@ -23,28 +23,28 @@ def _frame_filename_index(i: int, file_format: FileFormat) -> str: return f"{i:09}.{file_format.value}" -def _frame_filename(init: StepInit, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: +def _frame_filename(data: StepData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" - return f"{init.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" + return f"{data.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" -def frame(init: StepInit, indexes: Indexes) -> str: - return _frame_filename(init, indexes.frame.i) +def frame(data: StepData, indexes: Indexes) -> str: + return _frame_filename(data, indexes.frame.i) -def depth_frame(init: StepInit, indexes: Indexes) -> str: - return _frame_filename(init, indexes.frame.i, True) +def depth_frame(data: StepData, indexes: Indexes) -> str: + return _frame_filename(data, indexes.frame.i, True) -def tween_frame_name(init: StepInit, indexes: Indexes) -> str: - return _frame_filename(init, indexes.tween.i) +def tween_frame_name(data: StepData, indexes: Indexes) -> str: + return _frame_filename(data, indexes.tween.i) -def tween_depth_frame(init: StepInit, indexes: Indexes) -> str: - return _frame_filename(init, indexes.tween.i, True) +def tween_depth_frame(data: StepData, indexes: Indexes) -> str: + return _frame_filename(data, indexes.tween.i, True) -def preview_video_image_path(init: StepInit, indexes: Indexes) -> Path: - frame_name = get_frame_name(init.args.anim_args.video_init_path) +def preview_video_image_path(data: StepData, indexes: Indexes) -> Path: + frame_name = get_frame_name(data.args.anim_args.video_init_path) index = _frame_filename_index(indexes.frame.i, FileFormat.video_frame_format()) - return Path(init.output_directory) / "inputframes" / (frame_name + index) + return Path(data.output_directory) / "inputframes" / (frame_name + index) diff --git a/scripts/deforum_helpers/rendering/util/opt_utils.py b/scripts/deforum_helpers/rendering/util/opt_utils.py index 54403b2d6..30732fb76 100644 --- a/scripts/deforum_helpers/rendering/util/opt_utils.py +++ b/scripts/deforum_helpers/rendering/util/opt_utils.py @@ -12,9 +12,10 @@ def setup(init, schedule): put_if_present(data, "eta_ancestral", schedule.eta_ancestral) -def generation_info_for_subtitles(init): - return init.args.opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) +def generation_info_for_subtitles(render_data): + return render_data.args.opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) -def is_generate_subtitles(init): - return init.args.opts.data.get("deforum_save_gen_info_as_srt") +def is_generate_subtitles(render_data): + return render_data.args.opts.data.get("deforum_save_gen_info_as_srt") + From e288f25d41fe092e226ce7a44dfb1726b7df9614 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 18:48:29 +0200 Subject: [PATCH 073/132] Moved tween frame generation to TweenFrame class. --- scripts/deforum_helpers/render.py | 35 ++----------------- .../rendering/data/render_data.py | 2 +- .../rendering/data/step/step.py | 10 +++--- .../rendering/data/step/tween_step.py | 32 ++++++++++++++++- .../rendering/util/filename_utils.py | 14 ++++---- .../rendering/util/image_utils.py | 21 +++++------ 6 files changed, 59 insertions(+), 55 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 01d57fe3c..960c8779e 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -23,7 +23,6 @@ from modules.shared import opts, state from .colors import maintain_colors -from .rendering.data import Indexes from .rendering.data.render_data import RenderData from .rendering.data.step import Step, TweenStep from .rendering.img_2_img_tubes import (conditional_color_match_tube, conditional_frame_transformation_tube, @@ -32,7 +31,6 @@ from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils from .rendering.util.call.gen import call_generate from .rendering.util.call.video_and_audio import call_render_preview -from .rendering.util.image_utils import save_and_return_frame from .rendering.util.log_utils import (print_optical_flow_info, print_redo_generation_info, print_warning_generate_returned_no_image) from .save_images import save_image @@ -48,10 +46,10 @@ def run_render_animation(data: RenderData): web_ui_utils.init_job(data) last_preview_frame = 0 while data.indexes.frame.i < data.args.anim_args.max_frames: - step = Step.do_start_and_create(data) # TODO step should have an immutable init? - step.write_frame_subtitle(data) # TODO move step concerns from init to step.. + step = Step.do_start_and_create(data) + step.maybe_write_frame_subtitle() - maybe_emit_in_between_frames(data, step) + TweenStep.maybe_emit_in_between_frames(step) # TODO? make Step/TweenStep union and check if step is TweenStep data.images.color_match = Step.create_color_match_for_video(data) # TODO move to step? data.images.previous = transform_and_update_noised_sample(data, step) # TODO move to step? @@ -76,33 +74,6 @@ def run_render_animation(data: RenderData): data.animation_mode.unload_raft_and_depth_model() -def maybe_emit_in_between_frames(init, step): - if init.turbo.is_emit_in_between_frames(): - tween_frame_start_i = max(init.indexes.frame.start, init.indexes.frame.i - init.turbo.steps) - emit_frames_between_index_pair(init, step, tween_frame_start_i, init.indexes.frame.i) - - -def emit_frames_between_index_pair(init, last_step, tween_frame_start_i, frame_i): - """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" - tween_range = range(tween_frame_start_i, frame_i) - tween_indexes_list: List[Indexes] = TweenStep.create_indexes(init.indexes, tween_range) - tween_steps: List[TweenStep] = TweenStep.create_steps(last_step, tween_indexes_list) - init.indexes.update_tween_start(init.turbo) # TODO... - emit_tween_frames(init, tween_steps) - - -def emit_tween_frames(init, tween_steps): - """Emits a tween frame for each provided tween_step.""" - for tween_step in tween_steps: - tween_step.handle_synchronous_status_concerns(init) - tween_step.process(init) # side effects on turbo and on step - - new_image = tween_step.generate(init) - # TODO pass depth instead of tween_step.indexes - new_image = save_and_return_frame(init, tween_step.indexes, new_image) - init.images.previous = new_image # updating reference images to calculate hybrid motions in next iteration - - def generate_depth_maps_if_active(init): # TODO move all depth related stuff to new class. if init.args.anim_args.save_depth_maps: diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 104f676f0..0e5186bde 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -41,7 +41,7 @@ class RenderInitArgs: @dataclass(init=True, frozen=True, repr=False, eq=False) class RenderData: """The purpose of this class is to group and control all data used in render_animation""" - images: Images | None + images: Images | None # TODO rename to reference_images? turbo: Turbo | None indexes: Indexes | None mask: Mask | None diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 89ad77360..fded1ee1a 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -7,7 +7,7 @@ from ..render_data import RenderData from ..schedule import Schedule from ..turbo import Turbo -from ...util import memory_utils, opt_utils, web_ui_utils +from ...util import image_utils, memory_utils, opt_utils, web_ui_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.hybrid import ( call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, @@ -72,6 +72,7 @@ def _hybrid_comp_args(keys, i): @dataclass(init=True, frozen=False, repr=False, eq=False) class Step: step_data: StepData + render_data: RenderData schedule: Schedule depth: Any # TODO try to init early, then freeze class subtitle_params_to_print: Any @@ -84,10 +85,10 @@ def do_start_and_create(data: RenderData): print_animation_frame_info(data) web_ui_utils.update_job(data) # Actual create - step_init = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) + step_data = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) - return Step(step_init, schedule, None, None, "") + return Step(step_data, data, schedule, None, None, "") def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' @@ -102,7 +103,8 @@ def update_depth_prediction(self, data: RenderData, turbo: Turbo): precision = data.args.root.half_precision self.depth = data.depth_model.predict(image, weight, precision) - def write_frame_subtitle(self, data: RenderData): + def maybe_write_frame_subtitle(self): + data = self.render_data if data.turbo.is_first_step_with_subtitles(data): self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(data) self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_to_print) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 71a239389..3259c97d1 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -5,7 +5,7 @@ from ...data.indexes import Indexes from ...data.step import Step from ...img_2_img_tubes import conditional_force_tween_to_grayscale_tube, conditional_add_overlay_mask_tube -from ...util import log_utils, web_ui_utils +from ...util import image_utils, log_utils, web_ui_utils @dataclass(init=True, frozen=False, repr=False, eq=False) @@ -59,3 +59,33 @@ def handle_synchronous_status_concerns(self, init): self.last_step.write_frame_subtitle_if_active(init) # TODO decouple from execution and calc all in advance? log_utils.print_tween_frame_info(init, self.indexes, self.cadence_flow, self.tween) web_ui_utils.update_progress_during_cadence(init, self.indexes) + + @staticmethod + def maybe_emit_in_between_frames(step: Step): + # TODO? return the new frames + data = step.render_data + if data.turbo.is_emit_in_between_frames(): + tween_frame_start_i = max(data.indexes.frame.start, data.indexes.frame.i - data.turbo.steps) + TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, data.indexes.frame.i) + + @staticmethod + def emit_frames_between_index_pair(step: Step, tween_frame_start_i, frame_i): + """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" + tween_range = range(tween_frame_start_i, frame_i) + tween_indexes_list: List[Indexes] = TweenStep.create_indexes(step.render_data.indexes, tween_range) + tween_steps: List[TweenStep] = TweenStep.create_steps(step, tween_indexes_list) + step.render_data.indexes.update_tween_start(step.render_data.turbo) + TweenStep.emit_tween_frames(step, tween_steps) + + @staticmethod + def emit_tween_frames(step: Step, tween_steps): + """Emits a tween frame for each provided tween_step.""" + for tween_step in tween_steps: + tween_step.handle_synchronous_status_concerns(step.render_data) + tween_step.process(step.render_data) # side effects on turbo and on step + + new_image = tween_step.generate(step.render_data) + # TODO pass step and depth instead of data and tween_step.indexes + new_image = image_utils.save_and_return_frame(step.render_data, tween_step.indexes, new_image) + # updating reference images to calculate hybrid motions in next iteration + step.render_data.images.previous = new_image diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 1ece174d6..b0010c3ed 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -2,7 +2,7 @@ from pathlib import Path from ..data import Indexes -from ..data.step import StepData +from ..data.render_data import RenderData from ...video_audio_utilities import get_frame_name @@ -23,28 +23,28 @@ def _frame_filename_index(i: int, file_format: FileFormat) -> str: return f"{i:09}.{file_format.value}" -def _frame_filename(data: StepData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: +def _frame_filename(data: RenderData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" return f"{data.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" -def frame(data: StepData, indexes: Indexes) -> str: +def frame(data: RenderData, indexes: Indexes) -> str: return _frame_filename(data, indexes.frame.i) -def depth_frame(data: StepData, indexes: Indexes) -> str: +def depth_frame(data: RenderData, indexes: Indexes) -> str: return _frame_filename(data, indexes.frame.i, True) -def tween_frame_name(data: StepData, indexes: Indexes) -> str: +def tween_frame_name(data: RenderData, indexes: Indexes) -> str: return _frame_filename(data, indexes.tween.i) -def tween_depth_frame(data: StepData, indexes: Indexes) -> str: +def tween_depth_frame(data: RenderData, indexes: Indexes) -> str: return _frame_filename(data, indexes.tween.i, True) -def preview_video_image_path(data: StepData, indexes: Indexes) -> Path: +def preview_video_image_path(data: RenderData, indexes: Indexes) -> Path: frame_name = get_frame_name(data.args.anim_args.video_init_path) index = _frame_filename_index(indexes.frame.i, FileFormat.video_frame_format()) return Path(data.output_directory) / "inputframes" / (frame_name + index) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index 02a4624c5..be1e6fceb 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -3,21 +3,22 @@ import cv2 from .filename_utils import tween_frame_name +from ..data.render_data import RenderData -def save_cadence_frame(init, indexes, image): - filename = tween_frame_name(init, indexes) - save_path: str = os.path.join(init.args.args.outdir, filename) +def save_cadence_frame(data: RenderData, indexes, image): + filename = tween_frame_name(data, indexes) + save_path: str = os.path.join(data.args.args.outdir, filename) cv2.imwrite(save_path, image) -def save_cadence_frame_and_depth_map_if_active(init, indexes, image): - save_cadence_frame(init, indexes, image) - if init.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(init.output_directory, filename_utils.tween_depth_frame(init, indexes)) - init.depth_model.save(dm_save_path, step.depth) +def save_cadence_frame_and_depth_map_if_active(data: RenderData, indexes, image): + save_cadence_frame(data, indexes, image) + if data.args.anim_args.save_depth_maps: + dm_save_path = os.path.join(data.output_directory, filename_utils.tween_depth_frame(data, indexes)) + data.depth_model.save(dm_save_path, step.depth) -def save_and_return_frame(init, indexes, image): - save_cadence_frame_and_depth_map_if_active(init, indexes, image) +def save_and_return_frame(data: RenderData, indexes, image): + save_cadence_frame_and_depth_map_if_active(data, indexes, image) return image From f54887b6c399aa69ecb8ab38d135d2b528907cde Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 15 Jun 2024 20:37:33 +0200 Subject: [PATCH 074/132] Step logic moved. --- scripts/deforum_helpers/render.py | 133 +++------------- .../rendering/data/step/step.py | 142 +++++++++++++++--- .../rendering/data/step/tween_step.py | 42 +++--- .../rendering/img_2_img_tubes.py | 84 ++++++----- 4 files changed, 209 insertions(+), 192 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 960c8779e..db83354d6 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -14,27 +14,14 @@ # Contact the authors: https://deforum.github.io/ -import os - -import cv2 -import numpy as np -from PIL import Image # noinspection PyUnresolvedReferences from modules.shared import opts, state -from .colors import maintain_colors +from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import Step, TweenStep -from .rendering.img_2_img_tubes import (conditional_color_match_tube, conditional_frame_transformation_tube, - contrasted_noise_transformation_tube, optical_flow_redo_tube, - frame_transformation_tube) -from .rendering.util import generate_random_seed, memory_utils, filename_utils, web_ui_utils -from .rendering.util.call.gen import call_generate -from .rendering.util.call.video_and_audio import call_render_preview -from .rendering.util.log_utils import (print_optical_flow_info, print_redo_generation_info, - print_warning_generate_returned_no_image) -from .save_images import save_image -from .seed import next_seed +from .rendering.util import web_ui_utils +from .rendering.util.log_utils import print_warning_generate_returned_no_image def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -44,112 +31,30 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - last_preview_frame = 0 while data.indexes.frame.i < data.args.anim_args.max_frames: step = Step.do_start_and_create(data) step.maybe_write_frame_subtitle() - TweenStep.maybe_emit_in_between_frames(step) # TODO? make Step/TweenStep union and check if step is TweenStep - - data.images.color_match = Step.create_color_match_for_video(data) # TODO move to step? - data.images.previous = transform_and_update_noised_sample(data, step) # TODO move to step? - data.prepare_generation(data, step) # TODO move to step? - maybe_redo_optical_flow(data, step) # TODO move to step? - maybe_redo_diffusion(data, step) # TODO move to step? + # TODO? make Step/TweenStep union and check if step is TweenStep + grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube + overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube + step = TweenStep.maybe_emit_in_between_frames(step, grayscale_tube, overlay_mask_tube) - image = call_generate(data, step) # TODO move to step? + frame_tube = img_2_img_tubes.frame_transformation_tube + contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube + step.prepare_generation(frame_tube, contrasted_noise_tube) + image = step.do_generation() if image is None: print_warning_generate_returned_no_image() break - image = conditional_frame_transformation_tube(data, step)(image) # TODO move to step? - data.images.color_match = conditional_color_match_tube(data, step)(image) # TODO move to step? - next_frame, step.depth = progress_step(data, image, step.depth) # TODO move to step? - data.indexes.update_frame(next_frame) - state.assign_current_image(image) - # may reassign init.args.args and/or root.seed_internal - data.args.args.seed = next_seed(data.args.args, data.args.root) # TODO group all seeds and sub-seeds - last_preview_frame = call_render_preview(data, last_preview_frame) - web_ui_utils.update_status_tracker(data) - data.animation_mode.unload_raft_and_depth_model() - - -def generate_depth_maps_if_active(init): - # TODO move all depth related stuff to new class. - if init.args.anim_args.save_depth_maps: - memory_utils.handle_vram_before_depth_map_generation(init) - depth = init.depth_model.predict(opencv_image, init.args.anim_args.midas_weight, init.args.root.half_precision) - depth_filename = filename_utils.depth_frame(init, idx) - init.depth_model.save(os.path.join(init.output_directory, depth_filename), depth) - memory_utils.handle_vram_after_depth_map_generation(init) - return depth - - -def progress_step(init, image, depth): - """Will progress frame or turbo-frame step and return next index and `depth`.""" - opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) - if not init.animation_mode.has_video_input: - init.images.previous = opencv_image - if init.turbo.has_steps(): - return init.indexes.frame.i + init.turbo.progress_step(init.indexes, opencv_image), depth - else: - filename = filename_utils.frame(init, init.indexes) - save_image(image, 'PIL', filename, init.args.args, init.args.video_args, init.args.root) - depth = generate_depth_maps_if_active(init) - return init.indexes.frame.i + 1, depth # normal (i.e. 'non-turbo') step always increments by 1. - - -def transform_and_update_noised_sample(init, step): - if init.images.has_previous(): # skipping 1st iteration - transformed_image = frame_transformation_tube(init, step)(init.images.previous) - # TODO separate - noised_image = contrasted_noise_transformation_tube(init, step)(transformed_image) - init.update_sample_and_args_for_current_progression_step(step, noised_image) - return transformed_image - else: - return None - - -# Conditional Redoes -def maybe_redo_optical_flow(init, step): - optical_flow_redo_generation = init.optical_flow_redo_generation_if_not_in_preview_mode() - is_redo_optical_flow = step.is_optical_flow_redo_before_generation(optical_flow_redo_generation, init.images) - if is_redo_optical_flow: - init.args.root.init_sample = do_optical_flow_redo_before_generation(init, step) - - -def maybe_redo_diffusion(init, step): - is_diffusion_redo = init.has_positive_diffusion_redo and init.images.has_previous() and step.step_data.has_strength() - is_not_preview = init.is_not_in_motion_preview_mode() - if is_diffusion_redo and is_not_preview: - do_diffusion_redo(init, step) - - -def do_optical_flow_redo_before_generation(init, step): - stored_seed = init.args.args.seed # keep original to reset it after executing the optical flow - init.args.args.seed = generate_random_seed() # set a new random seed - print_optical_flow_info(init, optical_flow_redo_generation) # TODO output temp seed? - - sample_image = call_generate(init, init.indexes.frame.i, step.schedule) - optical_tube = optical_flow_redo_tube(init, optical_flow_redo_generation) - transformed_sample_image = optical_tube(sample_image) - - init.args.args.seed = stored_seed # restore stored seed - return Image.fromarray(transformed_sample_image) + image = img_2_img_tubes.conditional_frame_transformation_tube(step)(image) + step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(step)(image) + step.progress_and_save(image) + state.assign_current_image(image) + step.render_data.args.args.seed = step.next_seed() -def do_diffusion_redo(init, step): - stored_seed = init.args.args.seed - last_diffusion_redo_index = int(init.args.anim_args.diffusion_redo) - for n in range(0, last_diffusion_redo_index): - print_redo_generation_info(init, n) - init.args.args.seed = generate_random_seed() - diffusion_redo_image = call_generate(init, step) - diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) - # color match on last one only - is_last_iteration = n == last_diffusion_redo_index - if is_last_iteration: - mode = init.args.anim_args.color_coherence - diffusion_redo_image = maintain_colors(init.images.previous, init.images.color_match, mode) - init.args.args.seed = stored_seed - init.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) + step.update_render_preview() + web_ui_utils.update_status_tracker(step.render_data) + step.render_data.animation_mode.unload_raft_and_depth_model() diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index fded1ee1a..25591b3a4 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -7,17 +7,20 @@ from ..render_data import RenderData from ..schedule import Schedule from ..turbo import Turbo -from ...util import image_utils, memory_utils, opt_utils, web_ui_utils +from ...util import memory_utils, opt_utils, web_ui_utils from ...util.call.anim import call_anim_frame_warp +from ...util.call.gen import call_generate from ...util.call.hybrid import ( - call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, - call_get_matrix_for_hybrid_motion, + call_get_flow_for_hybrid_motion, call_get_flow_for_hybrid_motion_prev, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev, call_hybrid_composite) from ...util.call.images import call_add_noise from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle +from ...util.call.video_and_audio import call_render_preview from ...util.log_utils import print_animation_frame_info from ....hybrid_video import image_transform_ransac, image_transform_optical_flow +from ....save_images import save_image +from ....seed import next_seed @dataclass(init=True, frozen=True, repr=False, eq=False) @@ -46,17 +49,18 @@ def has_strength(self): @staticmethod def create(deform_keys, i): keys = deform_keys - return StepData(keys.noise_schedule_series[i], - keys.strength_schedule_series[i], - keys.cfg_scale_schedule_series[i], - keys.contrast_schedule_series[i], - int(keys.kernel_schedule_series[i]), - keys.sigma_schedule_series[i], - keys.amount_schedule_series[i], - keys.threshold_schedule_series[i], - keys.cadence_flow_factor_schedule_series[i], - keys.redo_flow_factor_schedule_series[i], - StepData._hybrid_comp_args(keys, i)) + return StepData( + keys.noise_schedule_series[i], + keys.strength_schedule_series[i], + keys.cfg_scale_schedule_series[i], + keys.contrast_schedule_series[i], + int(keys.kernel_schedule_series[i]), + keys.sigma_schedule_series[i], + keys.amount_schedule_series[i], + keys.threshold_schedule_series[i], + keys.cadence_flow_factor_schedule_series[i], + keys.redo_flow_factor_schedule_series[i], + StepData._hybrid_comp_args(keys, i)) @staticmethod def _hybrid_comp_args(keys, i): @@ -77,6 +81,7 @@ class Step: depth: Any # TODO try to init early, then freeze class subtitle_params_to_print: Any subtitle_params_string: str + last_preview_frame: int @staticmethod def do_start_and_create(data: RenderData): @@ -88,7 +93,7 @@ def do_start_and_create(data: RenderData): step_data = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) - return Step(step_data, data, schedule, None, None, "") + return Step(step_data, data, schedule, None, None, "", 0) def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' @@ -201,8 +206,8 @@ def apply_hybrid_motion_optical_flow(data: RenderData, image): else: return image - @staticmethod - def create_color_match_for_video(data: RenderData): + def create_color_match_for_video(self): + data = self.render_data if data.args.anim_args.color_coherence == 'Video Input' and data.is_hybrid_available(): if int(data.indexes.frame.i) % int(data.args.anim_args.color_coherence_video_every_N_frames) == 0: prev_vid_img = Image.open(preview_video_image_path(data, data.indexes)) @@ -210,3 +215,106 @@ def create_color_match_for_video(data: RenderData): data.images.color_match = np.asarray(prev_vid_img) return cv2.cvtColor(data.images.color_match, cv2.COLOR_RGB2BGR) return None + + def transform_and_update_noised_sample(self, frame_tube, contrasted_noise_tube): + data = self.render_data + if data.images.has_previous(): # skipping 1st iteration + transformed_image = frame_tube(data, self)(data.images.previous) + # TODO separate + noised_image = contrasted_noise_tube(data, self)(transformed_image) + data.update_sample_and_args_for_current_progression_step(self, noised_image) + return transformed_image + else: + return None + + def prepare_generation(self, frame_tube, contrasted_noise_tube): + self.render_data.images.color_match = self.create_color_match_for_video() + self.render_data.images.previous = self.transform_and_update_noised_sample(frame_tube, contrasted_noise_tube) + self.render_data.prepare_generation(self.render_data, self) + self.maybe_redo_optical_flow() + self.maybe_redo_diffusion() + + # Conditional Redoes + def maybe_redo_optical_flow(self): + data = self.render_data + optical_flow_redo_generation = data.optical_flow_redo_generation_if_not_in_preview_mode() + is_redo_optical_flow = self.is_optical_flow_redo_before_generation(optical_flow_redo_generation, data.images) + if is_redo_optical_flow: + data.args.root.init_sample = self.do_optical_flow_redo_before_generation() + + def maybe_redo_diffusion(self): + data = self.render_data + is_pos_redo = data.has_positive_diffusion_redo + is_diffusion_redo = is_pos_redo and data.images.has_previous() and self.step_data.has_strength() + is_not_preview = data.is_not_in_motion_preview_mode() + if is_diffusion_redo and is_not_preview: + self.do_diffusion_redo() + + def do_generation(self): + return call_generate(self.render_data, self) + + def progress_and_save(self, image): + next_index = self._progress_save_and_get_next_index(image) + self.render_data.indexes.update_frame(next_index) + + def _progress_save_and_get_next_index(self, image): + data = self.render_data + """Will progress frame or turbo-frame step, save the image, update `self.depth` and return next index.""" + opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + if not data.animation_mode.has_video_input: + data.images.previous = opencv_image + if data.turbo.has_steps(): + return data.indexes.frame.i + data.turbo.progress_step(data.indexes, opencv_image) + else: + filename = filename_utils.frame(data, data.indexes) + save_image(image, 'PIL', filename, data.args.args, data.args.video_args, data.args.root) + self.depth = generate_depth_maps_if_active(data) + return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. + + def next_seed(self): + return next_seed(self.render_data.args.args, self.render_data.args.root) + + def update_render_preview(self): + self.last_preview_frame = call_render_preview(self.render_data, self.last_preview_frame) + + def generate_depth_maps_if_active(self): + data = self.render_data + # TODO move all depth related stuff to new class. + if data.args.anim_args.save_depth_maps: + memory_utils.handle_vram_before_depth_map_generation(data) + depth = data.depth_model.predict(opencv_image, data.args.anim_args.midas_weight, + data.args.root.half_precision) + depth_filename = filename_utils.depth_frame(data, idx) + data.depth_model.save(os.path.join(data.output_directory, depth_filename), depth) + memory_utils.handle_vram_after_depth_map_generation(data) + return depth + + def do_optical_flow_redo_before_generation(self): + data = self.render_data + stored_seed = data.args.args.seed # keep original to reset it after executing the optical flow + data.args.args.seed = generate_random_seed() # set a new random seed + print_optical_flow_info(data, optical_flow_redo_generation) # TODO output temp seed? + + sample_image = call_generate(data, data.indexes.frame.i, self.schedule) + optical_tube = optical_flow_redo_tube(data, optical_flow_redo_generation) + transformed_sample_image = optical_tube(sample_image) + + data.args.args.seed = stored_seed # restore stored seed + return Image.fromarray(transformed_sample_image) + + def do_diffusion_redo(self): + data = self.render_data + stored_seed = data.args.args.seed + last_diffusion_redo_index = int(data.args.anim_args.diffusion_redo) + for n in range(0, last_diffusion_redo_index): + print_redo_generation_info(data, n) + data.args.args.seed = generate_random_seed() + diffusion_redo_image = call_generate(data, self) + diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) + # color match on last one only + is_last_iteration = n == last_diffusion_redo_index + if is_last_iteration: + mode = data.args.anim_args.color_coherence + diffusion_redo_image = maintain_colors(data.images.previous, data.images.color_match, mode) + data.args.args.seed = stored_seed + data.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 3259c97d1..9df2ca482 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -4,7 +4,6 @@ from ...data.indexes import Indexes from ...data.step import Step -from ...img_2_img_tubes import conditional_force_tween_to_grayscale_tube, conditional_add_overlay_mask_tube from ...util import image_utils, log_utils, web_ui_utils @@ -39,15 +38,15 @@ def create_directly(from_index, to_index): tween = TweenStep._calculate_tween_from_indices(from_index, to_index) return TweenStep(tween, None, None) - def generate_tween_image(self, init): + def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): is_tween = True - warped = init.turbo.do_optical_flow_cadence_after_animation_warping(init, self.indexes, self.last_step, self) - recolored = conditional_force_tween_to_grayscale_tube(init)(warped) - masked = conditional_add_overlay_mask_tube(init, is_tween)(recolored) + warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self.last_step, self) + recolored = grayscale_tube(data)(warped) + masked = overlay_mask_tube(data, is_tween)(recolored) return masked - def generate(self, init): - return self.generate_tween_image(init) + def generate(self, data, grayscale_tube, overlay_mask_tube): + return self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) def process(self, init): init.turbo.advance_optical_flow_cadence_before_animation_warping(init, self) @@ -55,37 +54,40 @@ def process(self, init): init.turbo.advance(init, self.indexes.tween.i, self.last_step.depth) init.turbo.do_hybrid_video_motion(init, self.indexes, init.images) # TODO remove self.indexes or init.indexes - def handle_synchronous_status_concerns(self, init): - self.last_step.write_frame_subtitle_if_active(init) # TODO decouple from execution and calc all in advance? - log_utils.print_tween_frame_info(init, self.indexes, self.cadence_flow, self.tween) - web_ui_utils.update_progress_during_cadence(init, self.indexes) + def handle_synchronous_status_concerns(self, data): + self.last_step.write_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? + log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.tween) + web_ui_utils.update_progress_during_cadence(data, self.indexes) @staticmethod - def maybe_emit_in_between_frames(step: Step): + def maybe_emit_in_between_frames(step: Step, grayscale_tube, overlay_mask_tube): # TODO? return the new frames - data = step.render_data - if data.turbo.is_emit_in_between_frames(): - tween_frame_start_i = max(data.indexes.frame.start, data.indexes.frame.i - data.turbo.steps) - TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, data.indexes.frame.i) + if step.render_data.turbo.is_emit_in_between_frames(): + tween_frame_start_i = max(step.render_data.indexes.frame.start, + step.render_data.indexes.frame.i - step.render_data.turbo.steps) + return TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, step.render_data.indexes.frame.i, + grayscale_tube, overlay_mask_tube) + return step @staticmethod - def emit_frames_between_index_pair(step: Step, tween_frame_start_i, frame_i): + def emit_frames_between_index_pair(step: Step, tween_frame_start_i, frame_i, grayscale_tube, overlay_mask_tube): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" tween_range = range(tween_frame_start_i, frame_i) tween_indexes_list: List[Indexes] = TweenStep.create_indexes(step.render_data.indexes, tween_range) tween_steps: List[TweenStep] = TweenStep.create_steps(step, tween_indexes_list) step.render_data.indexes.update_tween_start(step.render_data.turbo) - TweenStep.emit_tween_frames(step, tween_steps) + return TweenStep.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) @staticmethod - def emit_tween_frames(step: Step, tween_steps): + def emit_tween_frames(step: Step, tween_steps, grayscale_tube, overlay_mask_tube): """Emits a tween frame for each provided tween_step.""" for tween_step in tween_steps: tween_step.handle_synchronous_status_concerns(step.render_data) tween_step.process(step.render_data) # side effects on turbo and on step - new_image = tween_step.generate(step.render_data) + new_image = tween_step.generate(step.render_data, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes new_image = image_utils.save_and_return_frame(step.render_data, tween_step.indexes, new_image) # updating reference images to calculate hybrid motions in next iteration step.render_data.images.previous = new_image + return step diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index b58ebebf2..a3b9d52b4 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -4,6 +4,7 @@ import numpy as np from cv2.typing import MatLike +from .data.render_data import RenderData from .data.step.step import Step from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube @@ -26,93 +27,94 @@ ImageTube = Callable[[MatLike], MatLike] -def frame_transformation_tube(init, step) -> ImageTube: +def frame_transformation_tube(data: RenderData, step: Step) -> ImageTube: # make sure `img` stays the last argument in each call. - return tube(lambda img: step.apply_frame_warp_transform(init, img), - lambda img: step.do_hybrid_compositing_before_motion(init, img), - lambda img: Step.apply_hybrid_motion_ransac_transform(init, img), - lambda img: Step.apply_hybrid_motion_optical_flow(init, img), - lambda img: step.do_normal_hybrid_compositing_after_motion(init, img), - lambda img: Step.apply_color_matching(init, img), - lambda img: Step.transform_to_grayscale_if_active(init, img)) + return tube(lambda img: step.apply_frame_warp_transform(data, img), + lambda img: step.do_hybrid_compositing_before_motion(data, img), + lambda img: Step.apply_hybrid_motion_ransac_transform(data, img), + lambda img: Step.apply_hybrid_motion_optical_flow(data, img), + lambda img: step.do_normal_hybrid_compositing_after_motion(data, img), + lambda img: Step.apply_color_matching(data, img), + lambda img: Step.transform_to_grayscale_if_active(data, img)) -def contrast_transformation_tube(init, step) -> ImageTube: +def contrast_transformation_tube(data: RenderData, step: Step) -> ImageTube: return tube(lambda img: step.apply_scaling(img), - lambda img: step.apply_anti_blur(init, img)) + lambda img: step.apply_anti_blur(data, img)) -def noise_transformation_tube(init, step) -> ImageTube: - return tube(lambda img: step.apply_frame_noising(init, step, img)) +def noise_transformation_tube(data: RenderData, step: Step) -> ImageTube: + return tube(lambda img: step.apply_frame_noising(data, step, img)) -def optical_flow_redo_tube(init, optical_flow) -> ImageTube: +def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), - lambda img: image_transform_optical_flow( # TODO create img.get_flow - img, call_get_flow_from_images(init, init.images.previous, img, optical_flow), + lambda img: image_transform_optical_flow( + img, call_get_flow_from_images(data, data.images.previous, img, optical_flow), step.step_data.redo_flow_factor)) # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). -def conditional_hybrid_video_after_generation_tube(init, step) -> ImageTube: +def conditional_hybrid_video_after_generation_tube(step: Step) -> ImageTube: + data = step.render_data return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: call_hybrid_composite(init, init.indexes.frame.i, img, step.step_data.hybrid_comp_schedules), + lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step.step_data.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= - lambda: init.indexes.is_not_first_frame() and init.is_hybrid_composite_after_generation()) + lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) -def conditional_extra_color_match_tube(init) -> ImageTube: +def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output - return tube(lambda img: maintain_colors(img, init.images.color_match, init.args.anim_args.color_coherence), + return tube(lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: maintain_colors(img, init.images.color_match, init.args.anim_args.color_coherence), + lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= - lambda: init.indexes.is_first_frame() and init.is_initialize_color_match(init.images.color_match)) + lambda: data.indexes.is_first_frame() and data.is_initialize_color_match(data.images.color_match)) -def conditional_color_match_tube(init, step) -> ImageTube: +def conditional_color_match_tube(step: Step) -> ImageTube: # on strength 0, set color match to generation return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), - is_do_process=lambda: init.is_do_color_match_conversion(step)) + is_do_process=lambda: step.render_data.is_do_color_match_conversion(step)) -def conditional_force_to_grayscale_tube(init) -> ImageTube: +def conditional_force_to_grayscale_tube(data: RenderData) -> ImageTube: return tube(lambda img: ImageOps.grayscale(img), lambda img: ImageOps.colorize(img, black="black", white="white"), - is_do_process=lambda: init.args.anim_args.color_force_grayscale) + is_do_process=lambda: data.args.anim_args.color_force_grayscale) -def conditional_add_overlay_mask_tube(init, is_tween) -> ImageTube: - is_use_overlay = init.args.args.overlay_mask - is_use_mask = init.args.anim_args.use_mask_video or init.args.args.use_mask - index = init.indexes.tween.i if is_tween else init.indexes.frame.i +def conditional_add_overlay_mask_tube(data: RenderData, is_tween) -> ImageTube: + is_use_overlay = data.args.args.overlay_mask + is_use_mask = data.args.anim_args.use_mask_video or data.args.args.use_mask + index = data.indexes.tween.i if is_tween else data.indexes.frame.i is_bgr_array = True return tube(lambda img: ImageOps.grayscale(img), - lambda img: do_overlay_mask(init.args.args, init.args.anim_args, img, index, is_bgr_array), + lambda img: do_overlay_mask(data.args.args, data.args.anim_args, img, index, is_bgr_array), is_do_process=lambda: is_use_overlay and is_use_mask) -def conditional_force_tween_to_grayscale_tube(init) -> ImageTube: +def conditional_force_tween_to_grayscale_tube(data: RenderData) -> ImageTube: return tube(lambda img: cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY), lambda img: cv2.cvtColor(img, cv2.COLOR_GRAY2BGR), - is_do_process=lambda: init.args.anim_args.color_force_grayscale) + is_do_process=lambda: data.args.anim_args.color_force_grayscale) # Composite Tubes, made from other Tubes. -def contrasted_noise_transformation_tube(init, step) -> ImageTube: +def contrasted_noise_transformation_tube(data: RenderData, step: Step) -> ImageTube: """Combines contrast and noise transformation tubes.""" - contrast_tube: Tube = contrast_transformation_tube(init, step) - noise_tube: Tube = noise_transformation_tube(init, step) + contrast_tube: Tube = contrast_transformation_tube(data, step) + noise_tube: Tube = noise_transformation_tube(data, step) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(init, step, is_tween: bool = False) -> ImageTube: - hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(init, step) - extra_tube: Tube = conditional_extra_color_match_tube(init) - gray_tube: Tube = conditional_force_to_grayscale_tube(init) - mask_tube: Tube = conditional_add_overlay_mask_tube(init, is_tween) +def conditional_frame_transformation_tube(step: Step, is_tween: bool = False) -> ImageTube: + hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(step) + extra_tube: Tube = conditional_extra_color_match_tube(step.render_data) + gray_tube: Tube = conditional_force_to_grayscale_tube(step.render_data) + mask_tube: Tube = conditional_add_overlay_mask_tube(step.render_data, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) From 318bf992b8ec7765fa2b013e12deb69de60be079 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 00:10:57 +0200 Subject: [PATCH 075/132] Tween index generation detached. --- .../deforum_helpers/rendering/data/indexes.py | 12 +++++-- .../rendering/data/step/tween_step.py | 35 +++++++++++++------ .../deforum_helpers/rendering/data/turbo.py | 17 +++++---- .../rendering/util/call/resume.py | 6 ++-- 4 files changed, 48 insertions(+), 22 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index c2072145e..62c786b32 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -6,6 +6,9 @@ class IndexWithStart: start: int = 0 i: int = 0 + def copy(self): + return IndexWithStart(start=self.start, i=self.i) + @dataclass(init=True, frozen=False, repr=False, eq=False) class Indexes: @@ -14,7 +17,7 @@ class Indexes: @staticmethod def create(init, turbo): - frame_start = turbo.find_start(init, turbo) + frame_start = turbo.find_start(init) tween_start = 0 return Indexes(IndexWithStart(frame_start, 0), IndexWithStart(tween_start, 0)) @@ -23,7 +26,7 @@ def create_from_last(last_indexes, i: int): """Creates a new `Indexes` object based on the last one, but updates the tween start index.""" return Indexes(last_indexes.frame, IndexWithStart(last_indexes.tween.start, i)) - def create_next(self): + def create_next_tween(self): return Indexes(self.frame, IndexWithStart(self.tween.start, self.tween.i + 1)) def update_tween_start(self, turbo): @@ -38,3 +41,8 @@ def is_not_first_frame(self): def is_first_frame(self): return self.frame.i == 0 + + def copy(self): + return Indexes( + frame=self.frame.copy() if self.frame else None, + tween=self.tween.copy() if self.tween else None) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 9df2ca482..61672958b 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -2,7 +2,7 @@ from itertools import chain from typing import Any, Iterable -from ...data.indexes import Indexes +from ...data.indexes import Indexes, IndexWithStart from ...data.step import Step from ...util import image_utils, log_utils, web_ui_utils @@ -21,9 +21,25 @@ def _calculate_tween_from_indices(frame_difference, last_step) -> float: return min(0.0, max(1.0, float(last_step) / float(frame_difference))) @staticmethod - def create(indexes, last_step): - tween = float(indexes.tween.i - indexes.tween.start + 1) / float(indexes.frame.i - indexes.tween.start) - return TweenStep(indexes, last_step, tween, None, None) + def _calculate_expected_tween_frames(num_entries): + if num_entries <= 0: + raise ValueError("Number of entries must be positive") + offset = 1.0 / num_entries + positions = [offset + (i / num_entries) for i in range(num_entries)] + return positions + + @staticmethod + def _increment(original_indexes, tween_count, from_start): + inc = original_indexes.frame.i - tween_count - original_indexes.tween.start + from_start + original_indexes.tween = IndexWithStart(original_indexes.tween.start, original_indexes.tween.start + inc) + return original_indexes + + @staticmethod + def create_steps_from_values(last_step, values): + count = len(values) + r = range(count) + indexes_list = [TweenStep._increment(last_step.render_data.indexes.copy(), count, i + 1) for i in r] + return list((TweenStep(indexes_list[i], last_step, values[i], None, None) for i in r)) @staticmethod def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[Indexes]: @@ -31,12 +47,11 @@ def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[In @staticmethod def create_steps(last_step, tween_indexes_list: list[Indexes]) -> list['TweenStep']: - return [TweenStep.create(i.create_next(), last_step) for i in tween_indexes_list] - - @staticmethod - def create_directly(from_index, to_index): - tween = TweenStep._calculate_tween_from_indices(from_index, to_index) - return TweenStep(tween, None, None) + if len(tween_indexes_list) > 0: + expected_tween_frames = TweenStep._calculate_expected_tween_frames(len(tween_indexes_list)) + return TweenStep.create_steps_from_values(last_step, expected_tween_frames) + else: + return list() def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): is_tween = True diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 52c166b15..a250de9eb 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -124,18 +124,21 @@ def progress_step(self, indexes, opencv_image): self.next.image, self.next.index = opencv_image, indexes.frame.i return self.steps - def _set_up_step_vars(self, init, turbo): + def _set_up_step_vars(self, data): # determine last frame and frame to start on - prev_frame, next_frame, prev_img, next_img = call_get_resume_vars(init, turbo) + prev_frame, next_frame, prev_img, next_img = call_get_resume_vars(data, self) if self.steps > 1: - self.prev.image, self.prev.index = prev_img, prev_frame - self.next.image, self.next.index = next_img, next_frame + self.prev.image, self.prev.index = prev_img, prev_frame if prev_frame >= 0 else 0 + self.next.image, self.next.index = next_img, next_frame if next_frame >= 0 else 0 - def find_start(self, init, turbo) -> int: + def find_start(self, data) -> int: """Maybe resume animation (requires at least two frames - see function).""" # set start_frame to next frame - self._set_up_step_vars(init, turbo) - return self.next.index + 1 if init.is_resuming_from_timestring() else 0 + if data.is_resuming_from_timestring(): + self._set_up_step_vars(data) + return self.next.index + 1 + else: + return 0 def has_steps(self): return self.steps > 1 diff --git a/scripts/deforum_helpers/rendering/util/call/resume.py b/scripts/deforum_helpers/rendering/util/call/resume.py index c3cc3cbd1..75dd44510 100644 --- a/scripts/deforum_helpers/rendering/util/call/resume.py +++ b/scripts/deforum_helpers/rendering/util/call/resume.py @@ -1,7 +1,7 @@ from ....resume import get_resume_vars -def call_get_resume_vars(init, turbo): - return get_resume_vars(folder=init.args.args.outdir, - timestring=init.args.anim_args.resume_timestring, +def call_get_resume_vars(data, turbo): + return get_resume_vars(folder=data.args.args.outdir, + timestring=data.args.anim_args.resume_timestring, cadence=turbo.steps) From e50419a85d408062a990083a137a6aabc23eed2e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 00:25:07 +0200 Subject: [PATCH 076/132] Turbo cleanup. --- .../deforum_helpers/rendering/data/turbo.py | 66 ++++++++++--------- .../rendering/img_2_img_tubes.py | 3 +- 2 files changed, 36 insertions(+), 33 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index a250de9eb..fb454e28b 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -22,27 +22,27 @@ class Turbo: next: ImageFrame @staticmethod - def create(init): - steps = 1 if init.has_video_input() else init.cadence() + def create(data): + steps = 1 if data.has_video_input() else data.cadence() return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) - def advance(self, init, i: int, depth): + def advance(self, data, i: int, depth): if self.is_advance_prev(i): - self.prev.image, _ = call_anim_frame_warp(init, i, self.prev.image, depth) + self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) if self.is_advance_next(i): - self.next.image, _ = call_anim_frame_warp(init, i, self.next.image, depth) + self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) - def do_hybrid_video_motion(self, init, indexes, reference_images): + def do_hybrid_video_motion(self, data, indexes, reference_images): """Warps the previous and/or the next to match the motion of the provided reference images.""" - motion = init.args.anim_args.hybrid_motion + motion = data.args.anim_args.hybrid_motion def _is_do_motion(motions): return indexes.tween.i > 0 and motion in motions if _is_do_motion(['Affine', 'Perspective']): - self.advance_hybrid_motion_ransac_trasform(init, indexes, reference_images) + self.advance_hybrid_motion_ransac_trasform(data, indexes, reference_images) if _is_do_motion(['Optical Flow']): - self.advance_hybrid_motion_optical_tween_flow(init, indexes, reference_images, step) + self.advance_hybrid_motion_optical_tween_flow(data, indexes, reference_images, step) def advance_optical_flow(self, tween_step, flow_factor: int = 1): flow = tween_step.cadence_flow * -1 @@ -56,15 +56,15 @@ def advance_optical_tween_flow(self, step, flow): if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, ff) - def advance_hybrid_motion_optical_tween_flow(self, init, indexes, reference_images, step): - if init.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(init, indexes.tween.i - 1, reference_images.previous) + def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_images, step): + if data.args.anim_args.hybrid_motion_use_prev_img: + flow = call_get_flow_for_hybrid_motion_prev(data, indexes.tween.i - 1, reference_images.previous) turbo.advance_optical_tween_flow(self, step, flow) - init.animation_mode.prev_flow = flow + data.animation_mode.prev_flow = flow else: - flow = call_get_flow_for_hybrid_motion(init, indexes.tween.i - 1) + flow = call_get_flow_for_hybrid_motion(data, indexes.tween.i - 1) turbo.advance_optical_tween_flow(self, step, flow) - init.animation_mode.prev_flow = flow + data.animation_mode.prev_flow = flow def advance_cadence_flow(self, tween_step): ff = step.step_data.sub_step.cadence_flow_factor @@ -75,42 +75,44 @@ def advance_cadence_flow(self, tween_step): if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, inc, ff) - def advance_ransac_trasform(self, init, matrix): + # TODO? move to RenderData + def advance_ransac_trasform(self, data, matrix): i = indexes.tween.i - motion = init.args.anim_args.hybrid_motion + motion = data.args.anim_args.hybrid_motion if self.is_advance_prev(i): self.prev.image = image_transform_ransac(self.prev.image, matrix, motion) if self.is_advance_next(i): self.next.image = image_transform_ransac(self.next.image, matrix, motion) - def advance_hybrid_motion_ransac_trasform(self, init, indexes, reference_images): - if init.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(init, indexes.tween.i - 1, reference_images.previous) - turbo.advance_ransac_trasform(init, matrix) + # TODO? move to RenderData + def advance_hybrid_motion_ransac_trasform(self, data, indexes, reference_images): + if data.args.anim_args.hybrid_motion_use_prev_img: + matrix = call_get_matrix_for_hybrid_motion_prev(data, indexes.tween.i - 1, reference_images.previous) + turbo.advance_ransac_trasform(data, matrix) else: - matrix = call_get_matrix_for_hybrid_motion(init, indexes.tween.i - 1) - turbo.advance_ransac_trasform(init, matrix) + matrix = call_get_matrix_for_hybrid_motion(data, indexes.tween.i - 1) + turbo.advance_ransac_trasform(data, matrix) - def advance_optical_flow_cadence_before_animation_warping(self, init, tween_step): - if init.is_3d_or_2d() and init.has_optical_flow_cadence(): - has_tween_schedule = init.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0 + def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step): + if data.is_3d_or_2d() and data.has_optical_flow_cadence(): + has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0 has_images = self.prev.image is not None and self.next.image is not None has_step_and_images = tween_step.cadence_flow is None and has_images if has_tween_schedule and has_step_and_images: - cadence = init.args.anim_args.optical_flow_cadence - flow = call_get_flow_from_images(init, self.prev.image, self.next.image, cadence) + cadence = data.args.anim_args.optical_flow_cadence + flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) tween_step.cadence_flow = (flow / 2) self.next.image = advance_optical_flow self.advance_optical_flow(tween_step) self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow) - def do_optical_flow_cadence_after_animation_warping(self, init, indexes, step, tween_step): + def do_optical_flow_cadence_after_animation_warping(self, data, indexes, step, tween_step): if tween_step.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). - temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, init.width(), init.height()) - new_flow, _ = call_anim_frame_warp(init, indexes.tween.i, temp_flow, step.depth) + temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, step.depth) tween_step.cadence_flow = new_flow - abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, init.width(), init.height()) + abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween self.advance_cadence_flow(tween_step) self.prev.index = self.next.frame_idx = indexes.tween.i diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index a3b9d52b4..4973a20c8 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -58,8 +58,9 @@ def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). def conditional_hybrid_video_after_generation_tube(step: Step) -> ImageTube: data = step.render_data + step_data = step.step_data return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step.step_data.hybrid_comp_schedules), + lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), is_do_process= lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) From 661910b9dd4dee9a274d99514692c29ad76a88bc Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 05:44:09 +0200 Subject: [PATCH 077/132] Cadence logging combined. --- .../rendering/data/step/tween_step.py | 4 ++- .../deforum_helpers/rendering/data/turbo.py | 2 +- .../rendering/util/log_utils.py | 27 +++++++++++++++---- 3 files changed, 26 insertions(+), 7 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 61672958b..922b51725 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -80,7 +80,8 @@ def maybe_emit_in_between_frames(step: Step, grayscale_tube, overlay_mask_tube): if step.render_data.turbo.is_emit_in_between_frames(): tween_frame_start_i = max(step.render_data.indexes.frame.start, step.render_data.indexes.frame.i - step.render_data.turbo.steps) - return TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, step.render_data.indexes.frame.i, + return TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, + step.render_data.indexes.frame.i, grayscale_tube, overlay_mask_tube) return step @@ -91,6 +92,7 @@ def emit_frames_between_index_pair(step: Step, tween_frame_start_i, frame_i, gra tween_indexes_list: List[Indexes] = TweenStep.create_indexes(step.render_data.indexes, tween_range) tween_steps: List[TweenStep] = TweenStep.create_steps(step, tween_indexes_list) step.render_data.indexes.update_tween_start(step.render_data.turbo) + log_utils.print_tween_frame_from_to_info(step.render_data.turbo.steps, tween_frame_start_i, frame_i) return TweenStep.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) @staticmethod diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index fb454e28b..b342324e7 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -17,7 +17,7 @@ class ImageFrame: # TODO freeze.. @dataclass(frozen=False) class Turbo: - steps: int + steps: int # cadence prev: ImageFrame next: ImageFrame diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 3b4ff9fc8..691d76b73 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,5 +1,17 @@ -CYAN = "\033[36m" +RED = "\033[31m" +ORANGE = "\033[38;5;208m" YELLOW = "\033[33m" +GREEN = "\033[32m" +CYAN = "\033[36m" +BLUE = "\033[34m" +INDIGO = "\033[38;5;66m" +VIOLET = "\033[38;5;130m" +BLACK = "\033[30m" +WHITE = "\033[37m" + +BOLD = "\033[1m" +UNDERLINE = "\033[4m" + RESET = "\033[0m" @@ -7,10 +19,15 @@ def print_animation_frame_info(init): print(f"{CYAN}Animation frame: {RESET}{init.indexes.frame.i}/{init.args.anim_args.max_frames}") -def print_tween_frame_info(init, indexes, cadence_flow, tween): - msg_flow_name = '' if cadence_flow is None else init.args.anim_args.optical_flow_cadence + ' optical flow ' - msg_frame_info = f"cadence frame: {indexes.tween.i}; tween: {tween:0.2f};" - print(f"Creating in-between {msg_flow_name}{msg_frame_info}") +def print_tween_frame_from_to_info(cadence, start_i, end_i): + if end_i > 0: + print(f"{ORANGE}Creating in-between: {RESET}{cadence} frames ({start_i}-->{end_i}).") + +def print_tween_frame_info(data, indexes, cadence_flow, tween, if_disable=True): + if not if_disable: + msg_flow_name = '' if cadence_flow is None else data.args.anim_args.optical_flow_cadence + ' optical flow ' + msg_frame_info = f"cadence frame: {indexes.tween.i}; tween: {tween:0.2f};" + print(f"Creating in-between {msg_flow_name}{msg_frame_info}") def print_init_frame_info(init_frame): From 90ad9f1afd981fc7d02ab8de63511fc3b988ba76 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 06:43:31 +0200 Subject: [PATCH 078/132] Switched main render loop to process a list of precalculated steps. --- scripts/deforum_helpers/render.py | 9 +++---- .../rendering/data/step/step.py | 13 +++++++++- .../rendering/data/step/tween_step.py | 26 +++++++++---------- .../rendering/util/log_utils.py | 12 ++++++--- 4 files changed, 37 insertions(+), 23 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index db83354d6..ddc4a8c8f 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -21,7 +21,7 @@ from .rendering.data.render_data import RenderData from .rendering.data.step import Step, TweenStep from .rendering.util import web_ui_utils -from .rendering.util.log_utils import print_warning_generate_returned_no_image +from .rendering.util.log_utils import print_animation_frame_info, print_warning_generate_returned_no_image def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -31,14 +31,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - while data.indexes.frame.i < data.args.anim_args.max_frames: - step = Step.do_start_and_create(data) - step.maybe_write_frame_subtitle() - + steps = Step.create_all_steps(data) + for step in steps: # TODO? make Step/TweenStep union and check if step is TweenStep grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube step = TweenStep.maybe_emit_in_between_frames(step, grayscale_tube, overlay_mask_tube) + print_animation_frame_info(step.render_data) frame_tube = img_2_img_tubes.frame_transformation_tube contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/step.py index 25591b3a4..d449de4e4 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/step.py @@ -87,7 +87,6 @@ class Step: def do_start_and_create(data: RenderData): # Perform the necessary side effects memory_utils.handle_med_or_low_vram_before_step(data) - print_animation_frame_info(data) web_ui_utils.update_job(data) # Actual create step_data = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) @@ -95,6 +94,18 @@ def do_start_and_create(data: RenderData): data.args.anim_args, data.args.args) return Step(step_data, data, schedule, None, None, "", 0) + @staticmethod + def create_all_steps(data): + """Creates a list of steps for the entire animation.""" + steps = [] + max_steps = int(data.args.anim_args.max_frames / data.cadence()) + for step_index in range(max_steps): + step = Step.do_start_and_create(data) + step.maybe_write_frame_subtitle() + steps.append(step) + assert len(steps) == max_steps + return steps + def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' return has_flow_redo and images.has_previous() and self.step_data.has_strength() diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 922b51725..67ad5a733 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -1,6 +1,6 @@ from dataclasses import dataclass from itertools import chain -from typing import Any, Iterable +from typing import Any, Iterable, Tuple from ...data.indexes import Indexes, IndexWithStart from ...data.step import Step @@ -46,12 +46,12 @@ def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[In return list(chain.from_iterable([Indexes.create_from_last(base_indexes, i)] for i in frame_range)) @staticmethod - def create_steps(last_step, tween_indexes_list: list[Indexes]) -> list['TweenStep']: + def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['TweenStep'], list[float]]: if len(tween_indexes_list) > 0: expected_tween_frames = TweenStep._calculate_expected_tween_frames(len(tween_indexes_list)) - return TweenStep.create_steps_from_values(last_step, expected_tween_frames) + return TweenStep.create_steps_from_values(last_step, expected_tween_frames), expected_tween_frames else: - return list() + return list(), list() def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): is_tween = True @@ -78,21 +78,21 @@ def handle_synchronous_status_concerns(self, data): def maybe_emit_in_between_frames(step: Step, grayscale_tube, overlay_mask_tube): # TODO? return the new frames if step.render_data.turbo.is_emit_in_between_frames(): - tween_frame_start_i = max(step.render_data.indexes.frame.start, - step.render_data.indexes.frame.i - step.render_data.turbo.steps) - return TweenStep.emit_frames_between_index_pair(step, tween_frame_start_i, - step.render_data.indexes.frame.i, - grayscale_tube, overlay_mask_tube) + diff = step.render_data.indexes.frame.i - step.render_data.turbo.steps + tween_frame_start_i = max(step.render_data.indexes.frame.start, diff) + from_i = tween_frame_start_i + to_i = step.render_data.indexes.frame.i + return TweenStep.emit_frames_between_index_pair(step, from_i, to_i, grayscale_tube, overlay_mask_tube) return step @staticmethod - def emit_frames_between_index_pair(step: Step, tween_frame_start_i, frame_i, grayscale_tube, overlay_mask_tube): + def emit_frames_between_index_pair(step: Step, from_i, to_i, grayscale_tube, overlay_mask_tube): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" - tween_range = range(tween_frame_start_i, frame_i) + tween_range = range(from_i, to_i) tween_indexes_list: List[Indexes] = TweenStep.create_indexes(step.render_data.indexes, tween_range) - tween_steps: List[TweenStep] = TweenStep.create_steps(step, tween_indexes_list) + tween_steps, values = TweenStep.create_steps(step, tween_indexes_list) step.render_data.indexes.update_tween_start(step.render_data.turbo) - log_utils.print_tween_frame_from_to_info(step.render_data.turbo.steps, tween_frame_start_i, frame_i) + log_utils.print_tween_frame_from_to_info(step.render_data.turbo.steps, values, from_i, to_i) return TweenStep.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) @staticmethod diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 691d76b73..3ddc75842 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -15,14 +15,18 @@ RESET = "\033[0m" +def print_tween_frame_from_to_info(cadence, tween_values, start_i, end_i): + if start_i > 0: + print("Deforum progress: 100%|") + if end_i > 0: + formatted_values = [f"{val:.2f}" for val in tween_values] + print(f"{ORANGE}Creating in-between: {RESET}{cadence} frames ({start_i}-->{end_i}){formatted_values}") + + def print_animation_frame_info(init): print(f"{CYAN}Animation frame: {RESET}{init.indexes.frame.i}/{init.args.anim_args.max_frames}") -def print_tween_frame_from_to_info(cadence, start_i, end_i): - if end_i > 0: - print(f"{ORANGE}Creating in-between: {RESET}{cadence} frames ({start_i}-->{end_i}).") - def print_tween_frame_info(data, indexes, cadence_flow, tween, if_disable=True): if not if_disable: msg_flow_name = '' if cadence_flow is None else data.args.anim_args.optical_flow_cadence + ' optical flow ' From fb55f562b7c880f195a6e4d9e7cc98330b7169e5 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 19:09:34 +0200 Subject: [PATCH 079/132] Step objects renamed and moved step side effects from creation to execution. --- scripts/deforum_helpers/render.py | 37 +++++++++++-------- .../rendering/data/step/__init__.py | 4 +- .../data/step/{step.py => key_step.py} | 32 ++++++---------- .../rendering/data/step/tween_step.py | 36 +++++++++--------- .../rendering/img_2_img_tubes.py | 24 ++++++------ 5 files changed, 65 insertions(+), 68 deletions(-) rename scripts/deforum_helpers/rendering/data/step/{step.py => key_step.py} (94%) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index ddc4a8c8f..0e6996cce 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -19,8 +19,8 @@ from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData -from .rendering.data.step import Step, TweenStep -from .rendering.util import web_ui_utils +from .rendering.data.step import KeyStep, Tween +from .rendering.util import memory_utils, web_ui_utils from .rendering.util.log_utils import print_animation_frame_info, print_warning_generate_returned_no_image @@ -31,29 +31,34 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - steps = Step.create_all_steps(data) - for step in steps: - # TODO? make Step/TweenStep union and check if step is TweenStep + key_steps = KeyStep.create_all_steps(data) # TODO precalculate and add `tweens: List[Tween]` to KeyStep + for key_step in key_steps: + memory_utils.handle_med_or_low_vram_before_step(data) + print_animation_frame_info(key_step.render_data) + web_ui_utils.update_job(data) + + # TODO emit precalculated `key_step.tweens` if there are any grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - step = TweenStep.maybe_emit_in_between_frames(step, grayscale_tube, overlay_mask_tube) - print_animation_frame_info(step.render_data) + _ = Tween.maybe_emit_in_between_frames(key_step, grayscale_tube, overlay_mask_tube) + + key_step.maybe_write_frame_subtitle() frame_tube = img_2_img_tubes.frame_transformation_tube contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - step.prepare_generation(frame_tube, contrasted_noise_tube) - image = step.do_generation() + key_step.prepare_generation(frame_tube, contrasted_noise_tube) + image = key_step.do_generation() if image is None: print_warning_generate_returned_no_image() break - image = img_2_img_tubes.conditional_frame_transformation_tube(step)(image) - step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(step)(image) + image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) - step.progress_and_save(image) + key_step.progress_and_save(image) state.assign_current_image(image) - step.render_data.args.args.seed = step.next_seed() + key_step.render_data.args.args.seed = key_step.next_seed() - step.update_render_preview() - web_ui_utils.update_status_tracker(step.render_data) - step.render_data.animation_mode.unload_raft_and_depth_model() + key_step.update_render_preview() + web_ui_utils.update_status_tracker(key_step.render_data) + key_step.render_data.animation_mode.unload_raft_and_depth_model() diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py index 10dfb1e84..03d1723cf 100644 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -1,2 +1,2 @@ -from .step import StepData, Step -from .tween_step import TweenStep +from .key_step import KeyStepData, KeyStep +from .tween_step import Tween diff --git a/scripts/deforum_helpers/rendering/data/step/step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py similarity index 94% rename from scripts/deforum_helpers/rendering/data/step/step.py rename to scripts/deforum_helpers/rendering/data/step/key_step.py index d449de4e4..7ac200d5d 100644 --- a/scripts/deforum_helpers/rendering/data/step/step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -7,7 +7,7 @@ from ..render_data import RenderData from ..schedule import Schedule from ..turbo import Turbo -from ...util import memory_utils, opt_utils, web_ui_utils +from ...util import memory_utils, opt_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -17,14 +17,13 @@ from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.call.video_and_audio import call_render_preview -from ...util.log_utils import print_animation_frame_info from ....hybrid_video import image_transform_ransac, image_transform_optical_flow from ....save_images import save_image from ....seed import next_seed @dataclass(init=True, frozen=True, repr=False, eq=False) -class StepData: +class KeyStepData: noise: Any = None strength: Any = None scale: Any = None @@ -49,7 +48,7 @@ def has_strength(self): @staticmethod def create(deform_keys, i): keys = deform_keys - return StepData( + return KeyStepData( keys.noise_schedule_series[i], keys.strength_schedule_series[i], keys.cfg_scale_schedule_series[i], @@ -60,7 +59,7 @@ def create(deform_keys, i): keys.threshold_schedule_series[i], keys.cadence_flow_factor_schedule_series[i], keys.redo_flow_factor_schedule_series[i], - StepData._hybrid_comp_args(keys, i)) + KeyStepData._hybrid_comp_args(keys, i)) @staticmethod def _hybrid_comp_args(keys, i): @@ -74,8 +73,9 @@ def _hybrid_comp_args(keys, i): @dataclass(init=True, frozen=False, repr=False, eq=False) -class Step: - step_data: StepData +class KeyStep: + """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" + step_data: KeyStepData render_data: RenderData schedule: Schedule depth: Any # TODO try to init early, then freeze class @@ -84,25 +84,17 @@ class Step: last_preview_frame: int @staticmethod - def do_start_and_create(data: RenderData): - # Perform the necessary side effects - memory_utils.handle_med_or_low_vram_before_step(data) - web_ui_utils.update_job(data) - # Actual create - step_data = StepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) + def create(data: RenderData): + step_data = KeyStepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) - return Step(step_data, data, schedule, None, None, "", 0) + return KeyStep(step_data, data, schedule, None, None, "", 0) @staticmethod def create_all_steps(data): - """Creates a list of steps for the entire animation.""" - steps = [] + """Creates a list of key steps for the entire animation.""" max_steps = int(data.args.anim_args.max_frames / data.cadence()) - for step_index in range(max_steps): - step = Step.do_start_and_create(data) - step.maybe_write_frame_subtitle() - steps.append(step) + steps = [KeyStep.create(data) for _ in range(max_steps)] assert len(steps) == max_steps return steps diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 67ad5a733..7a5d1cc3e 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -3,18 +3,18 @@ from typing import Any, Iterable, Tuple from ...data.indexes import Indexes, IndexWithStart -from ...data.step import Step +from ...data.step import KeyStep from ...util import image_utils, log_utils, web_ui_utils @dataclass(init=True, frozen=False, repr=False, eq=False) -class TweenStep: +class Tween: """cadence vars""" - indexes: Indexes - last_step: Step + indexes: Indexes # TODO? replace `Indexes` with like separate `index`, `from_key_step_i` and `to_key_step_i` + last_step: KeyStep # TODO Tween shouldn't have a `KeyStep` but key_steps should have tweens. tween: float - cadence_flow: Any - cadence_flow_inc: Any + cadence_flow: Any # late init + cadence_flow_inc: Any # late init @staticmethod def _calculate_tween_from_indices(frame_difference, last_step) -> float: @@ -38,18 +38,18 @@ def _increment(original_indexes, tween_count, from_start): def create_steps_from_values(last_step, values): count = len(values) r = range(count) - indexes_list = [TweenStep._increment(last_step.render_data.indexes.copy(), count, i + 1) for i in r] - return list((TweenStep(indexes_list[i], last_step, values[i], None, None) for i in r)) + indexes_list = [Tween._increment(last_step.render_data.indexes.copy(), count, i + 1) for i in r] + return list((Tween(indexes_list[i], last_step, values[i], None, None) for i in r)) @staticmethod def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[Indexes]: return list(chain.from_iterable([Indexes.create_from_last(base_indexes, i)] for i in frame_range)) @staticmethod - def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['TweenStep'], list[float]]: + def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tween'], list[float]]: if len(tween_indexes_list) > 0: - expected_tween_frames = TweenStep._calculate_expected_tween_frames(len(tween_indexes_list)) - return TweenStep.create_steps_from_values(last_step, expected_tween_frames), expected_tween_frames + expected_tween_frames = Tween._calculate_expected_tween_frames(len(tween_indexes_list)) + return Tween.create_steps_from_values(last_step, expected_tween_frames), expected_tween_frames else: return list(), list() @@ -75,28 +75,28 @@ def handle_synchronous_status_concerns(self, data): web_ui_utils.update_progress_during_cadence(data, self.indexes) @staticmethod - def maybe_emit_in_between_frames(step: Step, grayscale_tube, overlay_mask_tube): + def maybe_emit_in_between_frames(step: KeyStep, grayscale_tube, overlay_mask_tube): # TODO? return the new frames if step.render_data.turbo.is_emit_in_between_frames(): diff = step.render_data.indexes.frame.i - step.render_data.turbo.steps tween_frame_start_i = max(step.render_data.indexes.frame.start, diff) from_i = tween_frame_start_i to_i = step.render_data.indexes.frame.i - return TweenStep.emit_frames_between_index_pair(step, from_i, to_i, grayscale_tube, overlay_mask_tube) + return Tween.emit_frames_between_index_pair(step, from_i, to_i, grayscale_tube, overlay_mask_tube) return step @staticmethod - def emit_frames_between_index_pair(step: Step, from_i, to_i, grayscale_tube, overlay_mask_tube): + def emit_frames_between_index_pair(step: KeyStep, from_i, to_i, grayscale_tube, overlay_mask_tube): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" tween_range = range(from_i, to_i) - tween_indexes_list: List[Indexes] = TweenStep.create_indexes(step.render_data.indexes, tween_range) - tween_steps, values = TweenStep.create_steps(step, tween_indexes_list) + tween_indexes_list: List[Indexes] = Tween.create_indexes(step.render_data.indexes, tween_range) + tween_steps, values = Tween.create_steps(step, tween_indexes_list) step.render_data.indexes.update_tween_start(step.render_data.turbo) log_utils.print_tween_frame_from_to_info(step.render_data.turbo.steps, values, from_i, to_i) - return TweenStep.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) + return Tween.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) @staticmethod - def emit_tween_frames(step: Step, tween_steps, grayscale_tube, overlay_mask_tube): + def emit_tween_frames(step: KeyStep, tween_steps, grayscale_tube, overlay_mask_tube): """Emits a tween frame for each provided tween_step.""" for tween_step in tween_steps: tween_step.handle_synchronous_status_concerns(step.render_data) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 4973a20c8..ed00b63f9 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -5,7 +5,7 @@ from cv2.typing import MatLike from .data.render_data import RenderData -from .data.step.step import Step +from .data.step.key_step import KeyStep from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube from ..masks import do_overlay_mask @@ -27,23 +27,23 @@ ImageTube = Callable[[MatLike], MatLike] -def frame_transformation_tube(data: RenderData, step: Step) -> ImageTube: +def frame_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: # make sure `img` stays the last argument in each call. return tube(lambda img: step.apply_frame_warp_transform(data, img), lambda img: step.do_hybrid_compositing_before_motion(data, img), - lambda img: Step.apply_hybrid_motion_ransac_transform(data, img), - lambda img: Step.apply_hybrid_motion_optical_flow(data, img), + lambda img: KeyStep.apply_hybrid_motion_ransac_transform(data, img), + lambda img: KeyStep.apply_hybrid_motion_optical_flow(data, img), lambda img: step.do_normal_hybrid_compositing_after_motion(data, img), - lambda img: Step.apply_color_matching(data, img), - lambda img: Step.transform_to_grayscale_if_active(data, img)) + lambda img: KeyStep.apply_color_matching(data, img), + lambda img: KeyStep.transform_to_grayscale_if_active(data, img)) -def contrast_transformation_tube(data: RenderData, step: Step) -> ImageTube: +def contrast_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: return tube(lambda img: step.apply_scaling(img), lambda img: step.apply_anti_blur(data, img)) -def noise_transformation_tube(data: RenderData, step: Step) -> ImageTube: +def noise_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: return tube(lambda img: step.apply_frame_noising(data, step, img)) @@ -56,7 +56,7 @@ def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). -def conditional_hybrid_video_after_generation_tube(step: Step) -> ImageTube: +def conditional_hybrid_video_after_generation_tube(step: KeyStep) -> ImageTube: data = step.render_data step_data = step.step_data return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), @@ -77,7 +77,7 @@ def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: lambda: data.indexes.is_first_frame() and data.is_initialize_color_match(data.images.color_match)) -def conditional_color_match_tube(step: Step) -> ImageTube: +def conditional_color_match_tube(step: KeyStep) -> ImageTube: # on strength 0, set color match to generation return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), is_do_process=lambda: step.render_data.is_do_color_match_conversion(step)) @@ -106,14 +106,14 @@ def conditional_force_tween_to_grayscale_tube(data: RenderData) -> ImageTube: # Composite Tubes, made from other Tubes. -def contrasted_noise_transformation_tube(data: RenderData, step: Step) -> ImageTube: +def contrasted_noise_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: """Combines contrast and noise transformation tubes.""" contrast_tube: Tube = contrast_transformation_tube(data, step) noise_tube: Tube = noise_transformation_tube(data, step) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(step: Step, is_tween: bool = False) -> ImageTube: +def conditional_frame_transformation_tube(step: KeyStep, is_tween: bool = False) -> ImageTube: hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(step) extra_tube: Tube = conditional_extra_color_match_tube(step.render_data) gray_tube: Tube = conditional_force_to_grayscale_tube(step.render_data) From fa50ec17e73af7721b3f659494d8c6629bf2d2bf Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 16 Jun 2024 20:57:10 +0200 Subject: [PATCH 080/132] Removed key step reference from tween step instances and fixed new combo tween progress printout. --- scripts/deforum_helpers/render.py | 2 +- .../rendering/data/step/key_step.py | 10 --- .../rendering/data/step/tween_step.py | 89 +++++++++++-------- .../deforum_helpers/rendering/data/turbo.py | 6 +- .../rendering/util/filename_utils.py | 2 +- .../rendering/util/image_utils.py | 17 ++-- .../rendering/util/log_utils.py | 3 +- .../rendering/util/web_ui_utils.py | 21 +++-- 8 files changed, 79 insertions(+), 71 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 0e6996cce..8d9141eb0 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -34,7 +34,6 @@ def run_render_animation(data: RenderData): key_steps = KeyStep.create_all_steps(data) # TODO precalculate and add `tweens: List[Tween]` to KeyStep for key_step in key_steps: memory_utils.handle_med_or_low_vram_before_step(data) - print_animation_frame_info(key_step.render_data) web_ui_utils.update_job(data) # TODO emit precalculated `key_step.tweens` if there are any @@ -42,6 +41,7 @@ def run_render_animation(data: RenderData): overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube _ = Tween.maybe_emit_in_between_frames(key_step, grayscale_tube, overlay_mask_tube) + print_animation_frame_info(key_step.render_data) key_step.maybe_write_frame_subtitle() frame_tube = img_2_img_tubes.frame_transformation_tube diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 7ac200d5d..a609b86f1 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -6,7 +6,6 @@ from ..render_data import RenderData from ..schedule import Schedule -from ..turbo import Turbo from ...util import memory_utils, opt_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate @@ -102,15 +101,6 @@ def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, i has_flow_redo = optical_flow_redo_generation != 'None' return has_flow_redo and images.has_previous() and self.step_data.has_strength() - def update_depth_prediction(self, data: RenderData, turbo: Turbo): - has_depth = data.depth_model is not None - has_next = turbo.next.image is not None - if has_depth and has_next: - image = turbo.next.image - weight = data.args.anim_args.midas_weight - precision = data.args.root.half_precision - self.depth = data.depth_model.predict(image, weight, precision) - def maybe_write_frame_subtitle(self): data = self.render_data if data.turbo.is_first_step_with_subtitles(data): diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 7a5d1cc3e..f2848fe2b 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -2,19 +2,22 @@ from itertools import chain from typing import Any, Iterable, Tuple +from ..turbo import Turbo from ...data.indexes import Indexes, IndexWithStart -from ...data.step import KeyStep -from ...util import image_utils, log_utils, web_ui_utils +from ...data.render_data import RenderData +from ...util import image_utils, log_utils, opt_utils, web_ui_utils +from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle @dataclass(init=True, frozen=False, repr=False, eq=False) class Tween: """cadence vars""" indexes: Indexes # TODO? replace `Indexes` with like separate `index`, `from_key_step_i` and `to_key_step_i` - last_step: KeyStep # TODO Tween shouldn't have a `KeyStep` but key_steps should have tweens. tween: float cadence_flow: Any # late init cadence_flow_inc: Any # late init + depth: Any + depth_prediction: Any @staticmethod def _calculate_tween_from_indices(frame_difference, last_step) -> float: @@ -39,7 +42,7 @@ def create_steps_from_values(last_step, values): count = len(values) r = range(count) indexes_list = [Tween._increment(last_step.render_data.indexes.copy(), count, i + 1) for i in r] - return list((Tween(indexes_list[i], last_step, values[i], None, None) for i in r)) + return list((Tween(indexes_list[i], values[i], None, None, last_step.depth, None) for i in r)) @staticmethod def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[Indexes]: @@ -53,58 +56,74 @@ def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tw else: return list(), list() - def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): + def generate_tween_image(self, data, depth, grayscale_tube, overlay_mask_tube): is_tween = True - warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self.last_step, self) + warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, depth, self.indexes, self) recolored = grayscale_tube(data)(warped) masked = overlay_mask_tube(data, is_tween)(recolored) return masked - def generate(self, data, grayscale_tube, overlay_mask_tube): - return self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) + def generate(self, data, depth, grayscale_tube, overlay_mask_tube): + return self.generate_tween_image(data, depth, grayscale_tube, overlay_mask_tube) - def process(self, init): - init.turbo.advance_optical_flow_cadence_before_animation_warping(init, self) - self.last_step.update_depth_prediction(init, init.turbo) - init.turbo.advance(init, self.indexes.tween.i, self.last_step.depth) - init.turbo.do_hybrid_video_motion(init, self.indexes, init.images) # TODO remove self.indexes or init.indexes + def process(self, data): + data.turbo.advance_optical_flow_cadence_before_animation_warping(data, self) + self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) + data.turbo.advance(data, self.indexes.tween.i, self.depth) + data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # TODO remove self.indexes or init.indexes + + @staticmethod + def calculate_depth_prediction(data, turbo: Turbo): + has_depth = data.depth_model is not None + has_next = turbo.next.image is not None + if has_depth and has_next: + image = turbo.next.image + weight = data.args.anim_args.midas_weight + precision = data.args.root.half_precision + return data.depth_model.predict(image, weight, precision) def handle_synchronous_status_concerns(self, data): - self.last_step.write_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? + self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.tween) web_ui_utils.update_progress_during_cadence(data, self.indexes) + def write_tween_frame_subtitle_if_active(self, data: RenderData): + if opt_utils.is_generate_subtitles(data): + params_to_print = opt_utils.generation_info_for_subtitles(data) + params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) + call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) + @staticmethod - def maybe_emit_in_between_frames(step: KeyStep, grayscale_tube, overlay_mask_tube): + def maybe_emit_in_between_frames(last_step, grayscale_tube, overlay_mask_tube): # TODO? return the new frames - if step.render_data.turbo.is_emit_in_between_frames(): - diff = step.render_data.indexes.frame.i - step.render_data.turbo.steps - tween_frame_start_i = max(step.render_data.indexes.frame.start, diff) + if last_step.render_data.turbo.is_emit_in_between_frames(): + diff = last_step.render_data.indexes.frame.i - last_step.render_data.turbo.steps + tween_frame_start_i = max(last_step.render_data.indexes.frame.start, diff) from_i = tween_frame_start_i - to_i = step.render_data.indexes.frame.i - return Tween.emit_frames_between_index_pair(step, from_i, to_i, grayscale_tube, overlay_mask_tube) - return step + to_i = last_step.render_data.indexes.frame.i + return Tween.emit_frames_between_index_pair(last_step, from_i, to_i, grayscale_tube, overlay_mask_tube) + return last_step @staticmethod - def emit_frames_between_index_pair(step: KeyStep, from_i, to_i, grayscale_tube, overlay_mask_tube): + def emit_frames_between_index_pair(last_step, from_i, to_i, grayscale_tube, overlay_mask_tube): """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" tween_range = range(from_i, to_i) - tween_indexes_list: List[Indexes] = Tween.create_indexes(step.render_data.indexes, tween_range) - tween_steps, values = Tween.create_steps(step, tween_indexes_list) - step.render_data.indexes.update_tween_start(step.render_data.turbo) - log_utils.print_tween_frame_from_to_info(step.render_data.turbo.steps, values, from_i, to_i) - return Tween.emit_tween_frames(step, tween_steps, grayscale_tube, overlay_mask_tube) + tween_indexes_list: List[Indexes] = Tween.create_indexes(last_step.render_data.indexes, tween_range) + tween_steps, values = Tween.create_steps(last_step, tween_indexes_list) + last_step.render_data.indexes.update_tween_start(last_step.render_data.turbo) + log_utils.print_tween_frame_from_to_info(last_step.render_data.turbo.steps, values, from_i, to_i) + return Tween.emit_tween_frames(last_step, tween_steps, grayscale_tube, overlay_mask_tube) @staticmethod - def emit_tween_frames(step: KeyStep, tween_steps, grayscale_tube, overlay_mask_tube): + def emit_tween_frames(last_step, tween_steps, grayscale_tube, overlay_mask_tube): """Emits a tween frame for each provided tween_step.""" for tween_step in tween_steps: - tween_step.handle_synchronous_status_concerns(step.render_data) - tween_step.process(step.render_data) # side effects on turbo and on step - - new_image = tween_step.generate(step.render_data, grayscale_tube, overlay_mask_tube) + tween_step.handle_synchronous_status_concerns(last_step.render_data) + tween_step.process(last_step.render_data) # side effects on turbo and on step + new_image = tween_step.generate(last_step.render_data, last_step.depth, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes - new_image = image_utils.save_and_return_frame(step.render_data, tween_step.indexes, new_image) + i = tween_step.indexes.tween.i + new_image = image_utils.save_and_return_frame(last_step.render_data, i, new_image) # updating reference images to calculate hybrid motions in next iteration - step.render_data.images.previous = new_image - return step + last_step.render_data.images.previous = new_image + return last_step diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index b342324e7..aecacb01f 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -106,16 +106,16 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step self.advance_optical_flow(tween_step) self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow) - def do_optical_flow_cadence_after_animation_warping(self, data, indexes, step, tween_step): + def do_optical_flow_cadence_after_animation_warping(self, data, indexes, depth, tween_step): if tween_step.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) - new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, step.depth) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, depth) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween self.advance_cadence_flow(tween_step) - self.prev.index = self.next.frame_idx = indexes.tween.i + self.prev.index = self.next.frame_idx = indexes.tween.i if indexes is not None else 0 if self.prev.image is not None and tween_step.tween < 1.0: return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween else: diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index b0010c3ed..8ee5ff141 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -23,7 +23,7 @@ def _frame_filename_index(i: int, file_format: FileFormat) -> str: return f"{i:09}.{file_format.value}" -def _frame_filename(data: RenderData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: +def frame_filename(data: RenderData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" return f"{data.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index be1e6fceb..3e1a2f2f0 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -1,24 +1,25 @@ import os import cv2 +from cv2.typing import MatLike -from .filename_utils import tween_frame_name +from . import filename_utils from ..data.render_data import RenderData -def save_cadence_frame(data: RenderData, indexes, image): - filename = tween_frame_name(data, indexes) +def save_cadence_frame(data: RenderData, i: int, image: MatLike): + filename = filename_utils.frame_filename(data, i) save_path: str = os.path.join(data.args.args.outdir, filename) cv2.imwrite(save_path, image) -def save_cadence_frame_and_depth_map_if_active(data: RenderData, indexes, image): - save_cadence_frame(data, indexes, image) +def save_cadence_frame_and_depth_map_if_active(data: RenderData, i, image): + save_cadence_frame(data, i, image) if data.args.anim_args.save_depth_maps: - dm_save_path = os.path.join(data.output_directory, filename_utils.tween_depth_frame(data, indexes)) + dm_save_path = os.path.join(data.output_directory, filename_utils.frame_filename(data, i, True)) data.depth_model.save(dm_save_path, step.depth) -def save_and_return_frame(data: RenderData, indexes, image): - save_cadence_frame_and_depth_map_if_active(data, indexes, image) +def save_and_return_frame(data: RenderData, i, image): + save_cadence_frame_and_depth_map_if_active(data, i, image) return image diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 3ddc75842..0247ed5d7 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -16,8 +16,7 @@ def print_tween_frame_from_to_info(cadence, tween_values, start_i, end_i): - if start_i > 0: - print("Deforum progress: 100%|") + print() # additional newline to skip out of progress bar. if end_i > 0: formatted_values = [f"{val:.2f}" for val in tween_values] print(f"{ORANGE}Creating in-between: {RESET}{cadence} frames ({start_i}-->{end_i}){formatted_values}") diff --git a/scripts/deforum_helpers/rendering/util/web_ui_utils.py b/scripts/deforum_helpers/rendering/util/web_ui_utils.py index d75a5612d..882b0d612 100644 --- a/scripts/deforum_helpers/rendering/util/web_ui_utils.py +++ b/scripts/deforum_helpers/rendering/util/web_ui_utils.py @@ -3,17 +3,16 @@ # noinspection PyUnresolvedReferences from modules.shared import state - WEB_UI_SLEEP_DELAY = 0.1 -def init_job(init): - state.job_count = init.args.anim_args.max_frames +def init_job(data): + state.job_count = data.args.anim_args.max_frames -def update_job(init): - frame = init.indexes.frame.i + 1 - max_frames = init.args.anim_args.max_frames +def update_job(data): + frame = data.indexes.frame.i + 1 + max_frames = data.args.anim_args.max_frames state.job = f"frame {frame}/{max_frames}" state.job_no = frame + 1 if state.skipped: @@ -24,11 +23,11 @@ def update_job(init): print("** RESUMING **") -def update_status_tracker(init): - progress = init.indexes.frame.i / init.args.anim_args.max_frames - JobStatusTracker().update_phase(init.args.root.job_id, phase="GENERATING", progress=progress) +def update_status_tracker(data): + progress = data.indexes.frame.i / data.args.anim_args.max_frames + JobStatusTracker().update_phase(data.args.root.job_id, phase="GENERATING", progress=progress) -def update_progress_during_cadence(init, indexes): - state.job = f"frame {indexes.tween.i + 1}/{init.args.anim_args.max_frames}" +def update_progress_during_cadence(data, indexes): + state.job = f"frame {indexes.tween.i + 1}/{data.args.anim_args.max_frames}" state.job_no = indexes.tween.i + 1 From 4f3b1b613c51282edb93f47ba576f3b29094b622 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 17 Jun 2024 02:48:07 +0200 Subject: [PATCH 081/132] Tween step precalculation (WIP). --- scripts/deforum_helpers/render.py | 69 ++++++++++++++----- .../deforum_helpers/rendering/data/indexes.py | 6 ++ .../rendering/data/step/key_step.py | 22 +++--- .../rendering/data/step/tween_step.py | 56 ++++++++++++--- .../deforum_helpers/rendering/data/turbo.py | 4 +- .../rendering/util/log_utils.py | 9 +++ 6 files changed, 126 insertions(+), 40 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 8d9141eb0..f8360eadf 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -20,7 +20,7 @@ from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import KeyStep, Tween -from .rendering.util import memory_utils, web_ui_utils +from .rendering.util import log_utils, memory_utils, web_ui_utils from .rendering.util.log_utils import print_animation_frame_info, print_warning_generate_returned_no_image @@ -31,33 +31,66 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - key_steps = KeyStep.create_all_steps(data) # TODO precalculate and add `tweens: List[Tween]` to KeyStep + start_index = data.turbo.find_start(data) + max_frames = data.args.anim_args.max_frames + + # create all key steps + key_steps = KeyStep.create_all_steps(data, start_index) + for i, key_step in enumerate(key_steps): + if i == 0: + continue + # reindex key steps # TODO calculate correct index right away. + key_step_index = 1 + start_index + int(i * (max_frames - 1 - start_index) / (len(key_steps) - 1)) + key_step.i = min(key_step_index, max_frames) + # add tweens to key steps + if data.turbo.is_emit_in_between_frames() and key_step.i > 0: + from_i = key_steps[i - 1].i # TODO? +1 + to_i = key_step.i # TODO? -1 + tweens, values = Tween.create_in_between_steps(key_step, data, from_i, to_i) + log_utils.info(f"Creating {len(tweens)} tween steps ({from_i}->{to_i}) for key step {key_step.i}") + for tween in tweens: + tween.indexes.update_tween_index(tween.i() + key_step.i) + key_step.tweens = tweens + key_step.tween_values = values + key_step.render_data.indexes.update_tween_start(data.turbo) + + # process key steps and their tweens for key_step in key_steps: memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - # TODO emit precalculated `key_step.tweens` if there are any - grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube - overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - _ = Tween.maybe_emit_in_between_frames(key_step, grayscale_tube, overlay_mask_tube) + is_step_having_tweens = len(key_step.tweens) > 0 + if is_step_having_tweens: + # print tween frame info + turbo_steps = key_step.render_data.turbo.steps + tween_values = key_step.tween_values + from_i = key_step.tweens[0].i() + to_i = key_step.tweens[-1].i() + log_utils.print_tween_frame_from_to_info(turbo_steps, tween_values, from_i, to_i) + # emit tweens + grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube + overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in key_step.tweens] print_animation_frame_info(key_step.render_data) key_step.maybe_write_frame_subtitle() - frame_tube = img_2_img_tubes.frame_transformation_tube - contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - key_step.prepare_generation(frame_tube, contrasted_noise_tube) - image = key_step.do_generation() - if image is None: - print_warning_generate_returned_no_image() - break + is_not_last_frame = key_step.i < max_frames + if is_not_last_frame: + frame_tube = img_2_img_tubes.frame_transformation_tube + contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube + key_step.prepare_generation(frame_tube, contrasted_noise_tube) + image = key_step.do_generation() + if image is None: + print_warning_generate_returned_no_image() + break - image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) - key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) + image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) - key_step.progress_and_save(image) - state.assign_current_image(image) - key_step.render_data.args.args.seed = key_step.next_seed() + key_step.progress_and_save(image) + state.assign_current_image(image) + key_step.render_data.args.args.seed = key_step.next_seed() key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index 62c786b32..26b1e69df 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -33,6 +33,12 @@ def update_tween_start(self, turbo): tween_start = max(self.frame.start, self.frame.i - turbo.steps) self.tween = IndexWithStart(tween_start, self.tween.i) + def update_tween_index(self, i): + self.tween = IndexWithStart(self.tween.start, i) + + def update_tween_start_index(self, i): + self.tween = IndexWithStart(i, self.tween.start) + def update_frame(self, i: int): self.frame = IndexWithStart(self.frame.start, i) diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index a609b86f1..c3c22a2b7 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -1,12 +1,13 @@ from dataclasses import dataclass -from typing import Any +from typing import Any, List import cv2 import numpy as np +from .tween_step import Tween from ..render_data import RenderData from ..schedule import Schedule -from ...util import memory_utils, opt_utils +from ...util import log_utils, memory_utils, opt_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -74,6 +75,7 @@ def _hybrid_comp_args(keys, i): @dataclass(init=True, frozen=False, repr=False, eq=False) class KeyStep: """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" + i: int step_data: KeyStepData render_data: RenderData schedule: Schedule @@ -81,19 +83,21 @@ class KeyStep: subtitle_params_to_print: Any subtitle_params_string: str last_preview_frame: int + tweens: List[Tween] + tween_values: List[float] @staticmethod def create(data: RenderData): step_data = KeyStepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) - schedule = Schedule.create(data, data.indexes.frame.i, - data.args.anim_args, data.args.args) - return KeyStep(step_data, data, schedule, None, None, "", 0) + schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) + return KeyStep(0, step_data, data, schedule, None, None, "", 0, list(), list()) @staticmethod - def create_all_steps(data): + def create_all_steps(data, start_index): """Creates a list of key steps for the entire animation.""" - max_steps = int(data.args.anim_args.max_frames / data.cadence()) - steps = [KeyStep.create(data) for _ in range(max_steps)] + max_steps = 1 + int((data.args.anim_args.max_frames - start_index) / data.cadence()) + steps = [KeyStep.create(data) for _ in range(0, max_steps)] + log_utils.info(f"Creating {len(steps)} KeySteps.") assert len(steps) == max_steps return steps @@ -115,7 +119,7 @@ def write_frame_subtitle_if_active(self, data: RenderData): call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) def apply_frame_warp_transform(self, data: RenderData, image): - previous, self.depth = call_anim_frame_warp(data, data.indexes.frame.i, image, None) + previous, self.depth = call_anim_frame_warp(data, self.i, image, None) return previous def _do_hybrid_compositing_on_cond(self, data: RenderData, image, condition): diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index f2848fe2b..077f1c267 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -12,13 +12,47 @@ @dataclass(init=True, frozen=False, repr=False, eq=False) class Tween: """cadence vars""" - indexes: Indexes # TODO? replace `Indexes` with like separate `index`, `from_key_step_i` and `to_key_step_i` + indexes: Indexes tween: float cadence_flow: Any # late init cadence_flow_inc: Any # late init depth: Any depth_prediction: Any + def i(self): + return self.indexes.tween.i + + def from_key_step_i(self): + return self.indexes.frame.start + + def to_key_step_i(self): + return self.indexes.frame.i + + def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): + """Emits this tween frame.""" + max_frames = last_step.render_data.args.anim_args.max_frames + if self.i() >= max_frames - 1: + return + + # log_utils.debug(f"Emitting tween frame {self.i()} for step {last_step.i}") + data = last_step.render_data + self.handle_synchronous_status_concerns(data) + self.process(data) + + new_image = self.generate(data, grayscale_tube, overlay_mask_tube) + # TODO pass step and depth instead of data and tween_step.indexes + new_image = image_utils.save_and_return_frame(data, self.i(), new_image) + + # updating reference images to calculate hybrid motions in next iteration + data.images.previous = new_image # FIXME + + @staticmethod + def create_in_between_steps(last_step, data, from_i, to_i): + tween_range = range(from_i, to_i) + tween_indexes_list: List[Indexes] = Tween.create_indexes(data.indexes, tween_range) + tween_steps_and_values = Tween.create_steps(last_step, tween_indexes_list) + return tween_steps_and_values + @staticmethod def _calculate_tween_from_indices(frame_difference, last_step) -> float: return min(0.0, max(1.0, float(last_step) / float(frame_difference))) @@ -56,21 +90,15 @@ def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tw else: return list(), list() - def generate_tween_image(self, data, depth, grayscale_tube, overlay_mask_tube): + def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): is_tween = True - warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, depth, self.indexes, self) + warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) recolored = grayscale_tube(data)(warped) masked = overlay_mask_tube(data, is_tween)(recolored) return masked - def generate(self, data, depth, grayscale_tube, overlay_mask_tube): - return self.generate_tween_image(data, depth, grayscale_tube, overlay_mask_tube) - - def process(self, data): - data.turbo.advance_optical_flow_cadence_before_animation_warping(data, self) - self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) - data.turbo.advance(data, self.indexes.tween.i, self.depth) - data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # TODO remove self.indexes or init.indexes + def generate(self, data, grayscale_tube, overlay_mask_tube): + return self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) @staticmethod def calculate_depth_prediction(data, turbo: Turbo): @@ -82,6 +110,12 @@ def calculate_depth_prediction(data, turbo: Turbo): precision = data.args.root.half_precision return data.depth_model.predict(image, weight, precision) + def process(self, data): + data.turbo.advance_optical_flow_cadence_before_animation_warping(data, self) + self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) + data.turbo.advance(data, self.indexes.tween.i, self.depth) + data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # TODO remove self.indexes or init.indexes + def handle_synchronous_status_concerns(self, data): self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.tween) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index aecacb01f..ad3ed10bb 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -106,11 +106,11 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step self.advance_optical_flow(tween_step) self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow) - def do_optical_flow_cadence_after_animation_warping(self, data, indexes, depth, tween_step): + def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): if tween_step.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) - new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, depth) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, self.depth) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 0247ed5d7..ec9d2d091 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -49,3 +49,12 @@ def print_redo_generation_info(init, n): def print_warning_generate_returned_no_image(): print(f"{YELLOW}Warning: {RESET}Generate returned no image. Skipping to next iteration.") + + +def info(s: str): + print(f"Info: {s}") + + +def debug(s: str): + eye_catcher = "###" + print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") From b161dbb37cc83aeb11d619298345dacf6e248840 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 18 Jun 2024 00:13:32 +0200 Subject: [PATCH 082/132] Tween step precalculation and randomized key frame distribution (proof of concept). --- scripts/deforum_helpers/render.py | 31 +---- .../rendering/data/render_data.py | 2 +- .../rendering/data/step/__init__.py | 2 +- .../rendering/data/step/key_step.py | 116 ++++++++++++++++-- .../rendering/util/call/gen.py | 8 +- 5 files changed, 119 insertions(+), 40 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index f8360eadf..88a69705c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -19,9 +19,8 @@ from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData -from .rendering.data.step import KeyStep, Tween +from .rendering.data.step import KeyIndexDistribution, KeyStep from .rendering.util import log_utils, memory_utils, web_ui_utils -from .rendering.util.log_utils import print_animation_frame_info, print_warning_generate_returned_no_image def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -34,34 +33,14 @@ def run_render_animation(data: RenderData): start_index = data.turbo.find_start(data) max_frames = data.args.anim_args.max_frames - # create all key steps - key_steps = KeyStep.create_all_steps(data, start_index) - for i, key_step in enumerate(key_steps): - if i == 0: - continue - # reindex key steps # TODO calculate correct index right away. - key_step_index = 1 + start_index + int(i * (max_frames - 1 - start_index) / (len(key_steps) - 1)) - key_step.i = min(key_step_index, max_frames) - # add tweens to key steps - if data.turbo.is_emit_in_between_frames() and key_step.i > 0: - from_i = key_steps[i - 1].i # TODO? +1 - to_i = key_step.i # TODO? -1 - tweens, values = Tween.create_in_between_steps(key_step, data, from_i, to_i) - log_utils.info(f"Creating {len(tweens)} tween steps ({from_i}->{to_i}) for key step {key_step.i}") - for tween in tweens: - tween.indexes.update_tween_index(tween.i() + key_step.i) - key_step.tweens = tweens - key_step.tween_values = values - key_step.render_data.indexes.update_tween_start(data.turbo) - - # process key steps and their tweens + key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.UNIFORM_SPACING) for key_step in key_steps: memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) is_step_having_tweens = len(key_step.tweens) > 0 if is_step_having_tweens: - # print tween frame info + # print tween frame info # TODO move to log_utils turbo_steps = key_step.render_data.turbo.steps tween_values = key_step.tween_values from_i = key_step.tweens[0].i() @@ -72,7 +51,7 @@ def run_render_animation(data: RenderData): overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in key_step.tweens] - print_animation_frame_info(key_step.render_data) + log_utils.print_animation_frame_info(key_step.render_data) key_step.maybe_write_frame_subtitle() is_not_last_frame = key_step.i < max_frames @@ -82,7 +61,7 @@ def run_render_animation(data: RenderData): key_step.prepare_generation(frame_tube, contrasted_noise_tube) image = key_step.do_generation() if image is None: - print_warning_generate_returned_no_image() + log_utils.print_warning_generate_returned_no_image() break image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 0e5186bde..880db3e03 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -342,7 +342,7 @@ def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, an @staticmethod def expand_prompts_out_to_per_frame(anim_args, root): - prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) + prompt_series = pd.Series([np.nan for _ in range(anim_args.max_frames)]) for i, prompt in root.animation_prompts.items(): if str(i).isdigit(): prompt_series[int(i)] = prompt diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py index 03d1723cf..cde4676a9 100644 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -1,2 +1,2 @@ -from .key_step import KeyStepData, KeyStep +from .key_step import KeyIndexDistribution, KeyStepData, KeyStep from .tween_step import Tween diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index c3c22a2b7..885f8fe1e 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -1,4 +1,6 @@ +import random from dataclasses import dataclass +from enum import Enum from typing import Any, List import cv2 @@ -72,6 +74,20 @@ def _hybrid_comp_args(keys, i): "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} +class KeyIndexDistribution(Enum): + UNIFORM_SPACING = "Uniform Spacing" + RANDOM_SPACING = "Random Spacing" + RANDOM_PLACEMENT = "Random Placement" + + @property + def name(self): + return self.value + + @staticmethod + def default(): + return KeyIndexDistribution.UNIFORM_SPACING + + @dataclass(init=True, frozen=False, repr=False, eq=False) class KeyStep: """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" @@ -93,13 +109,97 @@ def create(data: RenderData): return KeyStep(0, step_data, data, schedule, None, None, "", 0, list(), list()) @staticmethod - def create_all_steps(data, start_index): + def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIndexDistribution.default()): """Creates a list of key steps for the entire animation.""" - max_steps = 1 + int((data.args.anim_args.max_frames - start_index) / data.cadence()) - steps = [KeyStep.create(data) for _ in range(0, max_steps)] - log_utils.info(f"Creating {len(steps)} KeySteps.") - assert len(steps) == max_steps - return steps + max_frames = data.args.anim_args.max_frames + max_steps = 1 + int((max_frames - start_index) / data.cadence()) + log_utils.debug(f"max_steps {max_steps} max_frames {max_frames}") + key_steps = [KeyStep.create(data) for _ in range(0, max_steps)] + key_steps = KeyStep.recalculate_and_check_tweens(key_steps, start_index, max_frames, max_steps, index_dist) + + # Print message # TODO move to log_utils + tween_count = sum(len(key_step.tweens) for key_step in key_steps) - len(key_steps) + msg_start = f"Created {len(key_steps)} KeySteps with {tween_count} Tween frames." + msg_end = f"Key frame index distribution: '{index_dist.name}'." + log_utils.info(f"{msg_start} {msg_end}") + return key_steps + + @staticmethod + def recalculate_and_check_tweens(key_steps, start_index, max_frames, max_steps, index_distribution): + key_steps = KeyStep._recalculate_and_index_key_steps(key_steps, start_index, max_frames, index_distribution) + key_steps = KeyStep._add_tweens_to_key_steps(key_steps) + assert len(key_steps) == max_steps + + # FIXME seems to generate an unnecessary additional tween for value 1.0 (overwritten by actual frame?) + #log_utils.debug("tween count: " + str(sum(len(key_step.tweens) for key_step in key_steps))) + #expected_total_count = max_frames # Total should be equal to max_frames + #actual_total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) + #log_utils.debug("total count: " + str(actual_total_count)) + #assert actual_total_count == expected_total_count # Check total matches expected + + assert key_steps[0].i == 1 + assert key_steps[-1].i == max_frames + return key_steps + + @staticmethod + def _recalculate_and_index_key_steps(key_steps: List['KeyStep'], start_index, max_frames, index_distribution): + def calculate_uniform_indexes(): + return [1 + start_index + int(n * (max_frames - 1 - start_index) / (len(key_steps) - 1)) + for n in range(len(key_steps))] + + # TODO move logic into enum + if index_distribution == KeyIndexDistribution.UNIFORM_SPACING: + uniform_indexes = calculate_uniform_indexes() + for i, key_step in enumerate(key_steps): + key_step.i = uniform_indexes[i] + elif index_distribution == KeyIndexDistribution.RANDOM_SPACING: + uniform_indexes = calculate_uniform_indexes() + key_steps[0].i = start_index + 1 # Enforce first index + key_steps[-1].i = max_frames # Enforce last index + total_spacing = 0 # Calculate initial average spacing + for i in range(1, len(key_steps)): + total_spacing += key_steps[i].i - key_steps[i - 1].i + average_spacing = total_spacing / (len(key_steps) - 1) # Avoid division by zero + # Noise factor to control randomness (adjust as needed) + noise_factor = 0.5 # Higher value creates more variation + log_utils.debug(f"average_spacing {average_spacing}") + for i, key_step in enumerate(key_steps): + if i == 0 or i == len(key_steps) - 1: + continue # Skip first and last (already set) + base_index = uniform_indexes[i] + noise = random.uniform(-noise_factor, noise_factor) * average_spacing + log_utils.debug(f"base_index {base_index} noise {noise} i {int(base_index + noise)}") + key_step.i = int(base_index + noise) # Apply base index and noise + # Ensure index stays within frame bounds + key_step.i = max(start_index, min(key_step.i, max_frames - 1)) + elif index_distribution == KeyIndexDistribution.RANDOM_PLACEMENT: + key_steps[0].i = start_index + 1 # Enforce first index + key_steps[-1].i = max_frames # Enforce last index + # Randomly distribute indices for remaining keyframes + for i in range(1, len(key_steps) - 1): + key_step = key_steps[i] + key_step.i = random.randint(start_index + 1, max_frames - 2) + else: + raise KeyError(f"KeyIndexDistribution {index_distribution} doesn't exist.") + + is_random = index_distribution in [KeyIndexDistribution.RANDOM_SPACING, KeyIndexDistribution.RANDOM_PLACEMENT] + return key_steps.sort(key=lambda ks: ks.i) if is_random else key_steps + + @staticmethod + def _add_tweens_to_key_steps(key_steps): + for i in range(1, len(key_steps)): # skipping 1st key frame + data = key_steps[i].render_data + if data.turbo.is_emit_in_between_frames(): + from_i = key_steps[i-1].i + to_i = key_steps[i].i + tweens, values = Tween.create_in_between_steps(key_steps[i], data, from_i, to_i) + for tween in tweens: # TODO move to creation + tween.indexes.update_tween_index(tween.i() + key_steps[i].i) + log_utils.info(f"Creating {len(tweens)} tween steps ({from_i}->{to_i}) for key step {key_steps[i].i}") + key_steps[i].tweens = tweens + key_steps[i].tween_values = values + key_steps[i].render_data.indexes.update_tween_start(data.turbo) + return key_steps def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' @@ -177,7 +277,7 @@ def transform_to_grayscale_if_active(data: RenderData, image): @staticmethod def apply_hybrid_motion_ransac_transform(data: RenderData, image): - """hybrid video motion - warps images.previous to match motion, usually to prepare for compositing""" + """hybrid video motion - warps `images.previous` to match motion, usually to prepare for compositing""" motion = data.args.anim_args.hybrid_motion if motion in ['Affine', 'Perspective']: last_i = data.indexes.frame.i - 1 @@ -292,7 +392,7 @@ def do_optical_flow_redo_before_generation(self): data.args.args.seed = generate_random_seed() # set a new random seed print_optical_flow_info(data, optical_flow_redo_generation) # TODO output temp seed? - sample_image = call_generate(data, data.indexes.frame.i, self.schedule) + sample_image = call_generate(data, data.indexes.frame.i) optical_tube = optical_flow_redo_tube(data, optical_flow_redo_generation) transformed_sample_image = optical_tube(sample_image) diff --git a/scripts/deforum_helpers/rendering/util/call/gen.py b/scripts/deforum_helpers/rendering/util/call/gen.py index 54f6a0c91..049a6ac6a 100644 --- a/scripts/deforum_helpers/rendering/util/call/gen.py +++ b/scripts/deforum_helpers/rendering/util/call/gen.py @@ -1,7 +1,7 @@ from ....generate import generate -def call_generate(init, step): - ia = init.args - return generate(ia.args, init.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, - ia.root, init.parseq_adapter, init.indexes.frame.i, sampler_name=step.schedule.sampler_name) +def call_generate(data, step): + ia = data.args + return generate(ia.args, data.animation_keys.deform_keys, ia.anim_args, ia.loop_args, ia.controlnet_args, + ia.root, data.parseq_adapter, data.indexes.frame.i, sampler_name=step.schedule.sampler_name) From 6f7f995b95f6c4385652da5853367c88ad5de315 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 18 Jun 2024 01:03:20 +0200 Subject: [PATCH 083/132] Key step sorting fixed. --- scripts/deforum_helpers/rendering/data/step/key_step.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 885f8fe1e..5368a9099 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -183,7 +183,9 @@ def calculate_uniform_indexes(): raise KeyError(f"KeyIndexDistribution {index_distribution} doesn't exist.") is_random = index_distribution in [KeyIndexDistribution.RANDOM_SPACING, KeyIndexDistribution.RANDOM_PLACEMENT] - return key_steps.sort(key=lambda ks: ks.i) if is_random else key_steps + if is_random: + key_steps.sort(key=lambda ks: ks.i) + return key_steps @staticmethod def _add_tweens_to_key_steps(key_steps): From 952ba4bd3a14b12bf145c04738efb32dfa806d01 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 18 Jun 2024 02:10:57 +0200 Subject: [PATCH 084/132] Cleanup and fixed new info in log. --- scripts/deforum_helpers/render.py | 2 +- .../deforum_helpers/rendering/data/mask.py | 14 ------ .../rendering/data/step/tween_step.py | 45 +------------------ .../rendering/util/call/__init__.py | 6 +-- .../rendering/util/log_utils.py | 15 ++++--- 5 files changed, 13 insertions(+), 69 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 88a69705c..a53c14dc7 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -51,7 +51,7 @@ def run_render_animation(data: RenderData): overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in key_step.tweens] - log_utils.print_animation_frame_info(key_step.render_data) + log_utils.print_animation_frame_info(key_step.i, max_frames) key_step.maybe_write_frame_subtitle() is_not_last_frame = key_step.i < max_frames diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index f095aaccb..0a5c246a3 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -19,20 +19,6 @@ class Mask: def has_mask_image(self): return self.image is not None - @staticmethod - def assign_masks(init, i, is_mask_image, dicts): - # Grab the first frame masks since they wont be provided until next frame - # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set - # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 - key = 'video_mask' - if init.args.anim_args.use_mask_video: - mask = call_get_mask_from_file(init, i, True) - init.args.args.mask_file = mask - init.args.root.noise_mask = mask - put_all(dicts, key, mask) - elif is_mask_image is None and init.is_use_mask: - put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noisc mask - @staticmethod def _create_vals(count, dimensions): return list(map(lambda _: {'everywhere': create_img(dimensions)}, range(count))) diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 077f1c267..aade24d45 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -34,12 +34,11 @@ def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): if self.i() >= max_frames - 1: return - # log_utils.debug(f"Emitting tween frame {self.i()} for step {last_step.i}") data = last_step.render_data self.handle_synchronous_status_concerns(data) self.process(data) - new_image = self.generate(data, grayscale_tube, overlay_mask_tube) + new_image = self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes new_image = image_utils.save_and_return_frame(data, self.i(), new_image) @@ -53,10 +52,6 @@ def create_in_between_steps(last_step, data, from_i, to_i): tween_steps_and_values = Tween.create_steps(last_step, tween_indexes_list) return tween_steps_and_values - @staticmethod - def _calculate_tween_from_indices(frame_difference, last_step) -> float: - return min(0.0, max(1.0, float(last_step) / float(frame_difference))) - @staticmethod def _calculate_expected_tween_frames(num_entries): if num_entries <= 0: @@ -97,9 +92,6 @@ def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): masked = overlay_mask_tube(data, is_tween)(recolored) return masked - def generate(self, data, grayscale_tube, overlay_mask_tube): - return self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) - @staticmethod def calculate_depth_prediction(data, turbo: Turbo): has_depth = data.depth_model is not None @@ -126,38 +118,3 @@ def write_tween_frame_subtitle_if_active(self, data: RenderData): params_to_print = opt_utils.generation_info_for_subtitles(data) params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) - - @staticmethod - def maybe_emit_in_between_frames(last_step, grayscale_tube, overlay_mask_tube): - # TODO? return the new frames - if last_step.render_data.turbo.is_emit_in_between_frames(): - diff = last_step.render_data.indexes.frame.i - last_step.render_data.turbo.steps - tween_frame_start_i = max(last_step.render_data.indexes.frame.start, diff) - from_i = tween_frame_start_i - to_i = last_step.render_data.indexes.frame.i - return Tween.emit_frames_between_index_pair(last_step, from_i, to_i, grayscale_tube, overlay_mask_tube) - return last_step - - @staticmethod - def emit_frames_between_index_pair(last_step, from_i, to_i, grayscale_tube, overlay_mask_tube): - """Emits tween frames (also known as turbo- or cadence-frames) between the provided indices.""" - tween_range = range(from_i, to_i) - tween_indexes_list: List[Indexes] = Tween.create_indexes(last_step.render_data.indexes, tween_range) - tween_steps, values = Tween.create_steps(last_step, tween_indexes_list) - last_step.render_data.indexes.update_tween_start(last_step.render_data.turbo) - log_utils.print_tween_frame_from_to_info(last_step.render_data.turbo.steps, values, from_i, to_i) - return Tween.emit_tween_frames(last_step, tween_steps, grayscale_tube, overlay_mask_tube) - - @staticmethod - def emit_tween_frames(last_step, tween_steps, grayscale_tube, overlay_mask_tube): - """Emits a tween frame for each provided tween_step.""" - for tween_step in tween_steps: - tween_step.handle_synchronous_status_concerns(last_step.render_data) - tween_step.process(last_step.render_data) # side effects on turbo and on step - new_image = tween_step.generate(last_step.render_data, last_step.depth, grayscale_tube, overlay_mask_tube) - # TODO pass step and depth instead of data and tween_step.indexes - i = tween_step.indexes.tween.i - new_image = image_utils.save_and_return_frame(last_step.render_data, i, new_image) - # updating reference images to calculate hybrid motions in next iteration - last_step.render_data.images.previous = new_image - return last_step diff --git a/scripts/deforum_helpers/rendering/util/call/__init__.py b/scripts/deforum_helpers/rendering/util/call/__init__.py index 0f3ca1733..67ebc35d8 100644 --- a/scripts/deforum_helpers/rendering/util/call/__init__.py +++ b/scripts/deforum_helpers/rendering/util/call/__init__.py @@ -12,13 +12,13 @@ often just formatting or passing arguments, and directly call the corresponding method. - **Naming Convention:** - Function names begin with "call_", followed by the name of the actual method to call. - - The `init` object is always passed as the first argument. + - The `data` object is always passed as the first argument. - Frame indices (e.g., `frame_idx`, `twin_frame_idx`) are passed as the second argument "i", when relevant. **Example:** ```python # Example function in this module -def call_some_function(init, i, ...): - return some_module.some_function(init.arg77, init.arg.arg.whatever, i, ...) +def call_some_function(data, i, ...): + return some_module.some_function(data.arg77, data.arg.arg.whatever, i, ...) ``` """ diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index ec9d2d091..6e7dd097f 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -19,11 +19,12 @@ def print_tween_frame_from_to_info(cadence, tween_values, start_i, end_i): print() # additional newline to skip out of progress bar. if end_i > 0: formatted_values = [f"{val:.2f}" for val in tween_values] - print(f"{ORANGE}Creating in-between: {RESET}{cadence} frames ({start_i}-->{end_i}){formatted_values}") + count = end_i - start_i + 1 + print(f"{ORANGE}Creating in-between: {RESET}{count} frames ({start_i}-->{end_i}){formatted_values}") -def print_animation_frame_info(init): - print(f"{CYAN}Animation frame: {RESET}{init.indexes.frame.i}/{init.args.anim_args.max_frames}") +def print_animation_frame_info(i, max_frames): + print(f"{CYAN}Animation frame: {RESET}{i}/{max_frames}") def print_tween_frame_info(data, indexes, cadence_flow, tween, if_disable=True): @@ -37,14 +38,14 @@ def print_init_frame_info(init_frame): print(f"Using video init frame {init_frame}") -def print_optical_flow_info(init, optical_flow_redo_generation): +def print_optical_flow_info(data, optical_flow_redo_generation): msg_start = "Optical flow redo is diffusing and warping using" msg_end = "optical flow before generation." - print(f"{msg_start} {optical_flow_redo_generation} and seed {init.args.args.seed} {msg_end}") + print(f"{msg_start} {optical_flow_redo_generation} and seed {data.args.args.seed} {msg_end}") -def print_redo_generation_info(init, n): - print(f"Redo generation {n + 1} of {int(init.args.anim_args.diffusion_redo)} before final generation") +def print_redo_generation_info(data, n): + print(f"Redo generation {n + 1} of {int(data.args.anim_args.diffusion_redo)} before final generation") def print_warning_generate_returned_no_image(): From 23dea4d3a4255d6f73cf75af11a4f71f7f8b6775 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 19 Jun 2024 18:55:48 +0200 Subject: [PATCH 085/132] Fixed uniform index calculation and some issues with saving frames. --- scripts/deforum_helpers/render.py | 35 ++++----- .../rendering/data/render_data.py | 37 +++++----- .../rendering/data/step/key_step.py | 74 ++++++++++++------- .../rendering/data/step/tween_step.py | 10 +-- .../rendering/util/filename_utils.py | 8 +- .../rendering/util/image_utils.py | 5 +- 6 files changed, 96 insertions(+), 73 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index a53c14dc7..86bfeb69e 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -49,27 +49,28 @@ def run_render_animation(data: RenderData): # emit tweens grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in key_step.tweens] + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) + for tween in key_step.tweens] log_utils.print_animation_frame_info(key_step.i, max_frames) key_step.maybe_write_frame_subtitle() - is_not_last_frame = key_step.i < max_frames - if is_not_last_frame: - frame_tube = img_2_img_tubes.frame_transformation_tube - contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - key_step.prepare_generation(frame_tube, contrasted_noise_tube) - image = key_step.do_generation() - if image is None: - log_utils.print_warning_generate_returned_no_image() - break - - image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) - key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) - - key_step.progress_and_save(image) - state.assign_current_image(image) - key_step.render_data.args.args.seed = key_step.next_seed() + frame_tube = img_2_img_tubes.frame_transformation_tube + contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube + key_step.prepare_generation(frame_tube, contrasted_noise_tube) + + image = key_step.do_generation() # FIXME supress or fix inaccurate info to `print_combined_table` + if image is None: + log_utils.print_warning_generate_returned_no_image() + break + + image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) + + key_step.progress_and_save(image) + state.assign_current_image(image) + + key_step.render_data.args.args.seed = key_step.next_seed() key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 880db3e03..42b7d25d3 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -204,16 +204,14 @@ def update_sample_and_args_for_current_progression_step(self, step, noised_image self.args.root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) self.args.args.strength = max(0.0, min(1.0, step.step_data.strength)) - def update_some_args_for_current_step(self, indexes, step): - i = indexes.frame.i + def update_some_args_for_current_step(self, step, i): keys = self.animation_keys.deform_keys # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints self.args.args.pix2pix_img_cfg_scale = float(keys.pix2pix_img_cfg_scale_series[i]) self.args.args.prompt = self.prompt_series[i] # grab prompt for current frame self.args.args.scale = step.step_data.scale - def update_seed_and_checkpoint_for_current_step(self, indexes): - i = indexes.frame.i + def update_seed_and_checkpoint_for_current_step(self, i): keys = self.animation_keys.deform_keys is_seed_scheduled = self.args.args.seed_behavior == 'schedule' is_seed_managed = self.parseq_adapter.manages_seed() @@ -223,8 +221,7 @@ def update_seed_and_checkpoint_for_current_step(self, indexes): self.args.args.checkpoint = keys.checkpoint_schedule_series[i] \ if self.args.anim_args.enable_checkpoint_scheduling else None - def update_sub_seed_schedule_for_current_step(self, indexes): - i = indexes.frame.i + def update_sub_seed_schedule_for_current_step(self, i): keys = self.animation_keys.deform_keys is_subseed_scheduling_enabled = self.args.anim_args.enable_subseed_scheduling is_seed_managed_by_parseq = self.parseq_adapter.manages_seed() @@ -236,12 +233,11 @@ def update_sub_seed_schedule_for_current_step(self, indexes): self.args.root.subseed_strength = keys.subseed_strength_schedule_series[i] self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. - def prompt_for_current_step(self, indexes): + def prompt_for_current_step(self, i): """returns value to be set back into the prompt""" prompt = self.args.args.prompt max_frames = self.args.anim_args.max_frames seed = self.args.args.seed - i = indexes.frame.i return prepare_prompt(prompt, max_frames, seed, i) def _update_video_input_for_current_frame(self, i, step): @@ -261,8 +257,7 @@ def _update_video_mask_for_current_frame(self, i): self.args.root.noise_mask = new_mask mask.vals['video_mask'] = new_mask - def update_video_data_for_current_frame(self, indexes, step): - i = indexes.frame.i + def update_video_data_for_current_frame(self, i, step): if self.animation_mode.has_video_input: self._update_video_input_for_current_frame(i, step) if self.args.anim_args.use_mask_video: @@ -274,22 +269,24 @@ def update_mask_image(self, step, mask): has_sample = self.args.root.init_sample is not None if has_sample: mask_seq = step.schedule.mask_seq - sample = init.args.root.init_sample + sample = self.args.root.init_sample self.args.args.mask_image = call_compose_mask_with_check(self, mask_seq, mask.vals, sample) else: self.args.args.mask_image = None # we need it only after the first frame anyway - def prepare_generation(self, init, step): + def prepare_generation(self, data, step, i): # TODO move all of this to Step? - self.update_some_args_for_current_step(init.indexes, step) - self.update_seed_and_checkpoint_for_current_step(init.indexes) - self.update_sub_seed_schedule_for_current_step(init.indexes) - self.prompt_for_current_step(init.indexes) - self.update_video_data_for_current_frame(init.indexes, step) - self.update_mask_image(step, init.mask) - self.animation_keys.update(init.indexes.frame.i) + if i > self.args.anim_args.max_frames - 1: + return # FIXME? sus + self.update_some_args_for_current_step(step, i) + self.update_seed_and_checkpoint_for_current_step(i) + self.update_sub_seed_schedule_for_current_step(i) + self.prompt_for_current_step(i) + self.update_video_data_for_current_frame(i, step) + self.update_mask_image(step, data.mask) + self.animation_keys.update(i) opt_utils.setup(self, step.schedule) - memory_utils.handle_vram_if_depth_is_predicted(init) + memory_utils.handle_vram_if_depth_is_predicted(data) @staticmethod def create_output_directory_for_the_batch(directory): diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 5368a9099..98beb9795 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -1,3 +1,4 @@ +import os import random from dataclasses import dataclass from enum import Enum @@ -9,7 +10,7 @@ from .tween_step import Tween from ..render_data import RenderData from ..schedule import Schedule -from ...util import log_utils, memory_utils, opt_utils +from ...util import filename_utils, log_utils, memory_utils, opt_utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -112,30 +113,32 @@ def create(data: RenderData): def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIndexDistribution.default()): """Creates a list of key steps for the entire animation.""" max_frames = data.args.anim_args.max_frames - max_steps = 1 + int((max_frames - start_index) / data.cadence()) - log_utils.debug(f"max_steps {max_steps} max_frames {max_frames}") - key_steps = [KeyStep.create(data) for _ in range(0, max_steps)] - key_steps = KeyStep.recalculate_and_check_tweens(key_steps, start_index, max_frames, max_steps, index_dist) + num_key_steps = 1 + int((max_frames - start_index) / data.cadence()) + key_steps = [KeyStep.create(data) for _ in range(0, num_key_steps)] + key_steps = KeyStep.recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, index_dist) # Print message # TODO move to log_utils - tween_count = sum(len(key_step.tweens) for key_step in key_steps) - len(key_steps) + tween_count = sum(len(ks.tweens) for ks in key_steps) msg_start = f"Created {len(key_steps)} KeySteps with {tween_count} Tween frames." msg_end = f"Key frame index distribution: '{index_dist.name}'." log_utils.info(f"{msg_start} {msg_end}") return key_steps @staticmethod - def recalculate_and_check_tweens(key_steps, start_index, max_frames, max_steps, index_distribution): + def recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, index_distribution): key_steps = KeyStep._recalculate_and_index_key_steps(key_steps, start_index, max_frames, index_distribution) key_steps = KeyStep._add_tweens_to_key_steps(key_steps) - assert len(key_steps) == max_steps + assert len(key_steps) == num_key_steps - # FIXME seems to generate an unnecessary additional tween for value 1.0 (overwritten by actual frame?) - #log_utils.debug("tween count: " + str(sum(len(key_step.tweens) for key_step in key_steps))) - #expected_total_count = max_frames # Total should be equal to max_frames - #actual_total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) - #log_utils.debug("total count: " + str(actual_total_count)) - #assert actual_total_count == expected_total_count # Check total matches expected + # The number of generated tweens depends on index since last key-frame. The last tween has the same + # me index as the key_step it belongs to and is meant to replace the unprocessed original key frame. + total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) + assert total_count == max_frames + len(key_steps) - 1 # every key frame except the 1st has a tween double. + assert key_steps[0].tweens == [] # 1st key step has no tweens + + for i, ks in enumerate(key_steps): + tween_indices = [t.i() for t in ks.tweens] + log_utils.debug(f"Key step {i} at {ks.i}: {len(tween_indices)} tween indices: {tween_indices}") assert key_steps[0].i == 1 assert key_steps[-1].i == max_frames @@ -221,8 +224,10 @@ def write_frame_subtitle_if_active(self, data: RenderData): call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) def apply_frame_warp_transform(self, data: RenderData, image): - previous, self.depth = call_anim_frame_warp(data, self.i, image, None) - return previous + is_not_last_frame = self.i < data.args.anim_args.max_frames + if is_not_last_frame: # TODO? why not + previous, self.depth = call_anim_frame_warp(data, self.i, image, None) + return previous def _do_hybrid_compositing_on_cond(self, data: RenderData, image, condition): i = data.indexes.frame.i @@ -264,7 +269,8 @@ def apply_color_matching(data: RenderData, image): if data.images.color_match is None: # TODO questionable # initialize color_match for next iteration with current image, but don't do anything yet. - data.images.color_match = image.copy() + if image is not None: + data.images.color_match = image.copy() else: return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) return image @@ -320,16 +326,22 @@ def transform_and_update_noised_sample(self, frame_tube, contrasted_noise_tube): if data.images.has_previous(): # skipping 1st iteration transformed_image = frame_tube(data, self)(data.images.previous) # TODO separate - noised_image = contrasted_noise_tube(data, self)(transformed_image) - data.update_sample_and_args_for_current_progression_step(self, noised_image) - return transformed_image + if transformed_image is None: # FIXME? shouldn't really happen + log_utils.debug(f"transformed_image {transformed_image}") + noised_image = contrasted_noise_tube(data, self)(data.images.previous) + data.update_sample_and_args_for_current_progression_step(self, noised_image) + return data.images.previous + else: + noised_image = contrasted_noise_tube(data, self)(transformed_image) + data.update_sample_and_args_for_current_progression_step(self, noised_image) + return transformed_image else: return None def prepare_generation(self, frame_tube, contrasted_noise_tube): self.render_data.images.color_match = self.create_color_match_for_video() self.render_data.images.previous = self.transform_and_update_noised_sample(frame_tube, contrasted_noise_tube) - self.render_data.prepare_generation(self.render_data, self) + self.render_data.prepare_generation(self.render_data, self, self.i) self.maybe_redo_optical_flow() self.maybe_redo_diffusion() @@ -350,6 +362,10 @@ def maybe_redo_diffusion(self): self.do_diffusion_redo() def do_generation(self): + max_frames = self.render_data.args.anim_args.max_frames + if self.render_data.indexes.frame.i >= max_frames: + # TODO? prevents an error elsewhere when generating last frame, but `indexes`.. + self.render_data.indexes.update_frame(max_frames - 1) return call_generate(self.render_data, self) def progress_and_save(self, image): @@ -362,12 +378,20 @@ def _progress_save_and_get_next_index(self, image): opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) if not data.animation_mode.has_video_input: data.images.previous = opencv_image + filename = filename_utils.frame_filename(data, self.i) + + is_overwrite = False # replace the processed tween frame with the original one? (probably not) + if is_overwrite or not os.path.exists(os.path.join(data.args.args.outdir, filename)): + # In many cases, the original images may look more detailed or 'better' than the processed ones, + # but we only save the frames that were processed tough the flows to keep the output consistent. + # However, it may be preferable to use them for the 1st and for the last frame, or as thumbnails. + # TODO perhaps save original frames in a different sub dir? + save_image(image, 'PIL', filename, data.args.args, data.args.video_args, data.args.root) + + self.depth = self.generate_and_save_depth_map_if_active() if data.turbo.has_steps(): return data.indexes.frame.i + data.turbo.progress_step(data.indexes, opencv_image) else: - filename = filename_utils.frame(data, data.indexes) - save_image(image, 'PIL', filename, data.args.args, data.args.video_args, data.args.root) - self.depth = generate_depth_maps_if_active(data) return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. def next_seed(self): @@ -376,7 +400,7 @@ def next_seed(self): def update_render_preview(self): self.last_preview_frame = call_render_preview(self.render_data, self.last_preview_frame) - def generate_depth_maps_if_active(self): + def generate_and_save_depth_map_if_active(self): data = self.render_data # TODO move all depth related stuff to new class. if data.args.anim_args.save_depth_maps: diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index aade24d45..a0d854ed4 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -31,12 +31,12 @@ def to_key_step_i(self): def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): """Emits this tween frame.""" max_frames = last_step.render_data.args.anim_args.max_frames - if self.i() >= max_frames - 1: - return + if self.i() >= max_frames: + return # skipping tween emission on the last frame data = last_step.render_data self.handle_synchronous_status_concerns(data) - self.process(data) + self.process(last_step, data) new_image = self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes @@ -102,11 +102,11 @@ def calculate_depth_prediction(data, turbo: Turbo): precision = data.args.root.half_precision return data.depth_model.predict(image, weight, precision) - def process(self, data): + def process(self, last_step, data): data.turbo.advance_optical_flow_cadence_before_animation_warping(data, self) self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) data.turbo.advance(data, self.indexes.tween.i, self.depth) - data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # TODO remove self.indexes or init.indexes + data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # FIXME? remove self.indexes or init.indexes def handle_synchronous_status_concerns(self, data): self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 8ee5ff141..58dd619f3 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -29,19 +29,19 @@ def frame_filename(data: RenderData, i: int, is_depth=False, file_format=FileFor def frame(data: RenderData, indexes: Indexes) -> str: - return _frame_filename(data, indexes.frame.i) + return frame_filename(data, indexes.frame.i) def depth_frame(data: RenderData, indexes: Indexes) -> str: - return _frame_filename(data, indexes.frame.i, True) + return frame_filename(data, indexes.frame.i, True) def tween_frame_name(data: RenderData, indexes: Indexes) -> str: - return _frame_filename(data, indexes.tween.i) + return frame_filename(data, indexes.tween.i) def tween_depth_frame(data: RenderData, indexes: Indexes) -> str: - return _frame_filename(data, indexes.tween.i, True) + return frame_filename(data, indexes.tween.i, True) def preview_video_image_path(data: RenderData, indexes: Indexes) -> Path: diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index 3e1a2f2f0..ca550a325 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -7,10 +7,11 @@ from ..data.render_data import RenderData -def save_cadence_frame(data: RenderData, i: int, image: MatLike): +def save_cadence_frame(data: RenderData, i: int, image: MatLike, is_overwrite: bool = True): filename = filename_utils.frame_filename(data, i) save_path: str = os.path.join(data.args.args.outdir, filename) - cv2.imwrite(save_path, image) + if is_overwrite or not os.path.exists(save_path): + cv2.imwrite(save_path, image) def save_cadence_frame_and_depth_map_if_active(data: RenderData, i, image): From 3dce017efcc07139d0a3a54c0db90aab0f30ff9e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 19 Jun 2024 19:30:11 +0200 Subject: [PATCH 086/132] Table print to CLI suppressed, because it's not compatible with variable cadence. --- scripts/deforum_helpers/render.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 86bfeb69e..bf7b35f22 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -17,6 +17,7 @@ # noinspection PyUnresolvedReferences from modules.shared import opts, state +from . import generate from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import KeyIndexDistribution, KeyStep @@ -24,8 +25,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): + original_print_combined_table = suppress_table_printing() render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) run_render_animation(render_data) + reactivate_table_printing(original_print_combined_table) def run_render_animation(data: RenderData): @@ -75,3 +78,18 @@ def run_render_animation(data: RenderData): key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) key_step.render_data.animation_mode.unload_raft_and_depth_model() + + +def suppress_table_printing(): + # The combined table that is normally printed to the command line is suppressed, + # because it's not compatible with variable keyframe cadence. + def do_nothing(*args): + pass + + original_print_combined_table = generate.print_combined_table + generate.print_combined_table = do_nothing # Monkey patch with do_nothing + return original_print_combined_table + + +def reactivate_table_printing(original_print_combined_table): + generate.print_combined_table = original_print_combined_table From 2abe7371b410b0cef797ca1b9b224528dcf6fabf Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 19 Jun 2024 20:00:37 +0200 Subject: [PATCH 087/132] Cleanup. --- scripts/deforum_helpers/render.py | 31 +++---------------- .../deforum_helpers/rendering/data/images.py | 2 +- .../deforum_helpers/rendering/data/turbo.py | 16 +++++----- .../rendering/util/call/mask.py | 3 +- .../rendering/util/log_utils.py | 22 ++++++++++++- .../rendering/util/opt_utils.py | 1 - 6 files changed, 37 insertions(+), 38 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index bf7b35f22..1134cbd04 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -13,11 +13,8 @@ # along with this program. If not, see . # Contact the authors: https://deforum.github.io/ - -# noinspection PyUnresolvedReferences from modules.shared import opts, state -from . import generate from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import KeyIndexDistribution, KeyStep @@ -25,10 +22,10 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - original_print_combined_table = suppress_table_printing() + original_print_combined_table = log_utils.suppress_table_printing() render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) run_render_animation(render_data) - reactivate_table_printing(original_print_combined_table) + log_utils.reactivate_table_printing(original_print_combined_table) def run_render_animation(data: RenderData): @@ -43,12 +40,7 @@ def run_render_animation(data: RenderData): is_step_having_tweens = len(key_step.tweens) > 0 if is_step_having_tweens: - # print tween frame info # TODO move to log_utils - turbo_steps = key_step.render_data.turbo.steps - tween_values = key_step.tween_values - from_i = key_step.tweens[0].i() - to_i = key_step.tweens[-1].i() - log_utils.print_tween_frame_from_to_info(turbo_steps, tween_values, from_i, to_i) + log_utils.print_tween_frame_from_to_info(key_step) # emit tweens grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube @@ -62,7 +54,7 @@ def run_render_animation(data: RenderData): contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube key_step.prepare_generation(frame_tube, contrasted_noise_tube) - image = key_step.do_generation() # FIXME supress or fix inaccurate info to `print_combined_table` + image = key_step.do_generation() if image is None: log_utils.print_warning_generate_returned_no_image() break @@ -78,18 +70,3 @@ def run_render_animation(data: RenderData): key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) key_step.render_data.animation_mode.unload_raft_and_depth_model() - - -def suppress_table_printing(): - # The combined table that is normally printed to the command line is suppressed, - # because it's not compatible with variable keyframe cadence. - def do_nothing(*args): - pass - - original_print_combined_table = generate.print_combined_table - generate.print_combined_table = do_nothing # Monkey patch with do_nothing - return original_print_combined_table - - -def reactivate_table_printing(original_print_combined_table): - generate.print_combined_table = original_print_combined_table diff --git a/scripts/deforum_helpers/rendering/data/images.py b/scripts/deforum_helpers/rendering/data/images.py index 4471b5b3f..9b7e76e00 100644 --- a/scripts/deforum_helpers/rendering/data/images.py +++ b/scripts/deforum_helpers/rendering/data/images.py @@ -6,7 +6,7 @@ @dataclass(init=True, frozen=False, repr=False, eq=True) class Images: - previous: MatLike = None + previous: MatLike | None = None color_match: MatLike = None def has_previous(self): diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index ad3ed10bb..e9d15af0b 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -10,7 +10,7 @@ @dataclass(init=True, frozen=False, repr=False, eq=True) class ImageFrame: - image: MatLike + image: MatLike | None index: int @@ -40,7 +40,7 @@ def _is_do_motion(motions): return indexes.tween.i > 0 and motion in motions if _is_do_motion(['Affine', 'Perspective']): - self.advance_hybrid_motion_ransac_trasform(data, indexes, reference_images) + Turbo.advance_hybrid_motion_ransac_transform(data, indexes, reference_images) if _is_do_motion(['Optical Flow']): self.advance_hybrid_motion_optical_tween_flow(data, indexes, reference_images, step) @@ -76,7 +76,7 @@ def advance_cadence_flow(self, tween_step): self.next.image = image_transform_optical_flow(self.next.image, inc, ff) # TODO? move to RenderData - def advance_ransac_trasform(self, data, matrix): + def advance_ransac_transform(self, data, matrix): i = indexes.tween.i motion = data.args.anim_args.hybrid_motion if self.is_advance_prev(i): @@ -85,13 +85,14 @@ def advance_ransac_trasform(self, data, matrix): self.next.image = image_transform_ransac(self.next.image, matrix, motion) # TODO? move to RenderData - def advance_hybrid_motion_ransac_trasform(self, data, indexes, reference_images): + @staticmethod + def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): if data.args.anim_args.hybrid_motion_use_prev_img: matrix = call_get_matrix_for_hybrid_motion_prev(data, indexes.tween.i - 1, reference_images.previous) - turbo.advance_ransac_trasform(data, matrix) + turbo.advance_ransac_transform(data, matrix) else: matrix = call_get_matrix_for_hybrid_motion(data, indexes.tween.i - 1) - turbo.advance_ransac_trasform(data, matrix) + turbo.advance_ransac_transform(data, matrix) def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): @@ -104,7 +105,8 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step tween_step.cadence_flow = (flow / 2) self.next.image = advance_optical_flow self.advance_optical_flow(tween_step) - self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow) + flow_factor = 1 + self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): if tween_step.cadence_flow is not None: diff --git a/scripts/deforum_helpers/rendering/util/call/mask.py b/scripts/deforum_helpers/rendering/util/call/mask.py index b31e51223..856fcb964 100644 --- a/scripts/deforum_helpers/rendering/util/call/mask.py +++ b/scripts/deforum_helpers/rendering/util/call/mask.py @@ -10,4 +10,5 @@ def call_compose_mask_with_check(init, mask_seq, val_masks, image): def call_unsharp_mask(init, step, image, mask): kernel_size = (step.step_data.kernel, step.step_data.kernel) mask_image = mask.image if init.args.args.use_mask else None - return unsharp_mask(image, kernel_size, step.step_data.sigma, step.step_data.amount, step.step_data.threshold, mask_image) + return unsharp_mask(image, kernel_size, step.step_data.sigma, step.step_data.amount, + step.step_data.threshold, mask_image) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 6e7dd097f..0ab9af501 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,3 +1,5 @@ +from ... import generate + RED = "\033[31m" ORANGE = "\033[38;5;208m" YELLOW = "\033[33m" @@ -15,7 +17,10 @@ RESET = "\033[0m" -def print_tween_frame_from_to_info(cadence, tween_values, start_i, end_i): +def print_tween_frame_from_to_info(key_step): + tween_values = key_step.tween_values + start_i = key_step.tweens[0].i() + end_i = key_step.tweens[-1].i() print() # additional newline to skip out of progress bar. if end_i > 0: formatted_values = [f"{val:.2f}" for val in tween_values] @@ -59,3 +64,18 @@ def info(s: str): def debug(s: str): eye_catcher = "###" print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") + + +def suppress_table_printing(): + # The combined table that is normally printed to the command line is suppressed, + # because it's not compatible with variable keyframe cadence. + def do_nothing(*args): + pass + + original_print_combined_table = generate.print_combined_table + generate.print_combined_table = do_nothing # Monkey patch with do_nothing + return original_print_combined_table + + +def reactivate_table_printing(original_print_combined_table): + generate.print_combined_table = original_print_combined_table diff --git a/scripts/deforum_helpers/rendering/util/opt_utils.py b/scripts/deforum_helpers/rendering/util/opt_utils.py index 30732fb76..a93175eb2 100644 --- a/scripts/deforum_helpers/rendering/util/opt_utils.py +++ b/scripts/deforum_helpers/rendering/util/opt_utils.py @@ -18,4 +18,3 @@ def generation_info_for_subtitles(render_data): def is_generate_subtitles(render_data): return render_data.args.opts.data.get("deforum_save_gen_info_as_srt") - From 77b5f19b3fb5f64bbe1e4f6b4c4be349ebf43ec7 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 19 Jun 2024 20:50:51 +0200 Subject: [PATCH 088/132] New annotation to suppress table printing. --- scripts/deforum_helpers/render.py | 10 +++++----- .../deforum_helpers/rendering/util/log_utils.py | 14 ++++++++++++-- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 1134cbd04..14c13904c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -13,6 +13,8 @@ # along with this program. If not, see . # Contact the authors: https://deforum.github.io/ + +# noinspection PyUnresolvedReferences from modules.shared import opts, state from .rendering import img_2_img_tubes @@ -22,12 +24,11 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - original_print_combined_table = log_utils.suppress_table_printing() render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) run_render_animation(render_data) - log_utils.reactivate_table_printing(original_print_combined_table) +@log_utils.with_suppressed_table_printing def run_render_animation(data: RenderData): web_ui_utils.init_job(data) start_index = data.turbo.find_start(data) @@ -38,10 +39,9 @@ def run_render_animation(data: RenderData): memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - is_step_having_tweens = len(key_step.tweens) > 0 - if is_step_having_tweens: + is_step_with_tweens = len(key_step.tweens) > 0 + if is_step_with_tweens: # emit tweens log_utils.print_tween_frame_from_to_info(key_step) - # emit tweens grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 0ab9af501..85a709bee 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -66,7 +66,7 @@ def debug(s: str): print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") -def suppress_table_printing(): +def _suppress_table_printing(): # The combined table that is normally printed to the command line is suppressed, # because it's not compatible with variable keyframe cadence. def do_nothing(*args): @@ -77,5 +77,15 @@ def do_nothing(*args): return original_print_combined_table -def reactivate_table_printing(original_print_combined_table): +def _reactivate_table_printing(original_print_combined_table): generate.print_combined_table = original_print_combined_table + + +def with_suppressed_table_printing(func): + def wrapper(*args, **kwargs): + original_print_combined_table = _suppress_table_printing() + try: + return func(*args, **kwargs) + finally: + _reactivate_table_printing(original_print_combined_table) + return wrapper From 30f57453a823bf218408e9b9cac6f18c75b2fca6 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 19 Jun 2024 22:08:36 +0200 Subject: [PATCH 089/132] Moved index distribution logic to new module. --- .../rendering/data/step/__init__.py | 3 +- .../data/step/key_index_distribution.py | 56 +++++++++++++++ .../rendering/data/step/key_step.py | 69 ++----------------- 3 files changed, 63 insertions(+), 65 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/step/key_index_distribution.py diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py index cde4676a9..e55d647fd 100644 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ b/scripts/deforum_helpers/rendering/data/step/__init__.py @@ -1,2 +1,3 @@ -from .key_step import KeyIndexDistribution, KeyStepData, KeyStep +from .key_index_distribution import KeyIndexDistribution +from .key_step import KeyStepData, KeyStep from .tween_step import Tween diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py new file mode 100644 index 000000000..5aa3df6d0 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -0,0 +1,56 @@ +import random +from enum import Enum + + +class KeyIndexDistribution(Enum): + UNIFORM_SPACING = "Uniform Spacing" + RANDOM_SPACING = "Random Spacing" + RANDOM_PLACEMENT = "Random Placement" + + @property + def name(self): + return self.value + + @staticmethod + def default(): + return KeyIndexDistribution.UNIFORM_SPACING + + def calculate(self, start_index, max_frames, num_key_steps): + if self == KeyIndexDistribution.UNIFORM_SPACING: + return self._uniform_indexes(start_index, max_frames, num_key_steps) + elif self == KeyIndexDistribution.RANDOM_SPACING: + return self._random_spacing_indexes(start_index, max_frames, num_key_steps) + elif self == KeyIndexDistribution.RANDOM_PLACEMENT: + return self._random_placement_indexes(start_index, max_frames, num_key_steps) + else: + raise ValueError(f"Invalid KeyIndexDistribution: {self}") + + @staticmethod + def _uniform_indexes(start_index, max_frames, num_key_steps): + return [1 + start_index + int(n * (max_frames - 1 - start_index) / (num_key_steps - 1)) + for n in range(num_key_steps)] + + @staticmethod + def _random_spacing_indexes(start_index, max_frames, num_key_steps): + uniform_indexes = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + indexes = [start_index + 1, max_frames] # Enforce first and last indices + total_spacing = max_frames - start_index - 1 # Calculate initial total spacing + noise_factor = 0.5 # Higher value creates more variation + for i in range(1, num_key_steps - 1): + base_index = uniform_indexes[i] + noise = random.uniform(-noise_factor, noise_factor) * (total_spacing / (num_key_steps - 1)) + index = int(base_index + noise) + index = max(start_index + 1, min(index, max_frames - 1)) + indexes.append(index) + total_spacing -= index - indexes[i - 1] + indexes.sort(key=lambda key_index: key_index) + return indexes + + @staticmethod + def _random_placement_indexes(start_index, max_frames, num_key_steps): + indexes = [start_index + 1, max_frames] # Enforce first and last indices + for _ in range(1, num_key_steps - 1): + index = random.randint(start_index + 1, max_frames - 1) + indexes.append(index) + indexes.sort(key=lambda i: i) + return indexes diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 98beb9795..4a1b138b3 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -1,12 +1,11 @@ import os -import random from dataclasses import dataclass -from enum import Enum from typing import Any, List import cv2 import numpy as np +from . import KeyIndexDistribution from .tween_step import Tween from ..render_data import RenderData from ..schedule import Schedule @@ -75,20 +74,6 @@ def _hybrid_comp_args(keys, i): "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} -class KeyIndexDistribution(Enum): - UNIFORM_SPACING = "Uniform Spacing" - RANDOM_SPACING = "Random Spacing" - RANDOM_PLACEMENT = "Random Placement" - - @property - def name(self): - return self.value - - @staticmethod - def default(): - return KeyIndexDistribution.UNIFORM_SPACING - - @dataclass(init=True, frozen=False, repr=False, eq=False) class KeyStep: """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" @@ -126,7 +111,9 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn @staticmethod def recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, index_distribution): - key_steps = KeyStep._recalculate_and_index_key_steps(key_steps, start_index, max_frames, index_distribution) + key_indices = index_distribution.calculate(start_index, max_frames, num_key_steps) + for i, key_step in enumerate(key_indices): + key_steps[i].i = key_indices[i] key_steps = KeyStep._add_tweens_to_key_steps(key_steps) assert len(key_steps) == num_key_steps @@ -144,58 +131,12 @@ def recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_ste assert key_steps[-1].i == max_frames return key_steps - @staticmethod - def _recalculate_and_index_key_steps(key_steps: List['KeyStep'], start_index, max_frames, index_distribution): - def calculate_uniform_indexes(): - return [1 + start_index + int(n * (max_frames - 1 - start_index) / (len(key_steps) - 1)) - for n in range(len(key_steps))] - - # TODO move logic into enum - if index_distribution == KeyIndexDistribution.UNIFORM_SPACING: - uniform_indexes = calculate_uniform_indexes() - for i, key_step in enumerate(key_steps): - key_step.i = uniform_indexes[i] - elif index_distribution == KeyIndexDistribution.RANDOM_SPACING: - uniform_indexes = calculate_uniform_indexes() - key_steps[0].i = start_index + 1 # Enforce first index - key_steps[-1].i = max_frames # Enforce last index - total_spacing = 0 # Calculate initial average spacing - for i in range(1, len(key_steps)): - total_spacing += key_steps[i].i - key_steps[i - 1].i - average_spacing = total_spacing / (len(key_steps) - 1) # Avoid division by zero - # Noise factor to control randomness (adjust as needed) - noise_factor = 0.5 # Higher value creates more variation - log_utils.debug(f"average_spacing {average_spacing}") - for i, key_step in enumerate(key_steps): - if i == 0 or i == len(key_steps) - 1: - continue # Skip first and last (already set) - base_index = uniform_indexes[i] - noise = random.uniform(-noise_factor, noise_factor) * average_spacing - log_utils.debug(f"base_index {base_index} noise {noise} i {int(base_index + noise)}") - key_step.i = int(base_index + noise) # Apply base index and noise - # Ensure index stays within frame bounds - key_step.i = max(start_index, min(key_step.i, max_frames - 1)) - elif index_distribution == KeyIndexDistribution.RANDOM_PLACEMENT: - key_steps[0].i = start_index + 1 # Enforce first index - key_steps[-1].i = max_frames # Enforce last index - # Randomly distribute indices for remaining keyframes - for i in range(1, len(key_steps) - 1): - key_step = key_steps[i] - key_step.i = random.randint(start_index + 1, max_frames - 2) - else: - raise KeyError(f"KeyIndexDistribution {index_distribution} doesn't exist.") - - is_random = index_distribution in [KeyIndexDistribution.RANDOM_SPACING, KeyIndexDistribution.RANDOM_PLACEMENT] - if is_random: - key_steps.sort(key=lambda ks: ks.i) - return key_steps - @staticmethod def _add_tweens_to_key_steps(key_steps): for i in range(1, len(key_steps)): # skipping 1st key frame data = key_steps[i].render_data if data.turbo.is_emit_in_between_frames(): - from_i = key_steps[i-1].i + from_i = key_steps[i - 1].i to_i = key_steps[i].i tweens, values = Tween.create_in_between_steps(key_steps[i], data, from_i, to_i) for tween in tweens: # TODO move to creation From 05c5d9bf09d6307edbcacf7ea81f37f8c58b8ccb Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Thu, 20 Jun 2024 02:45:39 +0200 Subject: [PATCH 090/132] Progress bar for tween frames. --- scripts/deforum_helpers/render.py | 8 ++- .../rendering/util/log_utils.py | 55 ++++++++++--------- 2 files changed, 33 insertions(+), 30 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 14c13904c..954351a56 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -15,7 +15,8 @@ # Contact the authors: https://deforum.github.io/ # noinspection PyUnresolvedReferences -from modules.shared import opts, state +from modules.shared import cmd_opts, opts, progress_print_out, state +from tqdm import tqdm from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData @@ -44,8 +45,9 @@ def run_render_animation(data: RenderData): log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) - for tween in key_step.tweens] + tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tq] log_utils.print_animation_frame_info(key_step.i, max_frames) key_step.maybe_write_frame_subtitle() diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 85a709bee..cccc72d0d 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,5 +1,6 @@ from ... import generate +COLOUR_RGB = '\x1b[38;2;%d;%d;%dm' RED = "\033[31m" ORANGE = "\033[38;5;208m" YELLOW = "\033[33m" @@ -14,26 +15,27 @@ BOLD = "\033[1m" UNDERLINE = "\033[4m" -RESET = "\033[0m" +RESET = "\x1b[0m" -def print_tween_frame_from_to_info(key_step): - tween_values = key_step.tween_values - start_i = key_step.tweens[0].i() - end_i = key_step.tweens[-1].i() - print() # additional newline to skip out of progress bar. - if end_i > 0: - formatted_values = [f"{val:.2f}" for val in tween_values] - count = end_i - start_i + 1 - print(f"{ORANGE}Creating in-between: {RESET}{count} frames ({start_i}-->{end_i}){formatted_values}") +def print_tween_frame_from_to_info(key_step, is_disabled=True): + if not is_disabled: # replaced with prog bar, but value info print may be useful + tween_values = key_step.tween_values + start_i = key_step.tweens[0].i() + end_i = key_step.tweens[-1].i() + if end_i > 0: + formatted_values = [f"{val:.2f}" for val in tween_values] + count = end_i - start_i + 1 + print(f"{ORANGE}Creating in-between: {RESET}{count} frames ({start_i}-->{end_i}){formatted_values}") def print_animation_frame_info(i, max_frames): + print() # skipping out of the progress bar. print(f"{CYAN}Animation frame: {RESET}{i}/{max_frames}") -def print_tween_frame_info(data, indexes, cadence_flow, tween, if_disable=True): - if not if_disable: +def print_tween_frame_info(data, indexes, cadence_flow, tween, is_disabled=True): + if not is_disabled: # disabled because it's spamming the cli on high cadence settings. msg_flow_name = '' if cadence_flow is None else data.args.anim_args.optical_flow_cadence + ' optical flow ' msg_frame_info = f"cadence frame: {indexes.tween.i}; tween: {tween:0.2f};" print(f"Creating in-between {msg_flow_name}{msg_frame_info}") @@ -66,26 +68,25 @@ def debug(s: str): print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") -def _suppress_table_printing(): - # The combined table that is normally printed to the command line is suppressed, - # because it's not compatible with variable keyframe cadence. - def do_nothing(*args): - pass - - original_print_combined_table = generate.print_combined_table - generate.print_combined_table = do_nothing # Monkey patch with do_nothing - return original_print_combined_table - +def with_suppressed_table_printing(func): + def _suppress_table_printing(): + # The combined table that is normally printed to the command line is suppressed, + # because it's not compatible with variable keyframe cadence. + def do_nothing(*args): + pass -def _reactivate_table_printing(original_print_combined_table): - generate.print_combined_table = original_print_combined_table + original_print_combined_table = generate.print_combined_table + generate.print_combined_table = do_nothing # Monkey patch with do_nothing + return original_print_combined_table + def _reactivate_table_printing(original_print_combined_table): + generate.print_combined_table = original_print_combined_table -def with_suppressed_table_printing(func): def wrapper(*args, **kwargs): - original_print_combined_table = _suppress_table_printing() + original = _suppress_table_printing() try: return func(*args, **kwargs) finally: - _reactivate_table_printing(original_print_combined_table) + _reactivate_table_printing(original) + return wrapper From 7cf9b728ecd2090ad0b33102c21df06135292181 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 22 Jun 2024 10:25:30 +0200 Subject: [PATCH 091/132] Animation key regeneration. --- .../rendering/data/anim/animation_keys.py | 13 +++---------- .../deforum_helpers/rendering/data/render_data.py | 4 ++-- 2 files changed, 5 insertions(+), 12 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index 5d585240e..8697218e6 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -3,20 +3,13 @@ from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys -@dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=False) class AnimationKeys: deform_keys: DeformAnimKeys looper_keys: LooperAnimKeys - def update(self, i: int): - keys = self.looper_keys - keys.use_looper = keys.use_looper - keys.imagesToKeyframe = keys.imagesToKeyframe - keys.imageStrength = keys.image_strength_schedule_series[i] - keys.blendFactorMax = keys.blendFactorMax_series[i] - keys.blendFactorSlope = keys.blendFactorSlope_series[i] - keys.tweeningFramesSchedule = keys.tweening_frames_schedule_series[i] - keys.colorCorrectionFactor = keys.color_correction_factor_series[i] + def update_looper(self, loop_args, anim_args, seed): + self.looper_keys = LooperAnimKeys(loop_args, anim_args, seed) @staticmethod def _choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 42b7d25d3..64360351f 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -38,7 +38,7 @@ class RenderInitArgs: root: RootArgs = None -@dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=False) class RenderData: """The purpose of this class is to group and control all data used in render_animation""" images: Images | None # TODO rename to reference_images? @@ -284,7 +284,7 @@ def prepare_generation(self, data, step, i): self.prompt_for_current_step(i) self.update_video_data_for_current_frame(i, step) self.update_mask_image(step, data.mask) - self.animation_keys.update(i) + self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) opt_utils.setup(self, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(data) From e86325167f2d82b23655168fa69871d7539210b3 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 22 Jun 2024 12:21:57 +0200 Subject: [PATCH 092/132] Parseq key frame index distribution. --- scripts/deforum_helpers/render.py | 2 +- .../rendering/data/render_data.py | 2 +- .../data/step/key_index_distribution.py | 28 +++++++++++++++---- .../rendering/data/step/key_step.py | 9 ++++-- 4 files changed, 31 insertions(+), 10 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 954351a56..b072fa891 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -35,7 +35,7 @@ def run_render_animation(data: RenderData): start_index = data.turbo.find_start(data) max_frames = data.args.anim_args.max_frames - key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.UNIFORM_SPACING) + key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.UNIFORM_WITH_PARSEQ) for key_step in key_steps: memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 64360351f..7e927a8ed 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -47,7 +47,7 @@ class RenderData: mask: Mask | None seed: int args: RenderInitArgs - parseq_adapter: Any + parseq_adapter: ParseqAdapter srt: Any animation_keys: AnimationKeys animation_mode: AnimationMode diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index 5aa3df6d0..0d66a6054 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -1,11 +1,14 @@ import random from enum import Enum +from ...util import log_utils + class KeyIndexDistribution(Enum): - UNIFORM_SPACING = "Uniform Spacing" - RANDOM_SPACING = "Random Spacing" - RANDOM_PLACEMENT = "Random Placement" + UNIFORM_WITH_PARSEQ = "Uniform with Parseq" # similar to uniform, but parseq key frame diffusion is enforced. + UNIFORM_SPACING = "Uniform Spacing" # distance defined by cadence + RANDOM_SPACING = "Random Spacing" # distance loosely based on cadence (poc) + RANDOM_PLACEMENT = "Random Placement" # no relation to cadence (poc) @property def name(self): @@ -13,9 +16,11 @@ def name(self): @staticmethod def default(): - return KeyIndexDistribution.UNIFORM_SPACING + return KeyIndexDistribution.UNIFORM_WITH_PARSEQ # same as UNIFORM_SPACING, if no Parseq keys are present - def calculate(self, start_index, max_frames, num_key_steps): + def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter): + if self == KeyIndexDistribution.UNIFORM_WITH_PARSEQ: + return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) if self == KeyIndexDistribution.UNIFORM_SPACING: return self._uniform_indexes(start_index, max_frames, num_key_steps) elif self == KeyIndexDistribution.RANDOM_SPACING: @@ -30,6 +35,19 @@ def _uniform_indexes(start_index, max_frames, num_key_steps): return [1 + start_index + int(n * (max_frames - 1 - start_index) / (num_key_steps - 1)) for n in range(num_key_steps)] + @staticmethod + def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter): + parseq_key_frames = [keyframe["frame"] for keyframe in parseq_adapter.parseq_json["keyframes"]] + parseq_key_frames = parseq_key_frames[1:-1] # Ignore 1st and last parseq frame, because they're keys anyway. + num_parseq_key_frames = len(parseq_key_frames) + num_uniform_key_frames = num_key_steps - num_parseq_key_frames + uniform_indices = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_uniform_key_frames) + key_frames = parseq_key_frames.copy() + key_frames.extend(uniform_indices) # FIXME, if this replaces a frame, we need to generated an additional one. + key_frames.sort() + assert len(key_frames) == num_key_steps + return key_frames + @staticmethod def _random_spacing_indexes(start_index, max_frames, num_key_steps): uniform_indexes = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 4a1b138b3..604c96770 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -100,7 +100,9 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn max_frames = data.args.anim_args.max_frames num_key_steps = 1 + int((max_frames - start_index) / data.cadence()) key_steps = [KeyStep.create(data) for _ in range(0, num_key_steps)] - key_steps = KeyStep.recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, index_dist) + actual_num_key_steps = len(key_steps) + key_steps = KeyStep._recalculate_and_check_tweens(key_steps, start_index, max_frames, actual_num_key_steps, + data.parseq_adapter, index_dist) # Print message # TODO move to log_utils tween_count = sum(len(ks.tweens) for ks in key_steps) @@ -110,8 +112,9 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn return key_steps @staticmethod - def recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, index_distribution): - key_indices = index_distribution.calculate(start_index, max_frames, num_key_steps) + def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, + parseq_adapter, index_distribution): + key_indices = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) for i, key_step in enumerate(key_indices): key_steps[i].i = key_indices[i] key_steps = KeyStep._add_tweens_to_key_steps(key_steps) From f463ec34248600b689229d1dcf3844404e31fd92 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 22 Jun 2024 15:29:48 +0200 Subject: [PATCH 093/132] Parseq keyframe replacement. --- .../data/step/key_index_distribution.py | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index 0d66a6054..3e2e09f50 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -37,13 +37,21 @@ def _uniform_indexes(start_index, max_frames, num_key_steps): @staticmethod def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter): + """Calculates uniform indices according to cadence, but parseq key frames replace the closest deforum key.""" + uniform_indices = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) parseq_key_frames = [keyframe["frame"] for keyframe in parseq_adapter.parseq_json["keyframes"]] - parseq_key_frames = parseq_key_frames[1:-1] # Ignore 1st and last parseq frame, because they're keys anyway. - num_parseq_key_frames = len(parseq_key_frames) - num_uniform_key_frames = num_key_steps - num_parseq_key_frames - uniform_indices = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_uniform_key_frames) - key_frames = parseq_key_frames.copy() - key_frames.extend(uniform_indices) # FIXME, if this replaces a frame, we need to generated an additional one. + shifted_parseq_frames = [frame + 1 for frame in parseq_key_frames] + key_frames_set = set(uniform_indices) # set for faster membership checks + + # Insert parseq keyframes while maintaining keyframe count + for current_frame in shifted_parseq_frames: + if current_frame not in key_frames_set: + # Find the closest index in the set to replace + closest_index = min(key_frames_set, key=lambda x: abs(x - current_frame)) + key_frames_set.remove(closest_index) + key_frames_set.add(current_frame) + + key_frames = list(key_frames_set) key_frames.sort() assert len(key_frames) == num_key_steps return key_frames From eca0dadff148241d1dfc60b8048a23c297abb312 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 22 Jun 2024 23:23:30 +0200 Subject: [PATCH 094/132] Depth prediction fix. --- scripts/deforum_helpers/render.py | 6 ++++-- .../rendering/data/step/key_index_distribution.py | 4 ++++ .../rendering/data/step/tween_step.py | 3 ++- scripts/deforum_helpers/rendering/data/turbo.py | 12 ++++++++---- scripts/deforum_helpers/rendering/util/call/anim.py | 6 +++--- scripts/deforum_helpers/rendering/util/log_utils.py | 4 ++++ 6 files changed, 25 insertions(+), 10 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index b072fa891..c6d96b14c 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -45,8 +45,10 @@ def run_render_animation(data: RenderData): log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + # FIXME mute depth predictions, then reactivate?: + # tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + # disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + tq = key_step.tweens [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tq] log_utils.print_animation_frame_info(key_step.i, max_frames) diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index 3e2e09f50..f74ed46dd 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -39,6 +39,10 @@ def _uniform_indexes(start_index, max_frames, num_key_steps): def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter): """Calculates uniform indices according to cadence, but parseq key frames replace the closest deforum key.""" uniform_indices = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + if not parseq_adapter.use_parseq: + log_utils.warn("UNIFORM_WITH_PARSEQ, but Parseq is not active, using UNIFORM_SPACING instead.") + return uniform_indices + parseq_key_frames = [keyframe["frame"] for keyframe in parseq_adapter.parseq_json["keyframes"]] shifted_parseq_frames = [frame + 1 for frame in parseq_key_frames] key_frames_set = set(uniform_indices) # set for faster membership checks diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index a0d854ed4..cd66c39e3 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -117,4 +117,5 @@ def write_tween_frame_subtitle_if_active(self, data: RenderData): if opt_utils.is_generate_subtitles(data): params_to_print = opt_utils.generation_info_for_subtitles(data) params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) - call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) + is_cadence = float(self.tween) < 1.0 + call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index e9d15af0b..fbae3f923 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -6,7 +6,7 @@ from ..util.call.anim import call_anim_frame_warp from ..util.call.resume import call_get_resume_vars from ...hybrid_video import image_transform_ransac, image_transform_optical_flow - +from ..util import log_utils @dataclass(init=True, frozen=False, repr=False, eq=True) class ImageFrame: @@ -103,13 +103,17 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step cadence = data.args.anim_args.optical_flow_cadence flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) tween_step.cadence_flow = (flow / 2) - self.next.image = advance_optical_flow + log_utils.debug(f"cadence {cadence}") self.advance_optical_flow(tween_step) - flow_factor = 1 + diff = tween_step.indexes.frame.i - tween_step.i() + flow_factor = 1.0 / diff + log_utils.debug(f"flow_factor {flow_factor}") self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): - if tween_step.cadence_flow is not None: + is_enforce = True + if is_enforce and tween_step.cadence_flow is not None: + log_utils.debug("do_optical_flow_cadence_after_animation_warping") # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, self.depth) diff --git a/scripts/deforum_helpers/rendering/util/call/anim.py b/scripts/deforum_helpers/rendering/util/call/anim.py index 5c871499d..9452ea8d7 100644 --- a/scripts/deforum_helpers/rendering/util/call/anim.py +++ b/scripts/deforum_helpers/rendering/util/call/anim.py @@ -1,7 +1,7 @@ from ....animation import anim_frame_warp -def call_anim_frame_warp(init, i, image, depth): - ia = init.args - return anim_frame_warp(image, ia.args, ia.anim_args, init.animation_keys.deform_keys, i, init.depth_model, +def call_anim_frame_warp(data, i, image, depth): + ia = data.args + return anim_frame_warp(image, ia.args, ia.anim_args, data.animation_keys.deform_keys, i, data.depth_model, depth=depth, device=ia.root.device, half_precision=ia.root.half_precision) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index cccc72d0d..87b1f4646 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -63,6 +63,10 @@ def info(s: str): print(f"Info: {s}") +def warn(s: str): + print(f"{YELLOW}Warn: {RESET}{s}") + + def debug(s: str): eye_catcher = "###" print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") From 8d060155b848099302e38f799816cf39ec333e01 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 22 Jun 2024 23:58:43 +0200 Subject: [PATCH 095/132] Parseq key frame distribution described in README. --- README.md | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index bb8cef004..12a09296a 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ ## Experimental Deforum Fork -This is an experimental fork of the Deforum A1111 extension with the goal of refactoring the core and allow for direct control of "turbo-frames". -It will eventually also require an experimental version of Parseq to supply new "is_turbo_frame" values, future fork here: https://github.com/Tok/sd-parseq +This is an experimental fork of the Deforum A1111 extension with a refactored the render core that allows for +direct control of "turbo-frames" from Parseq. **Current Status:** -This is a work in progress, and installation is not recommended yet. +This is a work in progress, and installation is not really recommended yet. Please refer to the original project for a stable version: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) -## Neo-Core +## Refactored Render Core This section details the changes made to the render core in this fork. @@ -32,3 +32,23 @@ Existing code in all other Deforum modules remains untouched. **Implementation Details:** * **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. * **Style and Standards:** The code adheres to PEP 8 style guidelines and to other such practices. + +## Parseq Key Frame Distribution + +The [`KeyIndexDistribution.UNIFORM_WITH_PARSEQ`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) mode in the refactored render core ensures a uniform key frame distribution according to the specified cadence settings, while also prioritizing the inclusion of all key frames provided by Parseq. + +Here's how it works: + +1. The uniform key frame distribution is calculated based on the cadence settings, creating a set of non-cadence key frames. +2. The Parseq-provided key frames are then overlaid on top of this uniform distribution, replacing the closest non-cadence key frame. + +The key benefits of this approach are: + +1. **Parseq Precision**: All Parseq-provided key frames are guaranteed to be included in the final output, ensuring the highest possible sync precision. +2. **Cadence Flexibility**: The cadence settings can be set to exceptionally high values (e.g., 10, 20+) without losing Parseq sync, enabling faster generations. +3. **No Workarounds**: This approach eliminates the need for tricky workarounds when using Parseq with high cadence settings. + +The `UNIFORM_WITH_PARSEQ` distribution prioritizes Parseq key frames over the uniform cadence-based distribution, providing a balance between sync precision and generation speed. + +In essence, it means that **all Parseq key frames** (=frames that have a table entry in Parseq), will be **guaranteed to not +be a cadence-frames**. We gain Parseq precision at the tradeoff of cadence regularity. From 7b449b6415a38d071ac2ecc328a5cb27c2cab72e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 23 Jun 2024 06:38:01 +0200 Subject: [PATCH 096/132] Med & low VRAM fix. --- .../rendering/util/memory_utils.py | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index 76ef6e4ea..a07390865 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -1,7 +1,7 @@ # noinspection PyUnresolvedReferences from modules import lowvram, devices, sd_hijack # noinspection PyUnresolvedReferences -from modules.shared import cmd_opts # keep readonly +from modules.shared import cmd_opts, sd_model def is_low_or_med_vram(): @@ -10,36 +10,36 @@ def is_low_or_med_vram(): return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly -def handle_med_or_low_vram_before_step(init): - if init.is_3d_with_med_or_low_vram(): +def handle_med_or_low_vram_before_step(data): + if data.is_3d_with_med_or_low_vram(): # Unload the main checkpoint and load the depth model lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - if init.animation_mode.is_predicting_depths: - init.animation_mode.depth_model.to(init.args.root.device) + if data.animation_mode.is_predicting_depths: + data.depth_model.to(data.args.root.device) -def handle_vram_if_depth_is_predicted(init): - if init.animation_mode.is_predicting_depths: - if init.is_3d_with_med_or_low_vram(): - init.depth_model.to('cpu') +def handle_vram_if_depth_is_predicted(data): + if data.animation_mode.is_predicting_depths: + if data.is_3d_with_med_or_low_vram(): + data.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) -def handle_vram_before_depth_map_generation(init): +def handle_vram_before_depth_map_generation(data): if is_low_or_med_vram(): lowvram.send_everything_to_cpu() sd_hijack.model_hijack.undo_hijack(sd_model) devices.torch_gc() - init.depth_model.to(init.args.root.device) + data.depth_model.to(data.args.root.device) -def handle_vram_after_depth_map_generation(init): +def handle_vram_after_depth_map_generation(data): if is_low_or_med_vram(): - init.depth_model.to('cpu') + data.depth_model.to('cpu') devices.torch_gc() lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) sd_hijack.model_hijack.hijack(sd_model) From 748da5b97171a22ea4717360412c1608c89f7ec4 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 24 Jun 2024 01:11:01 +0200 Subject: [PATCH 097/132] Animation resume logic prepared. --- scripts/deforum_helpers/render.py | 25 +++++++++++++++---- .../rendering/img_2_img_tubes.py | 1 + .../rendering/util/log_utils.py | 2 +- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index c6d96b14c..9e36d17b5 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -14,14 +14,17 @@ # Contact the authors: https://deforum.github.io/ +import os +from pathlib import Path + +import PIL # noinspection PyUnresolvedReferences from modules.shared import cmd_opts, opts, progress_print_out, state -from tqdm import tqdm from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import KeyIndexDistribution, KeyStep -from .rendering.util import log_utils, memory_utils, web_ui_utils +from .rendering.util import filename_utils, log_utils, memory_utils, web_ui_utils def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -32,11 +35,21 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro @log_utils.with_suppressed_table_printing def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - start_index = data.turbo.find_start(data) + _ = data.turbo.find_start(data) # called because it sets up step vars + start_index = 0 max_frames = data.args.anim_args.max_frames key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.UNIFORM_WITH_PARSEQ) for key_step in key_steps: + # Check if resume + filename = filename_utils.frame_filename(data, key_step.i) + full_path = Path(data.output_directory) / filename + is_file_existing = os.path.exists(full_path) + if is_file_existing: + log_utils.warn(f"Frame {filename} exists, skipping to next key frame.") + key_step.render_data.args.args.seed = key_step.next_seed() + continue + memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) @@ -45,7 +58,7 @@ def run_render_animation(data: RenderData): log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - # FIXME mute depth predictions, then reactivate?: + # FIXME check if verbose console output is activated, otherwise reactivate: # tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, # disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') tq = key_step.tweens @@ -63,7 +76,9 @@ def run_render_animation(data: RenderData): log_utils.print_warning_generate_returned_no_image() break - image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + # TODO move check + if type(image) is not PIL.Image.Image: # check is required when resuming from timestring + image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) key_step.progress_and_save(image) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index ed00b63f9..6421e090f 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -8,6 +8,7 @@ from .data.step.key_step import KeyStep from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube +from ..colors import maintain_colors from ..masks import do_overlay_mask """ diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 87b1f4646..fc5b89ed5 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -64,7 +64,7 @@ def info(s: str): def warn(s: str): - print(f"{YELLOW}Warn: {RESET}{s}") + print(f"{ORANGE}Warn: {RESET}{s}") def debug(s: str): From 330dfbf138800e76f401a25bb9a97d9b3be4f304 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 29 Jun 2024 19:22:02 +0200 Subject: [PATCH 098/132] Parseq only keyframe distribution. --- scripts/deforum_helpers/render.py | 2 +- .../data/step/key_index_distribution.py | 19 +++++++++++++++++-- .../rendering/data/step/key_step.py | 12 ++++++++++-- .../rendering/data/step/tween_step.py | 1 + 4 files changed, 29 insertions(+), 5 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 9e36d17b5..06517bb1d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -39,7 +39,7 @@ def run_render_animation(data: RenderData): start_index = 0 max_frames = data.args.anim_args.max_frames - key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.UNIFORM_WITH_PARSEQ) + key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.PARSEQ_ONLY) for key_step in key_steps: # Check if resume filename = filename_utils.frame_filename(data, key_step.i) diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index f74ed46dd..abafdc97f 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -1,10 +1,12 @@ import random from enum import Enum +from typing import List from ...util import log_utils class KeyIndexDistribution(Enum): + PARSEQ_ONLY = "Parseq Only" # cadence is ignored. all frames not present in the Parseq table are handled as tweens. UNIFORM_WITH_PARSEQ = "Uniform with Parseq" # similar to uniform, but parseq key frame diffusion is enforced. UNIFORM_SPACING = "Uniform Spacing" # distance defined by cadence RANDOM_SPACING = "Random Spacing" # distance loosely based on cadence (poc) @@ -16,9 +18,11 @@ def name(self): @staticmethod def default(): - return KeyIndexDistribution.UNIFORM_WITH_PARSEQ # same as UNIFORM_SPACING, if no Parseq keys are present + return KeyIndexDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. - def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter): + def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: + if self == KeyIndexDistribution.PARSEQ_ONLY: + return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) if self == KeyIndexDistribution.UNIFORM_WITH_PARSEQ: return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) if self == KeyIndexDistribution.UNIFORM_SPACING: @@ -35,6 +39,17 @@ def _uniform_indexes(start_index, max_frames, num_key_steps): return [1 + start_index + int(n * (max_frames - 1 - start_index) / (num_key_steps - 1)) for n in range(num_key_steps)] + @staticmethod + def _parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter): + """Only Parseq key frames are used. Cadence settings are ignored.""" + if not parseq_adapter.use_parseq: + log_utils.warn("PARSEQ_ONLY, but Parseq is not active, using UNIFORM_SPACING instead.") + return KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + + parseq_key_frames = [keyframe["frame"] for keyframe in parseq_adapter.parseq_json["keyframes"]] + shifted_parseq_frames = [frame + 1 for frame in parseq_key_frames] + return shifted_parseq_frames + @staticmethod def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter): """Calculates uniform indices according to cadence, but parseq key frames replace the closest deforum key.""" diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 604c96770..974615a8f 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -98,9 +98,14 @@ def create(data: RenderData): def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIndexDistribution.default()): """Creates a list of key steps for the entire animation.""" max_frames = data.args.anim_args.max_frames + num_key_steps = 1 + int((max_frames - start_index) / data.cadence()) + if data.parseq_adapter.use_parseq and index_dist is KeyIndexDistribution.PARSEQ_ONLY: + num_key_steps = len(data.parseq_adapter.parseq_json["keyframes"]) + key_steps = [KeyStep.create(data) for _ in range(0, num_key_steps)] actual_num_key_steps = len(key_steps) + key_steps = KeyStep._recalculate_and_check_tweens(key_steps, start_index, max_frames, actual_num_key_steps, data.parseq_adapter, index_dist) @@ -114,9 +119,10 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn @staticmethod def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, parseq_adapter, index_distribution): - key_indices = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) + key_indices: List[int] = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) for i, key_step in enumerate(key_indices): key_steps[i].i = key_indices[i] + key_steps = KeyStep._add_tweens_to_key_steps(key_steps) assert len(key_steps) == num_key_steps @@ -131,7 +137,9 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st log_utils.debug(f"Key step {i} at {ks.i}: {len(tween_indices)} tween indices: {tween_indices}") assert key_steps[0].i == 1 - assert key_steps[-1].i == max_frames + if index_distribution != KeyIndexDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. + assert key_steps[-1].i == max_frames + return key_steps @staticmethod diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index cd66c39e3..c8679f943 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -35,6 +35,7 @@ def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): return # skipping tween emission on the last frame data = last_step.render_data + data.turbo.steps = len(last_step.tweens) self.handle_synchronous_status_concerns(data) self.process(last_step, data) From abfbe1f49981a2bdebc7832cde80b5dd418d77d7 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 30 Jun 2024 01:47:03 +0200 Subject: [PATCH 099/132] Variable pseudo cadence setup. --- scripts/deforum_helpers/render.py | 17 +++++++--- .../rendering/data/render_data.py | 5 +++ .../data/step/key_index_distribution.py | 2 ++ .../rendering/data/step/key_step.py | 32 ++++++++++-------- .../rendering/data/step/tween_step.py | 3 +- .../deforum_helpers/rendering/data/turbo.py | 33 ++++++++++++------- 6 files changed, 62 insertions(+), 30 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 06517bb1d..1d2e90780 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -20,6 +20,7 @@ import PIL # noinspection PyUnresolvedReferences from modules.shared import cmd_opts, opts, progress_print_out, state +from tqdm import tqdm from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData @@ -55,13 +56,21 @@ def run_render_animation(data: RenderData): is_step_with_tweens = len(key_step.tweens) > 0 if is_step_with_tweens: # emit tweens + # setup variable pseudo cadence + data.parseq_adapter.cadence = len(key_step.tweens) + data.parseq_adapter.a1111_cadence = len(key_step.tweens) + data.args.anim_args.diffusion_cadence = len(key_step.tweens) + data.args.anim_args.optical_flow_cadence = len(key_step.tweens) + data.args.anim_args.cadence_flow_factor_schedule = len(key_step.tweens) + # data.parseq_adapter.print_parseq_table() + log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - # FIXME check if verbose console output is activated, otherwise reactivate: - # tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - # disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') - tq = key_step.tweens + tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + # FIXME disable progress bar is verbose console output is activated: + # tq = key_step.tweens [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tq] log_utils.print_animation_frame_info(key_step.i, max_frames) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 7e927a8ed..8621d93a0 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -284,6 +284,11 @@ def prepare_generation(self, data, step, i): self.prompt_for_current_step(i) self.update_video_data_for_current_frame(i, step) self.update_mask_image(step, data.mask) + + if len(step.tweens) > 0: + data.args.anim_args.cadence_flow_factor_schedule = f"0: ({len(step.tweens) + 1})" + print(f"cadence_flow_factor_schedule: {data.args.anim_args.cadence_flow_factor_schedule}") + self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) opt_utils.setup(self, step.schedule) memory_utils.handle_vram_if_depth_is_predicted(data) diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index abafdc97f..a69750bae 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -72,6 +72,8 @@ def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_ key_frames = list(key_frames_set) key_frames.sort() + #key_frames[0] = start_index + 1 # Enforce first index + #key_frames[-1] = max_frames # Enforce last index assert len(key_frames) == num_key_steps return key_frames diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index 974615a8f..bd3315e68 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -111,7 +111,7 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn # Print message # TODO move to log_utils tween_count = sum(len(ks.tweens) for ks in key_steps) - msg_start = f"Created {len(key_steps)} KeySteps with {tween_count} Tween frames." + msg_start = f"Created {len(key_steps)} key frames with {tween_count} tweens." msg_end = f"Key frame index distribution: '{index_dist.name}'." log_utils.info(f"{msg_start} {msg_end}") return key_steps @@ -123,19 +123,20 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st for i, key_step in enumerate(key_indices): key_steps[i].i = key_indices[i] - key_steps = KeyStep._add_tweens_to_key_steps(key_steps) + is_enforce_tweens = index_distribution == KeyIndexDistribution.PARSEQ_ONLY + key_steps = KeyStep._add_tweens_to_key_steps(key_steps, is_enforce_tweens) assert len(key_steps) == num_key_steps + for i, ks in enumerate(key_steps): + tween_indices = [t.i() for t in ks.tweens] + log_utils.debug(f"Key frame {ks.i} has {len(tween_indices)} tweens: {tween_indices}") + # The number of generated tweens depends on index since last key-frame. The last tween has the same # me index as the key_step it belongs to and is meant to replace the unprocessed original key frame. total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) assert total_count == max_frames + len(key_steps) - 1 # every key frame except the 1st has a tween double. assert key_steps[0].tweens == [] # 1st key step has no tweens - for i, ks in enumerate(key_steps): - tween_indices = [t.i() for t in ks.tweens] - log_utils.debug(f"Key step {i} at {ks.i}: {len(tween_indices)} tween indices: {tween_indices}") - assert key_steps[0].i == 1 if index_distribution != KeyIndexDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. assert key_steps[-1].i == max_frames @@ -143,16 +144,17 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st return key_steps @staticmethod - def _add_tweens_to_key_steps(key_steps): + def _add_tweens_to_key_steps(key_steps, is_enforce_tweens): + log_utils.info(f"adding {len(key_steps)} tweens...") for i in range(1, len(key_steps)): # skipping 1st key frame data = key_steps[i].render_data - if data.turbo.is_emit_in_between_frames(): + if data.turbo.is_emit_in_between_frames() or is_enforce_tweens: from_i = key_steps[i - 1].i to_i = key_steps[i].i tweens, values = Tween.create_in_between_steps(key_steps[i], data, from_i, to_i) for tween in tweens: # TODO move to creation tween.indexes.update_tween_index(tween.i() + key_steps[i].i) - log_utils.info(f"Creating {len(tweens)} tween steps ({from_i}->{to_i}) for key step {key_steps[i].i}") + log_utils.info(f"Creating {len(tweens)} tweens ({from_i}->{to_i}) for key frame at {key_steps[i].i}") key_steps[i].tweens = tweens key_steps[i].tween_values = values key_steps[i].render_data.indexes.update_tween_start(data.turbo) @@ -165,14 +167,16 @@ def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, i def maybe_write_frame_subtitle(self): data = self.render_data if data.turbo.is_first_step_with_subtitles(data): - self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(data) - self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_to_print) + params_string = opt_utils.generation_info_for_subtitles(data) + self.subtitle_params_to_print = params_string + self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_string) call_write_frame_subtitle(data, data.indexes.frame.i, params_string) def write_frame_subtitle_if_active(self, data: RenderData): if opt_utils.is_generate_subtitles(data): - self.subtitle_params_to_print = opt_utils.generation_info_for_subtitles(data) - self.subtitle_params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) + params_string = opt_utils.generation_info_for_subtitles(data) + self.subtitle_params_to_print = params_string + self.subtitle_params_string = call_format_animation_params(data, self.indexes.tween.i, params_string) call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) def apply_frame_warp_transform(self, data: RenderData, image): @@ -332,7 +336,7 @@ def _progress_save_and_get_next_index(self, image): data.images.previous = opencv_image filename = filename_utils.frame_filename(data, self.i) - is_overwrite = False # replace the processed tween frame with the original one? (probably not) + is_overwrite = True # replace the processed tween frame with the original one? (probably not) if is_overwrite or not os.path.exists(os.path.join(data.args.args.outdir, filename)): # In many cases, the original images may look more detailed or 'better' than the processed ones, # but we only save the frames that were processed tough the flows to keep the output consistent. diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index c8679f943..57d4626a1 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -35,7 +35,7 @@ def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): return # skipping tween emission on the last frame data = last_step.render_data - data.turbo.steps = len(last_step.tweens) + #data.turbo.steps = len(last_step.tweens) self.handle_synchronous_status_concerns(data) self.process(last_step, data) @@ -89,6 +89,7 @@ def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tw def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): is_tween = True warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) + # print(f"warped {warped}") recolored = grayscale_tube(data)(warped) masked = overlay_mask_tube(data, is_tween)(recolored) return masked diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index fbae3f923..6641d76c0 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -4,6 +4,7 @@ from .subtitle import Srt from ..util.call.anim import call_anim_frame_warp +from ..util.call.hybrid import call_get_flow_from_images from ..util.call.resume import call_get_resume_vars from ...hybrid_video import image_transform_ransac, image_transform_optical_flow from ..util import log_utils @@ -27,9 +28,9 @@ def create(data): return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) def advance(self, data, i: int, depth): - if self.is_advance_prev(i): + if self.is_advance_prev(i) and self.prev.image is not None: self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) - if self.is_advance_next(i): + if self.is_advance_next(i) and self.next.image is not None: self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) def do_hybrid_video_motion(self, data, indexes, reference_images): @@ -96,23 +97,33 @@ def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): - has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[indexes.tween.start.i] > 0 + i = data.indexes.tween.start + has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[i] > 0 has_images = self.prev.image is not None and self.next.image is not None has_step_and_images = tween_step.cadence_flow is None and has_images - if has_tween_schedule and has_step_and_images: - cadence = data.args.anim_args.optical_flow_cadence + if has_tween_schedule and has_step_and_images and data.animation_mode.is_raft_active(): + cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) tween_step.cadence_flow = (flow / 2) - log_utils.debug(f"cadence {cadence}") - self.advance_optical_flow(tween_step) + if tween_step.cadence_flow is not None: + self.advance_optical_flow(tween_step) + diff = tween_step.indexes.frame.i - tween_step.i() flow_factor = 1.0 / diff - log_utils.debug(f"flow_factor {flow_factor}") - self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) + #flow_factor = tween_step.tween + #log_utils.info(f"flow_factor {flow_factor}") + if tween_step.cadence_flow is not None: + self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): - is_enforce = True - if is_enforce and tween_step.cadence_flow is not None: + if not data.animation_mode.is_raft_active(): + return self.next.image + log_utils.debug(f"tween_step.cadence_flow {tween_step.cadence_flow}") + if tween_step.cadence_flow is None: + cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence + flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) + tween_step.cadence_flow = (flow / 2) + if tween_step.cadence_flow is not None: log_utils.debug("do_optical_flow_cadence_after_animation_warping") # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) From 5817392ff411307a05142c8ae54a26aab3952766 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 30 Jun 2024 03:49:55 +0200 Subject: [PATCH 100/132] Readme update. --- README.md | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 12a09296a..3bbf4107b 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,19 @@ -## Experimental Deforum Fork - +# Experimental Deforum Fork This is an experimental fork of the Deforum A1111 extension with a refactored the render core that allows for direct control of "turbo-frames" from Parseq. -**Current Status:** +### Current Status This is a work in progress, and installation is not really recommended yet. Please refer to the original project for a stable version: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) ## Refactored Render Core - This section details the changes made to the render core in this fork. For easy integration, this fork isolates changes to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. Existing code in all other Deforum modules remains untouched. +![image](https://i.kym-cdn.com/photos/images/original/001/399/018/31f.gif) + * **Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. * **Key Improvements:** * Reduced cyclomatic complexity of the [`render_animation`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. @@ -35,20 +35,31 @@ Existing code in all other Deforum modules remains untouched. ## Parseq Key Frame Distribution -The [`KeyIndexDistribution.UNIFORM_WITH_PARSEQ`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) mode in the refactored render core ensures a uniform key frame distribution according to the specified cadence settings, while also prioritizing the inclusion of all key frames provided by Parseq. +### Purpose & Features +* Parseq Precision: All Parseq-provided key frames are guaranteed to be included in the final output, ensuring the highest possible sync precision. +* Cadence Flexibility: Cadence settings can be set to exceptionally high values (e.g., 10, 20+) or can be ignored completely without losing Parseq synchronization, enabling fast generations with less diffusion steps. +* No Workarounds: This approach eliminates the need for tricky workarounds when using Parseq with high or ignored cadence settings. -Here's how it works: +### Key Index Distribution Modes -1. The uniform key frame distribution is calculated based on the cadence settings, creating a set of non-cadence key frames. -2. The Parseq-provided key frames are then overlaid on top of this uniform distribution, replacing the closest non-cadence key frame. +#### PARSEQ_ONLY (without cadence) +[`KeyIndexDistribution.PARSEQ_ONLY`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) +This distribution **completely ignores cadence** and is only diffusing key frames defined in Parseq (=frames that have a table entry in Parseq). +All non-key frames are handled as if they would be cadence frames, although that term doesn't really apply when the key frames are not spaced out in fixed intervals. -The key benefits of this approach are: +#### UNIFORM_WITH_PARSEQ (variable pseudo cadence) +[`KeyIndexDistribution.UNIFORM_WITH_PARSEQ`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) +This mode ensures a uniform key frame distribution according to the specified cadence settings, while also prioritizing the inclusion of all key frames provided by Parseq. -1. **Parseq Precision**: All Parseq-provided key frames are guaranteed to be included in the final output, ensuring the highest possible sync precision. -2. **Cadence Flexibility**: The cadence settings can be set to exceptionally high values (e.g., 10, 20+) without losing Parseq sync, enabling faster generations. -3. **No Workarounds**: This approach eliminates the need for tricky workarounds when using Parseq with high cadence settings. +Here's how it works: +1. The uniform key frame distribution is calculated based on the cadence settings, creating a set of non-cadence key frames. +2. The Parseq-provided key frames are then overlaid on top of this uniform distribution, replacing the closest non-cadence key frame. The `UNIFORM_WITH_PARSEQ` distribution prioritizes Parseq key frames over the uniform cadence-based distribution, providing a balance between sync precision and generation speed. -In essence, it means that **all Parseq key frames** (=frames that have a table entry in Parseq), will be **guaranteed to not -be a cadence-frames**. We gain Parseq precision at the tradeoff of cadence regularity. +In essence, it means that **all Parseq key frames**, will be guaranteed to not +be a cadence-frames. We gain Parseq precision at the tradeoff of cadence regularity. + +##### Pseudo Cadence +`cadence` is still loosely observed in this mode, but since key frames are rearranged, the cadence settings should be understood as an average value. +In UNIFORM_WITH_PARSEQ mode, a cadence setting of "10" means "about 10". From 794d11758372dfcea6de5e0f0d143c949dc269f8 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 2 Jul 2024 01:36:10 +0200 Subject: [PATCH 101/132] RAFT fix (wip). --- scripts/deforum_helpers/render.py | 3 +- .../rendering/data/anim/animation_mode.py | 9 ++++-- .../rendering/data/step/key_step.py | 15 ++++++---- .../deforum_helpers/rendering/data/turbo.py | 30 +++++++++---------- .../rendering/img_2_img_tubes.py | 3 +- 5 files changed, 33 insertions(+), 27 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 1d2e90780..448917032 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -97,4 +97,5 @@ def run_render_animation(data: RenderData): key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) - key_step.render_data.animation_mode.unload_raft_and_depth_model() + + data.animation_mode.unload_raft_and_depth_model() diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index c7a0c1dcc..f3e5ff0f3 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -32,7 +32,7 @@ def unload_raft_and_depth_model(self): self.raft_model.delete_model() @staticmethod - def has_video_input(anim_args) -> bool: + def _has_video_input(anim_args) -> bool: return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod @@ -87,7 +87,10 @@ def from_args(step_args): # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') return AnimationMode( - AnimationMode.has_video_input(sa.anim_args), AnimationMode.initial_hybrid_files(sa), - hybrid_input_files, None, keep_3d_models_in_vram(sa), + AnimationMode._has_video_input(sa.anim_args), + AnimationMode.initial_hybrid_files(sa), + hybrid_input_files, + None, + keep_3d_models_in_vram(sa), AnimationMode.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), AnimationMode.load_raft_if_active(sa.anim_args, sa.args)) diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index bd3315e68..1aee9a31e 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -4,12 +4,14 @@ import cv2 import numpy as np +from PIL import Image from . import KeyIndexDistribution from .tween_step import Tween from ..render_data import RenderData from ..schedule import Schedule -from ...util import filename_utils, log_utils, memory_utils, opt_utils +from ... import img_2_img_tubes +from ...util import filename_utils, log_utils, memory_utils, opt_utils, utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -370,12 +372,13 @@ def generate_and_save_depth_map_if_active(self): def do_optical_flow_redo_before_generation(self): data = self.render_data + redo = data.args.anim_args.optical_flow_redo_generation stored_seed = data.args.args.seed # keep original to reset it after executing the optical flow - data.args.args.seed = generate_random_seed() # set a new random seed - print_optical_flow_info(data, optical_flow_redo_generation) # TODO output temp seed? + data.args.args.seed = utils.generate_random_seed() # set a new random seed + log_utils.print_optical_flow_info(data, redo) # TODO output temp seed? - sample_image = call_generate(data, data.indexes.frame.i) - optical_tube = optical_flow_redo_tube(data, optical_flow_redo_generation) + sample_image = call_generate(data, self) + optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, redo, self) transformed_sample_image = optical_tube(sample_image) data.args.args.seed = stored_seed # restore stored seed @@ -387,7 +390,7 @@ def do_diffusion_redo(self): last_diffusion_redo_index = int(data.args.anim_args.diffusion_redo) for n in range(0, last_diffusion_redo_index): print_redo_generation_info(data, n) - data.args.args.seed = generate_random_seed() + data.args.args.seed = utils.generate_random_seed() diffusion_redo_image = call_generate(data, self) diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) # color match on last one only diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 6641d76c0..7e87f3793 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -3,11 +3,13 @@ from cv2.typing import MatLike from .subtitle import Srt +from ..util import log_utils from ..util.call.anim import call_anim_frame_warp from ..util.call.hybrid import call_get_flow_from_images from ..util.call.resume import call_get_resume_vars -from ...hybrid_video import image_transform_ransac, image_transform_optical_flow -from ..util import log_utils +from ...hybrid_video import (image_transform_ransac, image_transform_optical_flow, + abs_flow_to_rel_flow, rel_flow_to_abs_flow) + @dataclass(init=True, frozen=False, repr=False, eq=True) class ImageFrame: @@ -21,6 +23,7 @@ class Turbo: steps: int # cadence prev: ImageFrame next: ImageFrame + # depth: None @staticmethod def create(data): @@ -67,10 +70,10 @@ def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_imag turbo.advance_optical_tween_flow(self, step, flow) data.animation_mode.prev_flow = flow - def advance_cadence_flow(self, tween_step): - ff = step.step_data.sub_step.cadence_flow_factor - i = indexes.tween.i - inc = tween_step.cadence_flow_inc + def advance_cadence_flow(self, data, tween_step): + ff = data.args.anim_args.cadence_flow_factor_schedule + i = tween_step.i() + inc = tween_step.cadence_flow_inc # FIXME if self.is_advance_prev(i): self.prev.image = image_transform_optical_flow(self.prev.image, inc, ff) if self.is_advance_next(i): @@ -118,20 +121,15 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): if not data.animation_mode.is_raft_active(): return self.next.image - log_utils.debug(f"tween_step.cadence_flow {tween_step.cadence_flow}") - if tween_step.cadence_flow is None: - cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence - flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) - tween_step.cadence_flow = (flow / 2) if tween_step.cadence_flow is not None: - log_utils.debug("do_optical_flow_cadence_after_animation_warping") - # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). - temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) - new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, self.depth) + # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abs conversions). + # temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) + # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween - self.advance_cadence_flow(tween_step) + self.advance_cadence_flow(data, tween_step) self.prev.index = self.next.frame_idx = indexes.tween.i if indexes is not None else 0 if self.prev.image is not None and tween_step.tween < 1.0: return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 6421e090f..8ad387733 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -9,6 +9,7 @@ from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube from ..colors import maintain_colors +from ..hybrid_video import image_transform_optical_flow from ..masks import do_overlay_mask """ @@ -48,7 +49,7 @@ def noise_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: return tube(lambda img: step.apply_frame_noising(data, step, img)) -def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: +def optical_flow_redo_tube(data: RenderData, optical_flow, step) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( From 753fd6dab988544aee6797ba59c59d1038d5fe70 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 6 Jul 2024 18:42:24 +0200 Subject: [PATCH 102/132] Added new tab and fixed some things. --- README.md | 20 ++++-- scripts/deforum_helpers/args.py | 3 +- scripts/deforum_helpers/render.py | 63 +++++++++++-------- .../rendering/data/anim/animation_mode.py | 9 ++- .../rendering/data/render_data.py | 8 ++- .../data/step/key_index_distribution.py | 13 ++-- .../rendering/data/step/key_step.py | 21 ++++--- .../rendering/data/step/tween_step.py | 2 +- .../deforum_helpers/rendering/data/turbo.py | 29 ++++----- scripts/deforum_helpers/ui_elements.py | 58 ++++++++++++++++- scripts/deforum_helpers/ui_left.py | 7 ++- 11 files changed, 164 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index 3bbf4107b..0c2510bcd 100644 --- a/README.md +++ b/README.md @@ -3,14 +3,24 @@ This is an experimental fork of the Deforum A1111 extension with a refactored th direct control of "turbo-frames" from Parseq. ### Current Status -This is a work in progress, and installation is not really recommended yet. -Please refer to the original project for a stable version: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) +The extension is work in progress, and may be unstable. +Installation is only recommended to for the purpose of using the experimental Parseq based key frame redistribution features. +Some Deforum features like hybrid video and coherence algorithms may currently not work and may need to be turned off. + +For a feature complete and stable version, please refer to the original project at: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) + +#### Installation +Since the name of this extension is shared with the original, it requires a full removal of the regular Deforum extension from Automatic 1111. +Then go to "Extensions" and "Install from URL": https://github.com/Tok/sd-webui-deforum +It will add a new "Neo Core" tab in Deforum with further information on how to properly use it. + +The rest of this Readme can be skipped if you're not interested in the code. ## Refactored Render Core This section details the changes made to the render core in this fork. -For easy integration, this fork isolates changes to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. -Existing code in all other Deforum modules remains untouched. +For easy integration, this fork isolates changes mostly to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. +Existing code in other Deforum modules, remains mostly unchanged (exception is UI and args related code). ![image](https://i.kym-cdn.com/photos/images/original/001/399/018/31f.gif) @@ -42,6 +52,8 @@ Existing code in all other Deforum modules remains untouched. ### Key Index Distribution Modes +New key in 'deforum_settings.txt': "neocore_key_index_distribution" + #### PARSEQ_ONLY (without cadence) [`KeyIndexDistribution.PARSEQ_ONLY`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) This distribution **completely ignores cadence** and is only diffusing key frames defined in Parseq (=frames that have a table entry in Parseq). diff --git a/scripts/deforum_helpers/args.py b/scripts/deforum_helpers/args.py index 5be0b3eac..cd7fd19fa 100644 --- a/scripts/deforum_helpers/args.py +++ b/scripts/deforum_helpers/args.py @@ -703,7 +703,8 @@ def DeforumAnimArgs(): "info": "", }, "hybrid_comp_mask_auto_contrast": False, - "hybrid_comp_save_extra_frames": False + "hybrid_comp_save_extra_frames": False, + "neocore_key_index_distribution": "Parseq Only" } def DeforumArgs(): diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 1d2e90780..d3fe19109 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -39,39 +39,22 @@ def run_render_animation(data: RenderData): _ = data.turbo.find_start(data) # called because it sets up step vars start_index = 0 max_frames = data.args.anim_args.max_frames - - key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.PARSEQ_ONLY) + key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.from_UI_tab(data)) for key_step in key_steps: - # Check if resume - filename = filename_utils.frame_filename(data, key_step.i) - full_path = Path(data.output_directory) / filename - is_file_existing = os.path.exists(full_path) - if is_file_existing: - log_utils.warn(f"Frame {filename} exists, skipping to next key frame.") - key_step.render_data.args.args.seed = key_step.next_seed() + if is_resume(data, key_step): continue memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - is_step_with_tweens = len(key_step.tweens) > 0 - if is_step_with_tweens: # emit tweens - # setup variable pseudo cadence - data.parseq_adapter.cadence = len(key_step.tweens) - data.parseq_adapter.a1111_cadence = len(key_step.tweens) - data.args.anim_args.diffusion_cadence = len(key_step.tweens) - data.args.anim_args.optical_flow_cadence = len(key_step.tweens) - data.args.anim_args.cadence_flow_factor_schedule = len(key_step.tweens) - # data.parseq_adapter.print_parseq_table() - + is_emit_tweens = len(key_step.tweens) > 0 + if is_emit_tweens: + setup_pseudo_cadence(data, len(key_step.tweens)) log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - tq = tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') - # FIXME disable progress bar is verbose console output is activated: - # tq = key_step.tweens - [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tq] + tweens = tweens_with_progress(key_step) + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] log_utils.print_animation_frame_info(key_step.i, max_frames) key_step.maybe_write_frame_subtitle() @@ -85,9 +68,9 @@ def run_render_animation(data: RenderData): log_utils.print_warning_generate_returned_no_image() break - # TODO move check if type(image) is not PIL.Image.Image: # check is required when resuming from timestring image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) key_step.progress_and_save(image) @@ -97,4 +80,32 @@ def run_render_animation(data: RenderData): key_step.update_render_preview() web_ui_utils.update_status_tracker(key_step.render_data) - key_step.render_data.animation_mode.unload_raft_and_depth_model() + + data.animation_mode.unload_raft_and_depth_model() + + +def is_resume(data, key_step): + filename = filename_utils.frame_filename(data, key_step.i) + full_path = Path(data.output_directory) / filename + is_file_existing = os.path.exists(full_path) + if is_file_existing: + log_utils.warn(f"Frame {filename} exists, skipping to next key frame.") + key_step.render_data.args.args.seed = key_step.next_seed() + return is_file_existing + + +def setup_pseudo_cadence(data, value): + data.parseq_adapter.cadence = value + data.parseq_adapter.a1111_cadence = value + data.args.anim_args.diffusion_cadence = value + data.args.anim_args.optical_flow_cadence = value + data.args.anim_args.cadence_flow_factor_schedule = value + # data.parseq_adapter.print_parseq_table() + + +def tweens_with_progress(key_step): + # only use tween progress bar when extra console output (aka "dev mode") is disabled. + is_not_verbose = not opts.data.get("deforum_debug_mode_enabled", False) + return tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') \ + if is_not_verbose else key_step.tweens diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index c7a0c1dcc..f3e5ff0f3 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -32,7 +32,7 @@ def unload_raft_and_depth_model(self): self.raft_model.delete_model() @staticmethod - def has_video_input(anim_args) -> bool: + def _has_video_input(anim_args) -> bool: return AnimationMode._is_2d_or_3d_mode(anim_args) and AnimationMode._is_using_hybrid_frames(anim_args) @staticmethod @@ -87,7 +87,10 @@ def from_args(step_args): # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') return AnimationMode( - AnimationMode.has_video_input(sa.anim_args), AnimationMode.initial_hybrid_files(sa), - hybrid_input_files, None, keep_3d_models_in_vram(sa), + AnimationMode._has_video_input(sa.anim_args), + AnimationMode.initial_hybrid_files(sa), + hybrid_input_files, + None, + keep_3d_models_in_vram(sa), AnimationMode.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), AnimationMode.load_raft_if_active(sa.anim_args, sa.args)) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 8621d93a0..5b21c4d64 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -286,8 +286,12 @@ def prepare_generation(self, data, step, i): self.update_mask_image(step, data.mask) if len(step.tweens) > 0: - data.args.anim_args.cadence_flow_factor_schedule = f"0: ({len(step.tweens) + 1})" - print(f"cadence_flow_factor_schedule: {data.args.anim_args.cadence_flow_factor_schedule}") + # FIXME it's not yet working as it is supposed to + # data.args.anim_args.cadence_flow_factor_schedule = f"0: ({len(step.tweens) + 1})" + data.args.anim_args.cadence_flow_factor_schedule = f"0: (1)" + step.step_data.cadence_flow_factor = 1.0 / len(step.tweens) + # print(f"cadence_flow_factor: {step.step_data.cadence_flow_factor}") + # print(f"cadence_flow_factor_schedule: {data.args.anim_args.cadence_flow_factor_schedule}") self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) opt_utils.setup(self, step.schedule) diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index a69750bae..937ff1262 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -16,6 +16,13 @@ class KeyIndexDistribution(Enum): def name(self): return self.value + @staticmethod + def from_UI_tab(data): + is_uniform_with_parseq = data.args.anim_args.neocore_key_index_distribution == "Uniform with Parseq" + return KeyIndexDistribution.UNIFORM_WITH_PARSEQ \ + if is_uniform_with_parseq \ + else KeyIndexDistribution.PARSEQ_ONLY + @staticmethod def default(): return KeyIndexDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. @@ -65,15 +72,13 @@ def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_ # Insert parseq keyframes while maintaining keyframe count for current_frame in shifted_parseq_frames: if current_frame not in key_frames_set: - # Find the closest index in the set to replace - closest_index = min(key_frames_set, key=lambda x: abs(x - current_frame)) + # Find the closest index in the set to replace (1st and last frame excluded) + closest_index = min(list(key_frames_set)[1:-1], key=lambda x: abs(x - current_frame)) key_frames_set.remove(closest_index) key_frames_set.add(current_frame) key_frames = list(key_frames_set) key_frames.sort() - #key_frames[0] = start_index + 1 # Enforce first index - #key_frames[-1] = max_frames # Enforce last index assert len(key_frames) == num_key_steps return key_frames diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index bd3315e68..d694fdc86 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -4,12 +4,14 @@ import cv2 import numpy as np +from PIL import Image from . import KeyIndexDistribution from .tween_step import Tween from ..render_data import RenderData from ..schedule import Schedule -from ...util import filename_utils, log_utils, memory_utils, opt_utils +from ... import img_2_img_tubes +from ...util import filename_utils, log_utils, memory_utils, opt_utils, utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -24,7 +26,7 @@ from ....seed import next_seed -@dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=False) class KeyStepData: noise: Any = None strength: Any = None @@ -34,7 +36,7 @@ class KeyStepData: sigma: Any = None amount: Any = None threshold: Any = None - cadence_flow_factor: Any = None + cadence_flow_factor: Any = None # FIXME re-assignable redo_flow_factor: Any = None hybrid_comp_schedules: Any = None @@ -134,7 +136,9 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st # The number of generated tweens depends on index since last key-frame. The last tween has the same # me index as the key_step it belongs to and is meant to replace the unprocessed original key frame. total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) + assert total_count == max_frames + len(key_steps) - 1 # every key frame except the 1st has a tween double. + assert key_steps[0].tweens == [] # 1st key step has no tweens assert key_steps[0].i == 1 @@ -370,12 +374,13 @@ def generate_and_save_depth_map_if_active(self): def do_optical_flow_redo_before_generation(self): data = self.render_data + redo = data.args.anim_args.optical_flow_redo_generation stored_seed = data.args.args.seed # keep original to reset it after executing the optical flow - data.args.args.seed = generate_random_seed() # set a new random seed - print_optical_flow_info(data, optical_flow_redo_generation) # TODO output temp seed? + data.args.args.seed = utils.generate_random_seed() # set a new random seed + log_utils.print_optical_flow_info(data, redo) # TODO output temp seed? - sample_image = call_generate(data, data.indexes.frame.i) - optical_tube = optical_flow_redo_tube(data, optical_flow_redo_generation) + sample_image = call_generate(data, self) + optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, redo, self) transformed_sample_image = optical_tube(sample_image) data.args.args.seed = stored_seed # restore stored seed @@ -387,7 +392,7 @@ def do_diffusion_redo(self): last_diffusion_redo_index = int(data.args.anim_args.diffusion_redo) for n in range(0, last_diffusion_redo_index): print_redo_generation_info(data, n) - data.args.args.seed = generate_random_seed() + data.args.args.seed = utils.generate_random_seed() diffusion_redo_image = call_generate(data, self) diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) # color match on last one only diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 57d4626a1..77caa1054 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -105,7 +105,7 @@ def calculate_depth_prediction(data, turbo: Turbo): return data.depth_model.predict(image, weight, precision) def process(self, last_step, data): - data.turbo.advance_optical_flow_cadence_before_animation_warping(data, self) + data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_step, self) self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) data.turbo.advance(data, self.indexes.tween.i, self.depth) data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # FIXME? remove self.indexes or init.indexes diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 6641d76c0..535e0b0f5 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -3,11 +3,13 @@ from cv2.typing import MatLike from .subtitle import Srt +from ..util import log_utils from ..util.call.anim import call_anim_frame_warp from ..util.call.hybrid import call_get_flow_from_images from ..util.call.resume import call_get_resume_vars -from ...hybrid_video import image_transform_ransac, image_transform_optical_flow -from ..util import log_utils +from ...hybrid_video import (image_transform_ransac, image_transform_optical_flow, + abs_flow_to_rel_flow, rel_flow_to_abs_flow) + @dataclass(init=True, frozen=False, repr=False, eq=True) class ImageFrame: @@ -21,6 +23,7 @@ class Turbo: steps: int # cadence prev: ImageFrame next: ImageFrame + # depth: None @staticmethod def create(data): @@ -67,10 +70,10 @@ def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_imag turbo.advance_optical_tween_flow(self, step, flow) data.animation_mode.prev_flow = flow - def advance_cadence_flow(self, tween_step): - ff = step.step_data.sub_step.cadence_flow_factor - i = indexes.tween.i - inc = tween_step.cadence_flow_inc + def advance_cadence_flow(self, data, tween_step): + ff = data.args.anim_args.cadence_flow_factor_schedule + i = tween_step.i() + inc = tween_step.cadence_flow_inc # FIXME if self.is_advance_prev(i): self.prev.image = image_transform_optical_flow(self.prev.image, inc, ff) if self.is_advance_next(i): @@ -95,7 +98,7 @@ def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): matrix = call_get_matrix_for_hybrid_motion(data, indexes.tween.i - 1) turbo.advance_ransac_transform(data, matrix) - def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step): + def advance_optical_flow_cadence_before_animation_warping(self, data, last_step, tween_step): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): i = data.indexes.tween.start has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[i] > 0 @@ -108,21 +111,13 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, tween_step if tween_step.cadence_flow is not None: self.advance_optical_flow(tween_step) - diff = tween_step.indexes.frame.i - tween_step.i() - flow_factor = 1.0 / diff - #flow_factor = tween_step.tween - #log_utils.info(f"flow_factor {flow_factor}") + flow_factor = 100.0 / len(last_step.tweens) if tween_step.cadence_flow is not None: self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): if not data.animation_mode.is_raft_active(): return self.next.image - log_utils.debug(f"tween_step.cadence_flow {tween_step.cadence_flow}") - if tween_step.cadence_flow is None: - cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence - flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) - tween_step.cadence_flow = (flow / 2) if tween_step.cadence_flow is not None: log_utils.debug("do_optical_flow_cadence_after_animation_warping") # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abc conversions). @@ -131,7 +126,7 @@ def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_s tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween - self.advance_cadence_flow(tween_step) + self.advance_cadence_flow(data, tween_step) self.prev.index = self.next.frame_idx = indexes.tween.i if indexes is not None else 0 if self.prev.image is not None and tween_step.tween < 1.0: return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween diff --git a/scripts/deforum_helpers/ui_elements.py b/scripts/deforum_helpers/ui_elements.py index f9467e7c5..8492295b0 100644 --- a/scripts/deforum_helpers/ui_elements.py +++ b/scripts/deforum_helpers/ui_elements.py @@ -579,4 +579,60 @@ def get_tab_output(da, dv): ffmpeg_stitch_imgs_but = gr.Button(value="*Stitch frames to video*") ffmpeg_stitch_imgs_but.click(fn=direct_stitch_vid_from_frames, inputs=[image_path, fps, add_soundtrack, soundtrack_path]) - return {k: v for k, v in {**locals(), **vars()}.items()} \ No newline at end of file + return {k: v for k, v in {**locals(), **vars()}.items()} + + +def get_tab_neocore(): + bars_mark = "📊" + check_mark = "✔️" + cross_mark = "❌" + warn_mark = "⚠️" + bulb_mark = "💡" + with gr.TabItem('Neo Core', elem_id='neocore_tab'): + gr.HTML(value=f"""

+

Parseq Instructions:
+
    +
  1. Setup your Parseq workflow as usual, but with high FPS (60?) and with high cadence (30?).
    2. +
  2. The refactored render core ensures that every frame in the Parseq table is diffused. It may therefore easily be used with just a fixed value for 'strengh' like perhaps '0.4'. +
+
Deforum Instructions:
+ Select one of the following key index distributions: +
    +
  1. {bars_mark} Parseq Only: Only frames with an entry in the Parseq table are diffused. Actual cadence settings are ignored and all frames not defined in Parseq are handled as if they would be cadence frames.
    2. +
  2. {bars_mark} Uniform with Parseq: Calculates uniform cadence distribution but rearranges them to preserve proper Parseq synchronization at high cadence. Cadence may be understood as 'pseudo cadence'. A cadence value of '30' may more correctly be understood as 'about 30' in this mode.
    3. +
+

""") + neocore_key_index_distribution = gr.Dropdown( + label="Select Key Index Distribution:", + choices=["Parseq Only", "Uniform with Parseq"], # TODO use KeyIndexDistribution enum + value="Parseq Only") + gr.HTML(value=f"""

+

Currently required settings to avoid errors or undefined behaviour:
+
    +
  1. {warn_mark} In both cases set "cadence" to a value larger than 1 and make sure it's the same in Parseq and in Deforum.
    2. +
  2. {warn_mark} Go to the Deforum "Keyframe" tab, select the "Coherence" tab and set everything to 'None'.
    3. +
+
Recommendations:
+
    +
  1. {bulb_mark} "'Uniform with Parseq' is able to handle high cadence values, but the number of actually diffused frames is still dictated by the cadence setting.
  2. +
  3. {bulb_mark} "Consider rendering at 60FPS. It shouldn't take much longer since it doesn't affect the number of diffusions."
  4. +
  5. {bulb_mark} "In 'Uniform with Parseq' mode, consider a cadence of 30, at 60FPS it will ensure two diffusions per seconds, which goes well for a clip synchonized to a 120BPM tune."
  6. +
+
Should be working:
+
    +
  1. {check_mark} Parseq based key frame distributions. Currently the only reason to use this build.
    2. +
  2. {check_mark} Subtitle generation (but may require manual cleanup to remove residual cadence values).
    3. +
  3. {check_mark} Depth warping.
    4. +
+
Currently untested or not working:
+
    +
  1. {cross_mark} Hybrid Video.
    2. +
  2. {cross_mark} Optical Flow and Color Coherence.
    3. +
  3. {cross_mark} Control Net (idk?).
    4. +
+
TL;DR
+
    +
  1. Setup Parseq workflow at 60 FPS with cadence 30, ideally with audio sync and 3d rotation etc, then set a fixed value for 'strength' at perhaps 0.4
    2. +
  2. Setup Deforum with 60 FPS and cadence 30 as well and disable everything in the "Keyframes" - "Coherence" tab. Then do the rest as usual and go...
    3. +

""") + return {k: v for k, v in {**locals(), **vars()}.items()} diff --git a/scripts/deforum_helpers/ui_left.py b/scripts/deforum_helpers/ui_left.py index aa1886372..205aabd30 100644 --- a/scripts/deforum_helpers/ui_left.py +++ b/scripts/deforum_helpers/ui_left.py @@ -20,7 +20,8 @@ from .gradio_funcs import change_css, handle_change_functions from .args import DeforumArgs, DeforumAnimArgs, ParseqArgs, DeforumOutputArgs, RootArgs, LoopArgs from .deforum_controlnet import setup_controlnet_ui -from .ui_elements import get_tab_run, get_tab_keyframes, get_tab_prompts, get_tab_init, get_tab_hybrid, get_tab_output +from .ui_elements import (get_tab_run, get_tab_keyframes, get_tab_prompts, get_tab_init, + get_tab_hybrid, get_tab_output, get_tab_neocore) def set_arg_lists(): # convert dicts to NameSpaces for easy working (args.param instead of args['param'] @@ -50,8 +51,10 @@ def setup_deforum_left_side_ui(): controlnet_dict = setup_controlnet_ui() # ControlNet tab tab_hybrid_params = get_tab_hybrid(da) # Hybrid tab tab_output_params = get_tab_output(da, dv) # Output tab + tab_neocore_params = get_tab_neocore() # Refactored Render Core tab # add returned gradio elements from main tabs to locals() - for key, value in {**tab_run_params, **tab_keyframes_params, **tab_prompts_params, **tab_init_params, **controlnet_dict, **tab_hybrid_params, **tab_output_params}.items(): + for key, value in {**tab_run_params, **tab_keyframes_params, **tab_prompts_params, **tab_init_params, + **controlnet_dict, **tab_hybrid_params, **tab_output_params, **tab_neocore_params}.items(): locals()[key] = value # Gradio's Change functions - hiding and renaming elements based on other elements From 62c9b8beea09ce642931060782bd6fe005ae8624 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 7 Jul 2024 00:20:25 +0200 Subject: [PATCH 103/132] Flow factor tuning. --- scripts/deforum_helpers/rendering/data/render_data.py | 6 +++--- scripts/deforum_helpers/rendering/data/turbo.py | 10 ++++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 5b21c4d64..a04c518ae 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -277,7 +277,7 @@ def update_mask_image(self, step, mask): def prepare_generation(self, data, step, i): # TODO move all of this to Step? if i > self.args.anim_args.max_frames - 1: - return # FIXME? sus + return self.update_some_args_for_current_step(step, i) self.update_seed_and_checkpoint_for_current_step(i) self.update_sub_seed_schedule_for_current_step(i) @@ -289,9 +289,9 @@ def prepare_generation(self, data, step, i): # FIXME it's not yet working as it is supposed to # data.args.anim_args.cadence_flow_factor_schedule = f"0: ({len(step.tweens) + 1})" data.args.anim_args.cadence_flow_factor_schedule = f"0: (1)" - step.step_data.cadence_flow_factor = 1.0 / len(step.tweens) - # print(f"cadence_flow_factor: {step.step_data.cadence_flow_factor}") # print(f"cadence_flow_factor_schedule: {data.args.anim_args.cadence_flow_factor_schedule}") + # step.step_data.cadence_flow_factor = 1.0 / len(step.tweens) + # print(f"cadence_flow_factor: {step.step_data.cadence_flow_factor}") self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) opt_utils.setup(self, step.schedule) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index d23ede221..d56060e1f 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -111,7 +111,9 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, last_step, if tween_step.cadence_flow is not None: self.advance_optical_flow(tween_step) - flow_factor = 100.0 / len(last_step.tweens) + # flow_factor = 1.0 / (last_step.i - tween_step.i() + 1) + # flow_factor = 100.0 / len(last_step.tweens) + flow_factor = 100.0 if tween_step.cadence_flow is not None: self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) @@ -120,9 +122,9 @@ def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_s return self.next.image if tween_step.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abs conversions). - # temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) - # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) - new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) + temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) + # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween From 76ff811bdf4072578a7c52fcd1d720b320223520 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 7 Jul 2024 04:09:35 +0200 Subject: [PATCH 104/132] Example clip added to readme. --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 0c2510bcd..1044651e0 100644 --- a/README.md +++ b/README.md @@ -75,3 +75,12 @@ be a cadence-frames. We gain Parseq precision at the tradeoff of cadence regular ##### Pseudo Cadence `cadence` is still loosely observed in this mode, but since key frames are rearranged, the cadence settings should be understood as an average value. In UNIFORM_WITH_PARSEQ mode, a cadence setting of "10" means "about 10". + +## Examples + +### Mark Fell - multistability 2-b +This music video was **generated in less than only 92 minutes on a 4070**, directly at 60 FPS and with a resolution of 1280x720. +It contains 6694 frames of which only 913 were actually diffused, but in full sync with Parseq. +Calculated pseudo cadence is 6694 / 913 = 7.33. More details in video description. + +[![Video Title](https://img.youtube.com/vi/K-9V5Tntck4/maxresdefault.jpg)](https://www.youtube.com/watch?v=K-9V5Tntck4) From 767dfd6e68eb73db6d05c5ae3524685e48281a5b Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 7 Jul 2024 04:19:30 +0200 Subject: [PATCH 105/132] Readme update. --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1044651e0..1a0e57086 100644 --- a/README.md +++ b/README.md @@ -78,9 +78,12 @@ In UNIFORM_WITH_PARSEQ mode, a cadence setting of "10" means "about 10". ## Examples -### Mark Fell - multistability 2-b +### Parseq Synchronized Music Video This music video was **generated in less than only 92 minutes on a 4070**, directly at 60 FPS and with a resolution of 1280x720. It contains 6694 frames of which only 913 were actually diffused, but in full sync with Parseq. Calculated pseudo cadence is 6694 / 913 = 7.33. More details in video description. -[![Video Title](https://img.youtube.com/vi/K-9V5Tntck4/maxresdefault.jpg)](https://www.youtube.com/watch?v=K-9V5Tntck4) +"neocore_key_index_distribution": "Parseq Only" + +#### Mark Fell - multistability 2-b +[![Mark Fell - multistability 2-b](https://img.youtube.com/vi/K-9V5Tntck4/maxresdefault.jpg)](https://www.youtube.com/watch?v=K-9V5Tntck4 "Click to watch on YouTube") From b98e01b70f521138c1f44389d6926687b27bda7d Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 7 Jul 2024 04:41:55 +0200 Subject: [PATCH 106/132] Readme update. --- README.md | 80 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 1a0e57086..bcf34fe7c 100644 --- a/README.md +++ b/README.md @@ -1,65 +1,40 @@ # Experimental Deforum Fork -This is an experimental fork of the Deforum A1111 extension with a refactored the render core that allows for +This is an experimental fork of the Deforum A1111 extension with a refactored render core that allows for direct control of "turbo-frames" from Parseq. ### Current Status -The extension is work in progress, and may be unstable. -Installation is only recommended to for the purpose of using the experimental Parseq based key frame redistribution features. -Some Deforum features like hybrid video and coherence algorithms may currently not work and may need to be turned off. +The forked extension is work in progress and may not be stable. +Installation is recommended **only for the purpose of using the Parseq based key frame redistribution features**. + +โš ๏ธ Some Deforum features like hybrid video and coherence algorithms are currently not working and may need to be turned off. For a feature complete and stable version, please refer to the original project at: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) #### Installation Since the name of this extension is shared with the original, it requires a full removal of the regular Deforum extension from Automatic 1111. -Then go to "Extensions" and "Install from URL": https://github.com/Tok/sd-webui-deforum -It will add a new "Neo Core" tab in Deforum with further information on how to properly use it. - -The rest of this Readme can be skipped if you're not interested in the code. - -## Refactored Render Core -This section details the changes made to the render core in this fork. +After that it can be installed from "Extensions" using "Install from URL": https://github.com/Tok/sd-webui-deforum +The forked extension adds a new "Neo Core" tab in Deforum, with further information on how to properly use it. -For easy integration, this fork isolates changes mostly to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. -Existing code in other Deforum modules, remains mostly unchanged (exception is UI and args related code). +๐Ÿ’ก The rest of this Readme can be skipped if you're not interested in the code. -![image](https://i.kym-cdn.com/photos/images/original/001/399/018/31f.gif) - -* **Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. -* **Key Improvements:** - * Reduced cyclomatic complexity of the [`render_animation`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. - * Improved separation of concerns (e.g., dedicated module for printing). - * Reduced argument complexity and improved scope control and variable organization using dataclasses. - * Enhanced code clarity with improved naming and removal of obsolete comments. - * But preserving domain specific lingo. - * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. - -**New Rendering Modules:** -* [`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. -* [`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. -* [`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. -* [`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. - -**Implementation Details:** -* **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. -* **Style and Standards:** The code adheres to PEP 8 style guidelines and to other such practices. +## ๐ŸŽ๏ธ Parseq Key Frame Distribution -## Parseq Key Frame Distribution +### Purpose & Features -### Purpose & Features * Parseq Precision: All Parseq-provided key frames are guaranteed to be included in the final output, ensuring the highest possible sync precision. * Cadence Flexibility: Cadence settings can be set to exceptionally high values (e.g., 10, 20+) or can be ignored completely without losing Parseq synchronization, enabling fast generations with less diffusion steps. * No Workarounds: This approach eliminates the need for tricky workarounds when using Parseq with high or ignored cadence settings. -### Key Index Distribution Modes +### ๐Ÿ“Š Key Index Distribution Modes New key in 'deforum_settings.txt': "neocore_key_index_distribution" -#### PARSEQ_ONLY (without cadence) +#### ๐Ÿ•ณ๏ธ PARSEQ_ONLY (without cadence) [`KeyIndexDistribution.PARSEQ_ONLY`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) This distribution **completely ignores cadence** and is only diffusing key frames defined in Parseq (=frames that have a table entry in Parseq). All non-key frames are handled as if they would be cadence frames, although that term doesn't really apply when the key frames are not spaced out in fixed intervals. -#### UNIFORM_WITH_PARSEQ (variable pseudo cadence) +#### ใ€ฐ๏ธ UNIFORM_WITH_PARSEQ (variable pseudo cadence) [`KeyIndexDistribution.UNIFORM_WITH_PARSEQ`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) This mode ensures a uniform key frame distribution according to the specified cadence settings, while also prioritizing the inclusion of all key frames provided by Parseq. @@ -76,7 +51,7 @@ be a cadence-frames. We gain Parseq precision at the tradeoff of cadence regular `cadence` is still loosely observed in this mode, but since key frames are rearranged, the cadence settings should be understood as an average value. In UNIFORM_WITH_PARSEQ mode, a cadence setting of "10" means "about 10". -## Examples +## ๐Ÿ“ผ Video Examples ### Parseq Synchronized Music Video This music video was **generated in less than only 92 minutes on a 4070**, directly at 60 FPS and with a resolution of 1280x720. @@ -87,3 +62,30 @@ Calculated pseudo cadence is 6694 / 913 = 7.33. More details in video descriptio #### Mark Fell - multistability 2-b [![Mark Fell - multistability 2-b](https://img.youtube.com/vi/K-9V5Tntck4/maxresdefault.jpg)](https://www.youtube.com/watch?v=K-9V5Tntck4 "Click to watch on YouTube") + +## ๐Ÿงน Refactored Render Core +This section details the changes made to the render core in this fork. + +For easy integration, this fork isolates changes mostly to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. +Existing code in other Deforum modules, remains mostly unchanged (exception is UI and args related code). + +![image](https://i.kym-cdn.com/photos/images/original/001/399/018/31f.gif) + +* **๐Ÿ› ๏ธ Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. +* **โ˜๏ธ Key Improvements:** + * Reduced cyclomatic complexity of the [`render_animation`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. + * Improved separation of concerns (e.g., dedicated module for printing). + * Reduced argument complexity and improved scope control and variable organization using dataclasses. + * Enhanced code clarity with improved naming and removal of obsolete comments. + * But preserving domain specific lingo. + * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. + +**๐Ÿ†• New Rendering Modules:** +* [`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. +* [`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. +* [`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. +* [`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. + +**๐Ÿ”Ž Implementation Details:** +* **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. +* **Style and Standards:** The code adheres to PEP 8 style guidelines and to other such practices. From 13dc5c4ce90e8bc1180ccc713bac4a4dd3148a5b Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 13 Jul 2024 18:15:20 +0200 Subject: [PATCH 107/132] Cleanup and fixed critical bug that would ignore transformation params in some cases with PARSEQ_ONLY setup. Parse table printout reactivated. --- scripts/deforum_helpers/render.py | 46 +++++----- .../rendering/data/anim/animation_mode.py | 3 +- .../rendering/data/render_data.py | 4 +- .../data/step/key_index_distribution.py | 35 ++++--- .../rendering/data/step/key_step.py | 92 ++++++++----------- .../rendering/data/step/tween_step.py | 66 ++++++------- .../deforum_helpers/rendering/data/turbo.py | 56 +++++------ .../rendering/util/image_utils.py | 5 + .../rendering/util/log_utils.py | 27 +++++- 9 files changed, 175 insertions(+), 159 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d3fe19109..153860d86 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -17,29 +17,27 @@ import os from pathlib import Path -import PIL # noinspection PyUnresolvedReferences -from modules.shared import cmd_opts, opts, progress_print_out, state +from modules.shared import cmd_opts, progress_print_out, state from tqdm import tqdm from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData from .rendering.data.step import KeyIndexDistribution, KeyStep -from .rendering.util import filename_utils, log_utils, memory_utils, web_ui_utils +from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) run_render_animation(render_data) -@log_utils.with_suppressed_table_printing +# @log_utils.with_suppressed_table_printing def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - _ = data.turbo.find_start(data) # called because it sets up step vars - start_index = 0 - max_frames = data.args.anim_args.max_frames + start_index = data.turbo.find_start(data) key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.from_UI_tab(data)) + for key_step in key_steps: if is_resume(data, key_step): continue @@ -47,16 +45,10 @@ def run_render_animation(data: RenderData): memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - is_emit_tweens = len(key_step.tweens) > 0 - if is_emit_tweens: - setup_pseudo_cadence(data, len(key_step.tweens)) - log_utils.print_tween_frame_from_to_info(key_step) - grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube - overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - tweens = tweens_with_progress(key_step) - [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] + if len(key_step.tweens) > 0: + emit_tweens(data, key_step) - log_utils.print_animation_frame_info(key_step.i, max_frames) + log_utils.print_animation_frame_info(key_step.i, data.args.anim_args.max_frames) key_step.maybe_write_frame_subtitle() frame_tube = img_2_img_tubes.frame_transformation_tube @@ -68,7 +60,7 @@ def run_render_animation(data: RenderData): log_utils.print_warning_generate_returned_no_image() break - if type(image) is not PIL.Image.Image: # check is required when resuming from timestring + if not image_utils.is_PIL(image): # check is required when resuming from timestring image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) @@ -94,18 +86,28 @@ def is_resume(data, key_step): return is_file_existing +def emit_tweens(data, key_step): + setup_pseudo_cadence(data, len(key_step.tweens) - 1) + if key_step.i == 1: + data.parseq_adapter.print_parseq_table() + log_utils.print_tween_frame_from_to_info(key_step) + grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube + overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube + tweens = _tweens_with_progress(key_step) + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] + + def setup_pseudo_cadence(data, value): + data.turbo.cadence = value data.parseq_adapter.cadence = value data.parseq_adapter.a1111_cadence = value data.args.anim_args.diffusion_cadence = value data.args.anim_args.optical_flow_cadence = value data.args.anim_args.cadence_flow_factor_schedule = value - # data.parseq_adapter.print_parseq_table() -def tweens_with_progress(key_step): +def _tweens_with_progress(key_step): # only use tween progress bar when extra console output (aka "dev mode") is disabled. - is_not_verbose = not opts.data.get("deforum_debug_mode_enabled", False) return tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') \ - if is_not_verbose else key_step.tweens + if not log_utils.is_verbose() else key_step.tweens diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index f3e5ff0f3..56808696d 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -78,8 +78,7 @@ def initial_hybrid_files(sa) -> list[Path]: # may cause side effects on args and anim_args. _, __, init_hybrid_input_files = hybrid_generation(sa.args, sa.anim_args, sa.root) return init_hybrid_input_files - else: - return [] + return [] @staticmethod def from_args(step_args): diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index a04c518ae..fb88d55f6 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -7,6 +7,8 @@ import numpy as np import pandas as pd from PIL import Image +# noinspection PyUnresolvedReferences +from modules.shared import opts from .anim import AnimationKeys, AnimationMode from .images import Images @@ -57,7 +59,7 @@ class RenderData: is_use_mask: bool @staticmethod - def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) -> 'RenderData': + def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) -> 'RenderData': ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) output_directory = args.outdir diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py index 937ff1262..b1118afae 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py @@ -12,34 +12,31 @@ class KeyIndexDistribution(Enum): RANDOM_SPACING = "Random Spacing" # distance loosely based on cadence (poc) RANDOM_PLACEMENT = "Random Placement" # no relation to cadence (poc) - @property - def name(self): - return self.value - @staticmethod def from_UI_tab(data): is_uniform_with_parseq = data.args.anim_args.neocore_key_index_distribution == "Uniform with Parseq" - return KeyIndexDistribution.UNIFORM_WITH_PARSEQ \ - if is_uniform_with_parseq \ - else KeyIndexDistribution.PARSEQ_ONLY + return (KeyIndexDistribution.UNIFORM_WITH_PARSEQ + if is_uniform_with_parseq + else KeyIndexDistribution.PARSEQ_ONLY) @staticmethod def default(): return KeyIndexDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: - if self == KeyIndexDistribution.PARSEQ_ONLY: - return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) - if self == KeyIndexDistribution.UNIFORM_WITH_PARSEQ: - return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) - if self == KeyIndexDistribution.UNIFORM_SPACING: - return self._uniform_indexes(start_index, max_frames, num_key_steps) - elif self == KeyIndexDistribution.RANDOM_SPACING: - return self._random_spacing_indexes(start_index, max_frames, num_key_steps) - elif self == KeyIndexDistribution.RANDOM_PLACEMENT: - return self._random_placement_indexes(start_index, max_frames, num_key_steps) - else: - raise ValueError(f"Invalid KeyIndexDistribution: {self}") + match self: + case KeyIndexDistribution.PARSEQ_ONLY: + return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) + case KeyIndexDistribution.UNIFORM_WITH_PARSEQ: + return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) + case KeyIndexDistribution.UNIFORM_SPACING: + return self._uniform_indexes(start_index, max_frames, num_key_steps) + case KeyIndexDistribution.RANDOM_SPACING: + return self._random_spacing_indexes(start_index, max_frames, num_key_steps) + case KeyIndexDistribution.RANDOM_PLACEMENT: + return self._random_placement_indexes(start_index, max_frames, num_key_steps) + case _: + raise ValueError(f"Invalid KeyIndexDistribution: {self}") @staticmethod def _uniform_indexes(start_index, max_frames, num_key_steps): diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/step/key_step.py index d694fdc86..8f4d0dc85 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/step/key_step.py @@ -107,15 +107,13 @@ def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIn key_steps = [KeyStep.create(data) for _ in range(0, num_key_steps)] actual_num_key_steps = len(key_steps) + #data.args.anim_args.max_frames = actual_num_key_steps + #max_frames = actual_num_key_steps - key_steps = KeyStep._recalculate_and_check_tweens(key_steps, start_index, max_frames, actual_num_key_steps, - data.parseq_adapter, index_dist) + key_steps = KeyStep._recalculate_and_check_tweens( # TODO remove actual_num_key_steps arg + key_steps, start_index, max_frames, actual_num_key_steps, data.parseq_adapter, index_dist) + log_utils.print_tween_step_creation_info(key_steps, index_dist) - # Print message # TODO move to log_utils - tween_count = sum(len(ks.tweens) for ks in key_steps) - msg_start = f"Created {len(key_steps)} key frames with {tween_count} tweens." - msg_end = f"Key frame index distribution: '{index_dist.name}'." - log_utils.info(f"{msg_start} {msg_end}") return key_steps @staticmethod @@ -125,22 +123,18 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st for i, key_step in enumerate(key_indices): key_steps[i].i = key_indices[i] - is_enforce_tweens = index_distribution == KeyIndexDistribution.PARSEQ_ONLY - key_steps = KeyStep._add_tweens_to_key_steps(key_steps, is_enforce_tweens) - assert len(key_steps) == num_key_steps - - for i, ks in enumerate(key_steps): - tween_indices = [t.i() for t in ks.tweens] - log_utils.debug(f"Key frame {ks.i} has {len(tween_indices)} tweens: {tween_indices}") + key_steps = KeyStep._add_tweens_to_key_steps(key_steps) + log_utils.print_key_step_debug_info_if_verbose(key_steps) # The number of generated tweens depends on index since last key-frame. The last tween has the same - # me index as the key_step it belongs to and is meant to replace the unprocessed original key frame. - total_count = len(key_steps) + sum(len(key_step.tweens) for key_step in key_steps) - - assert total_count == max_frames + len(key_steps) - 1 # every key frame except the 1st has a tween double. + # index as the key_step it belongs to and is meant to replace the unprocessed original key frame. + # TODO? make unit tests instead of asserts... + # total_count = len(key_steps) + sum(len(key_step.tweens) - 1 for key_step in key_steps) + # log_utils.info(f"total_count {total_count} len(key_steps) {len(key_steps)} max_frames {max_frames}") + # assert total_count == len(key_steps) + max_frames # every key frame except the 1st has a tween double. + assert len(key_steps) == num_key_steps assert key_steps[0].tweens == [] # 1st key step has no tweens - assert key_steps[0].i == 1 if index_distribution != KeyIndexDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. assert key_steps[-1].i == max_frames @@ -148,20 +142,17 @@ def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_st return key_steps @staticmethod - def _add_tweens_to_key_steps(key_steps, is_enforce_tweens): - log_utils.info(f"adding {len(key_steps)} tweens...") + def _add_tweens_to_key_steps(key_steps): + log_utils.info(f"Adding tweens to {len(key_steps)} keyframes...") for i in range(1, len(key_steps)): # skipping 1st key frame data = key_steps[i].render_data - if data.turbo.is_emit_in_between_frames() or is_enforce_tweens: - from_i = key_steps[i - 1].i - to_i = key_steps[i].i - tweens, values = Tween.create_in_between_steps(key_steps[i], data, from_i, to_i) - for tween in tweens: # TODO move to creation - tween.indexes.update_tween_index(tween.i() + key_steps[i].i) - log_utils.info(f"Creating {len(tweens)} tweens ({from_i}->{to_i}) for key frame at {key_steps[i].i}") - key_steps[i].tweens = tweens - key_steps[i].tween_values = values - key_steps[i].render_data.indexes.update_tween_start(data.turbo) + from_i = key_steps[i - 1].i + to_i = key_steps[i].i + tweens, values = Tween.create_in_between_steps(key_steps, i, data, from_i, to_i) + log_utils.debug(f"Creating {len(tweens)} tweens ({from_i}->{to_i}) for key frame at {key_steps[i].i}") + key_steps[i].tweens = tweens + key_steps[i].tween_values = values + key_steps[i].render_data.indexes.update_tween_start(data.turbo) return key_steps def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): @@ -195,8 +186,7 @@ def _do_hybrid_compositing_on_cond(self, data: RenderData, image, condition): if condition: _, composed = call_hybrid_composite(data, i, image, schedules) return composed - else: - return image + return image def do_hybrid_compositing_before_motion(self, data: RenderData, image): condition = data.is_hybrid_composite_before_motion() @@ -212,8 +202,7 @@ def apply_scaling(self, image): def apply_anti_blur(self, data: RenderData, image): if self.step_data.amount > 0: return call_unsharp_mask(data, self, image, data.mask) - else: - return image + return image def apply_frame_noising(self, data: RenderData, mask, image): is_use_any_mask = data.args.args.use_mask or data.args.anim_args.use_noise_mask @@ -225,23 +214,23 @@ def apply_frame_noising(self, data: RenderData, mask, image): @staticmethod def apply_color_matching(data: RenderData, image): - if data.has_color_coherence(): - if data.images.color_match is None: - # TODO questionable - # initialize color_match for next iteration with current image, but don't do anything yet. - if image is not None: - data.images.color_match = image.copy() - else: - return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) - return image + return apply_color_coherence(image, data) if data.has_color_coherence() else image + + @staticmethod + def apply_color_coherence(image, data: RenderData): + if data.images.color_match is None: + # Initialize color_match for next iteration with current image, but don't do anything yet. + if image is not None: + data.images.color_match = image.copy() + return image + return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) @staticmethod def transform_to_grayscale_if_active(data: RenderData, image): if data.args.anim_args.color_force_grayscale: grayscale = cv2.cvtColor(data.images.previous, cv2.COLOR_BGR2GRAY) return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) - else: - return image + return image @staticmethod def apply_hybrid_motion_ransac_transform(data: RenderData, image): @@ -268,8 +257,7 @@ def apply_hybrid_motion_optical_flow(data: RenderData, image): transformed = image_transform_optical_flow(images.previous, flow, step.step_data.flow_factor()) data.animation_mode.prev_flow = flow # side effect return transformed - else: - return image + return image def create_color_match_for_video(self): data = self.render_data @@ -295,8 +283,7 @@ def transform_and_update_noised_sample(self, frame_tube, contrasted_noise_tube): noised_image = contrasted_noise_tube(data, self)(transformed_image) data.update_sample_and_args_for_current_progression_step(self, noised_image) return transformed_image - else: - return None + return None def prepare_generation(self, frame_tube, contrasted_noise_tube): self.render_data.images.color_match = self.create_color_match_for_video() @@ -351,8 +338,7 @@ def _progress_save_and_get_next_index(self, image): self.depth = self.generate_and_save_depth_map_if_active() if data.turbo.has_steps(): return data.indexes.frame.i + data.turbo.progress_step(data.indexes, opencv_image) - else: - return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. + return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. def next_seed(self): return next_seed(self.render_data.args.args, self.render_data.args.root) @@ -380,7 +366,7 @@ def do_optical_flow_redo_before_generation(self): log_utils.print_optical_flow_info(data, redo) # TODO output temp seed? sample_image = call_generate(data, self) - optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, redo, self) + optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, redo) transformed_sample_image = optical_tube(sample_image) data.args.args.seed = stored_seed # restore stored seed diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/step/tween_step.py index 77caa1054..6adf61057 100644 --- a/scripts/deforum_helpers/rendering/data/step/tween_step.py +++ b/scripts/deforum_helpers/rendering/data/step/tween_step.py @@ -13,7 +13,7 @@ class Tween: """cadence vars""" indexes: Indexes - tween: float + value: float cadence_flow: Any # late init cadence_flow_inc: Any # late init depth: Any @@ -35,7 +35,7 @@ def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): return # skipping tween emission on the last frame data = last_step.render_data - #data.turbo.steps = len(last_step.tweens) + # data.turbo.steps = len(last_step.tweens) self.handle_synchronous_status_concerns(data) self.process(last_step, data) @@ -46,11 +46,41 @@ def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): # updating reference images to calculate hybrid motions in next iteration data.images.previous = new_image # FIXME + def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): + is_tween = True + warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) + # print(f"warped {warped}") + recolored = grayscale_tube(data)(warped) + masked = overlay_mask_tube(data, is_tween)(recolored) + return masked + + def process(self, last_step, data): + data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_step, self) + self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) + # log_utils.info(f"self.depth_prediction {self.depth_prediction}") + data.turbo.advance(data, self.indexes.tween.i, self.depth) + data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # FIXME? remove self.indexes or init.indexes + + def handle_synchronous_status_concerns(self, data): + self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? + log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.value) + web_ui_utils.update_progress_during_cadence(data, self.indexes) + + def write_tween_frame_subtitle_if_active(self, data: RenderData): + if opt_utils.is_generate_subtitles(data): + params_to_print = opt_utils.generation_info_for_subtitles(data) + params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) + is_cadence = self.value < 1.0 + call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) + @staticmethod - def create_in_between_steps(last_step, data, from_i, to_i): + def create_in_between_steps(key_steps, i, data, from_i, to_i): tween_range = range(from_i, to_i) tween_indexes_list: List[Indexes] = Tween.create_indexes(data.indexes, tween_range) + last_step = key_steps[i] tween_steps_and_values = Tween.create_steps(last_step, tween_indexes_list) + for tween in tween_steps_and_values[0]: + tween.indexes.update_tween_index(tween.i() + key_steps[i].i) return tween_steps_and_values @staticmethod @@ -83,16 +113,7 @@ def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tw if len(tween_indexes_list) > 0: expected_tween_frames = Tween._calculate_expected_tween_frames(len(tween_indexes_list)) return Tween.create_steps_from_values(last_step, expected_tween_frames), expected_tween_frames - else: - return list(), list() - - def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): - is_tween = True - warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) - # print(f"warped {warped}") - recolored = grayscale_tube(data)(warped) - masked = overlay_mask_tube(data, is_tween)(recolored) - return masked + return list(), list() @staticmethod def calculate_depth_prediction(data, turbo: Turbo): @@ -102,22 +123,5 @@ def calculate_depth_prediction(data, turbo: Turbo): image = turbo.next.image weight = data.args.anim_args.midas_weight precision = data.args.root.half_precision + # log_utils.info(f"weight {weight} precision {precision}") return data.depth_model.predict(image, weight, precision) - - def process(self, last_step, data): - data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_step, self) - self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) - data.turbo.advance(data, self.indexes.tween.i, self.depth) - data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # FIXME? remove self.indexes or init.indexes - - def handle_synchronous_status_concerns(self, data): - self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? - log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.tween) - web_ui_utils.update_progress_during_cadence(data, self.indexes) - - def write_tween_frame_subtitle_if_active(self, data: RenderData): - if opt_utils.is_generate_subtitles(data): - params_to_print = opt_utils.generation_info_for_subtitles(data) - params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) - is_cadence = float(self.tween) < 1.0 - call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index d56060e1f..731490b47 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -23,6 +23,7 @@ class Turbo: steps: int # cadence prev: ImageFrame next: ImageFrame + # depth: None @staticmethod @@ -31,9 +32,12 @@ def create(data): return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) def advance(self, data, i: int, depth): - if self.is_advance_prev(i) and self.prev.image is not None: - self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) - if self.is_advance_next(i) and self.next.image is not None: + # log_utils.info(f"i {i} is prev {self.is_advance_prev(i)} is next {self.is_advance_next(i)}") + # log_utils.info(f"i {i} has prev {self.prev.image is not None} has next {self.next.image is not None}") + # TODO test if not replacing prev here is fine.. + # if self.prev.image is not None: + # self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) + if self.next.image is not None: self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) def do_hybrid_video_motion(self, data, indexes, reference_images): @@ -61,14 +65,12 @@ def advance_optical_tween_flow(self, step, flow): self.next.image = image_transform_optical_flow(self.next.image, flow, ff) def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_images, step): - if data.args.anim_args.hybrid_motion_use_prev_img: - flow = call_get_flow_for_hybrid_motion_prev(data, indexes.tween.i - 1, reference_images.previous) - turbo.advance_optical_tween_flow(self, step, flow) - data.animation_mode.prev_flow = flow - else: - flow = call_get_flow_for_hybrid_motion(data, indexes.tween.i - 1) - turbo.advance_optical_tween_flow(self, step, flow) - data.animation_mode.prev_flow = flow + last_i = indexes.tween.i - 1 + flow = (call_get_flow_for_hybrid_motion(data, last_i) + if not data.args.anim_args.hybrid_motion_use_prev_img + else call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous)) + turbo.advance_optical_tween_flow(self, step, flow) + data.animation_mode.prev_flow = flow def advance_cadence_flow(self, data, tween_step): ff = data.args.anim_args.cadence_flow_factor_schedule @@ -91,12 +93,11 @@ def advance_ransac_transform(self, data, matrix): # TODO? move to RenderData @staticmethod def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): - if data.args.anim_args.hybrid_motion_use_prev_img: - matrix = call_get_matrix_for_hybrid_motion_prev(data, indexes.tween.i - 1, reference_images.previous) - turbo.advance_ransac_transform(data, matrix) - else: - matrix = call_get_matrix_for_hybrid_motion(data, indexes.tween.i - 1) - turbo.advance_ransac_transform(data, matrix) + last_i = indexes.tween.i - 1 + matrix = (call_get_matrix_for_hybrid_motion(data, last_i) + if not data.args.anim_args.hybrid_motion_use_prev_img + else call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous)) + turbo.advance_ransac_transform(data, matrix) def advance_optical_flow_cadence_before_animation_warping(self, data, last_step, tween_step): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): @@ -112,8 +113,8 @@ def advance_optical_flow_cadence_before_animation_warping(self, data, last_step, self.advance_optical_flow(tween_step) # flow_factor = 1.0 / (last_step.i - tween_step.i() + 1) - # flow_factor = 100.0 / len(last_step.tweens) - flow_factor = 100.0 + flow_factor = 100.0 / len(last_step.tweens) + # flow_factor = 100.0 if tween_step.cadence_flow is not None: self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) @@ -122,9 +123,9 @@ def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_s return self.next.image if tween_step.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abs conversions). - temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) - new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) - # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) + # temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) + # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) + new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) tween_step.cadence_flow = new_flow abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) tween_step.cadence_flow_inc = abs_flow * tween_step.tween @@ -132,8 +133,7 @@ def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_s self.prev.index = self.next.frame_idx = indexes.tween.i if indexes is not None else 0 if self.prev.image is not None and tween_step.tween < 1.0: return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween - else: - return self.next.image + return self.next.image def progress_step(self, indexes, opencv_image): self.prev.image, self.prev.index = self.next.image, self.next.index @@ -149,12 +149,12 @@ def _set_up_step_vars(self, data): def find_start(self, data) -> int: """Maybe resume animation (requires at least two frames - see function).""" - # set start_frame to next frame if data.is_resuming_from_timestring(): + # set start_frame to next frame self._set_up_step_vars(data) - return self.next.index + 1 - else: - return 0 + # instead of "self.next.index + 1" we always return 0, to print a message + # for every frame that is skipped because it already exists. + return 0 def has_steps(self): return self.steps > 1 diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index ca550a325..f62106e8f 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -1,5 +1,6 @@ import os +import PIL import cv2 from cv2.typing import MatLike @@ -24,3 +25,7 @@ def save_cadence_frame_and_depth_map_if_active(data: RenderData, i, image): def save_and_return_frame(data: RenderData, i, image): save_cadence_frame_and_depth_map_if_active(data, i, image) return image + + +def is_PIL(image): + return type(image) is PIL.Image.Image diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index fc5b89ed5..8d998cb9c 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,5 +1,8 @@ from ... import generate +# noinspection PyUnresolvedReferences +from modules.shared import opts + COLOUR_RGB = '\x1b[38;2;%d;%d;%dm' RED = "\033[31m" ORANGE = "\033[38;5;208m" @@ -18,6 +21,11 @@ RESET = "\x1b[0m" +def is_verbose(): + """Checks if extra console output is enabled in deforum settings.""" + return opts.data.get("deforum_debug_mode_enabled", False) + + def print_tween_frame_from_to_info(key_step, is_disabled=True): if not is_disabled: # replaced with prog bar, but value info print may be useful tween_values = key_step.tween_values @@ -30,7 +38,6 @@ def print_tween_frame_from_to_info(key_step, is_disabled=True): def print_animation_frame_info(i, max_frames): - print() # skipping out of the progress bar. print(f"{CYAN}Animation frame: {RESET}{i}/{max_frames}") @@ -55,6 +62,19 @@ def print_redo_generation_info(data, n): print(f"Redo generation {n + 1} of {int(data.args.anim_args.diffusion_redo)} before final generation") +def print_tween_step_creation_info(key_steps, index_dist): + tween_count = sum(len(ks.tweens) for ks in key_steps) + msg_start = f"Created {len(key_steps)} key frames with {tween_count} tweens." + msg_end = f"Key frame index distribution: '{index_dist.name}'." + info(f"{msg_start} {msg_end}") + + +def print_key_step_debug_info_if_verbose(key_steps): + for i, ks in enumerate(key_steps): + tween_indices = [t.i() for t in ks.tweens] + debug(f"Key frame {ks.i} has {len(tween_indices)} tweens: {tween_indices}") + + def print_warning_generate_returned_no_image(): print(f"{YELLOW}Warning: {RESET}Generate returned no image. Skipping to next iteration.") @@ -68,8 +88,9 @@ def warn(s: str): def debug(s: str): - eye_catcher = "###" - print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") + if is_verbose(): + eye_catcher = "###" + print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") def with_suppressed_table_printing(func): From a2a527cfcf3d867fe7f636cac95cf4b680aff14c Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 13 Jul 2024 18:41:54 +0200 Subject: [PATCH 108/132] Renamed key step to key frame and renamed step package to frame so it's not confused with diffusion steps. --- scripts/deforum_helpers/render.py | 32 +-- .../rendering/data/frame/__init__.py | 3 + .../{step/key_step.py => frame/key_frame.py} | 237 +++++++++--------- .../key_frame_distribution.py} | 24 +- .../tween_step.py => frame/tween_frame.py} | 0 .../rendering/data/step/__init__.py | 3 - .../rendering/img_2_img_tubes.py | 54 ++-- 7 files changed, 177 insertions(+), 176 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/frame/__init__.py rename scripts/deforum_helpers/rendering/data/{step/key_step.py => frame/key_frame.py} (92%) rename scripts/deforum_helpers/rendering/data/{step/key_index_distribution.py => frame/key_frame_distribution.py} (87%) rename scripts/deforum_helpers/rendering/data/{step/tween_step.py => frame/tween_frame.py} (100%) delete mode 100644 scripts/deforum_helpers/rendering/data/step/__init__.py diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 153860d86..5aedc28dd 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -23,7 +23,7 @@ from .rendering import img_2_img_tubes from .rendering.data.render_data import RenderData -from .rendering.data.step import KeyIndexDistribution, KeyStep +from .rendering.data.frame import KeyFrameDistribution, KeyFrame from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils @@ -36,42 +36,42 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro def run_render_animation(data: RenderData): web_ui_utils.init_job(data) start_index = data.turbo.find_start(data) - key_steps = KeyStep.create_all_steps(data, start_index, KeyIndexDistribution.from_UI_tab(data)) + key_frames = KeyFrame.create_all_steps(data, start_index, KeyFrameDistribution.from_UI_tab(data)) - for key_step in key_steps: - if is_resume(data, key_step): + for key_frame in key_frames: + if is_resume(data, key_frame): continue memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - if len(key_step.tweens) > 0: - emit_tweens(data, key_step) + if key_frame.has_tween_frames(): + emit_tweens(data, key_frame) - log_utils.print_animation_frame_info(key_step.i, data.args.anim_args.max_frames) - key_step.maybe_write_frame_subtitle() + log_utils.print_animation_frame_info(key_frame.i, data.args.anim_args.max_frames) + key_frame.maybe_write_frame_subtitle() frame_tube = img_2_img_tubes.frame_transformation_tube contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - key_step.prepare_generation(frame_tube, contrasted_noise_tube) + key_frame.prepare_generation(frame_tube, contrasted_noise_tube) - image = key_step.do_generation() + image = key_frame.do_generation() if image is None: log_utils.print_warning_generate_returned_no_image() break if not image_utils.is_PIL(image): # check is required when resuming from timestring - image = img_2_img_tubes.conditional_frame_transformation_tube(key_step)(image) + image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) - key_step.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_step)(image) + key_frame.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_frame)(image) - key_step.progress_and_save(image) + key_frame.progress_and_save(image) state.assign_current_image(image) - key_step.render_data.args.args.seed = key_step.next_seed() + key_frame.render_data.args.args.seed = key_frame.next_seed() - key_step.update_render_preview() - web_ui_utils.update_status_tracker(key_step.render_data) + key_frame.update_render_preview() + web_ui_utils.update_status_tracker(key_frame.render_data) data.animation_mode.unload_raft_and_depth_model() diff --git a/scripts/deforum_helpers/rendering/data/frame/__init__.py b/scripts/deforum_helpers/rendering/data/frame/__init__.py new file mode 100644 index 000000000..38d6f4344 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/frame/__init__.py @@ -0,0 +1,3 @@ +from .key_frame_distribution import KeyFrameDistribution +from .key_frame import KeyFrameData, KeyFrame +from .tween_frame import Tween diff --git a/scripts/deforum_helpers/rendering/data/step/key_step.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py similarity index 92% rename from scripts/deforum_helpers/rendering/data/step/key_step.py rename to scripts/deforum_helpers/rendering/data/frame/key_frame.py index 8f4d0dc85..1ebb379cf 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_step.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -6,8 +6,8 @@ import numpy as np from PIL import Image -from . import KeyIndexDistribution -from .tween_step import Tween +from . import KeyFrameDistribution +from .tween_frame import Tween from ..render_data import RenderData from ..schedule import Schedule from ... import img_2_img_tubes @@ -27,7 +27,7 @@ @dataclass(init=True, frozen=False, repr=False, eq=False) -class KeyStepData: +class KeyFrameData: noise: Any = None strength: Any = None scale: Any = None @@ -52,7 +52,7 @@ def has_strength(self): @staticmethod def create(deform_keys, i): keys = deform_keys - return KeyStepData( + return KeyFrameData( keys.noise_schedule_series[i], keys.strength_schedule_series[i], keys.cfg_scale_schedule_series[i], @@ -63,7 +63,7 @@ def create(deform_keys, i): keys.threshold_schedule_series[i], keys.cadence_flow_factor_schedule_series[i], keys.redo_flow_factor_schedule_series[i], - KeyStepData._hybrid_comp_args(keys, i)) + KeyFrameData._hybrid_comp_args(keys, i)) @staticmethod def _hybrid_comp_args(keys, i): @@ -77,10 +77,10 @@ def _hybrid_comp_args(keys, i): @dataclass(init=True, frozen=False, repr=False, eq=False) -class KeyStep: +class KeyFrame: """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" i: int - step_data: KeyStepData + step_data: KeyFrameData render_data: RenderData schedule: Schedule depth: Any # TODO try to init early, then freeze class @@ -90,70 +90,8 @@ class KeyStep: tweens: List[Tween] tween_values: List[float] - @staticmethod - def create(data: RenderData): - step_data = KeyStepData.create(data.animation_keys.deform_keys, data.indexes.frame.i) - schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) - return KeyStep(0, step_data, data, schedule, None, None, "", 0, list(), list()) - - @staticmethod - def create_all_steps(data, start_index, index_dist: KeyIndexDistribution = KeyIndexDistribution.default()): - """Creates a list of key steps for the entire animation.""" - max_frames = data.args.anim_args.max_frames - - num_key_steps = 1 + int((max_frames - start_index) / data.cadence()) - if data.parseq_adapter.use_parseq and index_dist is KeyIndexDistribution.PARSEQ_ONLY: - num_key_steps = len(data.parseq_adapter.parseq_json["keyframes"]) - - key_steps = [KeyStep.create(data) for _ in range(0, num_key_steps)] - actual_num_key_steps = len(key_steps) - #data.args.anim_args.max_frames = actual_num_key_steps - #max_frames = actual_num_key_steps - - key_steps = KeyStep._recalculate_and_check_tweens( # TODO remove actual_num_key_steps arg - key_steps, start_index, max_frames, actual_num_key_steps, data.parseq_adapter, index_dist) - log_utils.print_tween_step_creation_info(key_steps, index_dist) - - return key_steps - - @staticmethod - def _recalculate_and_check_tweens(key_steps, start_index, max_frames, num_key_steps, - parseq_adapter, index_distribution): - key_indices: List[int] = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) - for i, key_step in enumerate(key_indices): - key_steps[i].i = key_indices[i] - - key_steps = KeyStep._add_tweens_to_key_steps(key_steps) - log_utils.print_key_step_debug_info_if_verbose(key_steps) - - # The number of generated tweens depends on index since last key-frame. The last tween has the same - # index as the key_step it belongs to and is meant to replace the unprocessed original key frame. - # TODO? make unit tests instead of asserts... - # total_count = len(key_steps) + sum(len(key_step.tweens) - 1 for key_step in key_steps) - # log_utils.info(f"total_count {total_count} len(key_steps) {len(key_steps)} max_frames {max_frames}") - # assert total_count == len(key_steps) + max_frames # every key frame except the 1st has a tween double. - - assert len(key_steps) == num_key_steps - assert key_steps[0].tweens == [] # 1st key step has no tweens - assert key_steps[0].i == 1 - if index_distribution != KeyIndexDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. - assert key_steps[-1].i == max_frames - - return key_steps - - @staticmethod - def _add_tweens_to_key_steps(key_steps): - log_utils.info(f"Adding tweens to {len(key_steps)} keyframes...") - for i in range(1, len(key_steps)): # skipping 1st key frame - data = key_steps[i].render_data - from_i = key_steps[i - 1].i - to_i = key_steps[i].i - tweens, values = Tween.create_in_between_steps(key_steps, i, data, from_i, to_i) - log_utils.debug(f"Creating {len(tweens)} tweens ({from_i}->{to_i}) for key frame at {key_steps[i].i}") - key_steps[i].tweens = tweens - key_steps[i].tween_values = values - key_steps[i].render_data.indexes.update_tween_start(data.turbo) - return key_steps + def has_tween_frames(self): + return len(self.tweens) > 0 def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, images): has_flow_redo = optical_flow_redo_generation != 'None' @@ -212,53 +150,6 @@ def apply_frame_noising(self, data: RenderData, mask, image): data.args.root.noise_mask = call_compose_mask_with_check(data, seq, vals, contrast_image) return call_add_noise(data, self, image) - @staticmethod - def apply_color_matching(data: RenderData, image): - return apply_color_coherence(image, data) if data.has_color_coherence() else image - - @staticmethod - def apply_color_coherence(image, data: RenderData): - if data.images.color_match is None: - # Initialize color_match for next iteration with current image, but don't do anything yet. - if image is not None: - data.images.color_match = image.copy() - return image - return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) - - @staticmethod - def transform_to_grayscale_if_active(data: RenderData, image): - if data.args.anim_args.color_force_grayscale: - grayscale = cv2.cvtColor(data.images.previous, cv2.COLOR_BGR2GRAY) - return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) - return image - - @staticmethod - def apply_hybrid_motion_ransac_transform(data: RenderData, image): - """hybrid video motion - warps `images.previous` to match motion, usually to prepare for compositing""" - motion = data.args.anim_args.hybrid_motion - if motion in ['Affine', 'Perspective']: - last_i = data.indexes.frame.i - 1 - reference_images = data.images - matrix = call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ - if data.args.anim_args.hybrid_motion_use_prev_img \ - else call_get_matrix_for_hybrid_motion(data, last_i) - return image_transform_ransac(image, matrix, data.args.anim_args.hybrid_motion) - return image - - @staticmethod - def apply_hybrid_motion_optical_flow(data: RenderData, image): - motion = data.args.anim_args.hybrid_motion - if motion in ['Optical Flow']: - last_i = data.indexes.frame.i - 1 - reference_images = data.images - flow = call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ - if data.args.anim_args.hybrid_motion_use_prev_img \ - else call_get_flow_for_hybrid_motion(data, last_i) - transformed = image_transform_optical_flow(images.previous, flow, step.step_data.flow_factor()) - data.animation_mode.prev_flow = flow # side effect - return transformed - return image - def create_color_match_for_video(self): data = self.render_data if data.args.anim_args.color_coherence == 'Video Input' and data.is_hybrid_available(): @@ -388,3 +279,113 @@ def do_diffusion_redo(self): diffusion_redo_image = maintain_colors(data.images.previous, data.images.color_match, mode) data.args.args.seed = stored_seed data.args.root.init_sample = Image.fromarray(cv2.cvtColor(diffusion_redo_image, cv2.COLOR_BGR2RGB)) + + @staticmethod + def create(data: RenderData): + step_data = KeyFrameData.create(data.animation_keys.deform_keys, data.indexes.frame.i) + schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) + return KeyFrame(0, step_data, data, schedule, None, None, "", 0, list(), list()) + + @staticmethod + def apply_color_matching(data: RenderData, image): + return apply_color_coherence(image, data) if data.has_color_coherence() else image + + @staticmethod + def apply_color_coherence(image, data: RenderData): + if data.images.color_match is None: + # Initialize color_match for next iteration with current image, but don't do anything yet. + if image is not None: + data.images.color_match = image.copy() + return image + return maintain_colors(image, data.images.color_match, data.args.anim_args.color_coherence) + + @staticmethod + def transform_to_grayscale_if_active(data: RenderData, image): + if data.args.anim_args.color_force_grayscale: + grayscale = cv2.cvtColor(data.images.previous, cv2.COLOR_BGR2GRAY) + return cv2.cvtColor(grayscale, cv2.COLOR_GRAY2BGR) + return image + + @staticmethod + def apply_hybrid_motion_ransac_transform(data: RenderData, image): + """hybrid video motion - warps `images.previous` to match motion, usually to prepare for compositing""" + motion = data.args.anim_args.hybrid_motion + if motion in ['Affine', 'Perspective']: + last_i = data.indexes.frame.i - 1 + reference_images = data.images + matrix = call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ + if data.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_matrix_for_hybrid_motion(data, last_i) + return image_transform_ransac(image, matrix, data.args.anim_args.hybrid_motion) + return image + + @staticmethod + def apply_hybrid_motion_optical_flow(data: RenderData, image): + motion = data.args.anim_args.hybrid_motion + if motion in ['Optical Flow']: + last_i = data.indexes.frame.i - 1 + reference_images = data.images + flow = call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ + if data.args.anim_args.hybrid_motion_use_prev_img \ + else call_get_flow_for_hybrid_motion(data, last_i) + transformed = image_transform_optical_flow(images.previous, flow, step.step_data.flow_factor()) + data.animation_mode.prev_flow = flow # side effect + return transformed + return image + + @staticmethod + def create_all_steps(data, start_index, index_dist: KeyFrameDistribution = KeyFrameDistribution.default()): + """Creates a list of key steps for the entire animation.""" + num_key_steps = 1 + int((data.args.anim_args.max_frames - start_index) / data.cadence()) + if data.parseq_adapter.use_parseq and index_dist is KeyFrameDistribution.PARSEQ_ONLY: + num_key_steps = len(data.parseq_adapter.parseq_json["keyframes"]) + + key_steps = [KeyFrame.create(data) for _ in range(0, num_key_steps)] + actual_num_key_steps = len(key_steps) + + recalculated_key_steps = KeyFrame._recalculate_and_check_tweens(data, key_steps, + start_index, actual_num_key_steps, + data.parseq_adapter, index_dist) + log_utils.print_tween_step_creation_info(key_steps, index_dist) + + return recalculated_key_steps + + @staticmethod + def _recalculate_and_check_tweens(data, key_steps, start_index, num_key_steps, + parseq_adapter, index_distribution): + max_frames = data.args.anim_args.max_frames + key_indices: List[int] = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) + for i, key_step in enumerate(key_indices): + key_steps[i].i = key_indices[i] + + key_steps = KeyFrame._add_tweens_to_key_steps(key_steps) + log_utils.print_key_step_debug_info_if_verbose(key_steps) + + # The number of generated tweens depends on index since last key-frame. The last tween has the same + # index as the key_step it belongs to and is meant to replace the unprocessed original key frame. + # TODO? make unit tests instead of asserts... + # total_count = len(key_steps) + sum(len(key_step.tweens) - 1 for key_step in key_steps) + # log_utils.info(f"total_count {total_count} len(key_steps) {len(key_steps)} max_frames {max_frames}") + # assert total_count == len(key_steps) + max_frames # every key frame except the 1st has a tween double. + + assert len(key_steps) == num_key_steps + assert key_steps[0].tweens == [] # 1st key step has no tweens + assert key_steps[0].i == 1 + if index_distribution != KeyFrameDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. + assert key_steps[-1].i == max_frames + + return key_steps + + @staticmethod + def _add_tweens_to_key_steps(key_steps): + log_utils.info(f"Adding tweens to {len(key_steps)} keyframes...") + for i in range(1, len(key_steps)): # skipping 1st key frame + data = key_steps[i].render_data + from_i = key_steps[i - 1].i + to_i = key_steps[i].i + tweens, values = Tween.create_in_between_steps(key_steps, i, data, from_i, to_i) + log_utils.debug(f"Creating {len(tweens)} tweens ({from_i}->{to_i}) for key frame at {key_steps[i].i}") + key_steps[i].tweens = tweens + key_steps[i].tween_values = values + key_steps[i].render_data.indexes.update_tween_start(data.turbo) + return key_steps diff --git a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py similarity index 87% rename from scripts/deforum_helpers/rendering/data/step/key_index_distribution.py rename to scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py index b1118afae..98c1341df 100644 --- a/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py @@ -5,7 +5,7 @@ from ...util import log_utils -class KeyIndexDistribution(Enum): +class KeyFrameDistribution(Enum): PARSEQ_ONLY = "Parseq Only" # cadence is ignored. all frames not present in the Parseq table are handled as tweens. UNIFORM_WITH_PARSEQ = "Uniform with Parseq" # similar to uniform, but parseq key frame diffusion is enforced. UNIFORM_SPACING = "Uniform Spacing" # distance defined by cadence @@ -15,25 +15,25 @@ class KeyIndexDistribution(Enum): @staticmethod def from_UI_tab(data): is_uniform_with_parseq = data.args.anim_args.neocore_key_index_distribution == "Uniform with Parseq" - return (KeyIndexDistribution.UNIFORM_WITH_PARSEQ + return (KeyFrameDistribution.UNIFORM_WITH_PARSEQ if is_uniform_with_parseq - else KeyIndexDistribution.PARSEQ_ONLY) + else KeyFrameDistribution.PARSEQ_ONLY) @staticmethod def default(): - return KeyIndexDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. + return KeyFrameDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: match self: - case KeyIndexDistribution.PARSEQ_ONLY: + case KeyFrameDistribution.PARSEQ_ONLY: return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) - case KeyIndexDistribution.UNIFORM_WITH_PARSEQ: + case KeyFrameDistribution.UNIFORM_WITH_PARSEQ: return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) - case KeyIndexDistribution.UNIFORM_SPACING: + case KeyFrameDistribution.UNIFORM_SPACING: return self._uniform_indexes(start_index, max_frames, num_key_steps) - case KeyIndexDistribution.RANDOM_SPACING: + case KeyFrameDistribution.RANDOM_SPACING: return self._random_spacing_indexes(start_index, max_frames, num_key_steps) - case KeyIndexDistribution.RANDOM_PLACEMENT: + case KeyFrameDistribution.RANDOM_PLACEMENT: return self._random_placement_indexes(start_index, max_frames, num_key_steps) case _: raise ValueError(f"Invalid KeyIndexDistribution: {self}") @@ -48,7 +48,7 @@ def _parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) """Only Parseq key frames are used. Cadence settings are ignored.""" if not parseq_adapter.use_parseq: log_utils.warn("PARSEQ_ONLY, but Parseq is not active, using UNIFORM_SPACING instead.") - return KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + return KeyFrameDistribution._uniform_indexes(start_index, max_frames, num_key_steps) parseq_key_frames = [keyframe["frame"] for keyframe in parseq_adapter.parseq_json["keyframes"]] shifted_parseq_frames = [frame + 1 for frame in parseq_key_frames] @@ -57,7 +57,7 @@ def _parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) @staticmethod def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter): """Calculates uniform indices according to cadence, but parseq key frames replace the closest deforum key.""" - uniform_indices = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + uniform_indices = KeyFrameDistribution._uniform_indexes(start_index, max_frames, num_key_steps) if not parseq_adapter.use_parseq: log_utils.warn("UNIFORM_WITH_PARSEQ, but Parseq is not active, using UNIFORM_SPACING instead.") return uniform_indices @@ -81,7 +81,7 @@ def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_ @staticmethod def _random_spacing_indexes(start_index, max_frames, num_key_steps): - uniform_indexes = KeyIndexDistribution._uniform_indexes(start_index, max_frames, num_key_steps) + uniform_indexes = KeyFrameDistribution._uniform_indexes(start_index, max_frames, num_key_steps) indexes = [start_index + 1, max_frames] # Enforce first and last indices total_spacing = max_frames - start_index - 1 # Calculate initial total spacing noise_factor = 0.5 # Higher value creates more variation diff --git a/scripts/deforum_helpers/rendering/data/step/tween_step.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py similarity index 100% rename from scripts/deforum_helpers/rendering/data/step/tween_step.py rename to scripts/deforum_helpers/rendering/data/frame/tween_frame.py diff --git a/scripts/deforum_helpers/rendering/data/step/__init__.py b/scripts/deforum_helpers/rendering/data/step/__init__.py deleted file mode 100644 index e55d647fd..000000000 --- a/scripts/deforum_helpers/rendering/data/step/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -from .key_index_distribution import KeyIndexDistribution -from .key_step import KeyStepData, KeyStep -from .tween_step import Tween diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 6421e090f..3957bb6e4 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -5,7 +5,7 @@ from cv2.typing import MatLike from .data.render_data import RenderData -from .data.step.key_step import KeyStep +from .data.frame.key_frame import KeyFrame from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube from ..colors import maintain_colors @@ -28,24 +28,24 @@ ImageTube = Callable[[MatLike], MatLike] -def frame_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: +def frame_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: # make sure `img` stays the last argument in each call. - return tube(lambda img: step.apply_frame_warp_transform(data, img), - lambda img: step.do_hybrid_compositing_before_motion(data, img), - lambda img: KeyStep.apply_hybrid_motion_ransac_transform(data, img), - lambda img: KeyStep.apply_hybrid_motion_optical_flow(data, img), - lambda img: step.do_normal_hybrid_compositing_after_motion(data, img), - lambda img: KeyStep.apply_color_matching(data, img), - lambda img: KeyStep.transform_to_grayscale_if_active(data, img)) + return tube(lambda img: key_frame.apply_frame_warp_transform(data, img), + lambda img: key_frame.do_hybrid_compositing_before_motion(data, img), + lambda img: KeyFrame.apply_hybrid_motion_ransac_transform(data, img), + lambda img: KeyFrame.apply_hybrid_motion_optical_flow(data, img), + lambda img: key_frame.do_normal_hybrid_compositing_after_motion(data, img), + lambda img: KeyFrame.apply_color_matching(data, img), + lambda img: KeyFrame.transform_to_grayscale_if_active(data, img)) -def contrast_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: - return tube(lambda img: step.apply_scaling(img), - lambda img: step.apply_anti_blur(data, img)) +def contrast_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: + return tube(lambda img: key_frame.apply_scaling(img), + lambda img: key_frame.apply_anti_blur(data, img)) -def noise_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: - return tube(lambda img: step.apply_frame_noising(data, step, img)) +def noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: + return tube(lambda img: key_frame.apply_frame_noising(data, key_frame, img)) def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: @@ -57,9 +57,9 @@ def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). -def conditional_hybrid_video_after_generation_tube(step: KeyStep) -> ImageTube: - data = step.render_data - step_data = step.step_data +def conditional_hybrid_video_after_generation_tube(key_frame: KeyFrame) -> ImageTube: + data = key_frame.render_data + step_data = key_frame.step_data return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), @@ -78,10 +78,10 @@ def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: lambda: data.indexes.is_first_frame() and data.is_initialize_color_match(data.images.color_match)) -def conditional_color_match_tube(step: KeyStep) -> ImageTube: +def conditional_color_match_tube(key_frame: KeyFrame) -> ImageTube: # on strength 0, set color match to generation return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), - is_do_process=lambda: step.render_data.is_do_color_match_conversion(step)) + is_do_process=lambda: key_frame.render_data.is_do_color_match_conversion(key_frame)) def conditional_force_to_grayscale_tube(data: RenderData) -> ImageTube: @@ -107,16 +107,16 @@ def conditional_force_tween_to_grayscale_tube(data: RenderData) -> ImageTube: # Composite Tubes, made from other Tubes. -def contrasted_noise_transformation_tube(data: RenderData, step: KeyStep) -> ImageTube: +def contrasted_noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: """Combines contrast and noise transformation tubes.""" - contrast_tube: Tube = contrast_transformation_tube(data, step) - noise_tube: Tube = noise_transformation_tube(data, step) + contrast_tube: Tube = contrast_transformation_tube(data, key_frame) + noise_tube: Tube = noise_transformation_tube(data, key_frame) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(step: KeyStep, is_tween: bool = False) -> ImageTube: - hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(step) - extra_tube: Tube = conditional_extra_color_match_tube(step.render_data) - gray_tube: Tube = conditional_force_to_grayscale_tube(step.render_data) - mask_tube: Tube = conditional_add_overlay_mask_tube(step.render_data, is_tween) +def conditional_frame_transformation_tube(key_frame: KeyFrame, is_tween: bool = False) -> ImageTube: + hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(key_frame) + extra_tube: Tube = conditional_extra_color_match_tube(key_frame.render_data) + gray_tube: Tube = conditional_force_to_grayscale_tube(key_frame.render_data) + mask_tube: Tube = conditional_add_overlay_mask_tube(key_frame.render_data, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) From abf20f3d3ece2cefc067ce00b8c6531064148683 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 13 Jul 2024 19:43:22 +0200 Subject: [PATCH 109/132] More dataclasses frozen. Prepared fixing color coherence & more cleanup. --- .../rendering/data/anim/animation_keys.py | 5 +- .../rendering/data/anim/animation_mode.py | 5 +- .../rendering/data/frame/key_frame.py | 8 +-- .../rendering/data/frame/tween_frame.py | 32 ++++++------ .../deforum_helpers/rendering/data/turbo.py | 50 +++++++++---------- .../rendering/img_2_img_tubes.py | 7 +-- .../rendering/util/log_utils.py | 10 +++- .../rendering/util/memory_utils.py | 2 - 8 files changed, 60 insertions(+), 59 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py index 8697218e6..efce69056 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_keys.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_keys.py @@ -3,14 +3,11 @@ from ....animation_key_frames import DeformAnimKeys, LooperAnimKeys -@dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class AnimationKeys: deform_keys: DeformAnimKeys looper_keys: LooperAnimKeys - def update_looper(self, loop_args, anim_args, seed): - self.looper_keys = LooperAnimKeys(loop_args, anim_args, seed) - @staticmethod def _choose_default_or_parseq_keys(default_keys, parseq_keys, parseq_adapter): return default_keys if not parseq_adapter.use_parseq else parseq_keys diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 56808696d..4ffc864bb 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -8,13 +8,12 @@ from ....hybrid_video import hybrid_generation -# TODO FIXME find a way to assign prev_flow right away, then set frozen=true again, otherwise move prev_flow elsewhere. -@dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None hybrid_frame_path: str = "" - prev_flow: Any = None + prev_flow: Any | None = None is_keep_in_vram: bool = False depth_model: Any = None raft_model: Any = None diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index 1ebb379cf..ebc5188fe 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -26,7 +26,7 @@ from ....seed import next_seed -@dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class KeyFrameData: noise: Any = None strength: Any = None @@ -257,7 +257,7 @@ def do_optical_flow_redo_before_generation(self): log_utils.print_optical_flow_info(data, redo) # TODO output temp seed? sample_image = call_generate(data, self) - optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, redo) + optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, self, redo) transformed_sample_image = optical_tube(sample_image) data.args.args.seed = stored_seed # restore stored seed @@ -268,7 +268,7 @@ def do_diffusion_redo(self): stored_seed = data.args.args.seed last_diffusion_redo_index = int(data.args.anim_args.diffusion_redo) for n in range(0, last_diffusion_redo_index): - print_redo_generation_info(data, n) + log_utils.print_redo_generation_info(data, n) data.args.args.seed = utils.generate_random_seed() diffusion_redo_image = call_generate(data, self) diffusion_redo_image = cv2.cvtColor(np.array(diffusion_redo_image), cv2.COLOR_RGB2BGR) @@ -288,7 +288,7 @@ def create(data: RenderData): @staticmethod def apply_color_matching(data: RenderData, image): - return apply_color_coherence(image, data) if data.has_color_coherence() else image + return KeyFrame.apply_color_coherence(image, data) if data.has_color_coherence() else image @staticmethod def apply_color_coherence(image, data: RenderData): diff --git a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py index 6adf61057..fe55dcf12 100644 --- a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py @@ -17,7 +17,7 @@ class Tween: cadence_flow: Any # late init cadence_flow_inc: Any # late init depth: Any - depth_prediction: Any + depth_prediction: Any # reassigned def i(self): return self.indexes.tween.i @@ -28,16 +28,16 @@ def from_key_step_i(self): def to_key_step_i(self): return self.indexes.frame.i - def emit_frame(self, last_step, grayscale_tube, overlay_mask_tube): + def emit_frame(self, last_frame, grayscale_tube, overlay_mask_tube): """Emits this tween frame.""" - max_frames = last_step.render_data.args.anim_args.max_frames + max_frames = last_frame.render_data.args.anim_args.max_frames if self.i() >= max_frames: return # skipping tween emission on the last frame - data = last_step.render_data + data = last_frame.render_data # data.turbo.steps = len(last_step.tweens) self.handle_synchronous_status_concerns(data) - self.process(last_step, data) + self.process(last_frame, data) new_image = self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes @@ -54,12 +54,12 @@ def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): masked = overlay_mask_tube(data, is_tween)(recolored) return masked - def process(self, last_step, data): - data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_step, self) + def process(self, last_frame, data): + data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_frame, self) self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) # log_utils.info(f"self.depth_prediction {self.depth_prediction}") data.turbo.advance(data, self.indexes.tween.i, self.depth) - data.turbo.do_hybrid_video_motion(data, self.indexes, data.images) # FIXME? remove self.indexes or init.indexes + data.turbo.do_hybrid_video_motion(data, last_frame, self.indexes, data.images) def handle_synchronous_status_concerns(self, data): self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? @@ -74,13 +74,13 @@ def write_tween_frame_subtitle_if_active(self, data: RenderData): call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) @staticmethod - def create_in_between_steps(key_steps, i, data, from_i, to_i): + def create_in_between_steps(key_frames, i, data, from_i, to_i): tween_range = range(from_i, to_i) tween_indexes_list: List[Indexes] = Tween.create_indexes(data.indexes, tween_range) - last_step = key_steps[i] + last_step = key_frames[i] tween_steps_and_values = Tween.create_steps(last_step, tween_indexes_list) for tween in tween_steps_and_values[0]: - tween.indexes.update_tween_index(tween.i() + key_steps[i].i) + tween.indexes.update_tween_index(tween.i() + key_frames[i].i) return tween_steps_and_values @staticmethod @@ -98,21 +98,21 @@ def _increment(original_indexes, tween_count, from_start): return original_indexes @staticmethod - def create_steps_from_values(last_step, values): + def create_steps_from_values(last_frame, values): count = len(values) r = range(count) - indexes_list = [Tween._increment(last_step.render_data.indexes.copy(), count, i + 1) for i in r] - return list((Tween(indexes_list[i], values[i], None, None, last_step.depth, None) for i in r)) + indexes_list = [Tween._increment(last_frame.render_data.indexes.copy(), count, i + 1) for i in r] + return list((Tween(indexes_list[i], values[i], None, None, last_frame.depth, None) for i in r)) @staticmethod def create_indexes(base_indexes: Indexes, frame_range: Iterable[int]) -> list[Indexes]: return list(chain.from_iterable([Indexes.create_from_last(base_indexes, i)] for i in frame_range)) @staticmethod - def create_steps(last_step, tween_indexes_list: list[Indexes]) -> Tuple[list['Tween'], list[float]]: + def create_steps(last_frame, tween_indexes_list: list[Indexes]) -> Tuple[list['Tween'], list[float]]: if len(tween_indexes_list) > 0: expected_tween_frames = Tween._calculate_expected_tween_frames(len(tween_indexes_list)) - return Tween.create_steps_from_values(last_step, expected_tween_frames), expected_tween_frames + return Tween.create_steps_from_values(last_frame, expected_tween_frames), expected_tween_frames return list(), list() @staticmethod diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 731490b47..7daf0c64a 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -40,7 +40,7 @@ def advance(self, data, i: int, depth): if self.next.image is not None: self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) - def do_hybrid_video_motion(self, data, indexes, reference_images): + def do_hybrid_video_motion(self, data, last_frame, indexes, reference_images): """Warps the previous and/or the next to match the motion of the provided reference images.""" motion = data.args.anim_args.hybrid_motion @@ -50,32 +50,32 @@ def _is_do_motion(motions): if _is_do_motion(['Affine', 'Perspective']): Turbo.advance_hybrid_motion_ransac_transform(data, indexes, reference_images) if _is_do_motion(['Optical Flow']): - self.advance_hybrid_motion_optical_tween_flow(data, indexes, reference_images, step) + self.advance_hybrid_motion_optical_tween_flow(data, indexes, reference_images, last_frame) def advance_optical_flow(self, tween_step, flow_factor: int = 1): flow = tween_step.cadence_flow * -1 self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) - def advance_optical_tween_flow(self, step, flow): - ff = step.step_data.flow_factor() + def advance_optical_tween_flow(self, last_frame, flow): + ff = last_frame.step_data.flow_factor() i = indexes.tween.i if self.is_advance_prev(i): self.prev.image = image_transform_optical_flow(self.prev.image, flow, ff) if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, ff) - def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_images, step): + def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_images, last_frame): last_i = indexes.tween.i - 1 flow = (call_get_flow_for_hybrid_motion(data, last_i) if not data.args.anim_args.hybrid_motion_use_prev_img else call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous)) - turbo.advance_optical_tween_flow(self, step, flow) + turbo.advance_optical_tween_flow(self, last_frame, flow) data.animation_mode.prev_flow = flow - def advance_cadence_flow(self, data, tween_step): + def advance_cadence_flow(self, data, tween_frame): ff = data.args.anim_args.cadence_flow_factor_schedule - i = tween_step.i() - inc = tween_step.cadence_flow_inc # FIXME + i = tween_frame.i() + inc = tween_frame.cadence_flow_inc # FIXME if self.is_advance_prev(i): self.prev.image = image_transform_optical_flow(self.prev.image, inc, ff) if self.is_advance_next(i): @@ -99,40 +99,40 @@ def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): else call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous)) turbo.advance_ransac_transform(data, matrix) - def advance_optical_flow_cadence_before_animation_warping(self, data, last_step, tween_step): + def advance_optical_flow_cadence_before_animation_warping(self, data, last_frame, tween_frame): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): i = data.indexes.tween.start has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[i] > 0 has_images = self.prev.image is not None and self.next.image is not None - has_step_and_images = tween_step.cadence_flow is None and has_images + has_step_and_images = tween_frame.cadence_flow is None and has_images if has_tween_schedule and has_step_and_images and data.animation_mode.is_raft_active(): cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) - tween_step.cadence_flow = (flow / 2) - if tween_step.cadence_flow is not None: - self.advance_optical_flow(tween_step) + tween_frame.cadence_flow = (flow / 2) + if tween_frame.cadence_flow is not None: + self.advance_optical_flow(tween_frame) # flow_factor = 1.0 / (last_step.i - tween_step.i() + 1) - flow_factor = 100.0 / len(last_step.tweens) + flow_factor = 100.0 / len(last_frame.tweens) # flow_factor = 100.0 - if tween_step.cadence_flow is not None: - self.next.image = image_transform_optical_flow(self.next.image, -tween_step.cadence_flow, flow_factor) + if tween_frame.cadence_flow is not None: + self.next.image = image_transform_optical_flow(self.next.image, -tween_frame.cadence_flow, flow_factor) - def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_step): + def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_frame): if not data.animation_mode.is_raft_active(): return self.next.image - if tween_step.cadence_flow is not None: + if tween_frame.cadence_flow is not None: # TODO Calculate all increments before running the generation (and try to avoid abs->rel->abs conversions). # temp_flow = abs_flow_to_rel_flow(tween_step.cadence_flow, data.width(), data.height()) # new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, temp_flow, None) new_flow, _ = call_anim_frame_warp(data, indexes.tween.i, self.prev.image, None) - tween_step.cadence_flow = new_flow - abs_flow = rel_flow_to_abs_flow(tween_step.cadence_flow, data.width(), data.height()) - tween_step.cadence_flow_inc = abs_flow * tween_step.tween - self.advance_cadence_flow(data, tween_step) + tween_frame.cadence_flow = new_flow + abs_flow = rel_flow_to_abs_flow(tween_frame.cadence_flow, data.width(), data.height()) + tween_frame.cadence_flow_inc = abs_flow * tween_frame.value + self.advance_cadence_flow(data, tween_frame) self.prev.index = self.next.frame_idx = indexes.tween.i if indexes is not None else 0 - if self.prev.image is not None and tween_step.tween < 1.0: - return self.prev.image * (1.0 - tween_step.tween) + self.next.image * tween_step.tween + if self.prev.image is not None and tween_frame.value < 1.0: + return self.prev.image * (1.0 - tween_frame.value) + self.next.image * tween_frame.value return self.next.image def progress_step(self, indexes, opencv_image): diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 3957bb6e4..0257b35c4 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -4,11 +4,12 @@ import numpy as np from cv2.typing import MatLike -from .data.render_data import RenderData from .data.frame.key_frame import KeyFrame +from .data.render_data import RenderData from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite from .util.fun_utils import tube from ..colors import maintain_colors +from ..hybrid_video import image_transform_optical_flow from ..masks import do_overlay_mask """ @@ -48,12 +49,12 @@ def noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTub return tube(lambda img: key_frame.apply_frame_noising(data, key_frame, img)) -def optical_flow_redo_tube(data: RenderData, optical_flow) -> ImageTube: +def optical_flow_redo_tube(data: RenderData, key_frame: KeyFrame, optical_flow) -> ImageTube: return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( img, call_get_flow_from_images(data, data.images.previous, img, optical_flow), - step.step_data.redo_flow_factor)) + key_frame.step_data.redo_flow_factor)) # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 8d998cb9c..3ed9d2f76 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,8 +1,8 @@ -from ... import generate - # noinspection PyUnresolvedReferences from modules.shared import opts +from ... import generate + COLOUR_RGB = '\x1b[38;2;%d;%d;%dm' RED = "\033[31m" ORANGE = "\033[38;5;208m" @@ -79,6 +79,12 @@ def print_warning_generate_returned_no_image(): print(f"{YELLOW}Warning: {RESET}Generate returned no image. Skipping to next iteration.") +def print_cuda_memory_state(cuda): + for i in range(cuda.device_count()): + print(f"CUDA memory allocated on device {i}: {cuda.memory_allocated(i)} of {cuda.max_memory_allocated(i)}") + print(f"CUDA memory reserved on device {i}: {cuda.memory_reserved(i)} of {cuda.max_memory_reserved(i)}") + + def info(s: str): print(f"Info: {s}") diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index a07390865..de2170185 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -5,8 +5,6 @@ def is_low_or_med_vram(): - # TODO Ideally this should only be called once at the beginning of the render. - # Perhaps add a constant bool to RenderInit. return cmd_opts.lowvram or cmd_opts.medvram # cmd_opts are imported from elsewhere. keep readonly From 7d8584dd686de75775de762427cf58ba9946a002 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 13 Jul 2024 23:22:19 +0200 Subject: [PATCH 110/132] More cleanup and fixes. --- scripts/deforum_helpers/render.py | 22 +++++--------- .../rendering/data/anim/animation_mode.py | 2 +- .../rendering/data/frame/key_frame.py | 30 +++++++++---------- .../rendering/data/frame/tween_frame.py | 4 +-- .../deforum_helpers/rendering/data/images.py | 8 ++++- .../deforum_helpers/rendering/data/mask.py | 2 +- .../rendering/data/render_data.py | 9 +++--- .../rendering/data/schedule.py | 7 ++--- .../deforum_helpers/rendering/data/turbo.py | 21 ++++++------- .../rendering/img_2_img_tubes.py | 22 +++++++------- .../rendering/util/image_utils.py | 8 ++--- 11 files changed, 64 insertions(+), 71 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 5aedc28dd..9d7d2f6bc 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -22,8 +22,8 @@ from tqdm import tqdm from .rendering import img_2_img_tubes -from .rendering.data.render_data import RenderData from .rendering.data.frame import KeyFrameDistribution, KeyFrame +from .rendering.data.render_data import RenderData from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils @@ -35,16 +35,13 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro # @log_utils.with_suppressed_table_printing def run_render_animation(data: RenderData): web_ui_utils.init_job(data) - start_index = data.turbo.find_start(data) - key_frames = KeyFrame.create_all_steps(data, start_index, KeyFrameDistribution.from_UI_tab(data)) - + key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) for key_frame in key_frames: if is_resume(data, key_frame): continue memory_utils.handle_med_or_low_vram_before_step(data) web_ui_utils.update_job(data) - if key_frame.has_tween_frames(): emit_tweens(data, key_frame) @@ -55,7 +52,7 @@ def run_render_animation(data: RenderData): contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube key_frame.prepare_generation(frame_tube, contrasted_noise_tube) - image = key_frame.do_generation() + image = key_frame.generate() if image is None: log_utils.print_warning_generate_returned_no_image() break @@ -63,16 +60,12 @@ def run_render_animation(data: RenderData): if not image_utils.is_PIL(image): # check is required when resuming from timestring image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) + state.assign_current_image(image) key_frame.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_frame)(image) - key_frame.progress_and_save(image) - state.assign_current_image(image) - key_frame.render_data.args.args.seed = key_frame.next_seed() - key_frame.update_render_preview() web_ui_utils.update_status_tracker(key_frame.render_data) - data.animation_mode.unload_raft_and_depth_model() @@ -108,6 +101,7 @@ def setup_pseudo_cadence(data, value): def _tweens_with_progress(key_step): # only use tween progress bar when extra console output (aka "dev mode") is disabled. - return tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') \ - if not log_utils.is_verbose() else key_step.tweens + return (tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + if not log_utils.is_verbose() + else key_step.tweens) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 4ffc864bb..2bf359a65 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -8,7 +8,7 @@ from ....hybrid_video import hybrid_generation -@dataclass(init=True, frozen=True, repr=False, eq=False) +@dataclass(init=True, frozen=False, repr=False, eq=False) class AnimationMode: has_video_input: bool = False hybrid_input_files: Any = None diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index ebc5188fe..2171afd8c 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -2,6 +2,7 @@ from dataclasses import dataclass from typing import Any, List +import PIL import cv2 import numpy as np from PIL import Image @@ -21,6 +22,7 @@ from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.call.video_and_audio import call_render_preview +from ....colors import maintain_colors from ....hybrid_video import image_transform_ransac, image_transform_optical_flow from ....save_images import save_image from ....seed import next_seed @@ -105,13 +107,6 @@ def maybe_write_frame_subtitle(self): self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_string) call_write_frame_subtitle(data, data.indexes.frame.i, params_string) - def write_frame_subtitle_if_active(self, data: RenderData): - if opt_utils.is_generate_subtitles(data): - params_string = opt_utils.generation_info_for_subtitles(data) - self.subtitle_params_to_print = params_string - self.subtitle_params_string = call_format_animation_params(data, self.indexes.tween.i, params_string) - call_write_frame_subtitle(data, self.indexes.tween.i, params_string, sub_step.tween < 1.0) - def apply_frame_warp_transform(self, data: RenderData, image): is_not_last_frame = self.i < data.args.anim_args.max_frames if is_not_last_frame: # TODO? why not @@ -145,8 +140,9 @@ def apply_anti_blur(self, data: RenderData, image): def apply_frame_noising(self, data: RenderData, mask, image): is_use_any_mask = data.args.args.use_mask or data.args.anim_args.use_noise_mask if is_use_any_mask: - seq = self.schedule.noise_mask_seq + seq = self.schedule.noise_mask vals = mask.noise_vals + contrast_image = image data.args.root.noise_mask = call_compose_mask_with_check(data, seq, vals, contrast_image) return call_add_noise(data, self, image) @@ -154,7 +150,7 @@ def create_color_match_for_video(self): data = self.render_data if data.args.anim_args.color_coherence == 'Video Input' and data.is_hybrid_available(): if int(data.indexes.frame.i) % int(data.args.anim_args.color_coherence_video_every_N_frames) == 0: - prev_vid_img = Image.open(preview_video_image_path(data, data.indexes)) + prev_vid_img = Image.open(filename_utils.preview_video_image_path(data, data.indexes)) prev_vid_img = prev_vid_img.resize(data.dimensions(), PIL.Image.LANCZOS) data.images.color_match = np.asarray(prev_vid_img) return cv2.cvtColor(data.images.color_match, cv2.COLOR_RGB2BGR) @@ -199,7 +195,7 @@ def maybe_redo_diffusion(self): if is_diffusion_redo and is_not_preview: self.do_diffusion_redo() - def do_generation(self): + def generate(self): max_frames = self.render_data.args.anim_args.max_frames if self.render_data.indexes.frame.i >= max_frames: # TODO? prevents an error elsewhere when generating last frame, but `indexes`.. @@ -226,7 +222,7 @@ def _progress_save_and_get_next_index(self, image): # TODO perhaps save original frames in a different sub dir? save_image(image, 'PIL', filename, data.args.args, data.args.video_args, data.args.root) - self.depth = self.generate_and_save_depth_map_if_active() + self.depth = self.generate_and_save_depth_map_if_active(opencv_image) if data.turbo.has_steps(): return data.indexes.frame.i + data.turbo.progress_step(data.indexes, opencv_image) return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. @@ -237,14 +233,14 @@ def next_seed(self): def update_render_preview(self): self.last_preview_frame = call_render_preview(self.render_data, self.last_preview_frame) - def generate_and_save_depth_map_if_active(self): + def generate_and_save_depth_map_if_active(self, opencv_image): data = self.render_data # TODO move all depth related stuff to new class. if data.args.anim_args.save_depth_maps: memory_utils.handle_vram_before_depth_map_generation(data) depth = data.depth_model.predict(opencv_image, data.args.anim_args.midas_weight, data.args.root.half_precision) - depth_filename = filename_utils.depth_frame(data, idx) + depth_filename = filename_utils.depth_frame(data, data.indexes) data.depth_model.save(os.path.join(data.output_directory, depth_filename), depth) memory_utils.handle_vram_after_depth_map_generation(data) return depth @@ -320,7 +316,7 @@ def apply_hybrid_motion_ransac_transform(data: RenderData, image): return image @staticmethod - def apply_hybrid_motion_optical_flow(data: RenderData, image): + def apply_hybrid_motion_optical_flow(data: RenderData, key_frame, image): motion = data.args.anim_args.hybrid_motion if motion in ['Optical Flow']: last_i = data.indexes.frame.i - 1 @@ -328,14 +324,16 @@ def apply_hybrid_motion_optical_flow(data: RenderData, image): flow = call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous) \ if data.args.anim_args.hybrid_motion_use_prev_img \ else call_get_flow_for_hybrid_motion(data, last_i) - transformed = image_transform_optical_flow(images.previous, flow, step.step_data.flow_factor()) + transformed = image_transform_optical_flow( + reference_images.previous, flow, key_frame.step_data.flow_factor()) data.animation_mode.prev_flow = flow # side effect return transformed return image @staticmethod - def create_all_steps(data, start_index, index_dist: KeyFrameDistribution = KeyFrameDistribution.default()): + def create_all_frames(data, index_dist: KeyFrameDistribution = KeyFrameDistribution.default()): """Creates a list of key steps for the entire animation.""" + start_index = data.turbo.find_start(data) num_key_steps = 1 + int((data.args.anim_args.max_frames - start_index) / data.cadence()) if data.parseq_adapter.use_parseq and index_dist is KeyFrameDistribution.PARSEQ_ONLY: num_key_steps = len(data.parseq_adapter.parseq_json["keyframes"]) diff --git a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py index fe55dcf12..5be874674 100644 --- a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py @@ -1,6 +1,6 @@ from dataclasses import dataclass from itertools import chain -from typing import Any, Iterable, Tuple +from typing import Any, Iterable, Tuple, List from ..turbo import Turbo from ...data.indexes import Indexes, IndexWithStart @@ -41,7 +41,7 @@ def emit_frame(self, last_frame, grayscale_tube, overlay_mask_tube): new_image = self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) # TODO pass step and depth instead of data and tween_step.indexes - new_image = image_utils.save_and_return_frame(data, self.i(), new_image) + new_image = image_utils.save_and_return_frame(data, self, self.i(), new_image) # updating reference images to calculate hybrid motions in next iteration data.images.previous = new_image # FIXME diff --git a/scripts/deforum_helpers/rendering/data/images.py b/scripts/deforum_helpers/rendering/data/images.py index 9b7e76e00..628bcb171 100644 --- a/scripts/deforum_helpers/rendering/data/images.py +++ b/scripts/deforum_helpers/rendering/data/images.py @@ -1,8 +1,12 @@ from dataclasses import dataclass +import PIL import cv2 +import numpy as np from cv2.typing import MatLike +from ...load_images import load_image + @dataclass(init=True, frozen=False, repr=False, eq=True) class Images: @@ -16,7 +20,9 @@ def has_previous(self): def _load_color_match_sample(init) -> MatLike: """get color match for 'Image' color coherence only once, before loop""" if init.args.anim_args.color_coherence == 'Image': - raw_image = load_image(init.args.anim_args.color_coherence_image_path, None) + image_box: PIL.Image.Image = None + # noinspection PyTypeChecker + raw_image = load_image(init.args.anim_args.color_coherence_image_path, image_box) resized = raw_image.resize(init.dimensions(), PIL.Image.LANCZOS) return cv2.cvtColor(np.array(resized), cv2.COLOR_RGB2BGR) diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index 0a5c246a3..2bd03d25c 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -45,7 +45,7 @@ def _load_mask(init, args): @staticmethod def _create_mask_image(init): args = init.args.args - return call_or_use_on_cond(init.is_using_init_image_or_box(), lambda: _load_mask(init, args)) + return call_or_use_on_cond(init.is_using_init_image_or_box(), lambda: Mask._load_mask(init, args)) @staticmethod def _create(init, i, mask_image): diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index fb88d55f6..c9c84625e 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -16,7 +16,8 @@ from .mask import Mask from .subtitle import Srt from .turbo import Turbo -from ..util import memory_utils, opt_utils +from ..util import log_utils, memory_utils, opt_utils +from ..util.call.images import call_get_mask_from_file_with_frame from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame from ...args import DeforumArgs, DeforumAnimArgs, LoopArgs, ParseqArgs, RootArgs @@ -244,8 +245,8 @@ def prompt_for_current_step(self, i): def _update_video_input_for_current_frame(self, i, step): video_init_path = self.args.anim_args.video_init_path - init_frame = call_get_next_frame(init, i, video_init_path) - print_init_frame_info(init_frame) + init_frame = call_get_next_frame(self, i, video_init_path) + log_utils.print_init_frame_info(init_frame) self.args.args.init_image = init_frame self.args.args.init_image_box = None # init_image_box not used in this case self.args.args.strength = max(0.0, min(1.0, step.step_data.strength)) @@ -257,7 +258,7 @@ def _update_video_mask_for_current_frame(self, i): new_mask = call_get_mask_from_file_with_frame(self, mask_init_frame) self.args.args.mask_file = new_mask self.args.root.noise_mask = new_mask - mask.vals['video_mask'] = new_mask + self.mask.vals['video_mask'] = new_mask def update_video_data_for_current_frame(self, i, step): if self.animation_mode.has_video_input: diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index 51ec35864..e8997507c 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -12,7 +12,6 @@ class Schedule: eta_ancestral: float # TODO unify ddim- and a-eta to use one or the other, depending on sampler mask: Optional[Any] noise_mask: Optional[Any] - noise_mask_seq: Any @staticmethod def create(init, i, anim_args, args): @@ -27,10 +26,8 @@ def create(init, i, anim_args, args): eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) mask = Schedule.schedule_mask(keys, i, args) - noise_mask = Schedule.schedule_noise_mask(keys, i, anim_args) - noise_mask_seq = schedule.mask_seq if is_use_mask_without_noise else None - return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, - noise_mask, noise_mask_seq) + noise_mask = mask if is_use_mask_without_noise else Schedule.schedule_noise_mask(keys, i, anim_args) + return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) @staticmethod def _has_schedule(keys, i): diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 7daf0c64a..01d0838bd 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -3,12 +3,11 @@ from cv2.typing import MatLike from .subtitle import Srt -from ..util import log_utils from ..util.call.anim import call_anim_frame_warp -from ..util.call.hybrid import call_get_flow_from_images +from ..util.call.hybrid import call_get_flow_from_images, call_get_flow_for_hybrid_motion_prev, \ + call_get_flow_for_hybrid_motion, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev from ..util.call.resume import call_get_resume_vars -from ...hybrid_video import (image_transform_ransac, image_transform_optical_flow, - abs_flow_to_rel_flow, rel_flow_to_abs_flow) +from ...hybrid_video import image_transform_ransac, image_transform_optical_flow, rel_flow_to_abs_flow @dataclass(init=True, frozen=False, repr=False, eq=True) @@ -48,7 +47,7 @@ def _is_do_motion(motions): return indexes.tween.i > 0 and motion in motions if _is_do_motion(['Affine', 'Perspective']): - Turbo.advance_hybrid_motion_ransac_transform(data, indexes, reference_images) + self.advance_hybrid_motion_ransac_transform(data, indexes, reference_images) if _is_do_motion(['Optical Flow']): self.advance_hybrid_motion_optical_tween_flow(data, indexes, reference_images, last_frame) @@ -56,7 +55,7 @@ def advance_optical_flow(self, tween_step, flow_factor: int = 1): flow = tween_step.cadence_flow * -1 self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) - def advance_optical_tween_flow(self, last_frame, flow): + def advance_optical_tween_flow(self, indexes, last_frame, flow): ff = last_frame.step_data.flow_factor() i = indexes.tween.i if self.is_advance_prev(i): @@ -69,7 +68,7 @@ def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_imag flow = (call_get_flow_for_hybrid_motion(data, last_i) if not data.args.anim_args.hybrid_motion_use_prev_img else call_get_flow_for_hybrid_motion_prev(data, last_i, reference_images.previous)) - turbo.advance_optical_tween_flow(self, last_frame, flow) + self.advance_optical_tween_flow(indexes, last_frame, flow) data.animation_mode.prev_flow = flow def advance_cadence_flow(self, data, tween_frame): @@ -83,21 +82,19 @@ def advance_cadence_flow(self, data, tween_frame): # TODO? move to RenderData def advance_ransac_transform(self, data, matrix): - i = indexes.tween.i + i = data.indexes.tween.i motion = data.args.anim_args.hybrid_motion if self.is_advance_prev(i): self.prev.image = image_transform_ransac(self.prev.image, matrix, motion) if self.is_advance_next(i): self.next.image = image_transform_ransac(self.next.image, matrix, motion) - # TODO? move to RenderData - @staticmethod - def advance_hybrid_motion_ransac_transform(data, indexes, reference_images): + def advance_hybrid_motion_ransac_transform(self, data, indexes, reference_images): last_i = indexes.tween.i - 1 matrix = (call_get_matrix_for_hybrid_motion(data, last_i) if not data.args.anim_args.hybrid_motion_use_prev_img else call_get_matrix_for_hybrid_motion_prev(data, last_i, reference_images.previous)) - turbo.advance_ransac_transform(data, matrix) + self.advance_ransac_transform(data, matrix) def advance_optical_flow_cadence_before_animation_warping(self, data, last_frame, tween_frame): if data.is_3d_or_2d() and data.has_optical_flow_cadence(): diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 0257b35c4..dc1bb0b6b 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -2,6 +2,7 @@ import cv2 import numpy as np +from PIL import ImageOps, Image from cv2.typing import MatLike from .data.frame.key_frame import KeyFrame @@ -34,7 +35,7 @@ def frame_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTub return tube(lambda img: key_frame.apply_frame_warp_transform(data, img), lambda img: key_frame.do_hybrid_compositing_before_motion(data, img), lambda img: KeyFrame.apply_hybrid_motion_ransac_transform(data, img), - lambda img: KeyFrame.apply_hybrid_motion_optical_flow(data, img), + lambda img: KeyFrame.apply_hybrid_motion_optical_flow(data, key_frame, img), lambda img: key_frame.do_normal_hybrid_compositing_after_motion(data, img), lambda img: KeyFrame.apply_color_matching(data, img), lambda img: KeyFrame.transform_to_grayscale_if_active(data, img)) @@ -64,8 +65,7 @@ def conditional_hybrid_video_after_generation_tube(key_frame: KeyFrame) -> Image return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), - is_do_process= - lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) + is_do_process=lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: @@ -75,8 +75,8 @@ def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), - is_do_process= - lambda: data.indexes.is_first_frame() and data.is_initialize_color_match(data.images.color_match)) + is_do_process=lambda: data.indexes.is_first_frame() and data.is_initialize_color_match( + data.images.color_match)) def conditional_color_match_tube(key_frame: KeyFrame) -> ImageTube: @@ -110,14 +110,14 @@ def conditional_force_tween_to_grayscale_tube(data: RenderData) -> ImageTube: # Composite Tubes, made from other Tubes. def contrasted_noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: """Combines contrast and noise transformation tubes.""" - contrast_tube: Tube = contrast_transformation_tube(data, key_frame) - noise_tube: Tube = noise_transformation_tube(data, key_frame) + contrast_tube: ImageTube = contrast_transformation_tube(data, key_frame) + noise_tube: ImageTube = noise_transformation_tube(data, key_frame) return tube(lambda img: noise_tube(contrast_tube(img))) def conditional_frame_transformation_tube(key_frame: KeyFrame, is_tween: bool = False) -> ImageTube: - hybrid_tube: Tube = conditional_hybrid_video_after_generation_tube(key_frame) - extra_tube: Tube = conditional_extra_color_match_tube(key_frame.render_data) - gray_tube: Tube = conditional_force_to_grayscale_tube(key_frame.render_data) - mask_tube: Tube = conditional_add_overlay_mask_tube(key_frame.render_data, is_tween) + hybrid_tube: ImageTube = conditional_hybrid_video_after_generation_tube(key_frame) + extra_tube: ImageTube = conditional_extra_color_match_tube(key_frame.render_data) + gray_tube: ImageTube = conditional_force_to_grayscale_tube(key_frame.render_data) + mask_tube: ImageTube = conditional_add_overlay_mask_tube(key_frame.render_data, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index f62106e8f..1d715e372 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -15,15 +15,15 @@ def save_cadence_frame(data: RenderData, i: int, image: MatLike, is_overwrite: b cv2.imwrite(save_path, image) -def save_cadence_frame_and_depth_map_if_active(data: RenderData, i, image): +def save_cadence_frame_and_depth_map_if_active(data: RenderData, frame, i, image): save_cadence_frame(data, i, image) if data.args.anim_args.save_depth_maps: dm_save_path = os.path.join(data.output_directory, filename_utils.frame_filename(data, i, True)) - data.depth_model.save(dm_save_path, step.depth) + data.depth_model.save(dm_save_path, frame.depth) -def save_and_return_frame(data: RenderData, i, image): - save_cadence_frame_and_depth_map_if_active(data, i, image) +def save_and_return_frame(data: RenderData, frame, i, image): + save_cadence_frame_and_depth_map_if_active(data, frame, i, image) return image From 9cb15b8603f3b62a588a27c684a14675eb079949 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sun, 14 Jul 2024 02:15:07 +0200 Subject: [PATCH 111/132] More code simplification and cleanup. --- scripts/deforum_helpers/render.py | 69 +++++++++---------- .../rendering/data/frame/key_frame.py | 37 +++++----- .../data/frame/key_frame_distribution.py | 8 ++- .../rendering/data/frame/tween_frame.py | 7 +- .../deforum_helpers/rendering/data/images.py | 6 +- .../rendering/data/render_data.py | 3 + .../deforum_helpers/rendering/data/turbo.py | 24 +++---- .../rendering/util/log_utils.py | 6 ++ 8 files changed, 86 insertions(+), 74 deletions(-) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index 9d7d2f6bc..d50ad91d4 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -16,57 +16,57 @@ import os from pathlib import Path +from typing import List # noinspection PyUnresolvedReferences from modules.shared import cmd_opts, progress_print_out, state from tqdm import tqdm -from .rendering import img_2_img_tubes -from .rendering.data.frame import KeyFrameDistribution, KeyFrame -from .rendering.data.render_data import RenderData -from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils +from .rendering import img_2_img_tubes # noqa +from .rendering.data.frame import KeyFrameDistribution, KeyFrame # noqa +from .rendering.data.render_data import RenderData # noqa +from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils # noqa def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - render_data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) - run_render_animation(render_data) + data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) + web_ui_utils.init_job(data) + key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) + run_render_animation(data, key_frames) + data.animation_mode.unload_raft_and_depth_model() # @log_utils.with_suppressed_table_printing -def run_render_animation(data: RenderData): - web_ui_utils.init_job(data) - key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) +def run_render_animation(data: RenderData, key_frames: List[KeyFrame]): for key_frame in key_frames: if is_resume(data, key_frame): continue - - memory_utils.handle_med_or_low_vram_before_step(data) - web_ui_utils.update_job(data) - if key_frame.has_tween_frames(): - emit_tweens(data, key_frame) - - log_utils.print_animation_frame_info(key_frame.i, data.args.anim_args.max_frames) - key_frame.maybe_write_frame_subtitle() - - frame_tube = img_2_img_tubes.frame_transformation_tube - contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - key_frame.prepare_generation(frame_tube, contrasted_noise_tube) - + pre_process_key_frame_and_emit_tweens(data, key_frame) image = key_frame.generate() if image is None: log_utils.print_warning_generate_returned_no_image() break + post_process_key_frame(key_frame, image) - if not image_utils.is_PIL(image): # check is required when resuming from timestring - image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) - state.assign_current_image(image) - key_frame.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(key_frame)(image) - key_frame.progress_and_save(image) - key_frame.render_data.args.args.seed = key_frame.next_seed() - key_frame.update_render_preview() - web_ui_utils.update_status_tracker(key_frame.render_data) - data.animation_mode.unload_raft_and_depth_model() +def pre_process_key_frame_and_emit_tweens(data, key_frame): + memory_utils.handle_med_or_low_vram_before_step(data) + web_ui_utils.update_job(data) + if key_frame.has_tween_frames(): + emit_tweens(data, key_frame) + log_utils.print_animation_frame_info(key_frame.i, data.args.anim_args.max_frames) + key_frame.maybe_write_frame_subtitle() + frame_tube = img_2_img_tubes.frame_transformation_tube + contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube + key_frame.prepare_generation(frame_tube, contrasted_noise_tube) + + +def post_process_key_frame(key_frame, image): + if not image_utils.is_PIL(image): # check is required when resuming from timestring + image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) + state.assign_current_image(image) + key_frame.after_diffusion(image) + web_ui_utils.update_status_tracker(key_frame.render_data) def is_resume(data, key_step): @@ -80,9 +80,8 @@ def is_resume(data, key_step): def emit_tweens(data, key_step): - setup_pseudo_cadence(data, len(key_step.tweens) - 1) - if key_step.i == 1: - data.parseq_adapter.print_parseq_table() + _update_pseudo_cadence(data, len(key_step.tweens) - 1) + log_utils.print_parseq_table_at_start(data, key_step) log_utils.print_tween_frame_from_to_info(key_step) grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube @@ -90,7 +89,7 @@ def emit_tweens(data, key_step): [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] -def setup_pseudo_cadence(data, value): +def _update_pseudo_cadence(data, value): data.turbo.cadence = value data.parseq_adapter.cadence = value data.parseq_adapter.a1111_cadence = value diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index 2171afd8c..aedc88b78 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -202,6 +202,12 @@ def generate(self): self.render_data.indexes.update_frame(max_frames - 1) return call_generate(self.render_data, self) + def after_diffusion(self, image): + self.render_data.images.color_match = img_2_img_tubes.conditional_color_match_tube(self)(image) + self.progress_and_save(image) + self.render_data.args.args.seed = self.next_seed() + self.update_render_preview() + def progress_and_save(self, image): next_index = self._progress_save_and_get_next_index(image) self.render_data.indexes.update_frame(next_index) @@ -341,38 +347,29 @@ def create_all_frames(data, index_dist: KeyFrameDistribution = KeyFrameDistribut key_steps = [KeyFrame.create(data) for _ in range(0, num_key_steps)] actual_num_key_steps = len(key_steps) - recalculated_key_steps = KeyFrame._recalculate_and_check_tweens(data, key_steps, - start_index, actual_num_key_steps, - data.parseq_adapter, index_dist) + recalculated_key_steps = KeyFrame._recalculate_and_check_tweens( + data, key_steps, start_index, actual_num_key_steps, data.parseq_adapter, index_dist) log_utils.print_tween_step_creation_info(key_steps, index_dist) return recalculated_key_steps @staticmethod - def _recalculate_and_check_tweens(data, key_steps, start_index, num_key_steps, + def _recalculate_and_check_tweens(data, key_frames, start_index, num_key_steps, parseq_adapter, index_distribution): max_frames = data.args.anim_args.max_frames - key_indices: List[int] = index_distribution.calculate(start_index, max_frames, num_key_steps, parseq_adapter) - for i, key_step in enumerate(key_indices): - key_steps[i].i = key_indices[i] - - key_steps = KeyFrame._add_tweens_to_key_steps(key_steps) - log_utils.print_key_step_debug_info_if_verbose(key_steps) + key_frames = index_distribution.calculate(key_frames, start_index, max_frames, num_key_steps, parseq_adapter) + key_frames = KeyFrame._add_tweens_to_key_steps(key_frames) + log_utils.print_key_step_debug_info_if_verbose(key_frames) # The number of generated tweens depends on index since last key-frame. The last tween has the same # index as the key_step it belongs to and is meant to replace the unprocessed original key frame. - # TODO? make unit tests instead of asserts... - # total_count = len(key_steps) + sum(len(key_step.tweens) - 1 for key_step in key_steps) - # log_utils.info(f"total_count {total_count} len(key_steps) {len(key_steps)} max_frames {max_frames}") - # assert total_count == len(key_steps) + max_frames # every key frame except the 1st has a tween double. - - assert len(key_steps) == num_key_steps - assert key_steps[0].tweens == [] # 1st key step has no tweens - assert key_steps[0].i == 1 + assert len(key_frames) == num_key_steps + assert key_frames[0].i == 1 # 1st key frame is at index 1 + assert key_frames[0].tweens == [] # 1st key frame has no tweens if index_distribution != KeyFrameDistribution.PARSEQ_ONLY: # just using however many key frames Parseq defines. - assert key_steps[-1].i == max_frames + assert key_frames[-1].i == max_frames # last index is same as max frames - return key_steps + return key_frames @staticmethod def _add_tweens_to_key_steps(key_steps): diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py index 98c1341df..6fcaf2d60 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py @@ -23,7 +23,13 @@ def from_UI_tab(data): def default(): return KeyFrameDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. - def calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: + def calculate(self, key_frames, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: + key_indices: List[int] = self._calculate(start_index, max_frames, num_key_steps, parseq_adapter) + for i, key_step in enumerate(key_indices): + key_frames[i].i = key_indices[i] + return key_frames + + def _calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: match self: case KeyFrameDistribution.PARSEQ_ONLY: return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) diff --git a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py index 5be874674..3814f6dd0 100644 --- a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py @@ -47,17 +47,15 @@ def emit_frame(self, last_frame, grayscale_tube, overlay_mask_tube): data.images.previous = new_image # FIXME def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): - is_tween = True warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) - # print(f"warped {warped}") recolored = grayscale_tube(data)(warped) + is_tween = True masked = overlay_mask_tube(data, is_tween)(recolored) return masked def process(self, last_frame, data): data.turbo.advance_optical_flow_cadence_before_animation_warping(data, last_frame, self) self.depth_prediction = Tween.calculate_depth_prediction(data, data.turbo) - # log_utils.info(f"self.depth_prediction {self.depth_prediction}") data.turbo.advance(data, self.indexes.tween.i, self.depth) data.turbo.do_hybrid_video_motion(data, last_frame, self.indexes, data.images) @@ -73,6 +71,9 @@ def write_tween_frame_subtitle_if_active(self, data: RenderData): is_cadence = self.value < 1.0 call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) + def has_cadence(self): + return self.cadence_flow is not None + @staticmethod def create_in_between_steps(key_frames, i, data, from_i, to_i): tween_range = range(from_i, to_i) diff --git a/scripts/deforum_helpers/rendering/data/images.py b/scripts/deforum_helpers/rendering/data/images.py index 628bcb171..e9a444c40 100644 --- a/scripts/deforum_helpers/rendering/data/images.py +++ b/scripts/deforum_helpers/rendering/data/images.py @@ -10,8 +10,8 @@ @dataclass(init=True, frozen=False, repr=False, eq=True) class Images: - previous: MatLike | None = None color_match: MatLike = None + previous: MatLike | None = None def has_previous(self): return self.previous is not None @@ -27,5 +27,5 @@ def _load_color_match_sample(init) -> MatLike: return cv2.cvtColor(np.array(resized), cv2.COLOR_RGB2BGR) @staticmethod - def create(init): - return Images(None, Images._load_color_match_sample(init)) + def create(data): + return Images(Images._load_color_match_sample(data)) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index c9c84625e..364bcfd15 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -100,6 +100,9 @@ def is_3d_or_2d(self): def has_optical_flow_cadence(self): return self.args.anim_args.optical_flow_cadence != 'None' + def is_3d_or_2d_with_optical_flow(self): + return self.is_3d_or_2d() and self.has_optical_flow_cadence() + def is_3d_with_med_or_low_vram(self): return self.is_3d() and memory_utils.is_low_or_med_vram() diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 01d0838bd..6be5eb798 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -97,24 +97,24 @@ def advance_hybrid_motion_ransac_transform(self, data, indexes, reference_images self.advance_ransac_transform(data, matrix) def advance_optical_flow_cadence_before_animation_warping(self, data, last_frame, tween_frame): - if data.is_3d_or_2d() and data.has_optical_flow_cadence(): - i = data.indexes.tween.start - has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[i] > 0 - has_images = self.prev.image is not None and self.next.image is not None - has_step_and_images = tween_frame.cadence_flow is None and has_images - if has_tween_schedule and has_step_and_images and data.animation_mode.is_raft_active(): + if data.is_3d_or_2d_with_optical_flow(): + if self._is_do_flow(data, tween_frame): cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) tween_frame.cadence_flow = (flow / 2) - if tween_frame.cadence_flow is not None: + if tween_frame.has_cadence(): self.advance_optical_flow(tween_frame) - - # flow_factor = 1.0 / (last_step.i - tween_step.i() + 1) - flow_factor = 100.0 / len(last_frame.tweens) - # flow_factor = 100.0 - if tween_frame.cadence_flow is not None: + flow_factor = 1.0 # FIXME.. + # flow_factor = 1.0 / (len(last_frame.tweens) + 1.0) self.next.image = image_transform_optical_flow(self.next.image, -tween_frame.cadence_flow, flow_factor) + def _is_do_flow(self, data, tween_frame): + i = data.indexes.tween.start + has_tween_schedule = data.animation_keys.deform_keys.strength_schedule_series[i] > 0 + has_images = self.prev.image is not None and self.next.image is not None + has_step_and_images = tween_frame.cadence_flow is None and has_images + return has_tween_schedule and has_step_and_images and data.animation_mode.is_raft_active() + def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_frame): if not data.animation_mode.is_raft_active(): return self.next.image diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 3ed9d2f76..ffcb46575 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -26,6 +26,11 @@ def is_verbose(): return opts.data.get("deforum_debug_mode_enabled", False) +def print_parseq_table_at_start(data, key_step): + if key_step.i == 1: + data.parseq_adapter.print_parseq_table() + + def print_tween_frame_from_to_info(key_step, is_disabled=True): if not is_disabled: # replaced with prog bar, but value info print may be useful tween_values = key_step.tween_values @@ -99,6 +104,7 @@ def debug(s: str): print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") +# noqa def with_suppressed_table_printing(func): def _suppress_table_printing(): # The combined table that is normally printed to the command line is suppressed, From 568e5041efc3c0fa414ff25d098eec4100d62bb5 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Wed, 31 Jul 2024 16:42:13 +0200 Subject: [PATCH 112/132] Removed new tab and integrated keyframe redistrubution selection to existing Parseq tab. Moved new property to parseq namespace. Reverted render core back to original and added a switch for using the new core. --- scripts/default_settings.txt | 3 +- scripts/deforum_helpers/args.py | 20 +- scripts/deforum_helpers/defaults.py | 7 + scripts/deforum_helpers/render.py | 716 ++++++++++++++++-- .../data/frame/key_frame_distribution.py | 58 +- scripts/deforum_helpers/rendering/new_core.py | 90 +++ .../rendering/util/log_utils.py | 5 - scripts/deforum_helpers/ui_elements.py | 73 +- scripts/deforum_helpers/ui_left.py | 7 +- 9 files changed, 785 insertions(+), 194 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/new_core.py diff --git a/scripts/default_settings.txt b/scripts/default_settings.txt index 32e8eb914..a6c346dc0 100644 --- a/scripts/default_settings.txt +++ b/scripts/default_settings.txt @@ -143,7 +143,8 @@ "hybrid_comp_save_extra_frames": false, "parseq_manifest": "", "parseq_use_deltas": true, - "parseq_non_schedule_overrides": true, + "parseq_non_schedule_overrides": true, + "parseq_key_frame_redistribution": "off", "use_looper": false, "init_images": "{\n \"0\": \"https://deforum.github.io/a1/Gi1.png\",\n \"max_f/4-5\": \"https://deforum.github.io/a1/Gi2.png\",\n \"max_f/2-10\": \"https://deforum.github.io/a1/Gi3.png\",\n \"3*max_f/4-15\": \"https://deforum.github.io/a1/Gi4.jpg\",\n \"max_f-20\": \"https://deforum.github.io/a1/Gi1.png\"\n}", "image_strength_schedule": "0:(0.75)", diff --git a/scripts/deforum_helpers/args.py b/scripts/deforum_helpers/args.py index cd7fd19fa..5496ecee8 100644 --- a/scripts/deforum_helpers/args.py +++ b/scripts/deforum_helpers/args.py @@ -22,7 +22,8 @@ import modules.paths as ph import modules.shared as sh from modules.processing import get_fixed_seed -from .defaults import get_guided_imgs_default_json, mask_fill_choices, get_samplers_list, get_schedulers_list +from .defaults import (get_guided_imgs_default_json, get_parseq_keyframe_redistributions_list, + get_samplers_list, get_schedulers_list) from .deforum_controlnet import controlnet_component_names from .general_utils import get_os, substitute_placeholders @@ -703,8 +704,7 @@ def DeforumAnimArgs(): "info": "", }, "hybrid_comp_mask_auto_contrast": False, - "hybrid_comp_save_extra_frames": False, - "neocore_key_index_distribution": "Parseq Only" + "hybrid_comp_save_extra_frames": False } def DeforumArgs(): @@ -929,7 +929,7 @@ def DeforumArgs(): "type": "checkbox", "value": False, "info": "Preview motion only. Uses a static picture for init, and draw motion reference rectangle." - }, + }, } def LoopArgs(): @@ -991,7 +991,14 @@ def ParseqArgs(): "type": "checkbox", "value": True, "info": "Recommended. If you uncheck this, the FPS, max_frames and cadence in the Parseq doc are ignored, and the values in the A1111 UI are used instead." - } + }, + "parseq_key_frame_redistribution": { + "label": "Parseq key frame redistribution mode.", + "type": "dropdown", + "choices": get_parseq_keyframe_redistributions_list().values(), + "value": "None", + "info": "Gain Parseq precision at the cost of cadence regularity." + }, } def DeforumOutputArgs(): @@ -1120,8 +1127,7 @@ def DeforumOutputArgs(): "value": False, "info": "Interpolate upscaled images, if available", "visible": False - }, - + }, } def get_component_names(): diff --git a/scripts/deforum_helpers/defaults.py b/scripts/deforum_helpers/defaults.py index ac5f17199..007808732 100644 --- a/scripts/deforum_helpers/defaults.py +++ b/scripts/deforum_helpers/defaults.py @@ -47,6 +47,13 @@ def get_schedulers_list(): 'sgm uniform': 'SGM Uniform' } +def get_parseq_keyframe_redistributions_list(): + return { + 'off': 'Off', + 'parseq_only': 'Parseq Only (no cadence)', + 'uniform_with_parseq': 'Uniform with Parseq (pseudo-cadence)' + } + def DeforumAnimPrompts(): return r"""{ "0": "tiny cute bunny, vibrant diffraction, highly detailed, intricate, ultra hd, sharp photo, crepuscular rays, in focus", diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index d50ad91d4..c1fda7bb9 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -15,92 +15,642 @@ # Contact the authors: https://deforum.github.io/ import os -from pathlib import Path -from typing import List +import pandas as pd +import cv2 +import numpy as np +import numexpr +import gc +import random +import PIL +import time +from PIL import Image, ImageOps +from .generate import generate, isJson +from .noise import add_noise +from .animation import anim_frame_warp +from .animation_key_frames import DeformAnimKeys, LooperAnimKeys +from .video_audio_utilities import get_frame_name, get_next_frame, render_preview +from .depth import DepthModel +from .colors import maintain_colors +from .parseq_adapter import ParseqAdapter +from .seed import next_seed +from .image_sharpening import unsharp_mask +from .load_images import get_mask, load_img, load_image, get_mask_from_file +from .hybrid_video import ( + hybrid_generation, hybrid_composite, get_matrix_for_hybrid_motion, get_matrix_for_hybrid_motion_prev, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, image_transform_ransac, + image_transform_optical_flow, get_flow_from_images, abs_flow_to_rel_flow, rel_flow_to_abs_flow) +from .save_images import save_image +from .composable_masks import compose_mask_with_check +from .settings import save_settings_from_animation_run +from .deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled +from .subtitle_handler import init_srt_file, write_frame_subtitle, format_animation_params +from .resume import get_resume_vars +from .masks import do_overlay_mask +from .prompt import prepare_prompt +from modules.shared import opts, cmd_opts, state, sd_model +from modules import lowvram, devices, sd_hijack +from .rendering import new_core +from .RAFT import RAFT -# noinspection PyUnresolvedReferences -from modules.shared import cmd_opts, progress_print_out, state -from tqdm import tqdm +from deforum_api import JobStatusTracker -from .rendering import img_2_img_tubes # noqa -from .rendering.data.frame import KeyFrameDistribution, KeyFrame # noqa -from .rendering.data.render_data import RenderData # noqa -from .rendering.util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils # noqa +def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): + is_use_parseq = parseq_args.parseq_manifest and parseq_args.parseq_manifest.strip() + is_use_key_frame_redistribution = parseq_args.parseq_key_frame_redistribution != "Off" + is_use_new_render_core = is_use_parseq and is_use_key_frame_redistribution + if is_use_new_render_core: + new_core.render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root) + return + # initialise Parseq adapter + parseq_adapter = ParseqAdapter(parseq_args, anim_args, video_args, controlnet_args, loop_args) + + if opts.data.get("deforum_save_gen_info_as_srt", False): # create .srt file and set timeframe mechanism using FPS + srt_filename = os.path.join(args.outdir, f"{root.timestring}.srt") + srt_frame_duration = init_srt_file(srt_filename, video_args.fps) + + if anim_args.animation_mode in ['2D', '3D']: + # handle hybrid video generation + if anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Affine', 'Perspective', 'Optical Flow']: + args, anim_args, inputfiles = hybrid_generation(args, anim_args, root) + # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False + hybrid_frame_path = os.path.join(args.outdir, 'hybridframes') + # initialize prev_flow + if anim_args.hybrid_motion == 'Optical Flow': + prev_flow = None + + if loop_args.use_looper: + print("Using Guided Images mode: seed_behavior will be set to 'schedule' and 'strength_0_no_init' to False") + if args.strength == 0: + raise RuntimeError("Strength needs to be greater than 0 in Init tab") + args.strength_0_no_init = False + args.seed_behavior = "schedule" + if not isJson(loop_args.init_images): + raise RuntimeError("The images set for use with keyframe-guidance are not in a proper JSON format") + + # handle controlnet video input frames generation + if is_controlnet_enabled(controlnet_args): + unpack_controlnet_vids(args, anim_args, controlnet_args) + + # expand key frame strings to values + keys = DeformAnimKeys(anim_args, args.seed) if not parseq_adapter.use_parseq else parseq_adapter.anim_keys + loopSchedulesAndData = LooperAnimKeys(loop_args, anim_args, args.seed) if not parseq_adapter.use_parseq else parseq_adapter.looper_keys + + # create output folder for the batch + os.makedirs(args.outdir, exist_ok=True) + print(f"Saving animation frames to:\n{args.outdir}") + + # save settings.txt file for the current run + save_settings_from_animation_run(args, anim_args, parseq_args, loop_args, controlnet_args, video_args, root) + + # resume from timestring + if anim_args.resume_from_timestring: + root.timestring = anim_args.resume_timestring + + # Always enable pseudo-3d with parseq. No need for an extra toggle: + # Whether it's used or not in practice is defined by the schedules + if parseq_adapter.use_parseq: + anim_args.flip_2d_perspective = True + + # expand prompts out to per-frame + if parseq_adapter.manages_prompts(): + prompt_series = keys.prompts + else: + prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames)]) + for i, prompt in root.animation_prompts.items(): + if str(i).isdigit(): + prompt_series[int(i)] = prompt + else: + prompt_series[int(numexpr.evaluate(i))] = prompt + prompt_series = prompt_series.ffill().bfill() + + # check for video inits + using_vid_init = anim_args.animation_mode == 'Video Input' + + # load depth model for 3D + predict_depths = (anim_args.animation_mode == '3D' and anim_args.use_depth_warping) or anim_args.save_depth_maps + predict_depths = predict_depths or (anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth']) + predict_depths = predict_depths and not args.motion_preview_mode + if predict_depths: + keep_in_vram = opts.data.get("deforum_keep_3d_models_in_vram") + + device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else root.device) + depth_model = DepthModel(root.models_path, device, root.half_precision, keep_in_vram=keep_in_vram, depth_algorithm=anim_args.depth_algorithm, Width=args.W, Height=args.H, + midas_weight=anim_args.midas_weight) + + # depth-based hybrid composite mask requires saved depth maps + if anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth': + anim_args.save_depth_maps = True + else: + depth_model = None + anim_args.save_depth_maps = False + + raft_model = None + load_raft = (anim_args.optical_flow_cadence == "RAFT" and int(anim_args.diffusion_cadence) > 1) or \ + (anim_args.hybrid_motion == "Optical Flow" and anim_args.hybrid_flow_method == "RAFT") or \ + (anim_args.optical_flow_redo_generation == "RAFT") + load_raft = load_raft and not args.motion_preview_mode + if load_raft: + print("Loading RAFT model...") + raft_model = RAFT() + + # state for interpolating between diffusion steps + turbo_steps = 1 if using_vid_init else int(anim_args.diffusion_cadence) + turbo_prev_image, turbo_prev_frame_idx = None, 0 + turbo_next_image, turbo_next_frame_idx = None, 0 + + # initialize vars + prev_img = None + color_match_sample = None + start_frame = 0 + + # resume animation (requires at least two frames - see function) + if anim_args.resume_from_timestring: + # determine last frame and frame to start on + prev_frame, next_frame, prev_img, next_img = get_resume_vars( + folder=args.outdir, + timestring=anim_args.resume_timestring, + cadence=turbo_steps + ) + + # set up turbo step vars + if turbo_steps > 1: + turbo_prev_image, turbo_prev_frame_idx = prev_img, prev_frame + turbo_next_image, turbo_next_frame_idx = next_img, next_frame + + # advance start_frame to next frame + start_frame = next_frame + 1 + + frame_idx = start_frame + + # reset the mask vals as they are overwritten in the compose_mask algorithm + mask_vals = {} + noise_mask_vals = {} + + mask_vals['everywhere'] = Image.new('1', (args.W, args.H), 1) + noise_mask_vals['everywhere'] = Image.new('1', (args.W, args.H), 1) + + mask_image = None + + if args.use_init and ((args.init_image != None and args.init_image != '') or args.init_image_box != None): + _, mask_image = load_img(args.init_image, + args.init_image_box, + shape=(args.W, args.H), + use_alpha_as_mask=args.use_alpha_as_mask) + mask_vals['video_mask'] = mask_image + noise_mask_vals['video_mask'] = mask_image + + # Grab the first frame masks since they wont be provided until next frame + # Video mask overrides the init image mask, also, won't be searching for init_mask if use_mask_video is set + # Made to solve https://github.com/deforum-art/deforum-for-automatic1111-webui/issues/386 + if anim_args.use_mask_video: + + args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + + mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + noise_mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + elif mask_image is None and args.use_mask: + mask_vals['video_mask'] = get_mask(args) + noise_mask_vals['video_mask'] = get_mask(args) # TODO?: add a different default noisc mask + + # get color match for 'Image' color coherence only once, before loop + if anim_args.color_coherence == 'Image': + color_match_sample = load_image(anim_args.color_coherence_image_path, None) + color_match_sample = color_match_sample.resize((args.W, args.H), PIL.Image.LANCZOS) + color_match_sample = cv2.cvtColor(np.array(color_match_sample), cv2.COLOR_RGB2BGR) + + # Webui + state.job_count = anim_args.max_frames + last_preview_frame = 0 + + while frame_idx < anim_args.max_frames: + # Webui + + state.job = f"frame {frame_idx + 1}/{anim_args.max_frames}" + state.job_no = frame_idx + 1 + + if state.skipped: + print("\n** PAUSED **") + state.skipped = False + while not state.skipped: + time.sleep(0.1) + print("** RESUMING **") + + print(f"\033[36mAnimation frame: \033[0m{frame_idx}/{anim_args.max_frames} ") + + noise = keys.noise_schedule_series[frame_idx] + strength = keys.strength_schedule_series[frame_idx] + scale = keys.cfg_scale_schedule_series[frame_idx] + contrast = keys.contrast_schedule_series[frame_idx] + kernel = int(keys.kernel_schedule_series[frame_idx]) + sigma = keys.sigma_schedule_series[frame_idx] + amount = keys.amount_schedule_series[frame_idx] + threshold = keys.threshold_schedule_series[frame_idx] + cadence_flow_factor = keys.cadence_flow_factor_schedule_series[frame_idx] + redo_flow_factor = keys.redo_flow_factor_schedule_series[frame_idx] + hybrid_comp_schedules = { + "alpha": keys.hybrid_comp_alpha_schedule_series[frame_idx], + "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[frame_idx], + "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[frame_idx], + "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[frame_idx]), + "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[frame_idx]), + "flow_factor": keys.hybrid_flow_factor_schedule_series[frame_idx] + } + scheduled_sampler_name = None + scheduled_clipskip = None + scheduled_noise_multiplier = None + scheduled_ddim_eta = None + scheduled_ancestral_eta = None + + mask_seq = None + noise_mask_seq = None + if anim_args.enable_steps_scheduling and keys.steps_schedule_series[frame_idx] is not None: + args.steps = int(keys.steps_schedule_series[frame_idx]) + if anim_args.enable_sampler_scheduling and keys.sampler_schedule_series[frame_idx] is not None: + scheduled_sampler_name = keys.sampler_schedule_series[frame_idx].casefold() + if anim_args.enable_clipskip_scheduling and keys.clipskip_schedule_series[frame_idx] is not None: + scheduled_clipskip = int(keys.clipskip_schedule_series[frame_idx]) + if anim_args.enable_noise_multiplier_scheduling and keys.noise_multiplier_schedule_series[frame_idx] is not None: + scheduled_noise_multiplier = float(keys.noise_multiplier_schedule_series[frame_idx]) + if anim_args.enable_ddim_eta_scheduling and keys.ddim_eta_schedule_series[frame_idx] is not None: + scheduled_ddim_eta = float(keys.ddim_eta_schedule_series[frame_idx]) + if anim_args.enable_ancestral_eta_scheduling and keys.ancestral_eta_schedule_series[frame_idx] is not None: + scheduled_ancestral_eta = float(keys.ancestral_eta_schedule_series[frame_idx]) + if args.use_mask and keys.mask_schedule_series[frame_idx] is not None: + mask_seq = keys.mask_schedule_series[frame_idx] + if anim_args.use_noise_mask and keys.noise_mask_schedule_series[frame_idx] is not None: + noise_mask_seq = keys.noise_mask_schedule_series[frame_idx] + + if args.use_mask and not anim_args.use_noise_mask: + noise_mask_seq = mask_seq + + depth = None + + if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): + # Unload the main checkpoint and load the depth model + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() + if predict_depths: depth_model.to(root.device) + + if turbo_steps == 1 and opts.data.get("deforum_save_gen_info_as_srt"): + params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) + params_string = format_animation_params(keys, prompt_series, frame_idx, params_to_print) + write_frame_subtitle(srt_filename, frame_idx, srt_frame_duration, f"F#: {frame_idx}; Cadence: false; Seed: {args.seed}; {params_string}") + params_string = None + + # emit in-between frames + if turbo_steps > 1: + tween_frame_start_idx = max(start_frame, frame_idx - turbo_steps) + cadence_flow = None + for tween_frame_idx in range(tween_frame_start_idx, frame_idx): + # update progress during cadence + state.job = f"frame {tween_frame_idx + 1}/{anim_args.max_frames}" + state.job_no = tween_frame_idx + 1 + # cadence vars + tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx) + advance_prev = turbo_prev_image is not None and tween_frame_idx > turbo_prev_frame_idx + advance_next = tween_frame_idx > turbo_next_frame_idx + + # optical flow cadence setup before animation warping + if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None': + if keys.strength_schedule_series[tween_frame_start_idx] > 0: + if cadence_flow is None and turbo_prev_image is not None and turbo_next_image is not None: + cadence_flow = get_flow_from_images(turbo_prev_image, turbo_next_image, anim_args.optical_flow_cadence, raft_model) / 2 + turbo_next_image = image_transform_optical_flow(turbo_next_image, -cadence_flow, 1) + + if opts.data.get("deforum_save_gen_info_as_srt"): + params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) + params_string = format_animation_params(keys, prompt_series, tween_frame_idx, params_to_print) + write_frame_subtitle(srt_filename, tween_frame_idx, srt_frame_duration, f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}") + params_string = None + + print(f"Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};") + + if depth_model is not None: + assert (turbo_next_image is not None) + depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision) + + if advance_prev: + turbo_prev_image, _ = anim_frame_warp(turbo_prev_image, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + if advance_next: + turbo_next_image, _ = anim_frame_warp(turbo_next_image, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + + # hybrid video motion - warps turbo_prev_image or turbo_next_image to match motion + if tween_frame_idx > 0: + if anim_args.hybrid_motion in ['Affine', 'Perspective']: + if anim_args.hybrid_motion_use_prev_img: + matrix = get_matrix_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, prev_img, anim_args.hybrid_motion) + if advance_prev: + turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, anim_args.hybrid_motion) + if advance_next: + turbo_next_image = image_transform_ransac(turbo_next_image, matrix, anim_args.hybrid_motion) + else: + matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, anim_args.hybrid_motion) + if advance_prev: + turbo_prev_image = image_transform_ransac(turbo_prev_image, matrix, anim_args.hybrid_motion) + if advance_next: + turbo_next_image = image_transform_ransac(turbo_next_image, matrix, anim_args.hybrid_motion) + if anim_args.hybrid_motion in ['Optical Flow']: + if anim_args.hybrid_motion_use_prev_img: + flow = get_flow_for_hybrid_motion_prev(tween_frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + if advance_prev: + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) + if advance_next: + turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) + prev_flow = flow + else: + flow = get_flow_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + if advance_prev: + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, flow, hybrid_comp_schedules['flow_factor']) + if advance_next: + turbo_next_image = image_transform_optical_flow(turbo_next_image, flow, hybrid_comp_schedules['flow_factor']) + prev_flow = flow + + # do optical flow cadence after animation warping + if cadence_flow is not None: + cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.W, args.H) + cadence_flow, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model, depth=depth, device=root.device, half_precision=root.half_precision) + cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.W, args.H) * tween + if advance_prev: + turbo_prev_image = image_transform_optical_flow(turbo_prev_image, cadence_flow_inc, cadence_flow_factor) + if advance_next: + turbo_next_image = image_transform_optical_flow(turbo_next_image, cadence_flow_inc, cadence_flow_factor) + + turbo_prev_frame_idx = turbo_next_frame_idx = tween_frame_idx + + if turbo_prev_image is not None and tween < 1.0: + img = turbo_prev_image * (1.0 - tween) + turbo_next_image * tween + else: + img = turbo_next_image + + # intercept and override to grayscale + if anim_args.color_force_grayscale: + img = cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY) + img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) + + # overlay mask + if args.overlay_mask and (anim_args.use_mask_video or args.use_mask): + img = do_overlay_mask(args, anim_args, img, tween_frame_idx, True) + + # get prev_img during cadence + prev_img = img + + # current image update for cadence frames (left commented because it doesn't currently update the preview) + # state.current_image = Image.fromarray(cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2RGB)) + + # saving cadence frames + filename = f"{root.timestring}_{tween_frame_idx:09}.png" + cv2.imwrite(os.path.join(args.outdir, filename), img) + if anim_args.save_depth_maps: + depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{tween_frame_idx:09}.png"), depth) + + # get color match for video outside of prev_img conditional + hybrid_available = anim_args.hybrid_composite != 'None' or anim_args.hybrid_motion in ['Optical Flow', 'Affine', 'Perspective'] + if anim_args.color_coherence == 'Video Input' and hybrid_available: + if int(frame_idx) % int(anim_args.color_coherence_video_every_N_frames) == 0: + prev_vid_img = Image.open(os.path.join(args.outdir, 'inputframes', get_frame_name(anim_args.video_init_path) + f"{frame_idx:09}.jpg")) + prev_vid_img = prev_vid_img.resize((args.W, args.H), PIL.Image.LANCZOS) + color_match_sample = np.asarray(prev_vid_img) + color_match_sample = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2BGR) + + # after 1st frame, prev_img exists + if prev_img is not None: + # apply transforms to previous frame + prev_img, depth = anim_frame_warp(prev_img, args, anim_args, keys, frame_idx, depth_model, depth=None, device=root.device, half_precision=root.half_precision) + + # do hybrid compositing before motion + if anim_args.hybrid_composite == 'Before Motion': + args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, hybrid_comp_schedules, root) + + # hybrid video motion - warps prev_img to match motion, usually to prepare for compositing + if anim_args.hybrid_motion in ['Affine', 'Perspective']: + if anim_args.hybrid_motion_use_prev_img: + matrix = get_matrix_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, prev_img, anim_args.hybrid_motion) + else: + matrix = get_matrix_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, anim_args.hybrid_motion) + prev_img = image_transform_ransac(prev_img, matrix, anim_args.hybrid_motion) + if anim_args.hybrid_motion in ['Optical Flow']: + if anim_args.hybrid_motion_use_prev_img: + flow = get_flow_for_hybrid_motion_prev(frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, prev_img, anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + else: + flow = get_flow_for_hybrid_motion(frame_idx - 1, (args.W, args.H), inputfiles, hybrid_frame_path, prev_flow, anim_args.hybrid_flow_method, raft_model, + anim_args.hybrid_flow_consistency, anim_args.hybrid_consistency_blur, anim_args.hybrid_comp_save_extra_frames) + prev_img = image_transform_optical_flow(prev_img, flow, hybrid_comp_schedules['flow_factor']) + prev_flow = flow + + # do hybrid compositing after motion (normal) + if anim_args.hybrid_composite == 'Normal': + args, prev_img = hybrid_composite(args, anim_args, frame_idx, prev_img, depth_model, hybrid_comp_schedules, root) + + # apply color matching + if anim_args.color_coherence != 'None': + if color_match_sample is None: + color_match_sample = prev_img.copy() + else: + prev_img = maintain_colors(prev_img, color_match_sample, anim_args.color_coherence) + + # intercept and override to grayscale + if anim_args.color_force_grayscale: + prev_img = cv2.cvtColor(prev_img, cv2.COLOR_BGR2GRAY) + prev_img = cv2.cvtColor(prev_img, cv2.COLOR_GRAY2BGR) + + # apply scaling + contrast_image = (prev_img * contrast).round().astype(np.uint8) + # anti-blur + if amount > 0: + contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold, mask_image if args.use_mask else None) + # apply frame noising + if args.use_mask or anim_args.use_noise_mask: + root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq, noise_mask_vals, Image.fromarray(cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB))) + noised_image = add_noise(contrast_image, noise, args.seed, anim_args.noise_type, + (anim_args.perlin_w, anim_args.perlin_h, anim_args.perlin_octaves, anim_args.perlin_persistence), + root.noise_mask, args.invert_mask) + + # use transformed previous frame as init for current + args.use_init = True + root.init_sample = Image.fromarray(cv2.cvtColor(noised_image, cv2.COLOR_BGR2RGB)) + args.strength = max(0.0, min(1.0, strength)) + + args.scale = scale + + # Pix2Pix Image CFG Scale - does *nothing* with non pix2pix checkpoints + args.pix2pix_img_cfg_scale = float(keys.pix2pix_img_cfg_scale_series[frame_idx]) + + # grab prompt for current frame + args.prompt = prompt_series[frame_idx] + + if args.seed_behavior == 'schedule' or parseq_adapter.manages_seed(): + args.seed = int(keys.seed_schedule_series[frame_idx]) + + if anim_args.enable_checkpoint_scheduling: + args.checkpoint = keys.checkpoint_schedule_series[frame_idx] + else: + args.checkpoint = None + + # SubSeed scheduling + if anim_args.enable_subseed_scheduling: + root.subseed = int(keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = float(keys.subseed_strength_schedule_series[frame_idx]) + + if parseq_adapter.manages_seed(): + anim_args.enable_subseed_scheduling = True + root.subseed = int(keys.subseed_schedule_series[frame_idx]) + root.subseed_strength = keys.subseed_strength_schedule_series[frame_idx] + + # set value back into the prompt - prepare and report prompt and seed + args.prompt = prepare_prompt(args.prompt, anim_args.max_frames, args.seed, frame_idx) + + # grab init image for current frame + if using_vid_init: + init_frame = get_next_frame(args.outdir, anim_args.video_init_path, frame_idx, False) + print(f"Using video init frame {init_frame}") + args.init_image = init_frame + args.init_image_box = None # init_image_box not used in this case + args.strength = max(0.0, min(1.0, strength)) + if anim_args.use_mask_video: + args.mask_file = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + root.noise_mask = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + + mask_vals['video_mask'] = get_mask_from_file(get_next_frame(args.outdir, anim_args.video_mask_path, frame_idx, True), args) + + if args.use_mask: + args.mask_image = compose_mask_with_check(root, args, mask_seq, mask_vals, root.init_sample) if root.init_sample is not None else None # we need it only after the first frame anyway + + # setting up some arguments for the looper + loop_args.imageStrength = loopSchedulesAndData.image_strength_schedule_series[frame_idx] + loop_args.blendFactorMax = loopSchedulesAndData.blendFactorMax_series[frame_idx] + loop_args.blendFactorSlope = loopSchedulesAndData.blendFactorSlope_series[frame_idx] + loop_args.tweeningFrameSchedule = loopSchedulesAndData.tweening_frames_schedule_series[frame_idx] + loop_args.colorCorrectionFactor = loopSchedulesAndData.color_correction_factor_series[frame_idx] + loop_args.use_looper = loopSchedulesAndData.use_looper + loop_args.imagesToKeyframe = loopSchedulesAndData.imagesToKeyframe + + if 'img2img_fix_steps' in opts.data and opts.data["img2img_fix_steps"]: # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations + opts.data["img2img_fix_steps"] = False + if scheduled_clipskip is not None: + opts.data["CLIP_stop_at_last_layers"] = scheduled_clipskip + if scheduled_noise_multiplier is not None: + opts.data["initial_noise_multiplier"] = scheduled_noise_multiplier + if scheduled_ddim_eta is not None: + opts.data["eta_ddim"] = scheduled_ddim_eta + if scheduled_ancestral_eta is not None: + opts.data["eta_ancestral"] = scheduled_ancestral_eta + + if anim_args.animation_mode == '3D' and (cmd_opts.lowvram or cmd_opts.medvram): + if predict_depths: depth_model.to('cpu') + devices.torch_gc() + lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) + sd_hijack.model_hijack.hijack(sd_model) + + optical_flow_redo_generation = anim_args.optical_flow_redo_generation if not args.motion_preview_mode else 'None' + + # optical flow redo before generation + if optical_flow_redo_generation != 'None' and prev_img is not None and strength > 0: + stored_seed = args.seed + args.seed = random.randint(0, 2 ** 32 - 1) + print(f"Optical flow redo is diffusing and warping using {optical_flow_redo_generation} and seed {args.seed} optical flow before generation.") + + disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) + disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) + disposable_flow = get_flow_from_images(prev_img, disposable_image, optical_flow_redo_generation, raft_model) + disposable_image = cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB) + disposable_image = image_transform_optical_flow(disposable_image, disposable_flow, redo_flow_factor) + args.seed = stored_seed + root.init_sample = Image.fromarray(disposable_image) + del (disposable_image, disposable_flow, stored_seed) + gc.collect() + + # diffusion redo + if int(anim_args.diffusion_redo) > 0 and prev_img is not None and strength > 0 and not args.motion_preview_mode: + stored_seed = args.seed + for n in range(0, int(anim_args.diffusion_redo)): + print(f"Redo generation {n + 1} of {int(anim_args.diffusion_redo)} before final generation") + args.seed = random.randint(0, 2 ** 32 - 1) + disposable_image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) + disposable_image = cv2.cvtColor(np.array(disposable_image), cv2.COLOR_RGB2BGR) + # color match on last one only + if n == int(anim_args.diffusion_redo): + disposable_image = maintain_colors(prev_img, color_match_sample, anim_args.color_coherence) + args.seed = stored_seed + root.init_sample = Image.fromarray(cv2.cvtColor(disposable_image, cv2.COLOR_BGR2RGB)) + del (disposable_image, stored_seed) + gc.collect() + + # generation + image = generate(args, keys, anim_args, loop_args, controlnet_args, root, parseq_adapter, frame_idx, sampler_name=scheduled_sampler_name) -def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) - web_ui_utils.init_job(data) - key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) - run_render_animation(data, key_frames) - data.animation_mode.unload_raft_and_depth_model() - - -# @log_utils.with_suppressed_table_printing -def run_render_animation(data: RenderData, key_frames: List[KeyFrame]): - for key_frame in key_frames: - if is_resume(data, key_frame): - continue - pre_process_key_frame_and_emit_tweens(data, key_frame) - image = key_frame.generate() if image is None: - log_utils.print_warning_generate_returned_no_image() break - post_process_key_frame(key_frame, image) - - -def pre_process_key_frame_and_emit_tweens(data, key_frame): - memory_utils.handle_med_or_low_vram_before_step(data) - web_ui_utils.update_job(data) - if key_frame.has_tween_frames(): - emit_tweens(data, key_frame) - log_utils.print_animation_frame_info(key_frame.i, data.args.anim_args.max_frames) - key_frame.maybe_write_frame_subtitle() - frame_tube = img_2_img_tubes.frame_transformation_tube - contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube - key_frame.prepare_generation(frame_tube, contrasted_noise_tube) - - -def post_process_key_frame(key_frame, image): - if not image_utils.is_PIL(image): # check is required when resuming from timestring - image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) - state.assign_current_image(image) - key_frame.after_diffusion(image) - web_ui_utils.update_status_tracker(key_frame.render_data) - - -def is_resume(data, key_step): - filename = filename_utils.frame_filename(data, key_step.i) - full_path = Path(data.output_directory) / filename - is_file_existing = os.path.exists(full_path) - if is_file_existing: - log_utils.warn(f"Frame {filename} exists, skipping to next key frame.") - key_step.render_data.args.args.seed = key_step.next_seed() - return is_file_existing - - -def emit_tweens(data, key_step): - _update_pseudo_cadence(data, len(key_step.tweens) - 1) - log_utils.print_parseq_table_at_start(data, key_step) - log_utils.print_tween_frame_from_to_info(key_step) - grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube - overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube - tweens = _tweens_with_progress(key_step) - [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] - - -def _update_pseudo_cadence(data, value): - data.turbo.cadence = value - data.parseq_adapter.cadence = value - data.parseq_adapter.a1111_cadence = value - data.args.anim_args.diffusion_cadence = value - data.args.anim_args.optical_flow_cadence = value - data.args.anim_args.cadence_flow_factor_schedule = value - - -def _tweens_with_progress(key_step): - # only use tween progress bar when extra console output (aka "dev mode") is disabled. - return (tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') - if not log_utils.is_verbose() - else key_step.tweens) + + # do hybrid video after generation + if frame_idx > 0 and anim_args.hybrid_composite == 'After Generation': + image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + args, image = hybrid_composite(args, anim_args, frame_idx, image, depth_model, hybrid_comp_schedules, root) + image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) + + # color matching on first frame is after generation, color match was collected earlier, so we do an extra generation to avoid the corruption introduced by the color match of first output + if frame_idx == 0 and (anim_args.color_coherence == 'Image' or (anim_args.color_coherence == 'Video Input' and hybrid_available)): + image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, anim_args.color_coherence) + image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) + elif color_match_sample is not None and anim_args.color_coherence != 'None' and not anim_args.legacy_colormatch: + image = maintain_colors(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR), color_match_sample, anim_args.color_coherence) + image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)) + + # intercept and override to grayscale + if anim_args.color_force_grayscale: + image = ImageOps.grayscale(image) + image = ImageOps.colorize(image, black="black", white="white") + + # overlay mask + if args.overlay_mask and (anim_args.use_mask_video or args.use_mask): + image = do_overlay_mask(args, anim_args, image, frame_idx) + + # on strength 0, set color match to generation + if ((not anim_args.legacy_colormatch and not args.use_init) or (anim_args.legacy_colormatch and strength == 0)) and not anim_args.color_coherence in ['Image', 'Video Input']: + color_match_sample = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR) + + opencv_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR) + if not using_vid_init: + prev_img = opencv_image + + if turbo_steps > 1: + turbo_prev_image, turbo_prev_frame_idx = turbo_next_image, turbo_next_frame_idx + turbo_next_image, turbo_next_frame_idx = opencv_image, frame_idx + frame_idx += turbo_steps + else: + filename = f"{root.timestring}_{frame_idx:09}.png" + save_image(image, 'PIL', filename, args, video_args, root) + + if anim_args.save_depth_maps: + if cmd_opts.lowvram or cmd_opts.medvram: + lowvram.send_everything_to_cpu() + sd_hijack.model_hijack.undo_hijack(sd_model) + devices.torch_gc() + depth_model.to(root.device) + depth = depth_model.predict(opencv_image, anim_args.midas_weight, root.half_precision) + depth_model.save(os.path.join(args.outdir, f"{root.timestring}_depth_{frame_idx:09}.png"), depth) + if cmd_opts.lowvram or cmd_opts.medvram: + depth_model.to('cpu') + devices.torch_gc() + lowvram.setup_for_low_vram(sd_model, cmd_opts.medvram) + sd_hijack.model_hijack.hijack(sd_model) + frame_idx += 1 + + state.assign_current_image(image) + + args.seed = next_seed(args, root) + + last_preview_frame = render_preview(args, anim_args, video_args, root, frame_idx, last_preview_frame) + + JobStatusTracker().update_phase(root.job_id, phase="GENERATING", progress=frame_idx/anim_args.max_frames) + + + if predict_depths and not keep_in_vram: + depth_model.delete_model() # handles adabins too + + if load_raft: + raft_model.delete_model() diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py index 6fcaf2d60..dd48c3845 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame_distribution.py @@ -6,22 +6,26 @@ class KeyFrameDistribution(Enum): + OFF = "Off" PARSEQ_ONLY = "Parseq Only" # cadence is ignored. all frames not present in the Parseq table are handled as tweens. UNIFORM_WITH_PARSEQ = "Uniform with Parseq" # similar to uniform, but parseq key frame diffusion is enforced. - UNIFORM_SPACING = "Uniform Spacing" # distance defined by cadence - RANDOM_SPACING = "Random Spacing" # distance loosely based on cadence (poc) - RANDOM_PLACEMENT = "Random Placement" # no relation to cadence (poc) @staticmethod def from_UI_tab(data): - is_uniform_with_parseq = data.args.anim_args.neocore_key_index_distribution == "Uniform with Parseq" - return (KeyFrameDistribution.UNIFORM_WITH_PARSEQ - if is_uniform_with_parseq - else KeyFrameDistribution.PARSEQ_ONLY) + redistribution = data.args.parseq_args.parseq_key_frame_redistribution + match redistribution: + case "Off": + return KeyFrameDistribution.OFF + case "Parseq Only (no cadence)": + return KeyFrameDistribution.PARSEQ_ONLY + case "Uniform with Parseq (pseudo-cadence)": + return KeyFrameDistribution.UNIFORM_WITH_PARSEQ + case _: + raise ValueError(f"Invalid parseq_key_frame_redistribution from UI: {redistribution}") @staticmethod def default(): - return KeyFrameDistribution.PARSEQ_ONLY # same as UNIFORM_SPACING, if no Parseq keys are present. + return KeyFrameDistribution.OFF def calculate(self, key_frames, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: key_indices: List[int] = self._calculate(start_index, max_frames, num_key_steps, parseq_adapter) @@ -31,18 +35,15 @@ def calculate(self, key_frames, start_index, max_frames, num_key_steps, parseq_a def _calculate(self, start_index, max_frames, num_key_steps, parseq_adapter) -> List[int]: match self: - case KeyFrameDistribution.PARSEQ_ONLY: + case KeyFrameDistribution.PARSEQ_ONLY: # same as UNIFORM_SPACING, if no Parseq keys are present. return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) case KeyFrameDistribution.UNIFORM_WITH_PARSEQ: return self._uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_adapter) - case KeyFrameDistribution.UNIFORM_SPACING: - return self._uniform_indexes(start_index, max_frames, num_key_steps) - case KeyFrameDistribution.RANDOM_SPACING: - return self._random_spacing_indexes(start_index, max_frames, num_key_steps) - case KeyFrameDistribution.RANDOM_PLACEMENT: - return self._random_placement_indexes(start_index, max_frames, num_key_steps) + case KeyFrameDistribution.OFF: + log_utils.warn("Called new core without key frame redistribution. Using 'PARSEQ_ONLY'.") + return self._parseq_only_indexes(start_index, max_frames, num_key_steps, parseq_adapter) case _: - raise ValueError(f"Invalid KeyIndexDistribution: {self}") + raise ValueError(f"Invalid KeyFrameDistribution: {self}") @staticmethod def _uniform_indexes(start_index, max_frames, num_key_steps): @@ -84,28 +85,3 @@ def _uniform_with_parseq_indexes(start_index, max_frames, num_key_steps, parseq_ key_frames.sort() assert len(key_frames) == num_key_steps return key_frames - - @staticmethod - def _random_spacing_indexes(start_index, max_frames, num_key_steps): - uniform_indexes = KeyFrameDistribution._uniform_indexes(start_index, max_frames, num_key_steps) - indexes = [start_index + 1, max_frames] # Enforce first and last indices - total_spacing = max_frames - start_index - 1 # Calculate initial total spacing - noise_factor = 0.5 # Higher value creates more variation - for i in range(1, num_key_steps - 1): - base_index = uniform_indexes[i] - noise = random.uniform(-noise_factor, noise_factor) * (total_spacing / (num_key_steps - 1)) - index = int(base_index + noise) - index = max(start_index + 1, min(index, max_frames - 1)) - indexes.append(index) - total_spacing -= index - indexes[i - 1] - indexes.sort(key=lambda key_index: key_index) - return indexes - - @staticmethod - def _random_placement_indexes(start_index, max_frames, num_key_steps): - indexes = [start_index + 1, max_frames] # Enforce first and last indices - for _ in range(1, num_key_steps - 1): - index = random.randint(start_index + 1, max_frames - 1) - indexes.append(index) - indexes.sort(key=lambda i: i) - return indexes diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py new file mode 100644 index 000000000..ad3621b8e --- /dev/null +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -0,0 +1,90 @@ +import os +from pathlib import Path +from typing import List + +# noinspection PyUnresolvedReferences +from modules.shared import cmd_opts, progress_print_out, state +from tqdm import tqdm + +from . import img_2_img_tubes # noqa +from .data.frame import KeyFrameDistribution, KeyFrame # noqa +from .data.render_data import RenderData # noqa +from .util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils # noqa + + +def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): + log_utils.debug("Using new render core.") + data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) + web_ui_utils.init_job(data) + key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) + run_render_animation(data, key_frames) + data.animation_mode.unload_raft_and_depth_model() + + +# @log_utils.with_suppressed_table_printing +def run_render_animation(data: RenderData, key_frames: List[KeyFrame]): + for key_frame in key_frames: + if is_resume(data, key_frame): + continue + pre_process_key_frame_and_emit_tweens(data, key_frame) + image = key_frame.generate() + if image is None: + log_utils.print_warning_generate_returned_no_image() + break + post_process_key_frame(key_frame, image) + + +def pre_process_key_frame_and_emit_tweens(data, key_frame): + memory_utils.handle_med_or_low_vram_before_step(data) + web_ui_utils.update_job(data) + if key_frame.has_tween_frames(): + emit_tweens(data, key_frame) + log_utils.print_animation_frame_info(key_frame.i, data.args.anim_args.max_frames) + key_frame.maybe_write_frame_subtitle() + frame_tube = img_2_img_tubes.frame_transformation_tube + contrasted_noise_tube = img_2_img_tubes.contrasted_noise_transformation_tube + key_frame.prepare_generation(frame_tube, contrasted_noise_tube) + + +def post_process_key_frame(key_frame, image): + if not image_utils.is_PIL(image): # check is required when resuming from timestring + image = img_2_img_tubes.conditional_frame_transformation_tube(key_frame)(image) + state.assign_current_image(image) + key_frame.after_diffusion(image) + web_ui_utils.update_status_tracker(key_frame.render_data) + + +def is_resume(data, key_step): + filename = filename_utils.frame_filename(data, key_step.i) + full_path = Path(data.output_directory) / filename + is_file_existing = os.path.exists(full_path) + if is_file_existing: + log_utils.warn(f"Frame {filename} exists, skipping to next key frame.") + key_step.render_data.args.args.seed = key_step.next_seed() + return is_file_existing + + +def emit_tweens(data, key_step): + _update_pseudo_cadence(data, len(key_step.tweens) - 1) + log_utils.print_tween_frame_from_to_info(key_step) + grayscale_tube = img_2_img_tubes.conditional_force_tween_to_grayscale_tube + overlay_mask_tube = img_2_img_tubes.conditional_add_overlay_mask_tube + tweens = _tweens_with_progress(key_step) + [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] + + +def _update_pseudo_cadence(data, value): + data.turbo.cadence = value + data.parseq_adapter.cadence = value + data.parseq_adapter.a1111_cadence = value + data.args.anim_args.diffusion_cadence = value + data.args.anim_args.optical_flow_cadence = value + data.args.anim_args.cadence_flow_factor_schedule = value + + +def _tweens_with_progress(key_step): + # only use tween progress bar when extra console output (aka "dev mode") is disabled. + return (tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') + if not log_utils.is_verbose() + else key_step.tweens) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index ffcb46575..4016a1e82 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -26,11 +26,6 @@ def is_verbose(): return opts.data.get("deforum_debug_mode_enabled", False) -def print_parseq_table_at_start(data, key_step): - if key_step.i == 1: - data.parseq_adapter.print_parseq_table() - - def print_tween_frame_from_to_info(key_step, is_disabled=True): if not is_disabled: # replaced with prog bar, but value info print may be useful tween_values = key_step.tween_values diff --git a/scripts/deforum_helpers/ui_elements.py b/scripts/deforum_helpers/ui_elements.py index 8492295b0..768366a56 100644 --- a/scripts/deforum_helpers/ui_elements.py +++ b/scripts/deforum_helpers/ui_elements.py @@ -386,6 +386,10 @@ def get_tab_init(d, da, dp): parseq_non_schedule_overrides = create_gr_elem(dp.parseq_non_schedule_overrides) with FormRow(): parseq_use_deltas = create_gr_elem(dp.parseq_use_deltas) + gr.HTML(value=f"""
""") + with FormRow(): + parseq_key_frame_redistribution = create_gr_elem(dp.parseq_key_frame_redistribution) + create_keyframe_redistribution_info() return {k: v for k, v in {**locals(), **vars()}.items()} def get_tab_hybrid(da): @@ -582,57 +586,20 @@ def get_tab_output(da, dv): return {k: v for k, v in {**locals(), **vars()}.items()} -def get_tab_neocore(): +def create_keyframe_redistribution_info(): bars_mark = "📊" - check_mark = "✔️" - cross_mark = "❌" - warn_mark = "⚠️" - bulb_mark = "💡" - with gr.TabItem('Neo Core', elem_id='neocore_tab'): - gr.HTML(value=f"""

-

Parseq Instructions:
-
    -
  1. Setup your Parseq workflow as usual, but with high FPS (60?) and with high cadence (30?).
    2. -
  2. The refactored render core ensures that every frame in the Parseq table is diffused. It may therefore easily be used with just a fixed value for 'strengh' like perhaps '0.4'. -
-
Deforum Instructions:
- Select one of the following key index distributions: -
    -
  1. {bars_mark} Parseq Only: Only frames with an entry in the Parseq table are diffused. Actual cadence settings are ignored and all frames not defined in Parseq are handled as if they would be cadence frames.
    2. -
  2. {bars_mark} Uniform with Parseq: Calculates uniform cadence distribution but rearranges them to preserve proper Parseq synchronization at high cadence. Cadence may be understood as 'pseudo cadence'. A cadence value of '30' may more correctly be understood as 'about 30' in this mode.
    3. -
-

""") - neocore_key_index_distribution = gr.Dropdown( - label="Select Key Index Distribution:", - choices=["Parseq Only", "Uniform with Parseq"], # TODO use KeyIndexDistribution enum - value="Parseq Only") - gr.HTML(value=f"""

-

Currently required settings to avoid errors or undefined behaviour:
-
    -
  1. {warn_mark} In both cases set "cadence" to a value larger than 1 and make sure it's the same in Parseq and in Deforum.
    2. -
  2. {warn_mark} Go to the Deforum "Keyframe" tab, select the "Coherence" tab and set everything to 'None'.
    3. -
-
Recommendations:
-
    -
  1. {bulb_mark} "'Uniform with Parseq' is able to handle high cadence values, but the number of actually diffused frames is still dictated by the cadence setting.
  2. -
  3. {bulb_mark} "Consider rendering at 60FPS. It shouldn't take much longer since it doesn't affect the number of diffusions."
  4. -
  5. {bulb_mark} "In 'Uniform with Parseq' mode, consider a cadence of 30, at 60FPS it will ensure two diffusions per seconds, which goes well for a clip synchonized to a 120BPM tune."
  6. -
-
Should be working:
-
    -
  1. {check_mark} Parseq based key frame distributions. Currently the only reason to use this build.
    2. -
  2. {check_mark} Subtitle generation (but may require manual cleanup to remove residual cadence values).
    3. -
  3. {check_mark} Depth warping.
    4. -
-
Currently untested or not working:
-
    -
  1. {cross_mark} Hybrid Video.
    2. -
  2. {cross_mark} Optical Flow and Color Coherence.
    3. -
  3. {cross_mark} Control Net (idk?).
    4. -
-
TL;DR
-
    -
  1. Setup Parseq workflow at 60 FPS with cadence 30, ideally with audio sync and 3d rotation etc, then set a fixed value for 'strength' at perhaps 0.4
    2. -
  2. Setup Deforum with 60 FPS and cadence 30 as well and disable everything in the "Keyframes" - "Coherence" tab. Then do the rest as usual and go...
    3. -

""") - return {k: v for k, v in {**locals(), **vars()}.items()} + gr.HTML(value=f"""

+ Parseq keyframe redistribution ensures that every frame in the Parseq table is diffused. + It may easily be used at high FPS (e.g. '60') with just a fixed value for 'strength' in Parseq \ + (e.g. '0.4' for all frames). +

    +
  1. {bars_mark} Off: Key frames are not redistributed. Cadence settings are fully respected.
    2. +
  2. {bars_mark} Parseq Only: Only frames with an entry in the Parseq table are diffused. \ + Actual cadence settings are ignored and all frames not defined in Parseq are handled \ + as if they would be cadence frames. Recommended to be used with high FPS settings.
    3. +
  3. {bars_mark} Uniform with Parseq: Calculates uniform cadence distribution \ + but rearranges them to preserve proper Parseq synchronization at high cadence (e.g. '30'). \ + Cadence may be understood as 'pseudo cadence'. \ + A cadence value of '30' may more correctly be understood as 'about 30' in this mode.
    4. +
+

""") diff --git a/scripts/deforum_helpers/ui_left.py b/scripts/deforum_helpers/ui_left.py index 205aabd30..1f46cafc2 100644 --- a/scripts/deforum_helpers/ui_left.py +++ b/scripts/deforum_helpers/ui_left.py @@ -20,8 +20,8 @@ from .gradio_funcs import change_css, handle_change_functions from .args import DeforumArgs, DeforumAnimArgs, ParseqArgs, DeforumOutputArgs, RootArgs, LoopArgs from .deforum_controlnet import setup_controlnet_ui -from .ui_elements import (get_tab_run, get_tab_keyframes, get_tab_prompts, get_tab_init, - get_tab_hybrid, get_tab_output, get_tab_neocore) +from .ui_elements import (get_tab_run, get_tab_keyframes, get_tab_prompts, + get_tab_init, get_tab_hybrid, get_tab_output) def set_arg_lists(): # convert dicts to NameSpaces for easy working (args.param instead of args['param'] @@ -51,10 +51,9 @@ def setup_deforum_left_side_ui(): controlnet_dict = setup_controlnet_ui() # ControlNet tab tab_hybrid_params = get_tab_hybrid(da) # Hybrid tab tab_output_params = get_tab_output(da, dv) # Output tab - tab_neocore_params = get_tab_neocore() # Refactored Render Core tab # add returned gradio elements from main tabs to locals() for key, value in {**tab_run_params, **tab_keyframes_params, **tab_prompts_params, **tab_init_params, - **controlnet_dict, **tab_hybrid_params, **tab_output_params, **tab_neocore_params}.items(): + **controlnet_dict, **tab_hybrid_params, **tab_output_params}.items(): locals()[key] = value # Gradio's Change functions - hiding and renaming elements based on other elements From 41fae0d35d9275983e87e1d5dc7c8405fee8da9c Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 3 Aug 2024 14:46:01 +0200 Subject: [PATCH 113/132] Obsolete call wrapper removed. --- scripts/deforum_helpers/rendering/data/turbo.py | 11 ++++++----- scripts/deforum_helpers/rendering/img_2_img_tubes.py | 6 +++--- scripts/deforum_helpers/rendering/new_core.py | 8 ++++---- scripts/deforum_helpers/rendering/util/call/hybrid.py | 7 ------- 4 files changed, 13 insertions(+), 19 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 6be5eb798..7dc59593c 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -4,10 +4,11 @@ from .subtitle import Srt from ..util.call.anim import call_anim_frame_warp -from ..util.call.hybrid import call_get_flow_from_images, call_get_flow_for_hybrid_motion_prev, \ - call_get_flow_for_hybrid_motion, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev +from ..util.call.hybrid import (call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion, + call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev) from ..util.call.resume import call_get_resume_vars -from ...hybrid_video import image_transform_ransac, image_transform_optical_flow, rel_flow_to_abs_flow +from ...hybrid_video import (get_flow_from_images, image_transform_ransac, + image_transform_optical_flow, rel_flow_to_abs_flow) @dataclass(init=True, frozen=False, repr=False, eq=True) @@ -99,8 +100,8 @@ def advance_hybrid_motion_ransac_transform(self, data, indexes, reference_images def advance_optical_flow_cadence_before_animation_warping(self, data, last_frame, tween_frame): if data.is_3d_or_2d_with_optical_flow(): if self._is_do_flow(data, tween_frame): - cadence = "RAFT" # FIXME data.args.anim_args.optical_flow_cadence - flow = call_get_flow_from_images(data, self.prev.image, self.next.image, cadence) + cadence = data.args.anim_args.optical_flow_cadence + flow = get_flow_from_images(self.prev.image, self.next.image, cadence, data.animation_mode.raft_model) tween_frame.cadence_flow = (flow / 2) if tween_frame.has_cadence(): self.advance_optical_flow(tween_frame) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index dc1bb0b6b..14fa25a60 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -7,10 +7,10 @@ from .data.frame.key_frame import KeyFrame from .data.render_data import RenderData -from .util.call.hybrid import call_get_flow_from_images, call_hybrid_composite +from .util.call.hybrid import call_hybrid_composite from .util.fun_utils import tube from ..colors import maintain_colors -from ..hybrid_video import image_transform_optical_flow +from ..hybrid_video import get_flow_from_images, image_transform_optical_flow from ..masks import do_overlay_mask """ @@ -54,7 +54,7 @@ def optical_flow_redo_tube(data: RenderData, key_frame: KeyFrame, optical_flow) return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), lambda img: image_transform_optical_flow( - img, call_get_flow_from_images(data, data.images.previous, img, optical_flow), + img, get_flow_from_images(data.images.previous, img, optical_flow, data.animation_mode.raft_model), key_frame.step_data.redo_flow_factor)) diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index ad3621b8e..f1e64d49a 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -6,10 +6,10 @@ from modules.shared import cmd_opts, progress_print_out, state from tqdm import tqdm -from . import img_2_img_tubes # noqa -from .data.frame import KeyFrameDistribution, KeyFrame # noqa -from .data.render_data import RenderData # noqa -from .util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils # noqa +from . import img_2_img_tubes +from .data.frame import KeyFrameDistribution, KeyFrame +from .data.render_data import RenderData +from .util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): diff --git a/scripts/deforum_helpers/rendering/util/call/hybrid.py b/scripts/deforum_helpers/rendering/util/call/hybrid.py index b05f7c2fc..73fbb6cce 100644 --- a/scripts/deforum_helpers/rendering/util/call/hybrid.py +++ b/scripts/deforum_helpers/rendering/util/call/hybrid.py @@ -1,6 +1,5 @@ from ....hybrid_video import ( # Functions related to flow calculation - get_flow_from_images, get_flow_for_hybrid_motion, get_flow_for_hybrid_motion_prev, @@ -12,12 +11,6 @@ hybrid_composite) -def call_get_flow_from_images(init, prev_image, next_image, cadence): - # cadence is currently either "optical_flow_redo_generation" or "init.args.anim_args.optical_flow_cadence" - # TODO try to init "optical_flow_redo_generation" early, then remove the "cadence" arg again - return get_flow_from_images(prev_image, next_image, cadence, init.animation_mode.raft_model) - - def call_get_flow_for_hybrid_motion_prev(init, i, image): mode = init.animation_mode aa = init.args.anim_args From f852a788ac714c4caa686ed2920b4371a0afcfb1 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 3 Aug 2024 15:41:00 +0200 Subject: [PATCH 114/132] New type for PIL image tubes and improved type hints in img_2_img tubes. Conversion between PIL and numpy images extracted. --- .../rendering/img_2_img_tubes.py | 45 ++++++++++++------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 14fa25a60..4e981f486 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -28,6 +28,19 @@ # ImageTubes are functions that take a MatLike image and return a newly processed (or the same unchanged) MatLike image. ImageTube = Callable[[MatLike], MatLike] +PilImageTube = Callable[[Image.Image], Image.Image] + + +def _bgr_to_rgb(bgr_img): + return cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB) + + +def _numpy_to_pil(np_image: MatLike) -> Image.Image: + return Image.fromarray(_bgr_to_rgb(np_image)) + + +def _pil_to_numpy(pil_image: Image.Image) -> MatLike: + return np.array(pil_image) def frame_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: @@ -51,47 +64,45 @@ def noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTub def optical_flow_redo_tube(data: RenderData, key_frame: KeyFrame, optical_flow) -> ImageTube: - return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2RGB), + return tube(lambda img: _bgr_to_rgb(img), lambda img: image_transform_optical_flow( img, get_flow_from_images(data.images.previous, img, optical_flow, data.animation_mode.raft_model), key_frame.step_data.redo_flow_factor)) # Conditional Tubes (can be switched on or off by providing a Callable[Boolean] `is_do_process` predicate). -def conditional_hybrid_video_after_generation_tube(key_frame: KeyFrame) -> ImageTube: +def conditional_hybrid_video_after_generation_tube(key_frame: KeyFrame) -> PilImageTube: data = key_frame.render_data step_data = key_frame.step_data - return tube(lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), - lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), - lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), + return tube(lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), + lambda img: _numpy_to_pil(img), is_do_process=lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) -def conditional_extra_color_match_tube(data: RenderData) -> ImageTube: +def conditional_extra_color_match_tube(data: RenderData) -> PilImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output return tube(lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), - lambda img: cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR), + lambda img: _numpy_to_pil(_pil_to_numpy(img)), # TODO? remove lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), - lambda img: Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), + lambda img: _numpy_to_pil(img), is_do_process=lambda: data.indexes.is_first_frame() and data.is_initialize_color_match( data.images.color_match)) def conditional_color_match_tube(key_frame: KeyFrame) -> ImageTube: # on strength 0, set color match to generation - return tube(lambda img: cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR), + return tube(lambda img: _bgr_to_rgb(np.asarray(img)), is_do_process=lambda: key_frame.render_data.is_do_color_match_conversion(key_frame)) -def conditional_force_to_grayscale_tube(data: RenderData) -> ImageTube: +def conditional_force_to_grayscale_tube(data: RenderData) -> PilImageTube: return tube(lambda img: ImageOps.grayscale(img), lambda img: ImageOps.colorize(img, black="black", white="white"), is_do_process=lambda: data.args.anim_args.color_force_grayscale) -def conditional_add_overlay_mask_tube(data: RenderData, is_tween) -> ImageTube: +def conditional_add_overlay_mask_tube(data: RenderData, is_tween) -> PilImageTube: is_use_overlay = data.args.args.overlay_mask is_use_mask = data.args.anim_args.use_mask_video or data.args.args.use_mask index = data.indexes.tween.i if is_tween else data.indexes.frame.i @@ -115,9 +126,9 @@ def contrasted_noise_transformation_tube(data: RenderData, key_frame: KeyFrame) return tube(lambda img: noise_tube(contrast_tube(img))) -def conditional_frame_transformation_tube(key_frame: KeyFrame, is_tween: bool = False) -> ImageTube: - hybrid_tube: ImageTube = conditional_hybrid_video_after_generation_tube(key_frame) - extra_tube: ImageTube = conditional_extra_color_match_tube(key_frame.render_data) - gray_tube: ImageTube = conditional_force_to_grayscale_tube(key_frame.render_data) - mask_tube: ImageTube = conditional_add_overlay_mask_tube(key_frame.render_data, is_tween) +def conditional_frame_transformation_tube(key_frame: KeyFrame, is_tween: bool = False) -> PilImageTube: + hybrid_tube: PilImageTube = conditional_hybrid_video_after_generation_tube(key_frame) + extra_tube: PilImageTube = conditional_extra_color_match_tube(key_frame.render_data) + gray_tube: PilImageTube = conditional_force_to_grayscale_tube(key_frame.render_data) + mask_tube: PilImageTube = conditional_add_overlay_mask_tube(key_frame.render_data, is_tween) return tube(lambda img: mask_tube(gray_tube(extra_tube(hybrid_tube(img))))) From cda621510a0a4ede2e3eb5206147bb62ffc8ac2f Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 3 Aug 2024 15:44:40 +0200 Subject: [PATCH 115/132] Moved conversion methods to image utils. --- .../rendering/img_2_img_tubes.py | 23 +++++-------------- .../rendering/util/image_utils.py | 14 +++++++++++ 2 files changed, 20 insertions(+), 17 deletions(-) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 4e981f486..6677466d8 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -7,6 +7,7 @@ from .data.frame.key_frame import KeyFrame from .data.render_data import RenderData +from .util import image_utils from .util.call.hybrid import call_hybrid_composite from .util.fun_utils import tube from ..colors import maintain_colors @@ -31,18 +32,6 @@ PilImageTube = Callable[[Image.Image], Image.Image] -def _bgr_to_rgb(bgr_img): - return cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB) - - -def _numpy_to_pil(np_image: MatLike) -> Image.Image: - return Image.fromarray(_bgr_to_rgb(np_image)) - - -def _pil_to_numpy(pil_image: Image.Image) -> MatLike: - return np.array(pil_image) - - def frame_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTube: # make sure `img` stays the last argument in each call. return tube(lambda img: key_frame.apply_frame_warp_transform(data, img), @@ -64,7 +53,7 @@ def noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTub def optical_flow_redo_tube(data: RenderData, key_frame: KeyFrame, optical_flow) -> ImageTube: - return tube(lambda img: _bgr_to_rgb(img), + return tube(lambda img: image_utils.bgr_to_rgb(img), lambda img: image_transform_optical_flow( img, get_flow_from_images(data.images.previous, img, optical_flow, data.animation_mode.raft_model), key_frame.step_data.redo_flow_factor)) @@ -75,7 +64,7 @@ def conditional_hybrid_video_after_generation_tube(key_frame: KeyFrame) -> PilIm data = key_frame.render_data step_data = key_frame.step_data return tube(lambda img: call_hybrid_composite(data, data.indexes.frame.i, img, step_data.hybrid_comp_schedules), - lambda img: _numpy_to_pil(img), + lambda img: image_utils.numpy_to_pil(img), is_do_process=lambda: data.indexes.is_not_first_frame() and data.is_hybrid_composite_after_generation()) @@ -83,16 +72,16 @@ def conditional_extra_color_match_tube(data: RenderData) -> PilImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output return tube(lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), - lambda img: _numpy_to_pil(_pil_to_numpy(img)), # TODO? remove + lambda img: image_utils.numpy_to_pil(image_utils.pil_to_numpy(img)), # TODO? remove lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), - lambda img: _numpy_to_pil(img), + lambda img: image_utils.numpy_to_pil(img), is_do_process=lambda: data.indexes.is_first_frame() and data.is_initialize_color_match( data.images.color_match)) def conditional_color_match_tube(key_frame: KeyFrame) -> ImageTube: # on strength 0, set color match to generation - return tube(lambda img: _bgr_to_rgb(np.asarray(img)), + return tube(lambda img: image_utils.bgr_to_rgb(np.asarray(img)), is_do_process=lambda: key_frame.render_data.is_do_color_match_conversion(key_frame)) diff --git a/scripts/deforum_helpers/rendering/util/image_utils.py b/scripts/deforum_helpers/rendering/util/image_utils.py index 1d715e372..2dc9183b0 100644 --- a/scripts/deforum_helpers/rendering/util/image_utils.py +++ b/scripts/deforum_helpers/rendering/util/image_utils.py @@ -2,12 +2,26 @@ import PIL import cv2 +import numpy as np +from PIL import Image from cv2.typing import MatLike from . import filename_utils from ..data.render_data import RenderData +def bgr_to_rgb(bgr_img): + return cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB) + + +def numpy_to_pil(np_image: MatLike) -> Image.Image: + return Image.fromarray(bgr_to_rgb(np_image)) + + +def pil_to_numpy(pil_image: Image.Image) -> MatLike: + return np.array(pil_image) + + def save_cadence_frame(data: RenderData, i: int, image: MatLike, is_overwrite: bool = True): filename = filename_utils.frame_filename(data, i) save_path: str = os.path.join(data.args.args.outdir, filename) From cb5bce3408a8ec8ccc91503b29c4d77d8ff39849 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 3 Aug 2024 16:07:38 +0200 Subject: [PATCH 116/132] Improved description of the new key frame redistribution modes. --- scripts/deforum_helpers/ui_elements.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/scripts/deforum_helpers/ui_elements.py b/scripts/deforum_helpers/ui_elements.py index 768366a56..9327fde46 100644 --- a/scripts/deforum_helpers/ui_elements.py +++ b/scripts/deforum_helpers/ui_elements.py @@ -590,15 +590,19 @@ def create_keyframe_redistribution_info(): bars_mark = "📊" gr.HTML(value=f"""

Parseq keyframe redistribution ensures that every frame in the Parseq table is diffused. - It may easily be used at high FPS (e.g. '60') with just a fixed value for 'strength' in Parseq \ - (e.g. '0.4' for all frames). + It may easily be used at high FPS with just a fixed value for 'strength' in Parseq \ + (e.g. '0.33' for all frames with no logic to detect dips). + Since keyframe redistribution allows for Parseq synchronization at high or no cadence, \ + the generation can be performed much faster compared to a traditional low cadence setup. + Resulting videos tend to be less jittery at high or no cadence, \ + but may introduce 'depth smear' when combined with fast movement.

  1. {bars_mark} Off: Key frames are not redistributed. Cadence settings are fully respected.
  2. {bars_mark} Parseq Only: Only frames with an entry in the Parseq table are diffused. \ Actual cadence settings are ignored and all frames not defined in Parseq are handled \ - as if they would be cadence frames. Recommended to be used with high FPS settings.
    3. + as if they were cadence frames. Recommended to be used at high FPS settings (e.g. '60').
  3. {bars_mark} Uniform with Parseq: Calculates uniform cadence distribution \ - but rearranges them to preserve proper Parseq synchronization at high cadence (e.g. '30'). \ + but rearranges some keyframes to preserve proper Parseq synchronization at high cadence (e.g. '30'). \ Cadence may be understood as 'pseudo cadence'. \ A cadence value of '30' may more correctly be understood as 'about 30' in this mode.
From 6e960b0801a802428eca495bd29287c026b1d042 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 5 Aug 2024 20:34:20 +0200 Subject: [PATCH 117/132] Fixed crash when using keyframe redistribution logic combined with optical flows, but handling it as an experimental feature by printing a warning as it doesn't quite behave as one shoul expect and may also be quetionable in general when used in a variable cadence context. --- .../deforum_helpers/rendering/data/render_data.py | 14 ++++++++++---- scripts/deforum_helpers/rendering/data/turbo.py | 9 ++++----- .../deforum_helpers/rendering/img_2_img_tubes.py | 3 ++- scripts/deforum_helpers/rendering/new_core.py | 10 +++++++++- scripts/deforum_helpers/ui_elements.py | 2 ++ 5 files changed, 27 insertions(+), 11 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 364bcfd15..d2ab74835 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -97,15 +97,24 @@ def is_3d(self): def is_3d_or_2d(self): return self.args.anim_args.animation_mode in ['2D', '3D'] + def has_parseq_keyframe_redistribution(self): + return self.args.parseq_args.parseq_key_frame_redistribution != "Off" + def has_optical_flow_cadence(self): return self.args.anim_args.optical_flow_cadence != 'None' + def has_optical_flow_redo(self): + return self.args.anim_args.optical_flow_redo_generation != 'None' + def is_3d_or_2d_with_optical_flow(self): return self.is_3d_or_2d() and self.has_optical_flow_cadence() def is_3d_with_med_or_low_vram(self): return self.is_3d() and memory_utils.is_low_or_med_vram() + def has_keyframe_redistribution(self): + return self.args.parseq_args + def width(self) -> int: return self.args.args.W @@ -189,12 +198,9 @@ def diffusion_redo_as_int(self): def has_positive_diffusion_redo(self): return self.diffusion_redo_as_int() > 0 - def optical_flow_redo_generation(self): - return self.args.anim_args.optical_flow_redo_generation - def optical_flow_redo_generation_if_not_in_preview_mode(self): is_not_preview = self.is_not_in_motion_preview_mode() - return self.optical_flow_redo_generation() if is_not_preview else 'None' # don't replace with None value + return self.args.anim_args.optical_flow_redo_generation if is_not_preview else 'None' def is_do_color_match_conversion(self, step): is_legacy_cm = self.args.anim_args.legacy_colormatch diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 7dc59593c..767a46f66 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -100,13 +100,12 @@ def advance_hybrid_motion_ransac_transform(self, data, indexes, reference_images def advance_optical_flow_cadence_before_animation_warping(self, data, last_frame, tween_frame): if data.is_3d_or_2d_with_optical_flow(): if self._is_do_flow(data, tween_frame): - cadence = data.args.anim_args.optical_flow_cadence - flow = get_flow_from_images(self.prev.image, self.next.image, cadence, data.animation_mode.raft_model) - tween_frame.cadence_flow = (flow / 2) + method = data.args.anim_args.optical_flow_cadence # string containing the flow method (e.g. "RAFT"). + flow = get_flow_from_images(self.prev.image, self.next.image, method, data.animation_mode.raft_model) + tween_frame.cadence_flow = flow / len(last_frame.tweens) if tween_frame.has_cadence(): self.advance_optical_flow(tween_frame) - flow_factor = 1.0 # FIXME.. - # flow_factor = 1.0 / (len(last_frame.tweens) + 1.0) + flow_factor = 1.0 self.next.image = image_transform_optical_flow(self.next.image, -tween_frame.cadence_flow, flow_factor) def _is_do_flow(self, data, tween_frame): diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 6677466d8..29f12a4da 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -53,7 +53,8 @@ def noise_transformation_tube(data: RenderData, key_frame: KeyFrame) -> ImageTub def optical_flow_redo_tube(data: RenderData, key_frame: KeyFrame, optical_flow) -> ImageTube: - return tube(lambda img: image_utils.bgr_to_rgb(img), + return tube(lambda img: image_utils.pil_to_numpy(img), + lambda img: image_utils.bgr_to_rgb(img), lambda img: image_transform_optical_flow( img, get_flow_from_images(data.images.previous, img, optical_flow, data.animation_mode.raft_model), key_frame.step_data.redo_flow_factor)) diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index f1e64d49a..9dc911eb5 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -15,6 +15,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): log_utils.debug("Using new render core.") data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) + _check_experimental_render_conditions(data) web_ui_utils.init_job(data) key_frames = KeyFrame.create_all_frames(data, KeyFrameDistribution.from_UI_tab(data)) run_render_animation(data, key_frames) @@ -73,12 +74,19 @@ def emit_tweens(data, key_step): [tween.emit_frame(key_step, grayscale_tube, overlay_mask_tube) for tween in tweens] +def _check_experimental_render_conditions(data): + if data.has_parseq_keyframe_redistribution(): + if data.has_optical_flow_cadence(): + log_utils.warn("Using Parseq keyframe redistribution with optical flow cadence. Results may be unexpected.") + if data.has_optical_flow_redo(): + log_utils.warn("Using Parseq keyframe redistribution with optical flow redo. Results may be unexpected.") + + def _update_pseudo_cadence(data, value): data.turbo.cadence = value data.parseq_adapter.cadence = value data.parseq_adapter.a1111_cadence = value data.args.anim_args.diffusion_cadence = value - data.args.anim_args.optical_flow_cadence = value data.args.anim_args.cadence_flow_factor_schedule = value diff --git a/scripts/deforum_helpers/ui_elements.py b/scripts/deforum_helpers/ui_elements.py index 9327fde46..548620c74 100644 --- a/scripts/deforum_helpers/ui_elements.py +++ b/scripts/deforum_helpers/ui_elements.py @@ -596,6 +596,8 @@ def create_keyframe_redistribution_info(): the generation can be performed much faster compared to a traditional low cadence setup. Resulting videos tend to be less jittery at high or no cadence, \ but may introduce 'depth smear' when combined with fast movement. + Optical Flow related settings may not behave as expected and are recommended to be turned off \ + when keyframe redistribution is used (see tab "Keyframes", sub-tab "Coherence").
  1. {bars_mark} Off: Key frames are not redistributed. Cadence settings are fully respected.
  2. {bars_mark} Parseq Only: Only frames with an entry in the Parseq table are diffused. \ From 755f566e1860f6dc48f8118fc965728d343bc73b Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 5 Aug 2024 20:47:23 +0200 Subject: [PATCH 118/132] Obsolete table suppression code removed. --- scripts/deforum_helpers/rendering/new_core.py | 6 ++--- .../rendering/util/log_utils.py | 27 ------------------- 2 files changed, 3 insertions(+), 30 deletions(-) diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index 9dc911eb5..97050b51a 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -22,7 +22,6 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro data.animation_mode.unload_raft_and_depth_model() -# @log_utils.with_suppressed_table_printing def run_render_animation(data: RenderData, key_frames: List[KeyFrame]): for key_frame in key_frames: if is_resume(data, key_frame): @@ -76,10 +75,11 @@ def emit_tweens(data, key_step): def _check_experimental_render_conditions(data): if data.has_parseq_keyframe_redistribution(): + msg = "Using Parseq keyframe redistribution with {method}. Results may be unexpected." if data.has_optical_flow_cadence(): - log_utils.warn("Using Parseq keyframe redistribution with optical flow cadence. Results may be unexpected.") + log_utils.warn(msg.format(method="optical flow cadence")) if data.has_optical_flow_redo(): - log_utils.warn("Using Parseq keyframe redistribution with optical flow redo. Results may be unexpected.") + log_utils.warn(msg.format(method="optical flow generation")) def _update_pseudo_cadence(data, value): diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 4016a1e82..d68a1b211 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,8 +1,6 @@ # noinspection PyUnresolvedReferences from modules.shared import opts -from ... import generate - COLOUR_RGB = '\x1b[38;2;%d;%d;%dm' RED = "\033[31m" ORANGE = "\033[38;5;208m" @@ -97,28 +95,3 @@ def debug(s: str): if is_verbose(): eye_catcher = "###" print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") - - -# noqa -def with_suppressed_table_printing(func): - def _suppress_table_printing(): - # The combined table that is normally printed to the command line is suppressed, - # because it's not compatible with variable keyframe cadence. - def do_nothing(*args): - pass - - original_print_combined_table = generate.print_combined_table - generate.print_combined_table = do_nothing # Monkey patch with do_nothing - return original_print_combined_table - - def _reactivate_table_printing(original_print_combined_table): - generate.print_combined_table = original_print_combined_table - - def wrapper(*args, **kwargs): - original = _suppress_table_printing() - try: - return func(*args, **kwargs) - finally: - _reactivate_table_printing(original) - - return wrapper From 840140d8e7065fce6b03e62046b6477ab98f1084 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 5 Aug 2024 21:18:58 +0200 Subject: [PATCH 119/132] Tqdm behaviour fixed. --- scripts/deforum_helpers/rendering/new_core.py | 10 ++++++---- scripts/deforum_helpers/rendering/util/log_utils.py | 5 +++++ 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index 97050b51a..e870d84ad 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -92,7 +92,9 @@ def _update_pseudo_cadence(data, value): def _tweens_with_progress(key_step): # only use tween progress bar when extra console output (aka "dev mode") is disabled. - return (tqdm(key_step.tweens, position=1, desc="Tweens progress", file=progress_print_out, - disable=cmd_opts.disable_console_progressbars, leave=False, colour='#FFA468') - if not log_utils.is_verbose() - else key_step.tweens) + if not log_utils.is_verbose(): + log_utils.clear_previous_line() + return tqdm(key_step.tweens, desc="Tweens progress", file=progress_print_out, + disable=cmd_opts.disable_console_progressbars, colour='#FFA468') + else: + return key_step.tweens diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index d68a1b211..e61c449f7 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -24,6 +24,10 @@ def is_verbose(): return opts.data.get("deforum_debug_mode_enabled", False) +def clear_previous_line(): + print("\033[F\033[K", end="") # "\033[" is the ANSI escape sequence, "F" is cursor up, "K" is clear line. + + def print_tween_frame_from_to_info(key_step, is_disabled=True): if not is_disabled: # replaced with prog bar, but value info print may be useful tween_values = key_step.tween_values @@ -36,6 +40,7 @@ def print_tween_frame_from_to_info(key_step, is_disabled=True): def print_animation_frame_info(i, max_frames): + print("") print(f"{CYAN}Animation frame: {RESET}{i}/{max_frames}") From 7041ac1142a7600607d6f809a9d589e2e616e9ba Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 5 Aug 2024 21:40:09 +0200 Subject: [PATCH 120/132] Constants for ANSI codes. --- .../rendering/util/log_utils.py | 49 +++++++++++-------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index e61c449f7..8bfe401e2 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,22 +1,30 @@ # noinspection PyUnresolvedReferences from modules.shared import opts -COLOUR_RGB = '\x1b[38;2;%d;%d;%dm' -RED = "\033[31m" -ORANGE = "\033[38;5;208m" -YELLOW = "\033[33m" -GREEN = "\033[32m" -CYAN = "\033[36m" -BLUE = "\033[34m" -INDIGO = "\033[38;5;66m" -VIOLET = "\033[38;5;130m" -BLACK = "\033[30m" -WHITE = "\033[37m" +ESC = "\033[" # ANSI escape character, same as "\x1b[" +TERM = "m" # ANSI terminator -BOLD = "\033[1m" -UNDERLINE = "\033[4m" +EIGHT_BIT = "38;5;" +TEXT = "38;2;" +BACKGROUND = "48;2;" -RESET = "\x1b[0m" +COLOUR_RGB = f"{ESC}{TEXT}%d;%d;%d{TERM}" +BG_COLOUR_RGB = f"{ESC}{BACKGROUND}%d;%d;%d{TERM}" +RESET_COLOR = f"{ESC}0{TERM}" + +RED = f"{ESC}31{TERM}" +ORANGE = f"{ESC}{EIGHT_BIT}208{TERM}" +YELLOW = f"{ESC}33{TERM}" +GREEN = f"{ESC}32{TERM}" +CYAN = f"{ESC}36{TERM}" +BLUE = f"{ESC}34{TERM}" +INDIGO = f"{ESC}{EIGHT_BIT}66{TERM}" +VIOLET = f"{ESC}{EIGHT_BIT}130{TERM}" +BLACK = f"{ESC}30{TERM}" +WHITE = f"{ESC}37{TERM}" + +BOLD = f"{ESC}1{TERM}" +UNDERLINE = f"{ESC}4{TERM}" def is_verbose(): @@ -25,7 +33,7 @@ def is_verbose(): def clear_previous_line(): - print("\033[F\033[K", end="") # "\033[" is the ANSI escape sequence, "F" is cursor up, "K" is clear line. + print(f"{ESC}F{ESC}K", end="") # "F" is cursor up, "K" is clear line. def print_tween_frame_from_to_info(key_step, is_disabled=True): @@ -36,12 +44,12 @@ def print_tween_frame_from_to_info(key_step, is_disabled=True): if end_i > 0: formatted_values = [f"{val:.2f}" for val in tween_values] count = end_i - start_i + 1 - print(f"{ORANGE}Creating in-between: {RESET}{count} frames ({start_i}-->{end_i}){formatted_values}") + print(f"{ORANGE}Creating in-between: {RESET_COLOR}{count} frames ({start_i}-->{end_i}){formatted_values}") def print_animation_frame_info(i, max_frames): print("") - print(f"{CYAN}Animation frame: {RESET}{i}/{max_frames}") + print(f"{CYAN}Animation frame: {RESET_COLOR}{i}/{max_frames}") def print_tween_frame_info(data, indexes, cadence_flow, tween, is_disabled=True): @@ -79,7 +87,7 @@ def print_key_step_debug_info_if_verbose(key_steps): def print_warning_generate_returned_no_image(): - print(f"{YELLOW}Warning: {RESET}Generate returned no image. Skipping to next iteration.") + print(f"{YELLOW}Warning: {RESET_COLOR}Generate returned no image. Skipping to next iteration.") def print_cuda_memory_state(cuda): @@ -93,10 +101,11 @@ def info(s: str): def warn(s: str): - print(f"{ORANGE}Warn: {RESET}{s}") + eye_catcher = "###" + print(f"{ORANGE}{BOLD}{eye_catcher} Warning: {RESET_COLOR}{s}") def debug(s: str): if is_verbose(): eye_catcher = "###" - print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET}{s}") + print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET_COLOR}{s}") From 93501bb7f6ab24d6acd8e1c186421320415dc494 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Mon, 5 Aug 2024 23:32:27 +0200 Subject: [PATCH 121/132] Cleanup. --- .../rendering/data/frame/key_frame.py | 10 ++++++---- .../deforum_helpers/rendering/data/schedule.py | 16 +++++++++++----- .../deforum_helpers/rendering/img_2_img_tubes.py | 1 - 3 files changed, 17 insertions(+), 10 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index aedc88b78..ad4b6f5e2 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -22,6 +22,7 @@ from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.call.video_and_audio import call_render_preview +from ....animation_key_frames import DeformAnimKeys from ....colors import maintain_colors from ....hybrid_video import image_transform_ransac, image_transform_optical_flow from ....save_images import save_image @@ -52,8 +53,9 @@ def has_strength(self): return self.strength > 0 @staticmethod - def create(deform_keys, i): - keys = deform_keys + def create(data: RenderData): + i = data.indexes.frame.i + keys: DeformAnimKeys = data.animation_keys.deform_keys return KeyFrameData( keys.noise_schedule_series[i], keys.strength_schedule_series[i], @@ -284,8 +286,8 @@ def do_diffusion_redo(self): @staticmethod def create(data: RenderData): - step_data = KeyFrameData.create(data.animation_keys.deform_keys, data.indexes.frame.i) - schedule = Schedule.create(data, data.indexes.frame.i, data.args.anim_args, data.args.args) + step_data = KeyFrameData.create(data) + schedule = Schedule.create(data) return KeyFrame(0, step_data, data, schedule, None, None, "", 0, list(), list()) @staticmethod diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index e8997507c..d47e064f0 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -1,6 +1,10 @@ from dataclasses import dataclass from typing import Optional, Any +from .render_data import RenderData +from ...animation_key_frames import DeformAnimKeys +from ...args import DeforumAnimArgs, DeforumArgs + @dataclass(init=True, frozen=True, repr=False, eq=False) class Schedule: @@ -9,16 +13,17 @@ class Schedule: clipskip: int noise_multiplier: float eta_ddim: float - eta_ancestral: float # TODO unify ddim- and a-eta to use one or the other, depending on sampler + eta_ancestral: float mask: Optional[Any] noise_mask: Optional[Any] @staticmethod - def create(init, i, anim_args, args): - # TODO typecheck keys as DeformAnimKeys or provide key collection or something + def create(data: RenderData): """Create a new Schedule instance based on the provided parameters.""" - is_use_mask_without_noise = init.is_use_mask and not init.args.anim_args.use_noise_mask - keys = init.animation_keys.deform_keys + i = data.indexes.frame.i + args: DeforumArgs = data.args.args + anim_args: DeforumAnimArgs = data.args.anim_args + keys: DeformAnimKeys = data.animation_keys.deform_keys steps = Schedule.schedule_steps(keys, i, anim_args) sampler_name = Schedule.schedule_sampler(keys, i, anim_args) clipskip = Schedule.schedule_clipskip(keys, i, anim_args) @@ -26,6 +31,7 @@ def create(init, i, anim_args, args): eta_ddim = Schedule.schedule_ddim_eta(keys, i, anim_args) eta_ancestral = Schedule.schedule_ancestral_eta(keys, i, anim_args) mask = Schedule.schedule_mask(keys, i, args) + is_use_mask_without_noise = data.is_use_mask and not data.args.anim_args.use_noise_mask noise_mask = mask if is_use_mask_without_noise else Schedule.schedule_noise_mask(keys, i, anim_args) return Schedule(steps, sampler_name, clipskip, noise_multiplier, eta_ddim, eta_ancestral, mask, noise_mask) diff --git a/scripts/deforum_helpers/rendering/img_2_img_tubes.py b/scripts/deforum_helpers/rendering/img_2_img_tubes.py index 29f12a4da..efa24fd43 100644 --- a/scripts/deforum_helpers/rendering/img_2_img_tubes.py +++ b/scripts/deforum_helpers/rendering/img_2_img_tubes.py @@ -73,7 +73,6 @@ def conditional_extra_color_match_tube(data: RenderData) -> PilImageTube: # color matching on first frame is after generation, color match was collected earlier, # so we do an extra generation to avoid the corruption introduced by the color match of first output return tube(lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), - lambda img: image_utils.numpy_to_pil(image_utils.pil_to_numpy(img)), # TODO? remove lambda img: maintain_colors(img, data.images.color_match, data.args.anim_args.color_coherence), lambda img: image_utils.numpy_to_pil(img), is_do_process=lambda: data.indexes.is_first_frame() and data.is_initialize_color_match( From dbfb5164e3b28c61a3eba58871291ed4e7fb071a Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 00:13:53 +0200 Subject: [PATCH 122/132] Removed opts reference from render data and moved all global opts access to opt_utils. --- .../rendering/data/anim/animation_mode.py | 15 +++---- .../rendering/data/frame/key_frame.py | 2 +- .../rendering/data/render_data.py | 12 ++---- .../rendering/data/subtitle/srt.py | 10 ++--- .../deforum_helpers/rendering/data/turbo.py | 6 +-- scripts/deforum_helpers/rendering/new_core.py | 4 +- .../rendering/util/log_utils.py | 10 +---- .../rendering/util/memory_utils.py | 4 -- .../rendering/util/opt_utils.py | 39 ++++++++++++++----- 9 files changed, 52 insertions(+), 50 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 2bf359a65..33a40a7fd 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -3,7 +3,7 @@ from pathlib import Path from typing import Any -from ...util.memory_utils import keep_3d_models_in_vram +from ...util import opt_utils from ....RAFT import RAFT from ....hybrid_video import hybrid_generation @@ -66,9 +66,9 @@ def load_raft_if_active(anim_args, args): return RAFT() if is_load_raft else None @staticmethod - def load_depth_model_if_active(args, anim_args, opts): + def load_depth_model_if_active(args, anim_args): return AnimationMode._is_load_depth_model_for_3d(args, anim_args) \ - if opts.data.get("deforum_keep_3d_models_in_vram", False) else None + if opt_utils.keep_3d_models_in_vram() else None @staticmethod def initial_hybrid_files(sa) -> list[Path]: @@ -81,14 +81,15 @@ def initial_hybrid_files(sa) -> list[Path]: @staticmethod def from_args(step_args): - sa = step_args + sa = step_args # RenderInitArgs # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') + previous_flow = None # FIXME? return AnimationMode( AnimationMode._has_video_input(sa.anim_args), AnimationMode.initial_hybrid_files(sa), hybrid_input_files, - None, - keep_3d_models_in_vram(sa), - AnimationMode.load_depth_model_if_active(sa.args, sa.anim_args, sa.opts), + previous_flow, + opt_utils.keep_3d_models_in_vram(), + AnimationMode.load_depth_model_if_active(sa.args, sa.anim_args), AnimationMode.load_raft_if_active(sa.anim_args, sa.args)) diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index ad4b6f5e2..e4f6a168d 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -103,7 +103,7 @@ def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, i def maybe_write_frame_subtitle(self): data = self.render_data - if data.turbo.is_first_step_with_subtitles(data): + if data.turbo.is_first_step_with_subtitles(): params_string = opt_utils.generation_info_for_subtitles(data) self.subtitle_params_to_print = params_string self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_string) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index d2ab74835..f5f7424c1 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -7,8 +7,6 @@ import numpy as np import pandas as pd from PIL import Image -# noinspection PyUnresolvedReferences -from modules.shared import opts from .anim import AnimationKeys, AnimationMode from .images import Images @@ -37,7 +35,6 @@ class RenderInitArgs: video_args: Any = None controlnet_args: Any = None loop_args: LoopArgs = None - opts: Any = None # opts were passed from modules.shared. # TODO import and access directly? root: RootArgs = None @@ -61,12 +58,12 @@ class RenderData: @staticmethod def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) -> 'RenderData': - ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, opts, root) + ri_args = RenderInitArgs(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) output_directory = args.outdir is_use_mask = args.use_mask parseq_adapter = RenderData.create_parseq_adapter(ri_args) - srt = Srt.create_if_active(opts.data, output_directory, root.timestring, video_args.fps) + srt = Srt.create_if_active(output_directory, root.timestring, video_args.fps) animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) animation_mode = AnimationMode.from_args(ri_args) prompt_series = RenderData.select_prompts(parseq_adapter, anim_args, animation_keys, root) @@ -165,9 +162,6 @@ def is_resuming_from_timestring(self): def has_video_input(self): return self.animation_mode.has_video_input - def has_img2img_fix_steps(self): - return 'img2img_fix_steps' in self.args.opts.data and self.args.opts.data["img2img_fix_steps"] - def cadence(self) -> int: return int(self.args.anim_args.diffusion_cadence) @@ -306,7 +300,7 @@ def prepare_generation(self, data, step, i): # print(f"cadence_flow_factor: {step.step_data.cadence_flow_factor}") self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) - opt_utils.setup(self, step.schedule) + opt_utils.setup(step.schedule) memory_utils.handle_vram_if_depth_is_predicted(data) @staticmethod diff --git a/scripts/deforum_helpers/rendering/data/subtitle/srt.py b/scripts/deforum_helpers/rendering/data/subtitle/srt.py index cb90c0e1e..1e3594faf 100644 --- a/scripts/deforum_helpers/rendering/data/subtitle/srt.py +++ b/scripts/deforum_helpers/rendering/data/subtitle/srt.py @@ -2,6 +2,8 @@ from dataclasses import dataclass from decimal import Decimal + +from ...util import opt_utils from ....subtitle_handler import init_srt_file @@ -11,12 +13,8 @@ class Srt: frame_duration: Decimal @staticmethod - def is_subtitle_generation_active(opts_data): - return opts_data.get("deforum_save_gen_info_as_srt", False) - - @staticmethod - def create_if_active(opts_data, out_dir: str, timestring: str, fps: float) -> 'Srt | None': - if not Srt.is_subtitle_generation_active(opts_data): + def create_if_active(out_dir: str, timestring: str, fps: float) -> 'Srt | None': + if not opt_utils.is_subtitle_generation_active(): return None else: # create .srt file and set timeframe mechanism using FPS diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 767a46f66..77dbc92fe 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -2,7 +2,7 @@ from cv2.typing import MatLike -from .subtitle import Srt +from ..util import opt_utils from ..util.call.anim import call_anim_frame_warp from ..util.call.hybrid import (call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev) @@ -168,8 +168,8 @@ def is_advance_next(self, i: int) -> bool: def is_first_step(self) -> bool: return self.steps == 1 - def is_first_step_with_subtitles(self, render_data) -> bool: - return self.is_first_step() and Srt.is_subtitle_generation_active(render_data.args.opts.data) + def is_first_step_with_subtitles(self) -> bool: + return self.is_first_step() and opt_utils.is_subtitle_generation_active() def is_emit_in_between_frames(self) -> bool: return self.steps > 1 diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index e870d84ad..9b0127ed6 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -9,7 +9,7 @@ from . import img_2_img_tubes from .data.frame import KeyFrameDistribution, KeyFrame from .data.render_data import RenderData -from .util import filename_utils, image_utils, log_utils, memory_utils, web_ui_utils +from .util import filename_utils, image_utils, log_utils, opt_utils, memory_utils, web_ui_utils def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): @@ -92,7 +92,7 @@ def _update_pseudo_cadence(data, value): def _tweens_with_progress(key_step): # only use tween progress bar when extra console output (aka "dev mode") is disabled. - if not log_utils.is_verbose(): + if not opt_utils.is_verbose(): log_utils.clear_previous_line() return tqdm(key_step.tweens, desc="Tweens progress", file=progress_print_out, disable=cmd_opts.disable_console_progressbars, colour='#FFA468') diff --git a/scripts/deforum_helpers/rendering/util/log_utils.py b/scripts/deforum_helpers/rendering/util/log_utils.py index 8bfe401e2..a3c488919 100644 --- a/scripts/deforum_helpers/rendering/util/log_utils.py +++ b/scripts/deforum_helpers/rendering/util/log_utils.py @@ -1,5 +1,4 @@ -# noinspection PyUnresolvedReferences -from modules.shared import opts +from . import opt_utils ESC = "\033[" # ANSI escape character, same as "\x1b[" TERM = "m" # ANSI terminator @@ -27,11 +26,6 @@ UNDERLINE = f"{ESC}4{TERM}" -def is_verbose(): - """Checks if extra console output is enabled in deforum settings.""" - return opts.data.get("deforum_debug_mode_enabled", False) - - def clear_previous_line(): print(f"{ESC}F{ESC}K", end="") # "F" is cursor up, "K" is clear line. @@ -106,6 +100,6 @@ def warn(s: str): def debug(s: str): - if is_verbose(): + if opt_utils.is_verbose(): eye_catcher = "###" print(f"{YELLOW}{BOLD}{eye_catcher} Debug: {RESET_COLOR}{s}") diff --git a/scripts/deforum_helpers/rendering/util/memory_utils.py b/scripts/deforum_helpers/rendering/util/memory_utils.py index de2170185..5daf09535 100644 --- a/scripts/deforum_helpers/rendering/util/memory_utils.py +++ b/scripts/deforum_helpers/rendering/util/memory_utils.py @@ -45,7 +45,3 @@ def handle_vram_after_depth_map_generation(data): def select_depth_device(root): return 'cpu' if is_low_or_med_vram() else root.device - - -def keep_3d_models_in_vram(step_args): - return step_args.opts.data.get("deforum_keep_3d_models_in_vram", False) diff --git a/scripts/deforum_helpers/rendering/util/opt_utils.py b/scripts/deforum_helpers/rendering/util/opt_utils.py index a93175eb2..18f70658f 100644 --- a/scripts/deforum_helpers/rendering/util/opt_utils.py +++ b/scripts/deforum_helpers/rendering/util/opt_utils.py @@ -1,20 +1,39 @@ from .utils import put_if_present +# noinspection PyUnresolvedReferences +from modules.shared import opts -def setup(init, schedule): - data = init.args.opts.data - if init.has_img2img_fix_steps(): + +def is_subtitle_generation_active(): + return opts.data.get("deforum_save_gen_info_as_srt", False) + + +def is_verbose(): + """Checks if extra console output is enabled in deforum settings.""" + return opts.data.get("deforum_debug_mode_enabled", False) + + +def has_img2img_fix_steps(): + return 'img2img_fix_steps' in opts.data and opts.data["img2img_fix_steps"] + + +def keep_3d_models_in_vram(): + return opts.data.get("deforum_keep_3d_models_in_vram", False) + + +def setup(schedule): + if has_img2img_fix_steps(): # disable "with img2img do exactly x steps" from general setting, as it *ruins* deforum animations - data["img2img_fix_steps"] = False - put_if_present(data, "CLIP_stop_at_last_layers", schedule.clipskip) - put_if_present(data, "initial_noise_multiplier", schedule.noise_multiplier) - put_if_present(data, "eta_ddim", schedule.eta_ddim) - put_if_present(data, "eta_ancestral", schedule.eta_ancestral) + opts.data["img2img_fix_steps"] = False + put_if_present(opts.data, "CLIP_stop_at_last_layers", schedule.clipskip) + put_if_present(opts.data, "initial_noise_multiplier", schedule.noise_multiplier) + put_if_present(opts.data, "eta_ddim", schedule.eta_ddim) + put_if_present(opts.data, "eta_ancestral", schedule.eta_ancestral) def generation_info_for_subtitles(render_data): - return render_data.args.opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) + return opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) def is_generate_subtitles(render_data): - return render_data.args.opts.data.get("deforum_save_gen_info_as_srt") + return opts.data.get("deforum_save_gen_info_as_srt") From 5ab4e860d5a2509f44666a44c00d88ae9225293d Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 01:17:30 +0200 Subject: [PATCH 123/132] Flow factor schedule fix. --- .../rendering/data/render_data.py | 48 ++++++++----------- .../deforum_helpers/rendering/data/turbo.py | 17 +++---- scripts/deforum_helpers/rendering/new_core.py | 2 +- 3 files changed, 31 insertions(+), 36 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index f5f7424c1..a8269dae5 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -41,7 +41,7 @@ class RenderInitArgs: @dataclass(init=True, frozen=False, repr=False, eq=False) class RenderData: """The purpose of this class is to group and control all data used in render_animation""" - images: Images | None # TODO rename to reference_images? + images: Images | None turbo: Turbo | None indexes: Indexes | None mask: Mask | None @@ -88,28 +88,30 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, RenderData.maybe_resume_from_timestring(anim_args, root) return instance - def is_3d(self): + # The following methods are meant to provide easy and centralized access to the most important + # arguments and settings relevant for rendering. All bools use naming with 'is_' or 'has_'. + def is_3d(self) -> bool: return self.args.anim_args.animation_mode == '3D' - def is_3d_or_2d(self): + def is_3d_or_2d(self) -> bool: return self.args.anim_args.animation_mode in ['2D', '3D'] - def has_parseq_keyframe_redistribution(self): + def has_parseq_keyframe_redistribution(self) -> bool: return self.args.parseq_args.parseq_key_frame_redistribution != "Off" - def has_optical_flow_cadence(self): + def has_optical_flow_cadence(self) -> bool: return self.args.anim_args.optical_flow_cadence != 'None' - def has_optical_flow_redo(self): + def has_optical_flow_redo(self) -> bool: return self.args.anim_args.optical_flow_redo_generation != 'None' - def is_3d_or_2d_with_optical_flow(self): + def is_3d_or_2d_with_optical_flow(self) -> bool: return self.is_3d_or_2d() and self.has_optical_flow_cadence() - def is_3d_with_med_or_low_vram(self): + def is_3d_with_med_or_low_vram(self) -> bool: return self.is_3d() and memory_utils.is_low_or_med_vram() - def has_keyframe_redistribution(self): + def has_keyframe_redistribution(self) -> bool: return self.args.parseq_args def width(self) -> int: @@ -121,7 +123,7 @@ def height(self) -> int: def dimensions(self) -> tuple[int, int]: return self.width(), self.height() - # TODO group hybrid stuff elsewhere + # hybrid stuff def is_hybrid_composite(self) -> bool: return self.args.anim_args.hybrid_composite != 'None' @@ -139,8 +141,9 @@ def is_hybrid_composite_before_motion(self) -> bool: def is_hybrid_composite_after_generation(self) -> bool: return self.args.anim_args.hybrid_composite == 'After Generation' + # end hybrid stuff - def is_initialize_color_match(self, color_match_sample): + def is_initialize_color_match(self, color_match_sample) -> bool: """Determines whether to initialize color matching based on the given conditions.""" has_video_input = self.args.anim_args.color_coherence == 'Video Input' and self.is_hybrid_available() has_image_color_coherence = self.args.anim_args.color_coherence == 'Image' @@ -150,16 +153,16 @@ def is_initialize_color_match(self, color_match_sample): has_sample_and_match = has_any_color_sample and has_coherent_non_legacy_color_match return has_video_input or has_image_color_coherence or has_sample_and_match - def has_color_coherence(self): + def has_color_coherence(self) -> bool: return self.args.anim_args.color_coherence != 'None' - def has_non_video_or_image_color_coherence(self): + def has_non_video_or_image_color_coherence(self) -> bool: return self.args.anim_args.color_coherence not in ['Image', 'Video Input'] - def is_resuming_from_timestring(self): + def is_resuming_from_timestring(self) -> bool: return self.args.anim_args.resume_from_timestring - def has_video_input(self): + def has_video_input(self) -> bool: return self.animation_mode.has_video_input def cadence(self) -> int: @@ -177,7 +180,7 @@ def _has_init_image_or_box(self) -> bool: def is_using_init_image_or_box(self) -> bool: return self.args.args.use_init and self._has_init_image_or_box() - def is_not_in_motion_preview_mode(self): + def is_not_in_motion_preview_mode(self) -> bool: return not self.args.args.motion_preview_mode def color_coherence_mode(self): @@ -189,14 +192,14 @@ def diffusion_redo(self): def diffusion_redo_as_int(self): return int(self.diffusion_redo()) - def has_positive_diffusion_redo(self): + def has_positive_diffusion_redo(self) -> bool: return self.diffusion_redo_as_int() > 0 def optical_flow_redo_generation_if_not_in_preview_mode(self): is_not_preview = self.is_not_in_motion_preview_mode() return self.args.anim_args.optical_flow_redo_generation if is_not_preview else 'None' - def is_do_color_match_conversion(self, step): + def is_do_color_match_conversion(self, step) -> bool: is_legacy_cm = self.args.anim_args.legacy_colormatch is_use_init = self.args.args.use_init is_not_legacy_with_use_init = not is_legacy_cm and not is_use_init @@ -290,15 +293,6 @@ def prepare_generation(self, data, step, i): self.prompt_for_current_step(i) self.update_video_data_for_current_frame(i, step) self.update_mask_image(step, data.mask) - - if len(step.tweens) > 0: - # FIXME it's not yet working as it is supposed to - # data.args.anim_args.cadence_flow_factor_schedule = f"0: ({len(step.tweens) + 1})" - data.args.anim_args.cadence_flow_factor_schedule = f"0: (1)" - # print(f"cadence_flow_factor_schedule: {data.args.anim_args.cadence_flow_factor_schedule}") - # step.step_data.cadence_flow_factor = 1.0 / len(step.tweens) - # print(f"cadence_flow_factor: {step.step_data.cadence_flow_factor}") - self.animation_keys = AnimationKeys.from_args(self.args, self.parseq_adapter, self.seed) opt_utils.setup(step.schedule) memory_utils.handle_vram_if_depth_is_predicted(data) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 77dbc92fe..17d0b2dda 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -2,7 +2,7 @@ from cv2.typing import MatLike -from ..util import opt_utils +from ..util import opt_utils, log_utils from ..util.call.anim import call_anim_frame_warp from ..util.call.hybrid import (call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev) @@ -57,12 +57,12 @@ def advance_optical_flow(self, tween_step, flow_factor: int = 1): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) def advance_optical_tween_flow(self, indexes, last_frame, flow): - ff = last_frame.step_data.flow_factor() + flow_factor = last_frame.step_data.flow_factor() i = indexes.tween.i if self.is_advance_prev(i): - self.prev.image = image_transform_optical_flow(self.prev.image, flow, ff) + self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): - self.next.image = image_transform_optical_flow(self.next.image, flow, ff) + self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_images, last_frame): last_i = indexes.tween.i - 1 @@ -73,13 +73,14 @@ def advance_hybrid_motion_optical_tween_flow(self, data, indexes, reference_imag data.animation_mode.prev_flow = flow def advance_cadence_flow(self, data, tween_frame): - ff = data.args.anim_args.cadence_flow_factor_schedule + ff_string = data.args.anim_args.cadence_flow_factor_schedule + flow_factor = float(ff_string.split(": ")[1][1:-1]) i = tween_frame.i() - inc = tween_frame.cadence_flow_inc # FIXME + flow = tween_frame.cadence_flow_inc if self.is_advance_prev(i): - self.prev.image = image_transform_optical_flow(self.prev.image, inc, ff) + self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): - self.next.image = image_transform_optical_flow(self.next.image, inc, ff) + self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) # TODO? move to RenderData def advance_ransac_transform(self, data, matrix): diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index 9b0127ed6..e7d7a0fab 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -87,7 +87,7 @@ def _update_pseudo_cadence(data, value): data.parseq_adapter.cadence = value data.parseq_adapter.a1111_cadence = value data.args.anim_args.diffusion_cadence = value - data.args.anim_args.cadence_flow_factor_schedule = value + data.args.anim_args.cadence_flow_factor_schedule = f"0: ({value})" def _tweens_with_progress(key_step): From 59b92b245e50bde125ebc191031f39f7477dcc61 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 01:50:52 +0200 Subject: [PATCH 124/132] More cleanup. --- .../rendering/data/anim/animation_mode.py | 2 +- .../rendering/data/frame/key_frame.py | 29 +++++++++---------- .../rendering/data/frame/tween_frame.py | 7 ++--- .../deforum_helpers/rendering/data/indexes.py | 2 +- scripts/deforum_helpers/rendering/new_core.py | 3 +- .../rendering/util/opt_utils.py | 4 +-- 6 files changed, 22 insertions(+), 25 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py index 33a40a7fd..c0206bf7b 100644 --- a/scripts/deforum_helpers/rendering/data/anim/animation_mode.py +++ b/scripts/deforum_helpers/rendering/data/anim/animation_mode.py @@ -84,7 +84,7 @@ def from_args(step_args): sa = step_args # RenderInitArgs # path required by hybrid functions, even if hybrid_comp_save_extra_frames is False hybrid_input_files: Any = os.path.join(sa.args.outdir, 'hybridframes') - previous_flow = None # FIXME? + previous_flow = None return AnimationMode( AnimationMode._has_video_input(sa.anim_args), AnimationMode.initial_hybrid_files(sa), diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index e4f6a168d..1c6bba12a 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -39,7 +39,7 @@ class KeyFrameData: sigma: Any = None amount: Any = None threshold: Any = None - cadence_flow_factor: Any = None # FIXME re-assignable + cadence_flow_factor: Any = None redo_flow_factor: Any = None hybrid_comp_schedules: Any = None @@ -87,7 +87,7 @@ class KeyFrame: step_data: KeyFrameData render_data: RenderData schedule: Schedule - depth: Any # TODO try to init early, then freeze class + depth: Any # assigned during generation subtitle_params_to_print: Any subtitle_params_string: str last_preview_frame: int @@ -104,14 +104,14 @@ def is_optical_flow_redo_before_generation(self, optical_flow_redo_generation, i def maybe_write_frame_subtitle(self): data = self.render_data if data.turbo.is_first_step_with_subtitles(): - params_string = opt_utils.generation_info_for_subtitles(data) + params_string = opt_utils.generation_info_for_subtitles() self.subtitle_params_to_print = params_string self.subtitle_params_string = call_format_animation_params(data, data.indexes.frame.i, params_string) call_write_frame_subtitle(data, data.indexes.frame.i, params_string) def apply_frame_warp_transform(self, data: RenderData, image): is_not_last_frame = self.i < data.args.anim_args.max_frames - if is_not_last_frame: # TODO? why not + if is_not_last_frame: previous, self.depth = call_anim_frame_warp(data, self.i, image, None) return previous @@ -158,20 +158,19 @@ def create_color_match_for_video(self): return cv2.cvtColor(data.images.color_match, cv2.COLOR_RGB2BGR) return None + def _generate_and_update_noise(self, data, image, contrasted_noise_tube): + noised_image = contrasted_noise_tube(data, self)(image) + data.update_sample_and_args_for_current_progression_step(self, noised_image) + return image # return original as passed. + def transform_and_update_noised_sample(self, frame_tube, contrasted_noise_tube): data = self.render_data if data.images.has_previous(): # skipping 1st iteration transformed_image = frame_tube(data, self)(data.images.previous) - # TODO separate - if transformed_image is None: # FIXME? shouldn't really happen - log_utils.debug(f"transformed_image {transformed_image}") - noised_image = contrasted_noise_tube(data, self)(data.images.previous) - data.update_sample_and_args_for_current_progression_step(self, noised_image) - return data.images.previous - else: - noised_image = contrasted_noise_tube(data, self)(transformed_image) - data.update_sample_and_args_for_current_progression_step(self, noised_image) - return transformed_image + if transformed_image is None: + log_utils.warn("Image transformation failed, using fallback.") + transformed_image = data.images.previous + return self._generate_and_update_noise(data, transformed_image, contrasted_noise_tube) return None def prepare_generation(self, frame_tube, contrasted_noise_tube): @@ -334,7 +333,7 @@ def apply_hybrid_motion_optical_flow(data: RenderData, key_frame, image): else call_get_flow_for_hybrid_motion(data, last_i) transformed = image_transform_optical_flow( reference_images.previous, flow, key_frame.step_data.flow_factor()) - data.animation_mode.prev_flow = flow # side effect + data.animation_mode.prev_flow = flow return transformed return image diff --git a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py index 3814f6dd0..727154807 100644 --- a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py @@ -40,11 +40,10 @@ def emit_frame(self, last_frame, grayscale_tube, overlay_mask_tube): self.process(last_frame, data) new_image = self.generate_tween_image(data, grayscale_tube, overlay_mask_tube) - # TODO pass step and depth instead of data and tween_step.indexes new_image = image_utils.save_and_return_frame(data, self, self.i(), new_image) # updating reference images to calculate hybrid motions in next iteration - data.images.previous = new_image # FIXME + data.images.previous = new_image def generate_tween_image(self, data, grayscale_tube, overlay_mask_tube): warped = data.turbo.do_optical_flow_cadence_after_animation_warping(data, self.indexes, self) @@ -65,8 +64,8 @@ def handle_synchronous_status_concerns(self, data): web_ui_utils.update_progress_during_cadence(data, self.indexes) def write_tween_frame_subtitle_if_active(self, data: RenderData): - if opt_utils.is_generate_subtitles(data): - params_to_print = opt_utils.generation_info_for_subtitles(data) + if opt_utils.is_generate_subtitles(): + params_to_print = opt_utils.generation_info_for_subtitles() params_string = call_format_animation_params(data, self.indexes.tween.i, params_to_print) is_cadence = self.value < 1.0 call_write_frame_subtitle(data, self.indexes.tween.i, params_string, is_cadence) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index 26b1e69df..1369bbcfb 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -19,7 +19,7 @@ class Indexes: def create(init, turbo): frame_start = turbo.find_start(init) tween_start = 0 - return Indexes(IndexWithStart(frame_start, 0), IndexWithStart(tween_start, 0)) + return Indexes(IndexWithStart(frame_start), IndexWithStart(tween_start)) @staticmethod def create_from_last(last_indexes, i: int): diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/new_core.py index e7d7a0fab..e57fa63aa 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/new_core.py @@ -96,5 +96,4 @@ def _tweens_with_progress(key_step): log_utils.clear_previous_line() return tqdm(key_step.tweens, desc="Tweens progress", file=progress_print_out, disable=cmd_opts.disable_console_progressbars, colour='#FFA468') - else: - return key_step.tweens + return key_step.tweens diff --git a/scripts/deforum_helpers/rendering/util/opt_utils.py b/scripts/deforum_helpers/rendering/util/opt_utils.py index 18f70658f..995a0ba1f 100644 --- a/scripts/deforum_helpers/rendering/util/opt_utils.py +++ b/scripts/deforum_helpers/rendering/util/opt_utils.py @@ -31,9 +31,9 @@ def setup(schedule): put_if_present(opts.data, "eta_ancestral", schedule.eta_ancestral) -def generation_info_for_subtitles(render_data): +def generation_info_for_subtitles(): return opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed']) -def is_generate_subtitles(render_data): +def is_generate_subtitles(): return opts.data.get("deforum_save_gen_info_as_srt") From 1028aba768a7c5a700ef200b04bb5423315c11d3 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 02:26:57 +0200 Subject: [PATCH 125/132] Moved some depth related code to a new utility module and solved some more TODOs. Key frame data separated. --- .../rendering/data/__init__.py | 1 + .../rendering/data/frame/__init__.py | 3 +- .../rendering/data/frame/key_frame.py | 79 ++----------------- .../rendering/data/frame/key_frame_data.py | 55 +++++++++++++ .../rendering/util/depth_utils.py | 14 ++++ 5 files changed, 79 insertions(+), 73 deletions(-) create mode 100644 scripts/deforum_helpers/rendering/data/frame/key_frame_data.py create mode 100644 scripts/deforum_helpers/rendering/util/depth_utils.py diff --git a/scripts/deforum_helpers/rendering/data/__init__.py b/scripts/deforum_helpers/rendering/data/__init__.py index 82ae199ac..77adfb648 100644 --- a/scripts/deforum_helpers/rendering/data/__init__.py +++ b/scripts/deforum_helpers/rendering/data/__init__.py @@ -2,4 +2,5 @@ from .indexes import Indexes from .mask import Mask from .schedule import Schedule +from .render_data import RenderData from .turbo import Turbo diff --git a/scripts/deforum_helpers/rendering/data/frame/__init__.py b/scripts/deforum_helpers/rendering/data/frame/__init__.py index 38d6f4344..f2ec90d1b 100644 --- a/scripts/deforum_helpers/rendering/data/frame/__init__.py +++ b/scripts/deforum_helpers/rendering/data/frame/__init__.py @@ -1,3 +1,4 @@ from .key_frame_distribution import KeyFrameDistribution -from .key_frame import KeyFrameData, KeyFrame +from .key_frame_data import KeyFrameData +from .key_frame import KeyFrame from .tween_frame import Tween diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index 1c6bba12a..4b666008f 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -7,12 +7,11 @@ import numpy as np from PIL import Image -from . import KeyFrameDistribution +from . import KeyFrameData, KeyFrameDistribution from .tween_frame import Tween -from ..render_data import RenderData -from ..schedule import Schedule +from .. import RenderData, Schedule from ... import img_2_img_tubes -from ...util import filename_utils, log_utils, memory_utils, opt_utils, utils +from ...util import depth_utils, filename_utils, log_utils, opt_utils, utils from ...util.call.anim import call_anim_frame_warp from ...util.call.gen import call_generate from ...util.call.hybrid import ( @@ -22,64 +21,12 @@ from ...util.call.mask import call_compose_mask_with_check, call_unsharp_mask from ...util.call.subtitle import call_format_animation_params, call_write_frame_subtitle from ...util.call.video_and_audio import call_render_preview -from ....animation_key_frames import DeformAnimKeys from ....colors import maintain_colors from ....hybrid_video import image_transform_ransac, image_transform_optical_flow from ....save_images import save_image from ....seed import next_seed -@dataclass(init=True, frozen=True, repr=False, eq=False) -class KeyFrameData: - noise: Any = None - strength: Any = None - scale: Any = None - contrast: Any = None - kernel: int = 0 - sigma: Any = None - amount: Any = None - threshold: Any = None - cadence_flow_factor: Any = None - redo_flow_factor: Any = None - hybrid_comp_schedules: Any = None - - def kernel_size(self) -> tuple[int, int]: - return self.kernel, self.kernel - - def flow_factor(self): - return self.hybrid_comp_schedules['flow_factor'] - - def has_strength(self): - return self.strength > 0 - - @staticmethod - def create(data: RenderData): - i = data.indexes.frame.i - keys: DeformAnimKeys = data.animation_keys.deform_keys - return KeyFrameData( - keys.noise_schedule_series[i], - keys.strength_schedule_series[i], - keys.cfg_scale_schedule_series[i], - keys.contrast_schedule_series[i], - int(keys.kernel_schedule_series[i]), - keys.sigma_schedule_series[i], - keys.amount_schedule_series[i], - keys.threshold_schedule_series[i], - keys.cadence_flow_factor_schedule_series[i], - keys.redo_flow_factor_schedule_series[i], - KeyFrameData._hybrid_comp_args(keys, i)) - - @staticmethod - def _hybrid_comp_args(keys, i): - return { - "alpha": keys.hybrid_comp_alpha_schedule_series[i], - "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[i], - "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[i], - "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), - "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), - "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} - - @dataclass(init=True, frozen=False, repr=False, eq=False) class KeyFrame: """Key steps are the steps for frames that actually get diffused (as opposed to tween frame steps).""" @@ -226,10 +173,10 @@ def _progress_save_and_get_next_index(self, image): # In many cases, the original images may look more detailed or 'better' than the processed ones, # but we only save the frames that were processed tough the flows to keep the output consistent. # However, it may be preferable to use them for the 1st and for the last frame, or as thumbnails. - # TODO perhaps save original frames in a different sub dir? + # TODO? add option to save original frames in a different sub dir. save_image(image, 'PIL', filename, data.args.args, data.args.video_args, data.args.root) - self.depth = self.generate_and_save_depth_map_if_active(opencv_image) + self.depth = depth_utils.generate_and_save_depth_map_if_active(data, opencv_image) if data.turbo.has_steps(): return data.indexes.frame.i + data.turbo.progress_step(data.indexes, opencv_image) return data.indexes.frame.i + 1 # normal (i.e. 'non-turbo') step always increments by 1. @@ -240,24 +187,12 @@ def next_seed(self): def update_render_preview(self): self.last_preview_frame = call_render_preview(self.render_data, self.last_preview_frame) - def generate_and_save_depth_map_if_active(self, opencv_image): - data = self.render_data - # TODO move all depth related stuff to new class. - if data.args.anim_args.save_depth_maps: - memory_utils.handle_vram_before_depth_map_generation(data) - depth = data.depth_model.predict(opencv_image, data.args.anim_args.midas_weight, - data.args.root.half_precision) - depth_filename = filename_utils.depth_frame(data, data.indexes) - data.depth_model.save(os.path.join(data.output_directory, depth_filename), depth) - memory_utils.handle_vram_after_depth_map_generation(data) - return depth - def do_optical_flow_redo_before_generation(self): data = self.render_data redo = data.args.anim_args.optical_flow_redo_generation stored_seed = data.args.args.seed # keep original to reset it after executing the optical flow - data.args.args.seed = utils.generate_random_seed() # set a new random seed - log_utils.print_optical_flow_info(data, redo) # TODO output temp seed? + data.args.args.seed = utils.generate_random_seed() # create and set a new random seed + log_utils.print_optical_flow_info(data, redo) sample_image = call_generate(data, self) optical_tube = img_2_img_tubes.optical_flow_redo_tube(data, self, redo) diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame_data.py b/scripts/deforum_helpers/rendering/data/frame/key_frame_data.py new file mode 100644 index 000000000..a5eba3ec7 --- /dev/null +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame_data.py @@ -0,0 +1,55 @@ +from dataclasses import dataclass +from typing import Any + +from ....animation_key_frames import DeformAnimKeys + + +@dataclass(init=True, frozen=True, repr=False, eq=False) +class KeyFrameData: + noise: Any = None + strength: Any = None + scale: Any = None + contrast: Any = None + kernel: int = 0 + sigma: Any = None + amount: Any = None + threshold: Any = None + cadence_flow_factor: Any = None + redo_flow_factor: Any = None + hybrid_comp_schedules: Any = None + + def kernel_size(self) -> tuple[int, int]: + return self.kernel, self.kernel + + def flow_factor(self): + return self.hybrid_comp_schedules['flow_factor'] + + def has_strength(self): + return self.strength > 0 + + @staticmethod + def create(data): + i = data.indexes.frame.i + keys: DeformAnimKeys = data.animation_keys.deform_keys + return KeyFrameData( + keys.noise_schedule_series[i], + keys.strength_schedule_series[i], + keys.cfg_scale_schedule_series[i], + keys.contrast_schedule_series[i], + int(keys.kernel_schedule_series[i]), + keys.sigma_schedule_series[i], + keys.amount_schedule_series[i], + keys.threshold_schedule_series[i], + keys.cadence_flow_factor_schedule_series[i], + keys.redo_flow_factor_schedule_series[i], + KeyFrameData._hybrid_comp_args(keys, i)) + + @staticmethod + def _hybrid_comp_args(keys, i): + return { + "alpha": keys.hybrid_comp_alpha_schedule_series[i], + "mask_blend_alpha": keys.hybrid_comp_mask_blend_alpha_schedule_series[i], + "mask_contrast": keys.hybrid_comp_mask_contrast_schedule_series[i], + "mask_auto_contrast_cutoff_low": int(keys.hybrid_comp_mask_auto_contrast_cutoff_low_schedule_series[i]), + "mask_auto_contrast_cutoff_high": int(keys.hybrid_comp_mask_auto_contrast_cutoff_high_schedule_series[i]), + "flow_factor": keys.hybrid_flow_factor_schedule_series[i]} diff --git a/scripts/deforum_helpers/rendering/util/depth_utils.py b/scripts/deforum_helpers/rendering/util/depth_utils.py new file mode 100644 index 000000000..8884665a9 --- /dev/null +++ b/scripts/deforum_helpers/rendering/util/depth_utils.py @@ -0,0 +1,14 @@ +import os + +from . import filename_utils, memory_utils + + +def generate_and_save_depth_map_if_active(data, opencv_image): + if data.args.anim_args.save_depth_maps: + memory_utils.handle_vram_before_depth_map_generation(data) + depth = data.depth_model.predict(opencv_image, data.args.anim_args.midas_weight, + data.args.root.half_precision) + depth_filename = filename_utils.depth_frame(data, data.indexes) + data.depth_model.save(os.path.join(data.output_directory, depth_filename), depth) + memory_utils.handle_vram_after_depth_map_generation(data) + return depth From a5874f8c847f52163d280b7c6a71d104d1404b5e Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 03:03:45 +0200 Subject: [PATCH 126/132] Renamed new core to experimental core and removed unused transformations on prev.image. More cleanup. --- scripts/deforum_helpers/render.py | 4 +-- .../deforum_helpers/rendering/data/mask.py | 2 +- .../rendering/data/render_data.py | 27 +++---------------- .../deforum_helpers/rendering/data/turbo.py | 20 -------------- .../{new_core.py => experimental_core.py} | 2 +- .../rendering/util/depth_utils.py | 18 +++++++++++++ .../rendering/util/filename_utils.py | 13 +++++---- 7 files changed, 31 insertions(+), 55 deletions(-) rename scripts/deforum_helpers/rendering/{new_core.py => experimental_core.py} (98%) diff --git a/scripts/deforum_helpers/render.py b/scripts/deforum_helpers/render.py index c1fda7bb9..f6696f23d 100644 --- a/scripts/deforum_helpers/render.py +++ b/scripts/deforum_helpers/render.py @@ -48,7 +48,7 @@ from .prompt import prepare_prompt from modules.shared import opts, cmd_opts, state, sd_model from modules import lowvram, devices, sd_hijack -from .rendering import new_core +from .rendering import experimental_core from .RAFT import RAFT from deforum_api import JobStatusTracker @@ -58,7 +58,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, contro is_use_key_frame_redistribution = parseq_args.parseq_key_frame_redistribution != "Off" is_use_new_render_core = is_use_parseq and is_use_key_frame_redistribution if is_use_new_render_core: - new_core.render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root) + experimental_core.render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root) return # initialise Parseq adapter diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index 2bd03d25c..07cc0ad9c 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -35,7 +35,7 @@ def _assign(init, i, is_mask_image, dicts): init.args.root.noise_mask = mask put_all(dicts, key, mask) elif is_mask_image is None and init.is_use_mask: - put_all(dicts, key, get_mask(init.args.args)) # TODO?: add a different default noise mask + put_all(dicts, key, get_mask(init.args.args)) @staticmethod def _load_mask(init, args): diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index a8269dae5..2b39df549 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -14,13 +14,12 @@ from .mask import Mask from .subtitle import Srt from .turbo import Turbo -from ..util import log_utils, memory_utils, opt_utils +from ..util import depth_utils, log_utils, memory_utils, opt_utils from ..util.call.images import call_get_mask_from_file_with_frame from ..util.call.mask import call_compose_mask_with_check from ..util.call.video_and_audio import call_get_next_frame from ...args import DeforumArgs, DeforumAnimArgs, LoopArgs, ParseqArgs, RootArgs from ...deforum_controlnet import unpack_controlnet_vids, is_controlnet_enabled -from ...depth import DepthModel from ...generate import (isJson) from ...parseq_adapter import ParseqAdapter from ...prompt import prepare_prompt @@ -67,7 +66,7 @@ def create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, animation_keys = AnimationKeys.from_args(ri_args, parseq_adapter, args.seed) animation_mode = AnimationMode.from_args(ri_args) prompt_series = RenderData.select_prompts(parseq_adapter, anim_args, animation_keys, root) - depth_model = RenderData.create_depth_model_and_enable_depth_map_saving_if_active( + depth_model = depth_utils.create_depth_model_and_enable_depth_map_saving_if_active( animation_mode, root, anim_args, args) # Temporary instance only exists for using it to easily create other objects required by the actual instance. @@ -147,9 +146,9 @@ def is_initialize_color_match(self, color_match_sample) -> bool: """Determines whether to initialize color matching based on the given conditions.""" has_video_input = self.args.anim_args.color_coherence == 'Video Input' and self.is_hybrid_available() has_image_color_coherence = self.args.anim_args.color_coherence == 'Image' - has_any_color_sample = color_match_sample is not None # TODO extract to own method? has_coherent_non_legacy_color_match = (self.args.anim_args.color_coherence != 'None' and not self.args.anim_args.legacy_colormatch) + has_any_color_sample = color_match_sample is not None has_sample_and_match = has_any_color_sample and has_coherent_non_legacy_color_match return has_video_input or has_image_color_coherence or has_sample_and_match @@ -284,7 +283,6 @@ def update_mask_image(self, step, mask): self.args.args.mask_image = None # we need it only after the first frame anyway def prepare_generation(self, data, step, i): - # TODO move all of this to Step? if i > self.args.anim_args.max_frames - 1: return self.update_some_args_for_current_step(step, i) @@ -327,25 +325,6 @@ def select_prompts(parseq_adapter, anim_args, animation_keys, root): return animation_keys.deform_keys.prompts if parseq_adapter.manages_prompts() \ else RenderData.expand_prompts_out_to_per_frame(anim_args, root) - @staticmethod - def is_composite_with_depth_mask(anim_args): - return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' - - @staticmethod - def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args): - # depth-based hybrid composite mask requires saved depth maps - # TODO avoid or isolate side effect: - anim_args.save_depth_maps = (anim_mode.is_predicting_depths - and RenderData.is_composite_with_depth_mask(anim_args)) - return DepthModel(root.models_path, - memory_utils.select_depth_device(root), - root.half_precision, - keep_in_vram=anim_mode.is_keep_in_vram, - depth_algorithm=anim_args.depth_algorithm, - Width=args.W, Height=args.H, - midas_weight=anim_args.midas_weight) \ - if anim_mode.is_predicting_depths else None - @staticmethod def expand_prompts_out_to_per_frame(anim_args, root): prompt_series = pd.Series([np.nan for _ in range(anim_args.max_frames)]) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 17d0b2dda..3de57c778 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -24,19 +24,12 @@ class Turbo: prev: ImageFrame next: ImageFrame - # depth: None - @staticmethod def create(data): steps = 1 if data.has_video_input() else data.cadence() return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) def advance(self, data, i: int, depth): - # log_utils.info(f"i {i} is prev {self.is_advance_prev(i)} is next {self.is_advance_next(i)}") - # log_utils.info(f"i {i} has prev {self.prev.image is not None} has next {self.next.image is not None}") - # TODO test if not replacing prev here is fine.. - # if self.prev.image is not None: - # self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) if self.next.image is not None: self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) @@ -59,8 +52,6 @@ def advance_optical_flow(self, tween_step, flow_factor: int = 1): def advance_optical_tween_flow(self, indexes, last_frame, flow): flow_factor = last_frame.step_data.flow_factor() i = indexes.tween.i - if self.is_advance_prev(i): - self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) @@ -77,17 +68,12 @@ def advance_cadence_flow(self, data, tween_frame): flow_factor = float(ff_string.split(": ")[1][1:-1]) i = tween_frame.i() flow = tween_frame.cadence_flow_inc - if self.is_advance_prev(i): - self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) - # TODO? move to RenderData def advance_ransac_transform(self, data, matrix): i = data.indexes.tween.i motion = data.args.anim_args.hybrid_motion - if self.is_advance_prev(i): - self.prev.image = image_transform_ransac(self.prev.image, matrix, motion) if self.is_advance_next(i): self.next.image = image_transform_ransac(self.next.image, matrix, motion) @@ -157,12 +143,6 @@ def find_start(self, data) -> int: def has_steps(self): return self.steps > 1 - def _has_prev_image(self): - return self.prev.image is not None - - def is_advance_prev(self, i: int) -> bool: - return self._has_prev_image() and i > self.prev.index - def is_advance_next(self, i: int) -> bool: return i > self.next.index diff --git a/scripts/deforum_helpers/rendering/new_core.py b/scripts/deforum_helpers/rendering/experimental_core.py similarity index 98% rename from scripts/deforum_helpers/rendering/new_core.py rename to scripts/deforum_helpers/rendering/experimental_core.py index e57fa63aa..235988f52 100644 --- a/scripts/deforum_helpers/rendering/new_core.py +++ b/scripts/deforum_helpers/rendering/experimental_core.py @@ -13,7 +13,7 @@ def render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root): - log_utils.debug("Using new render core.") + log_utils.info("Using experimental render core.") data = RenderData.create(args, parseq_args, anim_args, video_args, controlnet_args, loop_args, root) _check_experimental_render_conditions(data) web_ui_utils.init_job(data) diff --git a/scripts/deforum_helpers/rendering/util/depth_utils.py b/scripts/deforum_helpers/rendering/util/depth_utils.py index 8884665a9..2a5c97fa1 100644 --- a/scripts/deforum_helpers/rendering/util/depth_utils.py +++ b/scripts/deforum_helpers/rendering/util/depth_utils.py @@ -1,6 +1,7 @@ import os from . import filename_utils, memory_utils +from ...depth import DepthModel def generate_and_save_depth_map_if_active(data, opencv_image): @@ -12,3 +13,20 @@ def generate_and_save_depth_map_if_active(data, opencv_image): data.depth_model.save(os.path.join(data.output_directory, depth_filename), depth) memory_utils.handle_vram_after_depth_map_generation(data) return depth + + +def is_composite_with_depth_mask(anim_args): + return anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth' + + +def create_depth_model_and_enable_depth_map_saving_if_active(anim_mode, root, anim_args, args): + # depth-based hybrid composite mask requires saved depth maps + anim_args.save_depth_maps = anim_mode.is_predicting_depths and is_composite_with_depth_mask(anim_args) + return DepthModel(root.models_path, + memory_utils.select_depth_device(root), + root.half_precision, + keep_in_vram=anim_mode.is_keep_in_vram, + depth_algorithm=anim_args.depth_algorithm, + Width=args.W, Height=args.H, + midas_weight=anim_args.midas_weight) \ + if anim_mode.is_predicting_depths else None diff --git a/scripts/deforum_helpers/rendering/util/filename_utils.py b/scripts/deforum_helpers/rendering/util/filename_utils.py index 58dd619f3..5a5fc7847 100644 --- a/scripts/deforum_helpers/rendering/util/filename_utils.py +++ b/scripts/deforum_helpers/rendering/util/filename_utils.py @@ -2,7 +2,6 @@ from pathlib import Path from ..data import Indexes -from ..data.render_data import RenderData from ...video_audio_utilities import get_frame_name @@ -23,28 +22,28 @@ def _frame_filename_index(i: int, file_format: FileFormat) -> str: return f"{i:09}.{file_format.value}" -def frame_filename(data: RenderData, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: +def frame_filename(data, i: int, is_depth=False, file_format=FileFormat.frame_format()) -> str: infix = "_depth_" if is_depth else "_" return f"{data.args.root.timestring}{infix}{_frame_filename_index(i, file_format)}" -def frame(data: RenderData, indexes: Indexes) -> str: +def frame(data, indexes: Indexes) -> str: return frame_filename(data, indexes.frame.i) -def depth_frame(data: RenderData, indexes: Indexes) -> str: +def depth_frame(data, indexes: Indexes) -> str: return frame_filename(data, indexes.frame.i, True) -def tween_frame_name(data: RenderData, indexes: Indexes) -> str: +def tween_frame_name(data, indexes: Indexes) -> str: return frame_filename(data, indexes.tween.i) -def tween_depth_frame(data: RenderData, indexes: Indexes) -> str: +def tween_depth_frame(data, indexes: Indexes) -> str: return frame_filename(data, indexes.tween.i, True) -def preview_video_image_path(data: RenderData, indexes: Indexes) -> Path: +def preview_video_image_path(data, indexes: Indexes) -> Path: frame_name = get_frame_name(data.args.anim_args.video_init_path) index = _frame_filename_index(indexes.frame.i, FileFormat.video_frame_format()) return Path(data.output_directory) / "inputframes" / (frame_name + index) From 59bd7e131a7e5efa160c97584d38dab495803e92 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 03:25:54 +0200 Subject: [PATCH 127/132] More cleanup. --- .../rendering/data/frame/key_frame.py | 4 ---- .../rendering/data/frame/tween_frame.py | 2 +- scripts/deforum_helpers/rendering/data/indexes.py | 2 +- scripts/deforum_helpers/rendering/data/mask.py | 3 +-- .../deforum_helpers/rendering/data/schedule.py | 2 -- scripts/deforum_helpers/rendering/data/turbo.py | 15 +++++++-------- .../deforum_helpers/rendering/util/call/resume.py | 2 +- 7 files changed, 11 insertions(+), 19 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/frame/key_frame.py b/scripts/deforum_helpers/rendering/data/frame/key_frame.py index 4b666008f..2b52fea73 100644 --- a/scripts/deforum_helpers/rendering/data/frame/key_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/key_frame.py @@ -144,10 +144,6 @@ def maybe_redo_diffusion(self): self.do_diffusion_redo() def generate(self): - max_frames = self.render_data.args.anim_args.max_frames - if self.render_data.indexes.frame.i >= max_frames: - # TODO? prevents an error elsewhere when generating last frame, but `indexes`.. - self.render_data.indexes.update_frame(max_frames - 1) return call_generate(self.render_data, self) def after_diffusion(self, image): diff --git a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py index 727154807..c15a9de97 100644 --- a/scripts/deforum_helpers/rendering/data/frame/tween_frame.py +++ b/scripts/deforum_helpers/rendering/data/frame/tween_frame.py @@ -59,7 +59,7 @@ def process(self, last_frame, data): data.turbo.do_hybrid_video_motion(data, last_frame, self.indexes, data.images) def handle_synchronous_status_concerns(self, data): - self.write_tween_frame_subtitle_if_active(data) # TODO decouple from execution and calc all in advance? + self.write_tween_frame_subtitle_if_active(data) # TODO? decouple from execution and calc all in advance. log_utils.print_tween_frame_info(data, self.indexes, self.cadence_flow, self.value) web_ui_utils.update_progress_during_cadence(data, self.indexes) diff --git a/scripts/deforum_helpers/rendering/data/indexes.py b/scripts/deforum_helpers/rendering/data/indexes.py index 1369bbcfb..9478c9123 100644 --- a/scripts/deforum_helpers/rendering/data/indexes.py +++ b/scripts/deforum_helpers/rendering/data/indexes.py @@ -30,7 +30,7 @@ def create_next_tween(self): return Indexes(self.frame, IndexWithStart(self.tween.start, self.tween.i + 1)) def update_tween_start(self, turbo): - tween_start = max(self.frame.start, self.frame.i - turbo.steps) + tween_start = max(self.frame.start, self.frame.i - turbo.cadence) self.tween = IndexWithStart(tween_start, self.tween.i) def update_tween_index(self, i): diff --git a/scripts/deforum_helpers/rendering/data/mask.py b/scripts/deforum_helpers/rendering/data/mask.py index 07cc0ad9c..68790e111 100644 --- a/scripts/deforum_helpers/rendering/data/mask.py +++ b/scripts/deforum_helpers/rendering/data/mask.py @@ -9,8 +9,7 @@ from ...rendering.util.call.images import call_get_mask_from_file -# TODO freeze? -@dataclass(init=True, frozen=False, repr=False, eq=False) +@dataclass(init=True, frozen=True, repr=False, eq=False) class Mask: image: Image vals: Any diff --git a/scripts/deforum_helpers/rendering/data/schedule.py b/scripts/deforum_helpers/rendering/data/schedule.py index d47e064f0..6cc5d3c56 100644 --- a/scripts/deforum_helpers/rendering/data/schedule.py +++ b/scripts/deforum_helpers/rendering/data/schedule.py @@ -83,12 +83,10 @@ def schedule_ancestral_eta(keys, i, anim_args): @staticmethod def schedule_mask(keys, i, args): - # TODO can we have a mask schedule without a normal schedule? if so check and optimize return keys.mask_schedule_series[i] \ if args.use_mask and Schedule._has_mask_schedule(keys, i) else None @staticmethod def schedule_noise_mask(keys, i, anim_args): - # TODO can we have a noise mask schedule without a mask- and normal schedule? if so check and optimize return keys.noise_mask_schedule_series[i] \ if anim_args.use_noise_mask and Schedule._has_noise_mask_schedule(keys, i) else None diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 3de57c778..78d24c23c 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -2,7 +2,7 @@ from cv2.typing import MatLike -from ..util import opt_utils, log_utils +from ..util import opt_utils from ..util.call.anim import call_anim_frame_warp from ..util.call.hybrid import (call_get_flow_for_hybrid_motion_prev, call_get_flow_for_hybrid_motion, call_get_matrix_for_hybrid_motion, call_get_matrix_for_hybrid_motion_prev) @@ -17,10 +17,9 @@ class ImageFrame: index: int -# TODO freeze.. @dataclass(frozen=False) class Turbo: - steps: int # cadence + cadence: int prev: ImageFrame next: ImageFrame @@ -122,12 +121,12 @@ def do_optical_flow_cadence_after_animation_warping(self, data, indexes, tween_f def progress_step(self, indexes, opencv_image): self.prev.image, self.prev.index = self.next.image, self.next.index self.next.image, self.next.index = opencv_image, indexes.frame.i - return self.steps + return self.cadence def _set_up_step_vars(self, data): # determine last frame and frame to start on prev_frame, next_frame, prev_img, next_img = call_get_resume_vars(data, self) - if self.steps > 1: + if self.cadence > 1: self.prev.image, self.prev.index = prev_img, prev_frame if prev_frame >= 0 else 0 self.next.image, self.next.index = next_img, next_frame if next_frame >= 0 else 0 @@ -141,16 +140,16 @@ def find_start(self, data) -> int: return 0 def has_steps(self): - return self.steps > 1 + return self.cadence > 1 def is_advance_next(self, i: int) -> bool: return i > self.next.index def is_first_step(self) -> bool: - return self.steps == 1 + return self.cadence == 1 def is_first_step_with_subtitles(self) -> bool: return self.is_first_step() and opt_utils.is_subtitle_generation_active() def is_emit_in_between_frames(self) -> bool: - return self.steps > 1 + return self.cadence > 1 diff --git a/scripts/deforum_helpers/rendering/util/call/resume.py b/scripts/deforum_helpers/rendering/util/call/resume.py index 75dd44510..c9d6e3115 100644 --- a/scripts/deforum_helpers/rendering/util/call/resume.py +++ b/scripts/deforum_helpers/rendering/util/call/resume.py @@ -4,4 +4,4 @@ def call_get_resume_vars(data, turbo): return get_resume_vars(folder=data.args.args.outdir, timestring=data.args.anim_args.resume_timestring, - cadence=turbo.steps) + cadence=turbo.cadence) From 766add55f4871062509c017f828edb7fdfb7ef01 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Tue, 6 Aug 2024 04:19:22 +0200 Subject: [PATCH 128/132] Reverted some of the removed code for prev transformation, but added a new bool to turn it on or off and disabled it by default. --- .../deforum_helpers/rendering/data/turbo.py | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 78d24c23c..84d50d026 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -17,6 +17,12 @@ class ImageFrame: index: int +# Disabling transformations of previous frames may not be suited for all scenarios, +# but depending on setup can speed up generations significantly and without changing +# the visual output in a noticeable way. Leaving it off should be fine for current use cases. +IS_TRANSFORM_PREV = False # TODO? benchmark and visually compare results. make configurable from UI? + + @dataclass(frozen=False) class Turbo: cadence: int @@ -29,7 +35,9 @@ def create(data): return Turbo(steps, ImageFrame(None, 0), ImageFrame(None, 0)) def advance(self, data, i: int, depth): - if self.next.image is not None: + if self._has_prev_image() and IS_TRANSFORM_PREV: + self.prev.image, _ = call_anim_frame_warp(data, i, self.prev.image, depth) + if self._has_next_image(): self.next.image, _ = call_anim_frame_warp(data, i, self.next.image, depth) def do_hybrid_video_motion(self, data, last_frame, indexes, reference_images): @@ -51,6 +59,8 @@ def advance_optical_flow(self, tween_step, flow_factor: int = 1): def advance_optical_tween_flow(self, indexes, last_frame, flow): flow_factor = last_frame.step_data.flow_factor() i = indexes.tween.i + if self.is_advance_prev(i): + self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) @@ -67,12 +77,16 @@ def advance_cadence_flow(self, data, tween_frame): flow_factor = float(ff_string.split(": ")[1][1:-1]) i = tween_frame.i() flow = tween_frame.cadence_flow_inc + if self.is_advance_prev(i): + self.prev.image = image_transform_optical_flow(self.prev.image, flow, flow_factor) if self.is_advance_next(i): self.next.image = image_transform_optical_flow(self.next.image, flow, flow_factor) def advance_ransac_transform(self, data, matrix): i = data.indexes.tween.i motion = data.args.anim_args.hybrid_motion + if self.is_advance_prev(i): + self.prev.image = image_transform_ransac(self.prev.image, matrix, motion) if self.is_advance_next(i): self.next.image = image_transform_ransac(self.next.image, matrix, motion) @@ -142,6 +156,15 @@ def find_start(self, data) -> int: def has_steps(self): return self.cadence > 1 + def _has_prev_image(self): + return self.prev.image is not None + + def is_advance_prev(self, i: int) -> bool: + return IS_TRANSFORM_PREV and self._has_prev_image() and i > self.prev.index + + def _has_next_image(self): + return self.next.image is not None + def is_advance_next(self, i: int) -> bool: return i > self.next.index From 0fd98781737b744e0dcc3ae76faac16758f0d537 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 16 Aug 2024 21:24:33 +0200 Subject: [PATCH 129/132] README reverted back to original (PR preps). --- README.md | 113 +++++++++++++++++++++++------------------------------- 1 file changed, 48 insertions(+), 65 deletions(-) diff --git a/README.md b/README.md index bcf34fe7c..230e8ca63 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,74 @@ -# Experimental Deforum Fork -This is an experimental fork of the Deforum A1111 extension with a refactored render core that allows for -direct control of "turbo-frames" from Parseq. -### Current Status -The forked extension is work in progress and may not be stable. -Installation is recommended **only for the purpose of using the Parseq based key frame redistribution features**. +# Deforum Stable Diffusion โ€” official extension for AUTOMATIC1111's webui -โš ๏ธ Some Deforum features like hybrid video and coherence algorithms are currently not working and may need to be turned off. +

    + Last Commit + GitHub issues + GitHub stars + GitHub forks + +

    -For a feature complete and stable version, please refer to the original project at: [https://github.com/deforum-art/sd-webui-deforum](https://github.com/deforum-art/sd-webui-deforum) +## Need help? See our [FAQ](https://github.com/deforum-art/sd-webui-deforum/wiki/FAQ-&-Troubleshooting) -#### Installation -Since the name of this extension is shared with the original, it requires a full removal of the regular Deforum extension from Automatic 1111. -After that it can be installed from "Extensions" using "Install from URL": https://github.com/Tok/sd-webui-deforum -The forked extension adds a new "Neo Core" tab in Deforum, with further information on how to properly use it. +## Getting Started -๐Ÿ’ก The rest of this Readme can be skipped if you're not interested in the code. +1. Install [AUTOMATIC1111's webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui/). -## ๐ŸŽ๏ธ Parseq Key Frame Distribution +2. Now two ways: either clone the repo into the `extensions` directory via git commandline launched within in the `stable-diffusion-webui` folder -### Purpose & Features +```sh +git clone https://github.com/deforum-art/sd-webui-deforum extensions/deforum +``` -* Parseq Precision: All Parseq-provided key frames are guaranteed to be included in the final output, ensuring the highest possible sync precision. -* Cadence Flexibility: Cadence settings can be set to exceptionally high values (e.g., 10, 20+) or can be ignored completely without losing Parseq synchronization, enabling fast generations with less diffusion steps. -* No Workarounds: This approach eliminates the need for tricky workarounds when using Parseq with high or ignored cadence settings. +Or download this repository, locate the `extensions` folder within your WebUI installation, create a folder named `deforum` and put the contents of the downloaded directory inside of it. Then restart WebUI. -### ๐Ÿ“Š Key Index Distribution Modes +Or launch A1111, navigate to the Extensions tab, choose Available, find deforum in the list of available extensions and install it. Restart A1111 once the extension has been installed. +3. Open the webui, find the Deforum tab at the top of the page. -New key in 'deforum_settings.txt': "neocore_key_index_distribution" +4. Enter the animation settings. Refer to [this general guide](https://docs.google.com/document/d/1pEobUknMFMkn8F5TMsv8qRzamXX_75BShMMXV8IFslI/edit) and [this guide to math keyframing functions in Deforum](https://docs.google.com/document/d/1pfW1PwbDIuW0cv-dnuyYj1UzPqe23BlSLTJsqazffXM/edit?usp=sharing). However, **in this version prompt weights less than zero don't just like in original Deforum!** Split the positive and the negative prompt in the json section using --neg argument like this "apple:\`where(cos(t)>=0, cos(t), 0)\`, snow --neg strawberry:\`where(cos(t)<0, -cos(t), 0)\`" -#### ๐Ÿ•ณ๏ธ PARSEQ_ONLY (without cadence) -[`KeyIndexDistribution.PARSEQ_ONLY`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) -This distribution **completely ignores cadence** and is only diffusing key frames defined in Parseq (=frames that have a table entry in Parseq). -All non-key frames are handled as if they would be cadence frames, although that term doesn't really apply when the key frames are not spaced out in fixed intervals. +5. To view animation frames as they're being made, without waiting for the completion of an animation, go to the 'Settings' tab and set the value of this toolbar **above zero**. Warning: it may slow down the generation process. -#### ใ€ฐ๏ธ UNIFORM_WITH_PARSEQ (variable pseudo cadence) -[`KeyIndexDistribution.UNIFORM_WITH_PARSEQ`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data/step/key_index_distribution.py) -This mode ensures a uniform key frame distribution according to the specified cadence settings, while also prioritizing the inclusion of all key frames provided by Parseq. +![adsdasunknown](https://user-images.githubusercontent.com/14872007/196064311-1b79866a-e55b-438a-84a7-004ff30829ad.png) -Here's how it works: -1. The uniform key frame distribution is calculated based on the cadence settings, creating a set of non-cadence key frames. -2. The Parseq-provided key frames are then overlaid on top of this uniform distribution, replacing the closest non-cadence key frame. -The `UNIFORM_WITH_PARSEQ` distribution prioritizes Parseq key frames over the uniform cadence-based distribution, providing a balance between sync precision and generation speed. +6. Run the script and see if you got it working or even got something. **In 3D mode a large delay is expected at first** as the script loads the depth models. In the end, using the default settings the whole thing should consume 6.4 GBs of VRAM at 3D mode peaks and no more than 3.8 GB VRAM in 3D mode if you launch the webui with the '--lowvram' command line argument. -In essence, it means that **all Parseq key frames**, will be guaranteed to not -be a cadence-frames. We gain Parseq precision at the tradeoff of cadence regularity. +7. After the generation process is completed, click the button with the self-describing name to show the video or gif result right in the GUI! -##### Pseudo Cadence -`cadence` is still loosely observed in this mode, but since key frames are rearranged, the cadence settings should be understood as an average value. -In UNIFORM_WITH_PARSEQ mode, a cadence setting of "10" means "about 10". +8. Join our Discord where you can post generated stuff, ask questions and more: https://discord.gg/deforum.
    +* There's also the 'Issues' tab in the repo, for well... reporting issues ;) -## ๐Ÿ“ผ Video Examples +9. Profit! -### Parseq Synchronized Music Video -This music video was **generated in less than only 92 minutes on a 4070**, directly at 60 FPS and with a resolution of 1280x720. -It contains 6694 frames of which only 913 were actually diffused, but in full sync with Parseq. -Calculated pseudo cadence is 6694 / 913 = 7.33. More details in video description. +## Known issues -"neocore_key_index_distribution": "Parseq Only" +* This port is not fully backward-compatible with the notebook and the local version both due to the changes in how AUTOMATIC1111's webui handles Stable Diffusion models and the changes in this script to get it to work in the new environment. *Expect* that you may not get exactly the same result or that the thing may break down because of the older settings. -#### Mark Fell - multistability 2-b -[![Mark Fell - multistability 2-b](https://img.youtube.com/vi/K-9V5Tntck4/maxresdefault.jpg)](https://www.youtube.com/watch?v=K-9V5Tntck4 "Click to watch on YouTube") +## Screenshots -## ๐Ÿงน Refactored Render Core -This section details the changes made to the render core in this fork. +Amazing raw Deforum animation by [Pxl.Pshr](https://www.instagram.com/pxl.pshr): +* Turn Audio ON! -For easy integration, this fork isolates changes mostly to the [`render.py`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py) module and introduces the [`rendering`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering) package. -Existing code in other Deforum modules, remains mostly unchanged (exception is UI and args related code). +(Audio credits: SKRILLEX, FRED AGAIN & FLOWDAN - RUMBLE (PHACE'S DNB FLIP)) -![image](https://i.kym-cdn.com/photos/images/original/001/399/018/31f.gif) +https://user-images.githubusercontent.com/121192995/224450647-39529b28-be04-4871-bb7a-faf7afda2ef2.mp4 -* **๐Ÿ› ๏ธ Focus on Maintainability:** The core rendering functionality is being refactored step-by-step with a focus on improved readability, testability, and easier future modifications. -* **โ˜๏ธ Key Improvements:** - * Reduced cyclomatic complexity of the [`render_animation`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/render.py#L41) method. - * Improved separation of concerns (e.g., dedicated module for printing). - * Reduced argument complexity and improved scope control and variable organization using dataclasses. - * Enhanced code clarity with improved naming and removal of obsolete comments. - * But preserving domain specific lingo. - * Improved unit testing capabilities due to a more modular structure and due to using expressions in place of statements where applicable. +Setting file of that video: [here](https://github.com/deforum-art/sd-webui-deforum/files/11353167/PxlPshrWinningAnimationSettings.txt). -**๐Ÿ†• New Rendering Modules:** -* [`rendering/img_2_img_tubes`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/img_2_img_tubes.py): Provides functions for conditionally processing images through various transformations. -* [`rendering/data`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/data): Provides a set of dataclasses specifically designed for data and logic handling within `render.py`. -* [`rendering/util`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util): Contains stateless utility functions for data transformation and manipulation used in `render.py`. -* [`rendering/util/call`](https://github.com/Tok/sd-webui-deforum/blob/automatic1111-webui/scripts/deforum_helpers/rendering/util/call): Provides modules to forward calls to other Deforum modules and adapt them to work with the new data structures without modifying the original code and without polluting any receiver namespace. +
    -**๐Ÿ”Ž Implementation Details:** -* **Multiparadigmatic:** The code leverages a procedural core with functional tools to transform object-oriented data structures. -* **Style and Standards:** The code adheres to PEP 8 style guidelines and to other such practices. +Main extension tab: + +![image](https://user-images.githubusercontent.com/121192995/226101131-43bf594a-3152-45dd-a5d1-2538d0bc221d.png) + +Keyframes tab: + +![image](https://user-images.githubusercontent.com/121192995/226101140-bfe6cce7-9b78-4a1d-be9a-43e1fc78239e.png) + +## License + +This program is distributed under the terms of the GNU Affero Public License v3.0, copyright (c) 2023 Deforum LLC. + +Some of its sublicensed integrated 3rd party components may have other licenses, see LICENSE for usage terms. \ No newline at end of file From 5846810c892df7e0866f8d447ed874ce5e3e8978 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 16 Aug 2024 21:30:12 +0200 Subject: [PATCH 130/132] User specific gitignores removed. --- .gitignore | 3 --- README.md | 4 ++-- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/.gitignore b/.gitignore index 8fcf14a14..1b0eb546e 100644 --- a/.gitignore +++ b/.gitignore @@ -32,6 +32,3 @@ htmlcov tests/results.xml .coverage* serverlog.txt - -# PyCharm config -.idea diff --git a/README.md b/README.md index 230e8ca63..a514500c4 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Or launch A1111, navigate to the Extensions tab, choose Available, find deforum 7. After the generation process is completed, click the button with the self-describing name to show the video or gif result right in the GUI! 8. Join our Discord where you can post generated stuff, ask questions and more: https://discord.gg/deforum.
    -* There's also the 'Issues' tab in the repo, for well... reporting issues ;) +* There's also the 'Issues' tab in the repo, for well... reporting issues ;) 9. Profit! @@ -71,4 +71,4 @@ Keyframes tab: This program is distributed under the terms of the GNU Affero Public License v3.0, copyright (c) 2023 Deforum LLC. -Some of its sublicensed integrated 3rd party components may have other licenses, see LICENSE for usage terms. \ No newline at end of file +Some of its sublicensed integrated 3rd party components may have other licenses, see LICENSE for usage terms. From ce02d25574bafc3d5176d92e3ff0cf6599a1b299 Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Fri, 16 Aug 2024 22:29:58 +0200 Subject: [PATCH 131/132] Logging a new warning for when Parseq keyframe redistribution is setup to be used together with hybrid video and updated the UI text. --- scripts/deforum_helpers/args.py | 4 ++-- scripts/deforum_helpers/rendering/experimental_core.py | 10 +++++++--- scripts/deforum_helpers/ui_elements.py | 3 +++ 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/scripts/deforum_helpers/args.py b/scripts/deforum_helpers/args.py index 5496ecee8..2b881e7a2 100644 --- a/scripts/deforum_helpers/args.py +++ b/scripts/deforum_helpers/args.py @@ -993,11 +993,11 @@ def ParseqArgs(): "info": "Recommended. If you uncheck this, the FPS, max_frames and cadence in the Parseq doc are ignored, and the values in the A1111 UI are used instead." }, "parseq_key_frame_redistribution": { - "label": "Parseq key frame redistribution mode.", + "label": "Parseq key frame redistribution.", "type": "dropdown", "choices": get_parseq_keyframe_redistributions_list().values(), "value": "None", - "info": "Gain Parseq precision at the cost of cadence regularity." + "info": "Gain Parseq precision at the cost of cadence regularity. Allows for fast generations at high cadence." }, } diff --git a/scripts/deforum_helpers/rendering/experimental_core.py b/scripts/deforum_helpers/rendering/experimental_core.py index 235988f52..6c74788d6 100644 --- a/scripts/deforum_helpers/rendering/experimental_core.py +++ b/scripts/deforum_helpers/rendering/experimental_core.py @@ -75,11 +75,15 @@ def emit_tweens(data, key_step): def _check_experimental_render_conditions(data): if data.has_parseq_keyframe_redistribution(): - msg = "Using Parseq keyframe redistribution with {method}. Results may be unexpected." + msg = "Experimental conditions: Using 'Parseq keyframe redistribution' together with '{method}'. {results}. \ + In case of problems, consider deactivating either one." + dark_or_dist = "Resulting images may quickly end up looking dark or distorted." if data.has_optical_flow_cadence(): - log_utils.warn(msg.format(method="optical flow cadence")) + log_utils.warn(msg.format(method="optical flow cadence", results=dark_or_dist)) if data.has_optical_flow_redo(): - log_utils.warn(msg.format(method="optical flow generation")) + log_utils.warn(msg.format(method="optical flow generation", results=dark_or_dist)) + if data.is_hybrid_available(): + log_utils.warn(msg.format(method="hybrid video", results="Render process may not run stable.")) def _update_pseudo_cadence(data, value): diff --git a/scripts/deforum_helpers/ui_elements.py b/scripts/deforum_helpers/ui_elements.py index 548620c74..11aa0705e 100644 --- a/scripts/deforum_helpers/ui_elements.py +++ b/scripts/deforum_helpers/ui_elements.py @@ -588,6 +588,7 @@ def get_tab_output(da, dv): def create_keyframe_redistribution_info(): bars_mark = "📊" + warn_mark = "⚠️" gr.HTML(value=f"""

    Parseq keyframe redistribution ensures that every frame in the Parseq table is diffused. It may easily be used at high FPS with just a fixed value for 'strength' in Parseq \ @@ -608,4 +609,6 @@ def create_keyframe_redistribution_info(): Cadence may be understood as 'pseudo cadence'. \ A cadence value of '30' may more correctly be understood as 'about 30' in this mode.

+ {warn_mark} It's currently not recommended to use keyframe redistribution together with optical flow \ + or with hybrid video.

""") From 5c34eb74722470e03f2b0a617fa32f1abe883edc Mon Sep 17 00:00:00 2001 From: Zir Teq Date: Sat, 17 Aug 2024 00:08:25 +0200 Subject: [PATCH 132/132] Final 0. --- scripts/deforum_helpers/rendering/data/render_data.py | 2 +- scripts/deforum_helpers/rendering/data/turbo.py | 2 +- scripts/deforum_helpers/rendering/util/call/__init__.py | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/deforum_helpers/rendering/data/render_data.py b/scripts/deforum_helpers/rendering/data/render_data.py index 2b39df549..1cb27f536 100644 --- a/scripts/deforum_helpers/rendering/data/render_data.py +++ b/scripts/deforum_helpers/rendering/data/render_data.py @@ -239,7 +239,7 @@ def update_sub_seed_schedule_for_current_step(self, i): self.args.root.subseed_strength = float(keys.subseed_strength_schedule_series[i]) if is_seed_managed_by_parseq: self.args.root.subseed_strength = keys.subseed_strength_schedule_series[i] - self.args.anim_args.enable_subseed_scheduling = True # TODO should be enforced in init, not here. + self.args.anim_args.enable_subseed_scheduling = True # TODO? move to init. def prompt_for_current_step(self, i): """returns value to be set back into the prompt""" diff --git a/scripts/deforum_helpers/rendering/data/turbo.py b/scripts/deforum_helpers/rendering/data/turbo.py index 84d50d026..b1270bb1b 100644 --- a/scripts/deforum_helpers/rendering/data/turbo.py +++ b/scripts/deforum_helpers/rendering/data/turbo.py @@ -20,7 +20,7 @@ class ImageFrame: # Disabling transformations of previous frames may not be suited for all scenarios, # but depending on setup can speed up generations significantly and without changing # the visual output in a noticeable way. Leaving it off should be fine for current use cases. -IS_TRANSFORM_PREV = False # TODO? benchmark and visually compare results. make configurable from UI? +IS_TRANSFORM_PREV = False # TODO? benchmark and visually compare results. make configurable from UI or remove? @dataclass(frozen=False) diff --git a/scripts/deforum_helpers/rendering/util/call/__init__.py b/scripts/deforum_helpers/rendering/util/call/__init__.py index 67ebc35d8..0e1fc9734 100644 --- a/scripts/deforum_helpers/rendering/util/call/__init__.py +++ b/scripts/deforum_helpers/rendering/util/call/__init__.py @@ -1,9 +1,9 @@ """ -This module provides utility functions for simplifying calls to other modules within the `render.py` module. +This module provides utility functions for simplifying calls to other modules within the `experimental_core.py` module. **Purpose:** - **Reduce Argument Complexity:** Provides a way to call functions in other modules without directly handling - a large number of complex arguments. This simplifies code within `render.py` by encapsulating argument management. + a large number of complex arguments. This simplifies code within the core by encapsulating argument management. - **Minimize Namespace Pollution:** Provides an alternative to overloading methods in the original modules, which would introduce the `RenderInit` class into namespaces where it's not inherently needed.