ValueError: bad marshal data (unknown type code) when dannce-predict

[yuan0821]

I use the dannce_master branch, and all steps follow the instruction, except I pip install tensorflow==2.3.1.

The information of traceback:
`Traceback (most recent call last):
File "E:\anaconda\envs\tf23\Scripts\dannce-predict-script.py", line 33, in
sys.exit(load_entry_point('dannce', 'console_scripts', 'dannce-predict')())

File "f:\finnaldannce\dannce_master\dannce\cli.py", line 54, in dannce_predict_cli
dannce_predict(params)

File "f:\finnaldannce\dannce_master\dannce\interface.py", line 1534, in dannce_predict
"centered_euclidean_distance_3D": losses.centered_euclidean_distance_3D,

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\save.py", line 182, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 178, in load_model_from_hdf5 custom_objects=custom_objects)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
return deserialize(config, custom_objects=custom_objects)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 175, in deserialize
printable_module_name='layer')

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 358, in deserialize_keras_object
list(custom_objects.items())))

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 617, in from_config
config, custom_objects)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1204, in reconstruct_from_config
process_layer(layer_data)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1186, in process_layer
layer = deserialize_layer(layer_data, custom_objects=custom_objects)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 175, in deserialize
printable_module_name='layer')

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 358, in deserialize_keras_object
list(custom_objects.items())))

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1006, in from_config
config, custom_objects, 'function', 'module', 'function_type')

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1058, in _parse_function_from_config
config[func_attr_name], globs=globs)

File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 457, in func_load
code = marshal.loads(raw_code)

ValueError: bad marshal data (unknown type code)`

[yuan0821]

• (tf23) F:\finnaldannce\demo_video\mouse_4>dannce-predict dannce_mouse_config_4.yaml
• 2023-05-04 01:56:56.987771: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
• io_config not found in io.yaml file, falling back to main config
• new_n_channels_out not found in io.yaml file, falling back to main config
• downfac not found in io.yaml file, falling back to main config
• extension not found in io.yaml file, falling back to main config
• batch_size not found in io.yaml file, falling back to main config
• n_views not found in io.yaml file, falling back to main config
• mono not found in io.yaml file, falling back to main config
• debug not found in io.yaml file, falling back to main config
• epochs not found in io.yaml file, falling back to main config
• net_type not found in io.yaml file, falling back to main config
• train_mode not found in io.yaml file, falling back to main config
• num_validation_per_exp not found in io.yaml file, falling back to main config
• vol_size not found in io.yaml file, falling back to main config
• nvox not found in io.yaml file, falling back to main config
• max_num_samples not found in io.yaml file, falling back to main config
• loss not found in io.yaml file, falling back to main config
• augment_brightness not found in io.yaml file, falling back to main config
• n_rand_views not found in io.yaml file, falling back to main config
• dannce_finetune_weights not found in io.yaml file, falling back to main config
• predict_mode not found in io.yaml file, falling back to main config
• com_train_dir set to: .\COM\train_results
• com_predict_dir set to: .\COM\predict_results
• dannce_train_dir set to: .\DANNCE\train_results\AVG
• dannce_predict_dir set to: .\DANNCE\predict_results
• exp set to: [{'label3d_file': 'F:\finnaldannce\demo_video\mouse_4\du_cam1_to_cam5_dannce.mat'}]
• io_config set to: io.yaml
• new_n_channels_out set to: 16
• downfac set to: 4
• extension set to: .avi
• batch_size set to: 1
• n_views set to: 5
• mono set to: True
• debug set to: True
• epochs set to: 10
• net_type set to: AVG
• train_mode set to: finetune
• num_validation_per_exp set to: 0
• vol_size set to: 240
• nvox set to: 80
• max_num_samples set to: 100
• loss set to: mask_nan_l1_loss
• augment_brightness set to: True
• n_rand_views set to: None
• dannce_finetune_weights set to: F:\finnaldannce\weights\max5
• predict_mode set to: torch
• base_config set to: dannce_mouse_config_4.yaml
• viddir set to: videos
• crop_height set to: None
• crop_width set to: None
• camnames set to: None
• n_channels_out set to: 20
• sigma set to: 10
• verbose set to: 1
• net set to: None
• gpu_id set to: 0
• immode set to: vid
• mirror set to: False
• start_batch set to: 0
• start_sample set to: None
• com_fromlabels set to: False
• medfilt_window set to: None
• com_file set to: None
• new_last_kernel_size set to: [3, 3, 3]
• n_layers_locked set to: 2
• vmin set to: None
• vmax set to: None
• interp set to: nearest
• depth set to: False
• comthresh set to: 0
• weighted set to: False
• com_method set to: median
• cthresh set to: None
• channel_combo set to: None
• dannce_predict_model set to: None
• expval set to: None
• from_weights set to: None
• write_npy set to: None
• n_channels_in set to: None
• vid_dir_flag set to: None
• num_train_per_exp set to: None
• chunks set to: None
• lockfirst set to: None
• load_valid set to: None
• augment_hue set to: False
• augment_hue_val set to: 0.05
• augment_bright_val set to: 0.05
• augment_rotation_val set to: 5
• drop_landmark set to: None
• raw_im_h set to: None
• raw_im_w set to: None
• n_instances set to: 1
• use_npy set to: False
• data_split_seed set to: None
• valid_exp set to: None
• metric set to: ['euclidean_distance_3D']
• lr set to: 0.001
• rotate set to: True
• augment_continuous_rotation set to: False
• com_thresh set to: None
• cam3_train set to: None
• debug_volume_tifdir set to: None
• dannce_predict_vol_tifdir set to: None
• rand_view_replace set to: True
• multi_gpu_train set to: False
• heatmap_reg set to: False
• heatmap_reg_coeff set to: 0.01
• save_pred_targets set to: False
• Using the following *dannce.mat files: .\du_cam1_to_cam5_dannce.mat
• Setting vid_dir_flag to True.
• Setting extension to .avi.
• Setting chunks to {'Camera1': array([0]), 'Camera2': array([0]), 'Camera3': array([0]), 'Camera4': array([0]), 'Camera5': array([0])}.
• Setting n_channels_in to 3.
• Setting raw_im_h to 2560.
• Setting raw_im_w to 2560.
• Setting expval to True.
• Setting net to finetune_AVG.
• Setting crop_height to [0, 2560].
• Setting crop_width to [0, 2560].
• Setting maxbatch to 100.
• Setting start_batch to 0.
• Setting vmin to -120.0.
• Setting vmax to 120.0.
• Setting n_rand_views to None.
• Using the following *dannce.mat files: .\du_cam1_to_cam5_dannce.mat
• Using torch predict mode
• Using camnames: ['Camera1', 'Camera2', 'Camera3', 'Camera4', 'Camera5']
• Experiment 0 using com3d: .\du_cam1_to_cam5_dannce.mat
• Removed 0 samples from the dataset because they either had COM positions over cthresh, or did not have matching sampleIDs in the
• COM file
• Saving 3D COM to .\DANNCE\predict_results\com3d_used.mat
• None
• 2023-05-04 01:57:01.592851: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI
• Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
• To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
• 2023-05-04 01:57:01.810827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2053bf83360 initialized for platform
• Host (this does not guarantee that XLA will be used). Devices:
• 2023-05-04 01:57:01.811082: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
• 2023-05-04 01:57:01.815819: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
• 2023-05-04 01:57:01.845859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
• pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 3080 computeCapability: 8.6
• coreClock: 1.8GHz coreCount: 68 deviceMemorySize: 10.00GiB deviceMemoryBandwidth: 707.88GiB/s
• 2023-05-04 01:57:01.846193: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
• 2023-05-04 01:57:01.846419: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
• 2023-05-04 01:57:01.846580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
• 2023-05-04 01:57:01.846729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
• 2023-05-04 01:57:01.846879: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
• 2023-05-04 01:57:01.847007: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
• 2023-05-04 01:57:01.847140: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
• 2023-05-04 01:57:01.847339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
• 2023-05-04 01:57:02.723605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
• 2023-05-04 01:57:02.723854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
• 2023-05-04 01:57:02.724007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
• 2023-05-04 01:57:02.724268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3080, pci bus id: 0000:41:00.0, compute capability: 8.6)
• 2023-05-04 01:57:02.740601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x205a265c200 initialized for platform
• CUDA (this does not guarantee that XLA will be used). Devices:
• 2023-05-04 01:57:02.740811: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3080, Compute Capability 8.6
• Init took 2.1324803829193115 sec.
• Initializing Network...
• Loading model from .\DANNCE\train_results\AVG\weights.9-7.06966.hdf5