Fix set_initialized_submodules bugs + improve asymptotic runtime#35698

Rocketknight1 wants to merge 5 commits intomainhuggingface/transformers:mainfrom

faster_set_initialized_submoduleshuggingface/transformers:faster_set_initialized_submodulesCopy head branch name to clipboard