Skip to content

Commit

Permalink
fix bugs
Browse files Browse the repository at this point in the history
  • Loading branch information
@juxuan27
juxuan27 committed Aug 24, 2023
1 parent 6464915 commit ac11aea
Show file tree
Hide file tree
Showing 532 changed files with 80,503 additions and 51 deletions.
6 changes: 3 additions & 3 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,6 @@ cython_debug/
.vscode/

# Data
humansd_data
logs
sd_logs
humansd_data/checkpoints
humansd_data/datasets
logs
7 changes: 2 additions & 5 deletions .gitmodules
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
[submodule "comparison_models/T2I-Adapter"]
path = comparison_models/T2I-Adapter
url = [email protected]:TencentARC/T2I-Adapter.git
[submodule "comparison_models/ControlNet"]
path = comparison_models/ControlNet
url = git@github.com:lllyasviel/ControlNet-v1-1-nightly.git
url = https://github.com/lllyasviel/ControlNet.git
[submodule "comparison_models/T2IAdapter"]
path = comparison_models/T2IAdapter
url = git@github.com:TencentARC/T2I-Adapter.git
url = https://github.com/TencentARC/T2I-Adapter.git
102 changes: 65 additions & 37 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@ HumanSD shows its superiorities in terms of (I) challenging poses, (II) accurate

**Table of Contents**

- [HumanSD](#humansd)
- [TODO](#todo)
- [Model Overview](#model-overview)
- [Getting Started](#getting-started)
- [Environment Requirement](#environment-requirement)
- [Model and Checkpoints](#model-and-checkpoints)
- [Quick Demo](#quick-demo)
- [Dataset](#dataset)
- [Training](#training)
- [Training](#training)
- [Quantitative Results](#quantitative-results)
- [Qualitative Results](#qualitative-results)
- [Natural Scene](#natural-scene)
Expand Down Expand Up @@ -72,7 +74,7 @@ HumanSD has been implemented and tested on Pytorch 1.12.1 with python 3.9.

Clone the repo:
```bash
git clone --recursive [email protected]:IDEA-Research/HumanSD.git
git clone [email protected]:IDEA-Research/HumanSD.git
```

We recommend you first install `pytorch` following [official instructions](https://pytorch.org/get-started/previous-versions/). For example:
Expand All @@ -88,15 +90,10 @@ Then, you can install required packages thourgh:
pip install -r requirements.txt
```

You also need to install MMPose following [here](https://github.com/open-mmlab/mmpose). Noted that you only need to install MMPose as a python package.
You also need to install MMPose following [here](https://github.com/open-mmlab/mmpose). Noted that you only need to install MMPose as a python package. PS: Because of the update of MMPose, we recommend you to install 0.29.0 version of MMPose.

### Model and Checkpoints




**Checkpoints**

Download necessary checkpoints of HumanSD, which can be found [here](https://drive.google.com/drive/folders/1NLQAlF7i0zjEpd-XY0EcVw9iXP5bB5BJ?usp=sharing). The data structure should be like:

```
Expand All @@ -110,26 +107,25 @@ Download necessary checkpoints of HumanSD, which can be found [here](https://dri
Noted that v2-1_512-ema-pruned.ckpt should be download from [Stable Diffusion](https://github.com/Stability-AI/stablediffusion).


**models**

### Quick Demo

You also need to prepare the configs of MMPose models. You can directly download [mmpose/configs](https://github.com/open-mmlab/mmpose/tree/main/configs) and put it into humansd_data. Then the data structure will be:
You can run demo either through command line or gradio.

You can run demo through command line with:

```
|-- humansd_data
|-- models
|-- mmpose
|-- configs
|-- _base_
|-- animal
|-- ...
python scripts/pose2img.py --prompt "oil painting of girls dancing on the stage" --pose_file assets/pose/demo.npz
```

You can also run demo compared with ControlNet and T2I-Adapter:

```
python scripts/pose2img.py --prompt "oil painting of girls dancing on the stage" --pose_file assets/pose/demo.npz --controlnet --t2i
```

### Quick Demo

You can run demo through:
You can run gradio demo through:

```
python scripts/gradio/pose2img.py
Expand Down Expand Up @@ -177,34 +173,44 @@ You may refer to the code [here](ldm/data/humansd.py) for loading the data.
**Laion-Human**


You may apply for access of Laion-Human [here](https://forms.gle/ANxDTjxcE2Ua45oU8). Noted that we have provide the pose annotations, images' .parquet file and mapping file, please download the images according to .parquet. The `key` in .parquet is the corresponding image index. For example, image with `key=338717` in 00033.parquet is corresponding to images/00000/000338717.jpg. If you download the LAION-Aesthetics in tar files, which is different from our data structure, we recommend you extract the tar file through code:
You may apply for access of Laion-Human [here](https://forms.gle/ANxDTjxcE2Ua45oU8). Noted that we have provide the pose annotations, images' .parquet file and mapping file, please download the images according to .parquet. The `key` in .parquet is the corresponding image index. For example, image with `key=338717` in 00033.parquet is corresponding to images/00000/000338717.jpg.

```python
import tarfile
tar_file="00000.tar" # 00000.tar - 00286.tar
present_tar_path=f"xxxxxx/{tar_file}"
save_dir="humansd_data/datasets/Laion/Aesthetics_Human/images"
with tarfile.open(present_tar_path, "r") as tar_file:
for present_file in tar_file.getmembers():
if present_file.name.endswith(".jpg"):
print(f" image:- {present_file.name} -")
image_save_path=os.path.join(save_dir,tar_file.replace(".tar",""),present_file.name)
present_image_fp=TarIO.TarIO(present_tar_path, present_file.name)
present_image=Image.open(present_image_fp)
present_image_numpy=cv2.cvtColor(np.array(present_image),cv2.COLOR_RGB2BGR)
if not os.path.exists(os.path.dirname(image_save_path)):
os.makedirs(os.path.dirname(image_save_path))
cv2.imwrite(image_save_path,present_image_numpy)
After downloading the images and pose, you need to extract zip files and make it looks like:


```
|-- humansd_data
|-- datasets
|-- Laion
|-- Aesthetics_Human
|-- images
|-- 00000.parquet
|-- 00001.parquet
|-- ...
|-- pose
|-- 00000
|-- 000000000.npz
|-- 000000001.npz
|-- ...
|-- 00001
|-- ...
|-- mapping_file_training.json
```

The file data structure should be like:
Then, you can use `python utils/download_data.py` to download all images.


Then, the file data structure should be like:

```
|-- humansd_data
|-- datasets
|-- Laion
|-- Aesthetics_Human
|-- images
|-- 00000.parquet
|-- 00001.parquet
|-- ...
|-- 00000
|-- 000000000.jpg
|-- 000000001.jpg
Expand All @@ -221,6 +227,28 @@ The file data structure should be like:
|-- mapping_file_training.json
```



If you download the LAION-Aesthetics in tar files, which is different from our data structure, we recommend you extract the tar file through code:

```python
import tarfile
tar_file="00000.tar" # 00000.tar - 00286.tar
present_tar_path=f"xxxxxx/{tar_file}"
save_dir="humansd_data/datasets/Laion/Aesthetics_Human/images"
with tarfile.open(present_tar_path, "r") as tar_file:
for present_file in tar_file.getmembers():
if present_file.name.endswith(".jpg"):
print(f" image:- {present_file.name} -")
image_save_path=os.path.join(save_dir,tar_file.replace(".tar",""),present_file.name)
present_image_fp=TarIO.TarIO(present_tar_path, present_file.name)
present_image=Image.open(present_image_fp)
present_image_numpy=cv2.cvtColor(np.array(present_image),cv2.COLOR_RGB2BGR)
if not os.path.exists(os.path.dirname(image_save_path)):
os.makedirs(os.path.dirname(image_save_path))
cv2.imwrite(image_save_path,present_image_numpy)
```

**Human-Art**


Expand Down
Binary file added assets/pose/demo.npz
View file
Binary file not shown.
143 changes: 143 additions & 0 deletions comparison_models/ControlNet/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
.idea/

training/
lightning_logs/
image_log/

*.pth
*.pt
*.ckpt
*.safetensors

gradio_pose2image_private.py
gradio_canny2image_private.py

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/
Loading

0 comments on commit ac11aea

Please sign in to comment.