Skip to content

Latest commit

 

History

History

humman_recon

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
tools
tools
 
 
README.md
README.md
 
 

README.md

HuMMan Release v1.0: Reconstruction Subset (HuMMan-Recon)

HuMMan v1.0: Reconstruction Subset (HuMMan-Recon) consists of 153 subjects and 339 sequences. Color images, masks (via matting), SMPL parameters, and camera parameters are provided. It is a challenging dataset for its collection of diverse subject appearance and expressive actions. Moreover, it unleashes the potential to benchmark reconstruction algorithms under realistic settings with commercial sensors, dynamic subjects, and computer vision-powered automatic annotations.

Installation

To use our visulization tools, relevant python packages need to be installed.

conda create -n humman python=3.9 -y
conda activate humman
pip install torch==1.12.1 opencv-python==4.10.0.84 smplx==0.1.28 chumpy==0.70 trimesh==4.4.3 tqdm==4.66.4 open3d==0.14.1 numpy==1.23.1

It is also highly recommended to install openxlab package to facilitate file downloading.

pip install openxlab

Downloads

Option 1: OpenXLab

HuMMan-Recon is currently hosted on OpenXLab. We recommend download files using CLI tools:

openxlab dataset download --dataset-repo OpenXDLab/HuMMan --source-path /humman_release_v1.0_recon --target-path /home/user/

You can selectively download files that you need, for example:

openxlab dataset download --dataset-repo OpenXDLab/HuMMan --source-path /humman_release_v1.0_recon/recon_kinect_color_part_1.zip --target-path /home/user/humman_release_v1.0_recon

Option 2: Hugging Face

HuMMan-Recon is also hosted on Hugging Face. Hugging Face uses git-lfs to manage large files.

Please make sure you have git-lfs installed. Then, follow the instructions below:

git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/caizhongang/HuMMan  # do not pull any large files yet
cd HuMMan

You may pull all files in HuMMan-Recon:

git lfs pull --include "humman_release_v1.0_recon/*"

Similarly, you can also selectively download files that you need, for example:

git lfs pull --include "humman_release_v1.0_recon/recon_iphone_color.7z"

Option 3: OneDrive

We have backed-up all files on OneDrive (except for depth images).

Color images:

Note that the 3 parts downloaded from OneDrive is the same to 13 parts downloaded from other data hosts.

Masks:

  • Manually annotated for color images in the test split only: OneDrive(CN) (~32 MB)
  • Generated via matting for color images in all splits: OneDrive(CN) (~2.1 GB)

SMPL parameters: OneDrive(CN) (~7.6 MB)

Camera parameters (world2cam): OneDrive(CN) (~1.3 MB).

Textured meshes: OneDrive(CN) (~22 GB)

Data Structure

Please download the .zip files and place in the same directory, note that you may not need all of them.

humman_release_v1.0_recon/   
├── recon_kinect_color_part_1.zip
├── recon_kinect_color_part_2.zip
├── recon_kinect_color_part_3.zip
├── recon_kinect_color_part_4.zip
├── recon_kinect_color_part_5.zip
├── recon_kinect_color_part_6.zip
├── recon_kinect_color_part_7.zip
├── recon_kinect_color_part_8.zip
├── recon_kinect_color_part_9.zip
├── recon_kinect_color_part_10.zip
├── recon_kinect_color_part_11.zip
├── recon_kinect_color_part_12.zip
├── recon_kinect_color_part_13.zip
├── recon_kinect_mask_manual.zip
├── recon_kinect_mask.zip
├── recon_smpl_params.zip 
├── recon_cameras.zip
├── recon_textured_meshes.zip
└── recon_kinect_depth.zip

Then decompress them:

unzip "*.zip"

The file structure should look like this:

humman_release_v1.0_recon/   
└── pxxxxxx_axxxxxx/  
    ├── kinect_color/
    │   ├── kinect_000/
    │   ...
    │   └── kinect_009/
    │       ├── 000000.png (uint8, 3 channels)
    │       ├── 000006.png
    │       ...
    │       └── xxxxxx.png
    │
    ├── kinect_mask_manual/
    │   ├── kinect_000/
    │   ...
    │   └── kinect_009/
    │       ├── 000000.png (uint8, 1 channel)
    │       ├── 000006.png
    │       ...
    │       └── xxxxxx.png
    │
    ├── kinect_mask/
    │   ├── kinect_000/
    │   ...
    │   └── kinect_009/
    │       ├── 000000.png (uint8, 1 channel)
    │       ├── 000006.png
    │       ...
    │       └── xxxxxx.png
    │
    ├── kinect_depth/
    │   ├── kinect_000/
    │   ...
    │   └── kinect_009/
    │       ├── 000000.png (uint16, 1 channel)
    │       ├── 000006.png
    │       ...
    │       └── xxxxxx.png
    │
    ├── smpl_params/
    │   ├── 000000.npz
    │   ├── 000006.npz
    │   ...
    │   └── xxxxxx.npz
    │
    ├── cameras.json
    │
    └── textured_meshes/
        ├── 000000.mtl
        ├── 000000.obj
        ├── 000000_0.png*
        ├── 000006.mtl
        ├── 000006.obj
        ├── 000006_0.png*
        ...
        ├── xxxxxx.mtl
        ├── xxxxxx.obj
        └── xxxxxx_0.png*
  • * indicates that there be multiple .png for one .obj file.

kinect_color/

import cv2
color_bgr = cv2.imread('/path/to/xxxxxxx.png')
color_rgb = cv2.cvtColor(color_bgr, cv2.COLOR_BGR2RGB)  # if RGB images are used

kinect_mask/ or kinect_mask_manual/

import cv2
mask = cv2.imread('/path/to/xxxxxxx.png', cv2.IMREAD_GRAYSCALE)  # grayscale

smpl_parms/

SMPL parameters are in the world coordinate system. The world coordinate system is the same as kinect_color_000 coordinate system. Each .npz consists of the following:

{
    'betas': np.array of shape (10,),
    'body_pose': np.array of shape (69,),
    'global_orient': np.array of shape (3,),
    'transl': np.array of shape (3,)
}

To read:

import numpy as np
smpl_params = np.load('/path/to/xxxxxx.npz')
global_orient = smpl_params['global_orient']
body_pose = smpl_params['body_pose']
betas = smpl_params['betas']
transl = smpl_params['transl']

cameras.json

Cameras use the OpenCV pinhole camera model.

  • K: intrinsic matrix
  • R: rotation matrix
  • T: translation vector

R, T form world2cam transformation. The world coordinate system is the same as kinect_color_000 coordinate system.

Each .json consists of parameters for 10 color cameras and 10 depth cameras:

{
    "kinect_color_000": {
        "K": np.array of shape (3,3),
        "R": np.array of shape (3,3),
        "T": np.array of shape (3,)
    },
    "kinect_depth_000": {
        "K": np.array of shape (3,3),
        "R": np.array of shape (3,3),
        "T": np.array of shape (3,)
    },
    ...
    "kinect_color_009": {...}
    "kinect_depth_009": {...}
}

To read:

import json
with open('/path/to/cameras.json', 'r') as f:
    cameras = json.load(f)
camera_params = cameras['kinect_color_000']  # e.g., Kinect ID = 0
K, R, T = camera_params['K'], camera_params['R'], camera_params['T']

kinect_depth/

To read the depth maps

import cv2
depth_image = cv2.imread('/path/to/xxxxxxx.png', cv2.IMREAD_UNCHANGED)

To convert the depth maps to point clouds in camera coordinate system

import open3d as o3d
import numpy as np
import json
import cv2

# load depth image
depth_image = cv2.imread('/path/to/xxxxxxx.png', cv2.IMREAD_UNCHANGED)

# load depth camera parameters
with open('/path/to/cameras.json', 'r') as f:
    cameras = json.load(f)
camera_params = cameras['kinect_depth_000']  # e.g., Kinect ID = 0
K, R, T = camera_params['K'], camera_params['R'], camera_params['T']

# initialize open3d camera
open3d_camera = o3d.camera.PinholeCameraParameters()
open3d_camera.intrinsic.set_intrinsics(
    width=640, height=576, fx=K[0, 0], fy=K[1, 1], cx=K[0, 2], cy=K[1, 2])

# generate point cloud
depth_image = o3d.geometry.Image(depth_image)
open3d_point_cloud = o3d.geometry.PointCloud.create_from_depth_image(
    depth_image, open3d_camera.intrinsic, depth_trunc=5.0)
point_cloud = np.array(open3d_point_cloud.points)

Visualization

We provide a simple 2D visualization tool for color images, masks, and SMPL vertices, and a simple 3D visualization tool for point clouds (from depth images) and SMPL mesh models.

Run 2D Visualizer

python tools/visualizer.py <root_dir> <seq_name> <kinect_id> <frame_id> \
  [--visualize_mask] [--visualize_smpl] [--smpl_model_path]
  • root_dir (str): root directory in which data is stored.
  • seq_name (str): sequence name, in the format 'pxxxxxx_axxxxxx'.
  • kinect_id (int): Kinect ID. Available range is [0, 9].
  • frame_id (int): frame ID. Available range varies for different sequences.
  • visualize_mask (flag): whether to overlay mask on color image.
  • visualize_mask_manual (flag): whether to overlay manually annotated mask on color image.
  • visualize_smpl (flag): whether to overlay SMPL vertices on color image.
  • smpl_model_path (str, optional): directory in which SMPL body models are stored. Defaults to /home/user/body_models/.

Example:

python tools/visualizer.py /home/user/humman_release_v1.0_recon p000455_a000986 0 0 \
  --visualize_mask_manual --visualize_smpl --smpl_model_path /home/user/body_models/

Note that the SMPL model path should consist the following structure:

body_models/   
└── smpl/  
    └── SMPL_NEUTRAL.pkl

SMPL_NEUTRAL.pkl may be downloaded from the official website.

Run 3D Visualizer

python tools/visualizer_3d.py <root_dir> <seq_name> <kinect_id> <frame_id> \
  [--visualize_smpl] [--smpl_model_path]
  • root_dir (str): root directory in which data is stored.
  • seq_name (str): sequence name, in the format 'pxxxxxx_axxxxxx'.
  • kinect_id (int): Kinect ID. Available range is [0, 9].
  • frame_id (int): frame ID. Available range varies for different sequences.
  • visualize_smpl (flag): whether to visualize SMPL 3D mesh model.
  • smpl_model_path (str, optional): directory in which SMPL body models are stored. Defaults to /home/user/body_models/.

Example:

python tools/visualizer_3d.py /home/user/humman_release_v1.0_recon p000455_a000986 0 0 \
  --visualize_smpl --smpl_model_path /home/user/body_models/