[BUG] MultiSystems load fparam shape mismatch

[anyangml]

Bug summary

When constructing the data set, fparam was registered

DataType(
            "fparam",
            np.ndarray,
            shape=(Axis.NFRAMES, 2),
            required=False,
        ),

...

system = dpdata.MultiSystems()
fparam = np.array([atoms.info["charge"],atoms.info["spin"]]).reshape(1,2)
assert fparam.shape == (1, 2)
frame.data["fparam"] = fparam
system.append(frame)
system.to_deepmd_npy_mixed(temp_out_pth)

When loading the system using

DataType(
            "fparam",
            np.ndarray,
            shape=(Axis.NFRAMES, 2),
            required=False,
        ),

sys = dpdata.MultiSystems()
sys.load_systems_from_file(filepth, fmt="deepmd/npy/mixed")

The following error occurs

dpdata.data_type.DataError: Shape of fparam is (2361, 2), but expected (6, 2)

dpdata Version

dpdata v0.2.24

Input Files, Running Commands, Error Log, etc.

Systems: 0%| | 0/1 [00:00<?, ?it/s]/mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10
Systems: 0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/mnt/data_nas/penganyang/LAMBench/lambench/models/ase_models.py", line 443, in
print(omol.run_ase_dptest(omol.calc, Path("/mnt/data_nas/public/public_training_data/OMol25_fparam/")))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/data_nas/penganyang/LAMBench/lambench/models/ase_models.py", line 288, in run_ase_dptest
sys.load_systems_from_file(filepth, fmt="deepmd/npy/mixed")
File "/mnt/data_nas/public/Miniconda/envs/omol25/lib/python3.11/site-packages/dpdata/system.py", line 1493, in load_systems_from_file
return self.from_fmt_obj(load_format(fmt), file_name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/data_nas/public/Miniconda/envs/omol25/lib/python3.11/site-packages/dpdata/system.py", line 1396, in from_fmt_obj
system_list.append(LabeledSystem(data=data_item, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/data_nas/public/Miniconda/envs/omol25/lib/python3.11/site-packages/dpdata/system.py", line 193, in init
self.check_data()
File "/mnt/data_nas/public/Miniconda/envs/omol25/lib/python3.11/site-packages/dpdata/system.py", line 221, in check_data
dd.check(self)
File "/mnt/data_nas/public/Miniconda/envs/omol25/lib/python3.11/site-packages/dpdata/data_type.py", line 119, in check
raise DataError(
dpdata.data_type.DataError: Shape of fparam is (2361, 2), but expected (6, 2)
(omol25) root@dsw-140567-9888d4895-8xtqz:/mnt/data_nas/penganyang/LAMBench/lambench/models# ^C
(omol25) root@dsw-140567-9888d4895-8xtqz:/mnt/data_nas/penganyang/LAMBench/lambench/models# ls /mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10
nopbc set.000 set.001 type_map.raw type.raw
(omol25) root@dsw-140567-9888d4895-8xtqz:/mnt/data_nas/penganyang/LAMBench/lambench/models# python -i
Python 3.11.0 (main, Mar 1 2023, 18:26:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
exit()
(omol25) root@dsw-140567-9888d4895-8xtqz:/mnt/data_nas/penganyang/LAMBench/lambench/models# ls /mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10/set.000
box.npy coord.npy energy.npy force.npy fparam.npy homo_energy.npy homo_lumo_gap.npy real_atom_types.npy
(omol25) root@dsw-140567-9888d4895-8xtqz:/mnt/data_nas/penganyang/LAMBench/lambench/models# python -i
Python 3.11.0 (main, Mar 1 2023, 18:26:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import numpy as np
np.load("/mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10/set.000/fparam.npy").shape
(2000, 2)
np.load("/mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10/set.001/fparam.npy").shape
(361, 2)
np.load("/mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10/set.001/real_atom_types.npy").shape
(361, 10)
np.load("/mnt/data_nas/public/public_training_data/OMol25_fparam/val_0/10/set.000/real_atom_types.npy").shape
(2000, 10)

Steps to Reproduce

debug.zip

from dpdata.data_type import (
    Axis,
    DataType,
)
import dpdata

datatypes_ef=[
        DataType(
            "fparam",
            np.ndarray,
            shape=(Axis.NFRAMES, 2),
            required=False,
        ),
    ]
    
for datatype in datatypes_ef:
    dpdata.System.register_data_type(datatype)
    dpdata.LabeledSystem.register_data_type(datatype)


sys = dpdata.MultiSystems()
sys.load_systems_from_file("debug", fmt="deepmd/npy/mixed")

Further Information, Files, and Links

No response

[anyangml]

My guess is there is something wrong with fparam saving as mixed type

atom_numbs (<Axis.NTYPES: 'ntypes'>,)
[82]
atom_names (<Axis.NTYPES: 'ntypes'>,)
[82]
atom_types (<Axis.NATOMS: 'natoms'>,)
[10]
orig (3,)
[3]
cells (<Axis.NFRAMES: 'nframes'>, 3, 3)
[6, 3, 3]
coords (<Axis.NFRAMES: 'nframes'>, <Axis.NATOMS: 'natoms'>, 3)
[6, 10, 3]
energies (<Axis.NFRAMES: 'nframes'>,)
[6]
forces (<Axis.NFRAMES: 'nframes'>, <Axis.NATOMS: 'natoms'>, 3)
[6, 10, 3]
fparam (<Axis.NFRAMES: 'nframes'>, 2)
[6, 2]

[anyangml]

likely we need to add fparam, aparam here

dpdata/dpdata/deepmd/mixed.py

Lines 59 to 70 in 08aefba

	if labels:
	if all_eners_concat is not None and all_eners_concat.size > 0:
	temp_data["energies"] = all_eners_concat[temp_idx]
	all_eners_concat = all_eners_concat[rest_idx]
	if all_forces_concat is not None and all_forces_concat.size > 0:
	temp_data["forces"] = all_forces_concat[temp_idx]
	all_forces_concat = all_forces_concat[rest_idx]
	if all_virs_concat is not None and all_virs_concat.size > 0:
	temp_data["virials"] = all_virs_concat[temp_idx]
	all_virs_concat = all_virs_concat[rest_idx]
	data_list.append(temp_data)
	return data_list