add DPA-ADAPT toolkit for downstream property adaptation

.github/workflows/test_python.yml

@@ -47,6 +47,7 @@ jobs:
           HOROVOD_WITH_MPI: 1
           # https://cmake.org/cmake/help/latest/variable/CMAKE_POLICY_VERSION_MINIMUM.html
           CMAKE_POLICY_VERSION_MINIMUM: 3.5
+          UV_HTTP_TIMEOUT: 120
       - run: dp --version
       - name: Get durations from cache
         uses: actions/cache@v6

README.md

@@ -24,6 +24,7 @@ For more information, check the [documentation](https://deepmd.readthedocs.io/).
 - **implements the Deep Potential series models**, which have been successfully applied to finite and extended systems, including organic molecules, metals, semiconductors, insulators, etc.
 - **implements MPI and GPU supports**, making it highly efficient for high-performance parallel and distributed computing.
 - **highly modularized**, easy to adapt to different descriptors for deep learning-based potential energy models.
+- **adapts pre-trained DPA models to downstream atomistic property prediction tasks with DPA-ADAPT**, a new Python API and CLI that supports frozen-descriptor scikit-learn heads, frozen property-head training, full end-to-end fine-tuning, and multi-task fine-tuning with an auxiliary force-field task. DPA-ADAPT trains on `deepmd/npy` systems and provides conversion pipelines for SMILES tables and structure or calculation files handled through dpdata. See the [DPA-ADAPT guide](doc/dpa_adapt/overview.md) and supported [input formats](doc/dpa_adapt/input_formats.md).
 
 ### License and credits
 
@@ -103,6 +104,7 @@ The code is organized as follows:
 
 - `examples`: examples.
 - `deepmd`: DeePMD-kit python modules.
+- `dpa_adapt`: DPA-ADAPT package for adapting pre-trained DPA models; see the [guide](doc/dpa_adapt/overview.md) and [input formats](doc/dpa_adapt/input_formats.md).
 - `source/lib`: source code of the core library.
 - `source/op`: Operator (OP) implementation.
 - `source/api_cc`: source code of DeePMD-kit C++ API.

deepmd/__about__.py

@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: LGPL-3.0-or-later
+__version__ = "0.0.0"

deepmd/entrypoints/test.py

@@ -307,8 +307,8 @@ def save_txt_file(
     append : bool, optional
         if true file will be appended instead of overwriting, by default False
     """
-    flags = "ab" if append else "w"
-    with fname.open(flags) as fp:
+    flags = "a" if append else "w"
+    with fname.open(flags, encoding="utf-8") as fp:
         np.savetxt(fp, data, header=header)
 
 

doc/cli.rst

@@ -1,7 +1,7 @@
 .. _cli:
 
-Command line interface
-======================
+DeePMD-kit command line interface
+=================================
 
 .. argparse::
    :module: deepmd.tf.entrypoints.main

doc/conf.py

@@ -7,6 +7,10 @@
 
 # -- Path setup --------------------------------------------------------------
 
+from __future__ import (
+    annotations,
+)
+
 import datetime
 
 # If extensions (or modules to document with autodoc) are in another directory,
@@ -224,3 +228,54 @@
 bibtex_bibfiles = ["../CITATIONS.bib"]
 
 remove_from_toctrees = ["autoapi/**/*", "API_CC/*", "api_c/*", "api_core/*"]
+
+
+# Auto-generated CLI reference pages (sphinx-argparse) nest a section per
+# subcommand and per argument group. Under the global ``:numbered:`` toctree
+# this explodes into unhelpful deep numbers (e.g. ``9.3.3.6.3.1.1.``). Cap the
+# section numbering at the given depth (number of dotted components); headings
+# deeper than that are left unnumbered. Only the listed pages are affected.
+from typing import (
+    TYPE_CHECKING,
+)
+
+from docutils import (
+    nodes,
+)
+
+if TYPE_CHECKING:
+    from sphinx.application import (
+        Sphinx,
+    )
+
+cli_secnumber_max_depth = {
+    "dpa_adapt/cli": 5,
+}
+
+
+def _cap_cli_secnumbers(app: Sphinx, doctree: nodes.document, docname: str) -> None:
+    """Drop section numbers below ``cli_secnumber_max_depth`` for CLI pages."""
+    max_depth = cli_secnumber_max_depth.get(docname)
+    if max_depth is None:
+        return
+    secnumbers = app.env.toc_secnumbers.get(docname)
+    if not secnumbers:
+        return
+    # The empty anchor "" holds the page chapter number (e.g. ``(9, 3)``).
+    # It must be dropped from the map, otherwise the writer falls back to it for
+    # the now-unnumbered deep sections; re-attach it to the page title instead.
+    page_number = secnumbers.get("")
+    app.env.toc_secnumbers[docname] = {
+        anchor: number
+        for anchor, number in secnumbers.items()
+        if anchor != "" and len(number) <= max_depth
+    }
+    if page_number:
+        for title in doctree.findall(nodes.title):
+            title["secnumber"] = page_number
+            break
+
+
+def setup(app: Sphinx) -> dict[str, bool]:
+    app.connect("doctree-resolved", _cap_cli_secnumbers)
+    return {"parallel_read_safe": True, "parallel_write_safe": True}

doc/dpa_adapt/cli.rst

@@ -0,0 +1,10 @@
+DPA-ADAPT command line interface
+================================
+
+The ``dpaad`` command is a short alias for ``dpa-adapt`` and exposes the same
+subcommands and options.
+
+.. argparse::
+   :module: dpa_adapt.cli
+   :func: get_parser
+   :prog: dpa-adapt

doc/dpa_adapt/index.rst

@@ -0,0 +1,15 @@
+DPA-ADAPT
+=========
+
+DPA-ADAPT adapts pretrained DPA models to downstream atomistic property
+prediction tasks through Python APIs and the standalone ``dpa-adapt`` CLI.
+It supports frozen-descriptor scikit-learn heads, frozen property-head
+training, full fine-tuning, and multi-task fine-tuning with an auxiliary
+force-field task.
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   input_formats
+   cli

doc/dpa_adapt/input_formats.md

@@ -0,0 +1,146 @@
+# Input Formats
+
+> **Project/package name:** `dpa-adapt`
+> **Python import:** `dpa_adapt`
+> **Main CLI:** `dpa-adapt`
+> **Optional short alias:** `dpaad`
+> **Display name:** DPA-ADAPT — Atomistic DPA Adaptation for Property Tasks
+
+`dpa-adapt data convert` and the Python `dpa_adapt.convert()` helper
+auto-detect the input type and route it to the correct pipeline:
+**SMILES table** → RDKit 3D conformer generation,
+**structure files** → dpdata (auto-detect or explicit `--fmt`).
+
+## SMILES Tables (CSV)
+
+**Trigger:** file extension `.csv` **and** a SMILES column.
+By default, the converter reads `SMILES`/`smiles`; use `--smiles-col` for
+other column names such as `smi` or `mol`. Or pass `--fmt smiles` explicitly.
+
+| Parameter          | Default       | Description                                                                                                        |
+| ------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `--smiles-col`     | `SMILES`      | Column name for SMILES strings                                                                                     |
+| `--property-col`   | `Property`    | Input table column to read target values from; also used as the output label name                                  |
+| `--train-ratio`    | `0.9`         | Fraction of samples used for training set                                                                          |
+| `--mol-dir`        | —             | Directory of pre-generated `.mol`, `.sdf`, `.xyz`, or `.pdb` structure files (skips RDKit 3D conformer generation) |
+| `--mol-template`   | `id{row}.mol` | Filename template under `--mol-dir`; use `{row}` for the CSV row index                                             |
+| `--split-seed`     | `42`          | Random seed for train/valid splitting                                                                              |
+| `--conformer-seed` | `42`          | Random seed for RDKit 3D conformer generation                                                                      |
+
+```bash
+# Auto-detected via SMILES column
+dpa-adapt data convert --input molecules.csv --output ./npy \
+    --property-col homo
+# Short alias
+dpaad data convert --input molecules.csv --output ./npy \
+    --property-col homo
+
+# Explicit fmt + custom column names
+dpa-adapt data convert --input data.csv --output ./npy --fmt smiles \
+    --smiles-col smi --property-col GAP --train-ratio 0.85 \
+    --split-seed 42 --conformer-seed 43
+# Short alias
+dpaad data convert --input data.csv --output ./npy --fmt smiles \
+    --smiles-col smi --property-col GAP --train-ratio 0.85 \
+    --split-seed 42 --conformer-seed 43
+```
+
+## Structure Files via dpdata
+
+**Trigger:** inputs not routed to the SMILES pipeline. This means `--fmt` is
+not `smiles`; when `--fmt` is omitted, CSV inputs are routed here only if they
+do not contain a recognized SMILES column.
+Calls dpdata for format auto-detection or explicit conversion.
+
+### Common Formats
+
+| `--fmt` value                                            | Typical file(s)               | Notes                                  |
+| -------------------------------------------------------- | ----------------------------- | -------------------------------------- |
+| `xyz`                                                    | `*.xyz`                       | Plain XYZ                              |
+| `vasp/poscar` / `vasp/contcar`                           | `POSCAR`, `CONTCAR`           | VASP input/final structure             |
+| `vasp/outcar`                                            | `OUTCAR`                      | VASP output (energies, forces, stress) |
+| `vasp/xml`                                               | `vasprun.xml`                 | VASP XML output                        |
+| `vasp/string`                                            | VASP structure string         | VASP structure from a string           |
+| `abacus/stru` / `stru`                                   | `STRU`                        | ABACUS input structure                 |
+| `abacus/scf` / `abacus/pw/scf` / `abacus/lcao/scf`       | SCF output                    | ABACUS SCF calculation                 |
+| `abacus/md` / `abacus/pw/md` / `abacus/lcao/md`          | MD output                     | ABACUS molecular dynamics              |
+| `abacus/relax` / `abacus/pw/relax` / `abacus/lcao/relax` | Relax output                  | ABACUS relaxation                      |
+| `cp2k/aimd_output`                                       | CP2K MD output                | CP2K AIMD output file                  |
+| `cp2k/output`                                            | CP2K SCF output               | CP2K single-point output               |
+| `deepmd/raw`                                             | `set.*/` dirs                 | DeePMD-kit raw format                  |
+| `deepmd/comp` / `deepmd/npy`                             | `set.*/` dirs                 | DeePMD-kit compressed/npy format       |
+| `deepmd/npy/mixed`                                       | mixed `deepmd/npy` dir        | DeePMD-kit mixed npy format            |
+| `deepmd/hdf5`                                            | `*.hdf5`                      | DeePMD-kit HDF5 format                 |
+| `lammps/dump` / `dump`                                   | `dump.*`                      | LAMMPS dump trajectory                 |
+| `lammps/lmp` / `lmp`                                     | `*.lmp`                       | LAMMPS data file                       |
+| `qe/cp/traj`                                             | CP trajectory                 | Quantum ESPRESSO Car-Parrinello MD     |
+| `qe/pw/scf`                                              | PWscf output                  | Quantum ESPRESSO PWscf                 |
+| `siesta/output`                                          | Siesta output                 | SIESTA SCF output                      |
+| `siesta/aimd_output`                                     | Siesta MD output              | SIESTA AIMD output                     |
+| `gaussian/log`                                           | `*.log`                       | Gaussian log file                      |
+| `gaussian/fchk`                                          | `*.fchk`                      | Gaussian formatted checkpoint          |
+| `gaussian/md`                                            | Gaussian MD output            | Gaussian MD trajectory                 |
+| `gaussian/gjf`                                           | `*.gjf`                       | Gaussian input file                    |
+| `amber/md`                                               | Amber MD output               | Amber MD trajectory                    |
+| `gromacs/gro` / `gro`                                    | `*.gro`                       | GROMACS coordinate file                |
+| `pwmat/output` / `pwmat/movement` / `pwmat/mlmd`         | `REPORT`, `MOVEMENT`, `MLMD`  | PWmat output / movement / MLMD         |
+| `pwmat/final.config` / `pwmat/atom.config`               | `final.config`, `atom.config` | PWmat final/input structure            |
+| `fhi_aims/output` / `fhi_aims/md`                        | FHI-aims output/MD            | FHI-aims calculation or MD trajectory  |
+| `fhi_aims/scf`                                           | FHI-aims SCF output           | FHI-aims SCF                           |
+| `psi4/out`                                               | Psi4 output                   | Psi4 calculation output                |
+| `psi4/inp`                                               | Psi4 input                    | Psi4 input file                        |
+| `orca/spout`                                             | ORCA output                   | ORCA single-point output               |
+| `sqm/out`                                                | SQM output                    | SQM output                             |
+| `sqm/in`                                                 | SQM input                     | SQM input                              |
+| `openmx/md`                                              | OpenMX MD output              | OpenMX MD trajectory                   |
+| `n2p2`                                                   | n2p2 output                   | n2p2/NNPack output                     |
+| `dftbplus`                                               | DFTB+ output                  | DFTB+ detailed.xml                     |
+| `mol` / `mol_file`                                       | `*.mol`                       | MDL Molfile                            |
+| `sdf` / `sdf_file`                                       | `*.sdf`                       | MDL SDFile                             |
+| `ase/structure`                                          | Any ASE format                | ASE structure (single frame)           |
+| `ase/traj`                                               | Any ASE trajectory            | ASE trajectory (multi-frame)           |
+| `pymatgen/structure`                                     | pymatgen objects              | pymatgen Structure                     |
+| `pymatgen/molecule`                                      | pymatgen objects              | pymatgen Molecule                      |
+| `pymatgen/computedstructureentry`                        | pymatgen objects              | pymatgen ComputedStructureEntry        |
+| `lmdb`                                                   | LMDB dir                      | DeePMD-kit LMDB format                 |
+| `list`                                                   | List-format dir               | List of system directories             |
+| `3dmol`                                                  | 3Dmol format                  | 3Dmol.js format                        |
+
+You can omit `--fmt` and let dpdata infer the input format from the file name
+or content. For example, files named `POSCAR`, `OUTCAR`, or `*.xyz` are often
+recognized automatically. Use `--fmt` when the file name is ambiguous or
+auto-detection fails.
+
+### Single file
+
+```bash
+dpa-adapt data convert --input POSCAR --output ./npy
+dpaad data convert --input POSCAR --output ./npy
+
+dpa-adapt data convert --input OUTCAR --output ./npy --fmt vasp/outcar
+dpaad data convert --input OUTCAR --output ./npy --fmt vasp/outcar
+
+dpa-adapt data convert --input traj.xyz --output ./npy --fmt xyz
+dpaad data convert --input traj.xyz --output ./npy --fmt xyz
+```
+
+### Glob patterns
+
+When `--input` contains wildcards (`*`, `?`, `[`), conversion uses mirrored
+batch output:
+
+- **1 or more matches** → each matched file is converted into an output
+  directory that mirrors its path relative to the non-wildcard prefix.
+- **0 matches** → `FileNotFoundError`.
+- A `manifest.json` is written into the output root, recording converted and
+  skipped files.
+
+```bash
+# Glob output mirrors the input tree under ./npy_root
+dpa-adapt data convert --input "calcs/**/OUTCAR" --output ./npy_root --fmt vasp/outcar
+dpaad data convert --input "calcs/**/OUTCAR" --output ./npy_root --fmt vasp/outcar
+```
+
+For example, `calcs/run1/OUTCAR` is written as `npy_root/run1/OUTCAR/`.
+When `--strict` is set, the first conversion error fails immediately. Without
+it, errors are skipped and logged in the manifest.