Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
dceadd7
Remove loop
DrLeucine Mar 17, 2025
1a7caba
Vectorise functions
DrLeucine Mar 17, 2025
5c04726
Clean up + remove redundant functions
DrLeucine Mar 18, 2025
0ef83ad
Simplify args + remove print statements in verbosity = 0
DrLeucine Mar 18, 2025
432e3cb
Remove unused args
DrLeucine Mar 18, 2025
c19d89f
Update versions
DrLeucine Mar 18, 2025
f7542ea
Update Github Workflow
DrLeucine Mar 18, 2025
eb27d1c
Add tqdm
DrLeucine Mar 18, 2025
a5978e0
Fix tests
DrLeucine Mar 18, 2025
3f274c5
Fix tests
DrLeucine Mar 18, 2025
cf559f0
Update test with new vectorised iswithin function
DrLeucine Mar 18, 2025
3230adc
Update README
DrLeucine Mar 18, 2025
f1a50b4
Update AMPAL
DrLeucine Mar 20, 2025
1ae096f
Add compression level
DrLeucine Mar 20, 2025
6f1c666
Update dependencies
DrLeucine Mar 20, 2025
45774bf
Fix numpy version to < 2.0
DrLeucine Mar 20, 2025
03377e8
Enforce HDF5 Typing
DrLeucine Mar 20, 2025
56ee079
Handle chunking and clean up memory
DrLeucine Apr 7, 2025
3e6ff50
Address I/O bottleneck leading to broken pipe with throttling and gc
DrLeucine Apr 8, 2025
77772f5
Add psutil
DrLeucine Apr 8, 2025
c168aa2
Fix memory fraction to 0.7 and reduce amino acid length
DrLeucine Apr 8, 2025
c348151
Fix memory fraction to 0.65 and reduce amino acid length
DrLeucine Apr 8, 2025
6f528b0
Avoid save_results bottleneck by writing to different files and then …
DrLeucine Apr 8, 2025
69ae5e5
Add better gc and error logging without mp.Manager().dict()
DrLeucine Apr 8, 2025
2aff43a
Add progress for errored structures
DrLeucine Apr 9, 2025
1535822
Add sanity checks + check if PDB is in the merged file already
DrLeucine Apr 9, 2025
c0984b7
Fix remerging logic
DrLeucine Apr 9, 2025
cfe1416
Add better checks if files are smaller than workers
DrLeucine Apr 9, 2025
0dacca7
Add recovery option + better deal with metadata
DrLeucine Apr 10, 2025
790f74e
Add further garbage collection and cache size for hdf5
DrLeucine Apr 11, 2025
8f17ebf
Add smaller chunking hdf5 + hd5 flush
DrLeucine Apr 11, 2025
90841a5
Avoid error after merging partials
DrLeucine Apr 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Avoid error after merging partials
  • Loading branch information
DrLeucine committed Apr 11, 2025
commit 90841a59af5dcd6b951f97e4af7f12a720ce6256
3 changes: 2 additions & 1 deletion src/aposteriori/data_prep/create_frame_data_set.py
Original file line number Diff line number Diff line change
Expand Up @@ -1299,7 +1299,8 @@ def process_paths(
if prior_partials:
merge_worker_hdf5_files(prior_partials, recovered_path, metadata=None, verbosity=verbosity)
for p in prior_partials:
p.unlink()
if p.exists():
p.unlink()

# Use recovered file to identify already-processed PDBs
existing_pdbs = set()
Expand Down