extended redshift range to 1.0 with new models - #447
Open
djones1040 wants to merge 25 commits into
Open
Conversation
added 18 commits
July 9, 2026 01:06
…logic to spec-z and photo-z models
…ding but in how out-of-distribution S/N is recognized
…ransients no longer needed
…erformance for 0.015 < z < 0.2 compared to the z < 1 training
djones1040
marked this pull request as ready for review
July 23, 2026 08:44
Collaborator
|
@djones1040 The initial data file system is described in this doc. You do not strictly need to use the Upload the replacement and new files: mc cp sbipp/SBI_model.pt js-blast/blast-astro-data/init/data/sbipp/SBI_model.pt
mc cp sbi_training_sets/ js-blast/blast-astro-data/init/data/sbi_training_sets/hatp_x_y_midz.pkl
...Look at each file and its previous versions, for example: $ mc stat --versions js-blast/blast-astro-data/init/data/cutout_cdn/2010H/2MASS/2MASS_K.fits
Name : 2MASS_K.fits
Date : 2025-05-14 15:53:12 CDT
Size : 256 KiB
ETag : ff6b28e951156bb6aab58633585580a1
VersionID : rIpbZq8byXi39znb3qiRToruJyTtosS
Type : file
Metadata :
Content-Type: application/octet-stream
Name : 2MASS_K.fits
Date : 2024-09-11 15:32:44 CDT
Size : 256 KiB
ETag : ff6b28e951156bb6aab58633585580a1
VersionID : tx4CUO97A8-5Rt72LjJTYe3RbiKm..r
Type : file
Metadata :
Content-Type: image/fits Update and extend the |
Collaborator
Author
This all worked, the files appear on full_dev up, and the blast-data.json file is now updated. However, seeing bash run/blastctl ci up fail with the following (old or new blast-data.json file, fails either way): |
Collaborator
|
Your error looks like a problem with the local MinIO object store. Try running My test of commit 663aff5 is successful. After a full reset using the Then I used |
Collaborator
|
In 7c773b9 I added a script to help us compare the transient data generated by different versions. I fully deleted and reprocessed 2026dix as an example transient on blast.scimma.org which is running v1.13.1 at the moment. I compared that data to the 2026dix.663aff5.json output shared earlier. The logic of the comparison algorithm is tricky, so the resulting differences it output as shown below are slightly misleading, because in the v1.13.1 output, the global sedfittingresult is associated with the "2026dix_2MASS_H_global" aperture", while in the djones-frankenblast output, it is under "2026dix_PanSTARRS_i_global". Basically, these lines need to be improved so we can have a robust comparison script. In any case, @djones1040 please examine the differences and ensure they are acceptable. Did you notice that the fit looks terrible on commit 663aff5? 
|
djones1040
commented
Jul 25, 2026
| download_file_from_s3(sed_results_tmp_filepath, sed_results_object_key) | ||
| download_file_from_s3(sed_modeldata_tmp_filepath, sed_modeldata_object_key) | ||
| if sed_obj[0].software_version is None or Version(sed_obj[0].software_version) <= Version('1.13.1'): | ||
| offset_sed_model = True |
Collaborator
Author
There was a problem hiding this comment.
@manning-ncsa this is the reason for the terrible looking plots -- I removed the offset_sed_model thing for the current version and later but I haven't updated the version number yet, meaning that the tests will look bad visually until we update. There was probably a more sensible way to do this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #426. New SBI++ model that extends the blast redshift range to z = 1. Updated S/N model, revised treatment of missing/noisy bands that better replaces out-of-distribution errors, finds better training sample 'neighbors' similar to the data, and treats uncertainties better. Currently can be somewhat noisy at high redshift, but will improve with LSST data and subsequent updates. Disables local SED model as global one works fine for everything.
Adds low-redshift model for z < 0.015 to replace hacky method of handling this previously. New SED files are here. Docs are updated.
Will need to add the new SBI++ model to the S3 bucket before deploy and then add the new files to app/entrypoints/blast-data.json. Because this changes how SED fits need to be plotted, code assumes this PR will be included in any version after 1.13.1, but this can be modified. Unfortunately, I also needed to add an additional trained model, which will add ~1 GB to the docker environment.