Skip to content

Extract contextual frames, perform IC with pybasic, and store target + IC frame in LanceDB#47

Merged
d33bs merged 8 commits into
WayScience:mainfrom
d33bs:ic_for_images
Jul 17, 2024
Merged

Extract contextual frames, perform IC with pybasic, and store target + IC frame in LanceDB#47
d33bs merged 8 commits into
WayScience:mainfrom
d33bs:ic_for_images

Conversation

@d33bs

@d33bs d33bs commented Jun 4, 2024

Copy link
Copy Markdown
Member

This PR addresses #43 by extracting contextual frames, perform IC with pybasic, and store target + IC frame in LanceDB. This work focuses on a limited set of data for development purposes, keeping concurrency in mind throughout to help prepare for #44 . Generally I also tried to be storage space conscious, removing files as they are no longer needed. I removed the earlier array storage work as it felt more appropriate to store image blobs instead (one could always extract the arrays afterwards).

To use pybasic in these changes I forked the repo under WayScience, created a branch from the submodule commit hash referenced in IDR_Stream, and modified the setup.py to be installable as a package (the exact reference may be found within the pyproject.toml file).

I'm attaching an example dataset created from this work here as a reference: example.zip.

Closes #43

@d33bs
d33bs requested review from gwaybio and jenna-tomkinson June 4, 2024 20:54

@jenna-tomkinson jenna-tomkinson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am going to leave a comment here as first reviewer since Greg is also assigned to review. I left some clarifying questions where I was a bit lost with what was happening. Feel free to reach out in case you have questions about my questions 😄 Otherwise, I am excited to see what a LanceDB file will look like!

Comment thread 5.data_packaging/constants.py
Comment thread 5.data_packaging/create_lancedb.py
Comment thread 5.data_packaging/gather_images.py Outdated
Comment thread 5.data_packaging/gather_images.py
Comment thread 5.data_packaging/gather_images.py
Comment thread 5.data_packaging/gather_images.py
Comment thread 5.data_packaging/gather_images.py Outdated
Comment thread 5.data_packaging/gather_images.py Outdated
Comment thread 5.data_packaging/gather_images.py
Comment thread 5.data_packaging/gather_images.py
@d33bs
d33bs requested a review from jenna-tomkinson June 10, 2024 13:58
@d33bs

d33bs commented Jun 10, 2024

Copy link
Copy Markdown
Member Author

Thanks @jenna-tomkinson for the review! There were a couple of comments which I left unresolved which I wanted to get your opinion on so I re-requested your review. Please don't hesitate to let me know if you have any questions / concerns.

@jenna-tomkinson jenna-tomkinson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR LGTM! I didn't see anything glaring in the code. This is very exciting work!

@d33bs

d33bs commented Jul 17, 2024

Copy link
Copy Markdown
Member Author

Thanks @jenna-tomkinson ! Merging this in.

@d33bs
d33bs merged commit b793d38 into WayScience:main Jul 17, 2024
@d33bs
d33bs deleted the ic_for_images branch July 17, 2024 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add IC image integration capabilities for LanceDB dataset

2 participants