visionOS Support

[ddhvr]

Will it be supported on this platform? It works fine in the editor but it always shows the errors as you can see in the attached files. Tryna to build a demo in the AVP(Apple Vision Pro), but nothing showed up in the headset. It seems like it cannot load data from StreamingAssets path.

[ChristopherRemde]

Hey! Yeah the file reading, as well as the Pointcloud shader support both are problems that we need to address. But luckily, we're trying to build an apple vision pro app with this package as well at the moment, so support for the AVP should come in the next weeks!

[ddhvr]

I'm excited that you're porting it to visionOS. If you need to test anything on the AVP, just let me know. I have the device and am ready to help.

[ChristopherRemde]

Hey thank you very much for the offer! Luckily, we also have a device available.

Just one more information that would be helpful for me: Was the sequence that you tried to load a Pointcloud, mesh, or mesh & texture sequence?

[ddhvr]

I tried the mesh and texture one, but they were not synced when I used "play to device" on the simulator. You might encounter the same issue, so I hope there's a good solution.

The attached file should help you see the difference more clearly.

Screen.Recording.2024-05-15.at.3.02.39.PM.mov

[ChristopherRemde]

Wow nice volumetric video quality!
I can't quite tell, but the mesh/geometry seems to be playing fine, right? It's just an issue with the textures?

[deephog]

Hi Chrostopher,

I'm DD's colleague Yao. Thank you for sharing your amazing project and being so helpful on solving this AVP issue.

It doesn't seem obvious in this sequence since the actor is not moving much, but geometry-wise, the sequence playback is normal. The problem is, because it is a per-frame reconstructed sequence, each frame has its own set of uv and texture map. When mesh and texture are temporally misaligned, textures may get mapped onto wrong portion of the mesh, which is our primary suspicion of what is happening. Actually we managed to make frames temporally coherent as much as possible, the mapping only drastically change every about 10 frames, which is why you see the change of texture pattern happens less frequently than the actual frame rate. But I guess the temporal misalignment is larger than 10 frames that we cannot even see one frame with proper texture mapping.

[ChristopherRemde]

Hey Yao, thank you very much! Great to see the package being used :)

I designed the playback in such a way that this de-synchronizing should never be occuring, unless something goes really wrong. In this case it seems like some textures can't be loaded, which throws an exception and disturbs the loading order. This is probably a fault with the metal API. I'll update and test the package with the Metal API in mind and will implement support for the AVP in the next weeks. Please bear with me, for some time, but I'll let you know as soon as possible when this is done!

If you encounter any other issue or have any other feedback or critique please let me know!

[deephog]

Thank you for the insights! Looking forward to your implementation, and if you need us to do more tests, please let us know.

[ChristopherRemde]

Thanks, this is great! :)
Will test your sequence on the AVP when everything is ready!

Just to be sure, this is a public forum and anybody can download your sequence. My email is in the bio if you want to take it private!

[ddhvr]

Thanks, I've emailed it to you.

[ddhvr]

Hi Chrostopher,

How's the process going?

Best,
DD

[ChristopherRemde]

Hey DD

I'm working on it! But it'll take some more time to complete it, as I can only work on it for a small portion of my time. I think it should be ready in around 3-4 weeks!

[ddhvr]

Hi Chrostopher,

Any updates?

Best,
DD

[ChristopherRemde]

Hey DD,

This update has unfortunately taken a lot more effort than I thought. I needed to rewrite a larger part of the plugin, but in return the performance has also increased quite a bit! (10-20x times in my test cases), which will also help a lot to run this on mobile devices, like the AVP.

I've finished about 80% of the features so far, so should be up and running soon!

[ChristopherRemde]

Finally got to test the new version and your sequence running on the AVP and I think I could pin down the main issue with your sequence.

The plugin currently uses .DDS textures, which are not supported on mobile Chipsets/devices. This is the reason why you could see the geometry, but not the textures.

I'll implement support for ATSC texture compression for mobile devices now, which should enable the plugin to run textured sequences on iOS and Android!