The result of depth_res is not good

[haojieh]

I use the camera realsense D435i to capture the depth_image and color_image from example of IntelRealSense Align & Background Removal.And then,I put depth_image and color_image in to def inference of class Inferencer but I think the result is not good.Can you tell me where I might be doing wrong?

[Galaxies99]

Hello, thank you for your interest in our work!

I would suggest to check the following points:

• The Background Removal may removes the background which has critical information (e.g., the background depth since the transparent objects lie on the background, causing the depths relevance). In our tests, we doesn't use the Background Removal function of the camera; instead, we set a certain workspace and clip the RGBD according to the workspace.
• Are you using the newest version of code and checkpoint? We have updated the code & checkpoint several days ago and fix the problem of the unexpected shift of depth map.
• Remember to scale the depth (for D435i, I think the scale coefficient is 1000) before sending depth to the network;
• Our model is applicable to depth of range [0.1, 1.5] meters, check whether the depth satisfies the conditions, since we have clip the depth in the preprocessing stage.

[haojieh]

Thank you for your response!

1. For 1 point I don't use Background Removal.
2. For 2 point, today I download again.
3. For 3 point,I don't scale the depth previously. I follow your suggestion to scale the depth but the result still same.
4. From 3. I successfully scale the original depth to real depth that values in array are real distance. I think that I am test in the range [0.1, 1.5] because the values of real distance in work space usually 0.4~0.6 meters. But, the values in refine_depth are still same and the values are not real distance.

[haojieh]

The picture in left of Align Example is depth_image after scaling, the picture in right of Align Example is refine_depth.Also have a problem is the little black dot which you can see it on the refine_depth.

[Galaxies99]

Could you provide the RGBD image you used here? I may try to reproduce the problem under my environments and investigate the reasons behind the problem.

Also, I notice that there are very-close foreground items (which might be camera supporters?) which affects the depth images (see the black holes in the depth image, i.e., the left sub-figure of the left figure). We do not test the performance under this circumstance so far (for grasping experiments, we use in-hand D435 camera which directly capture the workspace), so the problem might be related to this, but I think it would be more helpful if you could provide the test images for me to reproduce your problem and check what causes it.

By the way, although not related to the issue, it might be better to use the "Low Ambitient Light" mode or "Low Range" mode of the RealSense camera if there are very-close foreground items. The mode can be changed in realsense-viewer.

[haojieh]

OK, I will provide the rgb and depth image here. What kind of image do you need? jpg or png?
Also, the close foreground items are gripper. I fix the camera on the robot arm, so you think that black holes are caused by gripper?By the way, the black holes random move on the refine_depth image.

I list the following questions:
1.I think your network has a way to measure the depth of transparent objects but at the moment even scenes without transparent objects are wrong depth information.
2.Why reason cause the black holes?
If you need more information about my environment please tell me.
Thank you!

[Galaxies99]

OK, let us continue the discussions with E-mail.

[mtbui2010]

Hi Fang.
I appreciate your work.
I have the same problem with the above issue. Did you find a solution? Can you share the solution?

[Galaxies99]

Hi Fang. I appreciate your work. I have the same problem with the above issue. Did you find a solution? Can you share the solution?

Hi @mtbui2010, thank you for your interest.

After communications with @haojieh, I am working on this issue right now. If there is any progress, I will put the solution immediately here, and in the repo.

If you want to use the model now, I would suggest to use the stable version and try to solve the "shift" phenomenon in an engineering way. As stated in README, for a stable checkpoint that corresponds to the paper, please see this version of our repository. Our supplementary videos are recorded using that version of model. Shifting problems in this version may be solved by calculating the difference of the average depth before and after refining, and then subtract the difference from the refining depths.

[Galaxies99]

The problem has been greatly reduced in the new version of code and checkpoint.

In the new version, the shifting problem is slight, i.e., only ~2cm, which can be solved using engineering approaches, such as subtract 1cm for all depth map (this can be varied under different circumstances, but under our test the shifting problem is slight compared to the original problem).

In the new version, you can also use depth_coefficient to control the filling of invalid pixels in the inference stage, see L55 in sample_inference.py. If you find empty hole/black hole in the depth map, I would suggest to decrease the value of depth_coefficient to solve it.

Thanks for reporting issues!

[ZhiyangZhou24]

Hi Fang.
I appreciate your work.I have the same problem with the above issue(shifting&hole problems). And I tried your latest uploaded code and Checkpoint today, but I encountered the following problem, is it because the Checkpoint I am using is not the latest? Can you upload the latest Checkpoint? Thanks!
Traceback (most recent call last): File "/home/lab/TransCG/sample_inference.py", line 47, in <module> inferencer = Inferencer() File "/home/lab/zzy/TransCG/inference.py", line 61, in __init__ self.model.load_state_dict(checkpoint['model_state_dict']) File "/home/lab/anaconda3/envs/transcg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1605, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for DFNet: size mismatch for final.3.weight: copying a param with shape torch.Size([1, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 64, 1, 1]).

[Galaxies99]

Hi Fang. I appreciate your work.I have the same problem with the above issue(shifting&hole problems). And I tried your latest uploaded code and Checkpoint today, but I encountered the following problem, is it because the Checkpoint I am using is not the latest? Can you upload the latest Checkpoint? Thanks! Traceback (most recent call last): File "/home/lab/TransCG/sample_inference.py", line 47, in <module> inferencer = Inferencer() File "/home/lab/zzy/TransCG/inference.py", line 61, in __init__ self.model.load_state_dict(checkpoint['model_state_dict']) File "/home/lab/anaconda3/envs/transcg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1605, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for DFNet: size mismatch for final.3.weight: copying a param with shape torch.Size([1, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 64, 1, 1]).

Hi @ZhiyangZhou24, the checkpoint is the latest, and you also need to update the source code. I slightly change the last layer of the model (from kernel = 3 to kernel = 1) for performance improvement.

[Galaxies99]

Solved during e-mail conversation with @haojieh, and the solution has been released both in repo and in the previous comments. Thus, mark this issue as closed.

For further questions, please add new issues and describe your questions.