Request for new extension call "3D video/gif"?

[donlinglok1]

https://github.com/vt-vl-lab/3d-photo-inpainting

I saw a method to convert 2D images to 3D videos, does anyone plans to work on it?

I find a colab and tested with this SD image (37KB) from image source

Below is the result. (607KB)

image-46-33_swing.mp4

my windows test code has been uploaded.
https://github.com/donlinglok/3d-photo-inpainting

for windows just use git bash to run

sh download.sh

2022/11/05 update test case:

depth image MiDas vs LeRes:

08265-1049740864_dolly-zoom-in.mp4

08265-1049740864_circle
08265-1049740864_swing
08265-1049740864_zoom-in

[ClashSAN]

I failed to get it working on locally linux on cpu

[donlinglok1]

Sorry I only tested on colab, no sure is that colab script can work on pc 👀

[donlinglok1]

I just tested it on my windows+git bash, the project can choose CPU but seems the GPU is required by a sub-library called 'BoostingMonocularDepth'.

So the whole process will take around 4GB vram of a short time.
(around 3 mins on GPU and 30s on CPU)
my GPU only has 6GBs, so I need to close the torch to run this.

Not sure if can it make as an extension or post-script.

my test code has been uploaded.
https://github.com/donlinglok/3d-photo-inpainting

just use git bash to run

sh download.sh

2022/11/05 update
screen cap:
https://user-images.githubusercontent.com/6138806/200113012-df0ba26a-dff5-4c20-984f-b7634e5c2bf9.png

[donlinglok1]

Hurrah ! I managed to get it to work again and extract both a depth map and a PLY model !

The solution: I had to change the offscreen_rendering parameter in the argument.yml file from True to False.

...

File "C:\Users\User.conda\envs\3DP\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\Users\User.conda\envs\3DP\lib\site-packages\moviepy\decorators.py", line 135, in use_clip_fps_by_default
return f(clip, *new_a, **new_kw)
File "C:\Users\User.conda\envs\3DP\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\Users\User.conda\envs\3DP\lib\site-packages\moviepy\decorators.py", line 22, in convert_masks_to_RGB
return f(clip, *a, **k)
File "C:\Users\User.conda\envs\3DP\lib\site-packages\moviepy\video\VideoClip.py", line 307, in write_videofile
logger=logger)
File "C:\Users\User.conda\envs\3DP\lib\site-packages\moviepy\video\io\ffmpeg_writer.py", line 216, in ffmpeg_write_video
ffmpeg_params=ffmpeg_params) as writer:
File "C:\Users\User.conda\envs\3DP\lib\site-packages\moviepy\video\io\ffmpeg_writer.py", line 88, in init
'-r', '%.02f' % fps,
TypeError: must be real number, not NoneType

I recall the original colab I copied had the offscreen_rendering configuration as well, but my test code doesn't seem to have it. 🤔It's strange because I've run the program a few times later and I've already committed all files... I'm sorry to hear that you're having trouble.

https://colab.research.google.com/drive/1qaXo3D1-ta_bs7Ug9Aihb4XO9XjjaFj7#scrollTo=bWxDrXIHF36H

According to your last error, something went wrong during the ffmpeg conversion, seems fps number is not a number?

here is some result I search about the fps:
https://github.com/search?q=repo%3Adonlinglok%2F3d-photo-inpainting%20fps&type=code

[AugmentedRealityCat]

I've got everything to work and more than once this time, so let me share the solutions I had to deploy to make this work. And just to give you more context, after running the thing successfully once (except for the movie part at the end) I was stuck like the last time and could not make it work with new images at all.

Git Bash VS Anaconda Prompt Command Line
The biggest problem was that, running everything from an Anaconda prompt (Command Line), if you put a jpg in the KenBurns/Input folder, it will not get copied as a PNG in the 3d-photo-inpainting-master\BoostingMonocularDepth\inputs folder. In the log the system also says this at the beginning:
'cp' is not recognized as an internal or external command,
and later, when I finally got everything to work again with other images, it would first add this other error message before the other:
'rm' is not recognized as an internal or external command,
I understand now that those are made for other interfaces like Git Bash, and that CP should copy, and RM should delete.
If I run the application from a Git Bash interface (instead of an Anaconda prompt) then the system manages to remove the old files, and copy the new jpg from 3d-photo-inpainting-master\KenBurns\Input and paste them as PNGs in the 3d-photo-inpainting-master\BoostingMonocularDepth\inputs folder. But then Git Bash gives error messages and it stops running.
But since it has copied the jpg as a png, now the next time you run the whole thing from the Command Prompt, it works ! But only for the image that was converted to PNG, and then, later, if you try another image the name discrepancy between that new image and the name of the old PNG that was converted by Git Bash, and this naming difference will prevent the process from completing.
The solution for me, once I understood what was happening, was to convert any jpg I want to use as a PNG and put it myself into the 3d-photo-inpainting-master\BoostingMonocularDepth\inputs while making sure both have the same name and only the 3 letters extension differs. After that, everything works all the time, but you also have to remove your old images before starting a new one. You already had identified that removing images was essential, and you were totally right. Now I think we now why this is like that.

Movie Generation bug
The reason why the mp4 were not generated at the end of the process has something to do with ffmpeg. Reinstalling and then upgrading ffmpeg did not solve the problem. What worked was reinstalling and upgrading moviepy with this command:
pip install moviepy --upgrade
After that the movie generation never stopped working. When I do a Conda List command, it says the installed version of moviepy is 1.0.3

Character encoding bug
I had another bug with this and it was after I reinstalled and upgraded ffmpeg (in the 3DP venv) while trying to solve the movie generation bug. Or it might be after I upgraded CUDA (bad idea - when I tried to remove it after, the process was soo loooong, like it would have taken 48 hours of something to fix the dependencies - anyways I did something wrong.
Anyways, I looked up another error message I was getting and basically the problem was that the system now expected the command to include in which character encoding (UT8, etc.) the instructions it is receiving are compatible with. This error was coming from the .conda\envs\3DP\Lib_sitebuiltins.py file, so I found the faulty line (line 50) and changed it like this
with open(filename, "r", encoding='utf-8') as fp: instead of with open(filename, "r") as fp: which was missing the encoding parameter.
I found the solution here: https://bobbyhadz.com/blog/python-unicodedecodeerror-charmap-codec-cant-decode-byte

BoostingMonocularDepth requirements
I checked the prerequisites in the readme for BoostingMonocularDepth and I found that some things were missing, so I installed them. It did not solve my problem immediately, but it might have helped somehow (I just don't have the knowledge to discern that). Here are the packages that seemed to require an install or reinstall:

cudnnenv
gradio
gdown

Thing I should not have done: upgrade CUDA
Since there was a CUDA error when I tried to run the thing through Git Bash instead of the Conda Command Prompt, and after reading comments online, I thought that maybe I could solve the problem simply by upgrading CUDA. And I don't think I should have done that.

But NOW IT WORKS ALL THE TIME ! Thanks again for sharing this project. It was really hard for me to get this running, but it's because I am not a programmer at all so I have much to learn. One thing is for sure: I am not afraid of code anymore.

[donlinglok1]

I've got everything to work and more than once this time, so let me share the solutions I had to deploy to make this work. And just to give you more context, after running the thing successfully once (except for the movie part at the end) I was stuck like the last time and could not make it work with new images at all.
...
But NOW IT WORKS ALL THE TIME ! Thanks again for sharing this project. It was really hard for me to get this running, but it's because I am not a programmer at all so I have much to learn. One thing is for sure: I am not afraid of code anymore.

🥂Congratulations, now you can call yourself a programmer!

environment setup hell /finding problems in the log /encoding problem is something we've all experienced.

[AugmentedRealityCat]

Just to make things easier for other people who I'm sure will try this, here is what I believe you should look at first:

1. I think if we can replace the RM and CP commands with new commands like DELETE and COPY that would work with the Command Prompt instead of Bash, a lot of problems would go away. I don't know the proper commands or how to use them though, it's just the general idea to automate this part that I currently do by hand (converting JPG to PNG and put it in proper folder)

2. I think changing the offscreen_rendering parameter in the argument.yml file from True to False was important.

3. I am convinced the pip install moviepy --upgrade command solved the main problem that was preventing movie generation from working.

Thanks again.

[donlinglok1]

Just to make things easier for other people who I'm sure will try this, here is what I believe you should look at first:

1. I think if we can replace the RM and CP commands with new commands like DELETE and COPY that would work with the Command Prompt instead of Bash, a lot of problems would go away. I don't know the proper commands or how to use them though, it's just the general idea to automate this part that I currently do by hand (converting JPG to PNG and put it in proper folder)
2. I think changing the offscreen_rendering parameter in the argument.yml file from True to False was important.
3. I am convinced the pip install moviepy --upgrade command solved the main problem that was preventing movie generation from working.

Thanks again.

hey, I have updated the test code as your test result, thank you for spending lots of time and sharing the finding!

https://github.com/donlinglok/3d-photo-inpainting/commits/master

[Tollanador]

Deforum extension has 3D depth mapping, and I think can do this and more. Although it's not as simple to use, totally worth the initial learning curve though.
Or have I misunderstood what this technique is about?

[donlinglok1]

Thanks! I will try on Deforum too!

[ClashSAN]

@Tollanador have you been able to create something like this with deforum? Perhaps all the examples I've seen are not using this 3D depth mapping..

[barleyj21]

I've made it working on my win10 pc a couple of month ago. Should still have a working anaconda environment for it - I remember it was really hard to find versions of packages that would work together + some fixes to code i've looked up somewhere

[AugmentedRealityCat]

I was also able to make it work, but only once. I've been trying to reinstall properly since to identify where the problem was coming from, but I haven't had any more success since. What's encouraging is that now I know it can work - I just have to figure out how !
Thanks for confirming you had it running a couple of months ago as well, one more proof that there's light at the end of the tunnel (and that it is not a train coming at us !)

[donlinglok1]

#5056

Extraltodeus/depthmap2mask#11 (comment)

@AugmentedRealityCat LeRes depth image here, I cannot use the windows so I gen this using colab. If you want to try in windows you can check out my change here:

200113072-01c99308-03c0-44cb-b163-9b5de3357533_zoom-in.mp4

LeRes output:

inverted greyscale:

[AugmentedRealityCat]

It looks like that LeRes thing is actually working for you. This is GREAT ! I'll test your fix as soon as it's ready.

[donlinglok1]

It looks like that LeRes thing is actually working for you. This is GREAT ! I'll test your fix as soon as it's ready.

I see the videos you upload here

This is weird that we get different results using Windows and colab😅... anyway I will make a patch to avoid that.

[donlinglok1]

It looks like that LeRes thing is actually working for you. This is GREAT ! I'll test your fix as soon as it's ready.

I see the problem, you may not use the latest LeRes version code.
If you using git, you need the below command

git clone https://github.com/donlinglok/3d-photo-inpainting
cd 3d-photo-inpainting
git checkout LeRes 

If you download the zip on gitHub, make sure using this link (which selected the branch LeRes:
https://github.com/donlinglok/3d-photo-inpainting/tree/LeRes

[donlinglok1]

@AugmentedRealityCat I have implemented and updated the result above. Please use the latest code at https://github.com/donlinglok/3d-photo-inpainting/tree/LeRes

My result is 3d-photo-inpainting+LeRes is not quite good for anime images...

[AugmentedRealityCat]

I was just testing and I got the same result as before - but I had not seen your note above, and now I know why it was that way. Anyways I have pulled the proper branch now and I am going to make another test.

[AugmentedRealityCat]

First run did not work - something about a res101.pth file missing. I found a link to it on a boostingmonoculardepth colab and I'll download and put it where it seems to be going /BoostingMonocularDepth/res101.pth and I'll make another test.

[AugmentedRealityCat]

This is it TOTAL SUCCESS ! Thank you once again for this wonderful adventure.

The quality of the depthmap is outstanding for something extracted from a single picture.

[bugmelone]

This looks very impressive! Glad to see how fast new opportunities are opening up for us. Thank you so much!