RIFE vs Normal FFMPEG 15 - 60

[gateway]

So here is a video of both results of a video that was done at 15 fps, upscaled and then processed via normal ffmpeg command and RIFE.

https://youtu.be/H-LjB3fP6y0

I have noticed some weird wavy results on some of the videos .. Anyone else seeing this, its harder to see in this video but it was just what I had on hand.

[rkhamilton]

I'm not sure I'm seeing what you mean by wavy. One thing I'd suggest is you process videos with and without upscaling. I don't know how Real-ESRGAN works, but it could be introducing frame-to-frame differences if it makes different decisions about upscaling.

The video looks very cool to me btw.

[gateway]

Here is another one.. https://youtu.be/jUJXbOl9COc

btw I have been trying to figure out the best way to slowly morph from the original image to iteration x .. some of the tests I do its so damn quick you cant see the what i'll call evolution of the transformation.

Been trying like 15 fps, 500 iterations, but want it to save every 1 or so...

[rkhamilton]

Yeah the ripply stuff where the video never fully stabilizes is definitely a thing. Try using the adaptive learning rate feature.

config = VQGAN_CLIP_Config()
config.adaptiveLR = True
# rest of code here

What is happening is the optimizer is hitting a minimum (best image it can find) and kind of bouncing around that best point. It's taking big steps that sort of stride back and forth over the best spot. There's a variety of ways to make the stepsize change, and I implemented on that makes it take smaller steps when the image stops improving, and it will eventually stop changing at all.

I haven't tested adaptive learning rate with changing prompts over time, so I don't really know what it will do. It's worth a shot. It may work fine and reset after changing prompts, but it may just stall out totally.

If you want to try it, let me know how it goes. I've been working on evaluating different cut method algorithms, so my PC has been tied up for days generating test images for me to compare the methods.

[gateway]

ill give it a shot.. that last video deff shows it, some are worse than others.. thanks for the info on adaptiveLR, I havent personally looked into RIFE much yet for config/options etc..