Using RIFE for just optical flow

[vibhaa]

Hi, I'm interested in just getting the optical flow between frame 1 and frame 2 (and not necessarily producing a frame 1.5 in between). Is there a way to combine the two intermediate flows produced by your model to obtain this? I'm having a hard time interpreting the intermediate flows produced by the model at different scales, and am not sure how best to proceed.

Specifically, I visualize the grid supplied to grid_sample to produce each warped image, and am unable to make sense of the output (attached flow[:, :2] and flow[:, 2:4] passed to warp, and g visualized with flow_vis). Should I be subtracting identity? If I do that, I obtain a grid that looks like the identity deformation. Any help is greatly appreciated.

[hzwer]

I add a simple interface, I recommend to use RIFEm parameters => https://github.com/megvii-research/ECCV2022-RIFE#evaluation
And I wrote an example, not sure how it works

import torch
from torch.nn import functional as F
from model.RIFE import Model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = Model(arbitrary=True)
model.load_model('RIFE_m_train_log')
model.eval()
model.device()
...
I0 = torch.tensor(I0).to(device)
I1 = torch.tensor(I1).to(device)
with torch.no_grad():
            flow = model.flownet(I0, I1,timestep=1.0, returnflow=True)[:, :2] # will get flow1->0

[vibhaa]

Thank you for your help! I was able to get this working. If I understand this correctly, if I want flow 0->1, I repeat the same call with slightly different parameters?

flow = model.flownet(I0, I1, timestep=0.0, returnflow=True)[:, 2:4] # will get flow0->1

[hzwer]

Yes, I think you get it. I may add this issue to our README because it seems to be a useful function.

[hyc9]

Besides, I find the example code can't work, maybe the codebase has been modified? It can work like :
flow = model.inference(I0, I1,timestep=1.0)[:, :2]