KMT - Keras Motion Transfer: Animate Your Images with First-Order Motion

[abhaskumarsinha]

Hello Keras enthusiasts!

I'm excited to share my latest project, KMT: Keras Motion Transfer, an innovative deep learning pipeline that brings static images to life using motion dynamics from video! Inspired by Aliaksandr Siarohin's first-order motion model, this repo demonstrates how to animate images (or even create deep fakes) using Keras.

Repository Link: https://github.com/abhaskumarsinha/KMT

---

What It Does

• Keypoint Detection:
  Extracts (x, y) coordinates and local 2×2 Jacobians from both a grayscale source image (256×256×1) and the driving video frames. These keypoints serve as the foundation for transferring motion.

Motion Transfer Generator:

Using the keypoint information, the generator computes a first-order approximation (as detailed in the original motion transfer paper) to synthesize animated frames. This means each frame of the driving video influences the source image to produce a smooth, realistic motion effect.

Deep Learning & GAN Integration:

The model leverages a GAN training setup to refine both the keypoint detection and generation processes, ensuring realistic motion and appearance in the output frames.

---

Technical Highlights

• Backend Agnostic (Almost):
  The project is built entirely in Keras and is designed to work with TensorFlow, JAX, and PyTorch. Currently, it performs best on TensorFlow, while PyTorch and JAX encounter issues with tensor subtraction sizes.

• Grid Flow & Displacement Map Challenge:
  One of the most interesting hurdles was implementing grid flow maps. In PyTorch, functions like F.sample_grid() allow for seamless application of displacement maps using normalized coordinates (from -1 to +1). Keras, however, currently relies on keras.ops.image.map_coordinates(), which works with absolute values and lacks batch support.
  To tackle this, the model requires specifying the batch size during training (static graph computation) and switches to a batch size of 1 during inference.

---

Future Directions & Call for Collaboration

I'm planning to extend KMT by integrating techniques from the latest motion transfer research. If you have suggestions, improvements, or just want to chat about potential features, please drop a comment on the thread. Let's innovate together!

---

Examples:

Static Image Input

Animated output

---

Example Usage with Dummy Dataset