Google demonstrated the Lumiere spatio-temporal diffusion model. The new AI tool can create amazingly realistic videos up to five seconds long. The neural network animates still images or only parts of them in response to natural language text prompts. Unlike its predecessors, Lumiere builds the entire length of the video at once, rather than generating the first and last frame, trying to guess what happens between them. The development is a research project, and it is not yet known whether it will be available for widespread use.
Lumiere can copy the style of an image and then use that style to create a series of videos on other topics. A neural network can take a user’s original video and turn it into Lego, origami, or flowers.
Judging by the demonstrations, Lumiere has the most advanced drawing capabilities. You can close a part of the image, and Lumiere will automatically fill in that area – so organically that it’s impossible to see the artificial intelligence intervention.
The research team claims that U-net’s spatio-temporal architecture builds the entire length of the video at once, in one pass. This distinguishes the neural network from previous models, which often generated an initial and final frame and then tried to guess what would happen between them.
For now, this is just a research project. Therefore, Google does not have to aggressively neutralize the system in order to respect copyrights, privacy and security, as well as to prevent hate speech and nudity. This process invariably leads to a decrease in the quality of the result in generative models.