OpenAI, following in the footsteps of startups like Runway and technology giants like Google and Meta, is delving into video generation.
OpenAI has presented sora, a GenAI model that creates videos from text. Given a brief (or detailed) description or still image, Sora can generate 1080p movie-like scenes with multiple characters, different types of motion, and background details, OpenAI claims.
Sora can also "zoom in" on existing video clips, doing his best to fill in missing details.
“Sora has a deep understanding of language, allowing him to accurately interpret cues and generate compelling characters that express vibrant emotions,” OpenAI writes in a blog post. “The model understands not only what the user has requested in the message, but also how those things exist in the physical world.”
Now, there's a lot of grandstanding on OpenAI's demo page about Sora; The above statement is an example. But carefully selected samples of what makes the model They look pretty impressive, at least compared to the other text-to-video technologies seen.
For starters, Sora can generate videos in a variety of styles (e.g. photorealistic, animated, black and white, etc.) up to one minute, much longer than most. text conversion models to video. And these videos remain reasonably consistent in the sense that they don't always succumb to what might be called "AI oddities," such as objects moving in physically impossible directions.
For example, this tour of an art gallery, all generated by Sora (ignore the graininess, compression of the video-GIF conversion tool):
Image credits: OpenAI
Or this animation of a flower:
Image credits: OpenAI
Some of Sora's videos with a humanoid subject (a robot standing in front of a cityscape, for example, or a person walking down a snowy path) have a video game quality, perhaps because there isn't much going on around him. Additionally, the AI's weirdness manages to sneak into many clips, like cars driving in one direction and then suddenly reversing or arms melting into a duvet cover.
Image credits: OpenAI
The examples shown are animated gif excerpts from longer videos. One complete video could be this western town
Video credits: OpenAI
OpenAI, for all its superlatives, acknowledges that the model is not perfect. Writes:
“Sora may have difficulty accurately simulating the physics of a complex scene and may not understand specific cases of cause and effect. For example, a person may take a bite of a cookie, but afterward, the cookie may not have the bite mark. The model may also confuse the spatial details of a message, for example, mixing left and right, and may have difficulty with accurate descriptions of events taking place over time, such as following a specific camera path.”
OpenAI is positioning Sora as a research preview, revealing little about what data was used to train the model (less than ~10,000 hours of “high-quality” video), and refraining from making Sora widely available. The basis for it is the potential for abuse; OpenAI correctly points out that bad actors could misuse a model like Sora in countless ways.
OpenAI says it is working with experts to test the model for misuse, abuse, and create tools to detect if the video was generated by Sora. The company also says that if it decides to make the model a public product, it will ensure that provenance metadata is included in the generated results.
“We will engage policy makers, educators and artists from around the world to understand their concerns and identify positive use cases for this new technology,” writes OpenAI. “Despite extensive research and testing, we cannot predict all beneficial ways people will use our technology, nor all the ways it will be abused. “That’s why we believe learning from real-world usage is a critical component to building and launching increasingly safer AI systems over time.”
