Nvidia expands AI for Media with lip sync and 8K upscaling
Nvidia's AI for Media toolkit for broadcasters adds frame generation for slow motion, 8K upscaling and SDR-to-HDR conversion. It also adds real-time lip sync to translated audio and speaker detection.
What it does
Nvidia AI for Media is a toolkit for broadcasters and streamers. New pieces include Video Frame Generation (2x or 4x frame rate for smooth slow motion), Video Super Resolution (upscaling up to 8K), and TrueHDR (real-time SDR-to-HDR), which can run in one pipeline.
For localization, LipSync adjusts mouth movements in real time to match translated audio, and Active Speaker Detection finds who is talking.
Results and limits
The coverage gives no measured results. 6x and 8x frame generation are still experimental, and lip sync supports French, German and Spanish.
Worth knowing
The coverage doesn't say how to get it or what it costs. The toolkit also includes 3D Body Pose, Studio Voice audio cleanup and a Synthetic Video Detector.
Try it yourself
First step: run [a short low-res clip] through Video Super Resolution, then Video Frame Generation at 2x, and compare the slow-motion replay with the original.