Accelerating Video Production and Customization with GliaCloud and NVIDIA Omniverse Libraries

The proliferation of generative AI video models, along with the new workflows these models have introduced, has significantly accelerated production efficiency and elevated output quality in creative and marketing technology industries.

GliaCloud, a Taipei-based AI startup in the NVIDIA Inception program focused on autonomous marketer content creation, has recently open sourced Omniverse-ComfyUI Bridge. This NVIDIA Omniverse Kit-based extension template streams visual data directly from NVIDIA Omniverse RTX Renderer into ComfyUI, covering a wide array of data types including RGB images, normal maps, depth maps, instance ID segmentation, and semantic segmentation. This capability facilitates the direct extraction and practical utilization of ground truth viewport data within advanced, AI-driven image processing workflows.

Video 1. Walkthrough of the Omniverse-ComfyUI Bridge workflow. Credit: GliaCloud

Video-to-video generation pipeline with NVIDIA Omniverse libraries

The idea for the Omniverse-ComfyUI Bridge first emerged from the GliaCloud team’s experience in experimentation with various video-to-video (v2v) generative AI models in ComfyUI. They recognized the increasing demand for detailed visual data, such as depth maps and segmentation outputs, in current generative AI workflows.

“We see significant potential in hybrid workflows combining advanced 3D rendering engines like NVIDIA Omniverse with generative video models,” said David Chen, GliaCloud CEO and co-founder. “The granularity of creative controls available through Omniverse and generative AI models makes a compelling case for emerging marketing requirements, providing unprecedented levels of visual polish and scalability in generated content.”

Video 2. The 3D to Gen AI workflow. Credit: GliaCloud

Within internal experiments, the GliaCloud team has observed noticeable improvements in the refinement and object control of generated outputs by leveraging different data rendered from Omniverse RTX Renderer as v2v model inputs.

This was possible using NVIDIA Omniverse Replicator, which greatly improved the overall outputs from a range of open source video generation models by enabling dynamic masks to be created based on segmentation inputs. The v2v models can now iterate through multiple passes to one single input with separate prompts reserved for dedicated regions, resulting in object-based style controls.

Omniverse-ComfyUI Bridge for faster and more precise iterations 

The Omniverse-ComfyUI Bridge is specifically designed for flexibility, scalability, and ease of integration into customized ComfyUI workflows. Its lightweight architecture ensures seamless adoption without substantial resource overhead, enabling users to scale their workflows effectively according to specific requirements.

Core functionalities of the Omniverse-ComfyUI Bridge include:

Single and multi-frame captures: Capturing individual frames for precise edits or sequences for animations and dynamic visual content. Custom node integration for ComfyUI: Direct streaming of Omniverse RTX Renderer data into custom ComfyUI nodes, facilitating straightforward and efficient workflow setups. Real-time data processing: Immediate visual feedback for faster experimentation, reducing the iteration time significantly, thereby enhancing productivity.

Future iterations of the Omniverse-ComfyUI Bridge are anticipated to include further enhancements such as advanced animation controls, integration with human pose estimation models, and additional features for edge detection. These developments are poised to offer even greater control, accuracy, and creative freedom in AI-driven visual workflows.

As generative AI continues to evolve, the importance of precise, high-quality data streams like those provided by Omniverse-ComfyUI Bridge will become increasingly critical. Going forward, the team at GliaCloud anticipates a drastic increase in the speed of iterations in workflow structures, as the community simultaneously develops newer models and adapts their workflow accordingly.

Get started

With the GliaCloud open source Omniverse-ComfyUI Bridge, you can now bring your own OpenUSD asset and integrate it into a v2v workflow with ComfyUI nodes such as NVIDIA Cosmos. This solution combines the photorealistic rendering capabilities of Omniverse RTX Renderer and the advantages of the ComfyUI modular user interface, enabling you to experiment with custom pipelines, rapidly and precisely.

GliaCloud is committed to embracing and contributing to the open source community, building toward democratized creativity. 

Start building your own Omniverse Kit-based application with the NVIDIA Omniverse Kit App Template and join the conversation in the NVIDIA Omniverse developer community.

Acknowledgments

The authors would like to acknowledge Billy Ho, a creative technologist at GliaCloud, and Ying Xiang Zhao, a DevOps engineer at GliaCloud, for their contributions to this post.