NemoClaw Knowledge Wiki

Fahd Mirza - Videorefer model running locally

4 min read

Fahd Mirza - Videorefer model running locally

https://www.youtube.com/watch?v=OZnfsJjLH5k This video provides an overview and local installation guide for Alibaba’s “VideoRefer Suite,” a new video model (released under Apache 2 license) focused on enhancing Large Language Models (LLMs) with spatial-temporal object understanding. Here’s a detailed summary: 1. Introduction to VideoRefer:

  • Purpose: VideoRefer aims to improve video LLMs by enabling them to understand and reason about specific objects within a video at a fine-grained level, tracking them throughout the entire sequence.
  • Differentiation: Unlike other models that only provide general video comprehension, VideoRefer can segment objects, track their movement, and answer detailed questions about them over time.

2. Model Architecture (Key Components):

  • VideoRefer introduces three main components: VideoRefer-700K Dataset: A large-scale, high-quality dataset containing object-level video instructions, crucial for training the model’s fine-grained understanding. VideoRefer Model: Built for precise spatial-temporal perception. Object Encoder: Designed to process both single-frame and multi-frame region inputs. Spatial Token Extractor: Generates precise object-level feature embeddings from any given frame using pixel-level masks, allowing flexible region inputs. Adaptive Temporal Token Merge (TTM) Module: Captures object context across multiple frames, creating an enriched representation of the object’s movement and changes over time. VideoRefer Bench: A benchmark specifically designed to evaluate these new spatial-temporal reasoning capabilities.
  • The detailed object-level embeddings are then interleaved with general image-level embeddings and language instructions, which are fed into the LLM to facilitate detailed, object-focused reasoning and dialogue.

3. Local Installation & Setup:

  • The speaker demonstrated installation on an Ubuntu 22.04 LTS system with an NVIDIA H100 GPU (80GB VRAM).
  • Steps: Create a Conda virtual environment (conda create -n ai python=3.11 -y && conda activate ai). Clone the VideoRefer GitHub repository (git clone https://github.com/DAMO-NLP-SG/VideoRefer.git && cd VideoRefer). Install prerequisites (pip install -r requirements.txt). This step downloads various packages, including the model safetensors. Launch the Gradio demo (python demo/app.py). The model downloads further checkpoints (approx. 4.87GB).
  • VRAM Usage: During inference, the 7B parameter model consumed over 32GB of VRAM on the H100 GPU.

4. Demonstration of Capabilities:

  • Gradio Interface: The demo provides “Image” and “Video” modes, with options for “Caption” and “QA” (Question Answering). Users can upload media and use drawing tools to mask objects.

  • Video Mode - Captioning: Example 1 (Dog): The speaker uploaded a video of a dog. After masking the dog in the first frame, the model accurately captioned: “A small dog is sitting on a bed, looking at the camera. The dog’s fur is short and appears to be a mix of white and brown. It has a black collar around its neck. The dog’s ears are perked up, and it seems to be attentively watching something in front of it. The bed is covered with a floral-patterned bedspread.” (The video itself wasn’t visible in the VNC session but played in VLC.) Example 2 (AI-generated Dog): Another video of an AI dog was used. Masking the dog again produced an accurate and detailed caption: “A golden retriever is standing in a grassy field, looking directly at the camera with its tongue hanging out. The dog’s fur is light brown and it has a happy expression on its face. Its ears are floppy and it appears to be panting slightly. The background is blurred, emphasizing the dog’s features. As the video progresses, the dog remains in the same position, maintaining its cheerful demeanor.”

  • Video Mode - Question Answering (QA): The speaker attempted to ask about the relationship between two masked objects (himself and a monitor/wall). The model produced errors and struggled to correctly identify or infer the relationship, indicating a limitation in complex spatial reasoning within videos during the demo.

  • Image Mode - Captioning: Example 1 (Man in Suit): An image of a man in a suit was uploaded. Masking the man’s white shirt generated an accurate caption: “A white dress shirt with a pointed collar and a visible button placket.”

  • Image Mode - Question Answering (QA): When asked “What is the object?” after masking people in the background, the model failed to describe the people and instead captioned the wall behind them. When asked the same question after masking a candle in another image (two women, crystal ball, candles), the model again failed to identify the masked object and instead captioned one of the women in the image, missing the specific region of interest.

5. Conclusion:

  • The speaker finds the VideoRefer model “pretty interesting” from Alibaba, particularly for its object-level captioning capabilities in both images and videos.
  • However, he notes that its Visual Question Answering (VQA) performance and precise object-level understanding in QA mode, especially when multiple objects or complex relationships are involved, needs significant improvement and further refinement from the developers.

Graph View