Perceptual Graphics Quality Evaluation Metrics, Part 3

(work in progress…)

Previous posts included different perceptual quality metrics (Part 1, Part 2). In this post, I will continue with some other seminal perceptual graphics evaluation metrics, often works for video evaluation:

• Intel’s CGVQM
• Netflix’s VMAF
• ERQA

Intel’s CGVQM-D

CGVQM evaluates videos. It is especially handy for blurry, noisy, and aliased artifacts. Nonetheless, for a valid evaluation, each reference and test pair should contain the same camera motion, frame count, frame rate, resolution, and temporal alignment. It is designed specifically to capture both spatial and temporal artifacts from different graphics rendering techniques. There are two variations:

• CGVQM-5: Uses five layers of a 3D ResNet-18 model for maximum accuracy matching human testers.
• CGVQM-2: A lightweight, two-layer variant that is roughly 27% faster, ideal for fast feedback in continuous integration (CI) or build pipelines.

Use

• git clone --recursive https://github.com/IntelLabs/cgvqm.git
• pip install numpy scipy av
• However CGVQM is linked to (see requirements.txt).

  numpy==2.3.0
  pandas==2.3.0
  torch==2.8.0
  torchvision==0.23.0
  av==14.4.0
  

• Therefore, a virtual environment would be required.
  • first, & .\.venv\Scripts\python.exe -m pip install --upgrade pip
  • then, & .\.venv\Scripts\python.exe -m pip install -r .\requirements.txt (av 14.2.0), original was 14.4.0
  • finally, verify the virtual environment & .\.venv\Scripts\python.exe -c "import av, torch, torchvision; from torchvision.io.video import read_video, write_video; print('PyAV:', av.__version__); print('Torch:', torch.__version__); print('TorchVision:', torchvision.__version__); print('CUDA:', torch.cuda.is_available())". It should return

    PyAV: 14.2.0
    Torch: 2.8.0+cu128
    TorchVision: 0.23.0+cu128
    CUDA: True
    

  • Now, run & .\.venv\Scripts\python.exe -u .\cgvqm.py
  • From inside the code, the dist_vid_path, ref_vid_path, errmap_path, cgvqm_type = CGVQM_TYPE.CGVQM_5 should be adjusted. The CGVQM_5 is recommened over CGVQM_2 if computational power is not an issue.

  Uniform Rendering

  Foveated Rendering

  CGVQM Error Map (score 70.53/100)

N.B. for evaluating large size videos, the CGVQM might have memory issue.

Netflix’s VMAF

git