Time Warp, Temporal Reprojection, and Temporal Accumulation

(work in progress…)

While time warp, temporal reprojection, and temporal accumulation– all resue previous frames’ information, they try to solve different problems in global illumination algorithm. For example:

Time Warp

It is mostly targeted to the Extended Reality (XR) rendering to reduce the motion-to-photon latency. Sometimes also referred to as Asynchronous time warp. It uses the previously rendered image data. For example, the modern VR displays have generally 90 FPS or higher frame rate. That gives much smaller time budget (around 10 ms) to complete the rendering process, especially while the output is next to the eyes. Before the rendering process is complete, if the viewer moves their head or eye position, which has a very high chance to occur, the output image becomes outdated. However, instead of discarding the already rendered frame and start a new rendering, VR systems warp it to match the newest position. Consequently, it lowers the latency problem and expensive rendering cost. In addition, it may reduce the cybersickness and boost visual comfort. However, it has some drawbacks as well. For example, the occlusion (hidden surfaces) cannot be resolved and the warping could cause stretching near disocclusions.

TODO: Add figure

Temporal Reprojection

The target of temporal reprojection is to generate higher graphics fidelity output with smaller sample budget. Consequently, it reuses the previous frames’ samples to reduce Monte Carlo (MC) noise (aka. variance). Let’s look at the light transport algorithm:

\(L(x) = \int_{\Omega} f(\omega)L_i(\omega) \cos\theta d\omega\) Using the MC sampling, one frame estimates \(\hat{L}_t=\frac{1}{N}\sum_{i=1}^N L_i\) If the sample budget, $N=1$, the image is pretty noisy and normally unusable. However, there is a big catch. The latest Neural Network-based denoisers are pretty good to improve the graphics quality dramatically. However, that is another story. Back to the reprojection, instead of increasing the rays, we can reuse the previous frame’s samples. Suppose the previous frame, $t-1$ contains $L_{t-1}$ and the current frame, $t$ contains $L_t$, we can project previous pixels into the current frame, $L_{t-1}^{\rightarrow t}$ and BLEND it as follows

\(L_{\mathrm{final\,color}} = \alpha L_t + (1-\alpha)L_{t-1}^{\rightarrow t}\) where $0.0<\alpha<1.0$. For instance if $\alpha=0.5$, that means $50\%$ current frame’s samples and $50\%$ previous frame’s sample has been blended together to improve the graphics quality. Nevertheless, the blending technique and blending weight will have a significant role in this context.

TODO: Add figure

Another key factor is the Motion Vectors that impacts the final graphics quality. Motion vector fetch the frame buffer history, e.g., color, depth, motion, normal, variance, etc. In addition, the camera motion, object motion, and if there is any eye tracker available, gaze motion also align with the history.

Now, I have worked for long time in the perception-driven rendering (aka. foveated rendering) where the temporal reprojection even plays more valuable position as in the conventional foveated global illumination algorithm implementation, different image regions often receive different sample budget.

Temporal Accumulation

On compare to the Temporal Reprojection, temporal accumulation is more straight forward. There is no motion vector involved. Once the camera motion is constant, the temporal accumulation gathers the sampling data and converge gradually over time without increasing the sample count. Temporal accumulation does not require any blending operation.

TODO: add figure

The temporal reprojection and temporal accumulation are two commonly used techniques in real-time photorealistic rendering (global illumination) as each sample budget is costly. However, without proper reprojection, temporal accumulation alone could produce ghosting and smearing effect whenever the scene or viewpoint changes. On the other hand, without accumulation, reprojection alone does not reduce noise because it merely relocates previous information rather than integrating it.