Ray-tracing like Bokeh in Unity

developing a GPU compute shader to do stochastic samping focus blur

Published on April 5, 2021   unity · graphics

Focus blur is a technique in computer graphics to create the bokeh affect that occurs in photography when real lenses with (inevitably) non-zero sized apertures are used. The lens must be focused at a specific distance and any objects either in-front or behind that focus distance will be progressively blurred. The kind of bokeh that occurs depends upon the many tradeoffs that went into the design of the lens.

Focus blur as implemented in ray-tracers trades off performance for a smoother, less noisy result. The reason for this is that multiple rays are needed for each pixel to simulate the light captured by a non-zero sized (not pinhole) aperture. Usually the area of the aperture is stochastically (randomly) sampled and the more samples made the closer to a perfect capture you get. The results are then averaged. Hence the more samples the smoother the result. The effect of this is that out of focus areas get progressively more grainy the more out of focus they are.

In high performance rastering renderers, such as those in modern computer games, focus blur is a post-processing effect. After an image frame has been rendered the blur effect is applied to it. This requires a depth map to be captured as part of the rendering. The depth map is an image where each pixel contains the distance from the camera to the surface that provides the colour for that pixel. A depth map is quite cheap to produce as part of the modern rendering pipelines. The blur produced by this approach is very smooth.

Comparing Focus Blur Figure1 : ray-tracing focus blur (left) compared to post-processing focus blur (right)

I personally like the aesthetic of ray-traced focus blur much more. It has a more “analog” film grain like feel. When I moved to do more work with Unity I was dissapointed by the shiny smooth synthetic blur produced by it’s post-processing pipeline. That was the motivation for this project.

Unfortunately ray-tracing and raster pipelines are very different approaches to creating computer graphics. I couldn’t find any existing solutions or attempts at creating focus blur like a ray-tracer does. After a bit of head scratching this is the approach I came up with.

To get the performance needed we must use a compute shader. The compute shader runs in the GPU and has direct access to the image frame. The approach I developed still runs as a post-processing effect using the same image and depth map as the regular unity post-processing effect. It runs as two stages with an intermediate buffer connecting the stages.

The first stage calculates the “circle of confusion” for each pixel. The circle of confusion is a the radius of blur that the surface behind each pixel on the image will experience. The circle of confusion depends upon a number of factors including the focus distance, the aperture size and the lenses focal length (wide-angle to telephoto.)

Circle of Confusion Map Figure2 : Circle of Confusion map with yellow in front of focus and blue behind

The second stage takes the circle of confusion map along with the image. Each pixel is calculated in isolation and to all intents and purposes in parallel to maximum ability of the GPU. For each pixel we iterate number of times choosing a random offset from the target pixel. Two possibilities exist for valid interactions of the epixels. Firstly, if the circle of confusion of this pixel overlaps with the target pixel then this offset pixel will have some effect on the target pixel as long as it is in front. Secondly, if the circle of confusion of the target pixel overlaps with offset pixel then it will also be accumulated regardless of it being in front or behind. All light from relevant target pixels are weighted according to the kernel function and accumulated. The kernel function is used to weight the light received from a pixel according to it’s position in the circle of confusion. In an ideal bokeh all light within the circle of confusion is valued equally. Smooth bokeh occurs when the kernel has a gaussian falloff from the center. “Bad bokeh” occurs when light towards the edge of the circle of confusion has higher weighting. See Ken Rockwell for more information.

I have put the code for this compute shader up on GitLab along with the sample scene used in the youtube video.

Reference for shader parameters

  • Focus Distance - distance to the point of focus
  • Aperture - ratio of aperture (known as f-stop or f-number). The smaller the value is, the shallower the depth of field is.
  • Focal Length - the distance between the lens and the film. The larger the value is, the shallower the depth of field is
  • Kernel Shape - [Smooth, Circle, Outline] Smooth uses a gaussian blur. Circle simulates a traditional circular camera aperture. Outline creates a more sharp bokeh characteristic of mirror or overcorrected lenses.
  • Blur Radius - how large the maximum blur can be
  • Blur Radius Multipler - scales the actual blur to use more of the blur dynamic range set by blur radius
  • Sample Set Size - the number of random points to sample from. Large numbers will cause totally random looking noise, smaller values will start to look more like a pattern. This parameter doesn’t effect performance
  • Sample Count - the number of random samples tested for each pixel
  • Max Sample Count - many samples fall out of the range of the circle of confusion, once max sample count samples have fallen inside, and are used to effect the result, no more will be used. Keeping this value small keeps performance up.
  • Temporal Noise - When true the noise is varied on each frame, this looks more realistic and smoother but can cause a big increase in size and/or reduced quality of rendered video.

Read next

 

Caustics Simulation in Unity

- Part 4 in a series about creating a realistic water simulation and rendering

Caustics are ripples of light projected onto surfaces behind the refracting surface when light shines through. Caustics have the effect of concentrating light in very distinctive bands because when the refracting surface deforms it acts like a complex lens dynamically bending and focusing the light. Like the refraction, caustics are relatively straightforward to implement in ray-tracing though very computationally expensive. I could find no implementations of real caustics in Unity. The approaches I found use some kind of animation with static baked caustic textures that are warped in a wave-like fashion.
Published Feb 11, 2021  3 min read   unity · simulation · graphics
 

Refracting Water Surface in Unity

- Part 3 in a series about creating a realistic water simulation and rendering

Refraction is the physical phenomena that distorts views through the boundary between two transparent materials that have differing refractive indices. Common examples are looking into water or through a lens. This is very easy to achieve with ray-tracing because it closely simulates the physical process of light moving around a scene. Today (2021) ray-tracing can only be done offline or with limited complexity scenes on the latest hardware. Most games today use a technique called rasterization which is much faster but has many limitations.
Published Feb 11, 2021  3 min read   unity · simulation · graphics
 

Creating the Physics of Water Movement

- Part 2 in a series about creating a realistic water simulation and rendering

In this section we are going to look at simulating the movement of the water surface. We model the surface as a 2D grid of particles. These particles then form the basis of a mesh by rendering triangles between them. A particle is modelled as a point in space with a mass and velocity vector. Each particle is connected to each of its immediate eight neighbours. That is four to the left, right, in-front, and behind, then the diagonals.
Published Feb 11, 2021  3 min read   unity · simulation · graphics