Mental Ray - Ambient Occlusion Shader

In 3dsmax7 Mental Ray 3.3 gained a new shader called "Ambient / Reflective Occlusion" what can this be used for? Well, traditionally within Mental Ray GI has been handled photon mapping and Final Gather. In this case a GI effect can be generated via a shader, and in this way this system opens up some flexibility in how you generate your GI. So what does this shader do? Quite simply it shoots out rays from the pixel that is being shaded in a hemisphere based on the normal of the pixel. It then traces these rays to the scene bounds or to a fixed distance defined in the shader and if it hits a piece of geometry the ray is flagged as occluded. Mental Ray then works out the average number of rays that were occluded and uses this to work out a blend colour between the BRIGHT and DARK values setup in your shader. On the right is a diagram of a typical pixel being shaded. In the example shown you can see that our shader has thrown out a total of 5 rays. If the BRIGHT colour were white, and the DARK colour Black, we would have two DARK rays and three LIGHT rays. So our pixel would be shaded as 40% occluded, or in other words a grey of 40% black would be used. Due to the way Monti-Carlo sampling works, 5 rays would not give us a very good average, pretty much in the same way as a sampling of 5 people in a survey would not give a very good average opinion of Mr man in the street. So for a better quality render you would expect to use 24+ samples.

Example 1 On the left you can see a real world render of the diagrams setup. In Example 1 I have used no direct light at all. Only the shading that has been supplied by the occlusion shader can be seen on the ground plane and the sphere. One of the issues that you can see is that it is very speckly. This is due to the low sampling rate. Another issue is that the occlusion is actual not very ambient, in fact its rather focussed due the spread value of 0.6. Lets do another render, but this time we'll increase the number of samples to 64. This is totally overkill but it will show what increasing samples will do more clearly. Example 2 In you can see that by increasing the sample count to 64 we have made the finished render much smoother, this is due to the averages for each pixel being much closer to the real value due to their more extensive sampling of their occlusion hemisphere. This though comes at a cost. Example 1 rendered in roughly 5 seconds flat., whereas Example 2 rendered in 15 seconds. Of course all those extra samples take up extra processor time, so we usually have to make a trade-off between speed and quality (a re-occurring theme in CG it seems). So, the other issue is that the occlusion is currently very focussed. Lets remedy that by changing the spread value to something closer to a real hemispherical value such as 180 degrees. Now, a tip to remember here is that the wider you make the spread angle, the more scope you give the sampler to cover. In our survey example its like expanding the survey from just one small town, to now cover an entire city. The larger area to sample means we have to use more samples to maintain the same relative end quality. Another thing to consider is that the closer you get to 180 degrees, the more chance there is that a surface will simply occlude itself due to surface bumps and displacements. This can lead to diminishing returns, so I would suggest a range of from 90 degrees (0.5) to 160 degrees (0.88) with a probably sweet spot somewhere around 135 degrees (0.75). So, lets try changing our spread value to 0.9 and our sample value to 32 and see what we get. Example 3 In Example 3 you can see that we now have a much more ambient occlusion and that the ugly occlusion shadow on our sphere from the cube has now been smoothed out into a much more general occlusion shadow. But, even with 32 samples the shading has become more speckly, so we can see that for really high standard renders we might need to go up to 48 or maybe 64 samples and beyond again. This is not a big issue for still renders, but could be a bigger issue for animations where this specklyness can often boil from frame to frame.

Using Mapped BRIGHT and DARK values One of the features of the ambient occlusion map is that you can use a bitmap to define what colour should be used for each ray sample. In our first example the rays could return only one of two colours, which were BLACK or WHITE. Now we will use a bitmap set to ENVIRONMENT to define what the BRIGHT colour will be for each ray sample. The big issue with this is that the colour value for each sample can be wildly different from the last as each ray fires out in a random director and so may hit the map in regions with wildly differing luma and chroma values. Lets have a look at this in action. Example 4 In Example 4 you can see that our occlusion colour has become much darker. This is because our brightness map which can be seen below Example 4, includes a wide variety of shades and colours that change rapidly as you look across the image. You can see that it is a sunset, so it has a lot of warm colours in it and a dark horizon with a bright sun above. It may be hard to see but at 32 samples we now have a much more speckled finish. So in our next test I am going to brighten the BRIGHT map slightly using its OUTPUT value to show up this specklyness much more obviously. By the way, this render took 25 seconds, so mapping the BRIGHT and DARK values does come at a cost (as ever). Example 5 In Example 5 we can now clearly see just how speckly the ambient occlusions contribution to the shading has become. Also you can see that the ambient shading across the surface of the sphere changes colour quite rapidly, which is not really very desirable for what is meant to be ambient light. So, how can we get around this? Well there are two methods that we can use. The first is the brute force method of simply cranking up the number of samples. If you put the value high enough sooner or later the speckle level will drop to an acceptable one. However, by that point your render times will not be acceptable. So, on to the second method..... Simply take the BRIGHT map into a paint package and blur it very heavily. This will reduce the wide variations in luma and chroma within it and will immediately make your renders look both more ambient in their colouration and will also reduce the speckle problem. Example 6 In Example 6 you can see that I have done this and that the speckle levels are generally reduced and also that the lighting has become more ambient in nature. You can see the extent to which I have had to blur the BRIGHT map in the swatch below the main image. However, the ground plane is still pretty speckled and this seems to be caused by the pole of the BRIGHT map. How could we reduce this, well the simplest method is to use a bright map that uses a cubic mapping method and in that way remove the actual poles from the map. Lets do one final render with a direct light turned on, the BRIGHT map reduced in strength and with more sampling before moving on to a breakdown of what other features this shader offers.

Occlusion Shader Parameters The occlusion shader has a number of other parameters that allow it to be used for purposes other than for ambient shading. If you look at the shaders name you can see that it can also be used in reflections. Well lets quickly run through the parameters and then go on to see how the shader can be used to enhance your reflections. Samples: As we have seen earlier this parameter controls how many sample rays are cast by the shader during the shading of each pixel. It should be remembered that this value can be multiplied by the Anti Aliasing setting to actually cast many more samples than you specify, so you should probably aim for a value that produces slightly speckled results with an AA setting of 1/4 and which will render smoothly at an AA setting of 4/16. Bright: This colour is used to determine the colour contribution of the ambient shader for each pixel, it is used in proportion with the following DARK colour. The proportion is based on the number of occluded verses un-occluded rays. This colour can be mapped with a bitmap as we explored earlier. Dark: As discussed above, this colour is used for the occluded rays. Again it can be mapped for better control or for a special effect. Spread: This value determines the cone within which sample rays are cast. 0 will produce a ray only along the surface normal, whilst 1.0 represents a spread angle of 180 degrees. Max distance: Normally Rays will be sent out to the scene bounds before returning as occluded or not. By setting this value the shader will only trace up to the distance defined before returning a value. If it reaches the value without being occluded it returns a non occluded value. Of course this may produce a slightly false result as if the ray had been traced to its full extent it may have been in occluded. This will generally lead to a slight reduction in overall occlusion proportionate to the distance specified, but it will yield faster render times as each ray does not have to be traced so far. Reflective: This tick-box turns on a feature that allows the shader to be used for "Reflection occlusion" rather than ambient occlusion. Reflection occlusion is used where a hybrid raytrace / scanline pipeline is desired and reflection maps are used. Of course we all know that reflection maps are MUCH faster than raytraced reflections, but their downfall is that they do not reflect near by objects realistically or indeed change their reflection when those objects move. However they are often idea for reflections of a distant environment.

With reflection occlusion near by objects can in fact block out the reflection map and leave the DARK colour. This is usually desirable to just letting the reflection map always show. Another trick is to fill the occluded colour with an actual raytraced reflection, this has the benefit of a mainly reflection mapped reflection, with only occluding nearby geometry being raytraced, saving precious CPU cycles. You can see an example of both of these approaches on the right in examples 8 and 9. Type: The shader has four different modes in which it can operate. Until now we have been working with the shader using its first mode, mode 0. Mode 0: This mode simply sends out rays and uses a proportion of the BRIGHT and DARK colours to shade the pixel depending on the number of occluded to un-occluded rays. Mode 1: This mode does a similar thing, but this time it will use the Environment map for the scene or the one defined in its Mental Ray shader to define the colour value it returns for an un-occluded ray. The shader also uses the DARK colour as a base and multiplies the returned colour by the BRIGHT value. This can allow you to control the environment to some extent, for instance you can use an output shader in the BRIGHT slot set to 2 to double the brightness of the returned colour, and you could use a dark blue in the DARK slot to raise the occluded colour from a dull black. You can see both of these tweaks in action in Example 10 on the left. Mode 2: In this mode the shader returns an RGB value based on the averaged un-occluded world space coordinate, with Red being the X Axis, Green being the Y Axis and Blue being the Z axis. This map can be used in a compositing package using a suitable warp filter to actually apply the occlusion colour during compositing. This is common practice in film pipelines where it is desirable to break down lighting into its constituent parts. This allows for some quite major changes in lighting to be carried out at the compositing stage, which can often save a complete rerender. Mode 3: This mode is the same as Mode 2, but in this case the colours represent camera rather than world space. Application So, just where can the Ambient/Reflection shader be used? We'll it a pretty flexible shader in that it can be used in most map slots. Of course it will not always make sense to use it in a slot but experimentation is the key here. Generally you will use the shader in the self illumination slot of a blinn shader (with self illumination colour turned on). And in this way you can produce an ambient lighting effect for just that shader. Another area it can be used in is in the reflection slot, and here it can be used in with its reflection option turned on as a mask between a reflection map and a raytraced reflection. You can also apply it as the Diffuse or Self Illumination colour on a shader when the you are using modes 2 and 3. Finally a neat trick is to use the shader as a light shader on an omni light. Simply instance the shader into the lights LIGHT SHADER map, and turn on only its "Ambient" contribution under the "Advanced Effects" rollout. This will effectively make the occlusion shader into a scene wide occlusion shader that will shade all scene objects. I hope this tutorial has been of use to you. Please let me know what you think about it and if there are any areas that are a little unclear still.