CFN

WebGL + Rust: Basic Water Tutorial

Sun, 06 Jan 2019 18:00:00 +0000

In this tutorial we’ll discuss the ideas and concepts behind rendering water and then talk through some demo code.

By the end of this tutorial you should walk away with a good sense of how water is rendered as well as a reference implementation to serve as a jumping off point should you decide to dive into more complex water rendering techniques.

The full source code for the demo can be found on GitHub.

Using Rust + WebGL + WebAssembly

I asked and you decided!

Our past WebGL tutorials have all used JavaScript, but this time around we’ll be using Rust.

It's been a MINUTE (14 months) but I've freed up some time and I'm bringing back my step-by-step interactive WebGL tutorials !!

First on the list is rendering water using reflection, refraction, du/dv maps and all that good good !!

Which language would you prefer that I use ?
— C Francis Nwafili (@chinedufn) November 30, 2018

If you’ve never used Rust before don’t worry. All of the core concepts behind rendering water hold true no matter what language you’re using. We’re using WebGL but the same concepts hold for OpenGL or any other graphics API.

What goes into rendering water?

Our basic water renderer boils down to combining a few textures and tweaking numbers until we like what we see.

Click or touch and drag to move the camera. Use your mouse wheel to zoom in and out of the scene.

We first render the scene without the water and store this in a framebuffer that we call our refraction framebuffer. A framebuffer is a buffer that contains all of the data that we need to render a frame.

The refraction framebuffer tells us everything that is under the water.

In the real world, light from under the water is refracted, or deflected as it passes from the water into the air and to your eye. Hence the name refraction buffer.

At the top left of the interactive scene above you should see a rendering of the color texture that is attached to our refraction framebuffer.

Next we take the camera’s y position, multiply it by -1 while ensuring that we’re still looking at the same point and then re-render the scene. You can see an illustration of this at the top right of the demo above. This is our reflection framebuffer.

As you might guess from the name this is everything that could be reflected from the water’s surface and into the camera.

When rendering the water we blend the refraction and reflection textures onto the water quads surface. If you disable the Use Refraction and Use Reflection checkboxes in the demo you’ll see that in action.

The way that the water appear to move comes from slightly altering the point on the refraction and reflection textures that we sample from every frame.

These slight offsets comes from sampling a du/dv map, which is just a texture that encodes different x and y offsets.

Our du/dv map encodes small x/y offsets that will be used to distort our water’s surface textures, simulating waves.

We use a corresponding normal map to make the water appear to not be flat and for our specular highlights.

Our normal map gives our water quad more varried surface normals, allowing for better specular light simulation.

There are more advanced techniques that aid in simulating water with height and normals and waves such as distorting the actual quad to simulate waves with real heights.

Now that you’re a little more familiar with rendering water we’ll talk through the key pieces of the code that powers the water demo.

A good way to learn is to first follow along as we talk through the major pieces of the code, then clone the source code, run it locally and play around with the code and see your changes in the browser.

Walking through the demo code

#!/bin/bash

# File: build.sh

cd $(dirname $0)

# ./build.sh
if [ -z "$RELEASE"  ]; then
  # --------------------------------------------------
  # DEVELOPMENT BUILD
  # --------------------------------------------------

  RUST_BACKTRACE=1 cargo build --target wasm32-unknown-unknown
  wasm-bindgen ./target/wasm32-unknown-unknown/debug/webgl_water_tutorial.wasm --out-dir . --no-typescript --no-modules

# RELEASE=1 ./build.sh
else

  # --------------------------------------------------
  # RELEASE BUILD
  # --------------------------------------------------

  cargo build --target wasm32-unknown-unknown --release &&
  wasm-bindgen ./target/wasm32-unknown-unknown/release/webgl_water_tutorial.wasm --out-dir . --no-typescript --no-modules &&
  wasm-opt -O3 -o optimized.wasm webgl_water_tutorial_bg.wasm  &&
  mv optimized.wasm webgl_water_tutorial_bg.wasm
fi

Our build.sh script is used to compile our application and generate a .wasm file. We first use cargo to build our app for the wasm32 (32 bit WebAssembly) target.

The wasm-bindgen library annotated this .wasm file with different bits of information that will help us automatically generate the JS shims that we need for things such as accessing the DOM (in the future WebAssembly will have native DOM support without JS shims).

We then run the wasm-bindgen cli on our webgl_water_tutorial.wasm file in order to generate this webgl_water_tutorial.js shim file and a new webgl_water_tutorial_bg.wasm file with all of wasm-bindgen’s annotations removed.

We’ll later serve these two files to the client in our index.html file.

// File: build.rs

// ... trimmed ...

fn main() {
    let blender_files = vec!["./terrain.blend".to_string(), "./bird.blend".to_string()];

    // Only re-run this build script if we change our blender file
    for blender_file in blender_files.iter() {
        println!("{}", format!("cargo:rerun-if-changed={}", blender_file));
    }

    // ... trimmed ...

    let blender_stdout = landon::export_blender_data(&blender_files).unwrap();

    let meshes_by_file = blender_mesh::parse_meshes_from_blender_stdout(&blender_stdout).unwrap();
    let flattened_meshes = blender_mesh::flatten_exported_meshes(&meshes_by_file).unwrap();
    let flattened_meshes = bincode::serialize(&flattened_meshes).unwrap();

    let mut f = File::create("./meshes.bytes").unwrap();
    f.write_all(&flattened_meshes[..]).unwrap();

    // ... trimmed ...

    let mut f = File::create("./armatures.bytes").unwrap();
    f.write_all(&flattened_armatures[..]).unwrap();
}

A major component of of rendering water is rendering the reflections of the scenery around the water. We use Blender to create some scenery, and then a build.rs script that runs landon to export our scenery into bytes that we can download and deserialize on the client side.

This means that you can make changes to the .blend files and the next time that you build your application your application’s 3d models will be automatically updated to reflect your new changes.

<!-- File: index.html -->

<html>
  <head>
    <meta content="text/html;charset=utf-8" http-equiv="Content-Type"/>
  </head>
  <body>
    <div id="webgl-water-tutorial"></div>

    <script src='/webgl_water_tutorial.js'></script>

    <script>
      window.addEventListener('load', function () {
        window.wasm_bindgen('/webgl_water_tutorial_bg.wasm')
          .then(function () {
            // Start our rust application. You can find `WebClient` in `src/lib.rs`
            const webClient = new window.wasm_bindgen.WebClient()
            webClient.start()

            let time = Date.now();
            function render () {
              const dt = Date.now() - time

              webClient.update(dt)
              webClient.render()
              window.requestAnimationFrame(render)

              time = Date.now()
            }

            render()
          })
      })
    </script>
  </body>
</html>

index.html is where the story begins from the viewer’s perspective.

We create an empty div with id webgl-water-tutorial that our application will later embed itself into.

We then load up the webgl_water_tutorial.js file that we created earlier. Then we download and initialize our WebAssembly module using the window.wasm_bindgen helper function.

Once it’s been initialized we create a new instance of our Rust application and then begin the request animation frame loop that will render the application.

// File: src/lib.rs

// ... trimmed ...

/// Used to run the application from the web
#[wasm_bindgen]
pub struct WebClient {
    app: Rc<App>,
    gl: Rc<WebGlRenderingContext>,
    renderer: WebRenderer,
}

#[wasm_bindgen]
impl WebClient {
    /// Create a new web client
    #[wasm_bindgen(constructor)]
    pub fn new() -> WebClient {
        console_error_panic_hook::set_once();

        let app = Rc::new(App::new());

        let gl = Rc::new(create_webgl_context(Rc::clone(&app)).unwrap());
        append_controls(Rc::clone(&app)).expect("Append controls");

        let renderer = WebRenderer::new(&gl);

        WebClient { app, gl, renderer }
    }

    /// Start our WebGL Water application. `index.html` will call this function in order
    /// to begin rendering.
    pub fn start(&self) -> Result<(), JsValue> {
        let gl = &self.gl;

        load_texture_image(
            Rc::clone(gl),
            "/dudvmap.png",
            TextureUnit::Dudv,
        );
        load_texture_image(
            Rc::clone(gl),
            "/normalmap.png",
            TextureUnit::NormalMap,
        );
        load_texture_image(
            Rc::clone(gl),
            "/stone-texture.png",
            TextureUnit::Stone,
        );

        Ok(())
    }

    /// Update our simulation
    pub fn update(&self, dt: f32) {
        self.app.store.borrow_mut().msg(&Msg::AdvanceClock(dt));
    }

    /// Render the scene. `index.html` will call this once every requestAnimationFrame
    pub fn render(&mut self) {
        self.renderer
            .render(&self.gl, &self.app.store.borrow().state, &self.app.assets());
    }
}

Our application is a single Rust crate with all of the code in the src folder.

src/lib.rs is the entry point into our Rust application. You’ll recognize the WebClient struct since we made use of it in index.html. Towards the bottom we expose a render function that gets called every requestAnimationFrame from index.html.

We could’ve started the requestAnimationFrame loop from the Rust side but I didn’t learn how to do that until after I started working on this tutorial and I didn’t feel like changing it. Maybe next time.

// File: src/shader/water-vertex.glsl

attribute vec2 position;

uniform mat4 perspective;
uniform mat4 model;
uniform mat4 view;

uniform vec3 cameraPos;
varying vec3 fromFragmentToCamera;

varying vec4 clipSpace;
varying vec2 textureCoords;

const float tiling = 4.0;

void main() {
    vec4 worldPosition = model * vec4(position.x, 0.0, position.y, 1.0);

    clipSpace = perspective * view *  worldPosition;

    gl_Position = clipSpace;

    textureCoords = position + 0.5;
    textureCoords = textureCoords * tiling;

    fromFragmentToCamera = cameraPos - worldPosition.xyz;
}

Our water’s vertex shader is mainly used to pass data to the fragment shader where most of the interesting things happen.

The first key piece of our water’s vertex shader is the varying clipSpace;.

We’ll use the clipSpace coordinates in our fragment shader in order to figure out the water fragments x and y location in the screen between 0 and 1.

From there we’ll use those x and y values to sample from our refraction and reflection textures. This technique is known as projective texture mapping.

From there we’ll set our water’s texture coordinates that we’ll use for samping the du/dv map and normal map. Since our water’s position vector values range from -0.5 to 0.5 we’ll just add 0.5 to make them range from 0 to 1.

This means that our bottom left corner of our water is (0, 0) and our top right corner is (1, 1).

Instead of deriving the values you could just as easily have just passed them in as an attribute. Since we’re dealing with a single quad the calculation vs. memory trade-off isn’t relevant at all.

The fromFragmentToCamera = cameraPos - worldPosition.xyz; is responsible for setting the varying vector that points from this current fragment of water to our camera. This will be used for our lighting calculations as well as our Fresnel effect simulation.

Water Fragment Shader

Next we’ll do a deep dive into the water’s fragment shader. Read over the entire fragment shader once or twice, and then we’ll talk through the major pieces.

// File: src/shader/water-fragment.glsl

precision mediump float;

uniform sampler2D refractionTexture;
uniform sampler2D reflectionTexture;
uniform sampler2D dudvTexture;
uniform sampler2D normalMap;
uniform sampler2D waterDepthTexture;

vec3 sunlightColor = vec3(1.0, 1.0, 1.0);
vec3 sunlightDir = normalize(vec3(-1.0, -1.0, 0.5));

varying vec3 fromFragmentToCamera;

// Changes over time, making the water look like it's moving
uniform float dudvOffset;

varying vec4 clipSpace;

varying vec2 textureCoords;

const float waterDistortionStrength = 0.03;
const float shineDamper = 20.0;

uniform float waterReflectivity;
uniform float fresnelStrength;

vec4 shallowWaterColor =  vec4(0.0, 0.1, 0.3, 1.0);
vec4 deepWaterColor = vec4(0.0, 0.1, 0.2, 1.0);

vec3 getNormal(vec2 textureCoords);

void main() {
    // Normalized device coordinates - Between 0 and 1
    vec2 ndc = (clipSpace.xy / clipSpace.w) / 2.0 + 0.5;

    vec2 refractTexCoords = vec2(ndc.x, ndc.y);
    // Reflections are upside down
    vec2 reflectTexCoords = vec2(ndc.x, -ndc.y);

    float near = 0.1;
    float far = 50.0;

    // Get the distance from our camera to the first thing under this water fragment that a
    // ray would collide with. This might be the ground, the under water walls, a fish, or any
    // other thing under the water. This distance will depend on our camera angle.
    float cameraToFirstThingBehindWater = texture2D(waterDepthTexture, refractTexCoords).r;
    // Convert from our perspective transformed distance to our world distance
    float cameraToFirstThingUnderWater = 2.0 * near * far /
     (far + near - (2.0 * cameraToFirstThingBehindWater - 1.0)
      * (far - near));

    float cameraToWaterDepth = gl_FragCoord.z;
    float cameraToWaterDistance = 2.0 * near * far / (far + near - (2.0 * cameraToWaterDepth - 1.0) * (far - near));

    float angledWaterDepth = cameraToFirstThingUnderWater - cameraToWaterDistance;

    vec2 distortedTexCoords = texture2D(dudvTexture, vec2(textureCoords.x + dudvOffset, textureCoords.y)).rg * 0.1;
    distortedTexCoords = textureCoords + vec2(distortedTexCoords.x, distortedTexCoords.y + dudvOffset);

    // Between -1 and 1
    vec2 totalDistortion = (texture2D(dudvTexture, distortedTexCoords).rg * 2.0 - 1.0)
     * waterDistortionStrength;

    refractTexCoords += totalDistortion;
    reflectTexCoords += totalDistortion;

    // Prevent out distortions from sampling from the opposite side of the texture
    // NOTE: This will still cause artifacts towards the edges of the water. You can fix this by
    // making the water more transparent at the edges.
    // @see https://www.youtube.com/watch?v=qgDPSnZPGMA
    refractTexCoords = clamp(refractTexCoords, 0.001, 0.999);
    reflectTexCoords.x = clamp(reflectTexCoords.x, 0.001, 0.999);
    reflectTexCoords.y = clamp(reflectTexCoords.y, -0.999, -0.001);

    vec4 reflectColor = texture2D(reflectionTexture, reflectTexCoords);

    vec4 refractColor = texture2D(refractionTexture, refractTexCoords);

    // The deeper the water the darker the color
    refractColor = mix(refractColor, deepWaterColor, clamp(angledWaterDepth/10.0, 0.0, 1.0));

    vec3 toCamera = normalize(fromFragmentToCamera);

    vec3 normal = getNormal(distortedTexCoords);

    // Fresnel Effect. Looking at the water from above makes the water more transparent.
    float refractiveFactor = dot(toCamera, normal);

    // A higher fresnelStrength makes the water more reflective since the
    // refractive factor will decrease
    refractiveFactor = pow(refractiveFactor, fresnelStrength);

    vec3 reflectedLight = reflect(normalize(sunlightDir), normal);
    float specular = max(dot(reflectedLight, toCamera), 0.0);
    specular = pow(specular, shineDamper);
    vec3 specularHighlights = sunlightColor * specular * waterReflectivity;

    gl_FragColor = mix(reflectColor, refractColor, refractiveFactor);
    // Mix in a bit of blue so that it looks like water
    gl_FragColor = mix(gl_FragColor, shallowWaterColor, 0.2) + vec4(specularHighlights, 0.0);
}

vec3 getNormal(vec2 textureCoords) {
    vec4 normalMapColor = texture2D(normalMap, textureCoords);
    float makeNormalPointUpwardsMore = 2.6;
    vec3 normal = vec3(
      normalMapColor.r * 2.0 - 1.0,
      normalMapColor.b * makeNormalPointUpwardsMore,
      normalMapColor.g * 2.0 - 1.0
    );
    normal = normalize(normal);

    return normal;
}

And now the main pieces of the water fragment shader:

vec2 ndc = (clipSpace.xy / clipSpace.w) / 2.0 + 0.5;

We need to know which pixel on our refraction and reflection textures corresponds to the water fragment, so we’ll use the clip space coordinate (ranging from -1 to 1) and convert it into a texture coordinates (ranging from 0 to 1). This is known as projective texture mapping.

vec2 refractTexCoords = vec2(ndc.x, ndc.y);
vec2 reflectTexCoords = vec2(ndc.x, -ndc.y);

Here we find the point on our refraction and reflection coordinates that corresponds to this water fragment.

Note that we negate the y coordinate of the reflection texture coordinate. If you scroll up and look at the reflection texture (top right corner of demo) you’ll be able to better visualize why.

float cameraToWaterDepth = gl_FragCoord.z;
float cameraToWaterDistance = 2.0 * near * far / (far + near - (2.0 * cameraToWaterDepth - 1.0) * (far - near));

float angledWaterDepth = cameraToFirstThingUnderWater - cameraToWaterDistance;

Here we determine the distance from the camera to the first thing below the water.

If our camera is at an angle this won’t necessary be the water’s floor, it could be the side of the terrain under the water.

We can use this depth for different effects, such as making water that is closer to the shore more transparent.

float refractiveFactor = dot(toCamera, normal);
refractiveFactor = pow(refractiveFactor, fresnelStrength);
gl_FragColor = mix(reflectColor, refractColor, refractiveFactor);

Here we decide how much of the refraction texture to show vs. the reflection texture. We base this on the Fresnel effect, which means that looking at the water from above makes it more transparent and looking at the water from the sides makes it more reflective.

// File: src/shader/mesh-non-skinned-fragment.rs

// ... trimmed ...

void main(void) {
    if (dot(worldPosition, clipPlane) < 0.0) {
        discard;
    }

    // ... trimmed ...
}

If you look at the refraction texture (top left of demo scene) and the reflection texture (top right of demo scene) you’ll see that the refraction texture is only showing things below the water and the reflection texture is only showing things above the water.

With OpenGL you would accomplish this using gl_ClipDistance, but WebGL provides no such mechanism.

Instead, we determine whether or not the fragment is on the wrong side of our clipping plane in the fragment shader, and if it is we discard it.

// File: src/render/mod.rs

fn render_reflection_fbo(
    &mut self,
    gl: &WebGlRenderingContext,
    state: &State,
    assets: &Assets,
) {
    let Framebuffer { framebuffer, .. } = &self.reflection_framebuffer;
    gl.bind_framebuffer(GL::FRAMEBUFFER, framebuffer.as_ref());

    gl.viewport(0, 0, REFLECTION_TEXTURE_WIDTH, REFLECTION_TEXTURE_HEIGHT);

    gl.clear_color(0.53, 0.8, 0.98, 1.);
    gl.clear(GL::COLOR_BUFFER_BIT | GL::DEPTH_BUFFER_BIT);

    if state.water().use_reflection {
        let clip_plane = [0., 1., 0., -WATER_TILE_Y_POS];
        self.render_meshes(gl, state, assets, clip_plane, true);
    }
}

As shown above, we’ll pass the clip_plane into our mesh renderer and it’ll make its way into the fragment shader and power our clipping.

// File: src/render/water_tile.rs

impl<'a> Render<'a> for RenderableWaterTile<'a> {
    // ... trimmed ...

    fn buffer_attributes(&self, gl: &WebGlRenderingContext) {
        let shader = self.shader();

        let pos_attrib = gl.get_attrib_location(&shader.program, "position");
        gl.enable_vertex_attrib_array(pos_attrib as u32);

        // These vertices are the x and z values that create a flat square tile on the `y = 0`
        // plane. In our render function we'll scale this quad into the water size that we want.
        // x and z values, y is omitted since this is a flat surface. We set it in the vertex shader
        let vertices: [f32; 8] = [
            -0.5, 0.5, // Bottom Left
            0.5, 0.5, // Bottom Right
            0.5, -0.5, // Top Right
            -0.5, -0.5, // Top Left
        ];

        let mut indices: [u16; 6] = [0, 1, 2, 0, 2, 3];

        RenderableWaterTile::buffer_f32_data(&gl, &vertices, pos_attrib as u32, 2);
        RenderableWaterTile::buffer_u16_indices(&gl, &mut indices);
    }

    fn render(&self, gl: &WebGlRenderingContext, state: &State) {
        let shader = self.shader();

        let model_uni = shader.get_uniform_location(gl, "model");
        let view_uni = shader.get_uniform_location(gl, "view");
        let refraction_texture_uni = shader.get_uniform_location(gl, "refractionTexture");
        let reflection_texture_uni = shader.get_uniform_location(gl, "reflectionTexture");
        let dudv_texture_uni = shader.get_uniform_location(gl, "dudvTexture");
        let normal_map_uni = shader.get_uniform_location(gl, "normalMap");
        let water_depth_texture_uni = shader.get_uniform_location(gl, "waterDepthTexture");
        let dudv_offset_uni = shader.get_uniform_location(gl, "dudvOffset");
        let camera_pos_uni = shader.get_uniform_location(gl, "cameraPos");
        let perspective_uni = shader.get_uniform_location(gl, "perspective");
        let water_reflectivity_uni = shader.get_uniform_location(gl, "waterReflectivity");
        let fresnel_strength_unit = shader.get_uniform_location(gl, "fresnelStrength");

        let pos = (0., 0., 0.);

        let x_scale = 18.;
        let z_scale = 18.;
        let scale = Matrix4::new_nonuniform_scaling(&Vector3::new(x_scale, 1.0, z_scale));

        let model = Isometry3::new(Vector3::new(pos.0, pos.1, pos.2), nalgebra::zero());
        let model = model.to_homogeneous();
        let model = scale * model;
        let mut model_array = [0.; 16];
        model_array.copy_from_slice(model.as_slice());
        gl.uniform_matrix4fv_with_f32_array(model_uni.as_ref(), false, &mut model_array);

        let mut view = state.camera().view();
        gl.uniform_matrix4fv_with_f32_array(view_uni.as_ref(), false, &mut view);

        // ... trimmed ...

        gl.enable(GL::BLEND);
        gl.blend_func(GL::SRC_ALPHA, GL::ONE_MINUS_SRC_ALPHA);

        gl.draw_elements_with_i32(GL::TRIANGLES, 6, GL::UNSIGNED_SHORT, 0);

        gl.disable(GL::BLEND);
    }
}

src/render/water_tile.rs is where we pass all of our uniforms into our water shader. It is here that you’d do much of your experimenting if you try to extend this tutorial with more uniforms. For example, you might add a foam texture and sample it when the water is more shallow.

Before rendering we enable alpha blending, even though we aren’t actually making use of it in this tutorial.

If you look carefully you’ll sometimes see a little fuzziness near the edges of the water.

The water’s depth can also be used for fixing this fuzziness artifact by making the edges of the water more transparent. While we don’t do this in this demo, but ThinMatrix’s soft water edges tutorial explains how to accomplish this in your fragment shader.

Going forwards

You can find the rest of the demo’s code on GitHub.

If you have any questions or run into any challenges please feel free to open an issue and I’ll get back to you ASAP!

In the meantime, scroll down and join the mailing list and shoot me an email if there’s anything that you’d like to learn more about!

Til’ next time,

- CFN

Distributing Blender Scripts and Addons via npm

Wed, 04 Oct 2017 12:00:00 +0000

Hello, hello!

Right now the most popular way to distribute Blender addons and scripts is to give a download link that the user can use to get a copy of the script.

They can then add it into their user preferences

or use it from the Blender CLI

blender --python /path/to/my-downloaded-addon.py.

I wanted to share a slightly easier way to distribute your Blender addons and scripts by using npm (or any other package manager).

The idea is to be able to:

npm install -g my-blender-addon
# Run script one off in the Blender CLI
blender -P `my-blender-addon --run`
# or install and save in Blender user preferences
my-blender-addon --install

When you wrap a command in backticks in bash the output of the command gets used as an argument.

So the above my-blender-addon --run command simply prints the absolute file path of the python script that you want to run.

That way, the -P Blender CLI flag ends up getting called with the absolute path to your Blender script.

Let’s say you have a Blender addon package with a tree like so:

tree ./my-addon-package
├── cli.js
├── install-addon.js
├── install-addon.py
├── run-addon.js
├── run-addon.py
├── my-addon-script.py

There are two JavaScript scripts that you’ll expose via your npm module’s CLI. install-addon.js and run-addon.js. You can expose them behind any flag such as --install or --run, for example.

Both scripts simply console.log their corresponding Python script.

// install-addon.js
var path = require('path')
console.log(
  path.resolve(__dirname, './install-addon.py')
)

// run-addon.js
var path = require('path')
console.log(
  path.resolve(__dirname, './run-addon.py')
)

The addon install script takes the absolute path to your Blender addon and passes it into Blender’s addon_install method.

// install-addon.py
dir = os.path.dirname(__file__)
addonFilePath = dir + '/my-addon-script.py'
bpy.ops.wm.addon_install(filepath=addonFilePath)
bpy.ops.wm.addon_enable(module='my-addon-module-name')
bpy.ops.wm.save_userpref()

The run script installs and enables the addon and then executes it.

// run-addon.py
dir = os.path.dirname(__file__)
addonFilePath = dir + '/my-addon-script.py'
bpy.ops.wm.addon_install(filepath=addonFilePath)
bpy.ops.wm.addon_enable(module='my-addon-module-name')
bpy.ops.rigging.myAddonMethodName()

So if a user ran:

npm install g my-blender-addon
blender --python `my-blender-addon --run`

It would really resolve to:

blender --python `/path/to/run-addon.py`

You can check out blender-iks-to-fks and blender-actions-to-json for real examples of npm-installable Blender scripts/addons.

Hope it helps!

- CFN

Blender to WebGL Hot Reloading Experiment - Part 2/2

Sun, 24 Sep 2017 07:00:00 +0000

Welcome to Part 2 of my Blender -> WebGL hot reloading experiment. In part 1 we explored hot reloading wavefront OBJ models. This time around we gave skinned models with skeletal animations a try.

Hot reloading skeletal animation data from Blender to WebGL

I’ll be talking about the birds eye view details, but you can find the full source on GitHub. Feel free to fork and re-purpose the source code for your own implementation.

In Part 1 all we needed to export from Blender was an OBJ file. This functionality is built into Blender, so we just had to trigger an export whenever our Blender file was saved. We then parsed the OBJ data, sent it to the frontend, and boom - finished.

Handling skinned models was a bit trickier - let’s take a look at why.

Exporting Skinned Models

I use Blender’s COLLADA exporter which has thus far been rock solid for me. But there are a few drawbacks to it that left me a little stranded while starting on this experiment.

In Blender, animations are organized into groups of keyframes called “actions”

Blender has actions, which organize your animations into different named sets of animation keyframes. (Really actions are composed of fcurves, but it’s fine to ignore that detail unless you’re looking for extremely precise motions).

In your Blender file you might have dozens of actions, but when exporting to COLLADA using the native Blender exporter only one action can be exported.

This is an issue because you want to have access to all of your actions when you’re animating your model in your engine, not just whichever one you happen to have open.

In the past my solution was to manually copy all keyframes into one combined action and then export that combined action, but for automated hot reloading of new actions I needed a more automated process.

Exporting Actions

I started off trying to write a script to combine all actions into one combined action so that by the time I exported the combined action to COLLADA everything would be all together. This was essentially going to be an automated version of my old manual process.

I ran into some early troubles with this due to inexperience with the Blender API, and I’m really glad that I did.

Had I gone down that road I would’ve had a big collection of keyframes all under one action and no way to tell which keyframes belonged to which action without manually writing down the exact orders of the keyframes.

For example, I would have had to know that the idle animation was keyframes number 0-11, walk was 12-20 etc. Anytime I added new keyframes I would have needed to update this mapping.

After more experimentation I ended up leaning towards a solution of just exporting the keyframed bone pose matrices for all actions into a JSON file.

Then I could send that JSON down to the client, and the client would be able to reproduce all animations by accessing the bone pose matrices for any of the actions in the file.

We write all of our armature animation data to a JSON file

So after quite a bit of stumbling I made blender-actions-to-json a little Python script that exports all of an armature’s actions to a JSON file.

With actions out of the way, I needed to handle IKs (inverse kinematics).

Exporting IKs as FKs

I’ve written about converting IKs to FKs before while in Blender, but I now needed a way to do it from the command line.

After trying out a few approaches I settled on writing a script that would install my IK -> FK conversion addon and then run it. Installing an addon comes down to moving the addon file into Blender’s directory so this is fast.

The end process

The final steps were:

Start a Node.js process that watches for Blender file changes
Whenever a Blender File changes, spawn a Blender child process that converts the IK’d model into an FK model, and then exports both the FK armature actions into an action file and the model data COLLADA file.
Parsed and then send the action file and COLLADA file data to the client as JSON over an open websocket connection.
Client overwrites the vertex data GPU buffers with the new data that it received over websocket.
Hot reloading achieved!

Using data from blender-actions-to-json we overwrite the old animation

Future Performance Enhancements

In this experiment’s code we’re technically sending over some joint data twice since both the actions.json file and the COLLADA file hold joint data.

We don’t use any of the joint data in the COLLADA file, so a quick performance enhancement might be to delete the COLLADA keyframes before sending the model to the client. This could be accomplished by adding a flag to collada-dae-parser to not process or output any keyframes.

But yeah, that’s certainly for another day.

Great!

Thanks for reading! You should hit that little tweet button down there!

Cya next time,

- CFN

Unit Testing Blender Scripts and Addons with Node.js

Tue, 12 Sep 2017 17:26:00 +0000

Lately I’ve been working on automating the more tedious bits of my asset creation pipeline and while doing so I’ve been picking up a bit of Blender scripting.

Everything that you can do by hand in Blender can be done via API, so it’s been very fun being able to turn manual tasks into scripts without much headache.

Mousing over anything in Blender reveals its API

When I first started Blender scripting I tried to find some open source libraries to use as example references. Unfortunately none of the ones that I found were unit tested, so I didn’t have any immediate examples on best practices.

Now that I’ve written some of my own Blender scripts I want to quickly share two Blender script unit testing approaches that I took while working on blender-iks-to-fks and blender-actions-to-json.

I use Node.js and tape to run my tests, but the two approaches that we’ll discuss should transfer over to your language and tools of choice.

Two types of tests

Most Blender scripts fall into two buckets:

Bucket one is scripts that export data from Blender to the user’s filesystem. An example of this type of script is one that exports a JSON file with the names of all of the objects in the scene.

Bucket number two is scripts that modify or create visuals. An example of this is a script that creates and places a hat on anything that it believes is a head.

Depending on the nature of the script, you can either

Export the relevant data to a file, read that file in your test runner and verify that it has the contents that you expect.
Render an image of your Blender view and then use imagemagick to verify that the rendered image matches what you expect it to look like.

Data tests

Your data tests will boil down to something like this pseudocode:

var cp = require('child_process')
var fs = require('fs')
test('Exports a list of objects', function (t) {
  // Spawn headless blender in a child process
  cp.exec(
    'blender model.blend -b -P my-script.py -- ./script-output-file.txt',
    function (err, stdout, stderr) {
      // Read the data that your script wrote to the file system
      var output = fs.readFileSync('./script-output-file.json')
      // Verify that your scripts output matches what you expect
      t.deepEqual(output, expectedOutput)
    }
})

Blender has a CLI that allows you to run a headless instance of Blender and then run scripts from inside of this Blender instance.

So you spawn a child process that runs Blender’s CLI and your script in the background. You then write your script’s output to a file and then verify that you got what you expected.

The python addon or script that you’re testing might look something like this:

# In my-script.py

# Do a bunch of stuff

# Create a list of objects that we'll
# be exporting to a file
outputFileContents = 'mesh1, mesh2, mesh3'

# Write to the filesystem
argv = sys.argv
# Get all args after `--`
argv = argv[argv.index('--') + 1:]
file = argv[0] # script-output-file.txt

with open(file, 'w') as outputFile:
    outputFile.write(outputFileContents)

Visual tests

Visual testing involves rendering your Blender scene and verifying that the rendering matches an image that you have that you know is correct (usually by manually verifying it once).

Technically speaking every visual that you create in Blender is backed by bytes of data that you have full access to. So while anything can can be a data test, sometimes it is easier to test by just comparing renderings.

For example, in blender-iks-to-fks I’m taking an IK rig and creating an FK rig that should have the same poses.

While I could write all of my vertex positions to a file and compare the before and after, here it was easier to just take a before and after pic, compare them and call it a day. This meant that I didn’t need to maintain code in my script that exported data that I would only use for testing.

I have written about a similar visual testing approach while visually testing WebGL components.

Visual tests look something like this:

cp.exec(
  // `-o` means render output to an image
  // `-f 10` means render frame number 10
  // `-F PNG` means use the PNG image format
  // `-noaudio` makes this work if you don't have
  //   an audio card on your machine (Travis CI)
  'blender model.blend -b -o actual.png -f 10 -F PNG -noaudio,
  function () {
    cp.exec(
      // Use imagemagick to test that your image is what you want
      'compare -metric RMSE actual.png expected.png',
      function (_, stdout, err) {
        // Then read stdout to see get your image comparison
        // results
      }
    )
  }
)

You render your Blender view to a PNG, then test that PNG matches an expected PNG that you’ve commited.

If you’re expecting a perfect match, you should check stdout for 0. If a slight differene between the two images is acceptable, you can check stderr for the root mean square error, which is a number that represents how different the images are.

Nice!

And that wraps it up! Check out the two repos if you need some example code.

Also, scroll down and join my mailing list!

- CFN

Blender to WebGL Hot Reloading Experiment - Part 1/2

Tue, 05 Sep 2017 09:45:00 +0000

Over the weekend I experimented with hot reloading 3d models from Blender into a WebGL application.

In case you’ve never heard the term, hot reloading is when you reload something while your application is running. So, in our case, we’re updating our 3d model vertex data without refreshing the page.

Hot reloading a 3d model from Blender (right) -> Web Browser (left)

The experiment ended up coming together, so in this post I’ll explain how it works in case you ever want to do something similar.

Motivation

One of the challenges that I face with 3D modeling is the amount of hoops that I need to go through in order to get my model from Blender into my work-in-progress WebGL game.

After working on a model, I needed to export it using the correct settings and then run my converter on the correct file. Then I’d view my model in my game and if there were any issues I’d need to repeat the entire process.

Having this friction led me to dread making changes because of the repetitive work that needed to be done in order to finally see my changes.

As someone who is learning to model and to create art in general, having such a de-motivator stunts my growth. I end up wanting to work on something else that I’m good at instead of trying to incrementally improve my asset creation skills.

With this in mind, over the weekend I experimented with fully automating the export process.

The goal was to be able to hit save in Blender and have my updated model automatically appear in my game, without any additional effort from me.

The parameters

Right now my engine deals with two types of models - skinned and non-skinned.

Skinned models are models such as humans, wolves, ents or anything that use bones to control its movements.

Non-skinned models are models such as a house, a fence or a wall that don’t have bones for bending and deforming.

My process for getting a skinned model into the game is a bit more involved than my process for getting a non-skinned model into the game, so for Part 1 of this experiment I focused on hot-reloading non-skinned models.

If you’re interested in Part 2, where I’ll be experimenting with hot reloading models that have skeletal animations, join my mailing list!

How it works

The data flow is as follows:

I start a Node.js script that watches for changes to *.blend files.

chokidar.watch('./*.blend', {})
.on('change', ...)

Whenever there’s a change, our Node.js script spawns a child process that runs the a headless Blender instance. The instance executes a script that exports the Blender file into an .obj file.

require('child_process').exec(blender model.blend --background \
--python blender-to-obj.py -- model.obj, ...)

The next step is to convert the file into JSON using wavefront-obj-parser. I converted it in another child process using the wavefront-obj-parser CLI , but you could just read the .obj file into memory and convert it using the wavefront-obj-parser API.

require('child_process').exec(cat model.obj | obj2json > model.json)

I have a websocker server running, so next I send the new model.json to open websockets.

websocket.send(jsonFileBuffer.toString())

Lastly, the client receives the JSON data and overwrites the previous vertex data buffers on the GPU

clientWebsocket.onmessage = function (message) {
  var newVertexData = JSON.parse(message.data)
  // ...
  gl.bindBuffer(gl.ARRAY_BUFFER, vertexPosBuffer)
  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(newVertexData.positions), gl.STATIC_DRAW)
  gl.vertexAttribPointer(vertexPosAttrib, 3, gl.FLOAT, false, 0, 0)
  // ...
}

And voila, the client’s render loop is now rendering the model using the new vertex data!

I didn’t do any official benchmarking, but a model with around 8,000 vertices took just under two seconds after Blender file save to show up in my browser. Not bad!

You can check out the the complete source on GitHub.

Next steps

The steps are a bit coupled in the experimental code, but that should be fine for me for now. In the future I might separate the Blender -> OBJ step from the OBJ -> JSON step from the JSON -> websocket step. But for now I’ll probably just copy this code right into my game’s tooling.

It could be useful to abstract some of this into a standalone repo(s), but I’m not too interested in thinking through the proper abstraction right now.

You should of course feel totally free to take any of the code and build your own library. Let me know if you make something!

Thanks for reading!

- CFN

WebGL Tutorial: Skeletal Animation Sound Effects

Thu, 31 Aug 2017 16:54:00 +0000

Welcome back to another WebGL tutorial 🎉

This time around we’re going to learn how to trigger a sound effect at a precise moment during a skeletal animation.

We’ll combine the WebGL and Web Audio APIs in order to render an animated baseball player model and then play a sound effect whenever his baseball and bat make contact.

You can use this technique on a whole host of situations. For example, you might trigger sound effects precisely as your character’s feet make contact with the ground. Or play a clapping sound as your character claps her hands. Or any other use case that your imagination cooks up.

We will not be using any third party rendering libaries. We’ll be writing our own vertex and fragment shader from scratch.

You’ll hopefully walk away with a better sense of how to implement skeletal animation, as well as how to perfectly time your sound effects to your character’s activities.

Precisely triggering our sound effect

The concept behind our technique is simple. Our skeletal animation has 20 discrete keyframes. Each time we render our model we sample from a lower and upper keyframe, based on how much time has elapsed.

If our sound effect is meant to be played on the eigth keyframe, we’ll trigger it everytime our lower keyframe changes from keyframe #7 to keyframe #8.

click and drag to move the camera

If you wanted to you could trigger your animation at some offset amount of time in between your lower and upper keyframe, but this time around we’re going to keep our scope simple and only trigger sound effects on exact keyframes.

Alright, now that you have a sense of what we’re about to do, let’s dive in!

Getting set up

Before we can start writing code, we’ll need to set up our environment.

First, create a new file for this tutorial.

mkdir webgl-skeletal-sound-tutorial
cd webgl-skeletal-sound-tutorial
touch tutorial.js

Next, let’s download our 3d model and texture.

curl -OL https://github.com/chinedufn/\
webgl-skeletal-animation-sound-tutorial/raw/\
master/baseball-player.dae > baseball-player.dae

curl -OL https://github.com/chinedufn/\
webgl-skeletal-animation-sound-tutorial/raw\
/master/baseball-player-uvs.png > baseball-player-uvs.png

Next we’ll download our COLLADA parser so that we can get the vertex and joint data from our COLLADA .dae file.

npm install collada-dae-parser@0.12.0 \
expand-vertex-data@1.0.1 \
mat-to-dual-quat@1.0.0

gl-matrix, skeletal-animation-system we’ll power the math calculations that we’ll need for our experience.

npm install skeletal-animation-system@0.6.5 \
gl-mat4@1.1.4 gl-mat3@1.0.0 gl-vec3@1.0.3

Let’s install a development server so that we can easily view our work locally.

npm install budo@10.0.4

Lastly, we’ll want to parse our COLLADA file into JSON.

./collada-dae-parser/bin/dae2json.js \
baseball-player.dae > baseball-player.json

Creating our animation / sound effect experience

You’ll learn more if you type out the tutorial, but all of the source code is on GitHub should you need a full reference.

Now open up tutorial.js and let’s dive in!

var glMat4 = require('gl-mat4')
var glMat3 = require('gl-mat3')
var glVec3 = require('gl-vec3')
var expandVertexData = require('expand-vertex-data')
var mat4ToDualQuat = require('mat4-to-dual-quat')
var animationSystem = require('skeletal-animation-system')

We first require all of the dependencies that we’ll be using. gl-mat4 gl-mat3 and gl-vec3 are our standard matrix and vector libraries that we use to calculate the data for our uniforms.

expand-vertex-data de-compresses our JSON model vertex data so that we can use it for our vertex attributes.

mat-4-to-dual-quat is used to convert 4x4 matrices into dual quaternions. I’ve written about why I tend to prefer dual quaternions over matrices for vertex skinning, so we’re going to convert all of our joint matrices into dual quaternions.

And lastly, skeletal-animation-system is a wrapper around some math for interpolating and blending our model’s joint data over time.

var model = require('./baseball-player.json')
var baseballPlayer = expandVertexData(model)
baseballPlayer.keyframes = Object.keys(model.keyframes)
.reduce(function (dualQuats, keyframe) {
  dualQuats[keyframe] = []
  for (var k = 0; k < model.keyframes[keyframe].length; k++) {
    dualQuats[keyframe][k] = mat4ToDualQuat(
      model.keyframes[keyframe][k]
    )
  }
  return dualQuats
}, {})

Earlier we converted our COLLADA file into JSON, and here we require that JSON. As mentioned above, we’ll then convert all of the joint matrices into dual quaternions using mat4-to-dual-quat.

var canvas = document.createElement('canvas')
canvas.style.cursor = 'pointer'
canvas.width = 500
canvas.height = 500

var isDragging = false
var xCamRot = Math.PI / 20
var yCamRot = 0
var lastX
var lastY
canvas.onmousedown = function (e) {
  isDragging = true
  lastX = e.pageX
  lastY = e.pageY
}
canvas.onmouseup = function () {
  isDragging = false
}
canvas.onmousemove = function (e) {
  if (isDragging) {
    xCamRot += (e.pageY - lastY) / 60
    yCamRot -= (e.pageX - lastX) / 60

    xCamRot = Math.min(xCamRot, Math.PI / 2.5)
    xCamRot = Math.max(-0.5, xCamRot)

    lastX = e.pageX
    lastY = e.pageY
  }
}

canvas.addEventListener('touchstart', function (e) {
  lastX = e.touches[0].clientX
  lastY = e.touches[0].clientY
})
canvas.addEventListener('touchmove', function (e) {
  e.preventDefault()
  xCamRot += (e.touches[0].clientY - lastY) / 60
  yCamRot -= (e.touches[0].clientX - lastX) / 60

  xCamRot = Math.min(xCamRot, Math.PI / 2.5)
  yCamRot = Math.max(yCamRot, -0.5)

  lastX = e.touches[0].clientX
  lastY = e.touches[0].clientY
})

As per usual, we’ll set up our canvas events for detecting mouse and finger drags so that we can control our camera.

var gl = canvas.getContext('webgl')
gl.clearColor(0.0, 0.0, 0.0, 1.0)
gl.enable(gl.DEPTH_TEST)

var vertexGLSL = `
attribute vec3 aVertexPosition;
attribute vec3 aVertexNormal;
attribute vec2 aVertexUV;

attribute vec4 aJointIndex;
attribute vec4 aJointWeight;

varying vec3 vNormal;

uniform mat4 uMVMatrix;
uniform mat4 uPMatrix;
uniform mat3 uNMatrix;

uniform vec4 boneRotQuaternions[20];
uniform vec4 boneTransQuaternions[20];

varying vec3 vLightWeighting;
varying vec2 vUV;

void main (void) {
  vec4 weightedRotQuats =
    boneRotQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneRotQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneRotQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneRotQuaternions[int(aJointIndex.w)] * aJointWeight.w;

  vec4 weightedTransQuats =
    boneTransQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneTransQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneTransQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneTransQuaternions[int(aJointIndex.w)] * aJointWeight.w;

  float xRot = weightedRotQuats[0];
  float yRot = weightedRotQuats[1];
  float zRot = weightedRotQuats[2];
  float wRot = weightedRotQuats[3];
  float magnitude = sqrt(xRot * xRot + yRot * yRot + zRot * zRot +
  wRot * wRot);
  weightedRotQuats = weightedRotQuats / magnitude;
  weightedTransQuats = weightedTransQuats / magnitude;

  float xR = weightedRotQuats[0];
  float yR = weightedRotQuats[1];
  float zR = weightedRotQuats[2];
  float wR = weightedRotQuats[3];

  float xT = weightedTransQuats[0];
  float yT = weightedTransQuats[1];
  float zT = weightedTransQuats[2];
  float wT = weightedTransQuats[3];

  float t0 = 2.0 * (-wT * xR + xT * wR - yT * zR + zT * yR);
  float t1 = 2.0 * (-wT * yR + xT * zR + yT * wR - zT * xR);
  float t2 = 2.0 * (-wT * zR - xT * yR + yT * xR + zT * wR);

  mat4 convertedMatrix = mat4(
      1.0 - (2.0 * yR * yR) - (2.0 * zR * zR),
      (2.0 * xR * yR) + (2.0 * wR * zR),
      (2.0 * xR * zR) - (2.0 * wR * yR),
      0,
      (2.0 * xR * yR) - (2.0 * wR * zR),
      1.0 - (2.0 * xR * xR) - (2.0 * zR * zR),
      (2.0 * yR * zR) + (2.0 * wR * xR),
      0,
      (2.0 * xR * zR) + (2.0 * wR * yR),
      (2.0 * yR * zR) - (2.0 * wR * xR),
      1.0 - (2.0 * xR * xR) - (2.0 * yR * yR),
      0,
      t0,
      t1,
      t2,
      1
      );

  vec3 transformedNormal = (convertedMatrix *
    vec4(aVertexNormal, 0.0)).xyz;

  float y;
  float z;
  y = transformedNormal.z;
  z = -transformedNormal.y;
  transformedNormal.y = y;
  transformedNormal.z = z;

  transformedNormal = uNMatrix * transformedNormal;

  vec4 leftWorldSpace = convertedMatrix *
  vec4(aVertexPosition, 1.0);
  y = leftWorldSpace.z;
  z = -leftWorldSpace.y;
  leftWorldSpace.y = y;
  leftWorldSpace.z = z;

  vec4 leftHandedPosition = uPMatrix * uMVMatrix * leftWorldSpace;

  gl_Position = leftHandedPosition;

  vNormal = transformedNormal;
  vUV = aVertexUV;
}
`

I’ve written an explanation of a dual quaternion linear blending vertex shader so I won’t dive into detail here.

In short, we’re writing a vertex shader that will position all of our vertices based on where are bones are located in model space.

var fragmentGLSL = `
precision mediump float;

varying vec3 vLightWeighting;
varying vec3 vNormal;
varying vec2 vUV;

uniform vec3 uAmbientColor;
uniform vec3 uDirectionalColor;
uniform vec3 uLightingDirection;
uniform sampler2D uSampler;

void main(void) {
  float directionalLightWeighting = max(
    dot(vNormal, uLightingDirection),
  0.0);
  vec3 lightWeighting = uAmbientColor + uDirectionalColor *
  directionalLightWeighting;

  vec4 baseColor = texture2D(uSampler, vec2(vUV.s, vUV.t));
  gl_FragColor = baseColor * vec4(lightWeighting, 1.0);
}
`

Our simple fragment shader combines our model’s texture and some directional lighting in order to render our fragments.

var vertexShader = gl.createShader(gl.VERTEX_SHADER, vertexGLSL)
gl.shaderSource(vertexShader, vertexGLSL)
gl.compileShader(vertexShader)

var fragmentShader = gl.createShader(
  gl.FRAGMENT_SHADER,
  fragmentGLSL
)
gl.shaderSource(fragmentShader, fragmentGLSL)
gl.compileShader(fragmentShader)

var shaderProgram = gl.createProgram()
gl.attachShader(shaderProgram, vertexShader)
gl.attachShader(shaderProgram, fragmentShader)
gl.linkProgram(shaderProgram)

gl.useProgram(shaderProgram)

var vertexPosAttrib = gl.getAttribLocation(shaderProgram,
                                           'aVertexPosition')
var vertexNormalAttrib = gl.getAttribLocation(shaderProgram,
                                              'aVertexNormal')
var vertexUVAttrib = gl.getAttribLocation(shaderProgram,
                                          'aVertexUV')
var jointIndexAttrib = gl.getAttribLocation(shaderProgram,
                                            'aJointIndex')
var jointWeightAttrib = gl.getAttribLocation(shaderProgram,
                                             'aJointWeight')

gl.enableVertexAttribArray(vertexPosAttrib)
gl.enableVertexAttribArray(vertexNormalAttrib)
gl.enableVertexAttribArray(vertexUVAttrib)
gl.enableVertexAttribArray(jointIndexAttrib)
gl.enableVertexAttribArray(jointWeightAttrib)

var ambientColorUni = gl.getUniformLocation(shaderProgram,
                                            'uAmbientColor')
var lightingDirectionUni = gl.getUniformLocation(
  shaderProgram,
  'uLightingDirection')
var directionalColorUni = gl.getUniformLocation(
  shaderProgram,
  'uDirectionalColor')
var mVMatrixUni = gl.getUniformLocation(shaderProgram, 'uMVMatrix')
var pMatrixUni = gl.getUniformLocation(shaderProgram, 'uPMatrix')
var nMatrixUni = gl.getUniformLocation(shaderProgram, 'uNMatrix')
var uSampler = gl.getUniformLocation(shaderProgram, 'uSampler')

var boneRotQuaternions = {}
var boneTransQuaternions = {}
for (var i = 0; i < 20; i++) {
  boneRotQuaternions[i] = gl.getUniformLocation(
    shaderProgram,
    `boneRotQuaternions[${i}]`)
  boneTransQuaternions[i] = gl.getUniformLocation(
    shaderProgram,
    `boneTransQuaternions[${i}]`)
}

var vertexPosBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, vertexPosBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(
  baseballPlayer.positions
), gl.STATIC_DRAW)
gl.vertexAttribPointer(vertexPosAttrib, 3, gl.FLOAT, false, 0, 0)

var vertexNormalBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, vertexNormalBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(
  baseballPlayer.normals
), gl.STATIC_DRAW)
gl.vertexAttribPointer(vertexNormalAttrib, 3, gl.FLOAT, false,
                       0, 0)

var jointIndexBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, jointIndexBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(
  baseballPlayer.jointInfluences), gl.STATIC_DRAW)
gl.vertexAttribPointer(jointIndexAttrib, 4, gl.FLOAT, false, 0, 0)

var jointWeightBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, jointWeightBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(
  baseballPlayer.jointWeights), gl.STATIC_DRAW)
gl.vertexAttribPointer(jointWeightAttrib, 4, gl.FLOAT, false, 0, 0)

var vertexUVBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, vertexUVBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(
  baseballPlayer.uvs), gl.STATIC_DRAW)
gl.vertexAttribPointer(vertexUVAttrib, 2, gl.FLOAT, false, 0, 0)

var vertexIndexBuffer = gl.createBuffer()
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, vertexIndexBuffer)
gl.bufferData(gl.ELEMENT_ARRAY_BUFFER, new Uint16Array(
  baseballPlayer.positionIndices), gl.STATIC_DRAW)

gl.uniform3fv(ambientColorUni, [0.3, 0.3, 0.3])
var lightingDirection = [1, -1, -1]
glVec3.scale(lightingDirection, lightingDirection, -1)
glVec3.normalize(lightingDirection, lightingDirection)
gl.uniform3fv(lightingDirectionUni, lightingDirection)
gl.uniform3fv(directionalColorUni, [1, 1, 1])

gl.uniformMatrix4fv(pMatrixUni, false, glMat4.perspective(
  [], Math.PI / 3, 1, 0.1, 100))

var texture = gl.createTexture()
var textureImage = new window.Image()
var imageHasLoaded
textureImage.onload = function () {
  gl.bindTexture(gl.TEXTURE_2D, texture)
  gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true)
  gl.texParameteri(
    gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST)
  gl.texParameteri(
    gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST)
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA,
                gl.UNSIGNED_BYTE, textureImage)
  imageHasLoaded = true
}
textureImage.src = 'baseball-player-uvs.png'

Nothing too new here that we haven’t done in previous tutorials. We’re setting up all of our attributes and uniforms so that we can send the relevant data to the GPU.

var keyframesToPlaySoundOn = {
  7: true,
  9: true
}

var clockTime = 0
var lastStartTime = new Date().getTime()

var previousLowerKeyframe

And here’s where things start to come together. We have an object specifying the keyframes that we want to play our sound effect on. And then we’re getting set up to keep track of our lower keyframe after each render.

We’ll be keeping track of the lower keyframe that we’re sampling

Every frame we sample from two keyframes, lower and upper, based on the current clock time. When our lower keyframe is different from our previous lower keyframe, we know that we’ve started a new section of the animation. If this new section of the animation is the ball hitting the bat, we’ll play a sound effect.

function draw () {
  var currentTime = new Date().getTime()

  var timeElapsed = (currentTime - lastStartTime) / 1000 *
    playbackSpeed
  clockTime += timeElapsed
  lastStartTime = currentTime

  gl.clear(gl.COLOR_BUFFER_BIT, gl.DEPTH_BUFFER_BIT)

Here we advance the current time based on our playback speed. So as you drag the playback slider in the demo the clock time will increase more slowly.

  var animationData = animationSystem.interpolateJoints({
    currentTime: clockTime,
    keyframes: baseballPlayer.keyframes,
    jointNums: [
      0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
      13, 13, 14, 15, 16, 17, 18, 19, 20
    ],
    currentAnimation: {
      startTime: 0,
      range: [0, 12]
    }
  })

Next we use skeletal-animation-system to interpolate our lower and upper dual quaternions based on the current keyframes that we’re sampling. There are other options for skeletal-animation-system, but there are the only ones that we need.

  var newLowerKeyframe =
    animationData.currentAnimationInfo.lowerKeyframeNumber
  if (keyframesToPlaySoundOn[newLowerKeyframe] &&
      previousLowerKeyframe !== newLowerKeyframe) {
    if (!muted) {
      audio.play()
    }
  }
  previousLowerKeyframe = newLowerKeyframe

As mentioned above, we check if we’re transitioning over to the ball hitting the bat. If so, we play our bat hitting ball sound.

  for (var j = 0; j < 20; j++) {
    var rotQuat = animationData.joints[j].slice(0, 4)
    var transQuat = animationData.joints[j].slice(4, 8)

    gl.uniform4fv(boneRotQuaternions[j], rotQuat)
    gl.uniform4fv(boneTransQuaternions[j], transQuat)
  }

We push all of our interpolate dual quaternions to the GPU to be used during dual quaternion linear blending.

  var modelMatrix = [1, 0, 0, 0, 0, 1, 0,
    0, 0, 0, 1, 0, 0, 0, 0, 1]
  var nMatrix = glMat3.fromMat4([], modelMatrix)

  var camera = glMat4.create()
  glMat4.translate(camera, camera, [0, 0, 2.5])
  var yAxisCameraRot = glMat4.create()
  var xAxisCameraRot = glMat4.create()
  glMat4.rotateX(xAxisCameraRot, xAxisCameraRot, -xCamRot)
  glMat4.rotateY(yAxisCameraRot, yAxisCameraRot, yCamRot)
  glMat4.multiply(camera, xAxisCameraRot, camera)
  glMat4.multiply(camera, yAxisCameraRot, camera)
  glMat4.lookAt(camera, [camera[12], camera[13], camera[14]],
                [0, 0, 0], [0, 1, 0])
  var mVMatrix = glMat4.multiply([], camera, modelMatrix)

  gl.uniformMatrix3fv(nMatrixUni, false, nMatrix)
  gl.uniformMatrix4fv(mVMatrixUni, false, mVMatrix)

  // Once our texture has loaded we begin drawing our model
  if (imageHasLoaded) {
    gl.drawElements(
      gl.TRIANGLES,
      baseballPlayer.positionIndices.length, gl.UNSIGNED_SHORT, 0)
  }

  window.requestAnimationFrame(draw)
}

Here we set up our camera and render our model.

var audio = new window.Audio()
audio.crossOrigin = 'anonymous'
audio.src = 'bat-hit-ball.mp3'

var context = new window.AudioContext()
var analyzer = context.createScriptProcessor(1024, 1, 1)
var source = context.createMediaElementSource(audio)
var gainNode = context.createGain()

source.connect(analyzer)
analyzer.connect(gainNode)
gainNode.connect(context.destination)

Next we take advantage of the Web Audio API to create our volume bars display. We’ll use an analyzer node to retrieve all of the sound that is being emitted at different time samples. We’ll take the loudest sound and use that to determine how many bars to color.

var muted = true
var hasClickedBefore = false
var muteButton = document.createElement('button')
muteButton.innerHTML = 'Click to un-mute'
muteButton.style.cursor = 'pointer'
muteButton.style.marginRight = '10px'
muteButton.style.marginLeft = '10px'
muteButton.style.width = '100px'
muteButton.style.height = '40px'
muteButton.onclick = function () {
  if (!hasClickedBefore) {
    hasClickedBefore = true
    gainNode.gain.value = 0
    audio.play()
    setTimeout(function () {
      gainNode.gain.value = 1
    }, 500)
  }

  muted = !muted

  muteButton.innerHTML = muted ? 'Click to un-mute' :
    'Click to mute'
}

var playbackSlider = document.createElement('input')
playbackSlider.type = 'range'
playbackSlider.max = 100
playbackSlider.min = 0
playbackSlider.value = 85

var playbackSpeed = 0.85

var playbackDisplay = document.createElement('div')
playbackDisplay.innerHTML = 'Playback: 85%'

playbackSlider.oninput = function (e) {
  playbackSpeed = e.target.value / 100
  playbackDisplay.innerHTML = 'Playback: ' + e.target.value + '%'
}

var volumeBarContainer = document.createElement('div')
volumeBarContainer.style.display = 'flex'
volumeBarContainer.style.display = 'flex'
var volumeBars = []
for (var k = 0; k < 5; k++) {
  var volumeBar = document.createElement('div')
  volumeBar.style.width = '40px'
  volumeBar.style.height = '40px'
  volumeBar.style.border = 'solid #333 1px'
  volumeBar.style.transition = '0.9s ease background-color'
  volumeBar.style.marginRight = '2px'
  volumeBars.push(volumeBar)
  volumeBarContainer.appendChild(volumeBar)
}

Add our controls and volume display into the page.

analyzer.onaudioprocess = function (e) {
  var out = e.outputBuffer.getChannelData(0)
  var input = e.inputBuffer.getChannelData(0)
  var max = 0

  for (var i = 0; i < input.length; i++) {
    out[i] = input[i]
    max = input[i] > max ? input[i] : max
  }

  var volume = max * 50
  for (var j = 0; j < 5; j++) {
    if (j < volume) {
      volumeBars[j].style.backgroundColor = 'red'
    } else {
      volumeBars[j].style.backgroundColor = 'white'
    }
  }
}

As mentioned above, we use the audio analyzer to determine how many bars to render in our current volume display.

var controls = document.createElement('span')
controls.style.display = 'flex'
controls.style.marginBottom = '5px'
controls.style.alignItems = 'center'

var mountLocation = document.getElementById(
  'webgl-skeletal-sound-tutorial'
) || document.body
controls.appendChild(playbackSlider)
controls.appendChild(muteButton)
controls.appendChild(volumeBarContainer)
mountLocation.appendChild(controls)
mountLocation.appendChild(playbackDisplay)
mountLocation.appendChild(canvas)

draw()

And lastly we dump our controls into the page and begin our request animation frame render loop.

Whoa, you did it!

Congrats! What would you like to learn about next? Let me know on Twitter.

‘Til next time,

- CFN

WebGL Tutorial: Directional Shadow Mapping without extensions

Wed, 23 Aug 2017 06:00:00 +0000

Welcome back to another WebGL tutorial 🎉

In this tutorial we’ll introduce the concepts behind real time directional light shadow mapping and then walk through a full working example implementation.

The apple blocks the light, causing a shadow

We’ll be using a color texture technique that maximizes our compatability with different machines and mobile devices.

By the end of the tutorial you’ll hopefully walk away with an understanding of the core ideas behind shadow mapping as well as a reference implementation should you decide to implement shadow mapping yourself.

Achieving maximum compatability

While we won’t be using it this time around, the easiest way to implement shadow mapping in WebGL is by taking advantage of the WEBGL_depth_texture extension.

In a nutshell, WEBGL_depth_texture gives you easy access to the depth buffer.

The depth buffer helps you know when something is behind or in front of something else. This is the key information that powers shadow mapping, so being able to access the depth buffer easily is advantageous.

The problem with WEBGL_depth_texture, however, is that the extension might not be supported on your user’s device.

WEBGL_depth_texture support according to webglstats.com (probably even lower)

Luckily, with some workarounds you can accomplish shadow mapping without using WEBGL_depth_texture, and that is exactly what we’ll be doing in this tutorial.

In a real application you might prefer to take advantage of WEBGL_depth_texture, but still use the approach that we’ll learn in this tutorial as a fallback for when your user’s device doesn’t support the WEBGL_depth_texture.

What is a shadow?

Take a second and think of the last time you were outside on a sunny day (unless you happen to live in a cave, in which case how do you have wifi?).

Where does your shadow get cast? If you think carefully you’ll notice that you’re always right in between the sun and your shadow. Your shadow occurs wherever the sun’s light hits you before it has a chance to hit the ground.

click and drag to move the camera

It turns out that we can model this phenomenon in computer graphics.

Step one is to render the scene from the sun’s point of view. In our demo the sun can see the dragon, but it can’t see that piece of ground behind the dragon that ends up in the shadow.

Everythng behind this orthographic projection is in shadow

We encode the depth of the fragments of our dragon into a color texture. Later when we render the entire scene we’ll check where each fragment in our camera’s scene would have been if we had drawn it from the light’s view.

We encode object’s distances from the light’s view into a texture

If the fragment would have been behind the first thing that the light saw, it is in the shadow so we’ll render it a bit darker.

How we create our color texture

Here are the steps that we’ll take:

We render the scene from the light’s point of view in order to know what objects are blocking its rays.
We encode the distance of everything that the light sees into a texture. Every color in our texture represents some float distance away from the camera between 0.0 and 1.0. The texture ends up looking something like this:

We encode object’s distances from the light’s view into a texture
Later when we render our scene from our camera’s perspective, we use this texture to figure out whether or not the light can see the fragment in question. If it can’t, then the fragment is in the shadow so we darken it.

Now that we have a better sense of what we’ll be doing, let’s quickly grab our dependencies and start writing some code!

Downloading our dependencies

gl-mat4 helps us handle our matrix math, budo is our development server that helps us view our demo locally and stanford-dragon gives us the vertex data that we use to draw our demo dragon.

mkdir webgl-shadow-map-tutorial
cd webgl-shadow-map-tutorial
touch tutorial.js
npm install gl-mat4@1.1.4 budo@10.0.4 \
stanford-dragon@1.1.1

Now that we’ve downloaded our dependences, from here on out you will no longer need an internet connection.

Building our shadow map demo

We’re going to step through the creation of our demo. You’ll learn the most if you type along as you go, but you can also find the full shadow mapping source code on GitHub.

Open up the tutorial.js that you created above and let’s get started!

First we create our canvas and insert it into the page. We’ll later draw our scene onto this canvas every request animation frame.

var glMat4 = require('gl-mat4')
var stanfordDragon = require('stanford-dragon/4')

var canvas = document.createElement('canvas')
canvas.width = 500
canvas.height = 500
var mountLocation = document.getElementById(
  'webgl-shadow-map-tut'
) || document.body
mountLocation.appendChild(canvas)

Next we grab our WebGL context and enable depth testing.

We need depth testing because we need to be able to figure out if a fragment is in front of or behind another fragment.

var gl = canvas.getContext('webgl')
gl.enable(gl.DEPTH_TEST)

As we’ve done in the past, we set up some controls so that we can move our camera around by clicking and dragging our mouse or finger.

var canvasIsPressed = false
var xRotation = Math.PI / 20
var yRotation = 0
var lastPressX
var lastPressY
canvas.onmousedown = function (e) {
  canvasIsPressed = true
  lastPressX = e.pageX
  lastPressY = e.pageY
}
canvas.onmouseup = function () {
  canvasIsPressed = false
}
canvas.onmouseout = function () {
  canvasIsPressed = false
}
canvas.onmousemove = function (e) {
  if (canvasIsPressed) {
    xRotation += (e.pageY - lastPressY) / 50
    yRotation -= (e.pageX - lastPressX) / 50

    xRotation = Math.min(xRotation, Math.PI / 2.5)
    xRotation = Math.max(xRotation, 0.1)

    lastPressX = e.pageX
    lastPressY = e.pageY
  }
}

canvas.addEventListener('touchstart', function (e) {
  lastPressX = e.touches[0].clientX
  lastPressY = e.touches[0].clientY
})
canvas.addEventListener('touchmove', function (e) {
  e.preventDefault()
  xRotation += (e.touches[0].clientY - lastPressY) / 50
  yRotation -= (e.touches[0].clientX - lastPressX) / 50

  xRotation = Math.min(xRotation, Math.PI / 2.5)
  xRotation = Math.max(xRotation, 0.1)

  lastPressX = e.touches[0].clientX
  lastPressY = e.touches[0].clientY
})

Next we create the vertex shader that allows up to render the scene from the light’s viewpoint.

Notice our shadow depth color texture size is 1024x1024. Super old mobile devices from a decade ago such as the original iPhone have a limit of 1024x1024 textures. Phones from a few years ago might have a limit of 2048x2048. An iPhone 7 has a limit of 4096x4096.

But you don’t want to just choose the largest limit available. The larger your texture the more work your GPU will need to do to render it every frame.

The smaller your texture, the more crude your shadows look because you couldn’t fit enough information into your texture so many neighboring fragments end up using data that wasn’t truly meant for them.

I like to default to 512 and go up or down from there, but it comes down to the visual quality vs. the performance tradeoff of your application.

var shadowDepthTextureSize = 1024

var lightVertexGLSL = `
attribute vec3 aVertexPosition;

uniform mat4 uPMatrix;
uniform mat4 uMVMatrix;

void main (void) {
  gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);
}
`

Next we create out light’s fragment shader. It gets the depth into the scene of every fragment that it sees, encodes this depth into a color, and then writes this encoded distance color onto a texture.

var lightFragmentGLSL = `
precision mediump float;

vec4 encodeFloat (float depth) {
  const vec4 bitShift = vec4(
    256 * 256 * 256,
    256 * 256,
    256,
    1.0
  );
  const vec4 bitMask = vec4(
    0,
    1.0 / 256.0,
    1.0 / 256.0,
    1.0 / 256.0
  );
  vec4 comp = fract(depth * bitShift);
  comp -= comp.xxyz * bitMask;
  return comp;
}

void main (void) {
  gl_FragColor = encodeFloat(gl_FragCoord.z);
}
`

Our camera renders our scene, but also takes in the uniforms for our light’s projection and view matrices. We use these to determine where this vertex would have been if the light was looking at it.

Knowing where the light would have thought this vertex was allows us to look at the color that this vertex has in our light’s depth color texture.

For example, if the light has the closest distance when looking directly at this vertex encoded as a 0.2, but our current vertex would have been a 0.8 depth in the lights view, we know to darken it.

var cameraVertexGLSL = `
attribute vec3 aVertexPosition;

uniform mat4 uPMatrix;
uniform mat4 uMVMatrix;
uniform mat4 lightMViewMatrix;
uniform mat4 lightProjectionMatrix;

const mat4 texUnitConverter = mat4(0.5, 0.0, 0.0, 0.0, 0.0, 0.5, 
0.0, 0.0, 0.0, 0.0, 0.5, 0.0, 0.5, 0.5, 0.5, 1.0);

varying vec2 vDepthUv;
varying vec4 shadowPos;

void main (void) {
  gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);

  shadowPos = texUnitConverter * lightProjectionMatrix *
  lightMViewMatrix * vec4(aVertexPosition, 1.0);
}
`

Now that our vertex shader has figured out the coordinates of all of our vertices were they to be viewed from the light’s viewpoint, our fragment shader goes through every fragment and asks “Am I behind or in front of the first thing that the light saw?”

If a fragment is behind, we darken it.

We don’t just have a constant multiplier for dark or light though. Instead we use a technique called “Percentage Closer Filtering”.

In a nutshell, we loop through nearby fragments and find out on average how much all of them are in shadow.

This smooths out the edges of our shadow since with a limited resolution depth color texture some fragments at the edge might sample the wrong depth and thus lead to jagged edges.

var cameraFragmentGLSL = `
precision mediump float;

varying vec2 vDepthUv;
varying vec4 shadowPos;

uniform sampler2D depthColorTexture;
uniform vec3 uColor;

float decodeFloat (vec4 color) {
  const vec4 bitShift = vec4(
    1.0 / (256.0 * 256.0 * 256.0),
    1.0 / (256.0 * 256.0),
    1.0 / 256.0,
    1
  );
  return dot(color, bitShift);
}

void main(void) {
  vec3 fragmentDepth = shadowPos.xyz;
  float shadowAcneRemover = 0.007;
  fragmentDepth.z -= shadowAcneRemover;

  float texelSize = 1.0 / ${shadowDepthTextureSize}.0;
  float amountInLight = 0.0;

  for (int x = -1; x <= 1; x++) {
    for (int y = -1; y <= 1; y++) {
      float texelDepth = decodeFloat(texture2D(depthColorTexture,
      fragmentDepth.xy + vec2(x, y) * texelSize));
      if (fragmentDepth.z < texelDepth) {
        amountInLight += 1.0;
      }
    }
  }
  amountInLight /= 9.0;

  gl_FragColor = vec4(amountInLight * uColor, 1.0);
}
`

Next we link our two shader programs. Nothing too special going on here.

var cameraVertexShader = gl.createShader(gl.VERTEX_SHADER)
gl.shaderSource(cameraVertexShader, cameraVertexGLSL)
gl.compileShader(cameraVertexShader)

var cameraFragmentShader = gl.createShader(gl.FRAGMENT_SHADER)
gl.shaderSource(cameraFragmentShader, cameraFragmentGLSL)
gl.compileShader(cameraFragmentShader)

var cameraShaderProgram = gl.createProgram()
gl.attachShader(cameraShaderProgram, cameraVertexShader)
gl.attachShader(cameraShaderProgram, cameraFragmentShader)
gl.linkProgram(cameraShaderProgram)

var lightVertexShader = gl.createShader(gl.VERTEX_SHADER)
gl.shaderSource(lightVertexShader, lightVertexGLSL)
gl.compileShader(lightVertexShader)

var lightFragmentShader = gl.createShader(gl.FRAGMENT_SHADER)
gl.shaderSource(lightFragmentShader, lightFragmentGLSL)
gl.compileShader(lightFragmentShader)

var lightShaderProgram = gl.createProgram()
gl.attachShader(lightShaderProgram, lightVertexShader)
gl.attachShader(lightShaderProgram, lightFragmentShader)
gl.linkProgram(lightShaderProgram)

We initialize our vertex data for our dragon and our floor.

var floorPositions = [
  -30.0, 0.0, 30.0,
  30.0, 0.0, 30.0,
  30.0, 0.0, -30.0,
  -30.0, 0.0, -30.0
]
var floorIndices = [
  0, 1, 2, 0, 2, 3
]

var dragonPositions = stanfordDragon.positions
var dragonIndices = stanfordDragon.cells
dragonPositions = dragonPositions.reduce(function (all, vertex) {
  all.push(vertex[0] / 10)
  all.push(vertex[1] / 10)
  all.push(vertex[2] / 10)
  return all
}, [])
dragonIndices = dragonIndices.reduce(function (all, vertex) {
  all.push(vertex[0])
  all.push(vertex[1])
  all.push(vertex[2])
  return all
}, [])

Here we buffer our attribute data onto the GPU so that we can draw our floor and dragon.

var vertexPositionAttrib = gl.getAttribLocation(lightShaderProgram,
'aVertexPosition')
gl.enableVertexAttribArray(vertexPositionAttrib)

var dragonPositionBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, dragonPositionBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(dragonPositions),
gl.STATIC_DRAW)
gl.vertexAttribPointer(vertexPositionAttrib, 3, gl.FLOAT, false, 0,
0)

var dragonIndexBuffer = gl.createBuffer()
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, dragonIndexBuffer)
gl.bufferData(gl.ELEMENT_ARRAY_BUFFER,
new Uint16Array(dragonIndices), gl.STATIC_DRAW)

var floorPositionBuffer = gl.createBuffer()
gl.bindBuffer(gl.ARRAY_BUFFER, floorPositionBuffer)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(floorPositions),
gl.STATIC_DRAW)
gl.vertexAttribPointer(vertexPositionAttrib, 3, gl.FLOAT, false, 0,
0)

var floorIndexBuffer = gl.createBuffer()
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, floorIndexBuffer)
gl.bufferData(gl.ELEMENT_ARRAY_BUFFER,
new Uint16Array(floorIndices), gl.STATIC_DRAW)

This part is important for our color depth texture technique. We create a framebuffer that we can use to render our scene from our light’s viewpoint.

We set up a shadowDepthTexture variable that we’ll turn into a WebGL Texture. This texture will hold the output of our light’s shader program. Which means that it holds all of the encoded depth information of our scene.

gl.useProgram(lightShaderProgram)

var shadowFramebuffer = gl.createFramebuffer()
gl.bindFramebuffer(gl.FRAMEBUFFER, shadowFramebuffer)

var shadowDepthTexture = gl.createTexture()
gl.bindTexture(gl.TEXTURE_2D, shadowDepthTexture)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST)
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, shadowDepthTextureSize,
shadowDepthTextureSize, 0, gl.RGBA, gl.UNSIGNED_BYTE, null)

var renderBuffer = gl.createRenderbuffer()
gl.bindRenderbuffer(gl.RENDERBUFFER, renderBuffer)
gl.renderbufferStorage(gl.RENDERBUFFER, gl.DEPTH_COMPONENT16,
shadowDepthTextureSize, shadowDepthTextureSize)

gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0,
gl.TEXTURE_2D, shadowDepthTexture, 0)
gl.framebufferRenderbuffer(gl.FRAMEBUFFER, gl.DEPTH_ATTACHMENT,
gl.RENDERBUFFER, renderBuffer)

gl.bindTexture(gl.TEXTURE_2D, null)
gl.bindRenderbuffer(gl.RENDERBUFFER, null)

We create the orthographic projection matrix and view matrix for our light.

When shadow mapping a directional light you use an orthographic projection because you want to simulate parallel rays of light hitting your scene.

Put differently, because there is no perspective in an orthographic projection, only points that are directly behind the dragon will be considered in the shadow.

Contrast this with say, a spotlight, where a shadow typically grows as it gets farther away from the shadow caster. A spotlight shadow can be modeled with a perspective matrix.

Note that our method of creating an orthographic projection here is suboptimal! Here we just picked bounds that encompassed our scene, but in a resource constrained application you’ll want to dynamically calculate an orthographic projection that perfectly wraps your camera’s view frustrum (plus some offset if you want objects outside of the frustrum to be able to cast shadows. such as a tall tree in the distance).

If you’re intersted in learning more about this let me know!.

var lightProjectionMatrix = glMat4.ortho(
  [], -40, 40, -40, 40, -40.0, 80
)
var lightViewMatrix = glMat4.lookAt([], [0, 2, -3], [0, 0, 0],
[0, 1, 0])

Here we set up some of our uniform data for rendering.

The important piece is that we pass our texture uniform to our camera shader. This is how our camera has access to all of the depth information that we got from our light shader.

Our light wrote it’s depth information to a color texture, and our camera makes use of this color texture here.

var shadowPMatrix = gl.getUniformLocation(lightShaderProgram,
'uPMatrix')
var shadowMVMatrix = gl.getUniformLocation(lightShaderProgram,
'uMVMatrix')

gl.uniformMatrix4fv(shadowPMatrix, false, lightProjectionMatrix)
gl.uniformMatrix4fv(shadowMVMatrix, false, lightViewMatrix)

gl.bindBuffer(gl.ARRAY_BUFFER, floorPositionBuffer)
gl.vertexAttribPointer(vertexPositionAttrib, 3,
gl.FLOAT, false, 0, 0)

gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, floorIndexBuffer)
gl.bufferData(gl.ELEMENT_ARRAY_BUFFER,
new Uint16Array(floorIndices), gl.STATIC_DRAW)

gl.bindFramebuffer(gl.FRAMEBUFFER, null)

gl.useProgram(cameraShaderProgram)

var samplerUniform = gl.getUniformLocation(cameraShaderProgram,
'depthColorTexture')

gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, shadowDepthTexture)
gl.uniform1i(samplerUniform, 0)

var uMVMatrix = gl.getUniformLocation(cameraShaderProgram,
'uMVMatrix')
var uPMatrix = gl.getUniformLocation(cameraShaderProgram,
'uPMatrix')
var uLightMatrix = gl.getUniformLocation(cameraShaderProgram,
'lightMViewMatrix')
var uLightProjection = gl.getUniformLocation(cameraShaderProgram,
'lightProjectionMatrix')
var uColor = gl.getUniformLocation(cameraShaderProgram, 'uColor')

gl.uniformMatrix4fv(uLightMatrix, false, lightViewMatrix)
gl.uniformMatrix4fv(uLightProjection, false, lightProjectionMatrix)
gl.uniformMatrix4fv(uPMatrix, false, glMat4.perspective([],
Math.PI / 3, 1, 0.01, 900))

We create a function that will draw our shadow map. It first binds our framebuffer (which allows us to write to our dedicated color texture) and then renders our dragon.

We don’t need to render our floor since we know that our floor won’t need to cast any shadows.

var dragonRotateY = 0

function drawShadowMap () {
  dragonRotateY += 0.01

  gl.useProgram(lightShaderProgram)

  gl.bindFramebuffer(gl.FRAMEBUFFER, shadowFramebuffer)

  gl.viewport(0, 0, shadowDepthTextureSize, shadowDepthTextureSize)
  gl.clearColor(0, 0, 0, 1)
  gl.clearDepth(1.0)
  gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT)

  gl.bindBuffer(gl.ARRAY_BUFFER, dragonPositionBuffer)
  gl.vertexAttribPointer(vertexPositionAttrib, 3, gl.FLOAT, false,
  0, 0)
  gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, dragonIndexBuffer)

  var lightDragonMVMatrix = glMat4.create()
  glMat4.rotateY(lightDragonMVMatrix, lightDragonMVMatrix,
  dragonRotateY)
  glMat4.translate(lightDragonMVMatrix, lightDragonMVMatrix,
  [0, 0, -3])
  glMat4.multiply(lightDragonMVMatrix, lightViewMatrix,
  lightDragonMVMatrix)
  gl.uniformMatrix4fv(shadowMVMatrix, false, lightDragonMVMatrix)

  gl.drawElements(gl.TRIANGLES, dragonIndices.length,
  gl.UNSIGNED_SHORT, 0)

  gl.bindFramebuffer(gl.FRAMEBUFFER, null)
}

Here we use the shadowDepthTexture to draw the scene from our camera’s perspective.

function drawModels () {
  gl.useProgram(cameraShaderProgram)
  gl.viewport(0, 0, 500, 500)
  gl.clearColor(0.98, 0.98, 0.98, 1)
  gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT)

  var camera = glMat4.create()
  glMat4.translate(camera, camera, [0, 0, 45])
  var xRotMatrix = glMat4.create()
  var yRotMatrix = glMat4.create()
  glMat4.rotateX(xRotMatrix, xRotMatrix, -xRotation)
  glMat4.rotateY(yRotMatrix, yRotMatrix, yRotation)
  glMat4.multiply(camera, xRotMatrix, camera)
  glMat4.multiply(camera, yRotMatrix, camera)
  camera = glMat4.lookAt(camera,
  [camera[12], camera[13], camera[14]], [0, 0, 0], [0, 1, 0])

  var dragonModelMatrix = glMat4.create()
  glMat4.rotateY(dragonModelMatrix, dragonModelMatrix,
  dragonRotateY)
  glMat4.translate(dragonModelMatrix, dragonModelMatrix,
  [0, 0, -3])

  var lightDragonMVMatrix = glMat4.create()
  glMat4.multiply(lightDragonMVMatrix, lightViewMatrix,
  dragonModelMatrix)
  gl.uniformMatrix4fv(uLightMatrix, false, lightDragonMVMatrix)

  gl.uniformMatrix4fv(
    uMVMatrix,
    false,
    glMat4.multiply(dragonModelMatrix, camera, dragonModelMatrix)
  )

  gl.uniform3fv(uColor, [0.36, 0.66, 0.8])

  gl.activeTexture(gl.TEXTURE0)
  gl.bindTexture(gl.TEXTURE_2D, shadowDepthTexture)
  gl.uniform1i(samplerUniform, 0)

  gl.drawElements(gl.TRIANGLES, dragonIndices.length,
  gl.UNSIGNED_SHORT, 0)

  gl.bindBuffer(gl.ARRAY_BUFFER, floorPositionBuffer)
  gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, floorIndexBuffer)
  gl.vertexAttribPointer(vertexPositionAttrib, 3, gl.FLOAT, false,
  0, 0)

  gl.uniformMatrix4fv(uLightMatrix, false, lightViewMatrix)
  gl.uniformMatrix4fv(uMVMatrix, false, camera)
  gl.uniform3fv(uColor, [0.6, 0.6, 0.6])

  gl.drawElements(gl.TRIANGLES, floorIndices.length,
  gl.UNSIGNED_SHORT, 0)
}

And lastly we set things up so that our canvas re-renders every animation frame.

function draw () {
  drawShadowMap()
  drawModels()

  window.requestAnimationFrame(draw)
}
draw()

Now just enter ./node_modules/budo/bin/cmd.js tutorial.js --open --live to view your work.

Good stuff!

Just gimme the light!

Well done making it through - you now know how to implement shadow mapping!

There are many, many ways to improve your shadow maps such as front-face culling and calculating the perfect orthographic matrix to use for your light.

You can add in more and more techniques as you run into issues, but for most simple scenes the basics that you’ve learned here should have you covered!

What would you like to learn next? Let me know on Twitter :)

Til’ next time,

- CFN

WebGL Tutorial: Particle Effects via Billboards

Sat, 05 Aug 2017 11:00:00 +0000

Welcome back to another WebGL tutorial 🎉

This time around we’ll be learning to create particle effects by applying a technique called billboarding to a texture atlas of fire particles.

We’ll first explain the higher level concepts behind particle effects, and then we’ll step through a complete working example fire particle effect implementation.

You’ll hopefully walk away with an understanding of the core ideas behind particle effects and how to make use of them.

What is a particle effect?

In a nutshell, a particle effect is lots of smaller renderings that give you a desired look and feel when you put them all together.

In the canvas below we illustrate a simple fire particle effect. In our example the “fire” is actually 1000 little squares with small pictures of fire on them. We move them in different directions and when it all comes together it looks like one larger fire.

Click and drag to move the camera

Try turning billboarding on and off by clicking on the button above the canvas. Move the camera around and pay attention to how things look when billboarding is on vs. when billboarding is off.

Seeing that happens when billboarding is off vs. on should illustrate why billboarding is so important!

How does it work?

Each particle is a square composed of two triangles. On top of each square we paint a texture that looks like fire.

If our camera happens to be directly facing our particle then everything works as we’d expect.

Our camera is directly facing our particle and can see it

But a problem occurs when our camera begins to move. All of a sudden our camera is pointed at the side of our square and we can no longer see the fire.

Our camera has moved and it can’t see our particle

To resolve this, we’ll make our vertex shader move the four corners of our particle squares so that they’re always aligned with our camera.

We do this by getting the up and right vector for our now tilted camera and using them to place the four corners of our particle along the same plane as our tilted camera.

We move the particle’s corners to face the camera

Getting set up

Before we start coding we’ll set up our directory.

In your command line run the following:

mkdir webgl-particle-tutorial
cd webgl-particle-tutorial
touch tutorial.js

Next we install a couple of JavaScript modules.

gl-mat4 will help us with our matrix math.

We’ll use budo as a development server to view our work locally when we’re finished.

npm install gl-mat4@1.1.4 budo@10.0.4

Lastly we’ll download the texture atlas that we’ll use for our particle effect. Throughout the lifetime of a particle it will change from the first fire image to the 12th one.

curl -OL https://github.com/chinedufn/\
webgl-particle-effect-tutorial/raw/master/\
fire-texture-atlas.jpg > fire-texture-atlas.jpg

The fire texture atlas looks like this:

Alright now that you have all of your dependencies you will no longer need an internet connection for the rest of the tutorial.

Now that we’re all set up let’s write some shaders!

Let’s start coding!

Open your tutorial.js file. We’ll add code to it block by block, explaining it as we go.

You’ll likely learn the most if you type it out, but for convenience you can find the commented fire particle effect tutorial source code on GitHub.

var glMat4 = require('gl-mat4')

var billboardingEnabled = true

var canvas = document.createElement('canvas')
canvas.width = 500
canvas.height = 500
var mountLocation = document.getElementById(
  'webgl-particle-effect-tutorial'
) || document.body

First up we start by pulling in our gl-mat4 dependency that we’ll later use to position our camera. We then create the canvas that we’ll be drawing our particle effect onto.

var billboardButton = document.createElement('button')
billboardButton.innerHTML = 'Click to disable billboarding'
billboardButton.style.display = 'block'
billboardButton.style.cursor = 'pointer'
billboardButton.style.marginBottom = '3px'
billboardButton.style.height = '40px'
billboardButton.style.width = '160px'
billboardButton.onclick = function () {
  billboardingEnabled = !billboardingEnabled
  billboardButton.innerHTML = (
    billboardingEnabled ? 
      'Click to disable billboarding' :
      'Click to enable billboarding'
  )
}

Next we start creating the controls that power our little demo. Here we make the button that you can click to turn billboarding on and off. Seeing what happens when billboarding is turned off gives us a better idea of why it’s so useful to have.

mountLocation.appendChild(billboardButton)
mountLocation.appendChild(canvas)

Now that we have our button and canvas we insert them into the page.

var isDragging = false

var xRotation = 0
var yRotation = 0

var lastMouseX = 0
var lastMouseY = 0

canvas.onmousedown = function (e) {
  isDragging = true
  lastMouseX = e.pageX
  lastMouseY = e.pageY
}
canvas.onmousemove = function (e) {
  if (isDragging) {
    xRotation += (e.pageY - lastMouseY) / 50
    yRotation -= (e.pageX - lastMouseX) / 50

    xRotation = Math.min(xRotation, Math.PI / 2.5)
    xRotation = Math.max(xRotation, -Math.PI / 2.5)

    lastMouseX = e.pageX
    lastMouseY = e.pageY
  }
}
canvas.onmouseup = function (e) {
  isDragging = false
}

canvas.addEventListener('touchstart', function (e) {
  lastMouseX = e.touches[0].clientX
  lastMouseY = e.touches[0].clientY
})
canvas.addEventListener('touchmove', function (e) {
  e.preventDefault()
  xRotation += (e.touches[0].clientY - lastMouseY) / 50
  yRotation -= (e.touches[0].clientX - lastMouseX) / 50

  xRotation = Math.min(xRotation, Math.PI / 2.5)
  xRotation = Math.max(xRotation, -Math.PI / 2.5)

  lastMouseX = e.touches[0].clientX
  lastMouseY = e.touches[0].clientY
})

We want to be able to click and drag our canvas to move our camera, so we listen for onmousedown onmousemove and onmouseup to update our camera’s rotation about the world’s X and Y axes as we drag.

Similarly, we listen for touch events on the camera in order to move the camera as you drag your finger on a mobile device.

var gl = canvas.getContext('webgl')
gl.clearColor(0.0, 0.0, 0.0, 1.0)
gl.viewport(0, 0, 500, 500)

var vertexGLSL = `
uniform float uTime;

uniform vec3 uFirePos;

attribute float aLifetime;

attribute vec2 aTextureCoords;

attribute vec2 aTriCorner;

attribute vec3 aCenterOffset;

attribute vec3 aVelocity;

uniform mat4 uPMatrix;
uniform mat4 uViewMatrix;

uniform bool uUseBillboarding;

varying float vLifetime;
varying vec2 vTextureCoords;

void main (void) {
  float time = mod(uTime, aLifetime);

  vec4 position = vec4(
    uFirePos + aCenterOffset + (time * aVelocity),
    1.0
  );

  vLifetime = 1.3 - (time / aLifetime);
  vLifetime = clamp(vLifetime, 0.0, 1.0);
  float size = (vLifetime * vLifetime) * 0.05;

  if (uUseBillboarding) {
    vec3 cameraRight = vec3(
      uViewMatrix[0].x, uViewMatrix[1].x, uViewMatrix[2].x
    );
    vec3 cameraUp = vec3(
      uViewMatrix[0].y, uViewMatrix[1].y, uViewMatrix[2].y
    );

    position.xyz += (cameraRight * aTriCorner.x * size) +
     (cameraUp * aTriCorner.y * size);
  } else {
    position.xy += aTriCorner.xy * size;
  }

  gl_Position = uPMatrix * uViewMatrix * position;

  vTextureCoords = aTextureCoords;
  vLifetime = aLifetime;
}
`

Our vertex shader is the meat of our particle effect implementation.

We start by taking our current clock time modulus the lifetime of the particle. This makes our particles restart from the beginning of their motion whenever their life expires.

We then position our particle at the location of our fire, plus the offset from this fire location, plus the velocity of the particle times the amount of time that has elapsed in it’s lifetime simulation.

vLifetime is a number between 1 and 0 that decreases as the particle ages. We use it to size our particle proportionally to its age. It starts off large but then shrinks as it ages.

After that, if billboarding is turned on, we get the camera’s up and right direction in world space. Since we know the up and right direction of the camera we know the plane that the camera is facing. We use the billboards center position, the size of the billboard and the camera’s up and right vectors in order to position the current vertex along the plane that the camera is facing.

var fragmentGLSL = `
precision mediump float;

uniform vec4 uColor;

uniform float uTimeFrag;

varying float vLifetime;
varying vec2 vTextureCoords;

uniform sampler2D fireAtlas;

void main (void) {
  float time = mod(uTimeFrag, vLifetime);
  float percentOfLife = time / vLifetime;
  percentOfLife = clamp(percentOfLife, 0.0, 1.0);

  float offset = floor(16.0 * percentOfLife);
  float offsetX = floor(mod(offset, 4.0)) / 4.0;
  float offsetY = 0.75 - floor(offset / 4.0) / 4.0;

  vec4 texColor = texture2D(
    fireAtlas, 
    vec2(
      (vTextureCoords.x / 4.0) + offsetX,
      (vTextureCoords.y / 4.0) + offsetY
  ));
  gl_FragColor = uColor * texColor;

  gl_FragColor.a *= vLifetime;
}
`

As we simulate our particle our fragment shader calculates a number between 0 and 1 called percentOfLife. Based on the percentage of the particle’s life that has elapsed we choose one of the 16 textures in the texture atlas that we downloaded earlier.

Lastly we multiply gl_FragColor.a *= vLifetime;. This means that as vLifetime decreases the fragment fades away.

var vertexShader = gl.createShader(gl.VERTEX_SHADER)
gl.shaderSource(vertexShader, vertexGLSL)
gl.compileShader(vertexShader)

var fragmentShader = gl.createShader(gl.FRAGMENT_SHADER)
gl.shaderSource(fragmentShader, fragmentGLSL)
gl.compileShader(fragmentShader)

var shaderProgram = gl.createProgram()
gl.attachShader(shaderProgram, vertexShader)
gl.attachShader(shaderProgram, fragmentShader)
gl.linkProgram(shaderProgram)
gl.useProgram(shaderProgram)

In the usual boilerplate fashion here we compile the vertex and fragment shaders that we defined above.

var lifetimeAttrib = gl.getAttribLocation(
  shaderProgram, 'aLifetime'
)
var texCoordAttrib = gl.getAttribLocation(
  shaderProgram, 'aTextureCoords'
)
var triCornerAttrib = gl.getAttribLocation(
  shaderProgram, 'aTriCorner'
)
var centerOffsetAttrib = gl.getAttribLocation(
  shaderProgram, 'aCenterOffset'
)
var velocityAttrib = gl.getAttribLocation(
  shaderProgram, 'aVelocity'
)
gl.enableVertexAttribArray(lifetimeAttrib)
gl.enableVertexAttribArray(texCoordAttrib)
gl.enableVertexAttribArray(triCornerAttrib)
gl.enableVertexAttribArray(centerOffsetAttrib)
gl.enableVertexAttribArray(velocityAttrib)

And then we enable all of our vertex attributes.

var timeUni = gl.getUniformLocation(shaderProgram, 'uTime')
var timeUniFrag = gl.getUniformLocation(shaderProgram, 'uTimeFrag')
var firePosUni = gl.getUniformLocation(shaderProgram, 'uFirePos')
var perspectiveUni = gl.getUniformLocation(shaderProgram, 'uPMatrix')
var viewUni = gl.getUniformLocation(shaderProgram, 'uViewMatrix')
var colorUni = gl.getUniformLocation(shaderProgram, 'uColor')
var fireAtlasUni = gl.getUniformLocation(shaderProgram, 'uFireAtlas')
var useBillboardUni = gl.getUniformLocation(
  shaderProgram, 'uUseBillboarding'
)

Followed by getting the location of all of our uniforms so that we can later pass uniform data to the GPU.

var imageIsLoaded = false
var fireTexture = gl.createTexture()
var fireAtlas = new window.Image()
fireAtlas.onload = function () {
  gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true)
  gl.bindTexture(gl.TEXTURE_2D, fireTexture)
  gl.texImage2D(
    gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, fireAtlas
  )
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
  imageIsLoaded = true
}
fireAtlas.src = 'fire-texture-atlas.jpg'

In order to render our particles we need to have a texture. We load up our fire texture atlas and then create a WebGLTexture using the atlas.

var numParticles = 1000
var lifetimes = []
var triCorners = []
var texCoords = []
var vertexIndices = []
var centerOffsets = []
var velocities = []

var triCornersCycle = [
  -1.0, -1.0,
  1.0, -1.0,
  1.0, 1.0,
  -1.0, 1.0
]
var texCoordsCycle = [
  0, 0,
  1, 0,
  1, 1,
  0, 1
]

for (var i = 0; i < numParticles; i++) {
  var lifetime = 8 * Math.random()

  var diameterAroundCenter = 0.5
  var halfDiameterAroundCenter = diameterAroundCenter / 2

  var xStartOffset = diameterAroundCenter * 
    Math.random() - halfDiameterAroundCenter
  xStartOffset /= 3

  var yStartOffset = diameterAroundCenter * 
    Math.random() - halfDiameterAroundCenter
  yStartOffset /= 10

  var zStartOffset = diameterAroundCenter * 
    Math.random() - halfDiameterAroundCenter
  zStartOffset /= 3

  var upVelocity = 0.1 * Math.random()

  var xSideVelocity = 0.02 * Math.random()
  if (xStartOffset > 0) {
    xSideVelocity *= -1
  }

  var zSideVelocity = 0.02 * Math.random()
  if (zStartOffset > 0) {
    zSideVelocity *= -1
  }

  for (var j = 0; j < 4; j++) {
    lifetimes.push(lifetime)

    triCorners.push(triCornersCycle[j * 2])
    triCorners.push(triCornersCycle[j * 2 + 1])

    texCoords.push(texCoordsCycle[j * 2])
    texCoords.push(texCoordsCycle[j * 2 + 1])
    
    centerOffsets.push(xStartOffset)
    centerOffsets.push(yStartOffset + Math.abs(xStartOffset / 2.0))
    centerOffsets.push(zStartOffset)

    velocities.push(xSideVelocity)
    velocities.push(upVelocity)
    velocities.push(zSideVelocity)
  }

  vertexIndices = vertexIndices.concat([
    0, 1, 2, 0, 2, 3
  ].map(function (num) { return num + 4 * i }))
}

We create all of our particles and randomly choose how each one will move, where it will begin its life and how long it will live.

Each particle’s four corners are positioned identically relative to the particle’s center. So triCornersCycle gets re-used by every particle in order to position it’s four corners around it’s center.

function createBuffer (bufferType, DataType, data) {
  var buffer = gl.createBuffer()
  gl.bindBuffer(gl[bufferType], buffer)
  gl.bufferData(gl[bufferType], new DataType(data), gl.STATIC_DRAW)
  return buffer
}
createBuffer('ARRAY_BUFFER', Float32Array, lifetimes)
gl.vertexAttribPointer(lifetimeAttrib, 1, gl.FLOAT, false, 0, 0)

createBuffer('ARRAY_BUFFER', Float32Array, texCoords)
gl.vertexAttribPointer(texCoordAttrib, 2, gl.FLOAT, false, 0, 0)

createBuffer('ARRAY_BUFFER', Float32Array, triCorners)
gl.vertexAttribPointer(triCornerAttrib, 2, gl.FLOAT, false, 0, 0)

createBuffer('ARRAY_BUFFER', Float32Array, centerOffsets)
gl.vertexAttribPointer(centerOffsetAttrib, 3, gl.FLOAT, false, 0, 0)

createBuffer('ARRAY_BUFFER', Float32Array, velocities)
gl.vertexAttribPointer(velocityAttrib, 3, gl.FLOAT, false, 0, 0)

createBuffer('ELEMENT_ARRAY_BUFFER', Uint16Array, vertexIndices)

gl.enable(gl.BLEND)
gl.blendFunc(gl.ONE, gl.ONE)

gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, fireTexture)
gl.uniform1i(fireAtlasUni, 0)

gl.uniformMatrix4fv(
  perspectiveUni, 
  false, 
  glMat4.perspective([], Math.PI / 3, 1, 0.01, 1000)
)

First we push all of our particle attribute data to the GPU so that we can later draw our particles.

The important piece here is that we set gl.enable(gl.BLEND) and gl.blendFunc(gl.ONE, gl.ONE).

If you remember how the texture atlas looks, each fire particle square is some red in the center surrounded by black.

We want the red to show up as we render our particle effect, but we don’t want any of the black background. If we had rendered the black background it would overlap the particle squads behind it and we’d see something like this:

Improper blending

There are many, many ways to handle blending. Here we’re more or less saying:

(Update June 5, 2020) The blending explanation directly below this update is misguided. If you look at the demo above you will see that the center of the flame is white.

That is because those particles are being added with the other densely packed particles at the center of the flame behind them, leading to large color channel values that the GPU clamps back down to rgb(255, 255, 255).

A potentially better strategy (depending on what you’re aiming for) could be to discard fragments with color.a == 0 in the fragment shader.

Add our particle’s color with the color of whatever is behind it. So… If our particle color is black… (0, 0, 0) + whatever is behind it = whatever is behind it. So all black parts of our texture atlas are effectively transparent.

function createCamera () {
  var camera = glMat4.create()

  glMat4.translate(camera, camera, [0, 0.25, 1])

  var xAxisRotation = glMat4.create()
  var yAxisRotation = glMat4.create()
  glMat4.rotateX(xAxisRotation, xAxisRotation, -xRotation)
  glMat4.rotateY(yAxisRotation, yAxisRotation, yRotation)
  glMat4.multiply(camera, xAxisRotation, camera)
  glMat4.multiply(camera, yAxisRotation, camera)

  var cameraPos = [camera[12], camera[13], camera[14]]
  glMat4.lookAt(camera, cameraPos, redFirePos, [0, 1, 0])

  return camera
}

Next we create our view matrix so that we can render everything relative to our camera that we control by dragging our canvas.

var previousTime = new Date().getTime()

var clockTime = 3

var redFirePos = [0.0, 0.0, 0.0]
var redFireColor = [0.8, 0.25, 0.25, 1.0]

var purpFirePos = [0.5, 0.0, 0.0]
var purpFireColor = [0.25, 0.25, 8.25, 1.0]

function draw () {
  if (imageIsLoaded) {
    gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT)

    var currentTime = new Date().getTime()
    clockTime += (currentTime - previousTime) / 1000
    previousTime = currentTime

    gl.uniform1f(timeUni, clockTime)
    gl.uniform1f(timeUniFrag, clockTime)

    gl.uniformMatrix4fv(viewUni, false, createCamera())

    gl.uniform1i(useBillboardUni, billboardingEnabled)

    gl.uniform3fv(firePosUni, redFirePos)
    gl.uniform4fv(colorUni, redFireColor)

    gl.drawElements(gl.TRIANGLES, numParticles * 6, gl.UNSIGNED_SHORT, 0)

    gl.uniform3fv(firePosUni, purpFirePos)
    gl.uniform4fv(colorUni, purpFireColor)
    
    gl.drawElements(gl.TRIANGLES, numParticles * 6, gl.UNSIGNED_SHORT, 0)
  }

  window.requestAnimationFrame(draw)
}
draw()

In our final function we increment our simulation time every frame and pass that time into our vertex and fragment shader. Then we draw our red flame and our purple flame.

You made it! Now you can run ./node_modules/budo/bin/cmd.js tutorial.js --open --live and you should see your particle effect in your browser!

Great work!

You made it to the bottom. Pat yourself on the back.

What would you like to learn next? Let me know on Twitter :)

Til’ next time,

- CFN

Multitexturing a WebGL Terrain using a Blendmap

Sat, 15 Jul 2017 05:00:00 +0000

Welcome back to another WebGL tutorial 🎉

In this tutorial we’ll learn how to use a blendmap to multitexture a terrain.

Multitexturing is the practice of using more than one texture when rendering something. This allows you to create many different effects that you couldn’t achieve with just one texture.

In its most basic form, a blend map is an image that combines red, green, blue and black in order to encode the usage of four different textures. Each texture corresponding to one of those four colors.

You can create a blend map in any image editing tool such as Photoshop, but for this tutorial we’ll create our blend map using a custom 2d canvas tool that we’ll code up and then embed into our demo.

We’ll then use that blend map to multitexture a WebGL terrain on a separate 3d canvas.

You’ll hopefully walk away with an understanding of when you might use a blendmap, as well as a reference implementation for how to write and use a blendmap WebGL shader.

Why use a blendmap?

The big advantage to using a blendmap when texturing a terrain is that it makes it easy to tile multiple smaller textures across your large terrain.

You might use this technique to add paths, patches of dirt or other details to your terrain.

Let’s say that you want your terrain to be covered in mossy grass, with a few stone paths cutting through that moss. You want both of the moss and stone textures to tile, and you also want control over where they show up. How would you approach this?

One way would be to create a texture the size of your terrain that uses both your moss and stone texture. But what if your terrain is 10,000 x 10,000 pixels wide?

Today’s hardware won’t support a 10,000 x 10,000 sized texture, so you’d need to use a smaller texture image and then stretch it to fit your 10,000 x 10,000 terrain. This would lead to a much lower quality stretched texture on your terrain.

Alternatively, you can encode the location of the stone and moss and make your stone and moss textures tile themselves across anywhere on your terrain that they’ve been encoded. To do this we would use a third texture, our blendmap.

Our blendmap encodes which texture should be visible at which places on our terrain.

The interactive demo above will help us demonstrate. Click on the stone button and paint onto the painting canvas on the left.

The stone texture is being tiled on our WebGL terrain on the right canvas, and our blend map is encoding how much of the stone should show through. The color black tells our shader where to show the stone texture.

Now click on moss and paint on your blend map. You’re now using the color green in order to encode where the moss tiling should be shown.

Now drag the green slider to its halfway point and paint some more.

If you look closely you’ll see that your terrain is being painted with a mixture of stone and moss. This is because you’re using a mixture of black and green.

With blend maps you can encode exactly how much you want each texture to bleed through, allowing you to blend different textures! This is very useful for adding details such as paths that have a mix of dirt and grass.

With that out of the way, let’s jump into implementing the demo above.

Downloading our dependencies

Create a directory to hold the tutorial code.

mkdir blend-map-tutorial
cd blend-map-tutorial
touch tutorial.js

Download the texture atlas that we’ll use to texture our WebGL terrain.

curl -OL https://github.com/chinedufn/\
webgl-blend-map-tutorial/raw/master/terrain.jpg

We’ll install a development server so that we can view our tutorial in our browser, as well as a 4x4 matrix math library that we’ll use to create a perspective matrix.

npm install budo@10.0.3 gl-mat4@1.1.4

Implementing our blendmapped terrain

You’ll learn more if you type out the code as we go, but if you’d like a full, commented reference you can find all of the blend mapping source code on GitHub.

Open up tutorial.js and start by creating the canvas that we’ll be drawing onto and using as a blend map.

var blendCanvas = document.createElement('canvas')
blendCanvas.width = 256
blendCanvas.height = 256
blendCanvas.style.width = '256px'
blendCanvas.style.height = '256px'
blendCanvas.style.border = 'solid 1px rgba(0, 0, 0, 0.1)'

Next we set a variable that will hold the current brush color as we draw and change colors.

var brushColor = 'purple'

Next we create the slider controls that allow us to change the brush color and size. Dragging these controls will affect how much red, green and blue shows up in our image.

function createColorSlider () {
  var colorSlider = document.createElement('input')
  colorSlider.type = 'range'
  colorSlider.min = 0
  colorSlider.max = 255
  colorSlider.step = 1
  colorSlider.oninput = handleSliderChange

  return colorSlider
}
var redSlider = createColorSlider()
var blueSlider = createColorSlider()
var greenSlider = createColorSlider()

function handleSliderChange () {
  brushColor = 
   `rgb(${redSlider.value}, ${greenSlider.value}, ${blueSlider.value})`
  setColorDisplays()
}

function addSliderLabel (label, colorSlider) {
  var sliderLabel = document.createElement('label')
  sliderLabel.display = 'inline-block'
  sliderLabel.innerHTML = label

  var sliderWrapper = document.createElement('div')
  sliderWrapper.display = 'block'
  sliderWrapper.appendChild(sliderLabel)
  sliderWrapper.appendChild(colorSlider)

  return sliderWrapper
}

var brushSizeSlider = document.createElement('input')
brushSizeSlider.type = 'range'
brushSizeSlider.value = 20
brushSizeSlider.min = 0
brushSizeSlider.max = 40
brushSizeSlider.step = 1

var sliderContainer = document.createElement('div')
sliderContainer.appendChild(addSliderLabel('R', redSlider))
sliderContainer.appendChild(addSliderLabel('G', greenSlider))
sliderContainer.appendChild(addSliderLabel('B', blueSlider))
sliderContainer.appendChild(addSliderLabel('Brush Size', brushSizeSlider))

Next we create the display that shows us what color we currently have selected. This is composed of a little circular div that shows that has it’s background set to the current color, and then a span that shows the rgb(x, y, z) components of the current color.

function setColorDisplays () {
  colorDisplay.innerHTML = 
    `rgb(${redSlider.value}, ${greenSlider.value}, ${blueSlider.value})`
  swatchDisplay.style.backgroundColor = brushColor
}

var colorDisplay = document.createElement('span')
var swatchDisplay = document.createElement('div')
swatchDisplay.style.width = '20px'
swatchDisplay.style.height = '20px'
swatchDisplay.style.marginRight = '5px'
swatchDisplay.style.borderRadius = '50%'
setColorDisplays()

var colorDisplayContainer = document.createElement('div')
colorDisplayContainer.style.display = 'flex'
colorDisplayContainer.style.alignItems = 'center'
colorDisplayContainer.appendChild(swatchDisplay)
colorDisplayContainer.appendChild(colorDisplay)

Now let’s create the buttons that let us set black red green and blue as our brush color. These buttons directly correspond to our four textures - stone, lava, moss and water.

We also create the undo button. When we click this we’ll go back in time to our previous brush stroke.

var buttonContainer = document.createElement('div')
buttonContainer.style.display = 'flex'
buttonContainer.style.width = '500px'
buttonContainer.style.flexWrap = 'wrap'

function createColorButton (color, label) {
  var colorButton = document.createElement('button')

  colorButton.style.color = `rgb(${color.join(',')})`
  colorButton.style.fontSize = '24px'
  colorButton.style.cursor = 'pointer'
  colorButton.style.marginTop = '5px'
  colorButton.style.marginRight = '5px'
  colorButton.innerHTML = label
  colorButton.onclick = function () {
    brushColor = `rgb(${color.join(',')})`
    redSlider.value = color[0]
    greenSlider.value = color[1]
    blueSlider.value = color[2]
    setColorDisplays()
  }

  buttonContainer.appendChild(colorButton)
}

var undoButton = document.createElement('button')
undoButton.style.cursor = 'pointer'
undoButton.style.marginRight = '5px'
undoButton.innerHTML = 'Undo'
undoButton.onclick = undo
buttonContainer.appendChild(undoButton)

createColorButton([0, 0, 0], 'stone')
createColorButton([255, 0, 0], 'lava')
createColorButton([0, 255, 0], 'moss')
createColorButton([0, 0, 255], 'water')

Now that we’ve made all of our controls we add them into the page.

var mountElem = document.querySelector('#webgl-blend-map-tutorial') 
|| document.body
mountElem.appendChild(buttonContainer)
mountElem.appendChild(sliderContainer)
mountElem.appendChild(colorDisplayContainer)

Next we add our painting canvas and our WebGL terrain canvas into the DOM. We’ve now inserted all of our demo elements into the page.

var webGLCanvas = document.createElement('canvas')
webGLCanvas.width = 512
webGLCanvas.height = 512
webGLCanvas.style.cursor = 'not-allowed'
mountElem.appendChild(webGLCanvas)

var canvasContainer = document.createElement('div')
canvasContainer.style.display = 'flex'
canvasContainer.appendChild(blendCanvas)
canvasContainer.appendChild(webGLCanvas)
mountElem.appendChild(canvasContainer)

In order to paint onto our canvas we need to get our painting canvas’ 2d context. Our canvas’ 2d context holds all of the state and methods that determine what gets drawn onto the canvas at any given time.

var canvas2dContext = blendCanvas.getContext('2d')
canvas2dContext.fillStyle = brushColor
canvas2dContext.fillRect(
  0, 0, canvas2dContext.canvas.width, canvas2dContext.canvas.height
)

Now we create the controls that allow us to paint. When we click and move our mouse we’ll add points to an array. When we draw our painting canvas every request animation frame we’ll draw and connect these points.

We’ll also keep track of the lastMouseReleaseIndices array. Whenever we release our mouse or finger we’ll add to this array. Then when we hit our undo button we’ll use the last element in this array to know how far back to go when we remove points that have been drawn onto our painting canvas.

var isPainting = false

function startPainting (e) {
  isPainting = true
  addPoint(
      e.pageX - blendCanvas.offsetLeft,
      e.pageY - blendCanvas.offsetTop,
      false,
      brushSizeSlider.value
  )
}

function movePaintbrush (e) {
  if (isPainting) {
    addPoint(
      (e.pageX || e.changedTouches[0].pageX) - blendCanvas.offsetLeft,
      (e.pageY || e.changedTouches[0].pageY) - blendCanvas.offsetTop,
      true,
      brushSizeSlider.value
    )
  }
}

var lastMouseReleaseIndices = []
function stopPainting () {
  lastMouseReleaseIndices.push(allPoints.length - 1)
  isPainting = false
}

Now that we have our painting functions set up we add event listeners to our painting canvas to start and stop painting as we click or touch it.

blendCanvas.addEventListener('mousedown', startPainting)
blendCanvas.addEventListener('touchstart', startPainting)
blendCanvas.addEventListener('mousemove', movePaintbrush)
blendCanvas.addEventListener('touchmove', movePaintbrush)
blendCanvas.addEventListener('mouseup', stopPainting)
blendCanvas.addEventListener('touchend', stopPainting)

Here we create our function that adds a new point to our canvas. If connect is true we will draw a line between a new point and the point before it. This would happen if you were dragging the mouse or your finger.

var allPoints = []
function addPoint (x, y, connectWithPrevious, size) {
  allPoints.push({
    x: x,
    y: y,
    connect: connectWithPrevious,
    color: brushColor,
    size: size
  })
}

Here we set up our undo function. It looks at the second to last time that we lifted our mouse or finger and removes all points that have come since that time.

function undo () {
  if (lastMouseReleaseIndices.length > 1) {
    allPoints = allPoints.slice(
      0,
      lastMouseReleaseIndices[lastMouseReleaseIndices.length - 2]
    )
  } else if (lastMouseReleaseIndices.length === 1) {
    allPoints = []
  }

  lastMouseReleaseIndices.pop()
}

Here we redraw our painting canvas. We’ll do this every request animation frame. Whenever we redraw our canvas we set our blendmap image to read from whatever we’ve drawn onto our canvas. This has the effect of live reloading our WebGL terrain as we paint onto our 2d canvas since our WebGL terrain uses this blendmap image.

function redrawPaintCanvas () {
  canvas2dContext.fillRect(
    0, 0, canvas2dContext.canvas.width, canvas2dContext.canvas.height
  )
  
  canvas2dContext.lineJoin = 'round'

  for (var i = 0; i < allPoints.length; i++) {
    canvas2dContext.beginPath()
    if (allPoints[i].connect && allPoints[i - 1]) {
      canvas2dContext.moveTo(allPoints[i - 1].x, allPoints[i - 1].y)
    } else {
      canvas2dContext.moveTo(allPoints[i].x - 1, allPoints[i].y)
    }

    canvas2dContext.lineTo(allPoints[i].x, allPoints[i].y)
    canvas2dContext.closePath()
    canvas2dContext.strokeStyle = allPoints[i].color
    canvas2dContext.lineWidth = allPoints[i].size
    canvas2dContext.stroke()
  }

  blendmapImage.src = blendCanvas.toDataURL()
}

We’re done with our 2d painting canvas and we can now move onto our WebGL terrain. We start by grabbing our WebGL context.

Similar to our 2d context, our WebGL context is our handle into controlling all of the state that the underlying OpenGL implementation will use when drawing to our WebGL canvas.

var gl = webGLCanvas.getContext('webgl')
gl.clearColor(0.0, 0.0, 0.0, 1.0)
gl.viewport(0, 0, 512, 512)

We create three arrays to hold our vertex positions, our vertex uvs (texture coordinates) and our vertex indices (the other of vertices to draw).

var positions = []
var indices = []
var uvs = []

Here is where we create our terrain. We’re making a grid of 64x64 tiles. The bottom left corner of the grid has a [0, 0] uv coordinate, and the top right corner of the grid has a [1, 1] uv coordinate. That it to say that our blend map stretches across our terrain.

We’ll end up multiplying these coordinates by 6 when applying our texture atlas in order to tile our textures.

var distanceAway = -85

var tileNum = 0

var numRowsColumns = 64
for (var y = 0; y < numRowsColumns; y++) {
  for (var x = 0; x < numRowsColumns; x++) {
    positions = positions.concat([
      x, y, distanceAway,
      1 + x, y, distanceAway,
      1 + x, 1 + y, distanceAway,
      x, 1 + y, distanceAway
    ])

    uvs = uvs.concat([
      x / numRowsColumns, y / numRowsColumns,
      (x + 1) / numRowsColumns, y / numRowsColumns,
      (x + 1) / numRowsColumns, (y + 1) / numRowsColumns,
      x / numRowsColumns, (y + 1) / numRowsColumns
    ])

    indices = indices.concat([
      0, 1, 2, 0, 2, 3
    ].map(function (num) {
      return num + (4 * tileNum)
    }))

    tileNum++
  }
}

Now we create our blend map texture and terrain texture buffers and add our blend map and terrain images to them.

var blendmapTexture = gl.createTexture()
var blendmapImage = new window.Image()
blendmapImage.src = blendCanvas.toDataURL()
blendmapImage.onload = function () {
  handleLoadedTexture(blendmapTexture, blendmapImage)
}

var terrainTexture = gl.createTexture()
var terrainImage = new window.Image()
terrainImage.crossOrigin = 'anonymous'
terrainImage.onload = function () {
  handleLoadedTexture(terrainTexture, terrainImage)
}
terrainImage.src = '/terrain.jpg'

var bothImagesLoaded = false
function handleLoadedTexture (texture, image) {
  gl.bindTexture(gl.TEXTURE_2D, texture)
  gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true)
  gl.texImage2D(
    gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image
  )
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST)
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST)
  gl.bindTexture(gl.TEXTURE_2D, null)

  if (bothImagesLoaded) {
    setupWebGLState()
  }
  bothImagesLoaded = true
}

Let’s create our vertex shader. It is responsible for positioning all of our vertices.

var vertexShader = `
attribute vec3 aVertexPosition;
attribute vec2 aTextureCoord;

varying vec2 vTextureCoord;

uniform mat4 uMVMatrix;
uniform mat4 uPMatrix;

void main (void) {
  gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);
  vTextureCoord = aTextureCoord;
}
`

Our fragment shader is where the meat of our blend mapping lives. We sample our blend map based on where our fragment is. We then get the color of at that point.

We calculate how much black, red, green and blue are in the color the we sampled from our blend map. These four weightings are then used to determine how much of our stone, lava, moss and water textures get added into our final color.

When sampling from our stone, lava, moss and water textures, we restrict our texture coordinates the the section of our texture atlas that has the relevant texture, and then multiply these texture coordinates by 6. This makes our texture tile six times.

This would mean that if you paint the entire blend map with one color, say, black, you’ll see the stone texture repeated 6 times.

var fragmentShader = `
precision mediump float;

varying vec2 vTextureCoord;

uniform sampler2D uBlendmapSampler;
uniform sampler2D uTerrainSampler;

void main (void) {
   vec4 blendColor = 
     texture2D(uBlendmapSampler, vec2(vTextureCoord.s, vTextureCoord.t));

   float numRepeats = 6.0;
   vec4 rockColor = texture2D(uTerrainSampler, 
     vec2(
       mod(vTextureCoord.s * 0.5 * numRepeats, 0.5) + 0.5, 
       mod(vTextureCoord.t * 0.5 * numRepeats, 0.5)
   ));
   vec4 lavaColor = texture2D(uTerrainSampler, 
     vec2(
       mod(vTextureCoord.s * 0.5 * numRepeats, 0.5) + 0.5, 
       mod(vTextureCoord.t * 0.5 * numRepeats, 0.5) - 0.5
   ));
   vec4 mossColor = texture2D(uTerrainSampler, 
     vec2(
       mod(vTextureCoord.s * 0.5 * numRepeats, 0.5), 
       mod(vTextureCoord.t * 0.5 * numRepeats, 0.5) - 0.5
   ));
   vec4 waterColor = texture2D(uTerrainSampler, 
   vec2(
     mod(vTextureCoord.s * 0.5 * numRepeats, 0.5), 
     mod(vTextureCoord.t * 0.5 * numRepeats, 0.5)
   ));

   float blackWeight = 1.0 - blendColor.x - blendColor.y - blendColor.z;

   gl_FragColor = rockColor * blackWeight +
   lavaColor * blendColor.x +
   mossColor * blendColor.y +
   waterColor * blendColor.z;
}
`

Now that we have our vertex and fragment shaders we compile them into a shader program.

var vert = gl.createShader(gl.VERTEX_SHADER)
var frag = gl.createShader(gl.FRAGMENT_SHADER)
gl.shaderSource(vert, vertexShader)
gl.compileShader(vert)
gl.shaderSource(frag, fragmentShader)
gl.compileShader(frag)
var shaderProgram = gl.createProgram()
gl.attachShader(shaderProgram, vert)
gl.attachShader(shaderProgram, frag)
gl.linkProgram(shaderProgram)
gl.useProgram(shaderProgram)

Our WebGL context is a state machine. In order to draw to the page, we must set up all of the relevant state that we need before we can start drawing. Here we push all of the data that we need to draw our terrain onto the GPU.

function setupWebGLState () {
  var vertexPositionBuffer = gl.createBuffer()
  var vertexIndexBuffer = gl.createBuffer()
  var vertexUVsBuffer = gl.createBuffer()

  gl.bindBuffer(gl.ARRAY_BUFFER, vertexPositionBuffer)
  gl.bufferData(
    gl.ARRAY_BUFFER, new Float32Array(positions), gl.STATIC_DRAW
  )
  gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, vertexIndexBuffer)
  gl.bufferData(
    gl.ELEMENT_ARRAY_BUFFER, new Uint16Array(indices), gl.STATIC_DRAW
  )

  gl.bindBuffer(gl.ARRAY_BUFFER, vertexUVsBuffer)
  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(uvs), gl.STATIC_DRAW)

  var vertexPositionAttribute = gl.getAttribLocation(
    shaderProgram, 'aVertexPosition'
  )
  gl.bindBuffer(gl.ARRAY_BUFFER, vertexPositionBuffer)
  gl.enableVertexAttribArray(vertexPositionAttribute)
  gl.vertexAttribPointer(
    vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0
  )

  var textureCoordAttribute = gl.getAttribLocation(
    shaderProgram, 'aTextureCoord'
  )
  gl.bindBuffer(gl.ARRAY_BUFFER, vertexUVsBuffer)
  gl.enableVertexAttribArray(textureCoordAttribute)
  gl.vertexAttribPointer(
    textureCoordAttribute, 2, gl.FLOAT, false, 0, 0
  )

  var terrainSamplerUniform = gl.getUniformLocation(
    shaderProgram, 'uTerrainSampler'
  )
  gl.activeTexture(gl.TEXTURE1)
  gl.bindTexture(gl.TEXTURE_2D, terrainTexture)
  gl.uniform1i(terrainSamplerUniform, 1)

  var blendmapSamplerUniform = gl.getUniformLocation(
    shaderProgram, 'uBlendmapSampler'
  )
  gl.activeTexture(gl.TEXTURE0)
  gl.bindTexture(gl.TEXTURE_2D, blendmapTexture)
  gl.uniform1i(blendmapSamplerUniform, 0)

  var mvMatrixUniform = gl.getUniformLocation(shaderProgram, 'uMVMatrix')
  gl.uniformMatrix4fv(
    mvMatrixUniform, 
    false, 
    [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, -32, -32, 0, 1]
  )

  var pMatrixUniform = gl.getUniformLocation(shaderProgram, 'uPMatrix')
  gl.uniformMatrix4fv(pMatrixUniform, 
    false, 
    require('gl-mat4/perspective')(
      [], Math.PI / 4, 400 / 400, 0.1, 1000
    )
  )

  gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, vertexIndexBuffer)
}

Lastly we start up a request animation frame loop that redraws our painting canvas and our WebGL terrain canvas every animation frame.

function drawBothCanvases () {
  redrawPaintCanvas()

  if (bothImagesLoaded) {
    gl.clear(gl.COLOR_BUFFER_BIT)
    gl.drawElements(
      gl.TRIANGLES, indices.length, gl.UNSIGNED_SHORT, 0
    )
  }

  window.requestAnimationFrame(drawBothCanvases)
}
drawBothCanvases()

And that’s all of the code! In your command line run your development server using ./node_modules/budo/bin/cmd.js --open --live --host=127.0.0.1 tutorial.js. You should now be viewing your work in your browser.

Well done!

You made it through. Great work. What would you like to learn next? Let me know on Twitter!

Til’ next time,

- CFN

Dual Quaternion Shader Explained Line by Line

Sun, 09 Jul 2017 05:50:00 +0000

One of the cornerstones of skeletal animation is a vertex shader that blends the influences of your model’s bones and applies this blended influence to your model’s vertices in order to move and animate them.

A “bone” is just a rotation and translation in model space, so there are many different ways to represent a bone.

The most common representation is as a transformation matrix that encodes a desired rotation and translation. In recent years, however, the Dual Quaternion has proven itself as a very useful alternative.

When I first started using dual quaternions I struggled to get the math working in my vertex shaders. Most of the papers that I found explained the concepts, but didn’t provide a reference vertex shader.

So the goal of this post is to be a reference that I wish that I had while I was learning. It’s meant for the next person in my shoes that gets stuck while implementing dual quaternion linear blending.

In this post we’ll step through a dual quaternion vertex shader and explain how it works.

Why use Dual Quaternions?

There are a few benefits to using dual quaternions, but the one that first pulled my interest was how well suited they are for shortest path rotation interpolation.

Interpolating bones via matrices

Interpolating bones via dual quaternions

Above are two side-by-side gifs of a human rig bending forward.

The left rig is represented using matrices. The right gif takes the left gif’s matrices and converts them into dual quaternions using mat4-to-dual-quat.

You can see that the left rigs torso stretches as it bends, but the right rig remains rigid.

This is because when you interpolate matrices you are not guaranteed to have a rigid transform or to interpolate rotations along the shortest path.

In contrast, interpolating rotations via dual quaternions will always guarantee a rigid transform and a shortest path rotation. Hence the perfectly rigid bending of the torso.

If you want to explore further, here’s how to view both demo’s locally:

git clone https://github.com/chinedufn\
collada-dae-parser
cd collada-dae-parser
npm install

# Run the matrix interpolation demo
git checkout fe7cba26
npm run demo

# Run the dual quaternion interpolation demo
git checkout 02cfd8
npm run demo

With that quick background out of the way, let’s break down an example vertex shader for dual quaternion based skeletal animation (dual quaternion linear blending).

The vertex shader

Below is our entire dual quaternion linear blending vertex shader. Note that it isn’t very optimized and we’re creating more variables within the shader than we really need. That’s fine, we just want to understand the core concepts.

attribute vec3 aVertexPosition;
attribute vec3 aVertexNormal;
attribute vec4 aJointIndex;
attribute vec4 aJointWeight;

uniform mat4 uMVMatrix;
uniform mat4 uPMatrix;
uniform mat3 uNMatrix;
uniform vec4 boneRotQuaternions[18];
uniform vec4 boneTransQuaternions[18];

void main (void) {
  vec4 weightedRotQuats =
    boneRotQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneRotQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneRotQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneRotQuaternions[int(aJointIndex.w)] * aJointWeight.w;

  vec4 weightedTransQuats =
    boneTransQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneTransQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneTransQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneTransQuaternions[int(aJointIndex.w)] * aJointWeight.w;

  float xRot = weightedRotQuat[0];
  float yRot = weightedRotQuat[1];
  float zRot = weightedRotQuat[2];
  float wRot = weightedRotQuat[3];

  float rotQuatMagnitude = sqrt(xRot * xRot + yRot * yRot
   + zRot * zRot + wRot * wRot);
  weightedRotQuat = weightedRotQuat / rotQuatMagnitude;
  weightedTransQuat = weightedTransQuat / rotQuatMagnitude;

  float xR = weightedRotQuat[0];
  float yR = weightedRotQuat[1];
  float zR = weightedRotQuat[2];
  float wR = weightedRotQuat[3];
  float xT = weightedTransQuat[0];
  float yT = weightedTransQuat[1];
  float zT = weightedTransQuat[2];
  float wT = weightedTransQuat[3];
  float t0 = 2.0 * (-wT * xR + xT * wR - yT * zR + zT * yR);
  float t1 = 2.0 * (-wT * yR + xT * zR + yT * wR - zT * xR);
  float t2 = 2.0 * (-wT * zR - xT * yR + yT * xR + zT * wR);
  mat4 weightedJointMatrix = mat4(
        1.0 - (2.0 * yR * yR) - (2.0 * zR * zR),
        (2.0 * xR * yR) + (2.0 * wR * zR),
        (2.0 * xR * zR) - (2.0 * wR * yR),
        0,
        (2.0 * xR * yR) - (2.0 * wR * zR),
        1.0 - (2.0 * xR * xR) - (2.0 * zR * zR),
        (2.0 * yR * zR) + (2.0 * wR * xR),
        0,
        (2.0 * xR * zR) + (2.0 * wR * yR),
        (2.0 * yR * zR) - (2.0 * wR * xR),
        1.0 - (2.0 * xR * xR) - (2.0 * yR * yR),
        0,
        t0,
        t1,
        t2,
        1
        );

  gl_Position = uPMatrix * uMVMatrix *
  weightedJointMatrix * vec4(aVertexPosition, 1.0);
  
  vec3 blendedNormal = uNMatrix * 
  (weightedMatrix * vec4(aVertexNormal, 0.0)).xyz;
}

Take one more moment to scan through the entire shader again, then we’ll break it down piece by piece.

Breaking it down

We begin by defining our attributes. aVertexPosition and aVertexNormal hold the positions and normals for the mesh that we’re animating.

attribute vec3 aVertexPosition;
attribute vec3 aVertexNormal;
attribute vec4 aJointIndex;
attribute vec4 aJointWeight;

When skinning, you’ll usually allow each vertex to be influenced by up to three or four different bones. In our shader we’re allowing a maximum of four bones to influence a vertex, hence aJointIndex and aJointWeight being vec4s.

aJointIndex holds up to four numbers between 0 - 17 that correspond to one of our 18 bones.

aJointWeight encodes the weighting of each bone, or the amount that it should affect our vertex.

For example - let’s look at a set of joint indices and weights.

// Example values for a vertex
aJointIndex = [0, 13, 5, 1]
aJointWeight = [0.6, 0.2, 0.1, 0.1]

In this example our vertex is being affected 60% by bone zero, 20% by bone thirteen, ten% by bone 5 and 10% by bone 1. So if bone zero is moved of rotated, 60% of that transformation ends up being applied to our vertex.

Next we define our uniforms.

uniform mat4 uMVMatrix;
uniform mat4 uPMatrix;
uniform mat3 uNMatrix;
uniform vec4 boneRotQuaternions[18];
uniform vec4 boneTransQuaternions[18];

uMVMatrix and uPMatrix are our model view matrix and perspective matrix. These are used to know where to draw our model in our 3d world. uNMatrix is used to transform our normals as our model rotates.

We encode the rotational and translation components of our dual quaternions into boneRotQuaternions and boneTransQuaternions. We split our dual quaternion into two components because OpenGL ES 2.0 doesn’t support 2x4 matrices (which would have held both the four rotational and four translational values in a dual quaternion).

Since our model has 18 bones, we make our dual quaternion arrays hold 18 sets of 8 values.

Next we blend the dual quaternions that affect this vertex.

void main (void) {
  vec4 weightedRotQuats = 
    boneRotQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneRotQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneRotQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneRotQuaternions[int(aJointIndex.w)] * aJointWeight.w;

  vec4 weightedTransQuats = 
    boneTransQuaternions[int(aJointIndex.x)] * aJointWeight.x +
    boneTransQuaternions[int(aJointIndex.y)] * aJointWeight.y +
    boneTransQuaternions[int(aJointIndex.z)] * aJointWeight.z +
    boneTransQuaternions[int(aJointIndex.w)] * aJointWeight.w;

Lines like boneRotQuaternions[int(aJointIndex.x)] * aJointWeight.x are saying “get the rotation component of the dual quaternion of the joint that corresponds do this index and then multiply it by the weighting for this joint.

Once we add all of these weightings up we have our final blended dual quaternion.

One beautiful thing about dual quaternions is that blending (combining) them really is this simple. We just add up the influences of each joint and we’re done.

Now before we move forward we normalize our dual quaternion. This ensures that we have a rigid transform, which might not have been the case after blending our dual quaternions.

Basically, if you don’t do this you will see very unpleasant artifacts.

  float xRot = weightedRotQuat[0];
  float yRot = weightedRotQuat[1];
  float zRot = weightedRotQuat[2];
  float wRot = weightedRotQuat[3];

  float rotQuatMagnitude = sqrt(xRot * xRot + yRot * yRot
   + zRot * zRot + wRot * wRot);
  weightedRotQuat = weightedRotQuat / rotQuatMagnitude;
  weightedTransQuat = weightedTransQuat / rotQuatMagnitude;

I created a bunch of two letter floats here, but if you look passed that all we’re doing is turning our dual quaternion into a 4x4 matrix. This makes it very easy to transform our vertex using our newly blended joint matrix since matrix * vertex multiplication is already built into GLSL.

  float xR = weightedRotQuat[0];
  float yR = weightedRotQuat[1];
  float zR = weightedRotQuat[2];
  float wR = weightedRotQuat[3];
  float xT = weightedTransQuat[0];
  float yT = weightedTransQuat[1];
  float zT = weightedTransQuat[2];
  float wT = weightedTransQuat[3];
  float t0 = 2.0 * (-wT * xR + xT * wR - yT * zR + zT * yR);
  float t1 = 2.0 * (-wT * yR + xT * zR + yT * wR - zT * xR);
  float t2 = 2.0 * (-wT * zR - xT * yR + yT * xR + zT * wR);
  mat4 weightedJointMatrix = mat4(
        1.0 - (2.0 * yR * yR) - (2.0 * zR * zR),
        (2.0 * xR * yR) + (2.0 * wR * zR),
        (2.0 * xR * zR) - (2.0 * wR * yR),
        0,
        (2.0 * xR * yR) - (2.0 * wR * zR),
        1.0 - (2.0 * xR * xR) - (2.0 * zR * zR),
        (2.0 * yR * zR) + (2.0 * wR * xR),
        0,
        (2.0 * xR * zR) + (2.0 * wR * yR),
        (2.0 * yR * zR) - (2.0 * wR * xR),
        1.0 - (2.0 * xR * xR) - (2.0 * yR * yR),
        0,
        t0,
        t1,
        t2,
        1
        );

As a quick side note - You could also just transform your vertex’s position using your dual quaternion, instead of converting it into a matrix first. While writing this tutorial I found an example dual quaternion shader without any matrix conversions. If I ever run into performance issues with converting to a matrix here I’ll give this dual quaternion only approach a try.

Lastly we get our gl_Position by multiplying our perspective, model view and blended joint matrix with our original vertex position.

  gl_Position = uPMatrix * uMVMatrix *
  weightedJointMatrix * vec4(aVertexPosition, 1.0);
  
  vec3 blendedNormal = uNMatrix * 
  (weightedMatrix * vec4(aVertexNormal, 0.0)).xyz;
}

You made it!

Great work reading through the shader. If you’d like to play with a running demo of a dual quaternion shader, you can check out the collada-dae-parser demo and code.

Til next time,

- CFN