Project 5

INSTRUCTION.md

@@ -290,20 +290,20 @@ UI is accessible anywhere as `cfg.enableEffect0`, etc.
 
 **Pass 1:** Renders the scene geometry and its properties to the g-buffers.
 * `copy.vert.glsl`, `copy.frag.glsl`
-* The framebuffer object `pass_copy.fbo` must be bound during this pass.
+* The framebuffer object `pass_copy.fbo1` must be bound during this pass.
 * Renders into `pass_copy.depthTex` and `pass_copy.gbufs[i]`, which need to be
   attached to the framebuffer.
 
 **Pass 2:** Performs lighting and shading into the color buffer.
 * `quad.vert.glsl`, `deferred/blinnphong-pointlight.frag.glsl`
 * Takes the g-buffers `pass_copy.gbufs`/`depthTex` as texture inputs to the
   fragment shader, on uniforms `u_gbufs` and `u_depth`.
-* `pass_deferred.fbo` must be bound.
-* Renders into `pass_deferred.colorTex`.
+* `pass_deferred.fbo1` must be bound.
+* Renders into `pass_deferred.colorTex1`.
 
 **Pass 3:** Performs post-processing.
 * `quad.vert.glsl`, `post/one.frag.glsl`
-* Takes `pass_deferred.colorTex` as a texture input `u_color`.
+* Takes `pass_deferred.colorTex1` as a texture input `u_color`.
 * Renders directly to the screen if there are no additional passes.
 
 More passes may be added for additional effects (e.g. combining bloom with

README.md

@@ -3,26 +3,35 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Ethan Brooks
+* Tested on: Google Chrome Version 54.0.2840.71 (64-bit)
+  OS X Sierra, Intel Iris Pro 1536 MB
 
-### Live Online
+![](https://github.com/lobachevzky/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/vid2.gif)
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+### Summary
+For this project I implemented part of a deferred in WebGL. A deferred shader is a way to optimize shading in scenes with many lights. All geometries relevant to the fragment shading are calculated in a single pass and stored in a buffer (the g-buffer). Subsequently the fragment shader iterates through the lights in the scene and accesses the geometries in the buffer to color fragments lit by the light. In this project, I implemented the fragment shader, using the Blinn-Phong shading method, and two additional feature: toon shading and motion blur. I also made some additional adjustments to the scissor frame to ensure more accurate scissor clipping.
 
-### Demo Video/GIF
+## Fragment shader
+This shader uses the Blinn-Phong method. The brightness of an object is proportional to the angle between the surface normal and another angle, calculated as the sum of the vector from surface to eye and from surface to light.
 
-[![](img/video.png)](TODO)
+## Toon shading
+Toon shading discretizes light shading by bucketing shade levels into discrete buckets. In the simplest case -- two shade levels -- all cells lit above a certain threshold are shaded with the same amount of brightness and similarly for all cells lit below that threshold. My implementation of toon shading also uses an edge detector to outline edges. The edge detector applies a convolution over the depth associated with each pixel, picking out pixels adjacent to significant depth change and coloring these black.
 
-### (TODO: Your README)
+![](https://github.com/lobachevzky/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/duck.gif)
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+## Motion blur
+Motion blur subsamples pixels over time in the direction of their velocity. In order to get the previous position of a pixel, I stored the screen-to-world transformation matrices for the previous turn. This transformation sends a point from screen space to world space. Using the previous turn's transformation matrix, I was able to calculate the points position in world space for the current and the previous turn and then calculate the difference between the two, yielding a velocity vector. I then subdivided this vector and subsampled pixels between the current and previous position. I noticed that the camera blur seemed to be a little jittery. In order to mitigate this, instead of storing a moving average of the previous frame's transformation matrix. This helped slightly, but I speculate that one flaw with this approach is that the inverse transformations from screen to world position tend to produce a lot of rounding errors. I suspect that a better looking, but less performant method would simply store the previous world position of each pixel from turn to turn. This would require storing a `vec4` for every pixel instead of simply storing a `mat4` as the current method does. That said, it might run faster since the blur mechanism would be spared some matrix inversions.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+![](https://github.com/lobachevzky/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/blur.gif)
 
+## Scissor optimization
+Since light attenuates with distance in our model, at a certain distance, light ceases to have any effect on an area. Consequently our shader only scans pixels within the light radius of any given light, establishing a rectangle around the edges of the circle cast by the light radius. The original implementation simply took the cube circumscribing the light sphere and rendered pixels lying between the upper left corner of this cube and the lower right. In a few edge cases, however, the resulting rectangle failed to fully enclose the light halo. To correct the error, I iterated through all eight corners of the circumscribing cube to calculate the actual minima and maxima along the x and y axes.
+
+![](https://github.com/lobachevzky/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/scissor.gif)
+
+## G-buffer optimization
+In order to optimize memory usage in the g-buffer, I pre-calculated surface normals, combining texture normals with surface normals before adding them to the g-buffer. The naive method stores texture normals and surface normals separately and combines them in the fragment shader. My optimization combines them first and stores the result in the g-buffer. As a result the total time of the deferred render pass dropped from 36.6 ms to 29.4 ms.
 
 ### Credits
 

glsl/copy.frag.glsl

@@ -10,11 +10,23 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
 void main() {
+    vec3 texnor = texture2D( u_normap, vec2(v_uv) ).xyz;
+    vec3 nor = applyNormalMap(v_normal, texnor);
     // TODO: copy values into gl_FragData[0], [1], etc.
     // You can use the GLSL texture2D function to access the textures using
     // the UV in v_uv.
 
     // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+     gl_FragData[0] = vec4( v_position, 1.0 );
+     gl_FragData[1] = vec4( nor, 1.0 );
+     gl_FragData[2] = texture2D( u_colmap, vec2(v_uv) );
 }

glsl/deferred/ambient.frag.glsl

@@ -16,12 +16,14 @@ void main() {
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
+
     // TODO: Extract needed properties from the g-buffers into local variables
+    vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
 
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    gl_FragColor = vec4(colmap * 0.1, 1);  // TODO: replace this
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,38 +2,53 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
+#define NUM_CEL_CUTS 7
+#define GB(i) (texture2D(u_gbufs[i], v_uv).xyz)
+#define ROUND(n) (floor((n) + 0.5))
 
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
-uniform float u_lightRad;
+uniform vec3 u_camPos;
+
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
+uniform float u_lightRad;
+uniform int u_toon;
+
 varying vec2 v_uv;
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
-    normap = normap * 2.0 - 1.0;
-    vec3 up = normalize(vec3(0.001, 1, 0.001));
-    vec3 surftan = normalize(cross(geomnor, up));
-    vec3 surfbinor = cross(geomnor, surftan);
-    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
-}
+const vec4 SKY_COLOR = vec4(0.01, 0.14, 0.42, 1.0);
 
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
-    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
-    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
+
     // TODO: Extract needed properties from the g-buffers into local variables
+    // These definitions are suggested for starting out, but you will probably want to change them.
+    vec3 viewPos = GB(0);     // World-space position
+    vec3 nor = GB(1);  // Normals of the geometry as defined, without normal mapping
+    vec3 colmap = GB(2);  // The color map - unlit "albedo" (surface color)
 
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
     if (depth == 1.0) {
-        gl_FragColor = vec4(0, 0, 0, 0);
+        gl_FragColor = SKY_COLOR;
         return;
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    vec3 posRelLight = u_lightPos - viewPos;
+    vec3 V = normalize(viewPos);
+    vec3 L = normalize(posRelLight);
+    vec3 H = normalize(L + V); // mid angle beteen light and viewer
+    float intensity = dot(nor, H);
+    float attenuation = max(0.0, 1.0 - (length(posRelLight) / u_lightRad));
+    float shading = attenuation * intensity;
+
+    if (u_toon == 1) {
+        float cuts = float(NUM_CEL_CUTS);
+        shading = ROUND(shading * cuts) / cuts;
+    }
+
+    gl_FragColor = shading * vec4(colmap * u_lightCol, 1.0);
 }

glsl/deferred/blur.frag.glsl

@@ -0,0 +1,61 @@
+#version 100
+
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 3
+#define NUM_SAMPLES 13
+
+varying vec2 v_uv;
+
+uniform mat4 prevScreen2World;
+uniform mat4 newScreen2World;
+uniform mat4 world2Screen;
+
+uniform sampler2D u_depth;
+uniform sampler2D u_color;
+
+
+vec4 getWorldPos(vec4 H, mat4 transform) {
+
+    // Transform by the view-projection inverse.
+    vec4 D = transform * H;
+
+    // Divide by w to get the view position.
+    return D / D.w;
+}
+
+void main() {
+
+    // Get the depth buffer value at this pixel.
+    float zOverW = texture2D(u_depth, v_uv).x; // why .x ?
+
+    // H is the viewport position at this pixel in the range -1 to 1.
+    vec2 rescaled_uv = vec2(v_uv * 2.0 - 1.0);
+
+    // Screen pos
+    vec4 H = vec4(rescaled_uv, zOverW, 1);
+
+    vec4 prevPos = getWorldPos(H, prevScreen2World);
+    vec4 newPos = getWorldPos(H, newScreen2World);
+
+    // Use this frame's position and last frame's to compute the pixel velocity.
+    vec2 velocity = (world2Screen * (newPos - prevPos)).xy;
+    velocity *= velocity;
+
+    // Get the initial color at this pixel.
+    vec4 color = texture2D(u_color, v_uv); // TODO: not the color, but the output of blinnphong
+
+    vec2 v_uv_shifted = v_uv;
+    for(int i = 1; i < NUM_SAMPLES; ++i) {
+        v_uv_shifted += velocity / float(NUM_SAMPLES) / 10.0;
+
+        // Sample the color buffer along the velocity vector.
+        vec4 currentColor = texture2D(u_color, v_uv_shifted); // TODO: sample from somewhere else.
+
+        // Add the current color to our color sum.
+        color += currentColor;
+    }
+    // Average all of the samples to get the final blur color.
+    gl_FragColor = color / float(NUM_SAMPLES);
+}

glsl/deferred/debug.frag.glsl

@@ -38,15 +38,15 @@ void main() {
     if (u_debug == 0) {
         gl_FragColor = vec4(vec3(depth), 1.0);
     } else if (u_debug == 1) {
-        // gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+         gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
     } else if (u_debug == 2) {
-        // gl_FragColor = vec4(abs(geomnor), 1.0);
+         gl_FragColor = vec4(abs(geomnor), 1.0);
     } else if (u_debug == 3) {
-        // gl_FragColor = vec4(colmap, 1.0);
+         gl_FragColor = vec4(colmap, 1.0);
     } else if (u_debug == 4) {
-        // gl_FragColor = vec4(normap, 1.0);
+         gl_FragColor = vec4(normap, 1.0);
     } else if (u_debug == 5) {
-        // gl_FragColor = vec4(abs(nor), 1.0);
+         gl_FragColor = vec4(abs(nor), 1.0);
     } else {
         gl_FragColor = vec4(1, 0, 1, 1);
     }

glsl/deferred/edges.frag.glsl

@@ -0,0 +1,39 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 3
+#define NUM_CEL_CUTS 7
+
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+void main() {
+    int width = 800;
+    int height = 600;
+
+    mat3 kernel = mat3(
+          1,   1,   1,
+          1,  -8,   1,
+          1,   1,   1
+    );
+
+    float conv = 0.0;
+    for (int i = -1; i <= 1; i++) {
+        for (int j = -1; j <= 1; j++) {
+            vec2 offset = vec2(i, j) / vec2(width, height);
+            vec4 depth_at_offset = texture2D(u_depth, v_uv + offset);
+            vec4 product_at_offset = depth_at_offset * float(kernel[i + 1][j + 1]);
+            float reduce_sum = dot(vec4(1), product_at_offset);
+            conv += reduce_sum;
+        }
+    }
+
+    float depth = texture2D(u_depth, v_uv).x;
+    if (abs(conv) > 1.0 && depth < 1.0) {
+       gl_FragColor = vec4(1) * .065;
+       gl_FragColor.w = 1.0;
+    }
+}

glsl/deferred/edges.frag.glsl.orig

@@ -0,0 +1,61 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 3
+#define NUM_CEL_CUTS 7
+#define gb(i) (texture2D(u_gbufs[i], v_uv).xyz)
+#define round(n) (floor((n) + 0.5))
+<<<<<<< HEAD
+#define color_at_offset(i, j) (texture2D(u_gbufs[2], v_uv + vec2(i, j)))
+#define PRODUCT_AT_OFFSET(i, j, kernel) (dot(vec4(1), color_at_offset(i, j) * float(kernel[i][j])))
+
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+=======
+#define color_at_offset(i, j) (texture2D(u_depth, v_uv + vec2(i, j) / vec2(width, height)))
+#define PRODUCT_AT_OFFSET(i, j, kernel) (dot(vec4(1), color_at_offset(i, j) * float(kernel[i][j])))
+
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+>>>>>>> edges
+
+varying vec2 v_uv;
+
+void main() {
+    int width = 800;
+    int height = 600;
+
+    mat2 gx = mat2(
+         1,  0,
+         0, -1
+    );
+
+    mat2 gy = mat2(
+        0, -1,
+        1,  0
+    );
+
+    float gx_conv = PRODUCT_AT_OFFSET(0, 0, gx) +
+                    PRODUCT_AT_OFFSET(0, 1, gx) +
+
+                    PRODUCT_AT_OFFSET(1, 0, gx) +
+                    PRODUCT_AT_OFFSET(1, 1, gx) ;
+
+    float gy_conv = PRODUCT_AT_OFFSET(0, 0, gy) +
+                    PRODUCT_AT_OFFSET(0, 1, gy) +
+
+                    PRODUCT_AT_OFFSET(1, 0, gy) +
+                    PRODUCT_AT_OFFSET(1, 1, gy) ;
+
+<<<<<<< HEAD
+    if (gx_conv + gy_conv > -1000.0) {
+       gl_FragColor = vec4(1, 0, 0, 0);
+    }
+
+    gl_FragColor = vec4(1, 0, 0, 0);
+=======
+    if (gx_conv + gy_conv > 0.5) {
+       gl_FragColor = vec4(0, 0, 0, 1);
+    }
+>>>>>>> edges
+}

glsl/red.frag.glsl

@@ -3,5 +3,5 @@ precision highp float;
 precision highp int;
 
 void main() {
-    gl_FragColor = vec4(1, 0, 0, 1);
+    gl_FragColor = vec4(1, 0, 0, 0.1);
 }