Project 5: Daniel Krupka

README.md

@@ -3,31 +3,73 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Daniel Krupka
+* Tested on: Debian testing (stretch), Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz 8GB, GTX 850M
 
-### Live Online
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+# About
+This is a WebGL deferred renderer, featuring
+* GLTF model loading
+* Scissor optimization
+* G-Buffer optimization
+* Bloom lighting, with single or two-pass Gaussian blur
+* Cel shading
 
-### Demo Video/GIF
+# Screenshots
+![Sponza, Full](img/sponza_full.png "Sponza, Full")
+![Duck, Full](img/duck.png "Duck, Full")
 
-[![](img/video.png)](TODO)
+# Base Performance
+## Lights and Scissor Optimization
+As expected, increasing the number of lights degrades the performance of the deferred lighting pass.
+This graph shows the percentage of each frame's processing time that was spent on each pass. The number
+of lights ranged from 20 to 250.
 
-### (TODO: Your README)
+![Sponza, no scissor, light plot](img/plot_lights.png "Sponza, no scissor, light plot")
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+The other passes (G-Buffer copying, framebuffer moving) were unaffected - they operate only on pixels,
+independent of object count.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+Rendering only the region of the framebuffer affected by each light theoretically and practically improve
+the lighting performance, though not by much.
 
+![Sponza, scissor vs no scissor](img/plot_scissor.png "Sponza, scissor vs no scissor")
+
+This can likely be improved by calculating more accurate bounding boxes for each light, as well as by adding
+other culling, like removal of lights behind all objects.
+
+## G-Buffer Optimization
+
+To further improve performance, I consolidated two of the provided G-Buffers (geometry normals and the normal map).
+Theoretically, this should boost the copy pass. However this also turned out to provide even more benefit to the lighting
+pass. Likely, this is because computation of the combined normal and binding/sampling of the G-Buffer textures are practically
+more expensive than the memory transfer of a single buffer.
+
+![Sponza, 3v4](img/plot_3v4.png "Sponza, 3 vs 4 G-Buffers")
+
+Further improvement could be achieved by using a) the fact that normals are normalized to compute them from only two components, and
+b) that world-space coordinates can be computed from screen-space x,y, to eliminate the position G-Buffer and use only a two-float normal buffer.
+However, this will likely incur a cost of having to reconstruct the missing components.
+
+# Effects
+## Bloom
+
+I implemented Bloom by extracting 'bright' portions (according to standard sRGB luminance values) of the deferred-pass output,
+then performing a Gaussian blur and adding it to the rendered output. A togglable option switches between single-pass and two-pass blur,
+though the latter seemed to actually cause worse performance. This is likely an implementation flaw, due to switching framebuffers and rebinding textures.
+
+![Sponza bloom only](img/sponza_bloom_only.png )
+![Sponza bloom](img/sponza_bloom.png )
+![Duck glow](img/duck_glow.png )
+
+Bloom did not seem to affect overall performance.
+
+## Cel-shading
+I also implemented cel-shading, by creating a simple 1D stepped ramp texture, which was sampled in order to 'posterize' each RGB channel.
+Contouring is done with a seperate shader, which simply tests for large depth changes and subtracts from those (smoothed) locations.
+
+![Sponza cel](img/sponza_cel.png )
+![Duck cel](img/duck_cel_basic.png )
+![Duck cel full](img/duck_cel_full.png )
 
-### Credits
 
-* [Three.js](https://github.com/mrdoob/three.js) by [@mrdoob](https://github.com/mrdoob) and contributors
-* [stats.js](https://github.com/mrdoob/stats.js) by [@mrdoob](https://github.com/mrdoob) and contributors
-* [webgl-debug](https://github.com/KhronosGroup/WebGLDeveloperTools) by Khronos Group Inc.
-* [glMatrix](https://github.com/toji/gl-matrix) by [@toji](https://github.com/toji) and contributors
-* [minimal-gltf-loader](https://github.com/shrekshao/minimal-gltf-loader) by [@shrekshao](https://github.com/shrekshao)

glsl/copy.frag.glsl

@@ -1,5 +1,6 @@
 #version 100
 #extension GL_EXT_draw_buffers: enable
+
 precision highp float;
 precision highp int;
 
@@ -10,11 +11,23 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
 void main() {
     // TODO: copy values into gl_FragData[0], [1], etc.
     // You can use the GLSL texture2D function to access the textures using
     // the UV in v_uv.
 
     // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[0] = vec4(v_position, 1.0);
+    vec3 normap = texture2D(u_normap, v_uv).xyz;
+    vec3 nor = normalize(applyNormalMap(v_normal, normap));
+    gl_FragData[1] = vec4(nor, 0.0);
+    gl_FragData[2] = texture2D(u_colmap, v_uv);
 }

glsl/deferred/ambient.frag.glsl

@@ -3,18 +3,20 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
+uniform sampler2D u_celRamp;
+uniform bool u_celShade;
+
 varying vec2 v_uv;
 
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
 
@@ -23,5 +25,14 @@ void main() {
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    if (u_celShade) {
+      vec3 rampCol;
+      rampCol.r = texture2D(u_celRamp, vec2(gb2.r, 0.5)).r;
+      rampCol.g = texture2D(u_celRamp, vec2(gb2.g, 0.5)).g;
+      rampCol.b = texture2D(u_celRamp, vec2(gb2.b, 0.5)).b;
+      gl_FragColor = vec4(rampCol, 1);
+    }
+    else {
+      gl_FragColor = vec4(gb2.rgb, 1);  // TODO: replace this
+    }
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,29 +2,23 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
-varying vec2 v_uv;
+uniform sampler2D u_celRamp;
+uniform int u_celShade;
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
-    normap = normap * 2.0 - 1.0;
-    vec3 up = normalize(vec3(0.001, 1, 0.001));
-    vec3 surftan = normalize(cross(geomnor, up));
-    vec3 surfbinor = cross(geomnor, surftan);
-    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
-}
+varying vec2 v_uv;
 
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
 
@@ -35,5 +29,29 @@ void main() {
         return;
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 nor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+
+    vec3 lDir = normalize(u_lightPos.xyz - pos);
+    float nDot = dot(nor, lDir);
+    vec3 lamb, spec;
+    float att = max(0.0, u_lightRad - length(pos - u_lightPos.xyz));
+    vec3 vDir = normalize(-pos);
+    vec3 hDir = normalize(vDir + lDir);
+    float hDot = max(dot(hDir, nor), 0.0);
+
+    if (u_celShade == 0) {
+      lamb = u_lightCol.rgb * gb2.rgb * clamp(nDot, 0.0, 1.0);
+      spec = u_lightCol.rgb * gb2.rgb * pow(hDot, 16.0);
+    }
+    else {
+      vec3 rampCol = u_lightCol.rgb * gb2.rgb;
+      rampCol.r = texture2D(u_celRamp, vec2(rampCol.r, 0.5)).r;
+      rampCol.g = texture2D(u_celRamp, vec2(rampCol.g, 0.5)).g;
+      rampCol.b = texture2D(u_celRamp, vec2(rampCol.b, 0.5)).b;
+      lamb = rampCol;
+      spec = rampCol * pow(hDot, 16.0);
+    }
+
+    gl_FragColor = vec4(.5*spec + .5*lamb, 1.0) * att / u_lightRad;  // TODO: perform lighting calculations
 }

glsl/deferred/cel-contour.frag.glsl

@@ -0,0 +1,26 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 3
+
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+uniform vec2 u_texSize;
+
+varying vec2 v_uv;
+
+void main() {
+    vec3 pos = texture2D(u_gbufs[0], v_uv).xyz;
+    vec2 tOff = 1.0 / u_texSize;
+    float d1 = texture2D(u_depth, v_uv).x;
+    float n = 0.0;
+    for (int i = -1; i <= 1; i++) {
+    for (int j = -1; j <= 1; j++) {
+      float d = texture2D(u_depth, v_uv + vec2(float(i)*tOff.x, float(j)*tOff.y)).x;
+      n = max(n, abs(d-d1));
+    }}
+    n = 100.0*clamp(5.0*n - 1.0, 0.0, 1.0);
+
+    gl_FragColor = vec4(vec3(n), 0);
+}

glsl/deferred/debug.frag.glsl

@@ -2,52 +2,40 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 uniform int u_debug;
+uniform mat4 u_projMat;
+uniform mat4 u_viewMat;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
 const vec4 SKY_COLOR = vec4(0.66, 0.73, 1.0, 1.0);
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
-    normap = normap * 2.0 - 1.0;
-    vec3 up = normalize(vec3(0.001, 1, 0.001));
-    vec3 surftan = normalize(cross(geomnor, up));
-    vec3 surfbinor = cross(geomnor, surftan);
-    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
-}
-
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
     // These definitions are suggested for starting out, but you will probably want to change them.
+
     vec3 pos = gb0.xyz;     // World-space position
-    vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    vec3 nor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
     vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
-    vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
-    vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
 
     // TODO: uncomment
     if (u_debug == 0) {
         gl_FragColor = vec4(vec3(depth), 1.0);
     } else if (u_debug == 1) {
-        // gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+        gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
     } else if (u_debug == 2) {
-        // gl_FragColor = vec4(abs(geomnor), 1.0);
+        gl_FragColor = vec4(abs(nor), 1.0);
     } else if (u_debug == 3) {
-        // gl_FragColor = vec4(colmap, 1.0);
-    } else if (u_debug == 4) {
-        // gl_FragColor = vec4(normap, 1.0);
-    } else if (u_debug == 5) {
-        // gl_FragColor = vec4(abs(nor), 1.0);
+        gl_FragColor = vec4(colmap, 1.0);
     } else {
-        gl_FragColor = vec4(1, 0, 1, 1);
+        gl_FragColor = vec4(1, 0, 0, 1);
     }
 }

glsl/post/bloom.frag.glsl

@@ -0,0 +1,58 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define KSIZE 5
+
+uniform sampler2D u_color;
+uniform vec2 u_texSize;
+uniform float u_radius;
+uniform float u_kernel[KSIZE];
+uniform int u_pass;
+
+varying vec2 v_uv;
+
+const vec4 SKY_COLOR = vec4(0.01, 0.14, 0.42, 1.0);
+
+void main() {
+    vec3 color = texture2D(u_color, v_uv).rgb;
+    vec2 tOff = u_radius / u_texSize;
+
+    if (u_pass == 0) {
+      float bright = dot(color, vec3(0.2126, 0.7152, 0.0722));
+      if (bright < 1.0)
+        color = vec3(0,0,0);
+    }
+    else if (u_pass == 1) {
+      color *= u_kernel[0];
+      for (int i = 1; i < KSIZE; i++) {
+        color += texture2D(u_color, v_uv + vec2(tOff.x*float(i), 0.0)).rgb * u_kernel[i];
+        color += texture2D(u_color, v_uv - vec2(tOff.x*float(i), 0.0)).rgb * u_kernel[i];
+      }
+    }
+    else if (u_pass == 2) {
+      color *= u_kernel[0];
+      for (int i = 1; i < KSIZE; i++) {
+        color += texture2D(u_color, v_uv + vec2(0.0, tOff.y*float(i))).rgb * u_kernel[i];
+        color += texture2D(u_color, v_uv - vec2(0.0, tOff.y*float(i))).rgb * u_kernel[i];
+      }
+    }
+    else if (u_pass == -1) {
+      for (int i = 0; i < KSIZE; i++) {
+      for (int j = 0; j < KSIZE; j++) {
+        color += texture2D(u_color, v_uv + vec2(tOff.x*float(i), tOff.y*float(j))).rgb * u_kernel[i] * u_kernel[j];
+        color += texture2D(u_color, v_uv + vec2(tOff.x*float(i), -tOff.y*float(j))).rgb * u_kernel[i] * u_kernel[j];
+        color += texture2D(u_color, v_uv + vec2(-tOff.x*float(i), tOff.y*float(j))).rgb * u_kernel[i] * u_kernel[j];
+        color += texture2D(u_color, v_uv + vec2(-tOff.x*float(i), -tOff.y*float(j))).rgb * u_kernel[i] * u_kernel[j];
+      }}
+    }
+
+    /*
+    if (color.a == 0.0) {
+        gl_FragColor = SKY_COLOR;
+        return;
+    }
+    */
+
+    gl_FragColor = vec4(color, 1);
+}

glsl/red.frag.glsl

@@ -3,5 +3,5 @@ precision highp float;
 precision highp int;
 
 void main() {
-    gl_FragColor = vec4(1, 0, 0, 1);
+    gl_FragColor = vec4(1, 0, 0, .9);
 }