index.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="description"
        content="FaceDiffuser produces facial animation for 3D characters based on speech">
  <meta name="keywords" content="FaceDiffusion, Diffusion, Animation">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion</title>

  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
        rel="stylesheet">

  <link rel="stylesheet" href="./static/css/bulma.min.css">
  <link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
  <link rel="stylesheet" href="./static/css/bulma-slider.min.css">
  <link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
  <link rel="stylesheet"
        href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
  <link rel="stylesheet" href="./static/css/index.css">
  <link rel="icon" href="./static/images/favicon.svg">

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script defer src="./static/js/fontawesome.all.min.js"></script>
  <script src="./static/js/bulma-carousel.min.js"></script>
  <script src="./static/js/bulma-slider.min.js"></script>
  <script src="./static/js/index.js"></script>
</head>
<body>


<section class="hero">
  <div class="hero-body">
    <div class="container is-max-desktop">
      <div class="columns is-centered">
        <div class="column has-text-centered">
          <h1 class="title is-1 publication-title">FaceDiffuser</h1> 
          <h2 class="title is-3 publication-title">Speech-Driven 3D Facial Animation Synthesis Using Diffusion</h2>
          <div class="is-size-3 publication-authors">
            <span class="author-block"> <a href="https://www.linkedin.com/in/stefan-stan-824088159/"> Stefan Stan </a>, </span>
            <span class="author-block"> <a href="https://www.uu.nl/staff/KIHaque"> Kazi Injamamul Haque </a>, </span>
            <span class="author-block"> <a href="https://www.uu.nl/staff/ZYumak"> Zerrin Yumak </a> </span>
          </div>

          <div class="is-size-5 publication-authors">
            <span class="author-block">Utrecht University</span>
          </div>

          <div class="column has-text-centered">
            <div class="publication-links">
              <!-- PDF Link. -->
              <span class="link-block">
                <a href="https://dl.acm.org/doi/10.1145/3623264.3624447"
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="ai ai-google-scholar"></i>
                  </span>
                  <span>Paper</span>
                </a>
              </span>
              <span class="link-block">
                <a href="https://arxiv.org/pdf/2309.11306.pdf"
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="ai ai-arxiv"></i>
                  </span>
                  <span>arXiv</span>
                </a>
              </span>
              
              <!-- Code Link. -->
              <span class="link-block">
                <a href="https://github.com/uuembodiedsocialai/FaceDiffuser"
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="fab fa-github"></i>
                  </span>
                  <span>Code</span>
                  </a>
              </span>

              <!-- Demo Link. -->
              <span class="link-block">
                <a href="https://replicate.com/stefan-st/face-diffuser" target="_blank"
                class="external-link button is-normal is-rounded is-dark">
                <span class="icon">
                  <i class="fas fa-rocket"></i>
                </span>
                <span>Demo</span>
              </a>
          </div>
        </div>
      </div>
    </div>
  </div>
</section>

<section class="section">
  <div class="container is-max-desktop">
    <!-- Abstract. -->
    <div class="columns is-centered has-text-centered">
      <div class="column is-four-fifths">
        <h2 class="title is-3">Abstract</h2>
        <div class="content has-text-justified">
          <p>
            Speech-driven 3D facial animation synthesis has been a challenging task both in industry and research. Recent methods mostly focus on deterministic deep learning methods meaning that given a speech input, the output is always the same.
            However, in reality, the non-verbal facial cues that reside throughout the face are non-deterministic in nature. In addition, majority of the approaches focus on 3D vertex based datasets and methods that are compatible with existing facial animation pipelines with rigged characters is scarce.
            To eliminate these issues, we present FaceDiffuser, a non-deterministic deep learning model to generate speech-driven facial animations that is trained with both 3D vertex and blendshape based datasets.
            Our method is based on the diffusion technique and uses the pre-trained large speech representation model HuBERT to encode the audio input. To the best of our knowledge, we are the first to employ the diffusion method for the task of speech-driven 3D facial animation synthesis.
            We have run extensive objective and subjective analyses and show that our approach achieves better or comparable results in comparison to the state-of-the-art methods.  We also introduce a new in-house dataset that is based on a blendshape based rigged character.
          </p>
        </div>
      </div>
    </div>
    <!--/ Abstract. -->

    <!-- Paper video. -->
    <div class="columns is-centered has-text-centered">
      <div class="column is-four-fifths">
        <h2 class="title is-3">Video</h2>
        <div class="publication-video">
          <iframe src="./static/videos/FaceDiffuser.mp4"
                  frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
        </div>
      </div>
    </div>
    <!--/ Paper video. -->


    <!-- Methodology -->
    <div class="columns is-centered has-text-centered">
      <div class="column is-full">
        <h2 class="title is-3">Methodology - Training</h2>
        <div class="content has-text-justified">
          FaceDiffuser learns to denoise facial expressions and generate animations based on input speech.
          Audio speech embeddings from the pre-trained HuBERT model combined with embeddings from the noised ground truth animation sequence are used to train the Facial Decoder.
          The Facial Decoder is comprised of a sequence of GRU layers followed by a fully connected layer and learns to predict (i) vertex displacements or (ii) rig control (blendshape) values.
          The predicted sequence is compared with the ground truth sequence by computing the loss, which is then backpropagated to update the model parameters.
        </div>
        <div class="methodology-image">
        <img src="./static/images/training.png"> </img>
        </div>
      </div>
    </div>
    <div class="columns is-centered has-text-centered">
      <div class="column is-full">
        <h2 class="title is-3">Methodology - Inference</h2>
        <div class="content has-text-justified">
        FaceDiffuser inference is an iterative process from the maximum diffustion timestep T decreasing to 1. The initial noised animation input is represented by actual noise from the normal distribution.
          At each step, we provide the network with the audio and noised animation input. The predicted motion is then diffused again and fed to the next step of the iteration.
        </div>
        <div class="methodology-image">
        <img src="./static/images/inference.png"> </img>
        </div>
      </div>
    </div>
    <!--/ Methodology -->

  </div>
</section>



<section class="section" id="BibTeX">
  <div class="container is-max-desktop content">
    <h2 class="title">BibTeX</h2>
    <pre><code>@inproceedings{FaceDiffuser_Stan_MIG2023,
author = {Stan, Stefan and Haque,  Kazi Injamamul and Yumak,  Zerrin},
title = {FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion},
booktitle = {ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG '23), November 15--17, 2023, Rennes, France},
year = {2023},
location = {Rennes, France},
numpages = {11},
url = {https://doi.org/10.1145/3623264.3624447},
doi = {10.1145/3623264.3624447},
publisher = {ACM},
address = {New York, NY, USA},
}</code></pre>
  </div>
</section>


<footer class="footer">
  <div class="container">
    <div class="columns is-centered">
      <div class="column is-8">
        <div class="content">
          <p>
            This website is licensed under a <a rel="license"
                                                href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
            Commons Attribution-ShareAlike 4.0 International License</a>.
          </p>
          <p>
            Website based on the <a
              href="https://github.com/nerfies/nerfies.github.io">Nerfies</a> project page.
            If you want to reuse their source code, please credit them appropriately.
          </p>
        </div>
      </div>
    </div>
  </div>
</footer>

</body>
</html>