client.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "318c88c4-3b67-4a8a-9200-7e88ff9f08b2",
   "metadata": {},
   "source": [
    "# Warning\n",
    "\n",
    "[⚠️ For cost-efficiency, we no longer host a DALL-E Flow demo server. Click here to see how you can deploy it for yourself!](https://github.com/jina-ai/dalle-flow#server)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13ed1c4",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<p align=\"center\">\n",
    "<a href=\"https://github.com/jina-ai/dalle\"><img src=\"https://res.cloudinary.com/startup-grind/image/upload/c_fill,dpr_2.0,f_auto,g_xy_center,h_650,q_auto:good,w_1440,x_w_mul_0.5,y_h_mul_0.0/v1/gcs/platform-data-dsc/event_banners/banner_8XSoAdr.png?md\" alt=\"DALL·E Flow: A Human-in-the-Loop workflow for creating HD images from text\" width=\"100%\"></a>\n",
    "<br>\n",
    "</p>\n",
    "\n",
    "\n",
    "<b>A Human-in-the-loop<sup><a href=\"https://en.wikipedia.org/wiki/Human-in-the-loop\">?</a></sup> workflow for creating HD images from text</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0836667a",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "[![GitHub Repo stars](https://img.shields.io/github/stars/jina-ai/dalle-flow?style=social)](https://github.com/jina-ai/dalle-flow) [![Google Colab](https://img.shields.io/badge/Slack-2.8k-blueviolet?logo=slack&amp;logoColor=white&style=flat-square)](https://slack.jina.ai) [![GitHub last commit (branch)](https://img.shields.io/github/last-commit/jina-ai/dalle-flow/main)](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f11f924b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "\n",
    "🚧 If you find your request fails, it is possible the server is occasionally down for maintaince. Please give it a try in 5 minutes.\n",
    "\n",
    "- ⚠️ **2022/8/8** Started using CLIP-as-service as an [external executor](https://docs.jina.ai/fundamentals/flow/add-executors/#external-executors). Now you can easily [deploy your own CLIP executor](#run-your-own-clip) if you want. There is [a small breaking change](https://github.com/jina-ai/dalle-flow/pull/74/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R103) as a result of this improvement, so [please _reopen_ the notebook in Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n",
    "- ⚠️ **2022/7/6** Due to server migration to AWS EKS, server url changed to `grpcs://dalle-flow.dev.jina.ai`. All connections are now with TLS again, [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)\n",
    "- ⚠️ **2022/6/25** Unexpected downtime between 6/25 0:00 - 12:00 CET due to out of GPU quotas. The new server now has 2 GPUs, add healthcheck.\n",
    "- 2022/6/3 Reduce default number of images to 2 per pathway, 4 for diffusion.\n",
    "- ⚠️ 2022/5/23 Fix an upstream bug in CLIP-as-service. This bug makes the 2nd diffusion step irrelevant to the given texts. New Dockerfile proved to be reproducible on a AWS EC2 `p2.x8large` instance.\n",
    "- 2022/5/13b Removing TLS as Cloudflare gives 100s timeout, making DALLE Flow in usable [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n",
    "- 🔐 2022/5/13 New Mega checkpoint! All connections are now with TLS, [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n",
    "- 🌟 2022/5/10 [A Dockerfile is added! Now you can easily deploy your own DALL·E Flow](https://github.com/jina-ai/dalle-flow). New Mega checkpoint! Smaller memory-footprint, the whole Flow can now fit into **one GPU with 21GB memory**.\n",
    "- 🌟 2022/5/9 Less memory-footprint overall, the whole Flow can now fit into one GPU with 18GB memory!\n",
    "- 🌟 2022/5/7 DALL·E Flow just got updated!\n",
    "    - New DALL·E Mega checkpoint\n",
    "    - Improved GLID3 memory-efficiency and parameters\n",
    "- 🌟 2022/5/6 DALL·E Flow just got updated!\n",
    "    - The first step will generate 16 candidates: **8 from DALL·E Mega, 8 from GLID3-XL**; ranked by CLIP-as-service.\n",
    "    - Optimized the flow efficiency, diffusion and upscaling is much faster now!\n",
    "- ~~⚠️ 2022/5/3 **The number of images is restrict to 9 for DALL·E Mega, and 16 for GLID3-XL**~~\n",
    "- ⚠️ 2022/5/2 **Due to the massive requests now, the server is super busy.** You can deploy your own server by following [the instruction here](https://github.com/jina-ai/dalle-flow#server).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6af8cfd",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Using client is super easy. The following steps are best run in Jupyter notebook or [Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).  \n",
    "\n",
    "The only dependency you will need are [DocArray](https://github.com/jina-ai/docarray) and [Jina](https://github.com/jina-ai/jina), as DocArray is already included in Jina you only need to install `jina`.\n",
    "\n",
    "> On Google Colab, you will be asked to restart the kernel. Go ahead and restart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c95e266f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install jina"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3059e59c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "We have provided a demo server for you to play:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1bb189",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "server_url = 'grpcs://dalle-flow.dev.jina.ai'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72aacbed",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 1: Generate via DALL·E Mega\n",
    "\n",
    "Now let's define the prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac1ec094",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "prompt = 'an oil painting of a humanoid robot playing chess in the style of Matisse'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de6d7244",
   "metadata": {},
   "source": [
    "Do you need some hints and suggestions on the prompt? Check out those tricks:\n",
    " - [A Guide to Writing Prompts for Text-to-image AI](https://docs.google.com/document/d/17VPu3U2qXthOpt2zWczFvf-AH6z37hxUbvEe1rJTsEc/edit?usp=sharing)\n",
    " - [CLIP Templates](https://docs.google.com/document/d/1j2IAumYz4iZopOTAAOcCUKbFXP0jHK8mRgD4NLFKkaw/edit?usp=sharing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69d0b10b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's submit it to the server and visualize the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da672930",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "from docarray import Document\n",
    "\n",
    "doc = Document(text=prompt).post(server_url, parameters={'num_images': 2})\n",
    "da = doc.matches\n",
    "\n",
    "da.plot_image_sprites(fig_size=(10,10), show_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9298f59f",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here we generate 4 candidates, 2 from DALLE-mega and 2 from GLID3 XL, this is as defined in `num_images`, which takes about ~2 minutes. You can use a smaller value if it is too long for you. The results are sorted by [CLIP-as-service](https://github.com/jina-ai/clip-as-service), with index-`0` as the best candidate judged by CLIP. \n",
    "\n",
    "> You could generate up to 8 images per pathway via `num_images`, resulting 16 candidate images in total. But it will be much slower."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bf3a62b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 2: Select and refinement via GLID3 XL\n",
    "\n",
    "Of course, you may think differently. Notice the number in the top-left corner? Select the one you like the most and get a better view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e106718",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fav_id = 3\n",
    "\n",
    "fav = da[fav_id]\n",
    "fav.embedding = doc.embedding\n",
    "\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec3e4ea",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's submit the selected candidates to the server for diffusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13cf87ce",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "diffused = fav.post(f'{server_url}', parameters={'skip_rate': 0.6, 'num_images': 4}, target_executor='diffusion').matches\n",
    "\n",
    "diffused.plot_image_sprites(fig_size=(10,10), show_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb54ed9",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This will give 4 images based on the given image. You may allow the model to improvise more by giving `skip_rate` a near-zero value, or a near-one value to force its closeness to the given image. The whole procedure takes about ~1 minutes.\n",
    "\n",
    "> You could generate upto 8 images via `num_images`. But it will be slower."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "978ccf66",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 3: Select and upscale via SwinIR\n",
    "\n",
    "Select the image you like the most, and give it a closer look:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f00f44e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dfav_id = 2\n",
    "\n",
    "fav = diffused[dfav_id]\n",
    "\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab54a790",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If not satisfied, you can rerun the last cell in step 2 with this new `fav`.\n",
    "\n",
    "Finally, submit to the server for the last step: upscaling to 1024 x 1024px.\n",
    "\n",
    "> This step should be much faster (~10s) as the Flow is designed in unblocked manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60e8bfc",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "fav = fav.post(f'{server_url}/upscale')\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e910e8aa",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "> 💁‍♂️ On Google colab this image may render exactly the same size as before. But it is in 1024x1024 already. Right click on the image and copy/save it. You will see.\n",
    "\n",
    "That's it! It is _the one_.\n",
    "\n",
    "\n",
    "Btw, [DocArray is a powerful and easy-to-use data structure for unstructured data](https://github.com/jina-ai/docarray), that makes all of these possible. It is super productive for data scientists who work in cross-/multi-modal domain. To learn more about DocArray, [please check out the docs](https://docarray.jina.ai)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}