notebooks/Java Type Dependency Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "Recently I came over a great visualization of the relationships between classes made by Mike Bostock with his [Hierarchical Edge Bundling](https://bl.ocks.org/mbostock/7607999) in [D3](https://d3js.org/):\n",
    "\n",
    "<a href=\"https://bl.ocks.org/mbostock/7607999\"><img src=\"resources/java_type_dependency_analysis_example.png\" /></a>\n",
    "\n",
    "I wondered how hard it would be to reimplement this visualization with [jQAssistant](https://jqassistant.org/) and [Neo4j](https://neo4j.com/) and show actual dependencies between Java types. So let's have a look!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idea\n",
    "If you've scanned your project with jQAssistant, you get some graph data like the following:\n",
    "![](resources/java_type_dependency_analysis_dependency_pet.png)\n",
    "\n",
    "The graph contains the information about the dependencies of all software entities like Java classes or interfaces. This information is exactly what we need to create a Dependency Analysis between Java types. We just have to create the right Cypher query and the result that fits into the D3 visualization above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting the right Data\n",
    "What you need to do is to scan your software with jQAssistant to get the information about the dependencies between classes. Just download the demo project at https://github.com/buschmais/spring-petclinic, build it with Java & Maven (`mvn clean install`) and start the embedded Neo4j graph database with `mvn jqassistant:server` .\n",
    "\n",
    "With a running Neo4j database, we connect to it via  [py2neo](http://py2neo.org/v3/database.html) and get the relationship information between nodes. In this example, we want to retrieve the direct dependency between any Java type that belongs to our application:\n",
    "\n",
    "- We identify our application's `type`'s by using only artifacts that are contained in our project by using the corresponding node labels `Project` and `Artifact`.\n",
    "- We also filter out any test classes by skipping the non-relevant artifacts of the type `test-jar`.\n",
    "- With the remaining types, we search for all direct dependencies with the `DEPENDS_ON` relationship to the other types of our application. For reasons of completeness, we also provide the types that don't depend on any other type of our application. That's why we specify the `*0..1` parameter in the relationship.\n",
    "- Finally, we `COLLECT` all the dependencies for one type into a list (more exact, the `fqn` aka full qualified name of the type), because that's what the D3 visualization needs as input. \n",
    "\n",
    "The result is a dictionary with all the information needed for the D3 visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'imports': ['org.springframework.samples.petclinic.model.Pet',\n",
       "   'org.springframework.samples.petclinic.web.PetValidator'],\n",
       "  'name': 'org.springframework.samples.petclinic.web.PetValidator'},\n",
       " {'imports': ['org.springframework.samples.petclinic.repository.jdbc.JdbcPetVisitExtractor',\n",
       "   'org.springframework.samples.petclinic.repository.OwnerRepository',\n",
       "   'org.springframework.samples.petclinic.model.Owner',\n",
       "   'org.springframework.samples.petclinic.util.EntityUtils',\n",
       "   'org.springframework.samples.petclinic.repository.jdbc.JdbcOwnerRepositoryImpl',\n",
       "   'org.springframework.samples.petclinic.model.PetType',\n",
       "   'org.springframework.samples.petclinic.repository.jdbc.JdbcPet'],\n",
       "  'name': 'org.springframework.samples.petclinic.repository.jdbc.JdbcOwnerRepositoryImpl'},\n",
       " {'imports': ['org.springframework.samples.petclinic.util.EntityUtils',\n",
       "   'org.springframework.samples.petclinic.model.BaseEntity'],\n",
       "  'name': 'org.springframework.samples.petclinic.util.EntityUtils'}]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import py2neo\n",
    "\n",
    "query=\"\"\"\n",
    "MATCH\n",
    "    (:Project)-[:CONTAINS]->(artifact:Artifact)-[:CONTAINS]->(type:Type)\n",
    "WHERE\n",
    "    // we don't want to analyze test artifacts\n",
    "    NOT artifact.type = \"test-jar\" \n",
    "WITH DISTINCT type, artifact\n",
    "MATCH\n",
    "    (type)-[:DEPENDS_ON*0..1]->(directDependency:Type)<-[:CONTAINS]-(artifact)\n",
    "RETURN type.fqn as name, COLLECT(DISTINCT directDependency.fqn) as imports\n",
    "\"\"\"\n",
    "\n",
    "json_data = py2neo.Graph().run(query).data()\n",
    "json_data[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just write that data to a file that we read into our D3 HTML-Template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open ( \"vis/flare-imports.json\", mode='w') as json_file:\n",
    "    json_file.write(json.dumps(json_data, indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Result \n",
    "That's it: The [result](https://www.feststelltaste.de/wp-content/uploads/demos/java_type_dependency_analysis/spring-petclinic.html) is an interactive bundle diagram with all the dependencies between Java types of the application!\n",
    "\n",
    "<a href=\"https://www.feststelltaste.de/wp-content/uploads/demos/java_type_dependency_analysis/spring-petclinic.html\"><img src=\"resources/java_type_dependency_analysis_result.png\" /></a>\n",
    "\n",
    "In the figure above, I've hovered over the class `OwnerRepository` and I can immediately see the types that this class depends on (red) and the types that depend on the class `OwnerRepository` (green).\n",
    "\n",
    "**Next:** You can also query your software on different abstraction levels. As a proof of concept, I did that for [dependencies between packages](https://www.feststelltaste.de/wp-content/uploads/demos/java_type_dependency_analysis/spring-petclinic_packages.html) already:\n",
    "\n",
    "<a href=\"https://www.feststelltaste.de/wp-content/uploads/demos/java_type_dependency_analysis/spring-petclinic_packages.html\"><img src=\"resources/java_type_dependency_analysis_result_packages.png\" /></a>\n",
    "\n",
    "Stay tuned for more Dependency Analysis in a follow-up post!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}