notebooks/Building Higher-Level Abstractions of Source Code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "When working with bigger software systems, it's easy to get lost in all the source code that makes up the system. A good software system has to provide a structure that allows developers to quickly grasp the main ideas of a system. A proven  method to achieve this is using hierarchies and schemas. With this approach, tiny things can be summed up to bigger ones that make somehow sense in a more broader view. \n",
    "\n",
    "Schemas are conceptual units humans can reason about because they are familiar with it. If I say \"school\", you can quickly link some memories to that conceptual unit and attach new information to it. If I had started with all the details about my specific school &ndash; the building, my teachers, my pencil, every insignificance &ndash; without mentioning \"school\" at all, you wouldn't be able to follow my thoughts at all.\n",
    "\n",
    "Hierarchies are built upon conceptual units that are linked together in a tree-like structure. Conceptual units are part of multiple hierarchies depending on your perspective: In the example above, I'm speaking from the perspective of a pupil. But if you are an architect, you see the \"building\" school. If you are the school's district manager, you see it as a cost center. So there are multiple hierarchies, building up multiple schemas as well, leading to a more graph-like structure of the things we know.\n",
    "\n",
    "With these methods, we can produce nice little information packages (aka chunks) that are consumable way easier. And small is good! It's exactly what we need to process information quickly as well. Otherwise, we would be overwhelmed by pure information overload."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing hierarchies and schemas for source code\n",
    "Moving this view to software development, we see that there is a need for building up higher-level abstractions out of source code. We try to do it by creating a software architecture where its component are ordered in a hierarchical way. In most programming languages, this is will be directories and files in the end. For example, translated into the Java world, it means packages and source code files.\n",
    "\n",
    "But there is a big problem with this approach: Those hierarchies are one-dimensional. But you have multiple views on your application:\n",
    "\n",
    "- If you use the model view controller pattern, you see your components, well, as models, views and controllers. \n",
    "- If you want to add new business features, you see it divided into the subdomains of your business domain.\n",
    "- If you look at it from a performance's perspective, you see slow components, clients for external systems or database access components.\n",
    "- If you look at the code contributions of developers, you'll see different areas of code maintained by different teams.\n",
    "- If you are interested in the stable parts of your system, you will examine all the changes in a version control system.\n",
    "\n",
    "This \"multiperspectivity\" can't be expressed in source code directly. This also means that building up those hierarchies and schema for easier information processing isn't possible at all. This makes it so hard to understand and reason about all the interconnected, multifaceted problems in your application &ndash; making problem-solving harder than it has to be.\n",
    "\n",
    "Luckily, there are tools to reduce this ubiquitous problem in software development. Carola Lilienthal wrote about SotoGraph in her [book](https://www.amazon.de/Langlebige-Software-Architekturen-Technische-analysieren-begrenzen/dp/3864904943/) and Adam Thornhill will address this in his [upcoming book](http://www.adamtornhill.com/swevolution/reviewersprogress.html) as well. I want to show you an additional possibility based on [jQAssistant](http://jqassistant.com) and [Neo4j](https://neo4j.com). I wrote about my experience with this structural analysis tool in [My Experiences with jQAssistant](https://www.feststelltaste.de/my-experiences-with-jqassistant-so-far/) a while ago. Here I just scratched the surface regarding the main reasons why I like that tool so much: It is capable of creating different perspectives on your software system by using hierarchies (by using nodes and relationships) and schemas (by enabling so-called \"concepts\"). This is an extremely powerful way to abstract all software entities in your software to entities that software architects or even business expert can understand and thus reason about.\n",
    "\n",
    "Let's have a look how this works!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 1: Technical Concepts\n",
    "\n",
    "If we take a look at the living architecture documentation in [`jqassistant/structure.adoc`](https://github.com/buschmais/spring-petclinic/jqassistant/spring-mvc.adoc) of the [jQAssistant's version of Spring Petclinic](https://github.com/buschmais/spring-petclinic), we can see that jQAssistant can mark software entities (in this case Java types) depending on their relationship to other software entities:\n",
    "\n",
    "```\n",
    "[[spring-mvc:Controller]]\n",
    "[source,cypher,role=\"concept\"]\n",
    ".Labels all types annotated with \"org.springframework.stereotype.Controller\" \n",
    " with \"Spring\", \"Component\" and \"Controller\".\n",
    "----\n",
    "MATCH\n",
    "  (controller:Type)-[:ANNOTATED_BY]->()-[:OF_TYPE]->(annotationType:Type)\n",
    "WHERE\n",
    "  annotationType.fqn = \"org.springframework.stereotype.Controller\"\n",
    "SET\n",
    "  controller:Spring:Component:Controller\n",
    "RETURN\n",
    "  controller as Controller\n",
    "----\n",
    "```\n",
    "\n",
    "In the case above, jQAssistant assigns the labels `Spring`, `Component` and `Controller` to every entity in the software system that is annotated with the annotation's type `org.springframework.stereotype.Controller`. A `Controller` is a high-level technical concept as described above. Once marked in the database, you can find all other controllers as well and ask specific questions / create queries about them in a clear context. The marking mechanism gives you many different options for labeling and thus reason about all the technical entities in your software system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 2: Business Subdomains\n",
    "\n",
    "But often, business people don't want to talk about all the technical details like \"controllers\". We as software developers have to create other higher-level abstractions that mean something to non-technical people. In [Analyze Dependencies Between Business Subdomains](https://www.feststelltaste.de/analyze-dependencies-between-business-subdomains/), I sketched a way how you can achieve this based on package naming conventions. But as mentioned above, there are multiple ways how you can structure your software system. This means that relying on package naming conventions couldn't be possible.\n",
    "\n",
    "But even if there aren't any package naming conventions, you can identify some structure for example in class names or in your inheritance hierarchy that points you towards your subdomains in the code (if that isn't possible as well: I wrote my Master's thesis about mining cohesive concepts from source code via text mining, so you could use that as well :-D . There are also many other possibilities. And at the last resort, you have to do the mapping manually...).\n",
    "\n",
    "Let's see how this could work by mapping business subdomains to the class names of the Spring PetClinic project that we used above. In this project, the Java types look like this:\n",
    "\n",
    "* ```EntityUtils```\n",
    "* ```OwnerController```\n",
    "* ```PetRepository```\n",
    "* ```JpaVetRepositoryImpl```\n",
    "* ```Pet```\n",
    "* ```NamedEntity```\n",
    "* ```JpaVisitRepositoryImpl```\n",
    "* ```VisitController```\n",
    "* ```PetTypeFormatter```\n",
    "\n",
    "From the random list above, you can see that the names contain some information about the subdomain of the application like \"Pet\", \"Owner\" or \"Visit\". With a simple heuristic, we can map predefined subdomain names with the names of Java types by naming conventions. E. g. a rule in your living architecture documentation could be like this one:\n",
    "\n",
    "```\n",
    "The Spring PetClinic application consists of several business subdomains that can be identified by naming conventions.\n",
    "[[business:Subdomain]]\n",
    ".Create predefined Subdomain nodes and connect them to all Type nodes by naming conventions\n",
    "[source,cypher,role=concept]\n",
    "----\n",
    "UNWIND [\n",
    "\t{ name: \"Clinic\" },\n",
    "\t{ name: \"Owner\" },\n",
    "\t{ name: \"Person\" }, \n",
    "    { name: \"Pet\" },\n",
    "    { name: \"Specialty\" },\n",
    "\t{ name: \"Vet\" }, \n",
    "\t{ name: \"Visit\" }\n",
    "]\n",
    "AS properties\n",
    "CREATE (s:Subdomain) SET s = properties\n",
    "WITH s\n",
    "    MATCH (t:Type)\n",
    "\t    WHERE t.name CONTAINS s.name\n",
    "    MERGE (t)-[:BELONGS_TO]->(s)\n",
    "RETURN s.name, t.name\n",
    "----\n",
    "```\n",
    "Here, we first predefine all business subdomains as a map of properties and add new nodes for them. Additionally, we connect all the types that belong to a subdomain via the naming convention. \n",
    "\n",
    "In the database, we have now the information about the subdomain for each type in our software. The blue colored nodes are the ```Type``` nodes of the software system that are connected by a ```BELONGS_TO``` relationship to the grey node in the middle which is one of the ```Subdomain``` nodes.\n",
    "\n",
    "![](resources/higher_level_abstractions_with_jqassistant_subdomains.png)\n",
    "\n",
    "We can refine this model by adding information as we like and as long as there is some structural information existing for the data sources. For example, if we use the code analysis tools like FindBugs, it's possible to add existing coding violations into the graph and connect them to the types &ndash; and thus the subdomains. If we are using a bug tracking system like Bugzilla, we can list known bugs for specific subdomains. This enables us to look at internal code quality from a business' perspective! The problem space is reduced and can be communicated in a language that can be understood by business people as well. \n",
    "\n",
    "**This is the entry point into \"software triage\" which can help to focus on effective refactoring or reengineering tasks!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "In this blog post, you saw how you can create higher-level abstractions of your source code. With this approach, you can create schemas of known things by marking them either by additional node labels or by creating new conceptual nodes. Both approaches are fine to create more sophisticated hierarchies leading to chunks of information that can be processed by people who are unfamiliar with all the details of a software system. This enables discussions on a new level!\n",
    "\n",
    "By focusing on hierarchies that mean something to business people, a software architect can now discuss issues in the software system in terms of the business. \n",
    "\n",
    "With the graph-based approach in Neo4j, you can easily add additional information about open bug tickets, static code analysis results, latest code changes &ndash; everything! Discussions about code quality don't start with general statements like \n",
    "\n",
    "> *\"Ohhh, we are doomed, because we have so much technical debt!\"*\n",
    "\n",
    "or on a very detailed perspective like\n",
    "\n",
    "> *\"AbstractNotificationProxyServiceManager has a cyclomatic complexity of 144!\"*\n",
    "\n",
    "but on the right level:\n",
    "\n",
    "> *\"If you plan to change the notification feature in the next sprint, be sure to add additional time for cleaning up the code base for this component.\"*\n",
    "\n",
    "where you can go deeper if needed e. g. with \n",
    "\n",
    "> *\"There are 4 open bugs and the code is more complex than it has to be. Ohh, and the main developer leaves the company in three weeks, so it should be a high priority user story in your backlog as well!\"*\n",
    "\n",
    "I hope you can imagine the potential of higher-level abstractions of your source code. It's a very context-dependent approach that requires some customization because every business domain and thus software system is different. But that's exactly the reason why it is so powerful: It works in your context and it's tailored to your specific needs!\n",
    "\n",
    "What do you think about it? Are there any barriers that are still in the way? What are your ideas to bring developers and business people together to talk about internal software quality on the same level?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}