Timeline for visualizing jobs

[MOHITRAJANI]

Visualizing jobs on a timeline

1. As we know that Histories contains of Jobs, and these jobs have a start time and end time as returned by the Job Metrics, This visualization takes all these jobs and adds them to a timeline graph so-called "Gantt Chart".

2. For implementing the Gantt Chart we are using the so-called Frappe-Gantt. src: https://github.com/frappe/gantt

3. The main benefit for a user to use this feature is that they can easily study the gantt-chart and make out which jobs took long, which ones were short, which tasks overlapped, when did they start and when did they end etc. Apart from that the users can switch between different views to find out which jobs they have performed over a month, a week, in a day, or over the span of a few minutes. The different views that the user can switch between are:

4. There are many features that Frappe-Gantt doesn't possess but we needed them for our project and hence I have implemented them in the library, these features are:

• Hour and Minute view also added to the library
• Header row shows Date-Month-Hour-Seconds
• A new SVG (Diamond shaped) is added which gets shown whenever the task is too small for the Viewing-scale that is being used, eg. If a Hour or Week view is used and the tasks are only in seconds then you'd see a diamond instead of a bar
• Adding class-names for bars to give them random hsluv color.
• Minutes view scales start at the hour which the user selects from the date-time popup.
• dynamic decision of where to show the popup, based on how long the visualization bar is. If the right end is not visible to the user then the popup is shown on the left side of the bar, or it is shown on the right hand side of the bar.

• disabling drag and drop for the jobs.
• some css changes like making text colors black inside a bar, removing resizing cursor, etc.

Because of these changes to the original library, we will be using this modified version of frappe-gantt from https://github.com/MOHITRAJANI/gantt-advanced

5. This is how you navigate to this feature:

6. Minutes View only spans over an interval of 10 minutes, and hence the user decides the time span between which they want to see the minutes view task, (This is implemented in order to avoid infinite scrolling in minutes view). Frappe-Gantt was also customized again for changing the time from where the legend starts for minute view. After filtering the results that lie in the given range of time, the Minutes view looks like this.

7. As the user is going to pick the time span for Minutes view, there is a Date time picker modal which opens on click of Minutes View button which looks like this:

8. Small loader mechanism is attached to the page to indicate that the visualization is loading. This Loader does not only load until the histories are loading, but also until the entire gantt is initialized. The loader looks like the standard loader also used to show that the histories are loading

9. Instead of having just the boring single color bars and diamond svgs, we now use the hsluv colors used in galaxy to randomly assign colors based on task names. This looks much more elegant, for example:

10. Histories can be easily switched over and user will immediately see gantt chart for the new current history

11. Gantt visualization also shows a spanned message for indicating when the History is empty, i.e. It doesn't have any jobs. Also for empty history, the buttons become disabled.

12. Finally, this is how many overlapping jobs would look.

The tasks for this feature are performed by Me and @shubham4315 as a part of our Masters Project.

Details about all changes made in the original frappe/gantt library

As already mentioned that there were many additional functionalities that we needed which were unfortunately not present in the library, so I (Mohit Rajani) went ahead and started adding the functionalities in the library. Hence for this project, the src of frappe/gantt is not the original github of the library but rather it's my fork to the library with all the additional functionalities and improvements. This description also mentions the filenames where these particular changes can be found.

1. Hour scale was added as a view-mode to the library, For adding this functionality all the changes were made in the index.js file inside the src directory of the library, I had to work on the following tasks:

   • Deciding on what date format will be used in terms of Hour scale.
   • Then calculating how many columns need to be there when using a Hour scale, this calculation is based on how many blocks go between each hour and then finally calculating how many total hours we want to display on gantt.
   • Then I had to also decide how many hours or days should be additional added before the first chronological task to the columns, so that the user doesn't see the very first task at the very left corner of the page.
   • Calculating two variables Hour_low and Hour_upper, these parameters help the library decide where to draw column lines and where to display the next hour on the header respectively. So the Hour_upper variable tells the library, after how many columns does one hour change. And the Hour_low is for every stroke that comes between the current and the next Hour

2. Minute Scale was added to the View-mode. All changes for this functionality were done in src/index.js. The tasks for the functionality are same as above, but here the calculations were much more complex. Also, the start time before the first chronological task for minute scale is just another Minute so that the user does not have to scroll a lot only for finding the first task.

3. Addition of a new SVG i.e the Diamond SVG to the library, this functionality was added because in most of the original scales provided by the library, the tasks were too small as compared to the scales, and due to this, the bar would look as small as a dot in that scale, hence to have a unique indication for a very small task on a comparatively big scale this feature was added. The changes for this feature were made in the src/Bar.js file. The changes for this functionality involved these tasks:

   • Creating a diamond shaped svg by doing calculations to how much distance should be there between each points X and Y coordinates.
   • Deciding on which scenarios the diamond should be shown, for this decision also calculation was done as to how much is the duration between the task's start time and end time in the same scale which was used for the view, e.g. If you are on an Hour scale, then measuring how many minutes are there between the start time and the end time, If you are in a Minute scale then deciding based on how many seconds are there between the start time and the end time. And if this duration is less than 1, then use a flag to indicate that a diamond needs to be show.
   • Adding css-classes to the diamond svg programatically
   • Deciding the position of the popup for the diamond SVG

4. Dynamically deciding the position of the popover on the tasks. This functionality was added because of an observation made by @itisAliRH that when the tasks are quiet long, some or most part of the bar goes outside the screen and you have to scroll until the end of the task to see its popup as originally the popup was made on the right side by the library. The changes for this functionality were made in the src/popup.js and the changes are as under:

   • Finding the start and end position of the Task in terms of X and Y coordinates from the left corner of the screen.
   • Then mathematically calculating if the X coordinate of the end of the task is greater than the current screen's rightmost X-coordinate. If this is true, then showing the popup to the left of the start of the task.
   • Deciding the dimensions of the Popup on the left, designing an arrow to the right of the popup and placing the popup at the correct position based on the above calculations,

5. The original library had additional feature of dragging and dropping the tasks, and to resize them, but we did't want this functionality as our tasks were fixed and hence this functionality was removed from the sec/index.js file. Also now that we didn't want the drag and drop feature.

6. Some CSS changes were also done in the src/gantt.scss file. The main additions in css were:

   • making the color of text inside the bar as black so that it is clearly visible in the hsluv colored bars that we are using.
   • changing the overflow property to visible for the .gantt-container, because of which we can now continuously see the scrollbars which was originally not present in the library.
   • removing the resize cursor from the .handle class.

How to test the changes?

• I've included appropriate automated tests.
• Instructions for manual testing are as follows:
  1. yarn run test

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[hexylena]

Some comments from a discussion in IWC, after sharing my own implementation of similar, based around workflow invocations specifically

I'd recommend I guess for that PR to not use a gantt chart library but instead something like the chrome perf team work since that's going to be a lot more performant for dense traces and more appropriate for this sort of data, but, overall great to see that was being worked on at one point

alternatively with as easy as this json format is, one could imagine producing traces for a lot more than just jobs/workflow invocations, I think I would have loved this as an admin to generate a view of how my cluster was behaving. Or even to receive profiling data from a specific job if we instrumented something like optional profiling for specific jobs to see how they behave. This format is very amenable to stack traces and sampling profiling, you could really have a high to low level view which could be interesting :)

hope it gets picked up again / merged someday!!