Introduce markwhen for visualization?

[blaisep]

Has anyone already looked into visualization of timelines?
For example, I was hoping someone had already come up with an example of inserting some markwhen statements and creating a file that could be rendered into a graphic.

[yaythomas]

hey that's a pretty neat idea @blaisep! I've not done this myself, but you could go about it in a few ways:

1. use pypyr.steps.filewrite to create and then subsequently append whatever you want into your output file

2. Make it a bit easier to re-use in a pipeline by wrapping a mw "logger" into a custom step.

To begin with, for option 1, something like this works:

steps:
  - name: pypyr.steps.set
    comment: will be re-using this filename multiple places
    in:
      set:
        mw_path: my-markwhen-file.mw
        date_fmt: "%Y-%m-%dT%H:%M:%SZ"

  - name: pypyr.steps.py
    comment: silly function to pump out a markwhen entry - will be using
             it in subsequent !py strings conveniently to append lines to the
             mw file
    in:
      py: |
        from datetime import datetime
        
        def get_mw_entry(milestone, tag=None):
          now = datetime.utcnow().strftime(date_fmt)
          if tag:
            return f'{now}: {milestone} #{tag}\n'

          return f'{now}: {milestone}\n'

        # make the get_mw_entry func available to subsequent steps
        save('get_mw_entry')

  - name: pypyr.steps.filewrite
    comment: create markwhen output header
    in:
      fileWrite:
        path: '{mw_path}'
        payload: |
          title: my markwhen example
          #good: green
          #bad: red
  
  - name: pypyr.steps.echo
    in:
      echoMe: 1. do stuff. imagine there're multiple steps here.

  - name: pypyr.steps.filewrite
    comment: make markwhen timeline entry
    in:
      current_milestone: stage1
      fileWrite:
        path: '{mw_path}'
        append: True
        payload: !py get_mw_entry(current_milestone, "good")

  - name: pypyr.steps.echo
    in:
      echoMe: 2. do stuff. imagine there're multiple steps here.

  - name: pypyr.steps.filewrite
    comment: make markwhen timeline entry
    in:
      current_milestone: stage2
      fileWrite:
        path: '{mw_path}'
        append: True
        payload: !py get_mw_entry(current_milestone, "bad")

I checked that this works, but to my eye this is a bit. . . involved for a nice clean pipeline. Although saying that, you could create a "timed custom step-group" and call it using pypyr.steps.call.

Personally I'd probably go for option 2. Example to follow later. . .

[yaythomas]

okay, and for option 2 (using custom steps to tidy things up a bit):

Two steps in a mw subdir:

1. ./mw/init.py

from pathlib import Path


def run_step(context):
    context.assert_key_has_value(key='mw_init', caller=__name__)

    # applies {formatting expressions} if any
    mw_init = context.get_formatted('mw_init')

    path = mw_init.get('path', 'output.mw')
    # parent dirs for path must exist
    Path(path).parent.mkdir(parents=True, exist_ok=True)

    # subsequent mw.event steps will use path + date fmt.
    context['_mw_path'] = path
    context['_mw_dateformat'] = mw_init.get('date_fmt', '%Y-%m-%dT%H:%M:%SZ')

    headers = mw_init.get('headers')

    # only write if optional headers exist.
    # if no headers mw.event will create a headerless file on 1st run by itself
    if headers:
        with open(path, 'w') as f:
            f.write(headers)

2. ./mw/event.py

from datetime import datetime


def run_step(context):
    context.assert_key_has_value(key='mw_event', caller=__name__)

    # applies {formatting expressions} if any
    event = context.get_formatted('mw_event')

    # these values set in context by mw.init
    path = context.get('_mw_path')
    date_fmt = context.get('_mw_dateformat')

    if not path or not date_fmt:
        raise ValueError(
            'You have to initialize markwhen handling with mw.init before '
            + 'using mw.event')

    with open(path, 'a') as f:
        f.write(f'{datetime.utcnow().strftime(date_fmt)}: {event}\n')

And now you can use them from a pipeline like this:

steps:
  - name: mw.init
    comment: initialize the markwhen file with headers
    in:
      mw_init:
        path: my-markwhen-file-custom-steps.mw
        # note that headers passed through as str, not yaml
        headers: |
          title: my markwhen generated from pypyr
          description: with a description here
          #good: green
          #bad: red
  
  - name: pypyr.steps.echo
    in:
      echoMe: 1. do stuff. imagine there're multiple steps here.

  - name: mw.event
    in:
      mw_event: "stage1 #good"

  - name: pypyr.steps.echo
    in:
      echoMe: 2. do stuff. imagine there're multiple steps here.

  - name: mw.event
    in:
      mw_event: "stage2 #bad"

This is really just a quick and dirty sample - you can definitely do more cute/clever things with date-formatting, defaulting, setting ranges and tags (maybe a step to start a range, and another to end a range; also I just made the payloads string for headers and events, but you could structure that in yaml explicitly if you wanted to?)!

I just wanted to demonstrate the principle here - but even so, both these custom steps support the usual pypyr {formatting expressions} and of course you can use the usual decorators like foreach or while loops on them 😄

[blaisep]

Thank you Thomas. btw, I had suggested Markwhen for quick and pretty timelines. I suppose another perspective would be to have a more portable visualization toolkit using something like https://plantuml.com/timing-diagram where a pipeline could insert a text description which could be rendered by a wide range of tools to represent timing of events at varying levels of complexity.

…

On Fri, Jul 29, 2022 at 6:53 AM thomas ***@***.***> wrote: okay, and for option 2 (using custom steps to tidy things up a bit): Two steps in a mw subdir: 1. ./mw/init.py from pathlib import Path def run_step(context): context.assert_key_has_value(key='mw_init', caller=__name__) # applies {formatting expressions} if any mw_init = context.get_formatted('mw_init') path = mw_init.get('path', 'output.mw') # parent dirs for path must exist Path(path).parent.mkdir(parents=True, exist_ok=True) # subsequent mw.event steps will use path + date fmt. context['_mw_path'] = path context['_mw_dateformat'] = mw_init.get('date_fmt', '%Y-%m-%dT%H:%M:%SZ') headers = mw_init.get('headers') # only write if optional headers exist. # if no headers mw.event will create a headerless file on 1st run by itself if headers: with open(path, 'w') as f: f.write(headers) 1. ./mw/event.py from datetime import datetime def run_step(context): context.assert_key_has_value(key='mw_event', caller=__name__) # applies {formatting expressions} if any event = context.get_formatted('mw_event') # these values set in context by mw.init path = context.get('_mw_path') date_fmt = context.get('_mw_dateformat') if not path or not date_fmt: raise ValueError( 'You have to initialize markwhen handling with mw.init before ' + 'using mw.event') with open(path, 'a') as f: f.write(f'{datetime.utcnow().strftime(date_fmt)}: {event}\n') And now you can use them from a pipeline like this: steps: - name: mw.init comment: initialize the markwhen file with headers in: mw_init: path: my-markwhen-file-custom-steps.mw # note that headers passed through as str, not yaml headers: | title: my markwhen generated from pypyr description: with a description here #good: green #bad: red - name: pypyr.steps.echo in: echoMe: 1. do stuff. imagine there're multiple steps here. - name: mw.event in: mw_event: "stage1 #good" - name: pypyr.steps.echo in: echoMe: 2. do stuff. imagine there're multiple steps here. - name: mw.event in: mw_event: "stage2 #bad" This is really just a quick and dirty sample - you can definitely do more cute/clever things with date-formatting, defaulting, setting ranges and tags (maybe a step to start a range, and another to end a range; also I just made the payloads string for headers and events, but you could structure that in yaml explicitly if you wanted to?)! I just wanted to demonstrate the principle here. — Reply to this email directly, view it on GitHub <#285 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB6D6BJDRKZVPFSYE6IOBTVWOZYZANCNFSM543F6HKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- LinkedIn <https://www.linkedin.com/in/blaisepabon/> | Quora <https://www.quora.com/profile/Blaise-Pabon> | Github <https://github.com/blaisep> “If you want to go fast, go alone. If you want to go far, go together.” --African proverb

[yaythomas]

yep, for sure - and for any text-based output you can pretty much use the same principles/custom-steps illustrated above.

But really, the question is probably more what is suitable for your environment - i.e what's there already, and what exactly the data is for. Maybe there's some sort of ops dashboard/portal, connected to a structured logging provider, or maybe it is getting its data from various sources like json/a logging api/whatever. Like maybe Kibana/Elastic, or if this is a Microsoft estate something w PowerBI reading from a shared logging/timestamp db somewhere.

Regardless, from a pypyr perspective it's just a question of generating the required output format or calling the appropriate api - and you can pretty much follow the same sort of principle as the examples above with a custom step that does exactly what's appropriate for you.

On the cheap-and-cheerful side you could even use the built-in Python SQLLite db and then read it from Excel if your business users are more comfortable in spreadsheet reports. . .