Start formalizing documentation / protocol standards

[mr-c]

with examples in RST and MD

[ctb]

On Tue, Jan 27, 2015 at 10:14:19AM -0800, Michael R. Crusoe wrote:

with examples in RST and MD

Why are we doing protocols in MD? That is un-khmer-ly.

[mr-c]

RST alone is fine

On Tue Jan 27 2015 at 1:16:45 PM C. Titus Brown notifications@github.com
wrote:

On Tue, Jan 27, 2015 at 10:14:19AM -0800, Michael R. Crusoe wrote:

with examples in RST and MD

Why are we doing protocols in MD? That is un-khmer-ly.

—
Reply to this email directly or view it on GitHub
#743 (comment).

[ctb]

A few things coming up in dib-lab/khmer-protocols#148 -- cc @drtamermansour.

One issue is our relationship to Amazon, vs Rackspace, vs MSU HPC. I think we have to pick a platform and I'm +1 on Amazon, and defaulting to designing everything around Ubuntu 14.04 running on Amazon. No objections to thinking about a templating system that would let us generalize our protocols across multiple systems, but would like to talk about it before implementing. Thoughts?

[mr-c]

+1 for Ubuntu 14.04; installs using Debian packages; data mounted/linked
into homedir. Remaining software also installed to homedir without root.

This is the most portable way. We can provide instructions for Rackspace,
et cetera, by swapping out the setup instructions. Also allows for running
the protocols on Jenkins, as part of acceptance testing, and as part of
Debian's autopkgtest efforts.

See my work so far with Eelpond:
https://github.com/ged-lab/khmer-protocols/pull/147/files

On Mon Feb 09 2015 at 11:49:25 AM C. Titus Brown notifications@github.com
wrote:

A few things coming up in dib-lab/khmer-protocols#148
dib-lab/khmer-protocols#148 -- cc
@drtamermansour https://github.com/drtamermansour.

One issue is our relationship to Amazon, vs Rackspace, vs MSU HPC. I think
we have to pick a platform and I'm +1 on Amazon, and defaulting to
designing everything around Ubuntu 14.04 running on Amazon. No objections
to thinking about a templating system that would let us generalize our
protocols across multiple systems, but would like to talk about it before
implementing. Thoughts?

—
Reply to this email directly or view it on GitHub
#743 (comment).

[drtamermansour]

There are 2 virtualization types: The old paravirtual (PV) type and the
newer hardware virtual machine (HVM).
According to Amazon (
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html
), it is recommended to use HVM. However, the M1 instances that we used to
use does not support Linux HVM AMIs.

Does't make difference to standardize the virtualization types?

Side question: Are there preferable instance types?

On Mon, Feb 9, 2015 at 6:57 AM, Michael R. Crusoe notifications@github.com
wrote:

+1 for Ubuntu 14.04; installs using Debian packages; data mounted/linked
into homedir. Remaining software also installed to homedir without root.

This is the most portable way. We can provide instructions for Rackspace,
et cetera, by swapping out the setup instructions. Also allows for running
the protocols on Jenkins, as part of acceptance testing, and as part of
Debian's autopkgtest efforts.

See my work so far with Eelpond:
https://github.com/ged-lab/khmer-protocols/pull/147/files

On Mon Feb 09 2015 at 11:49:25 AM C. Titus Brown <notifications@github.com

wrote:

A few things coming up in dib-lab/khmer-protocols#148
dib-lab/khmer-protocols#148 -- cc
@drtamermansour https://github.com/drtamermansour.

One issue is our relationship to Amazon, vs Rackspace, vs MSU HPC. I
think
we have to pick a platform and I'm +1 on Amazon, and defaulting to
designing everything around Ubuntu 14.04 running on Amazon. No objections
to thinking about a templating system that would let us generalize our
protocols across multiple systems, but would like to talk about it before
implementing. Thoughts?

—
Reply to this email directly or view it on GitHub
#743 (comment).

—
Reply to this email directly or view it on GitHub
#743 (comment).

[mr-c]

@drtamermansour That's relatively easy to change later; I wouldn't worry about it for now.

When we are ready to do performance analysis we'll be able to come up with a good answer to this question.

[ctb]

A few things I noted --

My primary goal with the protocols is for them to be tutorials for doing practical data analysis. This may come in conflict with their use as acceptance tests; whenever this happens, we should choose the tutorial route, e.g. the less correct way that is easier to understand. (c.f. unnecessary use of multiple directories, unnecessary use of environment variables, extra jargon).

For the same reason, I've chosen not to be overly concerned with their cost to the user; we should choose a machine that is more expensive if it gets the job done with less latency. Obviously this involves judgement calls and tradeoffs.

More as I walk through the protocols :)

[mr-c]

Now that I have more experience with chroot's I'm fine with leaving in commands that run as root; that won't hamper the automated testing

Agreed that we need to favor understandability.

Here are changes I made in support of that:

• preferring the long version of options over the short
• using a new directory for each processing step
• using current versions of programs
• promoting good filesystem organization: making a top level folder for the pipeline, putting each program's source in a separate directory, not working outside of the user's home directory by mounting directly into it.

Sounds like you're -1 on explicitly changing into the directory at the beginning of each command block. That's fine with me.

Likewise it sounds like you'd like to avoid environmental variables such as ${HOME} and ${THREADS}. I'm okay with taking those out as well.

I was playing around with the parallel command and left implementations using it as comments. Those can be deleted (I wasn't expecting my pull request to be merged yet).

Here are some writing/formatting guidelines that I implemented:

• commands within text that a user could type in are quoted using double backticks
• asterisks that aren't inside double backticks or code blocks and aren't being used to indicate bold text need to be escaped using a backslash otherwise they won't show up in the output. Example: https://github.com/ged-lab/khmer-protocols/pull/147/files#diff-dbe37b50f17fded0319774772862ab6cL32
• using packaged version of software which eliminates both the download & compilation but also cleans up the command line as the full paths aren't required.

I also request the following stylistic change as it improves readability in vim's ReStructuredText mode:

• Indicate that a code block follows with the double colons at the end of the preceding paragraph not on a line by themselves. Sphinx will do reasonable auto formatting for you: If there is no period at the end of the preceding paragraph then Sphinx automatically puts a single colon in the output. If there is a period at the end then Sphinx won't add any punctuation.

Here's an example: https://github.com/ged-lab/khmer-protocols/pull/147/files#diff-888023f91d33aa4866c717764a7248a5L167

Other notes:

• the hardcoded paths to blast in blastkit need to be removed

[mr-c]

@ctb #743 (comment) is relevant to our meeting today