A common position on how to stratify risk and apply risk mitigation in Shiny/IDDs for GxP use

[epijim]

Proposal

An in person discussion would give a great opportunity to pool experiences with Shiny, Shiny validation/QC and it's place in decision making, and health authority interactions.

I would propose:

• In the lead up to the event, a small team pool existing efforts and prepare a draft decision tree for establishing risk in shiny apps/IDDs, and proposed strategies to mitigate risk including tool specific to shiny (e.g. shinytest2). We will also reach out to the R Validation Hub.
• At the event Dedicate ~2 hours at the event for an open discussion. With the smaller team presenting back the context and proposal, then opening to discussion and improvement.
• After the event Smaller team polish up the proposal to then circulate as a brief whitepaper/position statement.

Expected impact

This is a conversation happening across every company today, and is the elephant in the room when we talk about Shiny CSRs, use of IDDs in data monitoring committees.

The time is ideal for a focussed and achievable deliverable to be tackled in the precious time we can have F2F.

Prior work

• Working group with FDA on how to submit a shiny app
• Posit talk on how they test 2,500+ shiny apps
• More than one company seems to be looking at shiny for the DMC stage
• Roche uses Shiny (Teal) by default on clinical trials, including pivotal events like at unblinding
• Pfizer shared some principles around robust shiny
• Eli Lilly shared best practices for shiny](https://rinpharma.com/publication/rinpharma_34/)
• Merck shared an interactive, but js?, plot for interactive DMC work and github repo

Shared by Phil:

• @philbowsher has some links to examples/tools in a recent presentation
• Merck shared a framework for using shiny for HTA interactions
• idorsia has shared a framework for making the CSR in shiny

Would you be willing to potentially facilitate this discussion?

Yes definately!

[epijim]

From our roundtables, it sounded like Denali and Pfizer have internal documents already. At Roche we have had a working group on this topic, but it's not summarised into a simple decision tree - it's more split on department lines (e.g. a statistical programmer making a shiny app for use within their study is treated differently than a data manager making a shiny app for others to use across studies)

[harveyl888]

Oh man. I'd really want to be part of this discussion 😢 shinyValidator https://github.com/Novartis/shinyValidator might be a good starting point. David has since left Novartis but we could reach out to him to see if it's still under active development.

[matthiazzz]

I naively assume that risk mitigation of a shiny app mainly focuses on the GUI and its in- and outputs?

That is important, but I assume that it is worthwhile to at least mention mandatory pre-requisits like:

• reliable and re-producable system to spawn shiny app (i.e. a docker shiny server, posit connect, ...) and link to (re-producable/static and usable) data set
• shiny app for DMC is mentioned: incomplete, missing, false data coding will occur; how/what is risk mitigation strategy here?
• shiny apps use a ton of R packages and dependencies: controlled R package repository with all packages being already risk assessed?

It may be important to outline why and where risk mitigation of an R package and a shiny app are similar and where they are substantially different. Like static code/output vs interactive output challenges?

And, finally: can you do risk mitigation solely to the shiny app UI and completely ignore any "internals" like (very simplified) the underlying app.R, ui.R code, used R packages, deployed environment, etc?

[pzhang-cims]

Hi @matthiazzz, I'm interested in your shiny app application for DMC usage. Did you mean for closed session? I'm wondering if your shinyapps for DMC usage is host from unblinded statistician's organization and operated by unblinded statistician for display, or used by DMC statistician?

[rparmm]

Would love to be a part of this discussion. We had struggled with this. It resulted in some work building out UI based testing using Cypress. I feel that a lot end-to-end UI testing libraries (Cypress/Playwright) already do things like retryability/auto-wait really well compared to shinytest. The testing is also more realistic as browser based actions are done entirely on the browser as a user would interact with HTML elements rather than the abstraction of components through R. Furthermore, with R based deployments like POSITConnect, these testing libraries can also be made to do some load testing with multiple users and multiple scenarios.

[epijim]

This has been elevated to a workshop at the summit - thanks for your input!