Minimize app startup time

[aclark02-arcus]

@jthompson-arcus, I recall chatting with you once about reducing the amount of data that's processed at startup. I think you suggested that maybe that the objects at the top of app_server.R could be generated before calling run_app(), correct? Just thought I'd open an issue for it to keep that thought moving forward.

[LDSamson]

Do you mean before calling shinyApp within the run_app function? We can try that.

Pulling object generation outside of run_app is a bigger design change but can also be considered.

We should first find the largest bottlenecks I think.

Related:

• Check if figures can be improved #211
• DB Creation and Updating #210

[aclark02-arcus]

I want to give this issue a +1 - we went live for a larger study last week and the app is significantly lethargic, and not only at app start, but especially at app start.

Perhaps a different issue should be opened for this, but even navigating through the app is slow. For example, clicking a new tab that hasn't been opened yet, clicking "Save" on records, etc.

[LDSamson]

@aclark02-arcus can you perform some profiling and check what is the largest bottleneck?
How much data is in the app for the study that you are mentioning?
And how many items do you have per form?
Is it only slow in the figure view or also in the table view?

[aclark02-arcus]

703,118 items total.
Largest form is AEs @ 133,142 items.

The slowness is in the tables.

[LDSamson]

@aclark02-arcus what makes your user database so big? Do you think that is the biggest culprit?
Maybe it is the pivoting in create_table that causes the issue? Can you check how long it takes to call create_table on appdata[["Adverse events"]]?

Did you already update to the latest clinsight version? Using the latest version could have some performance gains due to updated R system dependencies in version 4.5.x (although I don't know how much)

[LDSamson]

Simple script to test big datasets using dummy data:

library(tidyr)
library(dplyr)
library(clinsight)
##
duplications <- 50
##
df <- clinsightful_data |> 
  select(-site_code) |> 
  tidyr::separate_wider_delim(subject_id, "_", names = c("country", "site", "subject"))
big_data <- lapply(
  seq_len(duplications), 
  \(x) {
    df |> 
    mutate(
      site = as.character(as.numeric(site) + 10*x),
      site_code = paste(country, site),
      subject_id = paste(country, site, subject, sep = "_")
      ) |> 
      select(-c(country, site, subject))
    }
) |> 
  dplyr::bind_rows()

test_clinsight(big_data, metadata)

This way, the medication table becomes the biggest table.

[aclark02-arcus]

Simple script to test big datasets using dummy data:

I didn't see your reprex above! Apologies. But I did go ahead and run profvis on our internal data using:

profvis::profvis(print(
+   clinsight::run_app(launch.browser = TRUE)
+ ))

Below is a high-level summary of anything that takes longer than 1 second, marked with a red dot. If there is anything arcus specific, where we are rendering multiple forms within a item_group / tab, then I marked that with 3 red dots.

So, in order of biggest to smallest bottlenecks, we have:

• 13.3s - run_app.R/readRDS
• 10.8s - fct_tables.R/useMthod("create_table")
• 6.3s - app_server.R/app_tables lapply()
• 5.6s - mod_review_form_tbl.R/get_form_table() call
• 3.9s - fct_tables.R/create_table.default() for 'common_form' methods
• 3.3s - utils.R/split_review_data() calling db_get_table()
• 2.7s - app_server.R/app_vars creation via get_meta_vars()
• 2.6s - fct_data_helpers.R/get_unique_vars() call
• 2.5s - fct_tables.R/table creation for 'medication' methods
• 2.1s - fct_tables.R/create_table.default() for 'adverse events' methods
• 2.0s - fct_db_collect_data.R/db_get_table()
• 1.6s - fct_tables.R/create_table.default() for 'continuous' methods
• 1.2s - mod_common_forms.R/moduleServer()
• 1.1s - app_server.R/lapply-ing thru mod_common_forms_server()
• 1.0s - mod_review_form_tbl.R/moduleServer()
• 0.9s - mod_common_forms.R/mod_review_form_tbl_server() call
• 0.8s - app_server.R/available_data creation via get_available_data()

These total 61.7s at startup alone. I would be interested to see if the flame graph is similar for your reprex above. I'll run that now using main.

[aclark02-arcus]

Whoops, not closed!

[aclark02-arcus]

Here are the reprex profvis results for start up (below). I marked red dots next to the rows that appear to have at least moderate overlap with our internal deployment bottlenecks. Obviously the two apps are pretty difference since one is creating a db from scratch.

[aclark02-arcus]

Any thoughts about how to attack this @LDSamson & @jthompson-arcus?