-
Notifications
You must be signed in to change notification settings - Fork 118
Agency Visualization Requirements
This is part of Task 4 of the Phase 6B work plan.
Stakeholders: Please fill out this survey of the existing tools and workflows in use at your agency. We are interested in how you manage your model runs, what tools you use when analyze the results of those runs, and what capabilities you particularly like or particularly feel are missing/need improvement.
After your agency has filled out the document, we will follow up with each of you for a brief interview or "show and tell" so we can better understand what is working and what is missing.
Please consider the use cases mentioned in the task order when you are answering:
- Organize/access model results
- Model debugging
- Calibration/validation
- Project analysis
- Scenario comparisons
- Results presentation
This is all going into one Wiki document so you can all see each other's responses: but PLEASE only edit the section for your own agency! Thanks! ✨
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
MWCOG currently manages model run inputs and outputs in file folders. The ActivitySim-based Gen3 Model that is currently under development will likely adopt this system. Cube does provide a scenario manager but MWCOG staff intentionally do not use it because of the added complexities.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
Currently, MWCOG mainly uses ArcGIS to visualize outputs from our trip-based Gen2 Model. We plan to adopt a R-based ABM Visualizer developed by RSG for the ActivitySim-based Gen3 Model.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
Gen3 Model is currently at the Phase 1 Model Deployment stage. We will further evaluate ABM Visualizer when we gain more hands-on experiences with the model and the visualization tool.
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
ABM Visualizer was mainly designed for internal use. We are interested in visualization tools that can convey model outputs (e.g., outputs from a race-related equity analysis) in less technical and more straightforward terms to the Board or the public.
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
At the early stage of Gen3 Model development, our agency is open to either option.
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
- Models are implemented in Cube’s Application Manager. A new .cat file is developed for each Model of Record (MOR), which happens when a base year is validated, or a new LRP is developed. All model runs are added in the Scenario Pane, and when input files are modified in Scenario Manager, it leaves documentation on what input files and what settings were run for each project/year/scenario.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
- Work to develop a Power-BI dashboard is ongoing. This will be used to show ABM trip list outputs. It will possibly be edited to show output network statistics as well. Currently, a set of Cube scripts are run at the end, which produces validation statistics (highway and transit). These .dbf tables can be imported via macro into an Excel workbook and graphs are updated. Additionally, because Ohio has network standards (with common attribute names and definitions), a standard .vpr file is available for all Ohio models that can display standard network fields, e.g. V/C ratio or difference for assigned links with counts. Some R scripts are also available to display outputs from the ABM.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
- Subarea assignment validation statistics were supposed to be included, but were forgotten. (Isn’t hard to add, but needs a day.) The Power-BI effort is to make the display of the trip list easier and to develop some standard displays. (Currently, cube scripts are used to tabulate the trip lists.)
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
- TAZ maps are developed for the Base, TIP and LRP years showing various land use and employment/population and their change/growth. These are used in our Design Traffic Early Coordination meetings so that all parties can determine whether development is included in the MOR or not. Additionally, all MPOs have an EJ process and maps and tables are generated for those. However, those are not standard across Ohio.
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
- Depends on what is developed. Probably both, and potentially the answer is different depending on whether we are discussing the MPO or Statewide implementation. (Statewide obviously has additional required visualization.)
1. Run management. How do you currently manage model run inputs and outputs?
In short the answer is "Informally".
In long - Anything that is spatial (input / output) we try and retain in our PTV (Visum) database - this includes all network and zonal (three zone system) inputs and outputs. Visum is our one stop shop for spatial input and output data.
There are a large number of inputs/outputs that aren't spatial for these we try to keep everything in easy to machine readable csvs (or text files). We attempt to keep all inputs and outputs documented on our wiki (for meta data). Matrices are stored in OMX. For calibration data we strive to put target data into the same format as ABM output - that way any visualization / comparison tool we develop for calibration also works for ABM-2-ABM run comparison.
I would say our outputs (from CT-RAMP) are currently a mess. What we are pushing ActivitySim towards is to have an output structure that looks just like an augmented survey (with a single household table, person table, trip table, vehicle table...). There will likely be hundreds to thousands of interim decision context fields on some of these tables (like the trip table potentially). These should be stored in hdf5, but be user specified for which fields are exported in the user friendly csv that is used for 99% of analysis / visualization. This is a wish list design element, not current practice.
2. Visualization now: which tools do you currently use for visualizing model outputs? We use PTV-Visum to visual spatial data, along with ESRI-ArcGIS and QGIS We have an RSG custom written (in R) html visualizer to review ABM-2-ABM outputs. Additionally, since we have processed our target data to look like ABM outputs, we use this visualizer to compare the ABM to target data during build / calibration.
Everything else is custom written scripts for common data analysis tasks/requests. We are working on building a documented wiki library of these various task/request specific "functions" we are putting together.
Nothing we have is web-based, but we are looking to put our functions / tools on the web (shared) and documented.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? As noted in 1 - we want the ABM outputs better formatted - I think we are working towards that with ActivitySim. Other than that, we are pretty happy with our visualization tools for CT-RAMP, we are hoping to get similar (maybe improved) functionality in ActivitySim. A critical element for us is to be able to do multiple scenario comparisons. At least 2 scenarios. Looking at the results from one run is of very little value for us. Our current visualization tools are really only built for looking at 2 scenarios. It would be a very interesting and meaningful upgrade to be able to look at X number of scenarios with the given tool.
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? I think "public ready" visualization is something that maybe gets built way down the road. I think analyst tools and assistance is the primary (lowest fruit). For Oregon, outward facing visualizations have been and are envisioned to be developed on an as needed basis and few (if any) follow the same approach/output - each one is it's own snowflake.
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? Our hope would be that the common viz toolkit would be the toolkit that we use. But if it falls short, we will be using existing tools or building work arounds. But the hope is that over time we have less and less Oregon specific tools and more and more we have common viz tools.
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
Inputs that may vary by scenario, which are typically land use and network files, are stored on a network drive in folders that are uniquely named for a given scenario. Land use and networks are stored as unique scenarios so that a given network can be run with any land use and vise versa (constrained by forecast year). The model config file has five parameters that must be set: inputs_dir, forecast_year, base_year, landuse_inputs & network_inputs. During a model run, these files are automatically copied to the model run location. Static files that do not change for a given scenario are stored in a sqlite database, which is accessed by the model throughout a run.
Outputs are stored in folders within the model run.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
We create pre-aggregated .csv outputs that are controlled by an expression file. These files are then used to create automated Jupyter notebook html files. So when a run is complete, there should be a bunch of html pages for various summaries, including base year validation comparisons. We can also visualize a run using our Plotly Dash dashboard, which uses the same .csv files and runs on an aws linux instance. The dashboard is a fully interactive and web-based app, so it offers a centralized, easily accessible place to examine/compare various scenarios.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
Visually tracing/mapping a person/household tours and trips is something that is missing from our visualization toolkit. We can do it, but it is not automated so it is pretty onerous.
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
Change is fine, but I think it is important that a run automatically creates summaries that are immediately accessible and I think Jupyter notebooks--> html outputs makes a lot of sense for this. I would say the majority of our runs are looked at quickly and then deleted, so they often do not make it to our dashboard. We like Plotly Dash for web-based dashboards and have a lot of experience with it, but we are very open to other frameworks. Shiny is used quite heavily by our organization and I see a lot of positives around it.
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!