customer pull: bayesian in b-school (2023-26)

[hyunjimoon]

before 2024, system dynamics audience has been my beach head in that I have MIT SD group who can give me feedback and collaborating closely with vensim which provides platform.

from 2024, i changed bayesian entrepreneurship #234 as my new beach head. but since both of them are in b-school, technology needed and the way it should be wrapped are not too different

---

Integrating Bayesian thinking with System dynamics: 10/20/2023 SD seminar

Building on #149, Tom and I are preparing SD seminar with, Charlie, Ozge, Travis, Abdullah, Vikash's help.

i wish to cover sd's position (which can or cannot be framed as statistical regression) and its implications in automation (including gen-AI).

q
1.1 what behavior should be learned for prescription? -> 1.2 minimum set of behaviors related to prescription?
2.1 how should it be learned for prescription? -> 2.2. minimum data samples for each learning?

• variable mapping (can we classify variables via their range)
• SD vs regression: complicated SD model can be perceived as exhaustive (mistakenly). And they may have experimenter bias.

[chasfine]

Possible paper outline/points

1. SD is a well-used and accepted modeling methodology in OR/MS and economics. Give a bunch of examples.
2. SD modeling tools have a somewhat less accessible UI/UX as compared with some other modeling approaches (e.g., linear programming and linear regression functionality build into Excel spreadsheet app.
3. There is opportunity to strengthen SD modeling tools and improve UI/UX, and here is what we (Angie/Tom?) did in terms of technical and usability contributions.
4. Case example: COVID paper by Hazhir, et al
5. Further possible applications and further opportunities for additional enhancements.

[hyunjimoon]

(name includes typo)

1. SD is a well-used and accepted modeling methodology in OR/MS and economics. Give a bunch of examples.

• Anderson POMS paper on Opportunities for System Dynamics Research in Operations Management for Public Policy
• OM: Sterman behav/bullwhip (1989?), Oliva: service
• orgsci: Rahmandad jobstrategy etc
• econ: kupman's phd-thesis
• climate, energy: john's phd-thesis, erling's amd chp.8

2. SD modeling tools have a somewhat less accessible UI/UX as compared with some other modeling approaches

• fig.1: vensim ui (prey-predator; hierarchical bayesian model)
• compared to the mathematical approach which include data variable (D) and can have closed form dF/dD, simulation is weaker in diagnostics for flowing in data. This requires simplified simulation tool that directly maps from which date (t) to real number (summary statistics)
• one way is to precompute f(x) for all disaggregated variables (pcsth: 223511071(3+11)) then return aggregated result upon request i.e. not on-demand

3. There is opportunity to strengthen SD modeling tools and improve UI/UX, and here is what we (Angie/Tom) did in terms of technical and usability contributions.

A. unified information unit across data-draws-decision-data loop

• fig.2 dynamic statistics
• hierarchical datastructure for convenient retreival (fx_pcsth)

B. statistical computation framework

• fig.3 sbc experimentation lab map
• expressing random variable as sampels (digitizing, stan rvar)
• how computation efficiency lead to better datastructure then visualization (xarray and arviz)
• starts from data standardization (xarray started from computation method)

C. decompose vensim functionality

• fig.4 mapping PAD modeling to vensim buttons
• as generator (p(y|theta) and p(theta)) and estimator (p_A(theta|y))
• need testable diagnostics which leads to D

D. dynamic diagnostics

• fig.5 visual diagnostics (poster second column)
• for data2draws estimator (5)
• for data2draws2decision generator, f, p (PA) (6~8)

4. Case example: Covid (+ CWD?)

• fig.6: workflow for covid model, datastructure, map it with fig.5 (estimator: 5, [generator, f, p]: 6~8)
• how ABCD improves problems (os-dependency, bugs, non-testability (+different opinions on reliability))
• suggest quantifiable boundary of reliable SD model size (5 stocks, 3 estimated parameters) with estimation under current vensim tool performance

5. Further possible applications and further opportunities for additional enhancements.

• fig.7: playground (with chatbot window?)
• llm model is another hierarchical model
• information unit from the decision to more data?

todo

• would limiting SD paper including estimation be better (1)?
• between Tom's example of Covid and CWD, which would fit better to our paper? (Covid: one story, can get deeper, CWD: diversity, more explicit stakeholders + have web publish)
• fig.4
• fig.7: benchmark openai playground

[hyunjimoon]

carefully selected; ui/ux is how to take this and glue it to
sd data: sparse time series (running model for twenty years, 5 data points, unequal spacing, big variations in variance - hetero skastacity, don't have one data; mix of sources, level of aggregation and time granularity),

interesting information (homogenous prior across the firms)

vertical + horizontal signal (granularity of parameterization emerge from the data )

[chasfine]

1. For list of papers, check whatvSD papers are most cited and in strong journals.
2. What is CWD paper?

[hyunjimoon]

Let data speak: automatically adjusting unit as data flows in

• Dynamic aggregation; best cluster may be different from clustered result #34

key idea / hypothesis

1. the hierarchical model provides a signal for vertical and horizontal focus

• vertical: institution from Jame's example in What entrepreneurial problem can Bayes solve? #157
• horizontal: DC.Washington (one of the 51 states) in Hazhir's example (vaccine taking population didn't change for DC - signalled bug in the model)
  Q1. How much outlier is a signal? which is better btw absolute and relative (e.g. variation in institution-level explains half the total variation)

2. signal can be connected to update the structural model
   Q2. does the model proceed to aggregate the unit until it finds something interesting? or elbow rule like clustering? model comparison measure p(D|M1) vs p(D|M2). start from no-pooling (as we can run parallel) then start aggregating by which standard? repeat this same process from disaggregating

3. slower clockspeed parameters' prior can be updated with higher clockspeed parameter's posterior

• tool: fx_pcsth data structure and plots
• e.g. of slower: structural (ode), unit of analysis (# of p, s, t and h), prior on variance of e.g. poisson data distribution (thickness of tail)
• e.g. of faster: prior of each parameter p
  Q3. does synthetic data play some role? is checking symmetry between generator and estimator important for the update process above?
  Q4. i) structure parameter (ode) ii) estimated parameter
• switch on/off (in xarray coords format)

case: covid model (SIR)

• f (summary statistics): ?
• x (time series value of component): 3
• p (parameter): 22 parameters -> 3?
• c (component): GDP, hospitalization, death cost (may not need)
• s (space): 51 states -> 7~9 (one group could stand out)?
• t (time): 1-1000
• h (hyperparameter, decision variable, action): vaccine amount, vaccine arrival time

figure:
data4dm.pdf
ref:
fig.1 vensim UI
fig.2 unified data structure: https://github.com/hyunjimoon/VaccineMisinf/blob/main/V81/tf_fx_pcsth.py
fig.3 SBC experiment lab: https://github.com/hyunjimoon/SBC/wiki/Customize-variability-and-target-testing
fig.4 parameter hierarchy from slow to fast, from generator to estimator: https://github.com/hyunjimoon/defense-reliability/blob/master/twin/Overview.ipynb
fig.5 how faster posterior triggers update of slower prior: https://amoon.world/%F0%9F%A7%ADamoon()/statistics/def(amoon(statistics%2C+dynamic))
fig.6 workflow for covid model: https://github.com/hyunjimoon/VaccineMisinf/tree/main#vaccinemisinf

[chasfine]

Is this the outline for a new /separate paper?

[hyunjimoon]

This shares the spirit but I need some feedback on the scope. From the diagram below, first proposal is 1, the second proposal is 2. Tom preferred 2 to 1 (reasons below), and I'm not sure which is better between 2 and 3 for the sole purpose of one paper. That being said, for SD seminar talk on 10/13, @tomfid and I are aiming for 2.

Tom shared his thoughts that designing UI/UX that supports Bayes workflow would be the first step. Meaning, first designing ui/ux that targets modeler (heavy user) which would ensure robust models to be produced, then moving on to a more general audience (light user) seems more reasonable. a.k.a starting the fix from one level deeper / closer to backend). That being said, as we are scaling 2, we are also nailing 1 e.g. this vensim webpublishing demo.

[hyunjimoon]

How would let data speak (LDS) modeling look like in econ growth context?

[hyunjimoon]

SD seminar on 10/20 12:30-2:00pm EST in the Jay W. Forrester conference room, E62-450

7 prep meeting

title:

goal:

[chasfine]

Maybe pick an industry (e.g., ready-to-eat cereals -- famous article by Dick Schmalensee of Sloan at https://www.jstor.org/stable/3003584) that has a lot of product variety added over time, and then try to get measures of whether consumers think quality has improved. My priors on the RTE cereal market is that variety did little to improve quality. I think Schmalensee would probably agree, but it has been very many years sinne I read his article. A more modern industry might be cell phones, where quality was increasing from the variety for a while but has perhaps slowed down now. Maybe there is a life cycle effect -- another of my priors.
Schmalensee-EntryDeterrenceReadytoEat-1978.pdf

[chasfine]

Maybe there is a life cycle effect -- essentially an S-curve effect

[okaranfil]

e.g. medical technology as an S-shaped curve, technology sector in my model, and Jack Homer's Ph.D. thesis on evolving medical technologies..

[tomfid]

[tomfid]

[hyunjimoon]

regarding oppty1, auto-subscript setter can be designed. Among the comments in #32, according to MIT city science lab, disaggregate approach has several important advantages over aggregate methods, which justifies the initial point as joint distribution / no pooling model, not full pooling. Although still there exist trade off between efficiency (computational) and effectiveness (hope of discovering the behavior from small level agent).

While aggregate approaches are primarily statistical in nature, disaggregate approaches directly explain why an individual makes a choice given their circumstances. Those analyzing the data are therefore better able to predict how those choices may change in different circumstances (Koppelman and Bhat 2006). Additionally, because disaggregate models represent the behavior of each individual, they allow us to visualize movement patterns in the city more effectively and to make more precise estimates of, for example, exposure to pollution as travelers make their way around the city. As discussed later in these notes, the mobility choices made by each individual can be further broken down into decisions regarding activities and locations, mode choices, and route choices. disaggregate model has an advantage of discovering the emerging behavior

[hyunjimoon]

From the above diagram, we decided that Angie leads application (lower right blue bubble), @tomfid leads modeling (upper left)

To be constructive for oppty 2 (make a bridge from bayes to dynamic models), I'm drafting the invitation from three Bayes crowds:

1. Stan (including StanKorea) audience
2. Bayes in business (have mailing list from AOM conference interested in Bayes)
3. statistical/computational/operations management Ph.D. friends and professors at MIT

What would be a good title for our talk?

• Prescriptive Bayes
• Data4DM^2 (decision making + dynamic model) System architecture
  -- Human (Community) system architecture (x-axis from the above: H1.tool builders, H2.model builders, H3.model policy designers, H4.model consumers)
  -- Machine (Computing) system architecture
  --- stat-dyn: M1.backend-estimator (HMC), M2.backend-generator (statistical model), M3.frontend (Stan language .stan),
  --- LLM: M1.backend-offline (documentation ingestion, model pre-training, data curation), M2. backend-data (vector db), M3. backend-generator + retriever, M4.backend-business logic, M3.backend-query M3.frontend, like that from Tom's last year slide)
  --- how the two hierarchies of human and machine interact

This could potentially be a preparation for Prescriptive Bayes conference drafted in #144.
Any suggestions, comments, or feedback would be helpful. @tomfid @chasfine

[chasfine]

Possible talk title? Integrating Bayesian thinking with systems dynamics for decision making

[hyunjimoon]

[hyunjimoon]

Integrating Bayesian thinking with systems dynamics for decision making

diagram above for ai-assisted decision making architecture

7 circles

• 5 decision makers (tool builder, model builder, policy designer, consumer, ai agent) + 2 products (database, ui/ux)

4 triangles

• 4 process(a1.define, b1.discuss, c1.define, b2.discuss)

3 squares

• highest uncertainty (e.g. culture, norm, behavior)
• needs clarification myself

17 arrows connecting 7 circles, 4 triangles, 3 squares

message

1. large statistical error for model builder and consumer

2. just as tool builder presents ai agent for b1.discuss as its agent, policy designer presents policy for b2.discuss. what difference can model builder make for b2 discuss i.e. is arrow11 necessary?

3. why can't we or shouldn't we replace human policy designer with ai agent? why do we need

• uk1~4 would be related to the area of the blue triangle?
• different direction of 3 vs 8. ai agent is output of a1 (input is tool builder and database) whereas ai agent serves as input for c1

[chasfine]

Angie, I find these notes to be too cryptic. Can you expand with sentences and/or short paragraphs?

[hyunjimoon]

@chasfine said it would be useful, with covid model, if we can describe how

1. decision (e.g. non-pharmaceutical intervention) based on generative program
2. generative program based on inflow of new information (e.g. decreasing uncertainty on covid being airborne disease).

today's agenda:

1. theoretical comparison of MCMC vs sequential MC
   a. Vikash thinks the relation between MCMC and MC is similar to gradient descent and stochastic gradient descent (the latter is much robust with part of the data)
   b. MCMC is slower but has clearer/graphical (divergence energy) convergence diagnostics
   c. many short parallel chains in MCMC ~ SMC?

2. Based on 1, design parallel workflow architecture that accounts for the convergence speed difference, in the covid context
   a. @tomfid can we measure the convergence speed difference e.g. MCMC takes 1 month, SMC takes 1 day (may increase as data increases)) in Vensim?
   b. account for 2.a in diagram here

---

below is version updates of @hazhirr's models (assumed it's sharable as he allowed to upload the model here):

V7 changes from V6: Changing the test and hospitalization sector completely

V8 with JS changes added include changing variable names and cleaning up the model, as well as new risk perception formulations.

V9 with HR modifying risk perception and hospitalization equations. Also deomgrpahic impact on fatality added in V9-HR

V10 with HR modifying post-mortem and fatality equations to fix a bug and be more transparent in fatality rates using more linear alterantives.

V11: Modified testing and calibration parts so that testing data, when availble, is used to drive the model (ratther than simulated testing data)

V13: Modified improved reading of testing from data (removing countries with only a couple of test data). Reduced free parameterse and ranges so calibration does not go to weird regions of parameter space.

V14: Update fatality equation to further reduce degrees of freedom and be mroe consistent with the analytical solution to death. Also further fine tuned the VOC ranges. Changed post-mortem test ratio to simplify. Added a new control panel for better tracking the data.

V15: Changed test capacity formulations to include a government response time for testing distinct from initial government response time. Also moved back to using simulation rather than data for testing, except for when there is a lot of data (US). Updated voc not letting non-covid acuity get too low. Made sensitivity to risk country specific.

V16: changed time constants for detection delay and resolution time to be based on data.; made the acuity level fixed at 1.4 , letting flu acuity to vary instead. Made the symptom density impact not increase beyong 1 .

V17: Removed travel section; Added pre-symptomatic stock different from those post onset but pre-testing and used a higher order delay; Removed Hospital Capacity Expansion; Added structure to capture hospital pressure depending on geographical distribution

V18: Added zero-inflated poisson structure to capture asymptomatics beyond poisson segment; Added increase in infectivity with symptoms; Added test sensitivity (false negatives) to the structure; cleaned up and improved robustness of extrapolation equations for testing and hospital $\mathrm{p}$ (missing symptoms)

V19: Add constraints for total death; fix sensitivity of testing that was missing for post-mortem tests;

V20: Updating parameter ranges based on V19 run; fixing hospital contact rate;

V21: Improving hospital demand to be limited to $10 %$ or so of Poisson part; Simplified hospital allocation problem which was too complex; fixed two errors in formulations for average acuity in not tested and related concepts; Changed post mortem testing to be based on coverage of testing in hospitals with a general parameter; Changed impact of acuity to fatality to enable higher order nonlinear effects; Changed subscripts so that baseline mortality can vary by country, but not impact of treatment on fatality.

V22: Limiting death drivers to only general parameters. Fixing post mortem testing equation.

V23: Fixing a problem in negative demand for test, to include recovered notest. Removing country-specificity in negative demand for testing; adding min contact fraction to list of calibrated parameters.

V24: Getting to negative binomial for calibration; correcting ranges for $\mathrm{R} 0$ not to go crazy high.; making testing sector all read from data files;

V25: Adding test capacity, rates, and cumulative to enhance calibration with test rate

V26 and 27: Fixing the inaccurate approximation of alloction of test; Adding changes in negative demand in resposne to recent infections.

V28: Making Total Asymptomatic Fracton a parameter; Removing time to test for negatives; cleaning unit errors; making dread factor country specific.

V29: Fixing an error in hospital capacity; Expanding minimum excess death attributable to COVID; Expanding Max Hospital Demand;

V30: Simplifying hospital capacity to not cut admissions in stock management.; Excluding data below 50 cumulative;

V31: Separating hosptial to be indepednent of non-hospital part

V33: Made infectiousness of symptomatic a step higher than asymptomatic.

V34: Make the hospital priority for untested dynamically respond to recent allocation received by non-covid hospital demand

V35: Add reality check test flags

V36: Fixing bugs in positive test fraction; making adj V48: $\mathrm{T}$ country specific.; adding priors for a few parameters with penalties for deviations

V37: Improving prior penalty deviation; adding post-mortem as country specific; removing weight on reports as country specific

V39: Adding impact of Weather; removing country-specific time constants.

V40: Making all parameters both general and country specific

V42: extending V40 architecture to all conceptual parameters that may be reported in the paper and updating the likelihood function to penaize larger variance.

V43: Simplifying variance formulation for random effects; making the penalty for excess mortality a real likelihood.

V44: Minor cleanups

V45: Update variable names based on documentation writup. Add structure for policy sensitivity at the end.

V46: Add flexibility to have real general variables as well

V47: Incorporating differences in asymptomatic infection into estimation

V48: Updating pre-symptomatic infectivity; adding parameter to stop calibration ; adding test capacity for future.

V49-51: changes in parameters for stdev of random effects and the excess death penalties.

V58: Add impact of learning and patient mix on IFR and adherence fatigue on behavioral response

V62: Create a simpler behavioral response function

V63: Cleanup mapping of behavioral response to policies and ready for Dec 23 analysis. Improve vaccination sector to make consistent with Stella model.

V64: Adding data to vaccination sector.

V65-66: Added error terms for initial unrealistic wave; added structure to use average recent test rates rather than last day.

V67: Add new variant rise to the model

V68: Add leaky vaccines, death despite vaccine, and loss of immunity

V70: Tighten the test demand variances; Changing post-mortem testing formulation; changing vaccination priority multiplier;

V71: Add new calibration component for excess mortality using total excess mortality rather than delta with covid counts

V72: A state resetting mechanism added to the model. Omicron variant added to model

73: Updating delays in measurement and death reporting; adding a saturation effect on risk perception so it does not go to very high levels and stay there forever

74: Separating Naive and NVx groups in the model

75: Adding country-level maxVacFrac coming from Vaccine model estimations; Adding projection process noise calibrated in ProcessNoiseGen model.

79: Adding structure for measurment timing of data; Changing total test allocation based on demand not population of each group; adding acuity loss due to deaths. Adding BA5 variant. Adding outcome measures needed.

[hyunjimoon]

Özge Karanfil and I had a chat on her paper with John on evidence-based testing which I feel can provide insight to our search for information sync between our two parallel cycles (1,2) and one large cycle.

1. how the policy designers and consumer interact
2. how the tool builder and modelers interact
3. how 1 and 2 interact to form a bigger cycle from

tool-builder:modeler+policy_maker:consumer from this figure is mapped to probabilistic computing system: SD+economics+statistics:practice

[okaranfil]

Hi Angie, we already exchanged some info on this but I'd like to reiterate that that's a fine analogy. Patients/ or the general public are "consumers of information" in our model for scientific evidence generation/ evidence synthesis & dissemination/ actual practice. It's a universal managerial problem seen in both medical & non-medical contexts, such as airport screening, background checks, tax audits, automotive emission tests, contentious jurisdiction, or on "consumers" of other kinds of the information who need to make a decision—on behalf of an individual, or for the whole population.. (You can read Pages 6-7 for background on judgment and decision-making lit and related references). Further layers of feedback may include cognitive & industry factors that may override scientific evidence (i.e. risk perception/fear and money and their interactions with the core evidence-based model), so the next stages of discussion with stakeholders immediately evolve into distrust in science, the pendulum in public opinion about the value of science, e.g. vaccine hesitancy, climate denial, guns in the U.S, etc.

So your cycle 1 may be similar to policy maker/ general public and cycle 2 policy maker/ clinician, I am not sure but probably no perfect analogy is needed there. There is a loss of efficiency at each of these interactions as there is no perfect adherence (clinicians don't perfectly follow policymakers, patients don't follow clinicians and/or policymakers, or only after a long delay), and even if there is perfect adherence, long time delays and bounded rationality (multiple stakeholders with different motivations such as in Beergame) leads to overshoot and undershoot, basically suboptimality in any of these areas we are looking at. Happy to provide more feedback as I see it evolving.. thanks

[okaranfil]

well, i understand because of the same analogy cancer screening tests are studied a lot in marketing literature..

[hyunjimoon]

You can read Pages 6-7 for background on judgment and decision-making lit and related references

What reference do you mean by pages 6-7? @okaranfil (you may already know but github supports pdf drag drop)

[hyunjimoon]

I (Angie) relocated @okaranfil 's comment below as a reply for better format

"Its the SDR 2020 paper with John"

[hyunjimoon]

ppl-sd/econ/stat-application for preparing 10/20 sd seminar talk

A: atom_supply: ppl, Tom
B: bit_supply: workflow, Angie
C: customer: sdeverywhere, Travis

Talk schedule (20mins each)
A,B: bit and atom supply circle (Tom)

B,C: matching supply with demand circle (Travis)

• how simulation affects policy

A,B,C: merging circles (Angie)

[hyunjimoon]

908. Tom and Angie
     idea: bayes in sd
     pd/commercialization: vensim_v11

atom, bit, customer

C: seminar audience
A:

• produce: vensim developers
• distribute: installed base of models and education (e.g. SD teachers around the world), Vensim manual (ui/ux. - https://mc-stan.org/docs/stan-users-guide/index.html)
• sell:
  B
• macro and cookbook

tools

Tool C1_segment

1. Are you selling into a beachhead market? yes to many (e.g. academic market (size??), other small markets(don't have specific market focus??))
2. Is the beachhead market saturated? everyone is vensim and stella users (double )

4. Expand to an adjacent market. - 1. replacing spreadsheet (1% excel) 2. datascience (e.g.bayes community)

Tool A1_Processify

1. Execute the process. Does it work well enough? - Yes: Move to step 2. - No: Tweak the process and return to step 1. (NO)
2. Standardize the process. (need to establish testing process, sales process)
   datastructure and ingest (sample .tab and get into pandas df)

Tool B1_Professionalize

1. Observe organizational performance. Any failures or shortcomings? - Yes: Move to step (YES), No: Continue with current team and processes.
2. Diagnose the organizational deficiency. - Identified: Move to step (other tools like probablistic program (ice sheet with MCMC) had better way of getting confidence interval) / not too much customer in the market (latent demand - sth they think they need is not actually needed) - resource allocation stuck in local opt.
3. Hire a professional to address the deficiency. - Successful: Return to step 1 to observe new performance.

Tool B2_Collaborate

1. Do you have potential partners in the value chain? - Yes: Move to step 2. - No: Build internal capabilities. (Stella is the same place in the value chain)
2. Can you share value without losing competitive advantage? - Yes: Establish a collaboration. - No: Reevaluate potential partners. (other consulting companies - sell tools / making the platform open enough to use other tools e.g. python, SDEverywhere? for this Tom used yellow: tool, blue: standard (file format))
   

Tool A2_Automate

1. Do you have a well-defined process for a task?
   • Yes: Move to step 2.
   • [-] No: Work on task standardization.
2. Is the process predictable and debugged?
   • Yes: Move to step 3.
   • [-] No: Refine and debug the process.
3. Do you have metrics to measure the process?
   • [-] Yes: Move to step 4. (regression set for vensim, macro, -- we need manufacturing, sales, hr, )
   • No: Establish relevant metrics.
4. Is there a process owner?
   • [-] Yes: Automate the process. (Tony kennedy vensim_v11 o)
   • No: Assign a process owner and then automate.

Tool B3_Acculturate

1. Do you have a defined company culture?
   • Yes: Move to step 2.
   • No: Define and communicate culture.
2. Is the culture being maintained during scaling?
   • Yes: Continue scaling.
   • No: Reinforce culture through regular communication.

Tool A3_Platformize

1. Does your business model allow for a platform structure?
   • Yes: Move to step 2.
   • No: Focus on product excellence.
2. Can you serve multiple customer segments?
   • Yes: Build a platform to cross economies of scale.
   • No: Reevaluate the business model.

Tool A4_Replicate

1. Is your process refined and documented?
   • Yes: Move to step 2.
   • No: Refine and document the process.
2. Can the process be replicated in different locations or settings?
   • Yes: Move to step 3.
   • No: Make necessary modifications.
3. Replicate the process and measure outcomes.
   • Successful: Continue replication.
   • Unsuccessful: Reevaluate and modify.

Tool A5_Capitalize

1. Do you need significant capital to scale?
   • Yes: Move to step 2.
   • No: Use existing resources to scale.
2. Are you willing to give up control for capital?
   • Yes: Seek external investment.
   • No: Explore bootstrapping or slower growth.

Tool C2_Evaluate

1. Do you have set milestones and KPIs?
   • Yes: Move to step 2.
   • No: Establish milestones and KPIs.
2. Are you meeting your milestones and KPIs?
   • Yes: Continue scaling.
   • No: Identify problems and adjust strategy.

[chasfine]

Regarding Can you share value without losing competitive advantage? : is this so relevant? Is your objective to build an empire or provide a social good for a community?

[hyunjimoon]

Are you selling into a beachhead market?

@tomfid what do you think? I looked up the exact concept of beachhead and Bill Autlet explains “A beachhead market is the place where, once you gain a dominant market share, you will have the strength to attack adjacent markets with different opportunities, building a larger company with each new following,” The three conditions that define a beachhead market: 1. The customers within the market all buy similar products 2. The customers within the market have a similar sales cycle and expect products to provide value in similar ways 3. Word of mouth communication exists between customers in the market.

[hyunjimoon]

Regarding replication: may not be so relevant for a purely digital product

Half joke: commercialization of bayes-based SD would be more welcomed in bayes' home country (England) compared to Norway. e.g. great supporter of Bayes-SD, Jim Duggins is near England. My point is location affects digital product adoption via customer/demand culture.

Musical-based movie (once, begin again, sing street) are more popular in Korea compared to U.S. but Barbie was less popular.

[hyunjimoon]

Can you share value without losing competitive advantage?

I interpret this chatGPT's logic (the above frame is chatGPT's first draft) as: is the rate of pie size increase faster than the rate of pie portion increase? Would Elon Musk sharing EV tech to grow the pie size and collaborate with new startups be an example?

• "As a business it seems smarter to grow the pie and increase your revenues even if Tesla's marketshare decreases." from here
• https://www.dailybreeze.com/2014/06/12/tesla-founder-elon-musk-giving-its-electric-technology-away-for-free/

[hyunjimoon]

regarding "resource allocation stuck in local opt" from Professionalize above, it would be interesting to connect to market failure (MR != MC) by framing tool_builder and policy maker as producer and model builder and consumer as consumer. Accepting econ convention that producer has smaller adjustment fraction (longe adjustment time) than consumer (which may be not true in cases where consumers has much lower agency power (e.g. pedestrian in transportation) meaning their underlying preferences change fast but revealed slowly), producer and consumer's delay may be connected to temperature delay and perception delay which can further be analyzed into four scenarios (red(osc), green(sat), blue (worse-better), yellow(better-worse (not nec. worse))).

[chasfine]

Regarding replication: may not be so relevant for a purely digital product

Half joke: commercialization of bayes-based SD would be more welcomed in bayes' home country (England) compared to Norway. e.g. great supporter of Bayes-SD, Jim Duggins is near England. My point is location affects digital product adoption via customer/demand culture.

Musical-based movie (once, begin again, sing street) are more popular in Korea compared to U.S. but Barbie was less popular.

[chasfine]

On replication vs segmentation: My thinking was that replication is entering a new (probably geographic) market with a (virtually) identical process, e.g, McDonald's builds virtually identical restaurants in new (USA locations) with the exact same marketing, product offerings, processes, etc. Pure replication. Software products, (e.g., Microsoft office) require even simpler replication -- essentially license and download a copy (which is identical to what everyone else gets). For Microsoft, replication is virtually effortless, for McDonald's it takes significant work.

Segmentation, on the other hand, in my thinking, requires a new product variant for a new market. So, if McDonald's or Microsoft is offering a Korean version of their model to that local market, they have to do (perhaps far) more customization in product, marketing, and perhaps process, to achieve adoption in the new language, culture, etc.

Perhaps this discussion raises the issue that replication and segmentation might occur on some continuum, rather than be orthogonal dimensions.

[hyunjimoon]

Agreed! The axis of that continuum may be ds_ratio.

uncertainty (=heterogeneity) is represented as sd.

1. does demand and supply mean match?
2. which is more informative btw demand and supply side?
   ds_ratio := demand sd/supply sd

supply_sd^2 = bit_supply_sd^2 + atom_supply_sd^2

replicate is prescribed when ds_ratio is large as it is more product agnostic hence process focused (eg IDEO) and push. segment when ds_ratio is small (~ customer pull). Recall plots we tried merging:right and left from

(Will replace to high resolution fig. later)

[hyunjimoon]

Attaching Charlie's diagram from Technology Roadmapping: A Tutorial

[hyunjimoon]

@tomfid asked for a brief feedback on his definition of SD (right). I have three and one question in the end.

1. I feel it has great overlap with the spirit of Bayesian Entrepreneurship: 🗺️map for startup compass🧭 #159, as colored below (A1-5 and T1-4). A4-T4 Orange for action-based/decision-making, A2-T12 blue for prior elicitation, A3-T3 green for testing/replication/quantifiable.

2. inference alg
   Considering the emphasis on empirics in T3, I hope different inference/calibration algorithm (MLE, MCMC, SMC) can be added to the definition as different algorithm returns different form of estimation which modelers should be aware of.

3. uncertainty representation may be harder in cont. time

Continuous time is a good example: nearly all SD models are continuous in time and value, and this is good; I’ve even written that Discrete Time Stinks (Fiddaman 2017). However, this need not be a defining characteristic of SD; it’s a technical choice governed by the need to achieve a good representation of a given system. Many SD games run in discrete time steps without harm, and a discrete interval might be a convenient representation for systems like an hourly electric power market.

Just a feeling, but fixing every objects to flow on continuous time may deter efficient uncertainty representation. Just as modeling large and slow world with quantum field model (not newtonian) is too luxurious and in fact can lower accuracy. Allowing different levels of discreteness to different clockspeed objects may be beneficial and based on this physics' quadrant (classic vs relativity vs quantum mechanics, field) I drafted below.

Q1. @tomfid from the sd genome doc you shared, what do you mean by “purely statistical models”? can’t all vensim models be cast as pure statistical model? (vice versa)

[tomfid]

Q1 Maybe I should have said "empirical." In any case, the idea is to exclude anything that has no a priori information about structure, where the model is derived exclusively from inserting data into some kind of generic structure (a linear regression or a random forest or example). The definition is also meant to exclude the opposite extreme, where you have no data, though that's rare in a sense, because if you venture to model something you almost surely have at least some kind of reference mode in mind.

[hyunjimoon]

Tom is visiting in person for the seminar! Hope I could have a meeting with Charlie and Tom to discuss the following roadmap of NSS with the help of SD (architecture framework is made by prob.comp lab and introduced in ppl minischool)

[hyunjimoon]

Based on Tom's draft of our talk's abstract, I connected this with economy growth model in the diagram below. In the seminar we could be the social welfare planner discuss what and how of "action parameters" in the context of SD community. Especially how vensim v11 can support $\psi$ increase.

Once upon a time, data was scarce, simulation was nearly unheard of, there were few ways to build and use quality dynamic models, and models rarely gave direct input to decisions. This made it easy for tool builders and tool users to coevolve within their own disciplinary stovepipes. The scale of our problems, and hopefully our opportunities, means that isolation is no longer viable. System Dynamics modelers can increasingly benefit from new tools in data science, AI and machine learning, and many fields could benefit from SD knowledge guiding better dynamics, realistic behavior, and useful interaction with decision makers.

Our future could be bright, with AI assistants freeing modelers from boring tasks, raising the productivity of thinking about systems, and making model results accessible to users. Or, it could be dark, with "ChatMDL" rapidly generating the simulation equivalent of 3-legged chicken images, models with superficial validity but more propaganda value than predictive accuracy, faster than we can debunk them.

In this seminar I'd like to explore the state of the SD tool portfolio and the requirements of modelers and model consumers. In passing, I will mention the near-term roadmap for Vensim, as well as the broader ecosystem of SD tools many of us use. But the real goal is to discuss the long term vision for SD. How do we realize a future in which, it's easy to make every model a Bayesian blend of structure and parameter priors from subject matter expertise and data likelihoods?
aggregation is not a dark art, but automated in a flexible and principled way?
every model run is a synthetic data experiment supporting decision making under uncertainty?
support for exploration of the state space yields as much understanding as we get from analytical methods on simple models?
we have the computing power to support these innovations, and we can explain what we're doing to influence people to solve our biggest problems.
I don't own the answers to all of these questions, so I hope we'll have an engaging discussion, and emerge with some common understanding of our prospects that will guide what we do next.

[hyunjimoon]

Framing tomorrow's talk as customer's prior elicitation (i.e. customer interview) it would be helpful if we can use today's session for constructing a signal to action map via simulated data testing.

setting

three actions

• $a_1$: improve infrastructure (bridge tools/standards/formats/apis)
• $a_2$: improve product
• $a_3$: communicate with end users, document

beachhead customer

• heavy-modelers that supply model to policy makers

product

BayesSD_vensim` with two main features:

• automate aggregation
• standardize prior elicitation (Model structure, Reference modes,Dynamic hypotheses, CLDs, Parameters sourced from SMEs, literature, other models), default priors,

meeting schedule

input from either Tom or Charlie are noted with @

1. sync Tom, Angie, Charlie's prior on BayesSD_vensim (hypothesis) - 20mins

• review our sd talk slide focusing on BayesSD_vensim @tomfid
• to design, develop, launch, produce, distribute, sell BayesSD_vensim product in the language of (evaluate, collaborate)|acculturate, (processify, replicate)|(automate, platformize)|capitalize

based on the synced prior of actions, customer, product, let's list critical hypothesis and their faithful experiments. These should be validated and verified. Plus each lower opportunity cost experiments are desirable. Three main tradeoffs are (details):

1. Increasing N decreases ${ValidStatBiasCost(\phi, N)}$, but increases ${OpportunityCost(\theta, \phi, N, M)}$
2. Increasing M decreases ${VerifConvgBiasCost(\theta|\phi, M)}$, but increases ${OpportunityCost(\theta, \phi, N, M)}$
3. Less specified hypothesis $\phi$ on customer, technology, organization, competition may increase ${VerifConvgBiasCost(\theta|\phi, M)}$, but generality of the test result may decrease ${OpportunityCost(\theta, \phi, N, M)}$

N is sample size for validating criticality, M is sample size for verifying faithfulness and detail of ValidStatBiasCost, VerifConvgBiasCost, VerifConvgBiasCost are in Table in the last.

For each actions, we'd repeat 2, 3. Then we'd evaluate the action

$a_1$: improve infrastructure (bridge tools/standards/formats/apis)

2. elicit hypotheses - 10mins

2.1 list one hypotheses for each sector - 5m

2.1.a: atom_supply (produce, distribute, sell -> efficiency): connecting the

2.1.b: bit_supply (design, develop, launch -> effectiveness)

2.1.c customer_demand (design, develop -> satisfiability): main users and influencers (John, Hazhir, Johan, Vicky, Ed, Travis) would find BayesSD_vensim usable

3.1.c should be based on critical elements of users (modelers)

2.2 test each hypotheses' criticality - 5m

validate each hypothesis for each sector's expert

2.2.a: atom_supply @tomfid

2.2.b: bit_supply @hyunjimoon

2.2.c customer_demand: @chasfine

2.3 sample number to validate hypotheses' criticality - 5m

2.3.a: N_a?

2.3.b: N_b?

2.3.c: N_c? @chasfine, @okaranfil, John, Hazhir, Johan, Vicky, Ed, Travis

based on the synced prior of product and critical hypotheses,

3. elicit experiment - 30min

3.1 list two experiments for each hypotheses - 5m * 3

3.1.a. atom_supply (produce, distribute, sell):

3.1.b. bit_supply (design, develop, launch):

3.1.c. customer_demand (design, develop):

3.2 test each experiment's faithfulness to each hypothesis

verify to each experiment's expert

3.2.a. atom_supply (produce, distribute, sell): @tomfid

• first experiment:
• second experiment:

3.2.b. bit_supply (design, develop, launch): @hyunjimoon

• first experiment:
• second experiment:

3.2.c. customer_demand (design, develop): @chasfine

• first experiment:
• second experiment:

3.3.a: M_a1, M_a2?

3.3.b: M_b1, M_b2?

3.3.c: M_c1, M_c2?

$a_2$: improve product

2. elicit hypotheses - 10mins

2.1 list one hypotheses for each sector - 5m

2.1.a: atom_supply (produce, distribute, sell -> efficiency): connecting the

2.1.b: bit_supply (design, develop, launch -> effectiveness)

2.1.c customer_demand (design, develop -> satisfiability): main users and influencers (John, Hazhir, Johan, Vicky, Ed, Travis) would find BayesSD_vensim usable

3.1.c should be based on critical elements of users (modelers)

2.2 test each hypotheses' criticality - 5m

validate each hypothesis for each sector's expert

2.2.a: atom_supply @tomfid

2.2.b: bit_supply @hyunjimoon

2.2.c customer_demand: @chasfine

2.3 sample number to validate hypotheses' criticality - 5m

2.3.a: N_a?

2.3.b: N_b?

2.3.c: N_c? @chasfine, @okaranfil, John, Hazhir, Johan, Vicky, Ed, Travis

based on the synced prior of product and critical hypotheses,

3. elicit experiment - 30min

3.1 list two experiments for each hypotheses - 5m * 3

3.1.a. atom_supply (produce, distribute, sell):

3.1.b. bit_supply (design, develop, launch):

3.1.c. customer_demand (design, develop):

3.2 test each experiment's faithfulness to each hypothesis

verify to each experiment's expert

3.2.a. atom_supply (produce, distribute, sell): @tomfid

• first experiment:
• second experiment:

3.2.b. bit_supply (design, develop, launch): @hyunjimoon

• first experiment:
• second experiment:

3.2.c. customer_demand (design, develop): @chasfine

• first experiment:
• second experiment:

3.3.a: M_a1, M_a2?

3.3.b: M_b1, M_b2?

3.3.c: M_c1, M_c2?

$a_3$: communicate with end users, document

2. elicit hypotheses - 10mins

2.1 list one hypotheses for each sector - 5m

2.1.a: atom_supply (produce, distribute, sell -> efficiency): connecting the

2.1.b: bit_supply (design, develop, launch -> effectiveness)

2.1.c customer_demand (design, develop -> satisfiability): main users and influencers (John, Hazhir, Johan, Vicky, Ed, Travis) would find BayesSD_vensim usable

3.1.c should be based on critical elements of users (modelers)

2.2 test each hypotheses' criticality - 5m

validate each hypothesis for each sector's expert

2.2.a: atom_supply @tomfid

2.2.b: bit_supply @hyunjimoon

2.2.c customer_demand: @chasfine

2.3 sample number to validate hypotheses' criticality - 5m

2.3.a: N_a?

2.3.b: N_b?

2.3.c: N_c? @chasfine, @okaranfil, John, Hazhir, Johan, Vicky, Ed, Travis

based on the synced prior of product and critical hypotheses,

3. elicit experiment - 30min

3.1 list two experiments for each hypotheses - 5m * 3

3.1.a. atom_supply (produce, distribute, sell):

3.1.b. bit_supply (design, develop, launch):

3.1.c. customer_demand (design, develop):

3.2 test each experiment's faithfulness to each hypothesis

verify to each experiment's expert

3.2.a. atom_supply (produce, distribute, sell): @tomfid

• first experiment:
• second experiment:

3.2.b. bit_supply (design, develop, launch): @hyunjimoon

• first experiment:
• second experiment:

3.2.c. customer_demand (design, develop): @chasfine

• first experiment:
• second experiment:

3.3.a: M_a1, M_a2?

3.3.b: M_b1, M_b2?

3.3.c: M_c1, M_c2?

#4. choose action: 10 * 3m
compute cost for each action

4.1 $a_1$: improve infrastructure (bridge tools/standards/formats/apis)

• N_a:
• M_a1:
• M_a2:
• N_b:
• M_b1:
• M_b2:
• N_c:
• M_c1:
• M_c2:
• opportunity cost?
  sum:

4.2 $a_2$: improve product

• N_a:

• M_a1:

• M_a2:

• N_b:

• M_b1:

• M_b2:

• N_c:

• M_c1:

• M_c2:

• opportunity cost?
  sum:

• $a_3$: communicate with end users, document

• N_a:

• M_a1:

• M_a2:

• N_b:

• M_b1:

• M_b2:

• N_c:

• M_c1:

• M_c2:

• opportunity cost?
  sum:

---

e.g. on desiging state2signal system anticipating the noisiness
An agent has to decide between two actions a 1 and a 2 , uncertain of the prevailing state of nature s, which can be either s 1 or s 2 . The agent’s payoff as a function of the action and the state of nature is as follows:

u(a 1 , s 1 ) = 10, u(a 1 , s 2 ) = 7

u(a 2 , s 1 ) = 5, u(a 2 , s 2 ) = 11

The prior probability of state s 1 is p = Pr(s 1 ).

a. Give a graphical representation of the agent’s decision problem as a function of the probability of state s 1 . If p = 0.4, what is the agent’s optimal decision?

b. Let the agent have access to a signal y before taking an action. Assume y has two possible outcomes with the following likelihoods:

Pr(y 1 | s 1 ) = λ 1 , Pr(y 1 | s 2 ) = λ2

How valuable is this information system if λ 1 = λ 2 = 0.5? How valuable is it if λ 1 = 0.5 and λ 2 = 0?

c. Show that, irrespectively of the prior, the information system λ 1 = 0.5 and λ 2 = 0 is preferred to the information system λ 1 = 0.5α + 0.5 β and λ 2 = β , where α and β are any numbers between 0 and 1. (Hint: show that latter system is garbling of former.)

Table 1: Definition of Costs associated with Entrepreneurial Experiment $\theta$ to Test Hypothesis $\phi$

-	def (related concept in E-strategy)
ValidApproxBiasCost($\theta$)	Cost of approximation bias caused by approximating utility with testing hypothesis $\phi$
ValidStatBiasCost($\theta, N$)	Cost of statistical bias caused by approximating utility of beachhead customer population with that of N sampled customers
VerifApproxBiasCost($\theta$ | $\phi$)	Cost of approximation bias caused by approximate testing hypothesis $\phi$ with designed experiment $\theta$
VerifConvgBiasCost($\theta$| $\phi, M$)	Cost of optimization bias caused by limited resource (represented as time $M$) early stops experiment $\theta$ to test $\phi$ before reaching to the optimal
OpportunityCost($\theta, \phi, N, M$)	Resource, time, and strategic costs of an experiment $\theta$ to test hypothesis $\phi$

[hyunjimoon]

Collected data on "Q1: What do your customers want and what are they willing to pay for?, Q2: One wish / your vision for using Vensim? What new tech would delight you?, Q3: What topics in system dynamics should Vensim 11+ focus on?" from on/offline audience (feel free to add, preferably with your name for us to folllow up)

Recording

Chat in the thread below.

[hyunjimoon]

Angie's digest - wish open discussion!

00:22:52 Angie Moon: https://docs.google.com/document/d/1y78ft5cn7OjycDE0HPbh3HyotohvyF7QWriT1h48eEE/edit?usp=sharing
00:23:00 Lucas Meneguim (UiB - System Dynamics Group): Reacted to "https://docs.googl..." with 👍
00:23:10 Angie Moon: Please let us know your opinion in the above document! Thanks
00:24:35 Al Thibeault: I’m working on the rare earth industry transition as it relates to the clean energy transition
00:24:52 Asmeret Naugle: I’d say my customers want solutions to problems. They want us to have the right solution method ready when a problem pops up, to answer it right away. They aren’t always willing to pay for the development to get it to that point, but they are willing to pay for the solutions when there is a problem.
00:25:54 jim: I'm a consultant. My clients want: Understanding, "the" answer, and affirmation. (Only one of which I want provide).
00:27:07 Asmeret Naugle: Adding to my previous comment: I have a few customers who are willing to fund an R&D “vision”, letting us create new capabilities. Those are usually my favorite customers!
00:29:02 Yuichiro Monden: I think typical template models for their own businesses will be very helpful for customers since they can save time to make models if they are not researcher.
00:31:37 Lucas Meneguim (UiB - System Dynamics Group): Reacted to "I think typical te..." with 👍
00:32:40 Angie Moon: Thanks for all the participation + valuable feedback!
00:33:05 Karim Chichakly: Reacted to "I'm a consultant. M..." with 👍
00:33:23 Lucas Meneguim (UiB - System Dynamics Group): If I could give a feedback directly to vensim, I would like to drag things without necessarily clickling on the "dragging" button
00:33:36 Karim Chichakly: Reacted to "If I could give a fe..." with 👍
00:33:38 George Papachristos: Reacted to "If I could give a fe..." with 👍
00:33:44 Mariya Andreeva: Reacted to "If I could give a fe..." with 👍
00:34:21 Zakie Mamashli: Reacted to "If I could give a fe..." with 👍
00:34:41 Mariya Andreeva: Reacted to "I think typical temp..." with 👍
00:34:57 Lucas Meneguim (UiB - System Dynamics Group): It is pretty annoying to do this each time. Plus. Adding equations and polarities in an easier way when im modelling. On vensim we need to open a pop up and manage it there rather than having an easily accessible tab on of the sides to do so
00:35:22 Mariya Andreeva: Reacted to "It is pretty annoyin..." with 👍
00:35:42 Lucas Meneguim (UiB - System Dynamics Group): And sometimes this kills momentum when you are feeling creative to model something, or simply wants to create a quick model to show an idea to someone
00:36:03 Celia Stafford: Replying to "It is pretty annoyin…"
The +- on the bottom of the screen are my favorite addition recently. You can click them after just selecting an arrow.
00:36:22 Lucas Meneguim (UiB - System Dynamics Group): Reacted to "The +- on the bott..." with ❤️
00:37:05 Karim Chichakly: Yes, then they lose confidence in the model.
00:39:55 Asmeret Naugle: Adding to the wish list: I would love to see better data analytics and data integration, especially data-inspired models. Our models often need to be supported by data, and it would be great to push toward things like structural validation (for example, causal inference methods). This area needs a lot more R&D, but it would really help with confidence
00:39:58 Travis Franck: Replying to "If I could give a fe..."

One undocumented Vensim feature is the ability to switch tools by hitting 1,2,3,4,5 etc keys. This makes switching tools much easier. And it actually wraps to Q,W,E,R,T keys too. Not fix your problem, but might help.
00:40:48 Lucas Meneguim (UiB - System Dynamics Group): Reacted to "Adding to the wish..." with ❤️
00:43:09 Tse Yang Lim: More than a threat of models as propaganda, I think, is just [unintentionally] bad models. But I guess that falls under 'replication crisis'
00:43:25 Dũng: Can you suggest any scholar or institution using system dynamics modelling for Marxism?
00:45:42 Esteban Ribero: Brain drain. It is hard to build a career using SD giving the low demand for it so, modelers who want to grow they have to abandon SD work to focus on other areas were the demand is.
00:46:17 Karim Chichakly: Reacted to "Brain drain. It is h..." with 👍
00:46:23 Esteban Ribero: I wish I could do SD for a living but, if not in academia then choices are really limited
00:47:12 Asmeret Naugle: Replying to "Brain drain. It is h…"
I think we could improve this by integrating more modern methods with SD. The more we can utilize LLMs, big data, etc., the more relevant we’ll be.
00:48:23 MJ: My observation shows that both have MUCH room for improvement but if this is about comparison, academics do slightly better
00:49:03 Karim Chichakly: I agree with that assessment, John. There is generally no peer review and things are always on a time deadline.
00:49:28 Karim Chichakly: I also agree on continuous automated testing.
00:49:39 Travis Franck: Reacted to "I also agree on cont..." with ❤️
00:50:05 Asmeret Naugle: I’d guess that a lot of it depends on the topic, too. Applications with easily available data and established theory can probably do better on quality/replication.
00:50:14 Travis Franck: Replying to "I also agree on cont..."

THIS. So important to have better continuous checking built into model development.
00:50:24 Karim Chichakly: Reacted to "THIS. So important ..." with 👍
00:50:36 Asmeret Naugle: Reacted to "I also agree on cont…" with ❤️
00:50:49 Angie Moon: Replying to "I also agree on cont..."

Agreed! Will covered in the talk soon e.g. standardization
00:51:08 Tse Yang Lim: "We need another half day of discussion" - these are critical topics for the community, it would be worth having some kind of dedicated workshop or such (as long as it's committed to actionable outputs) to discuss them
00:53:43 Tse Yang Lim: Reacted to "THIS. So important ..." with 👍
00:54:40 jim: If only one wish: Better tools for model analysis, meaning making it easier to trace a particular aspect of behavior to structure and understanding how the structure generates that aspect of behavior.
00:54:57 Cynthia: Reacted to ""We need another hal..." with 👍
00:55:24 Zakie Mamashli: Reacted to "If only one wish: B..." with 👍
00:56:03 WooJung Kim: Reacted to "If only one wish: B..." with 👍
00:56:24 WooJung Kim: Coming from learning design area, I wish there would be more online workshops (such as ArcGIS) and more intuitive UI(1. Easier short keys and manuals for them 2. Having instructions as shadowed fonts when you create a variable/flow/auxiliary variables) Cloud function for sharing models would be also great. + Providing templates like ArcGIS again would be an option that could be adopted — it would allow novice users to get the intuition of building models.
00:56:42 Tse Yang Lim: Not super pie in the sky but - good integration with visualisation esp. with uncertainty, e.g. for priors, posteriors, outputs over time
00:56:53 Angie Moon: Reacted to "Not super pie in the..." with 👍
00:57:33 Tse Yang Lim: Replying to "Coming from learning..."

My old GIS professor (Dana Tomlin) developed tool-by-tool, function-by-function tutorials for almost every functionality in ArcGIS. Have long wished we had an equivalent for Vensim
00:57:49 WooJung Kim: Reacted to "My old GIS professor..." with 👍
00:59:45 LAUBLE: I am satisfied with Vensim, and after more than 20 years of use, I am far from being able to use properly the tool. People expect too much from software when the solution depends from them and their ability to use it. I expect from the next Vensim versions, not more features, but a better explanation of the present features and how to use them properly.
01:00:23 jim: Wish two: Automatically increase robustness of a model -- i.e. make the modeled hypothesis robust to parameter changes and simple structural change (e.g. putting in a pecerption delay).
01:02:19 Al Thibeault: The brain drain discussion is related to discussions in the SD Society University Innovation Team meetings - contact Meagan Colvin

01:03:18 jim: Wish 3: Help creating policy (decision rule) chenages. Perhaps changing a single answer (e.g. introduce a product at this point in time) to a policy answer.

Not sure robustness is the goal. i don't like some myths around sensitivity check. natural for model to change according to priors. comparison and evaluation might be better happen in parameter space ("is this prior that results in certain outcome acceptable? within our reason?")

01:03:56 WooJung Kim: Reacted to "Wish 3: Help creati..." with 👍
01:07:36 Tse Yang Lim: I think the agent level helps with face validity
01:08:08 Tse Yang Lim: Even if you end up in the same place with aggregation, there are many audiences for whom thinking about individuals and agent behaviour is more intuitive than thinking about populations
01:08:10 Esteban Ribero: Reacted to "If only one wish: B…" with 👍
01:09:09 LAUBLE: A nice Vensim feature would be to find the best balance between making a model as simple as possible and still sufficiently useful relative to its purpose.
01:14:10 Asmeret Naugle: Does anyone here know if anyone has tried to fine tune a foundation model to develop CLDs from text?

01:14:25 Jeroen Struben: Aggregation flexibility also facilitates estimation when needing to move between partial model estimations, at more fine-grained level, and full closed-loop at more-aggregate levels.

@tomfid let's invite Jeroen to our Friday chat to discuss aggregation flexibility!

01:14:47 Tse Yang Lim: Reacted to "Does anyone here kno..." with 😮
01:15:14 Celia Stafford: Replying to "Does anyone here kno…"
Not that I know of but it’s something I’ve been looking a bit at if you want to help me with it :)
01:16:27 Asmeret Naugle: Replying to "Does anyone here kno…"
I’d definitely be interested in talking more!
01:17:26 Angie Moon: Reacted to "Aggregation flexibil..." with 👍
01:17:33 LAUBLE: Is reality check now running in compiled mode?
01:19:26 LAUBLE: Is it possible to run multiple reality checks in batch mode for instance using the DLL?
01:21:38 Karim Chichakly: Which was a major point of XMILE...
01:22:23 Tse Yang Lim: Very exciting list of soonish improvements!
01:23:27 Angie Moon: Arviz: https://python.arviz.org/en/stable/#
01:24:26 Travis Franck: sdeverywhere.org, for those interested. Ever evolving but some great features. Happy to try to demo it in a future webinar.
01:24:46 Asmeret Naugle: Reacted to "sdeverywhere.org, fo…" with ❤️
01:25:12 WooJung Kim: Reacted to "sdeverywhere.org, fo..." with 😮
01:28:54 Karim Chichakly: Anyone is welcome to join the SD Tools Users Group, where the tools discussed are 3rd party: https://groups.google.com/g/sd-tools
01:29:09 Travis Franck: Reacted to "Anyone is welcome to..." with 👍
01:30:02 Tse Yang Lim: Reacted to "Anyone is welcome to..." with 👍
01:30:45 Chris Campbell: Replying to "sdeverywhere.org, fo..."

There's a link on sdeverywhere.org to the SDEverywhere GitHub project, but here's the direct link (contains videos, quick start guide, and more): https://github.com/climateinteractive/SDEverywhere
01:32:56 Angie Moon: Test quantity-centered Workflow from the talk can be found in https://github.com/Data4DM/stanify#workflow

01:35:16 Asmeret Naugle: Left, but I suspect this audience isn’t representative :)

What does Asmeret mean by representative?

01:35:37 Tse Yang Lim: I was on the left
01:36:15 Tse Yang Lim: But it's partly because I find doing the stuff to the right to be more part of 'my job', to be done bespoke for any given audience

@tseyanglim what do you mean by " doing the stuff to the right to be more part of 'my job', to be done bespoke for any given audience"? not following..

01:36:28 Celia Stafford: Reacted to "But it's partly beca…" with ❤️
01:37:03 Asmeret Naugle: My guess is that the right side will be of most interest to the SD community, but the left side needs more focus to make SD relevant with the outside world.
01:37:33 Tse Yang Lim: Reacted to "My guess is that the..." with 👍
01:40:34 Karim Chichakly: For people like you and me, Tom (and Travis), it's necessarily not binary.

What information on Karim Chichakly and Asmeret Naugle would help me understand Karim's last comment?

01:41:02 Travis Franck: Reacted to "For people like you ..." with 👍
01:44:18 Karim Chichakly: Thanks Tom!
01:44:22 Esteban Ribero: Great discussion. Thanks!
01:44:25 Tse Yang Lim: Thanks Tom & Angie!
01:44:26 Oliva, Rogelio: Thank you, Tom!
01:44:29 Asmeret Naugle: Thanks Tom! Great presentation and conversation.
01:44:30 Yuichiro Monden: Thank you very much!
01:44:38 Rick Hubbard: Thank you very much to everyone involved in making this happen! Very valuable!
01:44:49 Zakie Mamashli: Thank you very much!
01:45:03 Jeroen Struben: Terrific thanks Tom, Angie, and all
01:45:27 jim: I think the turnout was not just due to the topic, but also due to the (reputation of) the presenter.

[hyunjimoon]

@chasfine @tomfid

partial realization of text-based diagram planner from feedback during talk from Johan Chu @advrk

https://diagrammergpt.github.io/?fbclid=IwAR3gw0iICjb7Ns5xCc-x2gQRZhbwtiMzEI2VNqRfUv3wDrbxqxpnVs7VSRc

[hyunjimoon]

I shared a recent bayesian software paper with hazhir and tom and was glad to identify interest in bayesflow and likelihood models.

Simulation-Based Prior Knowledge Elicitation for Parametric Bayesian Models is one result of the research direction I was working on with Andrew and Paul back in Columbia. I'd be interested in discussing ways to apply this to improve Vensim's data structure. As long as data structures are well enough as below, managerially and statistically, I'm all for reverse engineering the inference based on several graphical diagnostics.

invertible network by bayesflow. More approachable because it's python-based, stems from Stan, tested with posterior recovery plots and SBC.

Although its ode solver is in its infant step, we can learn the following:
i) rejection sampling based on an analytic solution which gives nonuniform joint prior as attached, from their notebook estimation for ode notebook
ii) plot design by prior location and scale, noise fraction as attached from their paper Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks
iii) feel for magnitude on prior location (5), scale (2.5), noise fraction (.5)

Based on Tom's draft, I developed SD parameter table for SBC as attached; for our next step, discussion on concrete hypothesis and plot will be greatly helpful for experiments. My hypothesis:

1. fractional time is inference-friendly (from parameter table, A3 instead of A2) - tested

2. configuration for lognormal likelihood exists which captures multiplicative noise and is inference-friendly as normal likelihood-tested

3. closer the estimated parameter is to the process noise source (e.g. production adjustment time is closer to process noise than target delivery delay) uglier parameter recovery becomes. if we can construct a controlled setting well, this sounds interesting

4. higher the multiplicative process noise is (e.g. process noise scale .1 vs 3), the uglier parameter recovery becomes

5. compare 2,3 results with process noise structure on and off (i.e. scale =0) hypothesis?

3-5 requires a thorough understanding of mechanism from your comment "process noise can change dynamics" but am lost

For speedup, reparameterization, not too diffuse prior, small HMC step size (.01), and reduce_sum are four strategies I am testing.

[hyunjimoon]

Integrating Bayesian Inference and System Dynamics with Case Studies in Epidemiology.pdf
got accepted to Stancon2024 (talk) and below is desire, belief, plan i prepared for a meeting with tom and jair

Desire

D1: develop an integrated workflow (from algorithm builders to stakeholders)
D2: extract general principles from Jair's five papers (p.13 from slide) by 2024 summer
D3: write a paper together on tool dynamics in an ecosystem

Belief

B1: tools will get integrated potentially with the help of word to world architecture that provide substrate for probabilistic reasoning, hence D1
B2: D2, D3 can help us plan our tools' (stanify, readsdr, vensim) future. I feel uncomfortable not using what I built from (build what we need and use what we've built philosophy I follow)
B3: D3 requires not only knowledge on emerging technology and its "meaning" to us, but also skills to manage process i.e. technology roadmapping

Planning

Plan3(B2, D2) for talk preparation, @jandraor, what input from @tomfid and I can help the most efficient generalization of your work?
Plan3(B3, D3): sec.6.2 Generate of roadmap template.pdf (excerpt from ai cofounder for pivot decisions) explains my approach to synthesize bayesian and system dynamics in the context of decision making. @tomfid and @jandraor 's input on below would be helpful
q1: how to design m_o,i (feedback receiver of policy use case and provider for software use case)
q2: pros and cons of integrated and modular management of dynamic and bayesian model tools? with the lens of #100

[hyunjimoon]

I reached out to Tom and Hazhir:

I'm reaching out to ask for help on literature review for barriers to adopting Bayesian reasoning. Based on our Bayes+SD journey (which I'm really enjoying, thank you!), I felt below are three root causes of low adoption, despite its theoretical optimality.

1. lack of product management training for modelers
2. compartmentalized inference and decision-making workflows
3. misconceptions about Bayesian methods among potential adopters
   To propose a more realistic solution (below), I'm eager to understand identified causes more. Could you please recommend relevant literature that comes to your mind?

Three integrations to adopt Bayesian reasoning

1. Continuous integration of new inference algorithms and observational data into individual modeler's reasoning processes. Treating reason as a product of each modeler, training modelers know-hows of continuously improving their product can help them adopt Bayesian reasoning.

2. Integrating upstream inference and downstream decision in the Bayesian ecosystem. Poor communication between algorithm/tool builder and model builder/policy designer/stakeholders increased system complexity by building what is not needed and not using what is built. By creating feedback loops for inference upstream and decision-making downstream, and connecting these loops, each ecosystem member can develop one's product (algorithms, tools, models, policies) that better address precise downstream needs.

3. Integrating distinct Bayesian reasoning techniques, such as probabilistic, relational, and social reasoning (per the probabilistic language of thought framework). This enables a more holistic, multi-faceted Bayesian approach to strategic planning, as demonstrated using a Tesla case study.

@tomfid answered and shared his diagrams which I like in that it gave me individual-based perspective (complementing idea or knowledge-based)

1. lack of product management training for modelers
   I think this maps to my "tech skills in ..." items. A key problem is that using data with typical SD models is hard, because they typically have broad scope, and therefore rely on data from a variety of sources with conflicting aggregation, widely varying quality, fuzzy interpretations, etc. This is not what normal data science is geared up to handle, which is massive monolithic datasets targeted at a single decision point in a system.

2. compartmentalized inference and decision-making workflows
   I think this relates to my "appetite" item. I think there may be several problems here. One is that a lot of Bayesian work focuses on measurement of nondecision features, like R0 in epidemiology. This is actually a useless number (though maybe informative to an expert) until you translate it into some decision like, how much should we spend on NPIs? Making that decision with full appreciation of the uncertainty that flows from priors and data is great, but relatively few end users can tell the difference between that and a best-guess answer or a data-free Monte Carlo approach, and obviously those are cheaper to get.

3. misconceptions about Bayesian methods among potential adopters
   I think there are two kinds of adopters here: 1) Decision makers consuming models of various sophistication 2) Modelers providing those
   Within SD, I think the modelers are easy to convince. We don't need to have a stupid SD frequentist vs. SD Bayesian debate because SD already has natural Bayesian inclinations and most models are going to be somewhat hybrid anyway (due to numerous non-estimated parameters).
   Within the consumers, I also don't think there's a reason to have the debate. The key point isn't really whether your methods are Bayesian, it's whether you recognize prior information, attempt to measure your uncertainty, and test decisions against those estimates. The hurdle is convincing people to consider the possibility that they might be wrong and to allocate some resources to constructing policies that are robust to that.

Hazhir liked Tom's operational graphs, and the distinct bottlenecks on both supply and demand side of getting Bayesian tools, data, and dynamic models together. He suggested, one may also note the distinction between decision makers and 'problems': a general issue may be the frequency of problems that may benefit from the full suite of tools on the demand side. He thinks for poorly defined problems with limited data it is not clear how much the full workflow adds value.

I agree with Hazhir, and this is where incrementally revealing workflow, along with its ability to identify modeler's state can be useful.

[hyunjimoon]

from the roadmap, i felt the customer pull might be the bottleneck so reached out to bayes in business supporters.

I'm reaching out to ask for advice on the subject line which I have developed as the preliminary plan below. Also I'd like to ask how I can persuade you to be a collaborator on this. My ultimate goal is to sell Bayesian computation technology (probabilistic program) to the entrepreneurship market in three years. To expedite this, I concluded the bottleneck breaking operations is to enhance the level of understanding of Bayes in the entrepreneurship market. Considering your expertise and passion for diffusing Bayesian statistics to management (our PDW with 12 yr history), I felt you'd be the best people to ask for help.Specific feedback I'd like to get from you is, two. If you feel zoom is better for this chat, just let me know. I'd be more than happy to send out zoom invites.

1. If you were me with the above ultimate goal and the constraint of ~ three hours a week this Fall, how would you sequence the two viable actions below? Three options: focus on writing paper, focus on journal club, do it parallelly.
2. If this plan sounds reasonable, what form of support/help may I ask to each of you?
3. writing a paper Bayesian Statistics in Entrepreneurship: Theory, Applications, and Opportunities (mock title). This would match solutions of Bayesian statistics in management (provided in our PDW slides and Jeff's recent paper attached (thanks Anup for introducing)) with needs of entrepreneurship (e.g. endogeneity, lack of data, heterogeneous prior) that can be much elegantly addressed with Bayesian statistics (identified in Bayesian conference slides attached) with your help. Thanks to the heavy lifting done by both sides, I don't think matching itself will take too much time when done together. We can aim to present this in the second Bayesian entrepreneurship conference, around next March.

2. start a journal club on this topic targeting Bayesian entrepreneurship conference attendees' phd students + our Bayesian stats in management PDW attendees interested in entrepreneurship. One output may be a syllabus that organizes a reading list relevant to Bayesian statistics for entrepreneurship that I can propose to organizers of the Bayesian entrepreneurship conference.

For both actions, I can frequently collect feedback from leaders of customers - Bayesian entrepreneurship conference organizers Scott Stern, Erin Scott, Arnaldo Camuffo.

and Jeff Dotson who just wrote MackeyDotson24_Bayesian Stats in Mangement.pdf was willing to chime in.

so I made below need and solution

Table 1: Bayesian Solutions and Corresponding Management Research Papers

Solution Classification	Paper Titles
1. Handling complex models and endogeneity	- Mackey et al. (2017): Corporate Diversification and the Value of Individual Firms: A Bayesian Approach - Arin et al. (2015): Revisiting the determinants of entrepreneurship: A Bayesian approach
2. Incorporating prior knowledge and dealing with small sample sizes	- Hansen et al. (2004): A Bayesian Operationalization of the Resource-Based View - Denrell et al. (2012): Inferring Superior Capabilities from Sustained Superior Performance: A Bayesian Analysis
3. Facilitating simulation studies and quantifying uncertainty	- Berchicci & King (2022): Building knowledge by mapping model uncertainty in six studies of social and financial performance - Anderson & Lenz (2001): Modeling the Impact of Organizational Change: A Bayesian Network Approach
4. Adaptability to different data types	- Casciaro & Lobo (2008): When Competence Is Irrelevant: The Role of Interpersonal Affect in Task-Related Ties - Cordery et al. (2015): The Operational Impact of Organizational Communities of Practice: A Bayesian Approach to Analyzing Organizational Change

Table 2: Entrepreneurship Research Needs and Corresponding Empirical Papers

Need Classification	Paper Titles
1. Priors formation	- Camuffo et al. (2023a): Theory-Driven Strategic Management Decisions - Felin & Zenger (2017): The Theory-Based View: Economic Actors as Theorists
2. Experimentation and Learning	- Camuffo et al. (2020): A Scientific Approach to Entrepreneurial Decision-Making: Evidence from a Randomized Control Trial - Koning et al. (2022): Experimentation and Startup Performance: Evidence from A/B Testing
3. Persuasion	- Gans (2023): Experimental Choice and Disruptive Technologies - Chavda et al. (2024): Theory-Driven Entrepreneurial Search
4. Pedagogy	- Camuffo et al. (2024): A Scientific Approach to Entrepreneurial Decision Making: Large Scale Replication and Extension - Zellweger & Zenger (2023): Entrepreneurs as Scientists: A Pragmatist Approach to Producing Value out of Uncertainty

with claude

[hyunjimoon]

#255