Simulation-Based Calibration to increase Decision Clockspeed for Entrepreneurs

[hyunjimoon]

This thread includes efforts to apply theory of Simulation-based calibration and its test quantity (accepted in Bayesian analysis) to entrepreneurship domain.

To start with, I collaborated upstream with claudeai to digest #156 and produce input for downstream gpt. I need @tomfid and @chasfine's input who are described as tool builders and policy designers in the gpt's final output.

claudeai

gpt

https://chat.openai.com/share/be0a5970-80a5-4929-a7ee-cedab6ca453a

description of below was #fig1 from the prompt

[hyunjimoon]

Simulation-based calibration (SBC) supported by computation power provides the tool to "look before leap". This is the only situation where Savage allowed using his theory "Savage says it would be preposterous to use his theory except in situations in which the decision-maker can look before she leaps". In that sense, SBC tool "massage her contingent snap beliefs into consistency".

Savage’s famous encounter with Maurice Allais in Paris illustrates the point. When it was pointed out that his answers to what is now called the Allais paradox were inconsistent, he revised them until they became consistent. Luce and Raiffa (1957, p. 302) summarize Savage’s views as follows: Once confronted with inconsistencies, one should, so the argument goes, modify one’s initial decisions so as to be consistent. Let us assume that this jockeying—making snap judgments, checking up on their consistency, modifying them, again checking on consistency etc—leads ultimately to a bona fide, prior distribution.
It is to this unformalized massaging process that Savage (1954, p. 16) is referring when he says that it would be both ‘‘preposterous’’ and ‘‘utterly ridiculous’’ to use his theory except in situations in which the decision-maker can look before she leaps. For only in a sufficiently small world can Alice examine every bridge that she might conceivably have to cross in advance of knowing what route she will find herself following in the future. I have borrowed this geographical metaphor from Savage, but it fails to capture the enormity of the cognitive task Alice would usually face in seeking to examine the implications of all possible future information she might receive when seeking to massage her contingent snap beliefs into consistency.

from this paper.

[chasfine]

Hmmm. Maybe the entrepreneur's challenge is "too big" for Bayesian Decision Theory. Perhaps the entrepreneur's challenge just cannot be a "small world" in its most general sense. Sub-problems that the entrepreneur faces might be "small enough", but perhaps the overall entrepreneurial challenge is "too big"?

[hyunjimoon]

"too big" for Bayesian Decision Theory

Theory can't cut it, but algorithm or program may! Boland et al proposed dynamic discretization algorithm that finds optimal solution for continuous time model without constructing a full model (summary below). Likewise, mathematical induction algorithmically covers "too big" (up to infinity). The following paper by business school professors) advocates "program" for pivoting.
Chen22_Programs of Experimentation and Pivoting for (Overconfident) Entrepreneurs.pdf

Sub-problems that the entrepreneur faces might be "small enough", but perhaps the overall entrepreneurial challenge is "too big"?

It's our role to segment "too big" until every segmented pieces hits "small enough" threshold. I was most impressed by your comment "the idea is to segment lifecycle and stitch them together to understand the whole" and your syllabus states "15.S10 builds a framework for 'nailing' ($0 to $1) and 'scaling' ($1 to $10M) the Operations Management capabilities required for success. We'll focus on the design, planning, control, and improvement of processes and systems to establish sound operations, to support the startup phase of the business, then drive scale.". profit of a firm is what you've chosen as conditional variable. Hence NSS is closer to condition-based not time based (preventive maintenance) approach.

The key is whether we can develop diagnostic tool that answers yes/no to "small enough?" question. e.g. how do you know enough is enough? (NSS's 3stage, 10 scaling tools)

[tomfid]

My conclusion, if the entrepreneur's challenge is too big for some kind of Bayesian dynamic model, would be that nothing else will work either - leaving only intuitive guessing, smart execution, and good luck in the face of natural selection.

I'm not quite ready to give up of the idea that we could formalize some of the procedures for identifying the state of the system and sensible strategic actions given that.

[tomfid]

My impression of the challenge is that the state space is big, and not all the concepts in it are easily observable, and maybe also they don't always have an agreed-upon operational definition. One possible response might be to try to reduce the dimensionality by trying to control a few things so you can explore/model simpler sub-problems.

[hyunjimoon]

Extraterrestrial-BAYES? (but not really)

Is luck-operation outside Bayes planet? Triggered by Charlie's question:

If co-opted nonaption is highly serendipitous, how does one explore this infinite-sized space?

This aligns with his previous comment:

Maybe the entrepreneur's challenge is "too big" for Bayesian Decision Theory.

And Tom's sarcasm:

Good luck in the face of natural selection.

I came across an intriguing concept called 🥅Two road chance from hyde08_🥅(two road_trickster makes the world).pdf. It describes the transformative potential of serendipity when independent causal paths intersect unpredictably. Unlike "one-road chance," which reflects predictable outcomes, absolute chance creates opportunities for newness and innovation. Exploring multiple paths increases the likelihood of surpassing critical thresholds, as seen in natural selection’s filtering of suboptimal paths, allowing unexpected intersections to drive creativity and change.

Recognizing how irresponsible it may sound, designing a wider net for luck 🎣fishing might be the best approach in some situations. However, even "designing the net" requires Bayes thinking—e.g., observing where luck is most likely to occur.

Returning to Bayes, entrepreneurial decision-making thrives by balancing structure and serendipity. Simulation-Based Calibration (SBC) acts as a QUALITY EVALUATION FRAMEWORK, systematically assessing the fit of inference algorithms (entrepreneurs’ resources), models of observation generation (business feedback models), and test quantities (evaluation metrics). SBC also functions as a LUCK-RECOGNIZER, leveraging exchangeability (rank uniformity) to detect discrepancies from homogeneity (hence is SBC frequentist.)). However, this "SBC net" is only as effective as the test statistics entrepreneurs choose to provide. The finer the structure, the better it captures serendipity in service of innovation.

[hyunjimoon]

Had a chat with Evy Chen who is very passionate about implementing simulation-based tools in retail business. Main argument is entrepreneurs should be able to test thier choices with simultaion tool that shows how their choice propagates through hierarchical decision structure which can cut down waste in capital, market, talent. To commercialize this idea, she needed a structured approach which is where she needs angie's input. Angie would benefit from Evy's 15yr insight, milestone and motivation to write her sloan fellow thesis, data. But most of all shared vision on the need for increased "access" and "simulation" as a rescue.

[hyunjimoon]

Angie's prompt to Evy

objective: document persuading Charlie (MIT professor passionate in operations for entrepreneurs) of Angie (MIT phd passionate in applying simulated based calibration to business theory) and Evy's collaboration

time constraint: by Sunday night
content constraint:

• length: 5pg
• input: Angie's summary of our discussion above
• format:

imagine EVY is entrepreneur in retail business and describe AGENT (1.1-4), ACTION2STATE(2.1-4), STATE2ACTION(3.1-2)

1. AGENT

1.1 five hierarchy of agents after EVY:
1.2 output (product design):
1.3 outcomes (product performance):
1.4 intermediate outcomes (how do structure capital to fund the product):

2. ACTION2STATE (blue simulation studies)

2.1 describe evy's ACTION in BIT_SUPPLY
2.1.1 design
2.1.2 develop
2.1.3 launch

2.2 describe evy's ACTION in BIT_SUPPLY
2.2.1 produce
2.2.2 distribute
2.2.3 sell

2.3 describe evy's ACTION in DEMAND_SIDE
2.2.1 design, develop

2.4 ACTION2TIME: map 10 ACTION from 2.1,2 with TIME

• 2024s (nail_stage):
• 2024f (scale_stage):
• 2025s (scale_stage):
• 2025f (scale_stage):
• 2026s (scale_stage):
• 2026f (sail_stage):

2.5 ACTION2TIME2STATE: map 2.3 ACTION to STATE 1.2-4 via TIME

• 2024s:
• 2024f:
• 2025s:
• 2025f:
• 2026s:
• 2026f:

3. STATE2(RE)ACTION (red data analysis)

3.1 STATE2TIME
assuming Evy can get any data from any five agents (don't be constained with existing data) what data should Evy gather? describe its quantity.
2024s:
2024f:
2025s:
2025f:
2026s:
2026f:

3.2 STATE2TIME2ACTION
how would the agent react to data flowing in 3.1 at each time?
2024s:
2024f:
2025s:
2025f:
2026s:
2026f:

3.3 STATE2TIME2ACTION counterfactual
what are data that would change Evy's ACTION in 3.2? (contrarian data)
2024s:
2024f:
2025s:
2025f:
2026s:
2026f:

[hyunjimoon]

red literature review

: computational representation of entrepreneurial, strategic, interactive agents

general idea that Bayesian nonparametric models (BNP) are more flexible than fixed-dimensional models and can adapt their internal structure to handle unlikely events or anomalous observations.

An example of BNP models handing rare or unlikely events can be seen
in Figure 4 of the Temporally-Reweighted CRP Mixture (AISTATS 2018;
http://proceedings.mlr.press/v84/saad18a/saad18a.pdf). Rather than
specify a single "fixed" time series model for all the GDP time
series, we use BNP in two ways (1) to non-parametrically partition the
collection of GDP time series and (2) to learn a separate
multivariate BNP model within each cluster (see Figure 3 tutorial).
The method finds that so-called "industrialized" nations have
different trajectories than say former Soviet nations or "rapidly
developing" economies; whereas countries like North Korea are
outliers.

An example of adapting to novelty adaption can be found on Pages 37-38 of my
doctoral dissertation, where online learning in a temporal BNP model
adapt to the changing structure in airline passenger data due to the
demand drop during COVID-19 (observe in particular how the ChangePoint
"CP" probabilities in Page 37 are evolving over time):
https://www.cs.cmu.edu/~fsaad/assets/Saad-fsaad-PhD-EECS-2022-thesis.pdf

Some other references of the general motivation and topics in BNP include:

1. High-level overview of BNP models---specifically for Dirichlet
   process, infinite mixture models, and random phylogenetic trees---can
   be found in Yee Why-Teh's 2011 summer school slides:

• https://www.stats.ox.ac.uk/~teh/teaching/npbayes/mlss2011F.pdf

2. More technical intro specifically for the DP (see references therein):

• https://fulbright.uark.edu/departments/math/_resources/pdf/event-related/sls/sls2020-lecture-prunster-1.pdf

3. Textbook references include:

• General: https://link.springer.com/book/10.1007/978-3-319-18968-0
• Biostatistics: https://link.springer.com/book/10.1007/978-3-319-19518-6

orange literature review

: entrepreneur's operations prior

green literature review

: entrepreneur's strategy prior

blue literature review

: social prior engineering

[evy-chen]

1. AGENT

hierarchies of agent groups
1.1 Brand
Decision Variables: Flavor, Shelf stability, SRP, Product segmentation, form factor, brand design, targeting audience, consumer trend, distribution method, dietary restrictions, store type, margin, COGS, volume, capital, and ingredients.

1.2 Manufacturer
Decision Variables: Volume, pipeline, form factor, fill size, kill steps, line time availability, packaging material availability, production schedule, seasonality, kosher/organic, and ingredients.

1.3 Distributor
Decision Variables: Product differentiation, product category, trend, product portfolio, retailer interest, volume, launch market, historical sales performance, room on shelf, trade spend, and margin.

1.4 Retailer
Decision Variables: Product differentiation, product category, trend, product portfolio, retailer interest, volume, launch market, historical sales performance, room on shelf, trade spend, ad spend, marketing budget, and broker involvement.

2. ACTION2STATE (Blue simulation studies).

2.1 Evy's ACTION in BIT_SUPPLY
2.1.1 Product design: Evolving prototypes.
2.1.2 R&D: Lab testing and modifications.
2.1.3 Manufacturer search: Finding the right production partner.

2.2 Evy's ACTION in BIT_SUPPLY
2.2.1 R&D: Refinement based on feedback.
2.2.2 Prototype: Final iteration.
2.2.3 Go-to-market strategy: Channel and audience identification.
2.2.4 Production: Mass manufacturing.
2.2.5 Pitch to buyers: Securing shelf space.

2.3 Evy's ACTION in DEMAND_SIDE
2.3.1 Design product and develop market plan: Adapting to market trends and preferences.

2.4 ACTION2TIME
Mapping actions across timeframes:

2024s (nail_stage): Finalize product design.
2024f (scale_stage): Start mass manufacturing.
2025s (scale_stage): Launch go-to-market strategy.
2025f (scale_stage): Secure major retailer partnerships.
2026s (scale_stage): Expand distribution channels.
2026f (sail_stage): Establish the brand as a market leader.

2.5 ACTION2TIME2STATE
Mapping actions to states through time: (Outlining how each action impacts the various agents over the timeline.)

3. STATE2(RE)ACTION (Red data analysis)

3.1 STATE2TIME
Data Evy should gather:
2024s: Historical purchasing data; consumer behavioral segmentation data.
2024f: Digital chase data; browsing history; smartphone GPS data; social media trends.
2025s: Concentrated niche market data.
2025f: Micro-marketing, localized demographic diversity data.
2026s: Micro-marketing, localized psychographic diversity data.
2026f: TBD based on the preceding data insights.

3.2 STATE2TIME2ACTION
Potential agent reactions to incoming data:
(Outlining how each agent might adjust their strategies or decisions based on the incoming data.)

3.3 STATE2TIME2ACTION Counterfactual
Data that could change Evy's actions:
(Identifying data trends that deviate from expectations and how they might steer Evy's decisions differently.)

[hyunjimoon]

meeting with - blue (wed) - red (mon)

[hyunjimoon]

Just had a talk to 26 hyundai mobis and the audience loved the diagram! Wish to get your input below to plan resource allocation + buy/make decision

1. difference between sampled customer and demand
2. how to best sample considering 1
3. what order should the green, red, purple be approached? This sequence is likely a function of unfair advantages and environments.

Wish to nail definition and three bias

[hyunjimoon]

1. prior (separate)

angie

evy

2. sync prior (together)

what to achieve and how

strategic parameters (A, D)

operational parameters (B,C)

3. generate data (Y_t) via simulation based on synced prior at each time t (together)

4. posterior action (separate)

• estimate what each individual can do to reproduce Y_t
• decompose Y_t into four orthogonal improve-ble biases

angie

evy