preparing stancon

[hyunjimoon]

FiddamanMoon23_BayesSD.pdf

[hyunjimoon]

summary of our last meeting

Angie Moon, Tom Fiddaman, and Jair discussed Bayesian methods in various fields, including epidemiological modeling. Jair shared his experience of defending his PhD early February and moving to England two weeks ago. In the first part of the conversation, the speakers discussed a project proposal for an AUC conference and an applications-oriented paper highlighting the challenges and common features of their work. In the second part, they discussed workflows in system dynamics modeling, emphasizing the importance of visualizing the logic of the model to avoid errors and improve speed, and leveraging visualization tools and fostering interaction between stakeholders to improve workflows.

Transcript

https://otter.ai/u/Vtto_SL0AnYjCYyzRe3lInrQDK4?view=transcript

Action Items
[ ] Continue discussion on GitHub with abstract and outline.
[ ] Send poster on modeling workflows to Jair.
[ ] Summarize relevant examples from thesis for talk proposal.
[ ] Schedule follow up meeting to discuss SD model.
Outline
Bayesian methods for data analysis and software development.
Angie Moon expresses concern about unexpected obituaries and Tom Fiddaman shares an unexpected snowstorm experience.
Angie Moon and Tom Fiddaman discuss potential conference topic for Bayesian modeling and methodological development.
Angie Moon and Tom Fiddaman discuss applications of Bayesian statistics in epidemiological modeling, with a focus on identifiability and practical challenges.
Improving workflows for epidemiology modeling using system dynamics software.
Jair and Tom discuss the need for faster modeling workflows in system dynamics and Bayesian analysis.
Angie Moon suggests using visualization tools to validate logic and reduce errors in model building.
Tom Fiddaman agrees, emphasizing the importance of interactivity and exploratory capabilities for model building.
Angie Moon and Tom Fiddaman discuss how to create an interactive epidemiology model that users can resonate with.
Using diagrams and code for modeling and selling ideas in different communities.
Jair and Tom discuss challenges in selling their product to different communities, with Jair highlighting the need to address both diagramming and coding needs.
Jair and Tom discuss the limitations of using code to model complex systems, with Jair highlighting the benefits of using diagrams for reproducibility.
Angie Moon and Tom Fiddaman discuss model size and complexity for epidemiology talks.
Angie Moon requests Jair to summarize their thesis for the proposed talk, connecting it to concrete examples from Jair's work.
Angie and Jair discuss meeting details, Tom provides advice on SD model.

This submission form asks only abstract so might be better submit after next week's meeting. @jandraor @tomfid

[hyunjimoon]

Tom, Jair, Angie

Tom Fiddaman and Angie Moon discussed improving SD model parameters with GPT and HMC, emphasizing the importance of data framework and infrastructure. They also discussed Bayesian inference and modeling in AI, including Jair's work and Nader's paper. The speakers also exchanged comments on optimization methods for Bayesian inference and prior modeling. Later, Angie and Tom discussed reality checks in modeling work, highlighting the need for replicable tests and evaluating the effectiveness of interventions through reality checks in epidemiology.

Transcript

https://otter.ai/u/APNuLVul0oAWu-l70_Vhh-7e7Fk?view=transcript

Action Items
[ ] Incorporate Tom's diagram inputs and get final check from Jair before submission [1:01:48]
[ ] Develop a representative diagram capturing the workflow and connections to inference/decision-making by Thursday [1:01:12]
[ ] Consider attending the conference in September if the submission is accepted [0:17:24]
[ ] Provide comments on the slides to iteratively improve the material [0:22:22]
Outline
Using GPT for SD model parameters with focus on data management and uncertainty.
Tom Fiddaman and Angie Moon discuss a slide deck with Jerry, with Tom questioning the outline's structure.
Angie Moon aims to provide a Python translation of SD model for GPT API connection.
Angie Moon and Tom Fiddaman discuss using PISD and GPT for SD modeling, with Jared highlighting the need for more data framework.
Bayesian inference and decision-making in epidemiology.
Angie Moon and Tom Fiddaman discuss the importance of interactive testing and deep inspection in code, with a focus on presenting their ideas in 15 minutes.
Angie Moon discusses Jairus's paper on Bayesian inference and decision support, focusing on inference methods such as HMC and anchoring.
Angie Moon discusses using Pathfinder for inference and approximation in machine learning.
Angie Moon proposes a unified framework for policy optimization and calibration in pandemic response.
Submitting a presentation to a conference, potential challenges and ideas for improvement.
Tom and Angie discuss submission options for an upcoming conference, including a 15-minute presentation and a tutorial.
Angie Moon suggests inviting simulation-based calibration team to collaborate on tutorials for SPC.
The team's submission for an SBC was not accepted due to a combination of factors, including poor marketing and a lack of readiness for the proposal.
Angie Moon and Tom Fiddaman discuss the challenges of communicating the value of a product to an unreceptive audience.
Developing a tool for policy designers and stakeholders, with a focus on interactions and feedback loops.
Angie Moon and Tom Fiddaman discuss a document for a presentation, including diagrams and slides.
Fiddaman and Moon discuss creating a hybrid chart for decision makers, prioritizing simplicity and ease of assimilation.
Tool and model builders need more insights on policy design and stakeholder outcomes to improve automation.
Angie Moon questions the relevance of the reality check feature for policy designers and stakeholders.
Using reality checks in modeling to ensure accuracy and avoid potential pitfalls.
Developing a way to record and automate repetitive tests to improve model accuracy.
Angie Moon suggests recording test data to improve understanding of model performance.
Tom Fiddaman explains the challenges of testing the effectiveness of vaccines in a simulated epidemic scenario.
Tom Fiddaman and Angie Moon discuss the potential risks of intervening in a pandemic, including the possibility of adherence fatigue leading to a worse outcome later on.
Using reality checks in Bayesian inference.
Tom Fiddaman explains how to use Reality Check with additional harvest fraction.
Tom Fiddaman explains the reality check for prevalence, expecting a 10% decrease.
Tom Fiddaman and Angie Moon discuss using soft constraints in Bayesian inference to model prior beliefs about a task's prevalence.
Angie Moon seeks input on diagram to connect workflow and Jair's paper, aiming to submit after quick check.

[tomfid]

https://mc-stan.org/events/stancon2024/

[hyunjimoon]

evidence of submitting below: https://capture.dropbox.com/rTeusGLLLEYWQqjs
Integrating Bayesian Inference and System Dynamics with Case Studies in Epidemiology.pdf

[hyunjimoon]

@tomfid Am reading the new Dotson24_Bayesian Stats in Mangement.pdf recommended from Bayes stats PDW at AOM conference and feel stats, cs (data management) education, especially on "slope endogeneity" are important for modelers who should serve the role of connecting software and policy feedback loops.

[hyunjimoon]

nate recommend https://pubmed.ncbi.nlm.nih.gov/38397684/

We further note aspects of model use in practice as an effective reporting tool in a manner that is parameterized by jurisdiction, including the support of a scripting pipeline that permits a fully automated reporting pipeline other than security-restricted new data retrieval, including automated model deployment, data validity checks, and automatic post-scenario scripting and reporting. As demonstrated by this multi-year deployment of the Bayesian machine learning algorithm of particle filtering to provide industrial-strength reporting to inform public health decision-making across Canada, such methods offer strong support for evidence-based public health decision-making informed by ever-current articulated transmission models whose probabilistic state and parameter estimates are continually regrounded by diverse data streams.

[hyunjimoon]

dumping erased but good figures

[hyunjimoon]

building on my theory on prob.prog for the next five years in #244 (reply in thread) i analyzed integration strategy for probabilistic programming based on clockspeed theory. below table shows "integral to modular to integral" movement might repeat in the Bayesian software industry based on evolving needs in bayesian software across generation cld:

Aspect	Past (Integral)	Present (Modular)	Future (Integral?)
Industry Structure	Vertical (Custom implementations)	Horizontal (Specialized tools and platforms)	Vertical (Integrated AI-Bayesian systems?)
Product Architecture	Monolithic (All-in-one custom solutions)	Modular (Mix-and-match components)	Integral (Unified probabilistic programming platforms?)
Key Players	Individual researchers, BUGS	Stan, PyMC, INLA, specialized tool developers	AI-enhanced Bayesian systems (e.g., W2W implementations)

[hyunjimoon]

Reflections

Amazing collaboration with @jandraor and @tomfid ! Thanks to you I had a meaningful conversation on connecting dynamics tool with stan tool, based on the talk (+ introduced our research to Stan community): Link for the final slide.

If Jair had given the talk it would have been much clearer talk for the audience. Change of presenter (due to Jair's sudden schedule), trigger the pivot of the talk's goal (in the very last minute). Looking back, deep inside my mind was the desire to use this presentation opportunity to push forward the technology I was developing (which downgraded the importance of customer (audience)'s comfort by sharing information in clean, simple, conventional format). Synthesizing the feedback from ten people, below might have been the better introduction:

i'm developer of simulation-based calibration library, and the main result of the paper was test quantities shapes sensitivity, but based on my interactions of model builders in business school, i found test quantities very hard to set. i found interactive simulation dashboard to be helpful tool in helping stakeholders realize what test quantities they are interested in. tools that elicits prior from stakeholders are more commonly developed in industry but most of them are in house. what i'm suggesting (where every user can make bespoke dynamic simulation-based tool which is most widely known for EnRoads for climate action may have great relevance with joint elicitation of feasibility and desirability prior)

Seven learnings:

🗣️ 1. honest communication of my situation relevant to the talk

i think audience might have understood if i said speaker changed, my major is business not infectious disease and i have less knowledge for the domain (😂 people now perceive me as infectious disease modeler)

🥶 2. cold call, culture, recognizing hierarchy to some extent

be careful about cold calling during the talk to the audience. try communicating in advance (I learned this from a moderator from bayesian entrepreneurship (BE) conference (#234) but knowing two difference in the situation (a not asking a very clear question, b not gained enough authority to cold call (in Bayesian entrepreneurship conference ##234, moderator who cold called was a renowned professor) might have changed my action.

⭐️ 3. making slide simple and emphasizing my contribution

spending more time walking through the cool translator (stanify, readsdr) we've developed might have been better

sean shared he was expecting to see how I made below - but the problem was we haven't translated this model to stan yet using our

🧠 4. Different background, different absorptive capacity, different satisfaction

Industry people who had first hand experience in the communication difficulty & people who were aware of system thinking gave more positive feedback (compared to pure statistician who were expecting to see concrete models, with much less meta-approach). However, for statisticians, my talk was too meta, businessy, abstract. Focusing on selling our translation tool might have been great alternative approach.

🌅 5. Stan needs a vision and is searching for scalability

• not sure how business model could gain legitimacy and support in nonprofit organization but I feel operational feasibility should be considered for nonprofits no matter how much fund they have.
• in front of Stan are two customer segments: academic's bespoke model (as is of now) and industry's automated, scalable model.
• steve verified people have great needs for bespoke simulation tool - industries build this, but it'd be great for modelers to have this (preferably scalable and automatable).

🐍 6. how i should flexibility explain definition

I couldn't find the best word to explain when Bob asked what i mean by "dynamics". It's different from just time series modeling and now come to think of it (especially after taking bayesian optimization tutorial), it may be the sequential decision making component that I felt lacking in stan e.g. Decision analysis in stan manual have rooms for improvement and I'd love to collaborate with Bob (or whoever is interested) to enhance this to make decision making more realistic e.g. first adding decision making structure (e.g. hunter and government stakeholder analysis from 🦌 chronic wasting example) then model evolving structure (Q to @tomfid - could hunter and government's relation change?) may be another keyword. i wonder how joint inference of desire and belief e.g. Online Bayesian Goal Inference for Boundedly-Rational Planning Agents could be modeled in stan. if this isn't easy, would making the language more flexible be one solution? i feel flexibility updating the goal of model is crucial for bayesian workflow, especially in the domain where model building clockspeed (upstream) is much slower than that of model consumption (downstream).

🙉 7. skeptical view of statisticians to business scholars/theories

There were some statistician who deeply trust in the uselessness of management/b-school scholar's work. the thing is, i've met b-school scholars saying the same things to the scholars in other domain. I'll go back home and buy this poster I think everybody should like everybody - Andy Warhol

---

Thanks for all who gave me honest feedback to help me learn: Mitzi Morris, Paul Burkner (should have made xmile - r/python- stan photo much bigger and stick with simple example (given tool is the key contribution)), Charles Margossian (people would have had better understanding if I showed how system dynamics people could benefit from Stan, if the goal was bridging - which makes sense since I'm bridging from vensim to stan, not the other way around), Liza (why and how weren't clear), Steve (industries make the dashboard inhouse, but it'd be useful to make it more accessible and automatable for individual modelers), Judith (background in infectious disease, she found talk to be interesting but mentioned audience might have been surprised by its format, think innovative format talk should be more welcomed), Sean (need clearer setup, was expecting me to explain how i made the interactive simulation tool using stan), Juan (he thought stan needs vision while listening to my talk), James (infectious disease modeler who knew system thinking), Etie (stan organizer who showed interest in being part if I organize stan connect with the topic of using interactive tool to elicit goal / communicate with stakeholders)

Actions

1. Invited Steve to our weekly meeting on 24th. Hope to seek potential collaboration path between Vensim and Stan with @tomfid's support (and the market formed from this connection can help me if I end up developing conversational modeling assistant) Steve gave feedback on trying to reach out to Ben and find master student (who need thesis topic) to work on this as Ben is interested in bayes workflow.

2. depending on 1, with Steve's help writing design doc and proposing might be a viability

[hyunjimoon]

Reflection on Stan Community Presentation

Introduction

Thank you for attending my presentation at the Stan Community event. I appreciate the opportunity to share our work on connecting dynamics tools with Stan. This document summarizes key points from the presentation, lessons learned, and proposed next steps.

Key Points from the Presentation

1. We've developed translation tools (stanify, readsdr) to bridge system dynamics and Stan.
2. Our research focuses on using interactive simulation dashboards to help stakeholders identify relevant test quantities for simulation-based calibration.
3. We aim to create bespoke dynamic simulation-based tools that can be widely accessible, similar to climate action models like EnRoads.

Lessons Learned

1. Clearer communication of the presenter's background and expertise is crucial.
2. Simplifying slides and emphasizing our main contributions would have improved understanding.
3. Balancing technical depth with accessibility for a diverse audience is challenging but necessary.
4. There's a need for scalable, automated modeling solutions in both academic and industry settings.
5. Flexibility in explaining concepts like "dynamics" is important, especially when bridging different fields.

Proposed Next Steps

1. Refine our presentation of the translation tools (stanify, readsdr) to make them more accessible to the Stan community.
2. Explore potential collaborations with Vensim and Stan to further develop these bridging technologies.
3. Investigate the possibility of enhancing Stan's capabilities in sequential decision-making and evolving model structures.
4. Consider writing a design document for integrating interactive simulation tools with Stan's Bayesian workflow.

Acknowledgments

I'd like to thank all those who provided valuable feedback, including Mitzi, Paul, Charles, Steve, Sean, Juan, James, Etie, and others. Your insights are crucial for improving our work and its presentation.

Contact

For further discussion or collaboration opportunities, please don't hesitate to reach out. I'm particularly interested in exploring how we can make dynamic simulation tools more accessible and integrated with Bayesian workflows.

[hyunjimoon]

@tomfid agenda1 for today - I'd like to analyze three specific people who I'm keeping in touch with as they liked our presentation / value proposition.organizing stanconnect with some synthesis with #144 sounds fun (ah if i could find a great overlap with this and my research it'd be so great)

---

using Designing a Survey for Stan Conference Attendees cld

1. Summary of our research:

• Horizontal integration of modeling tools (e.g., flu pandemic model)
• Vertical integration of modeling value chain members (e.g., chronic wasting disease policy)
• Bridging actions like stanify and readsdr to connect dynamic modeling tools with Stan
• Emphasis on purposeful modeling that connects diverse stakeholders and addresses challenges in model communication and transparency

2. Analysis of the three profiles:

a) Dr. Mphatso Phiri:

• Medical doctor and epidemiologist from Malawi
• Focuses on infectious disease research, particularly TB
• Uses mathematical and statistical modeling methods
• Interested in methodological issues related to designing and evaluating complex public health interventions
• Works at the interface of research, modeling, and policy

b) Hanna Fiegenbaum:

• Sustainability expert
• Works on circular construction projects and digital building passports
• Experience with life cycle assessments and ecosystem services
• Focuses on integrated solutions for people and nature's well-being
• Interested in connecting knowledge across different domains and sectors

c) Juan Ignacio de Oyarbide:

• Qualified Actuary specializing in General Insurance
• Interested in bridging risk theory and actuarial practice through innovative software solutions
• Focuses on ratemaking, consulting, and product lifecycle management

3. Survey design:

Subject: Follow-up Survey: Integrating Bayesian Inference and System Dynamics
hanna.fiegenbaum@gmail.com, mdphiri@mlw.mw,
Dear [Name],

Thank you for your interest in my presentation on integrating Bayesian inference and system dynamics at StanCon 2024. To better understand your needs and organize a Stan connect event, I'd appreciate your input on the following questions:

1. Which aspect of the integration between Bayesian inference and system dynamics interests you most?
   a) Horizontal integration of modeling tools
   b) Vertical integration of modeling value chain members
   c) Bridging actions (e.g., stanify, readsdr)
   d) Other (please specify)

2. what compartment do you associate yourself with?
   a) tool developers
   b) model builders
   c) policy designers
   d) stakeholders

3. How do you envision applying this integrated approach in your work?
   [Open-ended response]

4. What challenges do you face in your current modeling processes that this integrated approach might address?
   [Open-ended response]

5. Which of the following topics would you be most interested in exploring further? (Select all that apply)
   a) Translating system dynamics models to Stan
   b) Communicating model results to diverse stakeholders
   c) Incorporating expert knowledge into Bayesian models
   d) Applying integrated models to policy decisions
   e) Other (please specify)

6. What format would you prefer for a Stan connect event? (Select all that apply)
   a) Virtual workshop
   b) In-person conference
   c) Online discussion forum
   d) Collaborative project
   e) Other (please specify)

7. How familiar are you with Stan?
   a) Novice (little to no experience)
   b) Intermediate (some experience, but not proficient)
   c) Advanced (regularly use Stan in my work)

8. What specific tools or resources would you find most helpful in adopting this integrated approach?
   [Open-ended response]

9. Would you be interested in presenting your own work or case study at a future Stan connect event?
   a) Yes
   b) No
   c) Maybe, need more information

10. What is your preferred time commitment for a Stan connect event?
    a) 2-3 hours
    b) Half-day
    c) Full-day
    d) Multiple days
    e) Other (please specify)

11. Any additional comments or suggestions for the Stan connect event?
    [Open-ended response]

Thank you for your time and input. Your responses will help us tailor the Stan connect event to best meet the needs of our community.

Best regards,
[Your Name]

This survey is designed to gather specific information about the participants' interests, needs, and preferences related to the integration of Bayesian inference and system dynamics. It includes a mix of multiple-choice and open-ended questions to capture both quantitative and qualitative data, which will help you organize a more targeted and valuable Stan connect event.

[tomfid]

I think we didn't get to this but interested to hear more.

…

On October 16, 2024 7:04:47 AM MDT, "Angie.H Moon" ***@***.***> wrote: @tomfid agenda1 for today - I'd like to analyze three specific people who I'm keeping in touch with as they liked our presentation / value proposition.organizing stanconnect with some synthesis with #144 sounds fun (ah if i could find a great overlap with this and my research it'd be so great) --- using [Designing a Survey for Stan Conference Attendees cld](https://claude.ai/chat/43063888-c6e3-415b-93e5-d13aca39f200) 1. Summary of our research: - Horizontal integration of modeling tools (e.g., flu pandemic model) - Vertical integration of modeling value chain members (e.g., chronic wasting disease policy) - Bridging actions like stanify and readsdr to connect dynamic modeling tools with Stan - Emphasis on purposeful modeling that connects diverse stakeholders and addresses challenges in model communication and transparency 2. Analysis of the three profiles: a) Dr. Mphatso Phiri: - Medical doctor and epidemiologist from Malawi - Focuses on infectious disease research, particularly TB - Uses mathematical and statistical modeling methods - Interested in methodological issues related to designing and evaluating complex public health interventions - Works at the interface of research, modeling, and policy b) Hanna Fiegenbaum: - Sustainability expert - Works on circular construction projects and digital building passports - Experience with life cycle assessments and ecosystem services - Focuses on integrated solutions for people and nature's well-being - Interested in connecting knowledge across different domains and sectors c) Juan Ignacio de Oyarbide: - Qualified Actuary specializing in General Insurance - Interested in bridging risk theory and actuarial practice through innovative software solutions - Focuses on ratemaking, consulting, and product lifecycle management 3. Survey design: Subject: Follow-up Survey: Integrating Bayesian Inference and System Dynamics ***@***.***, ***@***.***, Dear [Name], Thank you for your interest in my presentation on integrating Bayesian inference and system dynamics at StanCon 2024. To better understand your needs and organize a Stan connect event, I'd appreciate your input on the following questions: 1. Which aspect of the integration between Bayesian inference and system dynamics interests you most? a) Horizontal integration of modeling tools b) Vertical integration of modeling value chain members c) Bridging actions (e.g., stanify, readsdr) d) Other (please specify) 2. How do you envision applying this integrated approach in your work? [Open-ended response] 3. What challenges do you face in your current modeling processes that this integrated approach might address? [Open-ended response] 4. Which of the following topics would you be most interested in exploring further? (Select all that apply) a) Translating system dynamics models to Stan b) Communicating model results to diverse stakeholders c) Incorporating expert knowledge into Bayesian models d) Applying integrated models to policy decisions e) Other (please specify) 5. What format would you prefer for a Stan connect event? (Select all that apply) a) Virtual workshop b) In-person conference c) Online discussion forum d) Collaborative project e) Other (please specify) 6. How familiar are you with Stan? a) Novice (little to no experience) b) Intermediate (some experience, but not proficient) c) Advanced (regularly use Stan in my work) 7. What specific tools or resources would you find most helpful in adopting this integrated approach? [Open-ended response] 8. Would you be interested in presenting your own work or case study at a future Stan connect event? a) Yes b) No c) Maybe, need more information 9. What is your preferred time commitment for a Stan connect event? a) 2-3 hours b) Half-day c) Full-day d) Multiple days e) Other (please specify) 10. Any additional comments or suggestions for the Stan connect event? [Open-ended response] Thank you for your time and input. Your responses will help us tailor the Stan connect event to best meet the needs of our community. Best regards, [Your Name] This survey is designed to gather specific information about the participants' interests, needs, and preferences related to the integration of Bayesian inference and system dynamics. It includes a mix of multiple-choice and open-ended questions to capture both quantitative and qualitative data, which will help you organize a more targeted and valuable Stan connect event. -- Reply to this email directly or view it on GitHub: #230 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>