Stan now supports log_prob, grad_log_prob, hessian, un/constrain pars

[hyunjimoon]

stan-dev/cmdstanr#701 and using cmdstanX wrappers, it will soon be exposed to cmdstanpy as well. This is very useful for variational bayes approximators (optimization alogrithm). #96

For stan math library in C++ and its autodiff, I summarized the latter part of Bob Carpenter's https://statmodeling.stat.columbia.edu/wp-content/uploads/2019/01/under-stan-hood-jan-19.pdf with some comments.

Under Stan’s Hood: HMC+NUTS+reverse mode autodiff implementation

• p.32 ode solver + reverse mode auto-diff exists. Because output (payoff, log posterior scalar) dimension (1) is lower than input (number of parameters), reverse mode autodiff is faster.
• p.36-38 second, third-order curvature calculation allows efficient move. reparameterization effect overlaps with this and is easier, which I think we should prioritize
• p.46, 49 speeds up comes from vectorization, dropping constants
• p.47 Stan autodiff is the fastest and uses least memory among C++ opensource. Hazhir was interested in benchmark plots by dimension; double system seems faster, but perhaps hard to install...?

Hard Models, Big Data: approximators with potential asymptotic bias

• p.52 canonical approximators like Laplace approximation, variational bayes
• p.59-62 seems promising but may be outdated as this is written in 2019; simulation-based inference (neural network, flow-based approximators for ode e.g. bayesflow) seems promising (before moving on to julia packages like SciML)
• p.63 cavity distribution separately computes partitioned y_k then combine likelihood focusing on overlap. seems relevant to SD's partial model calibration; are you aware of any mechanism for internal consistency in partial calibration? e.g. two models with the same data but different settings of estimated parameter settings (\theta_1, \theta_2 for the first, \theta_2, \theta_3 for the second) should result in a similar range of \theta_2 estimates.

Challenges: discrete parameter (policy) and hierarchical model

• p.66: discrete parameter needs marginalization: mixture models, survival models, HMMs, discrete measurement error models, missing data
• p.67: position-dependent curvature examples are banana-shaped densities from product of parameter and hierarchical model
  -- Tom's blog on policy optimization and banana show SD model possesses position-dependent curvature (related BayesSD discussion)
  -- market size * market share, first work accuracy * second checking accuracy could be examples for parameter products
  -- low-level parameter is less confusing than global parameters..
  -- dynamic target acceptance (adapt_delta) seems to be the key to speeding up hierarchical ode