Tuning hyper-parameters based on unified Bayesian and SD taxonomy

[hyunjimoon]

Goal

to figure out the best way to tune the following hyper-parameters and its useful resource. I left the link for each discussion (#12, #7, #9, #11) as they need active development, so please leave comment in each link! If you think of any other tunable parmameters, please leave a comment so that I can allocate a new discussion thread for that.

1. Typify model into PA or PAD based on the research purpose in #12

• Some models are useful, but how do we know which ones?Towards a unified Bayesian model taxonomy

2. Typify parameters to assumed or assumed time-series or estimated for testing in #7

• this discussion on holding constant (clamping) between two leaders (Andrew Gelman and Judea Pearl)

3. Tuning time-step for integration approximator in #9

4. Tuning window, the number of time-step, for behavior classifier (or pattern recognition) in #11

• Chp.6 Pattern recognition for model testing, calibration, and behavior analysis from Analytical methods for dynamic modelers

1~4 are all tuning, the only difference being the existence of pre-determined category or range; in 1 and 2, we are classifying into two or three buckets whereas for 3 and 4 we have larger number of choices, infinite in fact.

Another name for the above is, 1: including the real data or not (theoretical vs empirical), 2: parameter of interest or influence, 3: implementation of approximator, 4: resolution of classifier. All notation will follow (from the above paper in ###1):

[hyunjimoon]

The above table is for static Bayesian inference, but as SD (dynamic Bayesian inference) which has time-series format of data as both input and as generated, I extended the definition as follows. In Stats., target simulated msr and driving data are usually the same, but in SD, they are separated.

Example of the above classification is as below:

May I ask for some feedback @tomfid?

[tomfid]

Looks good. Just to be sure I understand the terminology:

"Latent Parameters" are latent because they're not directly observable, where as the "Observable" items are data or synthetic data (unrealized or realized observables in the lingo in table above)?

Msr = short for measurement?

Should the Observed y~ be split into something like "comparison data" (stuff you use to estimate the model) and "driving data" (stuff you use as exogenous inputs)? For example, if the Customer Order Rate is used to drive the model, presumably the Inventory, Shipment rate, etc. would have Target Simulated series (as shown), plus a companion Target Simulated Msr = Target Simulated + noise, plus comparison data from the real system (comparison data).

[tomfid]

If you point me to the inventory model, I'll mark up further according to my comments.

[hyunjimoon]

Latent Parameters" are latent because they're not directly observable,

Yes. I strictly followed the notation used in the paper I recommended in the paper I recommended above.

Msr = short for measurement?

Yes

[hyunjimoon]

Code-wise steps and in/outputs for three Bayesian checks (functions) prior predictive check (draws2data), posterior predictive check (data2draws), simulation-based calibration (draws2data2draws) are as follows:

draws2data

1. D ata block input of hyperparameter

# input: N (=timelength)
# output: `data_draws2data` (name datablock_draw2data? - hyperparameter is data here)

2. P(D) 1+ placeholder for latent parameter and unrealized-observable

# input: $p(\theta), p(y|\theta)$ 
# output: `draws2data_model.stan` representing P, p(theta, y) = p(theta)p(y|theta)

command and output of sub-procedure (need @Dashadower's help in implementing)

• set_prior_var: prior_variation
• set_prior_struc: functions.stan file
• set_prior_desire (demand?): ??
• `set_prior(, sigma): measurement error / sampling distribution (e.g. model.set_prior("prey_obs", "lognormal", "prey", "sigma"))

All in stan generated quantities block.

3. A(P(D)) 2+ Approximator

# input: choice of A (e.g. euler, euler_45, euler_auto)
# output: y_1..N

command for A(P(D)): draws2data_model.[sample/variational/optimize](data=data_draws2data)

data2draw

1. D ata block input of hyperparameter and realized observed quantities

# input: N, (other hps?),  \tilde{y}_{1..N}  hyperparameter + realized observable 
# output: `data_data2draws`

2. P(D): 1+ placeholder for latent parameter and unrealized-observable

# input: $p(\theta), p(y|\theta)$ 
# output: `data2draws.stan` representing P, $p(theta, y) = p(theta)p(y|theta)$

command and output of sub-procedure

in stan transformed parameter, model block

• set_prior_var: prior_variation
• set_prior_struc: functions.stan file
• set_prior_desire (demand?): ??
• set_prior(, sigma): measurement error / sampling distribution (e.g. model.set_prior("prey_obs", "lognormal", "prey", "sigma"))

in stan generative quantities block (??)

• loglik
• predictive checks

3. A(P(D)): 2 + Approximator

input: choice of inference algorithm e.g. MCMC, SMC, VI, Opt
output: $\theta_1..M |\tilde{y_1..N}$

draws2data2draws

1. data2draws

input:
output

2. draws2data

input:
output

3. data2draws(draws2data) vs p(theta)

input:
output

@Dashadower, I am making a function for each of the three and wanted to share the blueprint. P, A, D is from this @paul-buerkner 's paper. Any comments from any of you will be very appreciated!

Q. In our dynamic model context, what would

• be the difference btw PA and PAD? p(\theta|y), p(\theta|\tilde{y})
• PA be as an example model or code?