00-distributions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pixyzの確率分布の記述方法\n",
    "\n",
    "ここではまず，Pixyzにおける確率モデルの実装方法について説明します．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f64bc114670>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import argparse\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "import numpy as np\n",
    "\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 シンプルな確率分布の設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ガウス分布を作るためには，`Normal`をインポートして，平均（loc）と標準偏差（scale）を定義します．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pixyz.distributions import Normal\n",
    "\n",
    "x_dim = 50\n",
    "p1 = Normal(var=[\"x\"], loc=0, scale=1, dim=x_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "なお``var``には，変数の名前を設定します．ここでは`\"x\"`を設定しています．\n",
    "\n",
    "また，dimでは次元数を指定します．ここではdimが50となっていますから，50次元のサンプルを生成する形になります．\n",
    "\n",
    "上記で定義したp1の情報は次のようにみることができます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal\n",
      "p(x)\n"
     ]
    }
   ],
   "source": [
    "print(p1.distribution_name) \n",
    "print(p1.prob_text) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "distribution_nameでは，確率分布の名前を確認できます．\n",
    "\n",
    "prob_textでは，確率分布の形をテキストで出力できます．ここでテキストに書かれている確率変数は，上記のvarで指定したものです.\n",
    "\n",
    "また，p1を丸ごとprintすると，以下のように表示されます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(x) (Normal)\n",
      "Network architecture:\n",
      "  Normal()\n"
     ]
    }
   ],
   "source": [
    "print(p1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次に，定義した分布からサンプリングしてみましょう． サンプリングは，`sample()`によって実行します．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'x': tensor([[-1.5256, -0.7502, -0.6540, -1.6095, -0.1002, -0.6092, -0.9798, -1.6091,\n",
      "         -0.7121,  0.3037, -0.7773, -0.2515, -0.2223,  1.6871,  0.2284,  0.4676,\n",
      "         -0.6970, -1.1608,  0.6995,  0.1991,  0.8657,  0.2444, -0.6629,  0.8073,\n",
      "          1.1017, -0.1759, -2.2456, -1.4465,  0.0612, -0.6177, -0.7981, -0.1316,\n",
      "          1.8793, -0.0721,  0.0663, -0.4370,  0.7626,  0.4415,  1.1651,  2.0154,\n",
      "          0.2152, -0.5242, -0.1860, -0.6446,  1.5392, -0.8696, -3.3312, -0.7479,\n",
      "          1.1173,  0.2981]])}\n",
      "tensor([[-1.5256, -0.7502, -0.6540, -1.6095, -0.1002, -0.6092, -0.9798, -1.6091,\n",
      "         -0.7121,  0.3037, -0.7773, -0.2515, -0.2223,  1.6871,  0.2284,  0.4676,\n",
      "         -0.6970, -1.1608,  0.6995,  0.1991,  0.8657,  0.2444, -0.6629,  0.8073,\n",
      "          1.1017, -0.1759, -2.2456, -1.4465,  0.0612, -0.6177, -0.7981, -0.1316,\n",
      "          1.8793, -0.0721,  0.0663, -0.4370,  0.7626,  0.4415,  1.1651,  2.0154,\n",
      "          0.2152, -0.5242, -0.1860, -0.6446,  1.5392, -0.8696, -3.3312, -0.7479,\n",
      "          1.1173,  0.2981]])\n"
     ]
    }
   ],
   "source": [
    "samples = p1.sample()\n",
    "print(samples)\n",
    "print(samples[\"x\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "出力はdict形式になっています．\n",
    "\n",
    "サンプリング結果を確認したい変数について指定することで，中身を確認できます（ただし，この例では変数は\"x\"のみです）．\n",
    "\n",
    "なお，サンプリング結果は，PyTorchのtensor形式になっています．"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "続いて，尤度の計算方法を説明します．\n",
    "例えば，次のサンプルがあったとしましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "_sample1 = torch.Tensor([[-0.3030, -1.7618,  0.6348, -0.8044, -1.0371, -1.0669, -0.2085,\n",
    "         -0.2155,  2.2952,  0.6749,  1.7133, -1.7943, -1.5208,  0.9196,\n",
    "         -0.5484, -0.3472,  0.4730, -0.4286,  0.5514, -1.5474,  0.7575,\n",
    "         -0.4068, -0.1277,  0.2804,  1.7460,  1.8550, -0.7064,  2.5571,\n",
    "          0.7705, -1.0739, -0.2015, -0.5603, -0.6240, -0.9773, -0.1637,\n",
    "         -0.3582, -0.0594, -2.4919,  0.2423,  0.2883, -0.1095,  0.3126,\n",
    "         -0.3417,  0.9473,  0.6223, -0.4481, -0.2856,  0.3880, -1.1435,\n",
    "         -0.6512]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このサンプルにおける，p1の対数尤度を計算します．\n",
    "\n",
    "これは，log_likelihoodによって簡単に計算できます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-72.5003])\n"
     ]
    }
   ],
   "source": [
    "log_like = p1.log_likelihood({\"x\":_sample1})\n",
    "print(log_like)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "なお，log_likelihoodの引数はdictで与えることに注意してください． これは，どの確率変数かを指定するためです（この例では，xしかありませんが）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_dim = 50\n",
    "_p1 = Normal(cond_var=[\"mu\"], var=[\"x\"], loc=\"mu\", scale=1, dim=x_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(x|mu) (Normal)\n",
      "Network architecture:\n",
      "  Normal()\n"
     ]
    }
   ],
   "source": [
    "print(_p1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': tensor(2.2820), 'mu': 0}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_p1.sample({\"mu\":0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "_p2 = Normal(var=[\"mu\"], loc=0, scale=1, dim=x_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pixyzでは分布の積は，掛け算で表すことができます"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(x,mu) = p(x|mu)p(mu)\n",
      "Network architecture:\n",
      "  p(mu) (Normal): Normal()\n",
      "  p(x|mu) (Normal): Normal()\n"
     ]
    }
   ],
   "source": [
    "_p3 = _p1 * _p2\n",
    "print(_p3) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mu': tensor([[ 1.6734,  0.0103,  0.9837,  0.8793, -1.4504, -1.1802,  0.4100,  0.4085,\n",
       "           0.3956, -0.9823,  1.3264,  0.8547, -0.2805,  0.7000, -1.4567,  1.6089,\n",
       "           0.1716, -0.1600, -0.5047, -1.4746, -0.3416, -0.3003,  1.3075, -1.1628,\n",
       "           0.2911,  1.9907, -0.9247, -0.9301,  1.4301,  0.4208, -0.3538,  0.7639,\n",
       "          -0.9276,  1.1120,  0.1573,  1.2540,  1.3275, -0.4954,  1.5496,  0.3476,\n",
       "           0.0930,  0.6147, -0.6447, -0.2870,  3.3212, -0.4021, -0.7123, -0.6200,\n",
       "          -0.2281, -0.7893]]),\n",
       " 'x': tensor([[ 0.6363, -1.0566,  0.7752,  0.6638, -1.1799, -0.6205,  0.0916,  1.9202,\n",
       "          -1.1252, -0.0627,  0.7780,  0.5075, -1.0280, -0.2234, -0.8833,  1.4996,\n",
       "           0.9291, -0.5668, -0.6324, -1.1942, -0.3041, -0.9381,  0.4928, -1.8523,\n",
       "           1.0617,  0.9168, -1.1262, -1.4904,  2.1119, -0.0962,  1.4364,  1.3516,\n",
       "          -0.6771,  0.3189,  0.3995,  1.5423,  1.2180, -0.1828,  3.0534,  0.8514,\n",
       "          -0.4755,  1.4523, -0.9303,  0.1010,  2.1778, -1.0533, -0.8154,  0.0737,\n",
       "          -0.7694,  0.1059]])}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_p3.sample()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 深層ニューラルネットワークと組み合わせた確率分布の設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次に， 確率分布のパラメータを深層ニューラルネットワークで定義します．\n",
    "\n",
    "例えば，ガウス分布の平均$\\mu$と分散$\\sigma^2$は， パラメータ$\\theta$を持つ深層ニューラルネットワークによって，$\\mu=f(x;\\theta)$および$\\sigma^2=g(x;\\theta)$と定義できます．\n",
    "\n",
    "したがって，ガウス分布は${\\cal N}(\\mu=f(x;\\theta),\\sigma^2=g(x;\\theta))$となります．\n",
    "\n",
    "Pixyzでは，次のようなクラスを記述することで，これを実現できます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_dim = 20\n",
    "\n",
    "class P2(Normal):\n",
    "    def __init__(self):\n",
    "        super(P2, self).__init__(cond_var=[\"x\"], var=[\"a\"])\n",
    "\n",
    "        self.fc1 = nn.Linear(x_dim, 10)\n",
    "        self.fc21 = nn.Linear(10, a_dim)\n",
    "        self.fc22 = nn.Linear(10, a_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        h1 = F.relu(self.fc1(x))\n",
    "        return {\"loc\":self.fc21(h1), \"scale\":F.softplus(self.fc22(h1))} # mean and variance\n",
    "    \n",
    "p2 = P2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "まず， ガウス分布クラスを継承することで，ガウス分布のパラメータを深層ニューラルネットワークで定義することを明示します．\n",
    "\n",
    "次に，コンストラクタで，利用するニューラルネットワークを記述します．これは，通常のPyTorchと同じです．\n",
    "\n",
    "唯一異なる点は，superの引数にvarとcond_varの名前を指定している点です．\n",
    "\n",
    "varは先程見たように，出力する変数の名前を指定します．一方，cond_varではニューラルネットワークの入力変数の名前を指定します．これは，ここで定義する分布において，条件付けられる変数とみなすことができます．\n",
    "\n",
    "forwardについても，通常のPyTorchと同じです．ただし，注意点が2つあります．\n",
    "\n",
    "* 引数の名前と数は，cond_varで設定したものと同じにしてください． 例えば，cond_var=[\"x\", \"y\"]とした場合は，forward(self, x, y)としてください．ただし，引数の順番は変えても構いません．\n",
    "* 戻り値は，それぞれの確率分布のパラメータになります．上記の例ではガウス分布なので，平均と分散を指定しています．\n",
    "\n",
    "そして最後に，定義した確率分布クラスのインスタンスを作成します．\n",
    "\n",
    "次に，先程の例と同様，確率分布の情報を見てみましょう."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(a|x) (Normal)\n",
      "Network architecture:\n",
      "  P2(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=10, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=10, out_features=20, bias=True)\n",
      "  )\n"
     ]
    }
   ],
   "source": [
    "print(p2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p2の分布は，xで条件付けた形になっています．これらの表記は，superの引数で設定したとおりになっています．"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次に，先程の例のように，サンプリングしてみましょう．\n",
    "\n",
    "注意しなければならないのは，先ほどと異なり，条件づけた変数xがあるということです．\n",
    "\n",
    "先程おいた_samplesをxとしてサンプリングしましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': tensor([[ 0.6543,  0.4414,  0.4326,  0.6563,  1.6146,  0.7543, -0.4256, -0.5965,\n",
      "         -0.5136,  0.3094, -0.2979,  0.2488,  0.0521, -0.5897, -0.8235, -0.6019,\n",
      "         -0.4913, -0.8328, -0.5683, -0.8692]]), 'x': tensor([[-0.3030, -1.7618,  0.6348, -0.8044, -1.0371, -1.0669, -0.2085, -0.2155,\n",
      "          2.2952,  0.6749,  1.7133, -1.7943, -1.5208,  0.9196, -0.5484, -0.3472,\n",
      "          0.4730, -0.4286,  0.5514, -1.5474,  0.7575, -0.4068, -0.1277,  0.2804,\n",
      "          1.7460,  1.8550, -0.7064,  2.5571,  0.7705, -1.0739, -0.2015, -0.5603,\n",
      "         -0.6240, -0.9773, -0.1637, -0.3582, -0.0594, -2.4919,  0.2423,  0.2883,\n",
      "         -0.1095,  0.3126, -0.3417,  0.9473,  0.6223, -0.4481, -0.2856,  0.3880,\n",
      "         -1.1435, -0.6512]])}\n",
      "tensor([[ 0.6543,  0.4414,  0.4326,  0.6563,  1.6146,  0.7543, -0.4256, -0.5965,\n",
      "         -0.5136,  0.3094, -0.2979,  0.2488,  0.0521, -0.5897, -0.8235, -0.6019,\n",
      "         -0.4913, -0.8328, -0.5683, -0.8692]])\n",
      "tensor([[-0.3030, -1.7618,  0.6348, -0.8044, -1.0371, -1.0669, -0.2085, -0.2155,\n",
      "          2.2952,  0.6749,  1.7133, -1.7943, -1.5208,  0.9196, -0.5484, -0.3472,\n",
      "          0.4730, -0.4286,  0.5514, -1.5474,  0.7575, -0.4068, -0.1277,  0.2804,\n",
      "          1.7460,  1.8550, -0.7064,  2.5571,  0.7705, -1.0739, -0.2015, -0.5603,\n",
      "         -0.6240, -0.9773, -0.1637, -0.3582, -0.0594, -2.4919,  0.2423,  0.2883,\n",
      "         -0.1095,  0.3126, -0.3417,  0.9473,  0.6223, -0.4481, -0.2856,  0.3880,\n",
      "         -1.1435, -0.6512]])\n"
     ]
    }
   ],
   "source": [
    "samples = p2.sample({\"x\":_sample1})\n",
    "print(samples)\n",
    "print(samples[\"a\"])\n",
    "print(samples[\"x\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "出力には，aとxの２つのサンプルがあります．\n",
    "\n",
    "aが今回計算したサンプルで，xについては，引数として与えたサンプルがそのまま入っています．"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次に，尤度計算をします．\n",
    "\n",
    "尤度計算では， 全ての変数のデータを与える必要があります．\n",
    "\n",
    "aについては，上記でサンプルした値を入れて計算しましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-19.4727], grad_fn=<SumBackward1>)\n"
     ]
    }
   ],
   "source": [
    "_sample2 = samples[\"a\"]\n",
    "log_like = p2.log_likelihood({\"x\":_sample1, \"a\":_sample2})\n",
    "print(log_like)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "なお，これはもっと簡単に書くことができます．\n",
    "\n",
    "上記のサンプリングで全ての変数とその値がdict形式で出力されたので，それをそのままlog_likelihoodの引数とすればいいのです．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-19.4727], grad_fn=<SumBackward1>)\n"
     ]
    }
   ],
   "source": [
    "log_like = p2.log_likelihood(samples)\n",
    "print(log_like)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このように記述できる利点は， **変数の数が増えても同じ書き方で尤度計算を実行できる**ことです．\n",
    "\n",
    "サンプリング->尤度計算という処理は，深層生成モデルでは数多く登場します． このように記述できることで，どのような確率分布の形であっても容易に尤度を計算できるようになるのです．"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 確率分布の積の設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最初に書いたとおり，Pixyzの特徴の1つは，確率分布の積を簡単に記述できることです．\n",
    "\n",
    "ここで，これまで設定した確率分布を再度確認しましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(x) (Normal)\n",
      "Network architecture:\n",
      "  Normal()\n",
      "Distribution:\n",
      "  p(a|x) (Normal)\n",
      "Network architecture:\n",
      "  P2(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=10, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=10, out_features=20, bias=True)\n",
      "  )\n"
     ]
    }
   ],
   "source": [
    "print(p1) \n",
    "print(p2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数式的には，これらを掛け合わせることで同時分布を定義できます． \n",
    "\n",
    "$p(x,a) = p(a|x)p(x)$\n",
    "\n",
    "では，Pixyzではこれをどのように記述するのでしょうか．\n",
    "\n",
    "実は，それぞれの確率分布のインスタンスを**文字通り掛け合わせるだけ**でいいのです．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "p3 = p1 * p2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p3ではどのような確率分布が定義されているか確認しましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(a,x) = p(a|x)p(x)\n",
      "Network architecture:\n",
      "  p(x) (Normal): Normal()\n",
      "  p(a|x) (Normal): P2(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=10, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=10, out_features=20, bias=True)\n",
      "  )\n"
     ]
    }
   ],
   "source": [
    "print(p3) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "確かに，同時分布が定義されていることがわかります．\n",
    "\n",
    "このインスタンスp3からも，これまでと同様にサンプリングや尤度計算ができます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-82.8476], grad_fn=<ThAddBackward>)\n"
     ]
    }
   ],
   "source": [
    "samples = p3.sample()\n",
    "log_like = p3.log_likelihood(samples)\n",
    "print(log_like)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "この例をみてわかるように，サンプリングや尤度計算において，もはやp3を構成する各分布の形を気にする必要はありません．\n",
    "\n",
    "このような記述方法は，Pythonにおける確率モデリングライブラリでは**Pixyzが初めて採用しました**．"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "これは，確率分布が増えても同じです． 例えば，次の確率分布を定義しましょう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(y|a,x) (Normal)\n",
      "Network architecture:\n",
      "  P4(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc2): Linear(in_features=20, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=20, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=20, out_features=20, bias=True)\n",
      "  )\n"
     ]
    }
   ],
   "source": [
    "class P4(Normal):\n",
    "    def __init__(self):\n",
    "        super(P4, self).__init__(cond_var=[\"a\", \"x\"], var=[\"y\"])\n",
    "\n",
    "        self.fc1 = nn.Linear(x_dim, 10)\n",
    "        self.fc2 = nn.Linear(a_dim, 10)\n",
    "        self.fc21 = nn.Linear(10+10, 20)\n",
    "        self.fc22 = nn.Linear(10+10, 20)\n",
    "\n",
    "    def forward(self, a, x):\n",
    "        h1 = F.relu(self.fc1(x))\n",
    "        h2 = F.relu(self.fc2(a))\n",
    "        h12 = torch.cat([h1, h2], 1)\n",
    "        return {\"loc\":self.fc21(h12), \"scale\":F.softplus(self.fc22(h12))}\n",
    "    \n",
    "p4 = P4()\n",
    "\n",
    "print(p4) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p4は，aとxで条件付けられた確率分布です．他に$p(a|x)$，$p(x)$をつかって，同時分布は以下のように書けます．\n",
    "\n",
    "$p(y,a,z)=p(y|a,x)p(a|x)p(x)$\n",
    "\n",
    "同時分布が3つの確率分布の積になっていますが，これも上記と同様に書けます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution:\n",
      "  p(y,a,x) = p(y|a,x)p(a|x)p(x)\n",
      "Network architecture:\n",
      "  p(x) (Normal): Normal()\n",
      "  p(a|x) (Normal): P2(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=10, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=10, out_features=20, bias=True)\n",
      "  )\n",
      "  p(y|a,x) (Normal): P4(\n",
      "    (fc1): Linear(in_features=50, out_features=10, bias=True)\n",
      "    (fc2): Linear(in_features=20, out_features=10, bias=True)\n",
      "    (fc21): Linear(in_features=20, out_features=20, bias=True)\n",
      "    (fc22): Linear(in_features=20, out_features=20, bias=True)\n",
      "  )\n"
     ]
    }
   ],
   "source": [
    "p5 = p4 * p2 * p1\n",
    "print(p5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "distributionをみると，p5がどのような分布から構成されているかを表示します． サンプリングや尤度計算も全く同じようにできます．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-106.6829], grad_fn=<ThAddBackward>)\n"
     ]
    }
   ],
   "source": [
    "samples = p5.sample()\n",
    "log_like = p5.log_likelihood(samples)\n",
    "print(log_like)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このように，分布や深層ニューラルネットワークの形を気にせずに，同じ記述方法でサンプリングや尤度計算ができます．\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}