# Conditional UGMs¶

Consider the following simple pairwise UGM.

From the general formula for pairwise UGMs given in class, we have that: $$p(x_1, x_2, x_3, x_4) = \frac{1}{Z}\prod_{j=1}^4 \phi_j(x_j)\prod_{\{i,j\} \in E} \phi_{i,j}(x_i,x_j)$$

To see what happens when we condition on variables, let's write this formula out explicitly for the simple UGM shown above.

$$p(x_1, x_2, x_3, x_4) = \frac{1}{Z}\phi_1(x_1)\phi_2(x_2)\phi_3(x_3)\phi_4(x_4) \phi_{1,3}(x_1,x_3)\phi_{1,4}(x_1,x_4)\phi_{2,4}(x_2,x_4)\phi_{3,4}(x_3,x_4)\phi_{2,3}(x_2,x_3)$$

Now, in general, to get a particular marginal (e.g. $p(x_1)$) we have to sum over all the values the other random variables ($x_2, x_3, x_4$) can take on. I.e. $$p(x_1) = \sum_{x_2\in {X_2}}\sum_{x_3\in {X_3}}\sum_{x_4\in {X_4}}p(x_1, X_2 = x_2, X_3=x_3, X_4=x_4)$$

This is computationally hard in general - but can be much easier if we get to condition on variables that make our UGM simplier. To see how the conditional UGM may be simplier than the full UGM, let's let $x_3 = a$ and $x_4 = b$ (where $a$ and $b$ are some values in the support of $x_3$ and $x_4$ respectively) and see what the resulting conditional distribution looks like:

$$p(x_1, x_2| x_3=a, x_4=b) = \frac{1}{Z}\phi_1(x_1)\phi_2(x_2)\phi_3(a)\phi_4(b) \phi_{1,3}(x_1,a)\phi_{1,4}(x_1,b)\phi_{2,4}(x_2,b)\phi_{3,4}(a,b)\phi_{2,3}(x_2,b)$$

Now, notice that $\phi_3(a), \phi_4(b)$ and $\phi_{3,4}(a,b)$ are just some constants which we could push into our normalizing constant $Z$ and $\phi_{1,3}(x_1,a), \phi_{2,4}(x_2,b)$ and $\phi_{2,3}(x_2,b)$ are now just functions of $x_1$ and $x_2$ respectively. So we could rewrite our conditional distribution more simply as: $$p(x_1, x_2| x_3=a, x_4=b) = \frac{1}{Z'}\phi'_1(x_1)\phi'_2(x_2)$$ where $\phi'_{1}(x_1) = \phi_{1}(x_1)\phi_{1,3}(x_1,a)\phi_{1,4}(x_1,b)$ and simmilarly for $\phi'_{2}(x_2)$ and the new normalizing constant $Z'$ ensures everything sums to one. This formula is equivalent to an even simpler UGM where computing the marginals is trivial,

In question 1 you have to do this in a more complex UGM (you don't have to go through this tedious derivation - that was just for your understanding), but the procedure is the same - figure out what the conditional UGM looks like and then use what you know about the computational complexity of computing marginals for the conditional UGM to answer the question.

# Bayesian Inference¶

The conjugate priors question uses similar mechanical steps to those you've used in the past so we just looked at how you'd get started with each question and left the mechanics for you to figure out.

1. You have the likelihood $p(y | \theta, \alpha)$ and the prior $p(\theta | \alpha)$, so you can just use Bayes rule to get the posterior: $p(\theta | y, \alpha) = \frac{1}{Z}p(y | \theta, \alpha)p(\theta | \alpha)$ and then substitute in the formulae for the prior and likelihood and solve for the resulting distribution.

2. You're given $p(y|\alpha) = \int p(y,\theta | \alpha) d\theta$. By the chain rule of probability $\int p(y,\theta | \alpha) d\theta = \int p(y |\theta , \alpha)p(\theta | \alpha) d\theta$ and you have expressions for both of those terms. From there you just have to manipulate the expression into the form of a distribution that you recognise to solve the integral.

Questions 3 and 4 follow similar steps.