How does HMM reduce to GMM?

[alephpi]

In section 4.2.1 of the paper, you said

Therefore, we set $P_l = 0$ in all DVBx experiments. This further simplifies the inference, as the HMM becomes a Gaussian mixture model (GMM) (see Eq. 1)

I just don't see how the HMM reduces to GMM in this case. If the looping probability becomes 0, you still have non-zero transition probability and hence a complete temporal structure depending on time. How does it become a GMM where no temporal structure exists?

[domklement]

Hi,

Let's look at the equation (1) describing the transition probability: $p(z_t = s | z_{t-1} = s') = (1-P_l)\pi_s + \delta(s = s')P_l$. We can say that the probability of $p(z_t | z_{t-1})$ depends on both, $z_t$ and $z_{t-1}$, which makes it a function of both variables, i.e. $p(z_t | z_{t-1}) = f(z_t, z_{t-1})$ (mathematically speaking).

If we set $P_l = 0$, we can see that:
$p(z_t=s | z_{t-1}=s') = (1-0)\pi_s + \delta(s=s')\cdot 0 = \pi_s$. Thus, we can see that the transition probability collapses to a simple function $p(z_t = s| z_{t-1} = s') = \pi_s$, which makes it a function of $z_t$ only, i.e. $p(z_t | z_{t-1}) = f(z_t)$. Therefore, we can say that the distribution of the state at time $t$ is independend from the state at time $t-1$, i.e. it becomes a GMM and we no longer need forward-backward algorithm for the inference.

Just to summarize without any math, after setting $P_l=0$, the transitions become independent, which means there's no transition dependency between the states. Hence, HMM reduces to GMM.
It can be nicely visualized using Bayesian graphs (BG), i.e., if you draw a general HMM BG and then make the transitions independent (remove arcs between $z_{t-1}, z_t$, it becomes a BG of GMM.

I hope this explanation helps. If not, feel free to ask further questions and I will try to write down all of the equations and prove it rigorously.

Have a nice day!

Dominik

[alephpi]

Ok, now I understand the probability $p(z_t=s)$ is independent from state at time $t-1$. Which means for every timestep $t$, $z$ is generated from a categorical distribution $\text{Cat}(\boldsymbol{\pi})$, but I still don't see any GMM structure here.

Or do you actually mean that the model $p(x)$ = $\int p(x|z)p(z)\prod_s p(y_s) dy_1...dy_s dz$ is a GMM? I use the same notation as in the VBx report (Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks), where here the model only considers sample $x$ generated from one timestep rather than a sequence of $x_1,...x_T$, since they are mutually independent from each other.

[alephpi]

Hi @domklement, would you give some comments to my reply?

[domklement]

Hi @domklement, would you give some comments to my reply?

I appologize for not getting back to you sooner, I had quite busy schedule past few days. I'm going to write the math down today and will reply ASAP.

Dominik

[fnlandini]

Hi @alephpi I thought of commenting because maybe this is the missing link. The distributions in the states of the HMM are Gaussians. Now, if the states are independent, then you have each observation determined by the categorical distribution and the Gaussian itself so you end up with a Gaussian mixture model. The equations will probably clarify this, but I thought this might help.