Merge pull request #11 from kashefy/pca-svd

add note on svd vs eig

notes/01_pca/2_apply-pca.tex

@@ -111,12 +111,29 @@ \subsubsection{A note on implementation:}
 \item Numerical instabilities in eigendecomposition of very large covariance matrix $\rightarrow$ \textit{SVD}
 \item SVD not applicable to Kernel-PCA $\rightarrow$ \textit{Eig}
 \item Computational efficiency $\rightarrow$ \textit{SVD} (depends...?)
-\end{itemize}
 
 \slidesonly{
 $$
 \vec X \in \R^{N \times p} \quad \leadsto \quad \vec C = \frac{1}{p} \vec X~\vec X^\top
 $$
+}
+\item Possible differences between PCA via SVD and PCA via eig: SVD may flip the sign.
+
+\notesonly{
+
+The reason for the difference in sign between the eigenvector's you would get via \text{Eig} and the eigenvector you would get via \text{SVD}\footnote{Python's scikit-learn package uses SVD for its PCA implementation.}, is that SVD, in practice, may arbitrarily flip the direction of the eigenevectors.
+Possible reasons for the sign ambiguity according to \ref{bro2008resolving}:
+\begin{enumerate}
+\item  It's a mechanism in the SVD implementation (there's more than one SVD implementation) to mitigate numerical instabilities,
+\item Due to centering of the data,
+\item an SVD impelementation could have a random component.
+\end{enumerate}
+
+In both approaches you should be able to get the same reconstructions from both of them, i.e. the sign of the components along the flipped PC will also be flipped. Depending on the application, it may be necessary to "detect" if SVD flipped the direction or not. Numerical methods to detect this exist. A reason to care about what the sign should be, is when you want to interpret what an eigenvector represents to possibly assign meaning to the component along this PC: Is the component positive or negative along the PC, or this point has a larger component along the PC than another point. The classicial example of "interpreting" PCs would be "Eigenfaces". PCA is applied on images of faces. It is similar to what we ask you to do in Exercise 1.4. The Eigenvectors extracted form image data can be represented as images themselves. If SVD gives you "flipped" eigenvectors, visualizing them would give you the "negative" of that image. It's not as intuitive to look at negatives, but a person will still be able to make sense of them. However, imagine trying to interpret the eigenvector of other highdimensional data that isn't as pretty as pictures. Lnowing that SVD could have flipped the sign, could give you a hard time trying to make sense of what the eigenvector represents.
+
 }
 
+\end{itemize}
+
 \end{frame}
+

notes/01_pca/bibliography.bib

@@ -1,29 +1,11 @@
-@book{sutton1998introduction,
-  title={Introduction to reinforcement learning},
-  author={Sutton, Richard S and Barto, Andrew G and others},
-  volume={135},
-  year={1998},
-  publisher={MIT press Cambridge}
-}
-@Book{Bertsekas07,
-  author = 	 {D. P. Bertsekas},
-  title = 	 {Dynamic Programming and Optimal Control},
-  publisher ={Athena Scientific},
-  year = 	 {2007},
-  volume = 	 {2},
-  edition =  {3rd},
-  url = {http://www.control.ece.ntua.gr/UndergraduateCourses/ProxTexnSAE/Bertsekas.pdf}
-}
-@Article{Watkins92,
-  author = 	 {C. Watkins and P. Dayan},
-  title = 	 {Q-learning},
-  journal = 	 {Machine Learning},
-  year = 	 {1992},
-  OPTkey = 	 {},
-  volume = 	 {8},
-  OPTnumber = 	 {},
-  pages = 	 {279--292},
-  OPTmonth = 	 {},
-  OPTnote = 	 {},
-  OPTannote = 	 {}
+@article{bro2008resolving,
+  title={Resolving the sign ambiguity in the singular value decomposition},
+  author={Bro, Rasmus and Acar, Evrim and Kolda, Tamara G},
+  journal={Journal of Chemometrics: A Journal of the Chemometrics Society},
+  volume={22},
+  number={2},
+  pages={135--140},
+  year={2008},
+  publisher={Wiley Online Library},
+  url={https://prod-ng.sandia.gov/techlib-noauth/access-control.cgi/2007/076422.pdf}
 }