# Solution 7 Computational Intelligence Lab 2012

## Problem 1

• Given:
• $\mathbf {x} =\left\langle \mathbf {x} ,\ \mathbf {u} _{d(1)}\right\rangle \mathbf {u} _{d(1)}+\mathbf {r} ^{1}$
• $\mathbf {r} ^{1}$ is orthogonal to $\mathbf {u} _{d(1)}$
• For the next step we have: $\mathbf {r} ^{1}=\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}+\mathbf {r} ^{2}$
• $\mathbf {r} ^{2}$ is orthogonal to $\mathbf {u} _{d(2)}$
• Question: Is $\mathbf {r} ^{2}$ orthogonal to $\mathbf {u} _{d(1)}$? When is it true?
• Solution:
• $\mathbf {r} ^{2}=\mathbf {r} ^{1}-\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}$
• $\left\langle \mathbf {r} ^{2},\ \mathbf {u} _{d(1)}\right\rangle =\left\langle (r^{1}-\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}),\ \mathbf {u} _{d(1)}\right\rangle$
• $=\left\langle r^{1},\ \mathbf {u} _{d(1)}\right\rangle -\left\langle \left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle$
• since $\left\langle r^{1},\ \mathbf {u} _{d(1)}\right\rangle =0$:
• $\left\langle \mathbf {r} ^{2},\ \mathbf {u} _{d(1)}\right\rangle =\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \left\langle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle$
• = 0 if $\left\langle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle =0$ i.e. if $\mathbf {u} _{d(2)}$ and $\mathbf {u} _{d(1)}$ are orthogonal.

## Problem 2

• Given:
• Atom selection at iteration $t$: $d(t)^{*}={\underset {d}{argmax}}\left|\left\langle \mathbf {r} ^{t},\ \mathbf {u} _{d}\right\rangle \right|$
• Assume that $\mathbf {u} _{d}^{T}\mathbf {u} _{d}=w_{d}$ $\forall d$ where ${w_{d}}$ are some numbers not necessarily equal to $1$.
• Question: How will the atom selection rule change? What should you maximize now in order to minimize $\|\mathbf {r} ^{1}\|_{2}^{2}$?
• Solution:
• $d(t)^{*}={\underset {d}{argmax}}\left|\left\langle \mathbf {r} ^{t},\mathbf {u} _{d}\right\rangle \omega _{d}\right|$