# Solution 7 Computational Intelligence Lab 2012

## Problem 1

• Given:
• ${\displaystyle \mathbf {x} =\left\langle \mathbf {x} ,\ \mathbf {u} _{d(1)}\right\rangle \mathbf {u} _{d(1)}+\mathbf {r} ^{1}}$
• ${\displaystyle \mathbf {r} ^{1}}$ is orthogonal to ${\displaystyle \mathbf {u} _{d(1)}}$
• For the next step we have: ${\displaystyle \mathbf {r} ^{1}=\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}+\mathbf {r} ^{2}}$
• ${\displaystyle \mathbf {r} ^{2}}$ is orthogonal to ${\displaystyle \mathbf {u} _{d(2)}}$
• Question: Is ${\displaystyle \mathbf {r} ^{2}}$ orthogonal to ${\displaystyle \mathbf {u} _{d(1)}}$? When is it true?
• Solution:
• ${\displaystyle \mathbf {r} ^{2}=\mathbf {r} ^{1}-\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}}$
• ${\displaystyle \left\langle \mathbf {r} ^{2},\ \mathbf {u} _{d(1)}\right\rangle =\left\langle (r^{1}-\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)}),\ \mathbf {u} _{d(1)}\right\rangle }$
• ${\displaystyle =\left\langle r^{1},\ \mathbf {u} _{d(1)}\right\rangle -\left\langle \left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle }$
• since ${\displaystyle \left\langle r^{1},\ \mathbf {u} _{d(1)}\right\rangle =0}$:
• ${\displaystyle \left\langle \mathbf {r} ^{2},\ \mathbf {u} _{d(1)}\right\rangle =\left\langle \mathbf {r} ^{1},\ \mathbf {u} _{d(2)}\right\rangle \left\langle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle }$
• = 0 if ${\displaystyle \left\langle \mathbf {u} _{d(2)},\ \mathbf {u} _{d(1)}\right\rangle =0}$ i.e. if ${\displaystyle \mathbf {u} _{d(2)}}$ and ${\displaystyle \mathbf {u} _{d(1)}}$ are orthogonal.

## Problem 2

• Given:
• Atom selection at iteration ${\displaystyle t}$: ${\displaystyle d(t)^{*}={\underset {d}{argmax}}\left|\left\langle \mathbf {r} ^{t},\ \mathbf {u} _{d}\right\rangle \right|}$
• Assume that ${\displaystyle \mathbf {u} _{d}^{T}\mathbf {u} _{d}=w_{d}}$ ${\displaystyle \forall d}$ where ${\displaystyle {w_{d}}}$ are some numbers not necessarily equal to ${\displaystyle 1}$.
• Question: How will the atom selection rule change? What should you maximize now in order to minimize ${\displaystyle \|\mathbf {r} ^{1}\|_{2}^{2}}$?
• Solution:
• ${\displaystyle d(t)^{*}={\underset {d}{argmax}}\left|\left\langle \mathbf {r} ^{t},\mathbf {u} _{d}\right\rangle \omega _{d}\right|}$