Compressive Sampling of Analog Signals Moshe Mishali Yonina

Compressive Sampling (of Analog Signals) Moshe Mishali Yonina C. Eldar Technion – Israel Institute of Technology http: //www. technion. ac. il/~moshiko http: //www. ee. technion. ac. il/people/Yonina. Eldar moshiko@tx. technion. ac. il yonina@ee. technion. ac. il Advanced topics in sampling (Course 049029) Seminar talk – November 2008

Context - Sampling Analog world Continuous signal Digital world Sampling A 2 D Reconstruction D 2 A 2

Compression “Can we not just directly measure the part that will not end up being thrown away ? ” Donoho Original 2500 KB 100% Compressed 392 KB 148 950 15% 6% 38% 3

Outline • Mathematical background • From discrete to analog • Uncertainty principles for analog signals • Discussion 4

References • M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors, " IEEE Trans. on Signal Processing, vol. 56, no. 10, pp. 4692 -4702, Oct. 2008. • M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals, " CCIT Report #639, Sep. 2007, EE Dept. , Technion. • Y. C. Eldar, "Compressed Sensing of Analog Signals", submitted to IEEE Trans. on Signal Processing, June 2008. • Y. C. Eldar and M. Mishali, "Robust Recovery of Signals From a Union of Subspaces", ar. Xiv. org 0807. 4581, submitted to IEEE Trans. Inform. Theory, July 2008. # • Y. C. Eldar, "Uncertainty Relations for Analog Signals", submitted to IEEE Trans. Inform. Theory, Sept. 2008. 5

Mathematical background • • Basic ideas of compressed sensing Single measurement model (SMV) Multiple- and Infinite- measurement models (MMV, IMV) The “Continuous to finite” block (CTF) 6

Compressed Sensing “Can we not just directly measure the part that will not end up being thrown away ? ” Donoho “sensing … as a way of extracting information about an object from a small number of randomly selected Candès et. al. observations” Analog Audio Signal Nyquist rate Sampling High-rate Compressed Sensing Compression (e. g. MP 3) Low-rate 7

Concept Goal: Identify the bucket with fake coins. Nyquist: Weigh a coin from each bucket Compressed Sensing: numbers Compression Weigh a linear combination of coins from all buckets Bucket # 1 number 8

Mathematical Tools non-zero entries at least measurements Recovery: brute-force, convex optimization, greedy algorithms, and more… 9

CS theory – on 2 slides Compressed sensing (2003/4 and on) – Main results is uniquely determined by Donoho and Elad, 2003 Maximal cardinality of linearly independent column subsets Hard to compute ! 10

CS theory – on 2 slides Compressed sensing (2003/4 and on) – Main results is uniquely determined by Donoho and Elad, 2003 is random with high probability Donoho, 2006 and Candès et. al. , 2006 Convex and tractable Donoho, 2006 and Candès et. al. , 2006 NP-hard Greedy algorithms: OMP, FOCUSS, etc. Tropp, Cotter et. al. Chen et. al. and many other 11

Sparsity models measurements unknowns MMV SMV IMV = How can Joint sparsity Infinite Measurement Vectors (countable or uncountable) with joint sparsity prior be found ? Infinite many variables Infinite many constraints Exploit prior Reduce problem dimensions 12

Reduction Framework Find a frame for Solve MMV Theorem Mishali and Eldar (2008) IMV Deterministic reduction Infinite structure allows CS for analog signals MMV 13

From discrete to analog • Naïve extension • The basic ingredients of sampling theorem • • Sparse multiband model Rate requirements Multicoset sampling and unique representation Practical recovery with the CTF block • Sparse union of shift-invariant model • Design of sampling operator • Reconstruction algorithm 14

Naïve Extension to Analog Domain Standard CS Discrete Framework Analog Domain Sparsity prior what is a sparse analog signal ? Generalized sampling Infinite sequence Finite dimensional elements Random is stable w. h. p Stability Operator Continuous signal Randomness Infinitely many Need structure for efficient implementation Reconstruction Finite program, well-studied Undefined program over a continuous signal 15

Naïve Extension to Analog Domain Standard CS Discrete Framework Analog Domain Questions: Sparsity prior what is a sparse analog signal ? 1. What is the definition of analog sparsity ? Generalized sampling 2. How to select a sampling operator ? Infinite sequence Finite dimensional elements Random Operator Continuous signal 3. Can we introduce stucture in sampling and still preserve stability ? is stable w. h. p Stability Randomness Infinitely many Need structure for efficient implementation 4. How to solve infinite dimensional recovery problems ? Reconstruction Finite program, well-studied Undefined program over a continuous signal 16

A step backward Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than “Success has many fathers …” Whittaker 1915 Nyquist 1928 Kotelnikov Shannon 1933 1949 17

A step backward Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than Fundamental ingredients of a sampling theorm • A signal model • A minimal rate requirement • Explicit sampling and reconstruction stages 18

Discrete Compressed Sensing Analog Compressive Sampling 19

Analog Compressed Sensing What is the definition of analog sparsity ? A signal with a multiband structure in some basis 1. Each band has an uncountable number of non-zero elements 2. Band locations lie on an infinite grid 3. Band locations are unknown in advance no more than N bands, max width B, bandlimited to (Mishali and Eldar 2007) 20

Multi-Band Sensing: Goals bands Analog Sampling Infinite Reconstruction Analog Goal: Perfect reconstruction Constraints: 1. Minimal sampling rate What is the minimal rate ? What is the sensing mechanism ? 2. Fully blind system How to reconstruct from infinite sequences ? 21

Rate Requirement Theorem (non-blind recovery) Landau (1967) Subspace scenarios: Average sampling rate Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) Half blind system (Herley and Wong 99, Venkataramani and Bresler 00) 22

Sampling Multi-Coset: Periodic Non-uniform on the Nyquist grid In each block of samples, only are kept, as described by 2 3 Analog signal Point-wise samples 0 0 3 2 0 2 3 Bresler et. al. (96, 98, 00, 01) 23

The Sampler DTFT of sampling sequences Length. known Problems: in vector form unknowns Constant Observation: is sparse matrix known 1. Undetermined system – non unique solution is jointly sparse and unique under appropriate parameter selection ( 2. Continuous set of linear systems ) 24

Paradigm Solve finite problem Reconstruct 0 S = non-zero rows 1 2 3 4 5 6 25

Continuous to Finite CTF block MMV Solve finite problem Reconstruct span a finite space Any basis preserves the sparsity Continuous Finite 26

Algorithm CTF Continuous-to-finite block: Compressed sensing for analog signals Perfect reconstruction at minimal rate Blind system: band locations are unkown Can be applied to CS of general analog signals Works with other sampling techniques 27

Blind reconstruction flow Multi-coset with SBR 4 Yes CTF Universal No No SBR 2 Bi-section CTF Yes Uniform at Spectrum-blind Sampling Ideal low-pass filter Spectrum-blind Reconstruction 28

Final reconstruction (non-blind( Bresler et. al. (96, 00) 29

Framework: Analog Compressed Sensing Sampling signals from a union of shift-invariant spaces (SI) Subspace generators 30

Framework: Analog Compressed Sensing What happen if only K<

Framework: Analog Compressed Sensing Step 1: Compress the sampling sequences Step 2: “Push” all operators to analog domain CTF System A High sampling rate = m/T Only k sequences are non-zero Post-compression 32

Framework: Analog Compressed Sensing System B Low sampling rate = p/T Pre-compression CTF Theorem Eldar (2008) 33

Does it work ? 34

Simulations Minimal rate Sampling rate Brute-Force Minimal rate Sampling rate M-OMP 35

Simulations (2) 0% Recovery 100% Recovery Noise-free Sampling rate SBR 4 SBR 2 Empirical recovery rate 36

Simulations (3) Reconstruction filter Amplitude Signal Time (n. Secs) Output Time (n. Secs) 37

Break (10 min. please) 38

Uncertainty principles • • Coherence and the discrete uncertainty principle Analog coherence and principles Achieving the lower coherence bound Uncertainty principles and sparse representations 39

The discrete uncertainty principle Uncertainty principle 40

Discrete coherence Which bases achieve the lowest coherence ? 41

Discrete coherence Which signal achieves the uncertainty bound ? Spikes Fourier 42

Discrete to analog • Shift invariant spaces • Sparse representations Questions: • What is the analog uncertainty principle ? • Which bases has the lowest coherence ? • Which signal achieves the lower uncertainty bound ? 43

Analog uncertainty principle Theorem Eldar (2008) 44

Bases with minimal coherence In the DFT domain Fourier Spikes What are the analog counterparts ? • Constant magnitude • Modulation • “Single” component • Shifts 45

Bases with minimal coherence In the frequency domain 46

Tightness 47

Sparse representations • In discrete setting 48

Sparse representations • Analog counterparts Undefined program ! But, can be transformed into an IMV model 49

Discussion • IMV model as a fundamental tool for treating sparse analog signals • Should quantify the DSP complexity of the CTF block • Compare approach with the “analog” model • Building blocks of analog CS framework. 50

Thank you 51