2015年新书：Machine Learning A Bayesian and Optimization Perspective.pdf

【实例简介】
亚马逊很赞的5颗星星。好坏自己想吧。
This page intentionally left blank Machine Learning A Bayesian nd optimization erspective Sergios Theodoridis AMSTERDAM· BOSTON· HEIDELBERG· LONDON NEW YORK· OXFORD· PARIS· SAN DIEGO SAN FRANCISCO· SINGAPORE· SYDNEY· TOKYO ELSEVIER cademic Press is an imprint of Elsevie Academic Press is an imprint of Elsevier 125 London Wall. London EC2Y 5AS UK 525 B Street, Suite 1800, San Diego, CA 92101-4495, USA 225 Wyman Street, Waltham, MA 02451, USA The boulevard, Langford Lane, Kidlington, Oxford OX5 IGB, UK Copyright@ 2015 Elsevier Ltd. All rights reserved No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, withou permission in writing from the publisher. Details on how to seek permission, further information about the Publisher's permissions policies and our arrangements with organizations such as the copyright clearance center andtheCopyrightLicensingAgency,canbefoundatourwebsitewww.elsevier.com/permissions This book and the individual contributions contained in it are protected under copyright by the Publisher Cother than as may be noted herein) Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary Practitioners and researchers must always rely on their own experience and know ledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility To the fullest extent of the law, neither the Publisher nor the authors contributors or editors assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the library of Congress ISBN:978-0-12-801522-3 For information on all academic Press publications visitourwebsiteathttp://store.elseviercom/ Publisher: Jonathan Simpson Acquisition Editor: Tim Pitts Editorial Project Manager: Charlie Kent Production Project Manager.. Susan Li Designer: Greg harris Typeset by SPi Global, India Printed and bound in The United States 1516171819 10987654321 影 Working together to grow libraries in ELSEVIER Book Aid international developing countries www.elsevier.com.www.bookaid.org Contents Preface ,,,,,,,,,,,,,,,,,,.,,,,,,,,XV11 Acknowledgments... X Notation XXI CHAPTER 1 Introduction 1.1 What Machine Learning is About 1. 1.1 Classification 2 1.2 Regression 1.2 Structure and a road Map of the book References∴… 3589 CHAPTER 2 Probability and Stochastic Processes 2.1 Introduction 2.2 Probability and Random Variables 2. 2.1 Probability 2.2.2 Discrete random variables 12 2.2.3 Continuous Random Variables 2.2. 4 Mean and variance 2.2.5 Transformation of Random variables 2.3 Examples of Distributions....... 18 2.3.1 Discrete Variables 18 2.3.2 Continuous variables 20 2.4 Stochastic Processes 29 2. 4.1 First and second order statistics 30 2.4.2 Stationarity and Ergodicity 30 2.4.3 Power Spectral Density 33 2.4.4 Autoregressive Models 38 2.5 Information Theor 41 2.5.1 Discrete random variables 42 2.5.2 Continuous random variables 45 2.6 Stochastic Convergence 48 Problems 49 References 51 CHAPTER 3 Learning in Parametric Modeling: Basic concepts and directions 53 3. 1 Introductio 53 3.2 Parameter Estimation: The Deterministic point of view 54 Contents 3.3 Linear regression 57 3. 4 Classification 60 3.5 Biased Versus Unbiased Estimation 64 3.5.1 Biased or Unbiased Estimation? 65 3. 6 The Cramer -Rao Lower bound 67 3.7 Sufficient Statistic 70 3.8 Regularization 72 3.9 The Bias-Variance Dilemma 77 3.9.1 Mean-Square error estimation 77 3.9.2 Bias-Variance Tradeoff 3.10 Maximum Likelihood Method 82 3.10.1 Linear Regression: The Nonwhite Gaussian Noise Case 84 3. 11 Bayesian Inference 84 3.11.1 The Maximum a Posteriori Probability Estimation Method 88 3.12 Curse of Dimensionality 89 3. 13 Validation 3. 14 Expected and Empirical Loss functions 3. 15 Nonparametric Modeling and estimation Problems References 102 CHAPTER 4 Mean-Square Error Linear Estimation 105 4.1 Introduction ··········· ..105 4.2 Mean-Square Error Linear Estimation: The Normal Equations 106 4.2.1 The Cost function surface ..107 4.3 A Geometric Viewpoint: Orthogonality Condition 109 4.4 Extension to Complex-Valued Variables l11 4.4.1 Widely linear complex-Valued estimation 113 4.4.2 Optimizing with Respect to Complex-Valued Variables Wirtinger Calculus 4.5 Linear Filtering 4.6 MSE Linear Filtering: A Frequency Domain Point of View .120 4.7 Some Typical Applications ..124 4.7.1 Interference Cancellation 124 4.7.2 System Identification 4.7.3 Deconvolution: Channel Equalization.,. ·······;······ 12 126 4.8 Algorithmic Aspects: The Levinson and the Lattice-Ladder Algorithms 132 4.8.1 The Lattice-Ladder scheme 137 4.9 Mean-Square Error Estimation of Linear Models ····.··.·· 140 4.9.1 The Gauss-Markov Theorem 143 4.9.2 Constrained Linear estimation: The Beamforming case 145 Contents 4.10 Time-Varying Statistics: Kalman Filtering ...148 Problems 154 References ..158 CHAPTER 5 Stochastic Gradient Descent: The LMs Algorithm and its Family 161 5.1 Introduction .162 5.2 The Steepest Descent Method 163 5.3 Application to the Mean-Square Error Cost Function 167 5.3. 1 The Complex-Valued Case 175 5.4 Stochastic Approximation ..... .177 5.5 The Least-Mean-Squares adaptive algorithm 179 5.5.1 Convergence and Steady-State Performance of the lMs in Stationary Environments 181 5.5.2 Cumulative Loss bounds 186 5. 6 The Affine Projection Algorithm ..188 5.6.1 The Normalized LMs 193 5.7 The Complex-Valued Case ..194 5.8 Relatives of the Lms 196 5. 9 Simulation Examples ..199 5. 10 Adaptive decision Feedback equalization 202 5.11 The Linearly Constrained LMS 5.12 Tracking Performance of the LMS in Nonstationary Environments 206 5.13 Distributed Learning: The Distributed LMS 208 5.13.1 Cooperation Strategies ..209 5.13.2 The Diffusion LMs 211 5.13.3 Convergence and Steady-State Performance Some highlights ∴218 5.13. 4 Consensus-Based Distributed Schemes 220 5.14 A Case Study: Target Localization 222 5.15 Some Concluding Remarks: Consensus matrix.……,…,…,….223 Problems 224 References 227 CHAPTER 6 The Least-Squares Family 233 6.1 Introduction 234 6.2 Least-Squares Linear Regression: A Geometric Perspective 234 6.3 Statistical Properties of the LS Estimator 236 6.4 Orthogonalizing the Column Space of X: The SVD Method 239 6.5 Ridge regression 243 6.6 The Recursive Least-Squares algorithm 245 viii Contents 6.7 Newton's Iterative Minimization method 248 6.7.1 RLS and Newtons method 251 6.8 Steady-State Performance of the RLS 69 Complex- Valued data: The widely Linear RLi……… 52 254 6. 10 Computational Aspects of the Ls Solution 5 6. 11 The Coordinate and Cyclic Coordinate Descent Methods ............ 258 6.12 Simulation Examples……… 259 6.13 Total-Least-Squares 261 Problems 268 References 272 ChaPTeR Classification: a tour of the classics 275 7.1 Introduction ··· 275 7.2 Bayesian Classification 276 7.2.1 Average risk ········· 278 7.3 Decision(Hyper)Surfaces .280 7. 3.1 The Gaussian distribution Case 282 7.4 The Naive Bayes Classifie 287 7. 5 The Nearest Neighbor rule 288 7.6 Logistic Regression 290 7.7 Fishers Linear discriminant ········ 294 7.8 Classification Trees 300 7.9 Combining Classifiers ············ ..304 7.10 The Boosting Approach 307 7. 11 Boosting Trees ···· ..313 7.12 A Case Study: Protein Folding Prediction ..314 Probl 318 References 323 CHAPTER 8 Parameter Learning: A Convex Analytic Path ..327 8. 1 Introduction 328 8.2 Convex Sets and Functions 329 8.2.1 Convex Sets 329 8.2.2 Convex Functions ········· 330 8.3 Projections onto Convex Sets .··········· 333 8.3.1 Properties of Projections 337 8.4 Fundamental Theorem of Projections onto Convex Sets ..341 8.5 A Parallel Version of pocs ··· 344 8. 6 From Convex sets to parameter Estimation and Machine Learnin 345 8.6.1 Regression ..345 8.6.2 Classification .··········· 347 Contents X 8.7 Infinite Many Closed Convex Sets: The Online Learning Case......... 349 8.7.1 Convergence of APSM 351 8.8 Constrained Learning 356 8.9 The Distributed APSM ..357 8.10 Optimizing Nonsmooth Convex Cost Functions 358 8.10.1 Subgradients and subdifferentials 359 8.10.2 Minimizing Nonsmooth Continuous Convex Loss Functions The batch Learning Case 362 8.10.3 Online Learning for Convex Optimization 367 8.11 Regret Analysis ..370 8.12 Online Learning and Big Data Applications: A Discussion. ........... 374 8.13 Proximal Operators 379 8.13. 1 Properties of the Proximal Operator 382 8.13.2 Proximal minimization .383 8. 14 Proximal Splitting methods for Optimization 385 Problems 389 8. 15 Appendix to chapter 8 ...393 Referenc 398 CHAPTER 9 Sparsity-Aware Learning: Concepts and theoretical foundations 403 9. 1 Introduction ..403 .2 Searching for a 404 9.3 The Least Absolute Shrinkage and Selection Operator LASSO) ..407 9. 4 Sparse signal Representation 411 9.5 In Search of the Sparsest Solution 415 9.6 Uniqueness of the lo minimizer 422 9.6.1 Mutual Coherence 424 9.7 Equivalence of lo and eI Minimizers: Sufficiency Conditions 426 9.7.1 Condition Implied by the mutual Coherence Number 426 9. 7. 2 The Restricted Isometry Property(rIP) 427 9.8 Robust Sparse Signal Recovery from Noisy Measurements ...429 9.9 Compressed Sensing: The Glory of Randomness 430 9.9.1Dim ity Reduction and stable Embeddings 433 9.9.2 Sub-Nyquist Sampling: Analog-to-Information Conversion 434 9.10 A Case Study: Image De-Noising ...438 Problems 440 References 44 CHAPTER 10 Sparsity-Aware Learning: Algorithms and applications 449 10.1 Introduction 450 10.2 Sparsity-Promoting Algorithms 450 【实例截图】
【核心代码】