北大林宙辰：机器学习一阶算法的优化

【实例简介】
2016中国人工智能大会（CCAI 2016），机器学习的明天论坛，北京大学信息科学技术学院机器感知与智能教育部重点实验室教授林宙辰的演讲PPT。报道：http://geek.csdn.net/news/detail/97844
Nonlinear Optimization CCAI 2016 中国人工智能大会 Past (-1990s Major theories and techniques of optimization completed 1960s 1960s-1990s boom due to the invention of computers · Zeroth order methods Interpolation Methods Pattern Search Methods First order methods Coordinate descent Conjugate gradient (Stochastic) Gradient/Subgradient Descent Ellipsoid method Quasi-Newton Methods (Augmented)Lagrangian Method of multipliers Nonlinear Optimization CCAI 2016 中国人工智能大会 Past (-1990s Second order methods Newton's methods Sequential Quadratic Programming Interior point methods ·。 Nonlinear Optimization CCAI 2016 中国人工智能大会 Why first order methods? Converge relatively fast #iterations) Acceptable accuracy for machine learning Relatively cheap in storage and computation( complexity in each iteration) Important for big data era mIST nmAPG 400 800 1000 Iteration (a)Objective function value v.S. iteration Nonlinear Optimization CCAI 2016 中国人工智能大会 Present(1990s-now) Revive and refine of existing techniques Support vectors Spiral ascent Support vectors Application driven Support Vector Machines ° Deep Learning 32 Feature(F)maps 36X36 32 F. maps 18x18 8 F maps 32 F maps 32 F maps ° Big data 14X14 Gray scale 18x18 7X7 8 F maps 200 nodes 40X40 Image 7X7 40X40 55 5x5 2x2 C MP Fully connected(F) Convolution(C) Max pooling(MP) Max out(MO) Dropout(D) CCAI 2016 Present(1990s-now) 中国人工智能大会 Advances in first-order methods Smooth - nonsmooth · Convex-> Nonconvex Deterministic - Stochastic One/Two blocks-> Multiple blocks Synchronous-> Asynchronous Convergence Convergence rate CCAI 2016 Present(1990s-now) 中国人工智能大会 Smooth -> nonsmooth Sparsity LoW-Ran kness Smooth · Nonsmooth Gradient Subgradient Proximal Operator 0∈b(xk) ACrlk) minx f(x)+E/2∥x y∥2 Linearization c(x)≤9(xMk(g(xh),x-xM片D/2∥x h∥72 CCAI 2016 Present(1990s-now) 中国人工智能大会 Convex-> Nonconvex Nonconvex Convex Nonincreasing Objective Global optimal Cluster points are kkt points Converges to KKt point (Kurdyka-Lojasiewicz Condition) (x) y7((x)-f(x1米)).(0,0(x)≥1 CCAI 2016 Present(1990s-now) 中国人工智能大会 Deterministic-> Stochastic Deterministic · Stochastic fClk), f(xlk) Stochastic Gradient/ADMM 1m∑=11nl(x)→f kr(x Variance Reduction Stochastic Matrix Computation Randomized svd 【实例截图】
【核心代码】