崇志宏：强化学习和深度强化学习

【实例简介】
深度学习在强化学习方面的应用所产生的深度强化学习取得快速发展。如何解释深度强化学习优势产生的原因是理解技术的基本方法。
强化学习的问题 ntel ab 数据&智能实验室 Agent state reward S,||R +=PS=1=8:8:1an R, +1 , Environment 5o,A0,B1,S1,41,,S2,A2,R3 东南大学数据与智能实验室(D& nte lab) 强化学习算法的基本范式 ntel ab 数据&智能实验室 Generalized Policy Iteration 1. Policy Iteration 2. Value iteration evaluation T T U,丌 率 丌~ greedy(V dyu gre improvement 1. Policy Evaluation博弈 2. Policy Improvement 1.策略评价逼近策略价值 2.策略提高远离策略价值 Elements of rl ntel ab 数据&智能实验室 o Policy Valuation Oriented Methods Model-based or free estimation Exp ion or Exploit 0 nl ine or0千f|ine Expans ion and Backup Operation (q*) max nax 东南大学数据与智能实验室(D& Inte lab) Position of rl ntel ab 数据&智能实验室 Long short Term PixeICNN PixeIRNN Memory (LSTM) Convolution Networks with Fully Connected Neural Network Autoregressive Network Memory Network (CNN) (FCN) Stochastic Gradient Second Order Earth Mover Distance Descent (SGD) Algorithms (EMD) Networks <L Divergence Optimization Gradient Free Objective Algorthms Functions Algorithms Supervised Energy Based Model Learning Learning to Learn Unsupervised Learning Reinforcement earning Non-Probabilistic Models Probabilistic (Generative) Models Policy Optimization Dynamic Sparse Coding. Programming Autoencoder K-means Explicit Density Implicit Density Models Models Policy Gradients Policy Iteration Value iteration Derivative Free Optimization/Evolution Generative Adversarial Network (GAN) Moment Matchine Networks Actor Critic Methods -Learning Tractable Models Non-Tractable Models NADE PixeIRNN Boitzmann Machine Fully observed Bele variational Networks Autoencoder (vAE width Temporal-o of update Q Dynamic difference◆ R programming learning O OO O Which rl algorithm to use? are you learning 函数近似 in a simulator 样本效率 depth (length) how patient is simulation cost negigible compared 泛化问题 of update are you? 1to training cost? Explore/ model-based Q-Prop TRPO, PPO RL (GPS, etc DDPG. NAF A3C 目标/可训练性 BUT: if you have a simulator, you can compute gradients through it- do you need model-free rl? Exhaustive Monte I search Carlo 口口口□ 强化学习的主要问题和算法 ntel ab 数据&智能实验室 width of update Temporal Dynamic difference programming learning 基本问题 函数近似 Goal-or iented Try and Error Exploit and Explore 样本效率 depth (length) Extraplot and Generalize 泛化问题 of update 基本算法 Explore/it 动态规划 MC 目标/可训练性 Temperal Difference Q Exhaustive Monte search Carlo Agent 强化学习的问题 statereward S,R Ws,r.=P(S=,=r1-1=84-1= R .. Environment 1. State s)=PS=8=r1S1=84 2. Action ∑p(s,r1s,a)=1, for all s∈S,a∈A(s) 3. Reward s′∈S∈求 p(s|s,a)÷Pr{St=s′|S t-1 A t-1=C p(s, rs ∈求 r(s,a)=E[R:|S-1=s,A-1=a=∑r∑m(s,r|s,a) T∈灾s′∈S r(s,a,s)÷E[Rt|S A p(s,rs, t-1 t-1 t p(ss, a) 东南大学数据与智能实验室(D&| nte Lab) 强化学习的价值估计 G1=Bt+1+R+2+R+3+…+B G=B+1+R+2+7B+3+…=)R+k+1 k=0 Gt÷Rt+1+7Rt+2+2Rt+3+Rt+4+ =R+1+(R+2+R4+3+2R2+4+…) Rt+1 +?Gt+ T R1=+1 R2=+1 R 4 =0 2 R5=0 5 Gt ∑ k-t-1 R k=t+1 Un(s)÷EmG|St=]=E∑Rt+k+1S=s, for all s∈S, k=0 qr(s,a)= EIGt I St=s, At=a]=E>yRt+k+1 St=s, At=a k=0 【实例截图】
【核心代码】