陈天奇xgboost PPT

【实例简介】
陈天奇xgboost PPT.pdf
Elements in Supervised Learning o NotationS: i ra i-th training example Model: how to make prediction i given Linear model:!y=∑ wjNij(include linear/logistic regression The prediction score i can have different interpretations depending on the task Linear regression i is the predicted score Logistic regression: 1/(1+exp(yi))is predicted the probability of the instance being positive Others.for example in ranking i can be the rank score Parameters: the things we need to learn from data Linear model: 0= wilj=1, .,, d Elements continued: Objective Function Objective function that is everywhere Obi()=I(e)+9(6) Loss on training data: L=I(yi, ii) Square loss: l(9i, ii)=(yi-yi)2 Logistic loss: l(yi, gi)=i In(1+eyi)+(1-yi)ln(1+eyi) Regularization how complicated the model is? L2norm:92(v)=划ol|2 L1 norm(lasso): Q(w)=Xlwll 1 Putting known knowledge into context Ridge regression: >,(i -WTCi)2+All Linear model, square loss l2 regularization Lasso:∑ i=1(92 1x)2+l1 Linear model, square loss, L1 regularization o Logistic regression ∑=1{y;ln(1+e-0)+(1-y)ln(1+e)+川2 Linear model, logistic loss, L2 regularization o The conceptual separation between model, parameter objective also gives you engineering benefits Think of how you can implement Sgd for both ridge regression and logistic regression Objective and Bias variance Trade-off Obj(⊙)=L(Q)+(e o Why do we want to contain two component in the objective? o Optimizing training loss encourages predictive models Fitting well in training data at least get you close to training data which is hopefully close to the underlying distribution Optimizing regularization encourages simple models Simpler models tends to have smaller variance in future predictions, making prediction stable Outline o Review of key concepts of supervised learning Regression Tree and Ensemble(what are we learning o Gradient Boosting(How do we Learn ● Summary Regression Tree(CART) regression tree (also known as classification and regression tree) Decision rules same as in decision tree Contains one score in each leaf value Input: age, gender, occupation Regression Tree Ensemble Tree Ensemble methods Very widely used, look for gBM, random forest Almost half of data mining competition are won by using some variants of tree ensemble methods Invariant to scaling of inputs so you do not need to do careful features normalizatⅰon Learn higher order interaction between features Can be scalable, and are used in industry 【实例截图】
【核心代码】