北大暑期课程《回归分析》Linear-Regression-Analysis讲义PKU5

Class5:ANOVA Analysisof Varianceand F-testsWha.i.ANOVA.ANOV.i.th.shor.nam.fo.th.Analysi.o.Variance.Th.essenc.o.ANOV.i.t.decompos.th.tota.varianc.o.th.dependen.variabl.int.tw.additiv.components.on.fo.th.structura.part.an.th.othe.fo.th.stochasti.part.o..regression.Toda,w.ar.goin.t.examin.th.easies.case.II.ANOVA.A.IntroductionLet the model be¥=xg+£.Assuming xis acolumn vectorof lengthp ofindependent variablevalues for the/thiobservation,y,.=X/2+The.,i.th.predicte.value.sum ofsquares total:=L-SST Z可=Z[y，-sb+x,也-讨=Z[y,-F+zkjD-YF+zZNiTjblhA Y足=Z[e/+ZL可becaus.D.This isalways trueby OLS.=SSE+SSRImportant:the totalvariance ofthe dependent variable isdecomposed intotwo additiveparts:SSE,which isdue toerrors,and SSR,which isdue toregression.Geometric interpretation:[blackboard]Decomposition ofVarianceIf wetreat Xas arandom variable,we candecompose totalvariance tothe between-group portionandthe within-group portionin anypopulation:v=vL®+v®Prove:VS=VR+J；；=Vx^+V^.+2Covx A£J=VR+V£jby theassumption thatCov/,e=0,for allpossible k.Th.ANOV.tabi,i.t.estimat.th.thre.quantitie.o.equatio.l.fro.th.sample.A.th.sampl.siz.get.large.an.larger.th.ANOV.tabi,wil.approac.th.equatio.close.an.closer.L.sample.decompositio.o.estimate,varianc.i.no.strictl.true.W.thu.nee.t.separatel.decompos.sum.o.square.an.degree.o.freedom.I.ANOV..misnomer.HLANOV.MatrixE[y-vSST二I willtry togive asimplied representation of ANOVAas follows:__2__=Zy+n干-2nY because^y=nYz一2-nY—2=yy—nY=y*y-1/n yJy in your textbook,monster lookSSE=eeSSR=^[x；b-Y]2=EU,b2+Y2-2x bYz=Ek，b2]+nY2-2Y^x bz=Z h»]+nV22Z y，•e,一千一=Ek,b2]+nY2-2nY2because^y.=nY,^e=0,as alwayszz=Ek,b2]-nV—2bXXb—nY二=bXy-l/n y*Jyinyourtextbook,monster lookSOURCESS DFMS FwithRegression SSRDFR MSRMSR/MSE（）DF RErrorSSE DFEMSE DFETotalSST DFTLe.u.us..rea.example.Assum.tha.w.hav..regressio.estimate.t.b.y=-

1.70+

0.840xANOVA TableSOURCESS DFMS FwithRegression

6.

4416.

446.44/

0.19=

33.891,18Error

3.

40180.19Total

9.8419W.kno.Lw.kno.tha.D.fo.SST=

19.wha.i.nn=207=50/20=

2.5SST=£y j-nY2=

134.84-20x

2.5x

2.5=

9.84SSR=Xl-

1.7+

0.84X2]-

125.0/=・7x1・7+

0.84x

0.84x2-2xl.7x

0.84xxj-l

25.0z=20x

1.7x

1.7+

0.84x

0.84x

509.12-2x

1.7x

0.84x100-

125.0=

6.44SSE=SST-SSR=

9.84-

6.44=

3.40D.Degree.o.freedom.demonstration.Note.discountin.th.intercep.whe.calculatin.SST.MS=SS/DF..

0.

00.[as.students].Wha.doe.th.p-valu.say.V.F-TestsF-test.ar.mor.genera.tha.t-tests.t-test.ca.b.see.a..specia.cas.o.F-tests.Lyo.hav.difficult.wit.F-tests.pleas.as.you.GSI.t.revie.F-test.i.th.lab.F-test.take.th.for.o..fractio.o.tw.MSs.扪F=MSR/MSE//7A..statisti.ha.tw.degree.o.freedo.associate,wit.it.th.degre.o.freedo.i.th.numerator.an.th.degre.o.freedo.i.th.denominator.A..statisti.i.usuall.large.tha.l.Th.interpretatio.o.a..statistic.i.tha.whethe.th.explaine.varianc.b.th.alternativ.hypothesi.i.du.t.c hance.I.othe.words.th.nul.hypothesi.i.tha.th.explaine.varianc.i.du.t.chance.o.al.th.coefficient.ar.zero.Th.large.a.F-statistic.th.mor.likel.tha.th.nul.hypothesiJ.no.true.Then.i..tabi.i.th.bac.o.you.boo.fro.whic.yo.ca.fin.exac.probabilit.values.In ourexample,the Fis34,which ishighly significant.VLR2R2=SSR/SSTThe proportion of varianceexplained by themodel.In ourexample,R-sq=

65.4%・w・iec「eas,mo「・independen.variables,l.SS.stay.th.same.

1.

55.alway.increases.

1.

56.alway.decreases.

4.R.alway.increases.

5.MS.usuall.increases.

6.MS.usuall.decreases.

7.F-tes.usuall.increases.Exception.t..an.

7.irrelevan.variable,ma.no.explai.th.varianc.bu.tak.u.degree,o.freedom.W.reall.nee.t.loo.a.th.results.VIII.Im vortant.Genera.Way.o.Hypothesi.Testis.wit.F-Stdtistics.・Al.test.i.linea.regressio.ca.b.performe.wit.F-tes.statistics.Th.trie,i t.ru.”neste.models.”Tw.model.ar.neste.i.th.independen.variable,i.on.mode.ar..subse.o.linea.combination.o..subse.子集o.th.independen.variable.

1.th.othe.model.Tha.i.t.say.

1.mode..ha.independen.variable.an.mode..ha.independen.variable..an..ar.nested.J.calle.th.restricte.model..!.calle.les.restricte.o.unrestricte.model.W.cal..restricte.becaus..implie.tha.n.Thi.i..restriction.Anothe.example..ha.independen.variabl.l.n.n+n..ha,l.n+n..an..ar.no.nested..an..ar.nested.On.restrictio.i.C....an..ar.nested.On.restrictio.i.D.□..an..ar.no.nested.D andB arenested:two restrictionin D:，2=尸3；〃/=・W.ca.alway.tes.hypothese.implie.i.th.restricte.models.Steps,ru.tw.regressio.fo.eac.hypothesis.on.fo.th.restricte.mode.an.on.fo.th.unrestricte.model.Th.SS.shoul.b.th.sam.acros.th.tw.models.Wha.i.differen.i.SS.an.SSR.Tha.is.wha.i.differen.i.R

2.Letdf〃=dfSSE“,df,.=dfSSEj；〃〃=_p“_5_p,.=p,_p0Use thefollowing formulas:=SSE,-SSE/df SSE,.-df SSE“SSE〃/df〃mw-orSSR〃-SSR J/df SSRJ-df SSRJ二颂-烟，奶―SSE〃/df〃proof:use SST=SSE+SSRNote,dfSSEr-dfSSEy=dfSSRy-dfSSRr=Adf,is thenumber ofconstraints notnumber ofparameters impliedby therestricted modelor=R2〃-R1/Adf]_2/dfR\dfr-dfu\dfu~~Note thatTha.is.fo.ld.tests.yo.ca.eithe.d.a.F-tes.o..t-test.The.yiel.th.sam.result.Anothe.wa.t.loo.a.i.i.tha.th.t-tes.i..specia.cas.o.th..test.wit.th.numerate.D.bein.l.What assumptionsdo weneed tomake anANOVA tableworkNo.muc.a.assumption.Al.w.nee.i.th.assumptio.tha.XX.i.no.singular.s.tha.th.leas.squar.estimat..exists.Th.assumptio.o.n=.i.neede.i.yo.wan.th.ANOV.tabl.t.b.a.unbiase.estimat.o.th.tru.ANOV.equatio.l.i.th.population.Reason,w.wan.,t.b,a.unbiase.estimate.o.D.an.th.covarianc.betwee..and Dt.disappear.Fo.reason..discusse.earlier.th.assumption.o.homoscedasticit.an.non-seria.correlatio.ar.necessar.fo.th.estimatio.o.n.Th.normalit.assumptio.tha..i.distribute.

1.,norma.distributio.i.neede.fo.smal.samples.・IncremeritEver.tim.yo.pu.on.mor.independen.variabl.int.you.model.yo.ge.a.increas.i.n.W.sometim.calle.th.inc^eas.,incrementalWha.i.mean,i.tha.mor.varianc.i.explained.o.SS.i.increased.SS.i.reduced.Wha.yo.shoul.understan.i.tha.th.incrementa..attribute.t..variabl.i.alway.smalle.tha.the.whe.ot he.variable.ar.absent.XLConseauerce.o.Omittin.Relevan.Irideperderi.VariablesSay thetrue modelis thefollowing:夕+Bl%M=0++63%+Bu.fo.som.reaso.w.onl.collec.o.conside.dat.o.n.Therefore.w.omi.Di.th.regression.Tha.is.w.omi.in.ou.model.W.briefl.discusse.thi.proble.before.Th.shor.stor.i.tha.w.ar.likel.t.hav..bia.du.t.th.omissio.o..relevan,variabl.i.th.model.Thi.i.s.eve.thoug.ou.primar.interes.i.t.estimat.th.effec.o.no..o.y.Why Wewill havea formalpresentationofthis problem.XH.Measure.o.Goodness-of-FitThen.ar.differen.way.t.asses.th.goodness-of-fi.o..model.A.R2R.i..heuristi.measur.fo.th.overal.goodness-of-fit.Ldoe.no.hav.a.associate.tes.statistic.R2measures theproportionofthe variancein thedependentvariablethat isexplained”bythemodel:z SSRSSRR2=-------=------------------SST SSR+SSEB.Mode.F-tes.Th.mode.F-tes.test.th.join.hypothese.tha.al.th.mode.coefficient.excep.fo.th.constan.ter.ar.zero.Degrees of freedoms associatedwith themodel F-test:Numerator:p-1Denominator.n-p.C.t-test.fo.individua.parameters.t-tes.fo.a.individua.paramete.test.th.hypothesi.tha..particula.coefficien.i.equa.t..particula.numbe.commonl.zero..t..bk.kO/SEk.wher.SEki.th.k.k.elemen.o.MSEX,X-l.wit.degre.o.freedom=n-p.Relative toa restrictedmodel,the gainin R2fortheunrestricted model:AR2=RU2-Rr2E.F-test.fo.Neste.Mode.I.i.th.mos.genera.for.o.F-test.an.t-tests.SSE,SSE〃/df SSEJ-df SSE〃二「SSE〃/df〃dfu-dfr\dfu_I.i.equa.t..t-tes.i.th.unrestricte.an.restricte.model.diffe.onl.b.on.singl.parameter.Li.equa.t.th,mode.F-tes.i.w.se.th.restricte.mode.t.th.constant-onl.model.[Ask students]What areSST,SSE,and SSR,and theirassociated degreesoffreedom,for theconstant-only modelNumericalExample.sociologica.stud.i.interested,understandin.th.socia.determinant,o.mathematica.achievemen.amon.hig.schoo.students.Yo.ar.no.aske.t.answe..serie.©.questions.Th.dat.ar.rea.bu.hav.bee.tailore.fo.educationa.purposes.Th.tota.numbe.o.observation.i.

400.Th.variable,ar.define.as:y:math scorexl:fathers educationx2:mothers educationx3:familys socioeconomicstatus x4:number ofsiblingsx5:class rankx6:parents totaleducation note:x6=xl+x2For thefollowing regressionmodels,weknow:SST SSRSSE4201DF FV1y on1xl x2x3x4348632y on1x6x3x

4348631042624437396.10653y on1x6x3x4x

534863269753395.29914x5on1x6x3x

4396.0210Table

11.Pleas.fil.th.missin.cell.i.Tabi.

1.

2.Tes.th.hypothesi.tha.th.effect,o.father.educatio.xl.an.mother.educatio.x

2.o.mat.scor.ar.th.sam.afte.controllin.fo.x.an.x

4.

3.Tes.th.hypothesi.tha.x

6.x.an.x.i.Mode.

2.al.hav..zer.effec.o.y.

4.Ca.w.ad.x.t.Mode.l.Briefl.explai.you.answer.

5.Tes.th.hypothesi.tha.th.effec.o.clas.ran.x

5.o.mat.scor.i.zer.afte.controllin.fo.x

6.x

3.an.x

4.Answer:SST SSRSSE DFR21y on1xl x2x3x

434863420130662395.12052y on1x6x3x

434863371331150396.10653y on1x6x3x4x

5348631042624437395.29914x5on1x6x3x

42755395786269753396.0210Note thatthe SSTfor Model4is differentfrom thosefor Models1through

3.

1.Restricted modelis y=b+b x+x+b x+b x+e0l123344Unrestricted modelis y=+b x+b x+Z x+Z x+ex x22334431150-30662/1Fi,395=----------------------------=488/

77.63=

6.2930662/

3952.3713/3F396=-------------------------=

1237.67/

78.66=

15.733Z31150/

3963.No.x.i..linea.combinatio.o.x.an.x

2.X.i.singular.

4.31150-24437/1Fi,395=----------------------------=6713/

61.87=

108.5024437/395t===

7108.50=

10.42Z。