PPT-Paired Experiments and Interleaving for

Retrieval Evaluation Thorsten Joachims Madhu Kurup Filip Radlinski Department of Computer Science Department of Information Science Cornell University Decide between two Ranking Functions. The tradeoff between internal and external validity. Natural sciences. In natural sciences, most experiments have been undertaken in carefully controlled laboratory conditions. Seeking laws of physics, etc.. Gary Simpson. University of the West of Scotland. Fast-timing from below isomers ion-. g-g. Use selectivity of spectrometer + isomer decay. Know what isomer is. . 1. . 2. T. 1/2. . start. stop. Experimentation . Experiments and Torture. Some of the information is taken from Nazi Human Experimentation. Human Experimentation . Experiments on twins. Bone, muscle, limb and nerve transplant . Freezing experiments. . True Experiments. Characteristics. Threats to validity controlled by experiments. Threats not controlled by experiments. Obstacles to true experiments in the field. . Quasi-experiments. The logic of quasi-experiments . Unconscious Behavior. Criteria: . Real problem. Relevant to food (health). Actionable. Best method. Best journal. Method: . Pediatrician office flu/cold patients. 2x2: . teasp. vs. . tblsp. ; kid vs. no kid. Paired Difference in Means. Outline. Paired data. Confidence interval for difference in means based on paired data. Hypothesis test for difference in means based on paired data. Paired Data. Data . are . Testing Dimensions. Studying attributes of items in a . cultural domain. Free listing is a way to define the contents of a . domain. . Pile . sorts and triad tests are ways to measure . similarity. . Through Online Experimentation. WSDM Workshop on Web Search Click Data. February 12. th. , 2012. Yisong Yue. Carnegie Mellon University. Offline Post-hoc Analysis. Launch some ranking function on live traffic. MOOSE LASER. OFFICE FRAME. PHONE LICENSE. CLINIC EDITOR. METER CHART. Etc., etc.., etc.. List of Paired-Associates to be learned:. MOOSE. Response Interval. :. What word is associated with “moose”?. Prepared by Adam Bacchus. Agenda. Research findings on Spaced Practice. Research findings on Interleaving. Implementing interleaving strategy. Activity. Remember this?. Testing yourself on a regular basis, saves you from having to do any special test preparation.. Ronny Kohavi, Microsoft. Slides available at . http://exp-platform.com. Part 1. 2. Amazon . Shopping Cart . Recs. Add an item to your shopping cart at a website. Most sites show the cart. At Amazon, Greg Linden had the idea of showing recommendations based on cart items. . SYFTET. Göteborgs universitet ska skapa en modern, lättanvänd och . effektiv webbmiljö med fokus på användarnas förväntningar.. 1. ETT UNIVERSITET – EN GEMENSAM WEBB. Innehåll som är intressant för de prioriterade målgrupperna samlas på ett ställe till exempel:. to complete programming Figure 2 ! Margherita. . Polacci. F. . Arzilli, G. La Spina, N. Le . Gall. , . R. . Torres-. Orozco. , M. E. . Hartley. , D. Di Genova, . R. . C. . Atwood. , E. . W. . . Llewellin. , . R. . . Brooker. , H. M. .