PPT-Preserving Validity in Adaptive Data Analysis

Moritz Hardt IBM Research Almaden Joint work with Cynthia Dwork Vitaly Feldman Toni Pitassi Omer Reingold Aaron Roth Statistical Estimation Data domain X class labels Y Unknown distribution . -Chung Chow. Duke University. USA . 1. Dr. Shein-Chung Chow Biography . Shein-Chung Chow, PhD. is a Professor of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina. Prior to joining Duke University, he was Executive Director of National Clinical Trial Network Coordination Center of Taiwan. Prior to that, Dr. Chow held various management positions in the pharmaceutical industry. Dr. Chow is the Editor-in-Chief of the Journal of Biopharmaceutical Statistics and the Editor-in-Chief of the Biostatistics Book Series at Chapman and Hall/CRC Press of Taylor & Francis Group. He was elected Fellow of the American Statistical Association in 1995. He was the recipient of the DIA Outstanding Service Award (1996), and ICSA Extraordinary Achievement Award (1996). Dr. Chow is the author or co-author of over 200 methodology papers and 20 books, which include Design and Analysis of Bioavailability and Bioequivalence Studies, Design and Analysis of Clinical Trials, Sample Size Calculations in Clinical Research, and Adaptive Design Methods in Clinical Trials. Environmental issues. Naturally occurring variables: usual with several groups of people. Noise. Temperature. Light. Time of day. Seating arrangements. Artificiality:. Classroom or Office are not “real world” or “natural”. Teaching-as-Research Capstone Seminar. Survey, Data Collection, Instruments and Questionnaires. Director, . EERC. Associate Professor and Fulton C. . Noss. Faculty . Fellow. Department of Industrial . Research Data Management (RDM). Expert Centre Research Data. INTRODUCTION. Welcome. Structure of the course: focus on data life cycle . . Last week. : - Creating data, processing data. . In between. Danica Adams. Adaptive capacity. is one way to measure . resilience. .. Adaptive capacity. . is . the ability of a system . – in this case, Vermilion Parish as a political unit – . to adapt to changing circumstances while still retaining its fundamental functions as a system. . Vitaly Feldman. Accelerated Discovery Lab. IBM Research - . Almaden. . Cynthia . Dwork. Moritz . Hardt. Toni . Pitassi. Omer . Reingold. . Aaron Roth. Microsoft Res. Google Res. U. of Toronto Samsung Res. General Tools for Post-Selection Inference. Aaron Roth. What do we want to protect against?. Over-fitting from fixed algorithmic procedures (easiest – might hope to analyze exactly). e.g. variable/parameter selection followed by model fitting. Joe . Boninger. , . Joshua Brody. , Owen . Kephart. Swarthmore College. Cell Probe Model . [Yao81]. Memory consists of . w-bit. cells. Updates/queries charged for . # probes. All other computation . in Adaptive Data Analysis. Vitaly. Feldman. Overview. Adaptive data analysis. Motivation. Definitions. Basic techniques. With . Dwork. , . Hardt. , . Pitassi. , . Reingold. , Roth . [DFHPRR 14,15]. New results . Psychological Investigations . Research that uses the non-experimental method… . Content Page. SLIDES. . 1. Describe &. Evaluate. the Method. . 2. Hypothesis. & Variables. n/a. 3. Design . General Tools for Post-Selection Inference. Aaron Roth. From: . Trustworthy-broker@trustme.com. To: . aaroth@cis.upenn.edu. Date: 2/27/15. Subject: Gr8 investment tip!!!. Hi! You don’t know me, but here is a tip! . Why is this so important and. why is this so difficult?. Reliability and Validity. The language for this is from the quantitative tradition—scientific . verification—positivism. . Habermas. ’ cognitive interest of . Some types of validity. Criterion-related . Content. Construct. All part of . unitarian. view of validity. Constructs - theoretical abstractions aimed at organizing and making sense of our environment; they are. at time t.5 After aggregation into T regular time periods, we have a matrix of N items for T periods, xit where i indicates variables and t indicates period. Because no survey item is ever posed at e