Transparent Statistics in HCI

Workshop: Moving Transparent Statistics Forward @CHI 2017

Matthew Kay, Steve Haroz, Shion Guha, Pierre Dragicevic, and Chat Wacharamanotham

Call for Participation – Deadline Feb 15
Workshop at CHI 2017, Saturday, May 6, 2017

HCI is large and multidisciplinary, drawing on a variety of statistical practices. However, many of these existing practices have drawn increasing criticism within both HCI and related fields, including but not limited to: over-reliance on particular statistical methods, a lack of transparent reporting, a lack of replication and meta-analysis, few studies published with data or study materials, and inadequate education in statistics. These issues have even reached the popular press in coverage of the replication crisis in social science.

We are running a working workshop to develop concrete guidelines for improving statistical practice in HCI. Participants will work in groups to flesh out guidelines for helping reviewers fairly assess statistical reports in CHI papers, concrete suggestions for changes to review processes, resources for authors, and other relevant guidelines or proposals the group wishes to advance. We will seed writing with draft guidelines developed in the wake of last years’ SIG on Transparent Statistics in HCI. However, we will not be constrained to the outline in these draft documents if discussions take us elsewhere.

These documents will be shared publicly once finalized and we will engage the CHI Executive Committee and/or Program Chairs to discuss the feasibility of suggested changes to the review process, and the possibility of incorporating some of the recommendations into official documents like the SIGCHI reviewer guidelines. We will also continue hosting and drafting future revisions of the documents on a collaborative editing platform.

We are looking for a diverse set of perspectives on quantitative methods in the HCI community to develop these documents (while qualitative methods are important to HCI, our primary focus in this workshop is on improving the use and communication of quantitative work). If you are interested in improving the state of statistical practice in HCI—whether or not you attended last year’s SIG—submit a position statement (at most 2 pages in CHI Extended Abstracts format) containing:

Submit to chi2017workshop@transparentstatistics.org. The final deadline for submission is Feb 15, 2017.

As much as we would like as many CHI researchers as possible to get involved in this initiative, we may not be able to accept all applicants due to limits in the number of attendees at CHI workshops. If this happens, we will select applications based on expertise, diversity of perspectives in quantitative methods, diversity of HCI domains, and diversity of interest in specific artifacts (to ensure we have people interested in working on each artifact). Our goal is that the participants reflect the diversity in quantitative methods, statistics, and domains across HCI.

The purpose of the position statement is to help us determine specific participants, so we encourage one author per position statement. Position statements with multiple authors should clearly indicate which author will attend. If your position statement is accepted, the attending author must register for both the workshop and at least one day of the conference.