TailorEd Research Project
Overview
Our study uses automated aerial photography to document the instructional formats used in 74 courses taught in humanities, social sciences, engineering, and business disciplines in 12 active learning and 2 traditional (control) classrooms. We used the machine learning tools available in Matlab to analyze the photos to identify instances of lecture format, large group discussion, small group discussion, approximating different active learning formats. The next stage of the project will cross-reference these breakdowns of the kinds of pedagogical techniques used and their proportions, and correlate them to student learning outcomes (measured by grades), broken down by student demographic factors.
About
The TailorEd Study looks at the relationships between:
- classroom design (for example, mobile vs. stationary desks),
- teaching formats (lecture, discussion, small group work, use of classroom technology, etc.), and
- student performance (measured by grades), and student characteristics (race, gender, major, etc).
Its goal is to test the claims made in learning space design researchers that active learning formats (such as discussion and small group work) improve learning, especially for students marginalized in educational settings due to their gender, race, and other identity characteristics. Faculty and students teaching/taking classes included receive an email notifying them of inclusion of their class in the study, which directs them to this webpage, that provides information about the study, explains where to direct questions, and informs about how to opt out of the study. There are also signs posted in all study classrooms notifying faculty and students that the classroom is part of the TailorEd Study.
For frequently-asked questions about the study, see our FAQ page
Study Procedures
The study uses aerial classroom photography at 30-second intervals to capture classroom activities in real time. Researchers will then run these photos through a machine learning algorithm to categorize the activities depicted in order to determine the proportion of the class allocated to different activities. These course "pedagogical profiles" will be compared to student learning outcomes (in the form of grades) to identify relationships between different types and distributions of classroom activities and student learning, differentiated by:
- course factors (such as discipline and level),
- classroom characteristics (such as the presence/absence of mobile furniture, writable walls, multiple projectors, and smartboards), and
- student information (demographics, course grade and GPA, etc).
Confidentiality
Reports/publications based on this study will focus on aggregate findings across all courses included in the study, not on individual courses, faculty, or students. Faculty will not be identified at all, and students/student information will be identified using a unique identifier (NOT their student ID#) generated for the purpose of this study only. There is some small risk of identifiability in the photos as sample photos illustrate, people who know the individuals depicted may be able to recognize them. To mitigate this, individuals included in classroom photographs will never be identified in any part of this study, limiting the risk of identifiability more broadly. The photos also depict a "public" event (class meetings), and therefore do not contain sensitive or compromising information.
All records and research materials associated with this study will be kept in a secure location (password-protected computer/cloud backup). Only members of the research team will have access to project records and research materials.
Participants' Rights
Participation in this study is voluntary and won't affect your employment or performance review at Santa Clara University. You are free to opt out of the study at any time and to see research materials in which you are included (photos or reports on the study). If any member of the class (faculty or student) objects at any time to participating in the study, the course will be removed from the study and all data associated with that course will be permanently deleted.
Presentations
TailorEd in Course Hero's Education Summit 2020
In this session we will present TailorEd: A custom image recognition neural network applied to imagery of select classroom environments coupled with student demographic and grade data in a study of the effects of pedagogical practices and classroom technologies on learning. We will start with the motivation behind this project and explain how we have gathered our data, guide the attendees through a human qualitative data coding practice via a hands on exercise, introduce our Classroom Configuration Identifier (CCID) image classifier neural network via a live demo, share our preliminary results and explain our data modeling procedure, demo our resulting web tool for educators, and end by delineating the project’s future steps.
Publications
This study uses unique empirical methods to reconsider, for STEM disciplines, Russell's (1990) claim that WAC has failed to make a "permanent impact" on higher education by using photography to document classroom activities in real time and using machine learning to categorize these images to determine which learning activities are used in STEM instruction and in what proportions for a sample of 18 STEM courses at a private liberal arts university.
Journal Publication: Across the Disciplines from WAC Clearinghouse
The study of how classroom layout and activities affect learning outcomes of students with different demographics is difficult because it is hard to gather accurate information on the minute by minute progression of every class in a course. Furthermore, the process of data gathering must produce an abundance of data to work with and hence must be automated. A machine learning model trained on images of a classroom and thus capable of accurately labeling the classroom layout and activity of many thousands of images much faster and cheaper than employing a human.
Journal Publication: EAI Endorsed Transactions on Creative Technologies
Classroom Configuration Identifier (CCID) is an application utilizing AlexNet - an image recognition convolutional neural network (CNN) - trained and used for classification of classroom configurations. While CCID specifically deals with the categorical analysis of the classroom configuration itself, the next phases of the study will couple this classroom data with anonymized student data, in order to draw conclusions about the most optimal classroom configurations for enhancing learning. By analyzing correlations between different classroom configurations and corresponding student performance, the study will ultimately be able to supply educators with the information needed to setup more effective learning environments.
Presented at: The IEEE 11th International Conference on Engineering Education (ICEED 2019) at Kanazawa, Japan
Frequently Asked Questions
TailorEd takes only still photographs at 30-second intervals. We do not record any video or audio. If a class uses lecture capture or other classroom recording technologies, the study does not have access to those recordings, only to our own photographs.
No. Students and faculty are welcome to opt out of the study at any time, without any kind of penalty. To opt out, refer to the opt-out link provided in your study notification email, which has the subject line "Classroom Study Notification".
If anyone -- a student or a faculty member -- opts out of the study, their class will be removed from the study. If the class hasn't met yet, no data will be collected. If the quarter has already started and images have already been captured, we will stop data collection in that class and delete all existing data.
Alameda 103, Alumni Science 220, Daly Science 106, Graham 163, Graham 164, Kenna 107, Kenna 109, Kenna 308, Lucas 206, Lucas 210, Lucas 310, O'Connor 103, O'Connor 110, O'Connor 204, O'Connor 209, O'Connor 210, SCDI 1301, SCDI 1302, SCDI 1308, SCDI 2301, SCDI 2302, SCDI 3115, SCDI 3116, SCDI 3301, SCDI 3302, Vari 133, Varsi 114 (27 classrooms total).
Nothing. If you're no longer taking the class, you don't need to do anything, because no data involving you will be collected. Getting the study notification email was an error; it does not mean you're still enrolled or that you need to do anything to change or update your schedule. Sorry for the confusion!
Because students can add/drop classes until the end of Week 1, class rosters -- which we use to send out study notification emails -- are not finalized until Week 2. This ensures that we don't send emails to students who are no longer enrolled in study classes and that we don't miss any enrolled students who add the class late. (for study opt-outs that happen after the quarter has begun and data collection has begun, see "General FAQs" section)
Please notify one of the PIs (Julia Voss, jvoss@scu.edu , 408-554-2715 ; Navid Shaghaghi, nshaghaghi@scu.edu , 408-554-4179 ) so we can update the list of classes included in the study.
The ways in which faculty are designing courses that differ from Santa Clara's 100% in-person format is also of interest to the study, so we hope you'll agree to keep classes like this in the study.
You aren't obligated to tell them anything; the study's methods are designed so as not to impose on faculty or take up valuable class time. Students receive their own study email notification (sent during Week 2, after the add/drop date passes to ensure that all students receive the message). You're welcome to discuss the study with your students, however, if you like, and to direct any questions that come up to the study PIs Julia Voss ( jvoss@scu.edu , 408-554-2715 ) and Navid Shaghaghi ( nshaghaghi@scu.edu , 408-554-4179 ).
We're working on a data visualization report to offer participating faculty showing a breakdown of in-class activities across meeting days. Stay tuned, and if you have ideas and/or requests about this, please share them with study PIs Julia Voss ( jvoss@scu.edu , 408-554-2715 ) and Navid Shaghaghi ( nshaghaghi@scu.edu , 408-554-4179 ).
Contact
GET IN TOUCH
If you would like to receive more information or need to contact us, please fill out the form below and we will try to get back to you as soon as possible.
Questions, Concerns, and Opting Out of the Study
If you have questions, concerns, or complaints about this study, or if you wish to opt out, refer back to the email you received notifying you that a class you're taking/teaching is scheduled to be included in the TailorEd Study, titled "Classroom Study Notification". This email includes a personalized link allowing you to opt out of the study. If you cannot find that email, you can contact one of the investigators:
- Julia Voss ( jvoss@scu.edu , 408-554-2715 )
- Navid Shaghaghi ( nshaghaghi@scu.edu , 408-554-4179 )
Lastly, if you have questions about your rights as a participant in this study, or if you feel you have been placed at risk, contact the Chair of SCU's Human Subjects Committee through the Office of Research Compliance and Integrity at 408-554-5591 .