Publications

2023

1. Who refers whom? The effects of teacher characteristics on disciplinary office referrals

   Michael S Hayes, Jing Liu, and Seth Gershenson

   Economics of Education Review 2023

   Abs PDF

   "Teachers affect a wide range of students’ educational and social outcomes, but how they contribute to students’ involvement in school discipline is less understood. We estimate the impact of same-race teachers and other observed teacher qualifications on students’ likelihood of receiving a disciplinary referral. Using data that track all disciplinary referrals and the identity of both the referred and referring individuals from a large and diverse urban school district in California, we find that Black students’ probability of receiving at least one referral is about 3 percentage points (26.6% of Black students’ base rate) smaller than for white students when they have a Black teacher versus a white teacher. The reduced likelihoods of receiving referrals from same-race teachers also convert to reduced likelihoods of being suspended. These results are mostly driven by referrals for violence, interpersonal offences, and walkout infractions, middle school students, and students from high-poverty schools. Students are also less likely to be referred by more experienced teachers and by teachers who hold either an English language learners or special education credential. While it is unclear whether these findings are due to variation in teachers’ effects on actual student behavior, variation in teachers’ proclivities to make disciplinary referrals, or a combination of the two, these results nonetheless suggest that teachers play a central role in the prevalence of, and inequities in, office referrals and subsequent student discipline."

2. Troublemakers? The Role of Frequent Teacher Referrers in Expanding Racial Disciplinary Disproportionalities

   Jing Liu, Emily K Penner, and Wenjing Gao

   Educational Researchers 2023

   Abs PDF

   Teachers’ sensemaking of student behavior determines whether students get in trouble and are formally disciplined. Status categories, such as race, can influence perceptions of student culpability, but the degree to which teachers’ initial identification of student misbehavior exacerbates racial disproportionality in discipline receipt is unknown. This study provides the first systematic documentation of teachers’ use of office discipline referrals (ODRs) in a large, diverse urban school district in California that specifies the identity of both the referred and referring individuals in all ODRs. We identify teachers exhibiting extensive referring behavior, or the top 5% referrers, based on the number of ODRs they make in a given year and evaluate their contributions to disciplinary disparities. We find that ’top referrers’ effectively double the racial gaps in ODRs for both Black-White and Hispanic-White comparisons. These gaps are mainly driven by higher numbers of ODRs issued for Black and Hispanic students due to interpersonal offences and defiance and also partially convert to racial gaps in suspensions. Both the level and racial compositions of the school sites where top referrers serve and their personal traits seem to explain some of their frequent referring behavior. Targeting supports and interventions to top referrers might afford an important opportunity to reduce racial disciplinary gaps.

3. Can Automated Feedback Improve Teachers’ Uptake of Student Ideas? Evidence from a Randomized Controlled Trial in a Large-Scale Online Course.

   Dorottya Demszky, Jing Liu, Heather C Hill, and 2 more authors

   Educational Evaluation and Policy Analysis 2023

   Abs PDF

   Providing consistent, individualized feedback to teachers is essential for improving instruction but can be prohibitively resource intensive in most educational contexts. We develop an automated tool based on natural language processing to give teachers feedback on their uptake of student contributions, a high-leverage teaching practice that supports dialogic instruction and makes students feel heard. We conduct a randomized controlled trial as part of an online computer science course, Code in Place (n=1,136 instructors), to evaluate the effectiveness of the feedback tool. We find that the tool improves instructors’ uptake of student contributions by 27% and present suggestive evidence that our tool also improves students’ satisfaction with the course and assignment completion. These results demonstrate the promise of our tool to complement existing efforts in teachers’ professional development.

2022

1. More than shortages: The unequal distribution of substitute teaching

   Jing Liu, Susanna Loeb, and Ying Shi

   Education Finance and Policy 2022

   Abs Media PDF

   Classroom teachers in the US are absent on average approximately six percent of a school year. Despite the prevalence of teacher absences, surprisingly little research has assessed the key source of replacement instruction: substitute teachers. Using detailed administrative and survey data from a large urban school district, we document the prevalence, predictors, and variation of substitute coverage across schools. Less advantaged schools systematically exhibit lower rates of substitute coverage compared with peer institutions. Observed school, teacher, and absence characteristics account for only part of this school variation. In contrast, substitute teachers’ preferences for specific schools, mainly driven by student behavior and support from teachers and school administrators, explain a sizable share of the unequal distribution of coverage rates above and beyond standard measures in administrative data.

2. JUE insight: From referrals to suspensions: New evidence on racial disparities in exclusionary discipline

   Jing Liu, Michael S Hayes, and Seth Gershenson

   Journal of Urban Economics 2022

   Abs HTML PDF

   We use novel data on disciplinary referrals, including those that do not lead to suspensions, to better understand the origins of racial disparities in exclusionary discipline. We find significant differences between Black and white students in both referral rates and the rate at which referrals convert to suspensions. An infraction fixed-effects research design that compares the disciplinary outcomes of white and non-white students who were involved in the same multi-student incident identifies systematic racial biases in sentencing decisions. On both the intensive and extensive margins, Black and Hispanic students receive harsher sentences than their white co-conspirators. This result is driven by high school infractions and mainly applies to “more severe” infractions that involve fights or drugs. Reducing racial disparities in exclusionary discipline will require addressing underlying gaps in disciplinary referrals and the systematic biases that appear in the adjudication process.

3. Computationally Identifying Funneling and Focusing Questions in Classroom Discourse

   Sterling Alic, Dorottya Demszky, Zid Mancenido, and 3 more authors

   In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022) 2022

   Abs HTML

   Responsive teaching is a highly effective strategy that promotes student learning. In math classrooms, teachers might funnel students towards a normative answer or focus students to reflect on their own thinking depending their understanding of math concepts. When teachers focus, they treat students’ contributions as resources for collective sensemaking, and thereby significantly improve students’ achievement and confidence in mathematics. We propose the task of computationally detecting funneling and focusing questions in classroom discourse. We do so by creating and releasing an annotated dataset of 2,348 teacher utterances labeled for funneling and focusing questions, or neither. We introduce supervised and unsupervised approaches to differentiating these questions. Our best model, a supervised RoBERTa model fine-tuned on our dataset, has a strong linear correlation of .76 with human expert labels and with positive educational outcomes, including math instruction quality and student achievement, showing the model’s potential for use in automated teacher feedback tools. Our unsupervised measures show significant but weaker correlations with human labels and outcomes, and they highlight interesting linguistic patterns of funneling and focusing questions. The high performance of the supervised measure indicates its promise for supporting teachers in their instruction.

2021

1. Measuring conversational uptake: A case study on student-teacher interactions

   Dorottya Demszky, Jing Liu, Zid Mancenido, and 4 more authors

   arXiv preprint arXiv:2106.03873 2021

   Abs PDF

   "In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers’ uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers’ uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (pJSD), estimated via next utterance classification; (3) conducting a linguistically-motivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, pJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers. "

2. Engaging teachers measuring the impact of teachers on student attendance in secondary school

   Jing Liu, and Susanna Loeb

   Journal of Human Resources 2021

   Abs Media HTML PDF

   On average, secondary school students in the United States are absent from school three weeks per year. For this study, we are able to link middle and high school teachers to the class-attendance of students in their classrooms and create measures of teachers’ contributions to student class-attendance. We find systematic variation in teacher effectiveness at reducing unexcused class absences. These differences across teachers are as stable as those for student achievement, but teacher effectiveness on attendance only weakly correlates with their effects on achievement. A high value-added to attendance teacher has a stronger impact on students’ likelihood of finishing high school than does a high value-added to achievement teacher. Moreover, high value-added to attendance teachers can motivate students to pursue higher academic goals. These positive effects are particularly salient for low-achieving and low-attendance students.

3. The short-and long-run impacts of secondary school absences

   Jing Liu, Monica Lee, and Seth Gershenson

   Journal of Public Economics 2021

   Abs PDF

   We provide novel evidence on the causal impacts of student absences in middle and high school on state test scores, course grades, and educational attainment using a rich administrative dataset that tracks the date and class period of each absence. We use two similar but distinct identification strategies that address potential endogeneity due to time-varying student-level shocks by exploiting within-student, between-subject variation in class-specific absences. We also leverage information on the timing of absences to show that absences that occur after the annual window for state standardized testing do not affect test scores, providing a further check of our identification strategy. Both approaches yield similar results. We nd that absences in middle and high school harm contemporaneous student achievement and longer-term educational attainment: On average, missing 10 classes reduces math or English Language Arts test scores by 3-4% of a standard deviation and course grades by 17-18% of a standard deviation. 10 total absences across all subjects in 9th grade reduce both the probability of on-time graduation and ever enrolling in college by 2%. Learning loss due to school absences can have profound economic and social consequences.

4. Measuring teaching practices at scale: A novel application of text-as-data methods

   Jing Liu, and Julie Cohen

   Educational Evaluation and Policy Analysis 2021

   Abs PDF

   Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers, but conventional classroom observations are costly, prone to rater bias, and hard to implement at scale. Using nearly 1,000 word-to-word transcriptions of 4th- and 5th-grade English language arts classes, we apply novel text-as-data methods to develop automated, objective measures of teaching to complement classroom observations. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores.

5. Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

   Dorottya Demszky, Jing Liu, Zid Mancenido, and 4 more authors

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) 2021

   HTML Code

2020

1. Projecting the potential impacts of COVID-19 school closures on academic achievement

   Megan Kuhfeld, James Soland, Beth Tarasawa, and 3 more authors

   2020

   Abs Media PDF

   Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers, but conventional classroom observations are costly, prone to rater bias, and hard to implement at scale. Using nearly 1,000 word-to-word transcriptions of 4th- and 5th-grade English language arts classes, we apply novel text-as-data methods to develop automated, objective measures of teaching to complement classroom observations. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores.

2019

1. Admission mechanisms and the mismatch between colleges and students: Evidence from a large administrative dataset from China

   Shiyu Bo, Jing Liu, Ji-Liang Shiu, and 2 more authors

   Economics of Education Review 2019

   Abs HTML PDF

   This paper provides empirical evidence on how China’s transition from the Boston mechanism to the Chinese parallel mechanism (a simplified version of the Deferred Acceptance mechanism), along with changes to the information available to students on their entrance exam performance when they submit their college preferences, affect the academic match between colleges and students. Using data on students admitted to Chinese colleges from 2005 to 2011, we characterize the general patterns of mismatch between colleges and students based on students’ scores on China’s National College Entrance Exam and find evidence of substantial overmatch and undermatch. Results from a generalized difference-in-differences model indicate that switching from the Boston mechanism to the Chinese parallel mechanism lowered the probability of mismatch by approximately 6%. Allowing students to submit their college preferences after learning their exam scores rather than before the exam reduced the probability of mismatch by 18%.

2. Differing views of equity: How prospective educators perceive their role in closing achievement gaps

   Emily K Penner, Jane Rochmes, Jing Liu, and 2 more authors

   RSF: The Russell Sage Foundation Journal of the Social Sciences 2019

   Abs PDF

   Hiring is an opportunity for school districts to find educators with values and beliefs that align with district goals. Yet beliefs are difficult to measure. We use administrative data from more than ten thousand applications to certificated positions in an urban California school district in which applicants submitted essays about closing achievement gaps. Using structural topic modeling (STM) to code these essays, we examine whether applicants systematically differ in their use of these themes and whether themes predict hiring outcomes. Relative to white applicants, Hispanic and African American applicants are more likely to identify structural causes of inequities and discuss educators’ responsibilities for addressing inequality. Similar differences in themes emerge between applicants to schools with different student populations. Techniques like STM can decipher hard-to-measure beliefs from administrative data, providing valuable information for hiring and decision making.

2017

1. What We’re Missing: A Descriptive Analysis of Part-Day Absenteeism in Secondary School

   Camille R Whitney, and Jing Liu

   AERA Open 2017

   Abs Media PDF

   For schools and teachers to help students develop knowledge and skills, students need to show up to class. Yet absenteeism is prevalent, especially in secondary schools. This study uses a rich data set tracking class attendance by day for over 50,000 middle and high school students from an urban district in academic years 2007–2008 through 2012–2013. Our results extend and modify the extant findings on absenteeism that have been based almost exclusively on full-day absenteeism, missing class-by-class absences. Notably, part-day absenteeism is responsible for as many classes missed as full-day absenteeism, raising chronic absenteeism from 9% to 24% of secondary-grades students. Incorporating part-day absences sharply increases the chronic absenteeism gap between underrepresented minority students and their peers. Both full- and part-day absenteeism show a discrete jump at the point of transition from middle school to high school, but full-day absenteeism then declines whereas part-day absenteeism remains high in Grades 10 and 11 and increases again in Grade 12. Whereas 55% of full-day absences are unexcused, 92% of part-day absences are unexcused. Absenteeism from individual classes varies considerably by time of day but less by class subject matter.

2016

1. Connections Matter: How Interactive Peers Affect Students in Online College Courses

   Eric Bettinger, Jing Liu, and Susanna Loeb

   Journal of Policy Analysis and Management 2016

   Abs Media PDF

   Peers affect individual’s productivity in the workforce, in education, and in other team‐based tasks. Using large‐scale language data from an online college course, we measure the impacts of peer interactions on student learning outcomes and persistence. In our setting, students are quasi‐randomly assigned to peers, and as such, we are able to overcome selection biases stemming from endogenous peer grouping. We also mitigate reflection bias by utilizing rich student interaction data. We find that females and older students are more likely to engage in student interactions. Students are also more likely to interact with peers of the same gender and with peers from roughly the same geographic region. For students who are relatively less likely to be engaged in online discussion, exposure to more interactive peers increases their probabilities of passing the course, improves their grade in the course, and increases their likelihood of enrolling in the following academic term. This study demonstrates how the use of large‐scale, text‐based data can provide insights into students’ learning processes.

2012

1. Did VAT Reform Change Firms’ Employment Decisions?: Evidence from the Pilot VAT Reform in Northeast China

   Jing Liu, and Cheng Yuan

   Economic Science 2012