Read by QxMD icon Read

Evaluation Review

Vivian C Wong, Peter M Steiner
Over the last three decades, a research design has emerged to evaluate the performance of nonexperimental (NE) designs and design features in field settings. It is called the within-study comparison (WSC) approach or the design replication study. In the traditional WSC design, treatment effects from a randomized experiment are compared to those produced by an NE approach that shares the same target population. The nonexperiment may be a quasi-experimental design, such as a regression-discontinuity or an interrupted time-series design, or an observational study approach that includes matching methods, standard regression adjustments, and difference-in-differences methods...
January 1, 2018: Evaluation Review
Yang Tang, Thomas D Cook
The basic regression discontinuity design (RDD) has less statistical power than a randomized control trial (RCT) with the same sample size. Adding a no-treatment comparison function to the basic RDD creates a comparative RDD (CRD); and when this function comes from the pretest value of the study outcome, a CRD-Pre design results. We use a within-study comparison (WSC) to examine the power of CRD-Pre relative to both basic RDD and RCT. We first build the theoretical foundation for power in CRD-Pre, then derive the relevant variance formulae, and finally compare them to the theoretical RCT variance...
January 1, 2018: Evaluation Review
Yasemin Kisbu-Sakarya, Thomas D Cook, Yang Tang, M H Clark
Compared to the randomized experiment (RE), the regression discontinuity design (RDD) has three main limitations: (1) In expectation, its results are unbiased only at the treatment cutoff and not for the entire study population; (2) it is less efficient than the RE and so requires more cases for the same statistical power; and (3) it requires correctly specifying the functional form that relates the assignment and outcome variables. One way to overcome these limitations might be to add a no-treatment functional form to the basic RDD and including it in the outcome analysis as a comparison function rather than as a covariate to increase power...
January 1, 2018: Evaluation Review
Peter M Steiner, Vivian C Wong
In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate...
January 1, 2018: Evaluation Review
David Kaplan, Chansoon Lee
This article provides a review of Bayesian model averaging as a means of optimizing the predictive performance of common statistical models applied to large-scale educational assessments. The Bayesian framework recognizes that in addition to parameter uncertainty, there is uncertainty in the choice of models themselves. A Bayesian approach to addressing the problem of model uncertainty is the method of Bayesian model averaging. Bayesian model averaging searches the space of possible models for a set of submodels that satisfy certain scientific principles and then averages the coefficients across these submodels weighted by each model's posterior model probability (PMP)...
January 1, 2018: Evaluation Review
Robert Bifulco, Ross Rubenstein, Hosung Sohn
BACKGROUND: "Place-based" scholarships seek to improve student outcomes in urban school districts and promote urban revitalization in economically challenged cities. Say Yes to Education is a unique district-wide school reform effort adopted in Syracuse, NY, in 2008. It includes full-tuition scholarships for public and private universities, coupled with extensive wraparound support services in schools. OBJECTIVES: This study uses synthetic control methods to evaluate the effect of Say Yes on district enrollment and graduation rates...
December 2017: Evaluation Review
David A Bell
BACKGROUND: Assessing program effectiveness in human development is central to informing foreign aid policy-making and organizational learning. Foreign aid effectiveness discussions have increasingly given attention to the devaluing effects of aid flow volatility. This study reveals that the external evaluation event influences actor behavior, serving as a volatility-constraining tool. METHOD: A case study of a multidonor aid development mechanism served examining the influence of an evaluation event when considering anticipatory effects...
December 2017: Evaluation Review
Sharon Vandivere, Karin E Malm, Tiffany J Allen, Sarah Catherine Williams, Amy McKlindon
BACKGROUND: Youth who have experienced foster care are at risk of negative outcomes in adulthood. The family finding model aims to promote more positive outcomes by finding and engaging relatives of children in foster care in order to provide options for legal and emotional permanency. OBJECTIVES: The present study tested whether family finding, as implemented in North Carolina from 2008 through 2011, improved child welfare outcomes for youth at risk of emancipating foster care without permanency...
December 2017: Evaluation Review
T'Pring R Westbrook
No abstract text is available yet for this article.
October 2017: Evaluation Review
Jaime Thomas, Sarah A Avellar, John Deke, Philip Gleason
BACKGROUND: Systematic reviews assess the quality of research on program effectiveness to help decision makers faced with many intervention options. Study quality standards specify criteria that studies must meet, including accounting for baseline differences between intervention and comparison groups. We explore two issues related to systematic review standards: covariate choice and choice of estimation method. OBJECTIVE: To help systematic reviews develop/refine quality standards and support researchers in using nonexperimental designs to estimate program effects, we address two questions: (1) How well do variables that systematic reviews typically require studies to account for explain variation in key child and family outcomes? (2) What methods should studies use to account for preexisting differences between intervention and comparison groups? METHODS: We examined correlations between baseline characteristics and key outcomes using Early Childhood Longitudinal Study-Birth Cohort data to address Question 1...
June 2017: Evaluation Review
Nianbo Dong, Mark W Lipsey
BACKGROUND: It is unclear whether propensity score analysis (PSA) based on pretest and demographic covariates will meet the ignorability assumption for replicating the results of randomized experiments. PURPOSE: This study applies within-study comparisons to assess whether pre-Kindergarten (pre-K) treatment effects on achievement outcomes estimated using PSA based on a pretest and demographic covariates can approximate those found in a randomized experiment. METHODS: Data-Four studies with samples of pre-K children each provided data on two math achievement outcome measures with baseline pretests and child demographic variables that included race, gender, age, language spoken at home, and mother's highest education...
January 1, 2017: Evaluation Review
Randall Juras
INTRODUCTION: When cluster randomized trials are used to evaluate school-based nutritional interventions such as school lunch programs, design-stage estimates of the required sample size must take into account the correlation in outcomes among individuals within each cluster (e.g., classrooms, schools, or districts). Estimates of the necessary parameters have been carefully developed for educational interventions, but for nutritional interventions the literature is thin. METHODS: Using data from two large multi-school, multi-district impact evaluations conducted in the United States, this article calculates estimates of the design parameters required for sizing school-based nutritional studies...
November 20, 2016: Evaluation Review
Ben Kelcey, Zuchao Shen, Jessaca Spybrook
OBJECTIVE: Over the past two decades, the lack of reliable empirical evidence concerning the effectiveness of educational interventions has motivated a new wave of research in education in sub-Saharan Africa (and across most of the world) that focuses on impact evaluation through rigorous research designs such as experiments. Often these experiments draw on the random assignment of entire clusters, such as schools, to accommodate the multilevel structure of schooling and the theory of action underlying many school-based interventions...
October 26, 2016: Evaluation Review
John Deke
BACKGROUND: Cluster randomized controlled trials (CRCTs) often require a large number of clusters in order to detect small effects with high probability. However, there are contexts where it may be possible to design a CRCT with a much smaller number of clusters (10 or fewer) and still detect meaningful effects. OBJECTIVES: The objective is to offer recommendations for best practices in design and analysis for small CRCTs. RESEARCH DESIGN: I use simulations to examine alternative design and analysis approaches...
October 25, 2016: Evaluation Review
Jeffrey C Valentine, Sandra Jo Wilson, David Rindskopf, Timothy S Lau, Emily E Tanner-Smith, Martha Yeide, Robin LaSota, Lisa Foster
For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations...
October 25, 2016: Evaluation Review
Richard Dorsett, Richard Hendra, Philip K Robins
BACKGROUND: Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This article highlights a case in which full-sample results from a large-scale RCT in the United Kingdom differ from results for a subsample of survey respondents. OBJECTIVES: Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs...
October 25, 2016: Evaluation Review
Andrew P Jaciw, Li Lin, Boya Ma
BACKGROUND: Prior research has investigated design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Less is known about parameters important for assessing differential impacts. OBJECTIVES: This article develops a statistical framework for designing CRTs to assess differences in impact among student subgroups and presents initial estimates of critical parameters. RESEARCH DESIGN: Effect sizes and minimum detectable effect sizes for average and differential impacts are calculated before and after conditioning on effects of covariates using results from several CRTs...
October 18, 2016: Evaluation Review
Geoffrey Phelps, Benjamin Kelcey, Nathan Jones, Shuangshuang Liu
Mathematics professional development is widely offered, typically with the goal of improving teachers' content knowledge, the quality of teaching, and ultimately students' achievement. Recently, new assessments focused on mathematical knowledge for teaching (MKT) have been developed to assist in the evaluation and improvement of mathematics professional development. This study presents empirical estimates of average program change in MKT and its variation with the goal of supporting the design of experimental trials that are adequately powered to detect a specified program effect...
October 3, 2016: Evaluation Review
T'Pring R Westbrook, Sarah A Avellar, Neil Seftor
BACKGROUND: The federal government's emphasis on supporting the implementation of evidence-based programs has fueled a need to conduct and assess rigorous evaluations of programs. Through partnerships with researchers, policy makers, and practitioners, evidence reviews-projects that identify, assess, and summarize existing research in a given area-play an important role in supporting the quality of these evaluations and how the findings are used. These reviews encourage the use of sound scientific principles to identify, select, and implement evidence-based programs...
September 30, 2016: Evaluation Review
Nianbo Dong, Wendy M Reinke, Keith C Herman, Catherine P Bradshaw, Desiree W Murray
BACKGROUND: There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs). OBJECTIVES: We calculated the empirical reference values on critical research design parameters associated with statistical power for children's social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels (R (2))...
September 30, 2016: Evaluation Review
Fetch more papers »
Fetching more papers... Fetching...
Read by QxMD. Sign in or create an account to discover new knowledge that matter to you.
Remove bar
Read by QxMD icon Read

Search Tips

Use Boolean operators: AND/OR

diabetic AND foot
diabetes OR diabetic

Exclude a word using the 'minus' sign

Virchow -triad

Use Parentheses

water AND (cup OR glass)

Add an asterisk (*) at end of a word to include word stems

Neuro* will search for Neurology, Neuroscientist, Neurological, and so on

Use quotes to search for an exact phrase

"primary prevention of cancer"
(heart or cardiac or cardio*) AND arrest -"American Heart Association"