The evaluation of the SEMS‐World Bank pilot to reduce dropout in Upper‐Secondary
CIRO AVITABILE WORLD BANK
Three main questions 1) Why do we evaluate?
2) What do we evaluate?
3) How do we evaluate?
Why do we evaluate? 1. We want to know whether the program is working or not
2. If it is working, we would like to know what are the mechanisms Understanding the behavioral changes induced by the program, it can help us to design a program in a cost effective way It can help to assess whether the program has potential to be effective in other contexts
Just based on yesterday’s discussion • CBT
? ?
•Math Tutoring
? • CBT • Math Tutoring
Even when we get the intervention right…. In order to make it effective: 1. Do we always need 5 math sessions per week or 27 CBT sessions per year? 2. Do we always need 2 to 1 students/tutor for Math Tutoring or 12/1 students/counsellor for CBT sessions? 3. How much do we need to pay the counsellors?
In many cases, we do not have an answer
How can we know? Pilot alternative versions of the program Evaluate them
How do we choose the options to evaluate? In principle there are many versions of the same program that can be piloted, but we can realistically compare a few versions at the time 1. The theory (economic/psychological/pedagogical) put some restrictions 2. Most of the restrictions are institutional and context based: How much money do we have? Can schools hire external experts only for a few months a year? Are there potentially enough CBT experts in the country?
What are we piloting in Mexico?
The “golden” and the “silver” version THE GOLDEN VERSION
THE SILVER VERSION
1.
20 sessions of CBT per year in groups of 10 1. to 1
10 sessions of CBT per year in per year in groups of 10 to 1
2.
Math Tutoring
2.
Math Tutoring
3.
Basic Information Package:
3.
Basic Information Package:
a)
b)
How does High School work (promotion rules/potential resources in the school for different needs) Monetary Returns in the relevant Labor Market to different High School types
a)
b)
How does High School work (promotion rules/potential resources in the school for different needs) Monetary Returns in the relevant Labor Market to different High School types
It should be clear what our constraint was SEMS is already running a Math tutoring program, that will provide a basis for the new intervention
Information is cheap
Very difficult to hire external experts for the CBT sessions ‐> Teachers
What can we learn by comparing the “gold” and the “silver” version Assuming that we manage to adequately (Tony talks about fidelity) implement both versions of the program: 1.
If the impact of the “gold” is not statistically different from the “silver”, we have an easy choice
2.
If the impact of the “gold” is statistically larger than the “silver”:
3.
If SEMS has enough resources, it might think of scaling‐up the “gold” If not, the program can be twisted (changing some of the parameters)
If neither the “gold” nor the “silver” have a statistically significant impact
We should try other types of intervention
How do we evaluate our pilot?
Focalization 1.
We will pilot the program in the Mexico city metropolitan area
2.
We already agreed on a potential universe of schools
3.
The average number of students in 10th grade in our sample is 1,600 students
Comipems Students who want to attend a public high school in the urban area of Mexico City are required to enroll in a centralized and competitive assignment mechanism ◦ COMIPEMS, by its Spanish acronym
In March applicants fill a list of up to 20 educational options, which they rank in order of preference
In June students take a test consisting of 128 multiple choice questions
In July of the same year, the assignment process is carried out by an independent educational evaluation institution (CENEVAL)
The COMIPEMS score and HS dropout
Our population of interest
Evaluation design 1.
We will select a sample of 15 schools
2.
Within each school, we will focus on the 10th grade students in the bottom 2 quintiles of the COMIPEMS score
3.
Within each school, we will randomly allocate targeted students in 3 groups: The Golden Treatment The Silver Treatment Control group
Summarizing: Roughly 4,500 10th grade students in 15 schools in the Metropolitan area of
Mexico City Randomly allocated in 3 groups: 1. 1,500 Students in the Golden Treatment 2. 1,500 Students in the Silver Treatment 3. 1,500 Students in the Control group
Timeline First Results
Baseline • 4th week September 2016
Intervention starts • 2nd week October 2016
Intervention ends
• July 2017
• April 2017
Follow Up survey • May 2017
Two Options for Data Collection
Both Baseline and Follow Up based exclusively on Administrative Data Estimated cost: Almost 0
Combining Administrative and Survey Data both at the Baseline and the Follow Up Estimated cost: 220,000 USD
What do we get?
We can measure whether the intervention is affecting dropout We will not learn what is happening
We can measure whether the intervention is affecting the dropout and whether it is improving: 1. Socio‐Emotional Skills 2. Math knowledge We can collect information to understand the possible issues when scaling up