Student Assessment and Program Evaluation

Program:

The Immigrant Learning Center, Inc. (ILC)
442 Main Street
Malden, Massachusetts 02148
Tel. 781.322.9777 Fax. 781.321.1963
www.ilctr.org

Contact:

Diane Portnoy, Co-Founder and Director
dportnoy@ilctr.org

“Useful programming ideas can and do come from any number of sources—research, students, teachers, other staff, volunteers, the Board of Directors and student council. But if you don’t have systems in place not only to elicit that input, but then to respond to it, it’s as if you never asked those people, those resources for feedback in the first place.”

~ Diane Portnoy, Co-Founder and Director

Mission

Established in 1992, the mission of The Immigrant Learning Center, Inc. (ILC) is to provide enough knowledge of the English language to immigrant and refugee adults to help them lead productive lives in the United States and to become successful parents, workers and community members.

Services

The ILC serves 650 immigrants and refugees annually. Currently, students come from 58 countries, 37 communities in the Boston area and speak 37 native languages. The ILC offers five levels of ESL classes as well as ESL Family Literacy and Citizenship preparation classes. Each class meets five days a week for a total of 15 hours of weekly instruction. In addition, the ILC ABE program provides four levels of classes in Reading, Writing, Grammar, and Conversation, as well as GED and TOEFL preparation in the afternoon program for a total of ten hours weekly. All classes are complemented by computer literacy classes. The ILC provides support services to students through a full-time guidance counselor.

Rationale and Background of the Practice

The ILC views on-going student assessment and program evaluation as the means to demonstrate that its mission and goals are being reached. Data on student performance and other data direct program and policy decisions of the ILC. The ILC’s consultative process has led the agency to refine program evaluation and student assessment processes and tools over the years.

On-going program evaluation and student assessment allow the agency to be accountable to all its stakeholders (students, staff and funders). Furthermore, adult literacy research shows that adults persist in their studies when they can state their learning needs and goals and can see evidence that they are reaching those goals (Comings et al, 2000).

Description of the Practice

Student Assessment: The ILC’s assessment model translates into a three-part cycle. It includes initial intake and placement through an interview process; on-going, in-class articulation of student learning and vocational goals and subsequent refinement of classroom instruction, and a final stage of assessment that identifies students’ learning gains.

1. Initial intake and placement

Counselor meets with each incoming student for a twenty-minute interview that determines what classroom level the student will enter. No standardized tests are used during the initial intake. The interview consists of English reading, writing and conversational exercises, as well as a short activity meant to gather basic data about the student’s proficiency in his/her native language.

Once the student enters the program, the counselor and teachers administer the BEST or REEP test to validate that ESL students have been placed in correct levels.

2. Classroom assessments of student progress

Goal Sheet: Students complete a goal sheet that includes plans for meeting their personal, family and workplace literacy goals. Goal sheets become part of the student’s permanent record and are reviewed at the beginning and end of each session.

Teachers not only elicit students’ personal goals, but also their class goals: what they want to study, what they hope to learn in the class. Students discuss and share activities that help them in class and out of class with pronunciation, conversation, listening, grammar, reading, and writing. A variety of techniques are used to obtain this information: worksheets, problem posing activities, information grids, facilitated discussions, etc.

Portfolio: Program compiles students’ test results, evaluations, achievement and goal forms, worksheets, writing samples, or exercises that show progress.

3. End-of-session assessment and placement

Student Self-Evaluation: Students rate their grammar, speaking, listening, reading, writing and computer skill gains by indicating whether they “understand very well,” “understand a little,” or “don’t understand.”

Final Teacher Evaluation of Students: Teachers rate and comment on each student’s attendance, class participation, speaking, writing, listening and reading skills using a scale of “Below Average,” “Average,” or “Above Average.”

Student-Teacher Conference: The student self-evaluation and final teacher evaluation are discussed in this meeting.

Achievement Form: Teachers ask students a series of yes/no questions that provide information about workforce development goals (e.g., “Did you start your own business?”), educational goals (e.g. “Did you take the GED test?), civic goals (e.g. Did you take the citizenship exam?”), and in the case of family literacy classes, family-oriented goals (e.g. “Has your child applied to college?).

This three-part student assessment process is easy to adapt or replicate. It can be used as a template to compare what teachers, students and staff are already doing to identify and document student learning gains. Many programs will already have all or some aspects of a student assessment process in place. In developing or refining their student assessment process, programs should seek input from the staff, students, administration and perhaps also the agency’s governing board. After collecting various in-class assessment tools, the staff can compare their tools to those of the ILC and create ones that best meet their needs. From there, teachers can pilot the assessment tools, and after the first cycle of use, refine them. This process will likely require an investment of staff development and ongoing support.

Program Evaluation: The results of the student assessment process inform overall program evaluation efforts at the ILC. The ILC identifies on-going program evaluation and subsequent refinement of its services as one of the keys to successful organizational development and stability. The program evaluation consists of:

End-of-session classroom evaluations that inform teacher practices. This evaluation asks six questions. Following are two examples: “Please tell me what you think about the difficulty of the lessons. Are they too easy? Too difficult?” and, “Are the lessons helpful to you outside of school? Do you ever use what we learn in class after you go home? Can you give any examples?”
Yearly student surveys administered by upper level students and analyzed by staff. The survey instrument was initially developed by a subcommittee of the Board and refined over the years with input from students, a student council and teachers. This school-wide survey is six pages long and covers student demographic data, reasons for attending the ILC, areas in which students felt they had improved, and questions about instruction. Questions about instruction ask: how well students understand the lessons, how often class materials are useful and interesting, how often the teacher works with small groups or individual students, how often the teacher provides practice time, how often the teacher encourages and answers questions, and how has homework been helpful. Questions are multiple choice except for the last two, which ask students what they like about the Center and what they would change. Students complete the survey anonymously.
The ILC is conducting a five-year longitudinal study meant to demonstrate the impact of students’ improved English competence within family and work contexts. During FY 01, the ILC conducted in-depth, face-to-face interviews with a small sampling of former students and analyzed the results. Based on these interviews, a student survey was developed and piloted in 2002. The survey begins with currently enrolled students who determine their own goals through a goal-setting exercise. At that time, each student is advised that they will be requested to report back on how well they had achieved them over a three-year-period using postage-paid Student Update Cards. The cards ask, “Did the ILC help you with your goals? Are you working? If you have left the ILC, did you go to a school and/or training program?” and “Would you recommend the ILC to other people?”
The ILC recently completed a mailed survey to all its former students. Sample questions include: Have you started your own business? Do you have a job? Have you attended a training program? Have you attended other classes since leaving the ILC? Have you taken any tests (GED, TOEFL)? Are you a U.S. citizen? Do you volunteer? Do you help your children in school? Do you attend parent-teacher conferences? Do you think the ILC helped you learn English? With a 7% return rate, the results indicate that former ILC students are becoming successful workers, parents and community members. The next step is to determine if there is a cause and effect relationship between attending the ILC and student success and identifying those ILC practices that have the greatest impact.

The restructuring of the ILC’s afternoon ABE program, as well as the ILC’s new public education/civic outreach component are examples of changes that have been made based on student surveys, the longitudinal study, and input from the staff and advisory council.

Challenges

Implementing this type of student assessment and program evaluation system presents two challenges:

1. Program evaluation and systematic student assessment create additional staffing and data-collection expenses for the agency. This is addressed through fund raising efforts. The following are examples: 1) An assessment of the ABE program by students and teachers resulted in its restructure in January 2003 to better serve the needs of students. A fourth ABE teacher needed to be hired. The ILC’s Director of Development submitted proposals with a revised ABE program budget to major corporations and foundations requesting funding. The ILC received nearly $85,000 in grants from a major corporation and two foundations, including the Nellie Mae Education Foundation, in 2003 to fund this effort. 2) The ILC also received continued funding from the Nellie Mae Education Foundation to fund the Longitudinal Study to continue efforts to track the progress of ILC students and how ILC programs helped them to achieve their goals. 3) The Foundation also funds the ILC’s development capacity building efforts by supporting the salary of an Associate Director of Development who provides research, data base management and other administrative support to the Director of Development.

2. Staff must “buy-in” to the need for a student assessment and program evaluation process and trust the effectiveness of the particular assessment and evaluation tools that the agency chooses to use. This is addressed through staff input. Teachers have created some of the assessment tools. The culture of the ILC encourages staff suggestions and initiatives. Monthly staff meetings, scheduled teacher sharing time, and the easy accessibility of the director enhance the process. Their suggestions have led to changes.

Evidence of Impact and Effectiveness

“Detailed documentation of diverse outcomes has bolstered the ILC’s fundraising efforts. The ILC draws its funding from a very diverse pool of public and private sources, namely foundations, businesses and individuals.”

~ Diane Portnoy, Co-Founder and Director

The student assessment and program evaluation efforts enable the ILC to monitor and measure its impact and effectiveness in quantitative and qualitative ways. The ILC is able to present to all its stakeholders a myriad of countable outcomes as well as narrative student and program success stories. For example:

In fiscal year 2002, 93 students achieved their employment goals such as gaining employment, moving from part-time to full-time work, being promoted and starting businesses; 26 students entered colleges or training programs; 36 students became United States citizens; one passed the TOEFL; one passed the GED; one passed the GMAT, and one took the medical license exam; 11 students received scholarships to pursue further education or a civic goal such as becoming United States citizens; and 39 students, most of whom were non-literate in their native languages, were supported to meet their goals through the pre-literacy ESL class.
During fiscal year 2002, 101 students and their 191 children were served through the Family Literacy Program, achieving their self-identified goals of obtaining library cards, going to job interviews, getting new jobs, attending training programs, purchasing cars, receiving drivers’ licenses, buying homes, applying for credit cards, and meeting with their children’s teachers.

This detailed documentation of diverse outcomes has bolstered the ILC’s fundraising efforts. Statistics gathered by teachers, the guidance counselor and from student surveys are entered into a data base. Reports can be generated for combined or specific school programs and for the full or partial fiscal year. These reports accompany grant proposals to donors and ILC information packets to the media and other interested parties. They also are shared with the ILC Board of Trustees and staff.

Cost and Staffing

The ILC employs one full-time guidance counselor who provides student support services. She is responsible for all aspects of initial placement and testing, m aintaining student records as well as providing follow-up on attendance problems. She helps students to meet their identified long-term career and academic goals. She also c onducts in-class workshops on employment and health issues, and refers students to appropriate career, health and educational services.

The total cost of the program assessment and evaluation component is approximately $62,000 per year of which over 80% goes toward the full-time guidance counselor’s salary. In addition, the longitudinal study and other evaluation procedures cost the agency approximately $10,000 per year.

The ILC draws its funding from a very diverse pool of public and private sources, namely foundations, businesses and individuals. To meet its fundraising goals, the ILC employs a full-time Director of Development/ Community Planner and an Associate Director of Development.

Implications for Practice, Policy and Research

Nationally and locally, there is much debate about best ways to assess student progress. Most assessments do not capture well what is taught. And if they do, they are difficult to aggregate. While the search for the perfect assessment tools and processes continues, publicly funded programs must find feasible and adequate ways of assessing their impact and being accountable to their students and funders. The ILC’s integrated and on-going program evaluation system, which includes a multi-pronged student assessment process, can serve as a model for other adult literacy programs. Programs contemplating the adoption of a similar system would do well to reflect on what those tools do and do not capture, and whether there are other outcomes that are important to the program and the students.

The ILC’s longitudinal student survey as well as its survey of former students is an example of a practice where an individual agency’s research effort might well inform our understanding of the long-term impact of programs such as the Immigrant Learning Center.

A packet of sample intake, assessment and evaluation tools from the ILC is available from NELRC upon request. Email silja_kallenbach@worlded.org.

References

Comings, J., Parrella, A. & Soricone,. S. (2002). Helping adults persist: Four supports in Focus on Basics, Volume 4, Issue A. Boston, MA: National ILC for the Study of Adult Learning and Literacy.
http://www.ncsall.net/index.php?id=332