Vertex

The Arlington Education and Employment Program (REEP) in Arlington, Virginia, staff has developed a writing assessment for adult English language learners. The REEP Writing Assessment (RWA), as it has come to be known, is one of five assessment instruments approved by the Commonwealth of Virginia for English as a Second Language (ESOL) programs to use for pre- and post-testing for the National Reporting System (NRS)2.

The RWA is a performance-based writing test that measures writing gains for adult English language learners at NRS Beginning ESL through High ESL Advanced. The assessment consists of the REEP Writing Rubric3 and timed performance tasks (writing prompts). Students respond to a written prompt, which is scored using the Rubric. The Rubric articulates six performance levels across five specific dimensions of writing.

Why would REEP, a local service provider, get involved with test development of this kind? What does it take to develop a valid and reliable test?

 Who We Are 4

The Arlington Education and Employment Program (REEP) is an adult ESOL program within the Arlington Public Schools in Arlington, Virginia. Arlington, because of its close proximity to our nation’s capitol, has drawn large numbers of immigrants seeking jobs in our service industry and supporting our domestic and international organizations. In fact, immigrants comprise 23 percent of the total adult population in Arlington.

The REEP Program’s mission of providing for the education- and employment related needs of limited English proficient adult immigrants and refugees is accomplished through various educational components: intensive and non-intensive English as a Second Language (ESL), family literacy, workplace literacy, and computer- assisted instruction. Nine levels of ESL instruction are offered. Over 8,000 enrollments are provided annually at 14 locations throughout Arlington County. Fifty-five trained and experienced ESL teachers, who are supported by instructional coordinators, provide the instruction, and over 150 volunteers support instruction throughout the county.

Using a whole language approach, all levels of the REEP Curriculum focus on the speaking, listening, reading, and writing skills that adults need in their roles as workers, community members, and family members. Students vote on life skills topics such as health, employment, banking, children’s education, and transportation to determine the context through which they will develop these language skills. While students are learning language skills, they are also learning to access, evaluate, and use information and resources in a variety of life skills contexts and in a variety of ways, including computers and the Internet.

 Our Challenge

Performance-based assessments5 are popular because they are often program-based and learner-centered; however, funders tend to question their credibility. At REEP, we challenged ourselves to address this issue by finding a way to satisfy technical quality issues, such as validity and reliability, while also keeping in mind how assessment influences learning. We believed that this two-part approach would facilitate reporting student achievement both fairly and credibly.

 Our Story

In 1995, REEP staff started with the development of a writing rubric6. Staff was satisfied that the oral assessments we were using effectively placed students into our 9 levels of instruction with regard to their spoken ability. However, with regard to written skills, we hoped to fine-tune our placement of learners into appropriate class levels by articulating what our adult ESL learners could do in written English at various proficiency levels. The need for a more sensitive placement instrument was particularly apparent at our high intermediate through advanced levels, where there is a greater emphasis on developing writing skills.

Our rubric was developed by collecting multiple and extensive writing samples from each class level and analyzing them. We found that although we had nine instructional levels, our students’ writing fell into six distinct writing performance levels. The differences in these levels could be articulated using five characteristics (learning targets) of our learners’ writing: content and vocabulary, organization and development, structure, mechanics, and voice.

With a solidly crafted rubric in place, we were eager to determine its effectiveness. As part of our work with the What Works Literacy Partnership (WWLP)7, we designed and implemented a study to determine the effectiveness of using the REEP Writing Rubric to measure progress. With support from WWLP, we developed pre- and post-test writing tasks to assess writing gains.

Developing writing tasks that could be used for program-wide testing of beginning through advanced level students was challenging. The tasks needed to generate a wide variety of responses and enable students at different levels to demonstrate their abilities and life experiences. We decided that the performance task of writing a letter of advice based on their own experiences would meet the above criteria and be consistent with skills that students were practicing in class. Moreover, we structured the testing process to mirror instructional practice by engaging students in warm-up activities prior to the actual writing test. Warm-up activities include group brainstorming about the topic and pair conversation that focuses on the topic of the writing.

 What Works in Test Development

One of our main concerns in testing out the REEP Writing Assessment rubric was to determine its reliability. Reliability of test data is extremely important, especially in the context of reporting results to funders. To maximize the reliability of our results, WWLP researchers provided extensive guidance on field-testing, test administration procedures, scoring, performance task development, and rater training.

 Field-testing

Before administering the pre- and post-writing tests to hundreds of students, we conducted field-testing to answer the following questions:
1. Can we expect measurable progress within the specified test interval (120-180 hours of instruction)?

2. Can beginning through advanced level students demonstrate their writing skills in response to our writing tasks?

3. Are the pre- and post-test tasks equivalent? Do they represent the same level of difficulty?

To answer questions 1 and 2, a small group of experienced teachers administered the pre-test to approximately 100 REEP students, representing eight of the nine class levels8 at the beginning of an instructional cycle. At the end of the 12-week cycle of 120-180 hours of instruction, the teachers administered the post-test to the same group. Students were asked for feedback, and they responded that they felt they were able to demonstrate their writing skills with these tests. Teachers also thought that the tests demonstrated the students’ writing abilities. Experienced readers scored the tests, and then a WWLP researcher analyzed the results. The analysis showed that significant gains could be measured and that reliable results could be achieved using the scoring procedures we had implemented. With these promising preliminary findings, we were ready for large-scale testing.

To answer question 3, the same group of REEP students (5-6 from each class level) was given the pre-test followed by the post-test within a three-day period. A WWLP researcher analyzed the results and found no difference between students’ pre- and post-test scores, which demonstrated that the two tasks represented the same level of difficulty. One of the key elements in achieving equivalence was the use of the letter genre and parallel warm-up activities for both the pre- and post-tests.

 Test administration training

Prior to each test administration, testers participated in training on the testing procedures, including time limits, rules (such as no dictionaries), and how to conduct warm-up activities developed for the particular writing task. This ensured that all students completed the pre-writing activities and the test in a uniform way, thereby increasing the reliability of the REEP Writing Assessment.

 Determining scoring procedures

Each of the five writing characteristics receives a score between 0 and 6, with 6 the highest. The total score is determined by adding each characteristic score and dividing by 5. A sample scoring grid follows.

 Building scoring consensus

REEP staff were trained to use the writing rubric to score the performance tasks. Readers scored a range of essays. Scores for each writing characteristic were charted out as shown above, and the scoring rationale was discussed. (For an explanation of skills and scores, see the RWA Rubric.) This enabled the trainers to see how consistently the rubric was being interpreted, pinpoint areas of discrepancy, and build scoring consensus.

A shortened version of this reader calibration process was repeated prior to each scoring session9 to ensure continued consistency in rubric interpretation and scoring. Consistency among the readers was tracked to determine how many tests needed a third reader.

Each test was scored by two readers, and a third reader was used if the total score was more than one point different. Through the use of coded scoring, the second reader did not know how the first reader had scored the test. In this way, the first reader’s score did not influence the second reader. Similarly, students’ class levels were not indicated on the test paper.

Also in the interest of scoring consistency, scoring of the tests occurred in group sessions no longer than two hours each. This seemed to be the point at which readers began to “burn out.”

The training and scoring procedures described above resulted in an inter-rater reliability of 98 per cent. Only two per cent of the tests needed a third reader.

 Lessons Learned

The development of the REEP Writing Assessment taught us valuable lessons about the merits of working from the ground up. REEP students provided the data on which the rubric is based. REEP teachers were involved in every step of the RWA development process: creating writing tasks and warm-up activities, administering tests, developing scoring procedures, scoring tests, and analyzing data. Through this involvement, teachers developed a deeper appreciation of testing. They used their students’ test results to adjust their instruction so that they could better meet the needs of their students. Scoring tests written by beginning to advanced level students gave them a broader picture of writing levels within the program and informed their decisions about subsequent class placements.

The benefits of the RWA did not stop with the teachers. Teachers shared the writing rubric with their students, giving them a better sense of how they were being evaluated. Students at all levels started paying more attention to their writing as a result of the more formalized writing test. Many began to embrace writing instruction in the classroom. Learning English now meant more than learning to “speak” English.

We all gained a greater understanding of the testing process and the importance of showing that the assessment process is both fair and credible to all stakeholders. By participating in the test development process, the teachers developed skills and knowledge that also enable them to develop performance-based classroom assessments.

 Validation of the REEP Writing Assessment

Over a period of several years (1997-2002) and with assistance and guidance from WWLP researchers and researchers involved in further studies (see below), the assessment as well as test administration and scoring procedures studies were validated. The researchers concluded that:

• The REEP Pre and Post Writing Assessment (RWA) is a carefully designed and validated instrument with sufficiently high reliability.
• Writing gains can effectively and reliably be measured within a 12-week period of 120-180 hours of instruction,
• High inter-rater reliability can be achieved with the scoring and training procedures used, and
• Forms A, B, C, and D are equivalent forms of the test.

In 2002, CASAS selected REEP to participate in a study to show the appropriateness of the CASAS and REEP writing tests for adult ESOL programs and to demonstrate the validity of the two assessments. In their report, CASAS researchers concluded that: “…this study presents strong evidence for the validity of both the CASAS and REEP writing assessment rubrics for use with ESL students. Based on the observed correlations, the two rubrics and prompts are interchangeable.”

 Taking the Show on the Road

 Commonwealth of Massachusetts

In 2002, the Commonwealth of Massachusetts adopted the REEP Writing Assessment statewide. Massachusetts' Adult ESOL programs use our instrument for ESOL students (SPL 2+) to measure educational gains for the NRS. REEP staff provided training to practitioners and trainers and continue to provide technical assistance for administration and scoring. REEP and Massachusetts continue to collaborate on developing, piloting, and validating additional test prompts.

Our experience with Massachusetts enabled us to study the effectiveness of our instrument in program types and settings different from ours. It gave us the opportunity to think through procedures for programs with less intensive instruction, for open-entry programs, for very small programs, etc.

 Commonwealth of Virginia

Through a current EL/Civics grant, we double tested approximately 400 students representing multiple proficiency levels with both the BEST Plus and REEP Writing Assessment so that we could make an informed decision about which test to administer to which levels in the future. We found that both tests captured language learning gains at all levels, that students enjoyed taking both tests, and that the test training time was comparable for both. In the end, consideration of the students' language learning goals played a determining role in our decision. We now administer the BEST Plus10 to our literacy and beginning level students and the REEP Writing Assessment to our high intermediate and advanced level students. We continue to administer both tests to the intermediate level students since their goals tend to focus on both speaking and writing development.

 For More Information

If you would like more information about our story, our test, or the appropriateness of this instrument for your program, contact:
Suzanne Grant, Director
Arlington Education and Employment Program (REEP)
Office of Career, Technical, and Adult Education
Arlington Public Schools
2801 Clarendon Boulevard, #218
Arlington, VA 22201
703-228-4204
Sgrant@arlington.k12.va.us

If you would like information about scheduling a training in your area, please contact:
Nancy Faux
ESOL Specialist
Virginia Adult Learning Resource Center
Virginia Commonwealth University
Richmond, VA 23284-2020
nfaux@vcu.edu
1-800-237-0178