Flag Counter The field trial

The field trial

As an integral part of the programme, SEA-PLM provides opportunities for national experts who are responsible for the implementation of learning assessment reforms to benefit from an intense training and capacity development. Building from its long-lasting experience and expertise in conducting large-scale international assessments, the Australian Council for Educational Research (ACER) experts provide tailored regional and in-country level capacity trainings in different areas in learning assessments.

Following a collaborative approach with the respective Ministries of Education and staff responsible for implementing learning assessments, the field trial phase of SEA-PLM includes the development of all SEA-PLM instruments and procedures during a series of regional level workshops and meetings. It also includes the field trialing of the SEA-PLM tools and protocols in the participating countries. The field trial phase includes:

 

The Launching Ceremony of SEA-PLM

Hosted by the SEAMEO Secretariat and UNICEF Regional Office for East Asia and the Pacific (EAPRO), the Southeast Asia Primary Learning Metrics (SEA-PLM) Launching Ceremony took place on the 20 October 2014 at the Amari Watergate Hotel, Bangkok, Thailand.

The ceremony was attended by three Ministers of Education (Brunei Darussalam, Cambodia and Lao PDR) and representatives from the MOE Malaysia, Philippines and Thailand, as well as members from SEAMEO Centers, UNICEF representatives and other partners who expressed great enthusiasm for the initiative.

Supported by these six (6) countries as well as by additional countries that expressed interest at a later stage, the Ceremony corresponded to the official launch of all activities in relation to SEA-PLM and included the development of the Assessment Framework as well of the items and protocols in close collaboration with the countries. The process of establishing the National Technical Teams and National Steering Committees was also initiated in preparation for the implementation of all national activities for the field trial of SEA-PLM, e.g adaptation, translation, sampling, test administration, coding, data entry and analysis.

Domain Technical Review Panels (DTRP) Workshop

The Domain Technical Review Panels (DTRP) Workshop took place in February 2015 in Bangkok, Thailand and the main objective of the DTRP meeting was to provide advice on the development of the assessment frameworks for the four domains of SEA-PLM, including a consideration of linkages to national and international assessments in order to ensure regional suitability and acceptability of the frameworks. In order to do so, draft frameworks developed by ACER following review of curriculum documents from countries in the region were presented at the workshop for review and revision. The feedback from the representatives from the SEAMEO Member Countries and Partners was incorporated into the draft frameworks for each domain and submitted to the SEA-PLM Regional Steering Committee.

The following themes in relation to the frameworks were explored:

  • the rationale behind the assessment;
  • the definition of the content areas and competencies that the reporting scales will measure;
  • commonalities and distinctions across the countries, in terms of content, skills and standards;
  • linkages to / relevance of other assessments and reporting mechanisms;
  • the length and structure of the cognitive tests, and the number tasks and forms required in order to ensure adequate coverage of the tested domains;
  • the relationship between and purposes of field trial and main study.

Item Development Workshop

The item development workshop took place in June 2015 in Bangkok, Thailand and was designed to enhance the capacity of participating countries to develop reliable, valid and rigorous assessments for each of the domains, and to ensure that the SEA-PLM assessments reflect the values of the region. Attended by representatives from most SEAMEO Member Countries, the training in test development allowed participants to engage in the development of materials for the SEA-PLM domains through review of ACER developed materials and materials submitted by Member Countries, and further development of stimulus materials and test items. The participants were also introduced to language and translation issues relevant to a regional assessment.

Field Operations and Data Management Workshop

The Field Operations Planning and Data Management Workshop took place in August 2015 in Bangkok, Thailand and was aiming at providing a regional training to the Technical Teams from Brunei Darussalam, Cambodia, Lao PDR and Myanmar concerning all technical aspects for the preparation for the field trial of the SEA-PLM. This included the following objectives:

  1. Training of the country teams specifically in the technical aspects of assessment including standardised field operations, sampling, test administration, and data management.
  2. Finalisation and agreement on the technical standards of the assessment process.
  3. Facilitation of the building of a network of test implementation specialists so that they can share ideas and challenges and can propose solutions together.

The Field Operations and Data Management workshop for the second round of field trials, including Malaysia, Vietnam and the Philippines was organized at the national level and took place in February and March 2017.

Coder Training Workshop

This regional training was aiming at providing the Technical Teams with a training on the coding process of the data collected from the SEA-PLM field trial. It included the following components:

  1. Questionnaire development and coding conventions
  2. Coding open and constructed responses
  3. Standardised marking
  4. Methods to improve rater reliability
  5. Assessing inter-rater reliability

Participants of this training were the Technical Teams of the participating countries, including the Coding Coordinator and the Domain Specialists from Brunei Darussalam, Cambodia and Lao PDR. A similar training was/will be organized at the national level for Myanmar, Malaysia, Vietnam and the Philippines.

Data Entry Workshop

Data Management Workshops were held in each of the participating phase one SEA-PLM field trial countries during September and October 2016. The aim of the five-day workshops was to provide an overview of the KeyQuest Data Management Software, to facilitate data entry activities for both the test and questionnaire data, and to collaborate on processes for updating sampling data and undertaking quality control and data validation procedures.