The field trial

Capacity Building

As an integral part of the programme, SEA-PLM provides opportunities for national experts who are responsible for the implementation of learning assessment reforms to benefit from an intense training and capacity development. Building from its long-lasting experience and expertise in conducting large-scale international assessments, the Australian Council for Educational Research (ACER) experts provide tailored regional and in-country level capacity trainings in different areas in learning assessments.

Following a collaborative approach with the respective Ministries of Education and staff responsible for implementing learning assessments, the field trial phase of SEA-PLM includes the development of all SEA-PLM instruments and procedures during a series of regional level workshops and meetings. These include:

 

Domain Technical Review Panels (DTRP) Workshop

The Domain Technical Review Panels (DTRP) Workshop took place in February 2015 in Bangkok, Thailand and the main objective of the DTRP meeting was to provide advice on the development of the assessment frameworks for the four domains of SEA-PLM, including a consideration of linkages to national and international assessments in order to ensure regional suitability and acceptability of the frameworks. In order to do so, draft frameworks developed by ACER following review of curriculum documents from countries in the region were presented at the workshop for review and revision. The feedback from the representatives from the SEAMEO Member Countries and Partners was incorporated into the draft frameworks for each domain and submitted to the SEA-PLM Regional Steering Committee.

The following themes in relation to the frameworks were explored:

  • the rationale behind the assessment;
  • the definition of the content areas and competencies that the reporting scales will measure;
  • commonalities and distinctions across the countries, in terms of content, skills and standards;
  • linkages to / relevance of other assessments and reporting mechanisms;
  • the length and structure of the cognitive tests, and the number tasks and forms required in order to ensure adequate coverage of the tested domains;
  • the relationship between and purposes of field trial and main study.

Item Development Workshop

The item development workshop took place in June 2015 in Bangkok, Thailand and was designed to enhance the capacity of participating countries to develop reliable, valid and rigorous assessments for each of the domains, and to ensure that the SEA-PLM assessments reflect the values of the region. Attended by representatives from most SEAMEO Member Countries, the training in test development allowed participants to engage in the development of materials for the SEA-PLM domains through review of ACER developed materials and materials submitted by Member Countries, and further development of stimulus materials and test items. The participants were also introduced to language and translation issues relevant to a regional assessment.

Field Operations and Data Management Workshop

The Field Operations Planning and Data Management Workshop took place in August 2015 in Bangkok, Thailand and was aiming at providing a regional training to the Technical Teams from Brunei Darussalam, Cambodia, Lao PDR and Myanmar concerning all technical aspects for the preparation for the field trial of the SEA-PLM. This included the following objectives:

  1. Training of the country teams specifically in the technical aspects of assessment including standardised field operations, sampling, test administration, and data management.
  2. Finalisation and agreement on the technical standards of the assessment process.
  3. Facilitation of the building of a network of test implementation specialists so that they can share ideas and challenges and can propose solutions together.

The Field Operations and Data Management workshop for the second round of field trials, including Malaysia, Vietnam and the Philippines was organized at the national level and took place in February and March 2017.

Coder Training Workshop

This regional training was aiming at providing the Technical Teams with a training on the coding process of the data collected from the SEA-PLM field trial. It included the following components:

  1. Questionnaire development and coding conventions
  2. Coding open and constructed responses
  3. Standardised marking
  4. Methods to improve rater reliability
  5. Assessing inter-rater reliability

Participants of this training were the Technical Teams of the participating countries, including the Coding Coordinator and the Domain Specialists from Brunei Darussalam, Cambodia and Lao PDR. A similar training was/will be organized at the national level for Myanmar, Malaysia, Vietnam and the Philippines.

Data Entry Workshop

Data Management Workshops were held in each of the participating phase one SEA-PLM field trial countries during September and October 2016. The aim of the five-day workshops was to provide an overview of the KeyQuest Data Management Software, to facilitate data entry activities for both the test and questionnaire data, and to collaborate on processes for updating sampling data and undertaking quality control and data validation procedures.