Methodology

SEA-PLM is a common methodological approach to assessing learning outcomes of primary Grade 5 students in Southeast Asia. The assessment framework and tools are designed to meet international best practice and technical standards in large-scale assessment, delivering high quality and reliable data. 

Methodology, instruments, procedures and quality control processes are produced through a collaborative process involving countries and external assessment experts. Phases include framework and item development, translation and verification, sampling, data collection operations, data processing, scaling, data analysis and reporting of results.

The first assessment round, SEA-PLM 2019, is being implemented in six participating countries from Southeast Asia. Scientific expertise, technical support and quality control for the design and implementation of SEA-PLM 2019 is managed by the Australian Council for Educational Research (ACER).

Scientific protocol and survey parameters are outlined and reported for each round of SEA-PLM in the framework documentation, technical report, database, data user manual and released instruments. While main assessment methodology remains consistent from one round to the next, possibilities exist to revise, develop and explore new domain areas and techniques during a new assessment round. 

implementation process

About the methodology
  1. Tests and Questionnaires
  2. Sampling
  3. Data Collection and Coding
  4. Data Processing and Analysis
  5. Results Reporting

SEA-PLM develops cognitive and contextual questions on the basis of a conceptual framework designed by experts and countries. The framework defines the contents to be measured and is contextually suited to the region, national curricula and current research in education. Framework and instruments are designed to be used by all participating countries and between different assessment rounds allowing for comparison of trends over time.

SEA-PLM 2019 covers four content domains: mathematics, reading, writing, and global citizenship. For each of the domains, one review panel was established to advice the conceptual development of the framework based on a curriculum mapping of ASEAN member countries.

calculatingSEA-PLM Mathematical literacy is a person’s capacity, given a problem in a context that is of interest or importance to them to translate the problem into a suitable mathematical formulation, to apply mathematical knowledge and skills to find a solution, and to interpret the mathematical results in relation to the context and to review the merits or limitations of those results.
calculatingSEA-PLM Reading literacy is a person’s capacity, understanding, using and responding to a range of written texts, in order to meet personal, societal, economic and civic needs.
calculatingSEA-PLM Writing literacy is a person’s capacity, constructing meaning by generating a range of written texts to express oneself and communicate with others, in order to meet personal, societal, economic and civic needs.
calculatingSEA-PLM Global Citizenship literacy is a person’s attitude, appreciating and understanding the interconnectedness of all life on the planet to act and relate to others with this understanding to make the world a more peaceful, just, safe and sustainable place.

The instruments also include background questionnaires that gather important information from students, parents, teachers and school principals.

Questions and coding materials are developed by countries and experts on the basis of the SEA-PLM assessment framework to measure specific content and cognitive domain. All instruments are experimented during field trial operations on small samples of schools, ensuring that questions function well across the different participating countries and language of instruction, and to maintain consistency and reliability to the methodology.

SEA-PLM sampling obtains accurate estimates of Grade 5 learning outcomes in the Southeast Asian region. Sampling is consistent across countries and accommodates requested local and national objectives. Assessment data is collected from a representative sample of the national target population. This population is defined by UNESCO (2012) as “all students enrolled in the grade that represent five years of schooling counting from the first year of ISCED Level 1.” This is referred to in SEA-PLM as Grade 5.

All participating countries apply the same sampling procedures, which consists of the defining of national target population, construction of the sampling framework, definition of stratification variables, sampling of schools and classes and the allocation of booklets between students in a class.

SEA-PLM sampling has two stages:

1. Schools are selected following a systematic procedure whereby selection probability is proportional to the number of enrolled Grade 5 students. A minimum of 150 schools are sampled from each participating country.

2. In each sample school one Grade 5 class is selected at random. All students of the selected class participate in the assessment.

The final SEA-PLM booklets are administrated in the language(s) of instruction of each country to students via paper-pencil testing. Testing period is one hour, with students completing a series of multiple-choice and open-ended questions. The assessment uses a rotating booklet design (total of 18 cognitive booklets) that includes 30 minutes for each of the three learning areas (reading, writing and numeracy). The same instructions are provided to all students at the beginning of the test. As well as the test students, parents, teachers and principals complete contextual questionnaires to gather background information.

Standardized guidelines and instructions guide countries at each stage of the assessment administration. Countries are responsible for training government staff, school coordinators and test administrators, ensuring that the assessment standards are applied consistently. Quality control of the assessment is undertaken by government and external observers.

Data collected during the field trials and main survey follows the same process of entry, cleaning, verification, scaling and analysis. This process allows for the construction of valid measurements and datasets, including the generation of variance estimations and computing estimators of descriptive and factorial analysis.

Final reports are prepared through a collaborative process involving countries and experts. Results are reported in regional, national and specific reports (thematic report, policy brief and research paper). Researchers and education stakeholders are consulted to interpret and comment on the results, formulating messages and recommendations based on the data.