The Effect of National Examination and Curriculum on Students’ Performance in Solving TIMSS Model Mathematics Problems

This study was aimed to investigate effect of National Examination (UN) and curriculum also their interaction effect on students’ performance in solving TIMSS model Mathematics problems. This study was carried out with 300 8 graders of East Kalimantan students. The participants were from three mathematic National Examination result school categories on 2016 (low, moderate and high) and two implemented curriculum types (KTSP and Currciculum 2013). Data collection used 28 items TIMSS model Mathematics problems (α = 0.837) which contained content and cognitive domain. Data analysis used two ways analysis of variance. Data analysis revealed that there were significance main effect of national examination and curriculums, also interaction effect between of them on mathematic achievement. There were significance main effect of national examination and curriculums on content domain. But, there was no significance interaction effect between of them on performance in solving geometry. There were significance main effect of national examination and curriculums on cognitive domain. But, there was no significance interaction effect between of them on knowing thinking level.


INTRODUCTION
Globalization and science technology development in 21 st century strived international citizen for mastering some knowledges and skills, one of them is mathematics. National Council of Teacher of Mathematics (2000) explained that, In this changing world, those who understand and can do mathematics will have significantly enhanced opportunities and options for shaping their futures. Mathematical competence opens doors to productive futures. A lack of mathematical competence keeps those doors closed. In order to study and canal information about citizen readiness to overcome the challenges, there are many international studies about mathematics performance. One of them which participated by Indonesia is Trends in International Mathematics and Science (TIMSS).
TIMSS is a study which measures achievement of mathematics and science for 4 th and 8 th grade (Mullis, Martin, Ruddock, O'Sullivan and Preuschoff, 2009). The purpose of TIMSS is to inform about educational policies of participant countries. Yilmaz and Hanci (2015) stated that TIMSS study collects comparative data about national education systems so they contribute to improvement of education, especially in mathematics and science. The statement is parallel to Martin and Mullis (2006) that TIMSS was designed to collect information that could be used to improve mathematics and science learning in the world. By collecting data periodically, TIMSS present trend of education. The monitoring extends important information to develop educational policies, improve people accountability and identify increasing or decreasing of achievement.
Term of "achievement" relate to students' achievement in mastering several abilities in a subject which has formulated in curriculum (Mullis, et. al, 2009). Assessment of the achievement use problems which have developed based on the framework. The assessment framework was divided into two domain, conten and cognitive (Mullis et al, 2009). Content domain is collection of mathematical content which assessed by TIMSS, they are number, algebra, geometry and data & chance. Cognitive domain concern to skills in thinking process, from lower order to higher order (Budiman & Jailani, 2014), which include knowing, applying and reasoning.
Mathematics performance of Indonesian student are not only assessed by international assessment, but also national examination. National examination is national assessment that measure compentencies achievement in several subjects and referred to graduate competency standards (re: Standar Kompetensi Lulusan, SKL) (Badan Standar Nasional Pendidikan, 2017). SKL are parts of educational curriculum of Indonesia. Thus, based on frameworks problems, both national examination and TIMSS referred to educational curriculum. It means that mathematics performance which assessed by national examination related to TIMSS, although both of them have different type of problems (Stacey, 2011;Carnoy, et al., 2016).
Since 2014, TIMSS model Mathematics problems started embedded to national examination in order to improve mathematic achievement internationally. But, the result of mathematics national examination was not improved, year by year, even it was declined (Retnawati, Kartowagiran, Arlinwibowo and Sulistyaningsih, 2017). It was parallel to Indonesian mathematics achievement in TIMSS, which was still low and has not get significant improvement (Balitbang Kemendikbud, 2011;Mullis, Martin, Foy, and Arora, 2012). Thus, it assumed that there is a relationship between students' performance in solving mathematics national examination and mathematics achievement in TIMSS.
Still, mathematics achievement also affected by another factors, which come from school or student' characteristics. Several studies (ex. Carnoy, et.al, 2016;Lam and Lau, 2014;McConney and Perry, 2010;Jerrim, Lopez-Agudo, and Marcenaro-Gutierrez, 2020;Yun, 2019) imply that several factors which relate characteristics and quality of school and student contribute to students' performance. Study of Carnoy, et.al (2016) found that learning opportunity and teaching quality improvement generate vary effect to mathematics achievement, it depends to parents education level and mathematics school performance. One of Lam and Lau (2014) findings is number of student in a class and socio-economic status was strong predictor of PISA score. On the contrary, Ghasemi and Burley (2019) found that socio-cultural, politics, equity have not affected to mathematics performance. NCTM (2000) stated that main factor determined mathematics performance is applied curriculum. Scheerens (2002) also claimed that education effectivity which related to students achievement can be explained by three main factors, one of them is learning effectivity which related to curriculum. Curriculum consists of three aspects, there are the intended curriculum, the implemented curriculum and the achieved curriculum (Martin and Mullis, 2006). Thus, mathematics performance affected by planning, implementing and assessing through mathematics competencies which expected to be mastered by students.
At Indonesia, there are still different curriculum which implemented by schools, KTSP (School Based Curriculum) and Kurikulum 2013 (K13). Besides the emphasis on knowledge and skills, the difference between KTSP and K13 also lies in the learning approach and assessment system (Retnawati, Hadi, and Nugraha, 2016). K13 emphasizes student-centered learning rather than teacher-centered learning. In assessing students' performance, KTSP use tests that emphasize more aspects of knowledge and skills. Meanwhile, K13 use authentic assessments that measure students' attitudes, knowledge, and skills based on the learning process and outcomes. In addition, K13 learning promotes Higher Order Thinking Skills (HOTS) more than KTSP (Suryapuspitarini, Wardono, & Kartono, 2018).
Indonesian curriculum changing from KTSP to K13 was based on the low ranking of Indonesian students on PISA and TIMSS (Kemendikbud, 2012). That is, the implementation of the K13 aimed to improve students' mathematical performance internationally. Thus, the implementation of K13 should affect a positive influence on students' mathematics performance than KTSP. However, yet found studies that evaluating K13 in improving the ranking of Indonesian students in PISA and TIMSS.
Therefore, it is necessary to investigate the effect of the National Examination and curriculum on students' performance in solving TIMSS model Mathematics problems. This study was aimed to investigate effect of National Examination (UN) and curriculum also their interaction effect on students' mathematics achievements, that is students performance in solving TIMSS model Mathematics problems, both in general and based on domain. This study is different from previous TIMSS studies in Indonesia which still only describe students performance in solving TIMSS problems, but have not investigated the factors that affect it specifically.

METHOD
This study was a survey research. This study was conducted to 12 Junior High Schools distributed over East Kalimantan. This study was carried out with 300 8 th graders of East Kalimantan students. The participants were from three mathematic National Examination result school categories on 2016 (low, moderate and high) and two implemented curriculum types (KTSP and K13). Gender based distribution was fairly balanced, that is 151 boys and 149 girls.
Data collection used 28 items TIMSS model Mathematics task. The task was design based on TIMSS framework which contained content and cognitive domain. Content domain consist of number, algebra, geometry and data & chance. Cognitive domain consist of knowing, applying and reasoning. The task has been validated by expert judgement and restricted trial. After validation, the task has been revised based on expert suggestions. The trial found that the reliability estimation was 0.837. The measure meet the minimum standard reliability instrument to measure group of individuals (Ebel & Frisbie, 1991).
The obtained data was score test then converted to percentage of correct answer, both the TIMSS score in general and based on cognitive and content domain. The data were described by average percentage of correct answer based on National Examination and curriculum groups. Furthermore, the data were analyzed using two-way analysis of variance to investigate: 1) the effect of the National Examination, curriculum and their interactions on mathematics achievement in general; and 2) the effect of the National Examination, curriculum and their interaction on mathematics achievement based on content and cognitive domains. Post hoc test was conducted if the main effect was significant, both in general and based on cognitive and content domain.

RESULTS AND DISCUSSION Results
Generally, the average percentage of correct answers to the TIMSS model Mathematics problems is 27.76 with a standard deviation of 19.85. This percentage is relatively low because it is below to the 2015 international TIMSS average (Mullis, Martin, Foy, and Hooper, 2016). The highest percentage is 96.77, while the lowest is 3.23. The average percentage of correct answers to the TIMSS model Mathematics problems based on the National Examination results category and the implemented curriculum is shown in Table 1.  Based on the results of the analysis of variance (Table 2), the National Examination, curriculum and interaction both of them have a significant effect on the percentage of correct answers to the TIMSS model Mathematics problems in general. Based on Table 1 and Table  2, the percentage of correct answers to the TIMSS model Mathematics problems done by students from schools that have implemented K13 is significantly higher than students from schools that have implemented KTSP. Post hoc test results of the National Examination effect on the percentage of correct answers to the TIMSS model Mathematics problems are shown in Table 3.  Table 3 shows that the percentage of correct answers to the TIMSS model Mathematics problems done by students from high school category was significantly higher than students from moderate and low school category. While the percentage of correct answers to the TIMSS model Mathematics problems between students from the moderate and low category schools did not differ significantly, although the students percentage of correct answers from the low schools was higher than moderate schools.
In terms of the content domain, the average percentage of correct answers to the TIMSS model Mathematics problems is shown in Figure 1. It shows that the average percentage of correct answers to the TIMSS model Mathematics problems for each content domain is below to the 2015 TIMSS international average.  The average percentage of correct answers to the TIMSS model Mathematics problems for each content domain, the National Examination results category and the implemented curriculum also the analysis of variance are shown in Table 4 and Table 5. Based on the results of analysis of variance (Table 5), the National Examination and curriculum have a significant effect on the performance in solving each content domain, but the interaction of them did not significantly affect to the performance in solving geometry problems. Based on Table 4 and Table 5, the percentage of correct answers to the TIMSS model Mathematics problems done by students from schools which implemented K13 was significantly higher than students from schools which implemented KTSP for each content.  Table 6. It shows that the percentage of correct answers to the TIMSS model Mathematics problems done by students from high school category was significantly higher than students from moderate and low school category for each content domain. Meanwhile, the percentage of correct answers to the TIMSS model Mathematics problems between students from the moderate and low category schools did not differ significantly for each content domain, although the students percentage of correct answers from the low schools was higher than moderate schools. In terms of the cognitive domain, the average percentage of correct answers to the TIMSS model Mathematics problems is shown in Figure 2. It shows that the average percentage of correct answers to the TIMSS model Mathematics problems for each cognitive domain is below to the 2015 TIMSS international average.

Hamidy et al
The average percentage of correct answers to the TIMSS model Mathematics problems for each cognitive domain, the National Examination results category and the implemented curriculum also the analysis of variance are shown in Table 7 and Table 8. Based on the results of analysis of variance (Table 8), the National Examination and curriculum have a significant effect on the performance in solving each cognitive domain, but the interaction of them did not significantly affect to the performance in solving knowing problems. Based on Table 7 and Table 8, the percentage of correct answers to the TIMSS model Mathematics problems done by students from schools which implemented K13 was significantly higher than students from schools which implemented KTSP for each cognitive domain.  Table 9. It shows that the percentage of correct answers to the TIMSS model Mathematics problems done by students from high school category was significantly higher than students from moderate and low school category for each cognitive domain. Meanwhile, the percentage of correct answers to the TIMSS model Mathematics problems between students from the moderate and low category schools did not differ significantly for each cognitive domain, although the students percentage of correct answers from the low schools was higher than moderate schools.

Discussion
The overall TIMSS score shows that the mathematics achievement of East Kalimantan junior high school students is relatively low. Even so, the score was higher than the TIMSS score of Indonesian junior high school students in TIMSS 2011 (Mullis, Martin, Foy, and Aurora, 2012). In term of content and cognitive domains, students' mathematics achievements are still below the international average. Students' mathematics achievement which are below to the international average shows that most students have not yet reached the mathematical competencies formulated in the international curriculum.
The low mathematics achievement could be caused by various factors, include those from schools, students and environment. These various factors can affect mathematics achievement in general and specifically to the content and cognitive domains. The findings showed that the National Examination had a significant effect on students' mathematics achievements both in general and in terms of the content and cognitive domains. That is, the performance in solving the National Examination problems is an important factor that affect students' success in solving TIMSS problems. This is caused by the suitability of the mathematics problems characteristics in the National Examination and TIMSS, both in terms of the scope of the content and the level of thinking (Stacey, 2011;Carnoy, et.al, 2016).
Based on the post hoc tests, it was found that student success in the National Examination had a positive effect on students' mathematics achievement. It means that students from schools with a high National Examination category achieved higher mathematics achievements than students from the low and moderate school categories. So, if the higher the National Examination score in mathematics then the higher student's mathematics achievement in TIMSS. This is in line with the findings of Yilmaz and Hanci's (2015), that the higher the mathematics scores of students in school, the higher the TIMSS score that students can achieve. Thus, embedding TIMSS type problems into National Examination test is an appropriate effort to learn students in improving their mathematical abilities internationally (Retnawati, Kartowagiran, Arlinwibowo and Sulistyaningsih, 2017).
However, a significant positive effect was not shown between students from the moderate and low National Examination school categories. In each content and cognitive domain, the students performance from low National Examination school categories consistently higher than students from moderate National Examination schools, although it is not significant. It showed that there is a large performance gap between students from schools with a high National Examination category and students from schools with moderate and low National Examination categories.
Furthermore, the findings showed that the curriculum implemented by the school affected students' mathematics achievements in general and specifically in the content and cognitive domains. That is, the curriculum is also an important factor that affects students' mathematics achievement in TIMSS. This is in line with the purpose of TIMSS, which is to measure student achievement in mastering various abilities that have been formulated in the education curriculum (Mullis, et.al., 2009). Thus, mathematics achievement reflects the effectiveness of the curriculum applied (NCTM, 2000;Scheerens, 2002). The effectiveness of the curriculum includes the range of content and competencies to be achieved, the method of its implementation, as well as the form of evaluation and assessment of the competency outcomes that have been formulated.
Specifically, the findings also showed that the students mathematics achievements from schools which implemented K13 were significantly higher than students from KTSP one, both in general and based on content and cognitive domains. This confirms the assumption that have been proposed previously, that aspects, approaches and assessment methods developed in K13 are oriented towards students' preparation in facing global challenges that appropriatewith the objectives of TIMSS (Retnawati, Hadi, and Nugraha, 2016;2013 Curriculum Documents;Mullis, et.al., 2009). So, the scope of the content and the development of thinking skills in TIMSS are more accommodated in K13 than KTSP.
Although it is more in line with the TIMSS curriculum, K13 policy has not succeeded in improving Indonesian students mathematics achievements at TIMSS to come out from the bottom rank internationally. This is due to the implementation of K13 which is not yet appropriate and optimal (Jaedun, Hariyanto, and Rahardjo, 2014;Trisnawati, Gunawan, and Nongkeng, 2017), especially HOTS learning. Retnawati, Djidu, Kartianom, Apino, and Anazifa (2018) found that teachers' understanding and ability to teach, assess and solve HOTS problems was still low. HOTS questions have also not been introduced to students evenly (Witri, Putra, and Gustina, 2014). In addition, efforts to improve students' mathematical achievements internationally are not enough just to reform the education curriculum. Saoudi, Chroqui, and Okar, (2019) explained that other efforts needed to improve student achievement internationally, including the recruitment and training of teachers in order to achieve minimum standards of qualification, involving various stakeholders in formulating educational programs that fit the needs, school systems that reflect the real world, focus on aspects of learning and build a culture of evaluation and accountability.
The interaction between the factors of the National Examination and the implemented curriculum also has a significant effect on students' mathematics achievement in general. This significant interaction is similar to the findings by Indartono and Hamidy (2019) that curriculum differences have a moderating effect on the effect of National Examination results on international test results such as PISA and TIMSS. This is due to the National Examination which is still related to the curriculum, which is part of a centralized assessment by the government that refers to certain graduation standards (BSNP, 2017). Thus, the differences in the implemented curriculum by schools in Indonesia also have an impact on the results of the National Examination that they obtained, considering that the National Examination is still enforced equally to schools that implement either KTSP or K13.
In terms of content domain, the interaction between the factors of the National Examination and curriculum has no significant effect on the content of the geometry. It occurred because there are no differences in the scope of geometry content between those formulated in the curriculum and those tested on the National Examination (Permendiknas No. 22/2006;Permendikbud RI No. 24/2016). Whereas in other content domains, there is a change in content coverage both in K13 compared to KTSP and in the National Examination problems.
In terms of cognitive domain, the interaction between the factors of the National Examination and curriculum has no significant effect on the level of thinking knowing. It shows that the ability to think low level has been accommodated in the curriculum and the National Examination. While the abilities to think at a higher level are more emphasized on K13 than KTSP. In addition, the ability to solve problems with a higher level of thinking is the key to students' success in achieving high scores in National Examination and TIMSS.

CONCLUSION
Based on the findings and discussion, it concluded that: 1) National Examination and curriculum as well as the interaction of both of them significantly affect students' mathematics achievement; 2) National Examination and curriculum have a significant effect on the ability for each content domain, but the interaction between them did not significantly affect to the ability to solve geometry problems; 3) National Examination and curriculum have a significant effect on the ability for each cognitive domain, but the interaction of them did not significantly affect to the ability to know (knowing).

RECOMMENDATION
The results of the study have implications for improving the quality of the National Examination and education curriculum in Indonesia in order to improve the mathematics achievement of Indonesian students at the international level. On the one hand, the sample used in this study is different from the sample used by TIMSS to assess the mathematics achievements of Indonesian students. So that the comparison of performance which presented between East Kalimantan and Indonesian students is not entirely equivalent. Therefore, it is necessary to study the data sources and sampling techniques used in TIMSS for next subsequent research.