Flipped Learning Instruction to Enhance University Students ’ Higher Order Thinking Skills

This study aims to shed some light on the significant impact of content videos in flipped learning classroom on students’ higher order thinking skills. The secondary aim of this study is to portray students’ perception toward the practice of flipped learning in assisting their higher order thinking skill development. A total of 36 elementary teacher education students at IAIN Surakarta are involved as participants in this pre-experimental study. Drawing on pre-test, post-test, and focus group discussion, the results of paired sample ttest show a significant difference on the students’ score of pre-test and post-test, meaning that flipped learning with content videos is effective to enhance the students’ higher order thinking skills. Furthermore, the focus group discussion reveals that the students positively perceive the practice of flipped learning instruction to enhance their higher order thinking skills. These results contribute to providing fruitful insights for teachers and school administrators as a teaching model to enhance students’ higher order thinking skills. Article History Received: 31-12-2020 2


Introduction
The twenty-first century learning is the witness of the learning process shift from a teacher-centered paradigm into a student-centered model (Hembrough & Jordan, 2020). In a student-centered learning, teachers are urged to design interactive teaching activities which put a lot of exposures on students' shoulders to improve their knowledge and skills . One of the skills required by university students is higher order thinking skill. Higher order thinking is considered as a twenty-first century skill that should be acquired by students, especially in university level (Margana & Widyantoro, 2017). Therefore, developing students' higher order thinking skills becomes a challenge for teachers particularly in terms of designing appropriate classroom activities.
In the era of advanced Information Communication and Technology (ICT), various teaching styles and methods have been invented and offer ample opportunities to solve current challenges in the process of teaching and learning (Satar, 2018). The innovation of student-centered learning activity using content videos represent a novel way of flipped learning (Kay, 2012). The idea of flipped learning was firstly coming up when (Bergmann & Sams, 2012) looked for an effective teaching method for the absent students to have an access on the class lessons and materials. They recorded the lessons and made use of a digital platform to post the recordings online so that the students who did not join the class still be able to follow the learning materials by watching the videos at any time and any place. Then, they found out that the model of learning greatly helped students who could not attend the class. Surprisingly, the students who attended the class could also watch the videos and this activity helped them review and revisit the day's lessons. It initiates the concept of massive integration of digital technology in the process of teaching and learning called flipped learning instruction (Bergmann & Sams, 2012;Chen et al., 2014).
This study comes up with the idea of potential impact of flipped learning instruction on students' higher order thinking skills. Considering that critical thinking is a crucial skill that should be acquired by university students, it is highly necessary to find an alternative way of teaching to assist students develop their higher order thinking ability. Previous research has documented that flipped learning is considered as an effective teaching methods to enhance learning efficacy and students' knowledge, such as language skills (Singay, 2020), communicative competence (Lee & Wallace, 2018), students' active engagement (Roach, 2014), and students' motivation and self-efficacy (Chun, 2014). Therefore, it is such a worth inquiry to examine the significant effect of flipped learning on students' higher order thinking skills.
Research on the empirical correlation between flipped learning and higher order thinking skills have been conducted for years in the various context of education in the world. Alsowat (2016) investigated the effect of flipped classroom teaching model on graduate students' higher order thinking skills (HOTS), engagement, and satisfaction in the Kingdom of Saudi Arabia (KSA). Drawing on an experimental study, the results showed a statistically significant difference between students' achievements in HOTS group and non-HOTS group. This study indicated that flipped learning was an effective method to enhance students' higher order thinking skills. Similarly, (Lee & Lai, 2017) conducted an explanatory study on the role of flipped learning to improve higher order thinking skills of secondary students in Hong Kong. The results conclude that it is possible to improve students' higher order thinking skills using the flipped learning classroom approach of teaching.
In a similar direction, Heo and Chun (2018) examined the students' change of highorder thinking ability after being taught using flipped learning method. Considering Psychological students in South Korea as the respondents, the study found that students were actively participated and engaged in a flipped learning classroom activity, and had the initiative in learning and felt confident with their learning. Confirming the results of Alsowat (2016), Heo and Chun (2018), and Lee and Lai (2017), Annan et al. (2019) shed some light on the effect of student-created content videos in flipped learning on business students' higher order thinking skills. Using an explorative qualitative study, Annan et al. (2019) revealed that flipped learning with student-created content videos is effective as a teaching method to improve students' higher order thinking skills. Also, the students were more engaged and actively participated in learning activities designed using flipped method.
Having explored the above-mentioned previous literature, a research gap is observable. Although research examining the impact of flipped learning on students' higher order thinking skills has been enormously conducted, the use of this blended learning method to improve elementary teacher education students' higher order thinking skills still remains a paucity of evidence. In addition, to the best of the researchers' knowledge, an inquiry on the relation between flipped learning and higher order thinking skills in Indonesian educational context remains unexplored. Therefore, this study aims to close the gap by examining the potential effect of flipped learning instruction on elementary teacher education students' higher order thinking skills. The results of this study contribute as fruitful insights for teachers and university administrators to acknowledge a promising benefit of flipped learning in higher education context, especially to develop students' higher order thinking skills.

Research Method
The present study aims to examine the effect of content videos in flipped learning on the level of university students' higher order thinking skills. An additional objective of this study is to reveal the students' perception about the implementation of flipped learning instruction in their classroom. To deal with the objectives, the pre-experimental research (Phakiti, 2015) was conducted to examine the effect of flipped learning on students' higher order thinking skills. Following the experimental research, a Focus Group Discussion (FGD) was administered to depict an in-depth understanding of the students' perception about the implementation of flipped learning. This study was based in the context of Indonesian university teaching at the department of elementary teacher education of IAIN Surakarta Indonesia. This study was conducted in "Innovation and Method of Teaching" course of third-year students of undergraduate program during September -December 2020. The Innovation and Method of Teaching is a compulsory subject that must be taken by the students with the aim to equip students with the knowledge of various method and innovation in teaching. The participants involved in this study were 36 elementary teacher education students at IAIN Surakarta, consisting of 28 female (78%) and 8 male (22%). The participants' average age was 20.
The instruments used in this study were higher order thinking skill test and Focus Group Discussion (FGD). The first instrument, higher order thinking skill test, was used to examine the participants' thinking skills in terms of evaluation, analysis, and creation. The test was adapted from Annan et al. (2019) and developed by referring to Bloom's taxonomy and only questions representing higher order thinking skills were selected. The test consisted of 30 questions that were equally divided into three levels, i.e. evaluating (10 questions), analyzing (10 questions), and creating (10 questions). Every question was followed by four options as distractors (A, B, C, and D) and there was only one correct answer. The questions forced the participants to demonstrate their best ability to develop new ideas before reaching the answer. For scoring, the correct answer was one point and the false answer was zero. Moreover, detailed investigation on the students' perception about the practice of flipped learning in their classroom was measured using an FGD consisting of 8 students who were conveniently selected based on their active engagement during the classroom activities. Considering the chance for all participants to speak and convey their perception, the FGD last for about 90 minutes. This study employed FGD because it offered the researchers short duration and low budget research in gathering qualitative data (Nugroho, 2020). In addition, this type of data collection provides rich and in-depth results about phenomena being studied since the participants can freely interact with each other in the same time to discuss the same topic (Yin, 2015).
To collect the data, first of all, the higher order thinking skill test was administered as a pre-test in the first meeting of the class. After conducting the pre-test, the participants were taught using flipped learning instruction for 6 meetings. In each meeting, the participants experienced different topics of learning, i.e. Contemporary Teaching and Learning, Communicative Teaching Method, Modern Teaching, Learning Media Development, Innovative Learning Media, and Creating Digital Learning Environment. In the eighth meeting, another set of higher order thinking skill test was administered as a post-test to examine the participants' improvement in terms of their higher order thinking skills after being taught using flipped learning. Following the post-test, the researchers conducted the FGD for 90 minutes to reveal how the students' feel after experiencing the flipped learning instruction. 8 participants of the FGD shared, argued, and discussed their opinion about the type of teaching and learning method that allowed the integration of inside and outside classroom activities.
As for the data analysis, this study invited two raters to evaluate the results of pre-test and post-test performed by the students. The first rater was one of the researchers and the other was a lecturer of Education Management department at IAIN Surakarta Indonesia. The participants' scores resulted from the higher order thinking test were converted into 1-30 based on the number of correct answers, one for the correct answer and zero for the false answer. Thereafter, the mean score along with the standard deviation and standard error mean of the pre-test and post-test were calculated using SPSS program. A paired sample t-test was further administered to determine the significant difference of the two mean scores. As for the qualitative data obtained from FGD, the researchers transcribed and coded the data to see the emerging themes related to the students' perception about the flipped learning classroom. The researchers analyzed, simplified, and reviewed the most relevant data that later were used as the basis of conclusion drawing. To ensure the validity and reliability of the data, the three researchers did analysis of the data from FGD independently, and the analysis process was continued by stages of discussion to reach the consensus on the final results.

Finding and Discussion
This section highlights the empirical answers of the research questions related to the impact of flipped learning on students' higher order thinking skills and their perception about the implementation of the teaching method in their classroom. Following the results, this section presents interpretation and discussion in accordance to current related theories and findings of previous studies. The participants' utterances are quoted verbatim to support the description of the findings of the qualitative data.

Impact of Flipped learning on students' higher order thinking skills
The results of pre-test and post-test as the basis to answer the first research question with regard to the impact of flipped learning on students' higher order thinking skills are presented in table 1. The mean scores of both tests are calculated to depict the students' improvement. As portrayed in table 1, there is a difference between the students' scores of pre-test and post-test. The mean score of the students' pre-test is 18.05, while the mean score of their post-test is 25.69. Furthermore, to determine whether the two mean scores have a significant difference, a paired sample t-test was administered. The results of paired sample t-test analysis are clearly presented in table 2.  Table 2 reveals that the students' score achieved in the post-test outperform the score of the pre-test. It emphasizes the significant difference between the results of pre-test and post-test, which means that the students perform better in terms of higher order thinking skills after being taught using flipped learning instruction. As shown in table 2, the significant level is .000 (< .05) and t-test statistics is 3.791 (> t-table: 2.023), that indicates the significant difference. Table 2 further presents that the mean score of the students in the post-test is 7.64 higher than in the pre-test. Therefore, it can be concluded that the students of elementary teacher education at IAIN Surakarta who participated in this study experience significant enhancement on their higher order thinking skills after being taught using flipped learning instruction.
The significant improvement of the students' higher order thinking skills after being taught using flipped learning as depicted in this study confirms the results of previous study conducted by Heo and Chun (2018) that reported a positive effect of flipped learning on Psychological students' higher order thinking ability. The result of this study is also in accordance to the finding of Annan et al. (2019) that flipped learning classroom using student-created content videos successfully develops business students' higher order thinking skills. These results indicate that flipped learning can potentially become an alternative to develop the students' higher order thinking skills which is urgently required in this twentyfirst century teaching. The possible reason for the positive impact is that the concept of flipped learning emphasizes on digital learning environment and active collaboration among students (Hasanudin & Fitrianingsih, 2018;Katchamat, 2018). These characteristics match with the needs of twenty-first century students who are closely related to digital native and pair works (Nugroho & Atmojo, 2020).
The significant impact of flipped learning on students' higher order thinking skills as resulted in this study is also in line with the result reported by Lee and Lai (2017) who conduct a research in a Hong Kong secondary school. Lee and Lai's (2017) study concluded that it was possible to accept flipped learning model to improve secondary school students' higher order thinking capability. In a similar direction, the result of this study is also consistent with the finding of Alsowat (2016) that flipped classroom teaching provided a significant effect on English learners' higher order thinking skills, engagement, and satisfaction in Saudi Arabia. From the above-mentioned results, it is suggested that twentyfirst century teaching and learning must be able to equip students with digital literacy and innovative learning to enhance their critical thinking Webb & Doman, 2020). Therefore, it is a worth saying that flipped learning offers an ample opportunities for teachers and students to achieve the goals of twenty-first century teaching and learning, where a higher order thinking skill is highly required (Hwang et al., 2019;Singay, 2020).

Students' perception about Flipped learning instruction
The data analysis of FGD reveals students' perception about the practice of flipped learning to improve the level of their higher order thinking skills. In general, the students deliver positive attitudes with regard to their experiences in joining learning activities using flipped method. More than half of the students opine that they were having joyous and interactive learning activities when they were taught using this flipped model. This integrated-digital learning inside and outside classroom seems to offer ample opportunities for the students to have an active engagement in the learning activities.
Based on the results of data analysis, there are three main themes emphasized by the students in the FGD about the practice of Flipped learning. First, flipped learning classroom provides more communication opportunities and time between teacher and students. The results of this study suggest that a flipped learning dramatically offers the amount of time the teacher spends with the students. It is also found that a flipped learning creates a platform for teacher and students to ask questions and discuss them with an issue they are finding challenging.
" Second, from the results of FGD it is also revealed that the flipped learning allows the students to have more collaboration among them. By performing peer works, discussion, presentation, and debate during the class time, the flipped learning enables the students to spend more time collaborating with one another. It suggests that flipped learning not only facilitates the students a great way to learn, but also improves their team working skills.
"Since I participated in this teaching model, I feel that I often collaborate with my pairs and friends in performing tasks or conducting learning activities in class. Most of my time is to experience group tasks such as discussion, peer feedbacks, and presentation." ("P1. FGD). "…….. and flipped learning gives me opportunity to spend more time to have collaboration with my friends to do assignments and class projects." (P4. FGD). Third, the results of FGD further depict that flipped learning encourages students to come to class with a good preparation. After the students have engaged with the digital contents through videos at home, they come to the classroom prepared with new ideas and questions in mind. The results indicate that flipped learning is a great way to foster the students' readiness to the classroom session, and thus nurture their responsibility and engagement.
"I always come to the class with questions or comments in my mind about the videos related to the topic of that day meeting after I watch that videos before class. Having questions and background in my mind motivates me to join active participation and discussion in the class." (P8. FGD). "I always feel confidence because I am ready to go to the class with ideas and questions I get from the videos that I watch at home. I think it is one of the benefits of flipped learning that we experience." (P5. FGD). This results confirms the finding of Hwang et al. (2019) that flipped learning practice in the context of university level offers students opportunity to experience joyful, fun, and interesting learning activities. The present-day students are well-known as digital native; thus, integrating technology in learning activities can be an alternative to enhance their critical thinking skills (Mutiaraningrum & Cahyono, 2015). The integration of digital technology in a university learning seems to be a promising way to achieve the success of twenty-first century learning (Nartiningrum & Nugroho, 2020;Zainuddin & Perera, 2019). This has been proven by the results of this study that flipped learning, which is the integration of digital technology and classroom-based activities, can improve the levels of university students' higher order thinking skills.
The results of this study have several implications for educational system in university level. First, examining the significant impact of flipped learning instruction on students' higher order thinking skills is a compelling effort to seek for the alternative way of teaching higher order thinking skills. By referring to the results of this study and relevant previous findings, university teachers can design teaching and learning activities using flipped method to assist students develop their higher order thinking skills. Second, the results of this study is considered as "a wake-up call" for university teachers and authorities to begin acknowledging the potential benefit of flipped learning to enhance students' critical thinking skills. With this regards, this study's results suggest that flipped learning can be integrated in the teaching curricula of university education, especially to facilitate the development of students' higher order thinking skills.

Conclusion
The results of this study indicate that flipped learning instruction can be a promising teaching method to improve the university students' higher order thinking skills. It is proven by the result of paired sample t-test that indicates a significant difference between the students' scores of pre-test and post-test. In a nutshell, it can be concluded that the levels of students' higher order thinking skills improve after being taught using content videos in a flipped learning instruction. As for the second research question, the results of FGD show that the students convey positive perceptions about the implementation of flipped learning in their classroom. Therefore, the results of this study imply that flipped learning instruction can be used to enhance higher order thinking skills of university students, especially in this twentyfirst century era where digital literacy and critical thinking ability become crucial skills.

Suggestion
Apart from the compelling results, the present study acknowledges some limitations. First, the number of sample used in this study cannot be used as a basis of generalization of the results in a large population of university students around the world. Therefore, future studies are recommended to conduct an immense study by involving larger number of participants coming from various backgrounds and cultural characteristics. Second, this study is conducted in the context of elementary teacher education department at a university in Surakarta Indonesia. Hence, further research is strongly suggested to examine the effect of flipped learning on higher order thinking skills of students coming from other departments such as Mathematics, Biology, Economy, etc. Lastly yet importantly, future studies are also encouraged to develop an alternative instrument to measure higher order thinking skills other than the instrument used in this study to enrich the literature on the measurement of this twenty-first century skill.