Evaluating ChatGPT Response Quality in Ethnomathematics Learning

Abstract

This study analyzes the quality of ChatGPT responses in generating ethnomathematical explanations in mathematics learning. A descriptive qualitative approach with a case study design was employed. The research participants consisted of one mathematics teacher and 30 eleventh-grade students at SMAI Nurul Huda. Data were collected through classroom observation, semi-structured interviews, and documentation of prompt-response transcripts. Two ethnomathematics-based plane geometry problems, related to a traditional engklek arena and a traditional-house pattern, were explored using ChatGPT through a text-based interface. The documented responses were evaluated based on three indicators: scientific accuracy, logical coherence, and cultural contextual relevance. The findings show that ChatGPT produced stronger responses in terms of scientific accuracy and logical coherence, particularly for lower-complexity problems. In Problem 2, 97% of the responses were categorized as accurate and 100% were highly coherent. In contrast, responses to the more complex Problem 1 were mostly categorized as fairly accurate and coherent, with 7% of the responses identified as inaccurate and incoherent. The weakest performance appeared in cultural contextual relevance, as 93% of responses to Problem 1 were less relevant and 100% of responses to Problem 2 were irrelevant. Observations and interviews further revealed that ChatGPT can increase student engagement and support procedural understanding, but it may also encourage dependence on instant answers if not accompanied by teacher guidance. Therefore, ChatGPT should be positioned as a supplementary learning tool whose use requires mathematical verification, prompt scaffolding, and explicit cultural reflection.