Cognitive Levels of Questions By Indonesian Teachers of English

According to the 2013 English curriculum (K-13), teachers were required to create English test items covering higher-order thinking skills (HOTS). By answering HOTS questions, students are expected to understand information deeper rather than only focusing on recalling facts. This case study aimed to find the cognitive levels of questions used in teacher-made English final tests in a private senior high school in Sampit, Central Kalimantan, Indonesia. Using the revised Bloom's taxonomy, this qualitative study investigated the written documents of teacher-made English final tests for grades X and XI in the academic years 2018-2019 for social and science programs. The findings showed that the cognitive levels of questions in the final test were dominated by the lower order of thinking skills (LOTS), with understanding as the highest level, followed by remembering and applying. The researchers identified potential sources of preference for LOTS questions. The paper concludes with research recommendations and a professional development program for English teachers.


Introduction
The Indonesian government has implemented the 2013 curriculum (K-13) in all secondary educational units. K-13 prepares students to have competencies needed in the 21 st century, such as critical thinking, creativity, collaboration, and communication (Cahyana & Sugiarto, 2019). According to the K-13, teachers should develop students' higher-order thinking skills (HOTS). HOTS are stated in 2. Literature Review

Cognitive Levels of Revised Bloom's Taxonomy
The revised Bloom's taxonomy has the same classification and number of cognitive levels as the original version. It was divided into LOTS and HOTS (Adams, 2015;Reeves, 2012;Soltis et al., 2015). Anderson et al. (2001) categorized the cognitive levels of remembering, understanding, and applying as LOTS while analyzing, evaluating, and creating as HOTS. HOTS requires students to interpret, analyze, and evaluate information, whereas LOTS requires students to rely on what they have memorized and understood from prior knowledge (Anderson et al., 2001). LOTS only asks students to repeat what has been taught and does not ask them to construct their thinking (Riazi & Mosalanejad, 2010).
HOTS questions in English instruction for Indonesian students were encouraged (Cahyana & Sugiarto, 2019). According to Serevina et al. (2019), HOTS encourage students to focus on knowing how rather than what. HOTS enable students to solve problems through critical thinking, innovation, and creativity (Sofyatiningrum et al., 2018). Students are expected to be able to think deeply, critically, and creatively to solve complex problems. Unfortunately, in practice, HOTS questions were still a challenge for teachers worldwide. In Indonesia, this is because the teachers lack understanding of the concept of HOTS, making and implementing HOTS questions in assessments (Abkary & Purnawarman, 2020).
Furthermore, in Malaysia, Shafeei et al. (2017) found that English teachers preferred LOTS questions to HOTS questions. Teachers in Malaysia have difficulty using HOTS because they lack knowledge about it, and the students have a low proficiency level in understanding the questions. LOTS are commonly used in the classroom to teach English. In Jordan, LOTS of questions dominated the English course books for secondary and university levels (Freahat & Smadi, 2014). In Uganda, Mitana et al. (2018) found that teachers used more LOTS than HOTS questions. In Korea, LOTS questions were used more than HOTS questions in English course books (Park & Kang, 2008). Questions in textbooks tend to focus on the lower level because acquiring knowledge through memorizing is more emphasized than practicing analyzing, evaluating, or creating the problem. In Saudi Arabia, when students have an examination, the teachers let the students learned the knowledge in their textbook (Alnofal, 2018) Previous studies in Indonesia (Dewi & Sukarni, 2020;Pratiwi et al., 2019;Utami et al., 2019;Widiati, 2012) revealed that most of the questions in English final examinations used in Senior High Schools were dominated by LOTS. Such lower-level questions cannot help students to learn and think critically (Abosalem, 2016;Park & Kang, 2008). Janah (2020) found that the questions in English books for secondary schools published by the Indonesian government were dominated by LOTS; on the other hand, Febrina et al. (2019) found that reading comprehension questions in an English textbook for grade 11 contained more HOTS questions than LOTS.
Delivering LOTS of questions to the students could not help them find a solution to the problem in different ways. Students tend to create a process of convergent thinking skills that focus on finding the correct answer using previously learned information (de Vries & Lubart, 2019). On the other hand, as a process of complex thinking, HOTS enables students to break down information, draw a conclusion, analyze data, and relate information involving the basic levels of thinking (Ariyana et al., 2018). Prasetianto (2019) argued that HOTS are classified as divergent questions requiring many possible responses to solve the problem. In other words, LOTS assesses students' ability to remember and use the information to complete tasks. In contrast, HOTS encourages students to think critically; they must process, relate, and apply data from one source to another.
Bloom's taxonomy has been claimed to be crucial to measuring students' mastery of knowledge (Reeves, 2012). Previous studies suggested using Bloom's taxonomy to investigate the levels of thinking skills in English test items (Köksal & Ulum, 2018;Mitana et al., 2018). Crowe et al. (2008) claimed that Bloom's taxonomy could help teachers measure students' Biology mastery levels. Kusuma et al. (2021) revealed that the HOTS assessment was able to help learners train their high-level cognitive thinking abilities.

Final English Test Items
As stated in K-13, English teachers must develop test items to assess student competence. English tests are expected to contain items that measure students' higher cognitive levels (Cahyana & Sugiarto, 2019). Previous research analyzed the cognitive levels of questions in different schools in Indonesia. Widiati (2012) found that English teachers in Surabaya developed more LOTS questions than HOTS questions. Utami et al. (2019) conducted content analysis research on four sets of multiple-choice English tests made by English teachers from 2016 to 2019 in one of the senior high schools in Indonesia. They found that most yearly test items covered LOTS and were dominated by understanding level.
Furthermore, Dewi & Sukarni (2020) found that the final English test items in Kebumen were dominated by understanding level. These are similar to a study by Pratiwi et al. (2019), who found that understanding level was the most dominant category in the English final test items in three different JELTL (Journal of English Language Teaching and Linguistics), 8(2), 2023 senior high schools in Singaraja. Putra & Abdullah (2019) found that LOTS questions were higher than HOTS questions in English examination items in Indonesia.
Many questions about remembering were found in the previous studies. For example, the question asks about elements. The following are the basic elements of a short story, except … belongs to remembering because it requires students to recall what was taught during the learning activity (Utami et al., 2019). Dewi & Sukarni (2020) categorized the question, saying that the text gives us information about … as remembering. The question asks students to recall the information from the reading text, and they can easily find the answer from the text given (Dewi & Sukarni, 2020;Utami et al., 2019;Widiati, 2012).
Besides remembering levels, most of the questions in the English final test were about levels of understanding. Utami et al. (2019) categorized questions such as finding a synonym, finding the main idea, and finding a reference as understanding. Finding a synonym requires students to guess the meaning of the word based on the context. Dewi & Sukarni (2020) found questions asking the students to find the synonym of a word with the word in the provided list. Finding the main idea belongs to understanding because it requires an understanding of the whole text. Understanding questions were in open questions and multiple-choice questions. Wisrance & Semiun, (2020) found an open question like "What is the topic of the text". An example of finding a reference is "These are soon absorbed by the vill…." The underlined word refers to … (Utami et al., 2019).
Grammar test items also appeared in teacher-made English items used in different schools in Indonesia. Completing a sentence with an appropriate word is considered applying (Dewi & Sukarni, 2020;Kamlasi, 2018;Utami et al., 2019). Here's an example of an application question: The trees would have produced more fruits if you … regularly. In this question, students must understand the conditional sentence and its pattern and apply their knowledge about this type of grammar to the sentence. Students cannot use their knowledge at this level without understanding what should be applied (Bloom et al., 1956).
The English test items developed by the English teacher in previous studies in Indonesia contained LOTS and HOTS questions. Utami et al., (2019) argued that finding the title of the text such as, The best title for the story above is …, or the purpose of the text such as What is the purpose of the text? is included in the analyzing level because students are required to analyze the relevant information of the whole text and make a conclusion. Besides, making a good paragraph by arranging jumbled sentences belongs to analyzing because students should organize the jumbled sentences to construct a correct paragraph (Pratiwi et al., 2019;Putra & Abdullah, 2019;Utami et al., 2019).
Another higher-level cognitive skill found in the English final test is evaluating. Asking students' judgment about the quality of a thing and its reason belongs to evaluating, for example, "What is hunting? Do you think it is a good activity? Why or why not? because students need to understand the facts of hunting and give their judgment about the quality of hunting activities based on those facts (Widiati, 2012). In addition, evaluating allows students to judge an object by giving their arguments, evaluation, or opinion (Dewi & Sukarni, 2020).
The previous studies also found few essay questions in the English test at the level of creating, the highest level in the revised Bloom's taxonomy. Asking students to converse or find a solution to a specific problem is part of creating. A question, for example, Please give your suggestions at least three to overcome the problem in the picture above! belongs to creating because students need to give their opinions about the picture and find the solutions to overcome the problem (Ramadhana et al., 2018). In addition, a question like, Make a conversation by using expressing suggestion and its response, is considered as creating because students have to critically and creatively create the conversation (Dewi & Sukarni, 2020).

Research Methods
This case study analyzed the cognitive levels in the teacher-made English final test items. A qualitative content analysis was applied to investigate teacher-made English test items (Kohlbacher, 2006). The test items were administered as multiple-choice and essay questions to grade X and XI students in Sampit, Central Kalimantan, during the even semester of the academic years 2018-2019 and 2020-2021. Accordingly, five sets of teacher-made English final tests containing 210 test items with 195 questions in multiple-choice form and 15 questions in essay form were analyzed. The number of test items was the total after 24 questions were eliminated because they repeated the other previous questions that had been used.
The procedures for data collection and data analysis were as follows: A research tool in the form of a checklist, coded with C1 for remembering, C2 for understanding, C3 for applying, C4 for analyzing, C5 for evaluating, and C6 for creating, was developed in advance to investigate the cognitive levels of the collected data. Then, the data were collected from the English teachers at the school, who were responsible for making the English final test items for grades 10 and 11 used in the even semesters of 2018-2019 and 2020-2021. Next, the collected data were read, and similar questions used in the academic year 2018-2019 and used again in the academic year 2020-2021 were eliminated. Then, the remaining data were analyzed according to the revised Bloom's taxonomy by using a checklist. Investigator triangulation by employing multiple evaluators was done to ensure the results could be trusted. The results of the analysis were presented in the form of a finding table. Finally, the conclusion of the investigation was reached by counting the number of each cognitive level category.

Results
This study found that the test items in the teacher-made English final tests appeared in different forms and cognitive levels. There were more LOTS questions than HOTS questions. They were mostly found in the form of multiple-choice and open-ended questions in reading comprehension, grammar, and vocabulary. Besides, other LOTS questions found were about asking about students' knowledge of the world, such as characteristics of a particular text, social functions and generic structures of a text, and writing test items. Writing test items that were found in this study were in the form of reordering jumbled words into correct sentences and jumbled sentences into a meaningful dialogue.
Each type of test item appeared in different categories of cognitive levels of LOTS and HOTS. Table 1 shows the test items and cognitive levels for each test item. As shown in Table 1, LOTS questions are higher than HOTS questions. Questions dealing with reading mainly belong to understanding followed by remembering, analyzing, and evaluating. Vocabulary questions, like reading, require more knowledge than remembering and applying. Unlike reading and vocabulary, all grammar items were at the level of applying. Moreover, most of the questions categorized as 'others' were about remembering besides applying and analyzing. None of the questions were categorized as creating. Accordingly, there were a lot more LOTS questions in the level of understand as the highest followed by remember and apply than HOTS in the level of analyze and evaluate.

Remembering Questions
The recall questions primarily sought explicit, detailed information, the text's main ideas, and The answer to question 69 was explicitly stated in the sentence, "The battle in which he was killed occurred near Vlakke Hoek, or Etna Bay, in the Arafura Sea." Question number 25 was not a reading comprehension question because the question did not ask about the information in the text. Instead of finding information in the text, the students answered question 25 by recalling their knowledge about the social function of narrative text.

Understanding Questions
Understanding questions mainly were about finding a synonym, the meaning of a sentence, completing a paragraph with the correct words, finding a reference, making an inference and a conclusion, finding the topic, comparing two ideas, exemplifying, explaining information, and guessing the action of the character. Below are examples of reading comprehension questions at the level of understanding. To answer question 65, students had to read the whole text and find the stated words related to Stephen Hawking's profession. The sentences contained words referring to Stephen Hawking, such as "…, so he pursued physics instead. …, he was awarded a first class honors degree in natural science", "His many publications together with other scientists". Students can answer question 182 by comprehending the function of the pronoun it in the sentence, which refers to something that has been mentioned before. Students should be able to read carefully and understand the sentence to determine which noun or phrase the pronoun it refers to.

Applying Questions
Applying questions dealt with grammar and writing a correct sentence. Students were instructed to complete a text with correct grammar or reorder scrambled words.

Analyzing Questions
Analyzing questions involved arranging jumbled sentences into a good dialogue, determining the reason, opinion, or idea of the writer who wrote the text, and finding the topic of a reading text requiring students' analysis.
-Why does the song writer entitle the song "Superman"? (item 160) a. The writer wants to show that Superman is a perfect hero b. The writer shows that Superman always becomes the winner c. The writer is going to tell us that superman is the symbol of victory d. The writer is going to tell us that Superman always has dreams in his life e. The writer shows that Superman gives his best although he faces many troubles -How are the writers feeling? (item 203) The answer to question number 160, The writer shows that Superman gives his best although he faces many troubles was not stated explicitly in the song. The question asked for implicit information about song lyrics. To answer the question, students should analyze and infer the verse, chorus, and bridge in the song to find the underlying reason for the songwriter. Essay question number 203 requires JELTL (Journal of English Language Teaching and Linguistics), 8(2), 2023 students to understand and analyze the poem. The correct answer is that she is psychologically hurt lonesome. This answer is not stated explicitly in the poem. Therefore, students should be able to identify the idea developed throughout the poem by assessing the title, rhythm, setting, tone, diction, and mood

Evaluating Questions
Evaluating questions asked for value from the text and finding the best solutions in the alternative answers to the problem. The examples for the questions are as follows: -What can a school management do to prevent violence at school? (item 133) a. Treat students fairly b. Have a strict inspection c. Do a test on new students d. Give compliments to students e. Appreciate students' achievement -What can we learn from the dialog? (item 157) a. Lifestyle is very important b. Online shopping is dangerous c. We should deposit our money d. We should buy cheap products e. We should spend our money wisely The correct answer to question 133 is having a strict inspection of which the information is not stated in the text. This question is asking for a judgment, so the students have to evaluate as well as analyze. To answer the question, students should be able to analyze the factual information stated in the text, such as the fact that the presence of a high amount of violence is bad for the school's reputation, so parents will try to remove their children from the school which means violence at school is a serious problem. This fact helps students evaluate the best and most effective way in which the school management can prevent violence at school. In addition, question 157 requires students to find evidence from the dialogue showing what the character in the dialogue has done to support the answer.
To answer it correctly, we should spend our money wisely, students should be able to find the underlying message of the dialog by understanding the implied meaning of the dialogue lines. It helps students evaluate and determine the dialogue message, which the readers can use to learn about and be aware of similar behaviors that they are doing.
In summary, the majority of the questions on teacher-created English final exams are from LOTS. The answers to the questions were provided in the text, which allowed students to use their prior knowledge to find the keywords or clues in the text. Students were also asked to apply their knowledge of certain topics, such as grammar rules. A few HOTS questions were found, analyzing and evaluating, requiring students to use their thinking skills to answer the questions. Instead of relying on their memory to answer the questions, they should critically analyze and interpret the information in the text to answer the questions.

Discussion
The findings of this study showed that the cognitive levels of questions in the teacher-made English final test were dominated by LOTS (92%) rather than HOTS (8%) questions. These findings are in line with the results of Dewi & Sukarni (2020), Utami et al. (2019), Pratiwi et al. (2019), Shafeei et al. (2017), Freahat & Smadi (2014), Widiati (2012), and Park & Kang (2008) which stated that the majority of teachers use lower-cognitive level questions.
The test items used in this study were mostly found in multiple-choice rather than essay questions. It follows Daeli et al. (2020), who found that multiple-choice was a dominant test item used in reading comprehension. Unfortunately, the test forms did not affect the level of questions in this study. Both multiple-choice and essay questions mainly required students to find a single answer, some of which were mentioned in the text or based on their previously learned knowledge, which led them to think convergently (de Vries & Lubart, 2019;Prasetianto, 2019). A limited number of questions thinking divergently, i.e., find implicit ideas in the reading passages.
Many questions about remembering, understanding, and applying led to the discovery of divergent thinking. Remembering questions in this study required the students to recall familiar or repeated information in the text (Anderson et al., 2001;Dewi & Sukarni, 2020;Utami et al., 2019). Furthermore, this study confirmed that understanding questions were the most frequent cognitive level found in the previous studies (Dewi & Sukarni, 2020;Pratiwi et al., 2019;Utami et al., 2019). Finally, this study showed that applying questions mostly asked students to complete a sentence or paragraph with an appropriate type of grammar, as found by (Dewi & Sukarni, 2020;Kamlasi, 2018;Utami et al., 2019). Anderson et al. (2001) stated that applying requires students to perform a task using a standard procedure.
As mentioned, most analyzing questions require students to explore, relate, and draw a conclusion. However, the word 'analyze' does not always indicate HOTS. Analyzing questions about arranging jumbled sentences into a coherent dialogue can be classified as HOTS while arranging jumbled words into a correct sentence belongs to applying. Arranging jumbled sentences into a coherent conversation requires students to understand the meaning of each sentence and how they are related to each other while arranging jumbled words into a good sentence asks students to be able to understand and apply the rule of how each word is used as an element of a sentence. Although rearranging scrambled words and sentences may look like similar questions, they appear different on cognitive levels. Anderson et al. (2001) stated that analyzing means organizing information into a good structure by determining how the sentences are connected. On the other hand, applying means putting into place a specific procedure to solve the problem.
Misunderstanding the concept of analyzing questions was found in this study. It was found in this study that the word "analyze" was not used to analyze the information in the reading text but to ask students to recall their knowledge about text structure. The teachers in this study seemed to misunderstand the Basic Competence stated in grades K-13. Instead of raising a reading comprehension question, this current study asked a general question about text structure.

Conclusion
Similar to the previous findings, the teacher-made test used in a private school in Sampit, Central Kalimantan, tended to cover LOTS. The demands of K-13 were not met. Instead of helping students to think critically and practice reasoning skills, the questions developed by the teachers exercised the students' ability to remember and understand facts. The dominant use of LOTS questions indicated that English teachers must learn to create HOTS questions according to the revised Bloom's taxonomy. Considering the related studies, English teachers and schools in Indonesia should be more aware of the significance of the cognitive levels used to develop questions. To help English teachers write HOTS questions in their teacher-made tests, the schools can facilitate the teachers with teacher professional development programs on writing HOTS questions. An internal professional development program on developing questions requiring HOTS for English teachers is recommended.
Moreover, further research on teachers' abilities to create HOTS questions should be conducted to ensure the teachers gain skills from the professional development program. Teacher-made English final tests prepare students to meet the challenges of Industrial Revolution 4.0. Furthermore, since this study only figured out the cognitive level appearing in English tests, future researchers interested in this topic can explore the English teacher's competence of HOTS in developing the assessment questions.