|
 |
Is comprehension measurable? | ||
Share with:
| ||
 Language Learning Forum : General discussion |  | |
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes  Speaks: French*, English, Spanish Studies: Polish | Message 57 of 211 09 August 2014 at 10:38pm | IP Logged | ||
I agree with most of these observations. One of the reasons I believe that comprehension is not measurable exactly is precisely this difficulty of defining what comprehension is. But since comprehension, however it is defined, is an important element in any discussion of the effectiveness of methods of language learning, I prefer the broad categories that I've mentioned: Nothing, Some and All. I'll leave the more subtle subcategories to other observers. 1 person has voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 58 of 211 10 August 2014 at 7:59am | IP Logged | ||
|
In a recent class discussion on complexity and understanding of written language, I looked at an excerpt from
quite an interesting book They Laughed When I Sat Down, An informal history of advertising in words and pictures. The author is Frank Rowsome, Jr. Here is the first paragraph of the Acknowledgments: "This book proposes neither to attack nor defend advertising. To assail an activity on which billions of dollars are spent annually would call, at the very least, for rolling up the polemic sleeves and spitting on the polemic hands: and to defend billions of dollars seems supererogatory, like being in favour of the west wind. Instead, the objective here is to examine informally how a single segment of the industry, magazine advertising, grew in America in the years between the Civil War and World War II. No oxen are gored, no moments erected." It should be pointed out that the book was published in 1959 and this is reflected in the writing style. There are 95 words in this paragraph. How well do non-native speakers of English understand this text? As a matter of fact, how well do native speakers of English understand it? Actually, the text is straight-forward and most people got the gist of it. The only unusual word is supererogatory 'unnecessary'. It was when trying to translate this into French, especially spontaneously without a dictionary, that certain issues of understanding arose. There were long discussions on how to render "call for, at the very least, rolling up the polemic sleeves and spitting on the polemic hands", "being in favour of the west wind", and "No oxen was gored, no monument erected." I'll skip all the details, but much of the discussion centred on what what essential to understanding this text and what could be ignored. For example, the last line 'No oxen gored, no monument erected' could be safely ignored. Now, how to measure or test for comprehension of this paragraph? Edited by s_allard on 10 August 2014 at 8:00am 2 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 59 of 211 10 August 2014 at 10:54pm | IP Logged | ||
|
Since I personally know some of the people behind the Canadian federal government official languages tests, I have an idea of their thinking and their strategies for testing of comprehension. As I understand it, the idea is not to measure comprehension per se but instead assess what the user can do in terms of reaction to a written text. In other words, if the test-taker has understood the text correctly, they are able to answer certain questions. How they go about understanding is basically irrelevant. What counts is ultimately how well this future government employee will be able to do their job in the second language. This, by the way, is a fundamental component of the CEFR perspective in assessing language proficiency. It's what people can do with the language that is important not the language in itself. For those people who think these tests are easy and one can easily guess one's way through, I'll remind them that their are 60 questions that must be answered in 90 minutes. And the texts get progressively harder, that is more complex, more subtle and longer. Note that there are 10 pilot or dummy tests that are used for research by the test makers. These are assessment tests. Everybody gets the same number of questions, and it's the number of right answers that determine one's level. Here is a condensed version of the official document: "SLE-Test of Reading Comprehension in the Second Official Language The SLE - Test of Reading Comprehension in the Second Official Language assesses the ability to understand texts written in your second official language. In this test, you are presented with a number of texts (e.g., e-mails, notes, letters, information bulletins, excerpts from reports, research papers). Your task is to read each text carefully and respond to the questions by selecting the best answer to each question. There are a total of 60 multiple-choice questions of two different types in this test: Fill in the blank; and Answer the question. Of the 60 questions, 50 will count towards your score. Ten (10) questions are pilot questions that will not count towards your score. The questions comply with the test’s specifications and have been approved for inclusion by a multidisciplinary team of subject-matter experts. While these questions will not count towards your final score, they may be used in future versions of this test. You will have 90 minutes to complete all the questions on the test, in addition to 20 to 30 minutes for administration. Example questions There are two types of questions in this test. Examples of texts with these types of questions can be found after the descriptions. Question type 1 For this type of question, you will see a blank line. Choose the best word or group of words to complete the sentence from among the four response options. Question type 2 For this type of question, you will see a question about a text. Choose the response option that best answers the question. EXAMPLES of question type 1 Expéditeur : Pat Bégin Destinataire : Liste des employées et des employés Objet : Système informatique de gestion des congés Bonjour, Veuillez ______(A) que le Système informatique de gestion des congés ne sera pas disponible demain. Vous pourrez accéder de nouveau à votre compte dès mardi prochain. Vous devrez alors ______(B) vos soldes de congés pour l’année prochaine. Pat Bégin 1. Choisissez le mot ou le groupe de mots le plus approprié à insérer dans l'espace en blanc « A ». voir marquer laisser noter In this case, answer choice four best completes the text. Therefore, you would select number four. 2. Choisissez le mot ou le groupe de mots le plus approprié à insérer dans l'espace en blanc « B ». augmenter confirmer manipuler ignorer In this case, answer choice two best completes the text. Therefore, you would select number two. EXAMPLES of question type 2 Communiqué concernant le dépôt du Rapport annuel du ministère Le Ministère des finances a terminé le 30 avril dernier son Rapport annuel. Ce rapport couvre toutes les activités des ministères et des agences du gouvernement fédéral en vertu de la nouvelle Loi sur les finances dans la fonction publique (LFFP). Dans ce rapport, les statistiques récentes de la dernière année fiscale sur les dépenses de la fonction publique fédérale montrent de meilleurs résultats. Plus exactement, 90% des ministères et des agences se soumettent à leurs engagements financiers conformément à la loi, une hausse de 6% par rapport à l’année précédente. 3. Selon le texte, que montrent les statistiques? le pourcentage de conformité à la LFFP devrait diminuer l’an prochain la majorité des ministères et des agences respectent la LFFP le pourcentage de conformité à la LFFP ne devrait pas s’accroître l’an prochain la plupart des ministères et des agences ne respectent pas la LFFP In this case, answer choice two is the best answer. Therefore, you would select number two. 4. Que dit le texte sur le respect de la LFFP en général? Il est en augmentation. Il est en réduction. Il est en stagnation. Il est en alourdissement. In this case, answer choice one is the best answer. Therefore, you would select number one." 1 person has voted this message useful | |||
| shapd Senior Member United Kingdom Joined 5959 days ago 126 posts - 208 votes Speaks: English* Studies: German, Italian, Spanish, Latin, Modern Hebrew, French, Russian | Message 60 of 211 13 August 2014 at 1:08pm | IP Logged | ||
|
I have come a bit late to this discussion but would like to point out that the professional linguists who study the number of words needed for comprehension do not try to assess the percentage of a text which can be understood, which has been the focus of this thread. Read the freely available papers on Paul Nation's home page.
What he has done is to present a group of learners with a series of texts with different numbers of words that they do not know (read the papers for details of how he assesses that). He then asks them to summarise the argument or the story that has been given. From that he can work out what percentage of unknown words prevents accurate reproduction of the story i.e. what is the minimum percentage of known words required for accurate extensive reading without a dictionary - almost the opposite of the original question here. Note that Nation's group do not expect to be able to assess in a simple study like this the more subtle aspects discussed here like knowledge of grammatical peculiarities or the associations that words express. They are only interested in whether a word is correctly interpreted in a given context. He has come up with a figure of at least 95% known words and preferably 98% before even the weaker students can understand a text. So if you skim a text and find that it has significantly fewer words that you recognise, you will not be able to get more than a rough idea of its content without study. It is generally recognised that it is much more difficult to assess active skills scientifically, and it is much more subjective. However, there are CEFR interviews available on the web at different grades and it is quite obvious that there is a clear difference between A, B and C - and yes both increasing sophistication in use of grammar and wider vocabulary are seen at each stage. On a slightly different point, S Allard's frequent statements about knowing the idiomatic uses of common words are usually addressed by regarding them as set phrases in their own right to be treated as any other special vocabulary, not as extensions of the basic meaning. 8 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 61 of 211 13 August 2014 at 4:46pm | IP Logged | ||
I think that the paper being referred to here is " Unknown Vocabulary Density and Reading Comprehension" by Marcella Hu Hsueh-chao and Paul Nation. In this study, students are presented with texts containing different percentages of nonsense words and are then tested with a) multiple choice tests and b) cued recall tests. If this is the right paper, the researchers do not ask the test takers to summarise the argument of the story. If this is not the right paper, I would appreciate the correct reference. The authors conclude: "This study shows that the density of unknown words has a marked effect on reading comprehension...This (research) provides support for the position taken by Hirsch and Nation (1992) namely that learners need to have around 98% coverage of words of a text to be able to read for pleasure." It should be noted that the text in question was a piece of fiction. What I found interesting in the conclusion was the next paragraph that begins as follows: "This conclusion must not be interpreted as say that with 98% coverage of the vocabulary no other skills or knowledge are needed to gain adequate comprehension. All of the subjects in this study were readers in their first language, had considerable knowledge of English grammar, were experienced in reading English, and brought considerable background knowledge to their reading. These all contribute to their skill in comprehending text and account for some learners reading the 95% and 90% versions getting high scores. However, as readability studies show, vocabulary knowledge is a critical component in reading." I certainly agree with the observations of this study. As I have mentioned, word counting methods, of which Paul Nation is certainly the most well known exponent, can have all kinds of uses, especially for language teaching. Nation's work has focused on vocabulary size necessary for comprehension and not on measuring comprehension. Edited by s_allard on 13 August 2014 at 4:52pm 1 person has voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 62 of 211 13 August 2014 at 5:05pm | IP Logged | ||
I disagree with this assertion that "It is generally recognised that it is much more difficult to assess active skills scientifically, and it is much more subjective." Where is this written? I think the the reality is exactly the opposite. The reason active skills are easier to assess is that they are observable. We can read a person's writing and listen to a recording of their speech whereas we have to resort to indirect methods of trying to "measure" a person's reading and listening comprehension. As a matter of fact, shapd rightly points out that the sample CEFR interviews show different levels of productive skills. There are no videos of people demonstrating different grades of reading or listening comprehension. 1 person has voted this message useful | |||
| smallwhite Pentaglot Senior Member Australia Joined 5118 days ago 537 posts - 1045 votes Speaks: Cantonese*, English, Mandarin, French, Spanish | Message 63 of 211 13 August 2014 at 5:17pm | IP Logged | ||
|
Perhaps the reason I am happy with "comprehension based on word count" is that I count myself A LOT and I have a pretty concrete idea of how much comprehension each % figure represents. Sort of like how the weather report tells you it's 20*C / 68*F tomorrow, and you will interpret for yourself how cold that is and whether you'd need a jacket, based on your past experience.
And I count myself a lot because it's excellent motivation to see the number improve. And because I'm an accountant. 1 person has voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 64 of 211 13 August 2014 at 5:34pm | IP Logged | ||
My insistence on the importance of idiomatic expressions using common words is based on contemporary work in applied linguistics on what is called formulaic language. This includes the well known things like phrasal verbs in English, collocations, idioms, metaphors and proverbs. It's definitely not as simple as treating these things as set phrases. For example, how does one distinguish between Get to the bottom of the hill and Get to the bottom of a scandal? As a matter of fact, it is interesting to note that in his study of the vocabulary of IELTS Speaking Testtest candidates, Paul Nation concludes: "It showed that the sophistication in vocabulary use of high-proficiency candidates was characterised by the fluent use of various formulaic expressions, often composed of high-frequency words, perhaps more so than any noticeable amount of low-frequency words in their speech." For those readers who still believe that high-proficiency means large vocabulary, here you have it from the expert himself. It's not necessarily the number of different words that count, it's the "fluent use of various formulaic expressions, often composed of high-frequency words", i.e. ordinary simple words, that is the indicator of proficiency. 1 person has voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
