|
 |
Is comprehension measurable? | ||
Share with:
| ||
 Language Learning Forum : General discussion |  | |
 emk Diglot Moderator United States Joined 5342 days ago 2615 posts - 8806 votes  Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 137 of 211 21 August 2014 at 7:20pm | IP Logged | ||
If you want to know what reading with partial comprehension actually feels like subjectively, various folks have provided examples: Harry Potter with key words semi-decipherable or opaque Egyptian with a majority of sentences decipherable, but others opaque (parallel text) Latin children's book which is only occasionally decipherable French with high comprehension More French with high comprehension In each case, these examples include include the actual thought process of the reader. You can see why sometimes it makes sense to count words (generally at higher levels, where the sentences themselves rarely "fall apart"), and when it make sense to count sentences (generally when a good number of sentences are ultimately decipherable—at least appropximately—but a fraction are beyond all hope). The exact numbers you'd give in each case would be highly dependent on your counting methodology. If these examples aren't enough to explain what people have been trying to say, there's certainly not much more that I can do. If it doesn't make sense after all this, it's not going to. My general philosophy of numerical measurements: 1. State a measuring procedure that could at least hypothetically be reproduced. It's OK to use different measures for different purposes. 2. Take a sample and count. 3. Do math, approximately. 4. Round to an appropriate degree of precision. This will generally work out for most readers, as long as nobody confuses "about 75%" with "75.18%." The former means "roughly 3 out of 4" and the latter means "7518 out of 10,000 +/- 0.5 (or a standard deviation, if specified)." If this still doesn't make sense, well, that's OK. I enjoyed explaining things as best as I could, and I hope that helped somebody along the way. 4 persons have voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 4819 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 138 of 211 21 August 2014 at 7:48pm | IP Logged | ||
|
The trouble is that you cannot simply demonstrate 75% understanding of a whole book on a small excerpt, in my opinion.
The only way would be to copy the whole text or a part large enough to represent the whole text (which would still be quite long to work with), divide it into the required pieces (perhaps paragraphs for overall comprehension, but most likely sentences, for some uses even individual words), assess their difficulty and importance for the overall understanding of the text (perhaps give them a numeric value based on it), decide how to count things you more or less understand from the context but couldn't exactly and automatically translate with all the nuances (probably through some coefficients), and then count. Such a process should give you numbers exact enough to satisfy you. But the process would be very time consuming. But the basic conditio sine qua non is the choice of a sample representing the whole book, which wouldn't be an easy task. The shortest option would be a few chapters, in my opinion. The longer the sample, the more precise results. Therefore I'm convinced those guesses "out of the thin air" based on subjective view on our own comprehension of whole book are actually much more precise than any funny counts based on a few sentences and black/white decisions. 4 persons have voted this message useful | |||
| sctroyenne Diglot Senior Member United StatesRegistered users can see my Skype Name Joined 5201 days ago 739 posts - 1312 votes Speaks: English*, French Studies: Spanish, Irish | Message 139 of 211 21 August 2014 at 9:00pm | IP Logged | ||
I think I can identify what would give an English learner trouble in this passage: "Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much." This whole clause would probably throw me off. I'd understand the beginning of the sentence but I wouldn't understand what this is doing here and which would make me feel shaky about this sentence as a whole (and it is a bit odd for an omniscient 3rd person narrator to suddenly switch narrative voices within a single sentence, which happens to be the first sentence of the whole book/series and enter into the head of this couple, without quotes to make it more clear). Also, while I would understand the words and even the phrase "thank you very much" I wouldn't really understand the sort of "prideful" tone and may have trouble discovering the correct meaning from reference materials. As a native speaker I understand it but still find it a bit odd. "He was a big, beefy man with hardly any neck, although he did have a very large moustache." The problem in the sentence wouldn't be the word "beefy" (which is easy to look up and is redundant in this sentence), but rather how/why the two clauses are linked together with the word "although". I'd look up all my material on the usage of "although" but not really understand why it's used here. So decipherable yet still puzzling. "Mrs Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours." The first portion is an odd, yet decipherable description. Though as a learner I wouldn't know that this isn't a typical way to say that someone has a long neck (and as a native, I was picturing her with a fat neck rather than a long one until I got to "craning") and attribute my unfamiliarity with it to my status as a learner and not one of Rowling's particularisms as an author. "Craning" is only one word but it's a pretty key one in this sentence so not understanding it may make it hard to really understand the sentence as a whole. Beyond Harry Potter, I can think of whole passages that go on for pages in Lord of the Rings where Tolkien describes the characters walking through a valley/up a hill/through a forest/through a cave, etc where there would be a lot that I wouldn't understand (and may even have difficulty following as a native). These are passages you either just slog through or skip. 2 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 140 of 211 21 August 2014 at 11:52pm | IP Logged | ||
|
Talk is cheap. I like concrete examples. I'm still trying to figure out how one can arrive at 75% comprehension of
a Harry Potter novel. When I tried to eliminate a quarter of the words of a Harry Potter excerpt, everybody pooh poohed me. serpent came up with some quaint variants that I quite haven't figured out yet. Following up on emk's experience of understanding two thirds of the sentences of a Jules Verne novel, I tried the sentence counting approach by eliminating two out of eight sentences from the Harry Potter excerpt. That didn't seem to work either. emk has explained the math and tells us to choose a counting methodology. I agree but where is the counting methodology? But I'm still stuck with the same problem of demonstrating 75% comprehension of a specific piece of text. Then we're told that that you can't demonstrate 75% of a book by looking a an small excerpt. And after all this, the key observation is "Therefore I'm convinced those guesses "out of the thin air" based on subjective view on our own comprehension of whole book are actually much more precise than any funny counts based on a few sentences and black/white decisions." We're back to square one. Since no-one seems to be able to demonstrate just what 75% comprehension means, we fall back on my original observation: it's all guessing based on feeling of the moment. When I read that someone has just finished reading a French novel with 95% comprehension I take that to mean that that figure was based on a feeling for 95% and not on any real counting. A newcomer to this thread, sctroyenne, has made an interesting contribution by trying to identify passages in the excerpt that are probably very challenging for the English learner. I particularly like the observation of how the verb craning is key to understanding the entire sentence. In my opinion, what sctroyenne has done is raise the issue of what it means to really understand a passage. One could say that a native speaker - and most likely a young and educated one - would read the same words as our English learner and have a different comprehension experience. When our English learner says that they understand a sentence or a passage, we have to take their word for it, but we can ask what exactly are they understanding. For example, I would add '',,,they just didn't hold with such nonsense" in the very first line to sctroyenne's comments as a bit odd turn of phrase that was unfamiliar to me. I do believe that some English learner would have some difficulty fully appreciating this text because of the language used. How to calculate that 75% is beyond me. 1 person has voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 4819 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 141 of 211 22 August 2014 at 1:05am | IP Logged | ||
|
I don't agree with scrtoyenne's observation at all. No offence meant but it is a totally wrong guess about what a learner would or wouldn't understand. I was one of such learners trying HP as even their first foreign book, so I am speaking here from experience and not guesses, and while I had trouble with vocabulary and some grammar when reading the book (in 2002 or so), such passages were actually very clear. Why do you think a learner cannot understand irony or humour here? It works just the same way in many other languages. The "thank you very much" exemple is nothing difficult to understand in the context, unless you are a person without any sense of humour. I've even heard it being used many times exactly the same way in Czech and a few times in French.
Quite the same applies to the other exemple. "Beefy" would be very likely a new word for most intermediate learners but easily associated with beef to give at least some clue. "Although" would only be a trouble if it was a new word for the learner. Again, learners are new to the language, not to existence of sense of humour. The word might require the learner to use a vocabulary but its use is not much of a mystery. Sure, a learner may be confused by such phrases in a descriptive serious non fiction but probably not while reading a book whose author may fancy to make a joke from time to time. Yes, craning would almost certainly be a new word for most learners. But looking it up in a dictionary would clarify the meaning of "twice the usual amount of neck" for anyone interested. Those not interested would miss nothing important for the story. There are many such passages in the LotR, so a larger part of the book wouldn't be understood, given the asumption that all parts of the text are of the same importance and you are interested only in the quantity. But a learner who would skip most descriptions due to low comprehension wouldn't actually lose much of the story and "real content" of the book. Therefore he or she could fully enjoy the experience even with larger parts skipped. Yes, we are back to square one again because, as was already explained many times by many people including Iversen, the percentages, as they are usually used by many htlalers, are just a metaphore used to avoid lenghty pointless worded statements. And we are getting to more and more ridiculous methods of measurement, such as native speakers trying to imagine they are learners who discover existence of irony for the first time in their lives.
Yes, that is how you were supposed to take it right from the beginning. This fact has already been clarified many pages ago and repeated again and again :-D Just to make sure everything is clear now after all the exemples and repetitions ;-): 1.The percentages are usually just a metaphore to avoid vague worded statements which wouldn't allow a finer distinction of progress. People who do some real counts usually inform others about the fact and about their methodology. 2.The percentage metaphore is used for subjective assessment of one's skill and/or progress, therefore it is based on a feeling and there is nothing wrong about it. It doesn't hurt its usefulness in the slightest. 3.Most counting methods to get "real results" have serious flaws and, while amusing theoretically, have no real use, at least for most learners. 4.There are more options than just understand/not understand. There is a huge difference in self assessment caused by the non consistency of what individual learners prefer to count as understood and what not. 5.Not all words, not all sentence and even not all paragraphs have the same value for overall comprehension of the text. So purely quantitative approach to sum up the % of words known or % of fully understood sentences actually doesn't mean that much when it comes to overall comprehension of the text. 3 persons have voted this message useful | |||
| James29 Diglot Senior Member United States Joined 5185 days ago 1265 posts - 2113 votes Speaks: English*, Spanish Studies: French | Message 142 of 211 22 August 2014 at 1:06am | IP Logged | ||
|
S_allard has the better analysis here. There is simply not a way to calculate a percentage of understanding. No matter how hard someone tries that simply cannot be demonstrated. That's why all the other "explanations" simply don't hold water and s_allard is having so much fun pointing it out.
But, the percentages ARE meaningful. The percentages are meaningful when compared to something that can give them real meaning. That can only be that same person's ordinal scale. So, I think s_allard is a bit incorrect when he/she says: "Since no-one seems to be able to demonstrate just what 75% comprehension means, we fall back on my original observation: it's all guessing based on feeling of the moment." It is definitely based on just a "feeling", but it's based on the specific feeling of how it compares to their other "understanding" experiences. The percentages are based on the order in which that person would put different experiences. The number makes sense and are useful when the person says "the first time I read that author I understood 70%" and "the third time I read that author I understood 80%." This is why the other folks have such a hard time believing that s_allard is right... because the numbers are useful and definitely do have meaning. 2 persons have voted this message useful | |||
| YnEoS Senior Member United States Joined 4064 days ago 472 posts - 893 votes Speaks: English* Studies: German, Russian, Cantonese, Japanese, French, Hungarian, Czech, Swedish, Mandarin, Italian, Spanish | Message 143 of 211 22 August 2014 at 2:46am | IP Logged | ||
|
[Previous Post was deleted because I think this manner of inquiry will ultimate be more productive]
I think the word "understanding" has a lot more usages than its being given here, I'm not going to attempt to note them all but I want to draw a distinction that I think will be useful. That is the difference between understanding "words" and understanding a "story" Storytelling 101 So storytelling is a complex and nuanced subject that I can't do justice to in post but here are some basic principles that I think will be useful for this discussion. Storytelling often relies on the reader making tons of assumptions based on what's written in the text, and afterwords the reader cannot recite all the words, but is capable of summarizing the story. For some simple examples, in one chapter a character goes off to war, in a later chapter the character returns and its mentioned that they are limping around with a cane. The reader assumes they've received some injury in the war even though it's never stated in the text. Or more complexly Character A tells Character B they're going to the movies and instead they walk into a bar. This could cause the reader to draw many different conclusions, maybe A is a drunk and wants to keep it secret, maybe B is morally opposed to drinking and A doesn't want B to think less of them. They may be able to answer this with information they've already read, or it might be left a mystery until later in the text. A good deal of what's essential to a story isn't directly in the text, but the text gives us information and we form ideas about who the characters are and whats "going on". When asked to recount stories, people will often say what was essential to the story "he was a shady character", "they keep lying to each other" and a lot of times that's information that's not directly in the text, but inferred by the situations. Using words to understand a story A lot of words aren't essential for understanding a story Original Text: John is a lawyer and today was the most important case of his career. Reader Comprehension: John is a ____ and today was the more important case of his career. Most readers will be able to guess John's profession even without knowing the word for lawyer. Even with a much lower comprehension, if the whole story revolves around the trial, then if they can comprehend that John is in a trial, they will probably assume that he wants to win the trial, that he needs to argue his point and submit evidence, and that problems will get in his way. Even lacking some of the nuance, a person might still be able to put together a rough idea of what's going on. This is probably a big reason why people will read translations of books they've read before in their native language before moving into new books. They're transferring their knowledge from Known Story ---> Unknown Words ---> Known Words ---> Unknown Story If I understand 70% of the words in my favorite book translated into French, I can enjoy it despite the low word comprehension, I already know the story, so all I need to get is enough information to know what part of the story I'm in. Then I can guess all kinds of words through that knowledge. Even if its a familiar genre, say like an adventure story, I know a lot of things going in. A character is going to go from simple circumstances to much more dangerous ones and is going to have to learn new skills to overcome these challenges, and probably attract a mate in the process. Notice a lot of sci-fi and fantasy stories in your native language will require you to learn tons of new terminology about different creatures/species/planets/tribes/spells/technologies, you expect this and its easy to figure out because it doesn't change much of what's essential to the story, it's just detail. As long as you recognize these terms when they re-occur you can figure out their role in the story. What do People mean when they say % comprehension and why do they read things at low comprehension I think the most sensible answer is that they're referring to words, because this is easiest to quantify, but they may understand a lot more of the story, if they've read it before, or if its their favorite genre. Why do people read things they understand very little of? Who the hell knows, just ask them. The first time I opened a French book and understood more words than I didn't understand I was freaking thrilled just to be able to parse a sentence here or there and didn't really care how much of the story I could put together. What is understanding and how much do native speakers understand Understanding is a complex process that refers to different things, and if we want to break it down its better not to assume it's a single simple phenomenon that we can break down and analyze. Even if a native speaker knows all the words and grammar, its hard to tell if they'll extract all the detail they possible could out of something. Whenever I re-read a book or re-watch a movie I always noticed details I missed before and develop a more nuanced understanding of the characters. I think if one is going to embark on a critique of how much learners actually understand, one needs to do a lot more studying of what it means to understand a story and how much native speakers understand story, and develop a much more specific and useful terminology. It's also strange to critique the idea that someone would read a story at 75% comprehension in general, because there are tons of factors and reasons why they might do it, IE: They read it before, they're using a pop up dictionary, or they're just happy to be reading in their language. Edited by YnEoS on 22 August 2014 at 2:53am 3 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5240 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 144 of 211 22 August 2014 at 2:50am | IP Logged | ||
|
Since I'm the perpetual bad boy here, let me state my position. My fundamental starting point is that a basic
element of effective communication is good terminology. This is pretty much a given in science. We can only "understand" each other if we agree on a common language. Those who have been around HTLAL long enough know that I have had two causes : the proper usage of the word fluency and the promotion of the CEFR as a standardized system for describing language proficiency. I don't think I've made any difference insofar as fluency is concerned. I've accepted the fact that the world outside of science uses fluency as a synonym of proficiency. After all, can't one become fluent in only three months? The CEFR was developed to provide a state of the art system for describing and assessing foreign language proficiency in the countries of the European Union. Even though I can't take any credit for this, I am heartened to see that it has become widely used here at HTLAL. Although most of the time people assess their own levels, there is at least some effort to refer to the content of the various levels. And some people do pass the tests for various reasons. The wonderful thing about all this is that we are referring to the same system of measurement. Individual self- assessment will vary but the various levels are not defined in terms of subjective feelings. At any given moment, one can actually sit a CEFR exam. A bit like the use of the word fluency, this business of percent comprehension has always intrigued me because I could never understand exactly what it meant. I've been told a gazillion times that it's all subjective and it's just a metaphor but it still is a figure and by putting it on a document one is communicating something to other people. When someone writes that they have read a document or listened to something with a certain percentage comprehension, what am I supposed to think? If the figure is not really a true value, then what is it? I get that it's some sort of handy tool that people used to supposedly indicate their progress. The other thing that made me think about this is that nowhere in the world of applied linguistics and second language acquisition is there anything remotely resembling this idea of percentage measurement of comprehension. This is not to confused with the vast world of vocabulary size assessment. For example, does Krashen ever speak of percent comprehension? I also note that some people here used percentages extensively and others very rarely, if at all. My initial interest in this topic stemmed from my own attempts at measuring my comprehension of Spanish texts and television. When I attempted of quantify how much I understood of Gabriel García Marquéz's Cien años de soledad, I saw major methodological uses that led me to abandon the whole project. So, I basically stopped worrying about how much I was understanding. Since I couldn't do it myself, I decided to see how other people did it. And that's why I'm here, kicking up the dust. I'm obviously not going to be satisfied if somebody tells me it's just a matter of feeling. As I sit here contemplating my Spanish book, I don't feel anything coming on me. How much longer do I have to wait? Although I believe measuring comprehension directly is impossible, I am open to looking at possible markers or indices. This is why I have adamantly asked for people to show me how to calculate that famous 75% comprehension of Harry Potter. Mention was made of understanding 2/3 rds of the sentences of a Jules Verne novel. I went to have a look at the novel and didn't really understand how this calculation worked. I am fully cognizant of that fact that we have different levels of understanding in our target languages. That's not the issue. The issue is how and why put a percentage on something that seems so difficult to measure. Edited by s_allard on 22 August 2014 at 5:34am 1 person has voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
