|
 |
Is comprehension measurable? | ||
Share with:
| ||
 Language Learning Forum : General discussion |  | |
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes  Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 105 of 211 19 August 2014 at 7:55pm | IP Logged | ||||||||
|
Neither have fans of non-numeric statements. No logical or scientific bases.
Really. There was no viable alternative proposed on this thread, therefore we are left with 2 options: 1.Stick with a representation people understand. 2.Have nothing. Just say "a little" or "not much", keep insisting that nothing but full comprehension (which is not defined as you've just said yourself) is enjoyable and so on. I believe many learners would just rather give up and start knitting if we had such an attitude ;-) You know, lots of things are dependent on a feeling. You believe science is totally quantified and feeling dependence free? Go to any lab and watch them add a few drops, name the color of the result, change a protocol because it feels right.The same can be said about most other professions which are feeling-free at first sight. It's the same thing. Something feels right based on experience and a scale that may just not be that easy to clearly define to others. And there is nothing wrong with that, unless you've got a better option, which you obviously haven't got. All your arguments have been made and, no offence meant, I cannot find anything of value in them. All you did say (and repeat endlessly) is: 1.The % aren't based on any objectively measurable scale that would be the same for everyone. That much I agree with. 2.Using % scale is worthless and stupid and you've basically painted us as morons daydreaming about progressing in a language. Well, that I cannot agree with that and I would be quite curious to see a log of yours to compare how is your method better. 3.Non-numerical definitions are much better than the %. No reason, they are just as vague but they are better. Well, I think this thread just proved that both are of same value and just a matter of preference. Actually, I wonder why are some people so obsessed with counting, assessing and critisising the methods used by others instead of just actually learning a language or speaking about learning a language :-) 3 persons have voted this message useful | |||||||||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 106 of 211 20 August 2014 at 1:14am | IP Logged | ||||||||
|
Well, the cat is out of the bag and has gone ballistic. As to be expected, when some people run out of logical
arguments, they get into a huff and resort to insults, innuendos and discourtesies. As I always say, if you can't stand the heat, get out of the kitchen. Let's move on. For those who still may be interested, I have noted that Hu and Nation concluded in their study that 98% word coverage was necessary to enjoy the piece of fiction in question. As I have already quoted, the authors also pointed out that some students were able to understand or enjoy the text with less coverage, as low as 90%, because they brought other language skills to bear on this task. This 98% word coverage figure is very high but is also in line with all the other word counting studies that Paul Nation has done. To be able to understand a written English text well you need to know around at least 95% of the words. If you take an average of 350 words per printed page, you can miss 7 words and still understand or enjoy the work. As I look at Harry Potter and the Philosopher's Stone, I think that not knowing 7 words per page is not that bad after all. What happens when the word coverage goes below 95%? Comprehension goes down of course. But is the relationship between comprehension and word coverage linear? For example, does 80% word coverage mean 80% comprehension? If the people who proudly display their percentages actually did some word counting instead of just going by the feeling in their bones, then we would know more about this. But what Hu and Nation have shown is that below that 95% threshold, generally speaking, comprehension becomes difficult and nearly impossible below 80%. In other words, if you tried to read Harry Potter and did not know one out of five words, you would basically not understand anything. You might as well not bother reading the book. This corresponds to what most of us experience when reading in our target languages. It's something like an all or nothing proposition. You appreciate, enjoy or understand the text when you know nearly all the words. Otherwise the text is in a sort of fog where you may have an idea of what is going on but it is hardly enjoyable reading. It's tedious. And sometimes, as happens to me quite often, you get stuck on something you don't understand at all and is key to understanding the whole text. Or I may ask myself, Did I really understand what I thought I understood? The importance of all this depends on the nature of the text. For something like a railway schedule or a price list, one could get by with a low word coverage but legal texts demand a very high level word coverage and comprehension. Would you sign a contract where you don't understand half of the document? Actually, some people do. Take jokes. We know how they work. A little story followed by a punch line that is usually a play on words or on some cultural reference. Can you comprehend 50%, 75% or 85% of a joke? I don't think so. You either get the joke 100% or you don't. It doesn't make a difference if you understood everything except the punch line When people say they understand 75% of Harry Potter, what they are really saying is that they recognize 75% or the words - what some people call sight vocabulary. But truth be told, at that level of word coverage - missing one word out of four, comprehension is negligible. But could one have 98% word coverage and still not understand the text? Actually, I think one could. It's probably rare but I think one could know the main meaning of all the words and not be able to put everything together properly. This is why one may have to read a text many times in order to understand everything. This is also probably where the level 2 comprehension kicks in and where subtleties of grammar and usage make the difference. To be explored. Edited by s_allard on 20 August 2014 at 6:56am 2 persons have voted this message useful | |||||||||
 Iversen Super Polyglot Moderator Denmark berejst.dk Joined 6705 days ago 9078 posts - 16473 votes Speaks: Danish*, French, English, German, Italian, Spanish, Portuguese, Dutch, Swedish, Esperanto, Romanian, Catalan Studies: Afrikaans, Greek, Norwegian, Russian, Serbian, Icelandic, Latin, Irish, Lowland Scots, Indonesian, Polish, Croatian Personal Language Map | Message 107 of 211 20 August 2014 at 1:34am | IP Logged | ||||||||
|
You can let people read a certain text and then ask control questions. A weakness with this procedure is that the creator (or owner) of the test assumes that his/her/their answers are the only correct ones, and I have seen cases where this was a moot point, to put it bluntly. But fair or not fair, the outcome is a score, and you can rank both persons taking a particular text after their scores AND the tests after the scores obtained by the test persons. But rankings aren't measurements with a welldefined scale, and there is no way you can translate your position on such tests into claims about your comprehension of other texts than those used in the test.
Those who state that you need to know at least 95 or 98% of a text to understand it have backed this on studies where they have replaced the rarest 5 or 2% of the words with nonsense words. They then let GOOD readers read the stuff and test their understanding by asking them directly whether they understand the stuff and by posing control questions (under the assumption that your own answers actually are the only correct ones). And then you can of course count those who don't understand something and award scores for the number of things they seem to understand. But these figures are tied to the concrete tests and the texts they are based on, and they certainly can't provide objective measures of comprehension with welldefined scales for any other works. So saying that you understand 75% of Kritik der Reinen Vernunft or Dantes Commeddia or Winnie the Pooh is just a metaphor for claiming that you more or less understand those works, but run into problems fairly often, and claiming that you understand 99% just means that you think you understand almost everything - although you are too cautious to claim you understand everything. If you fell good about attaching percentages then by all means do it, but ultimately the numbers are just metaphors for your subjective feeling. Cavesa asks why some people are obsessed with counting and assessing learning methods (in some cases combined with criticism of rival methods). There is no doubt that I'm one of those obsessed persons, and I can just say that I'm interested in learning methods as such and want to study them with the nearest approximation to scientific mathods I can muster. Of course I also expect to get information about ways to learn languages efficiently, but my interest in learning methods is something that can't simply be identified with my interest in the languages themselves. It is a separate study field. Edited by Iversen on 20 August 2014 at 1:42am 4 persons have voted this message useful | |||||||||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 108 of 211 20 August 2014 at 3:12am | IP Logged | ||||||||
|
Let's have go at trying to see how a person could understand 75% of a Harry Potter text. As Iversen has pointed
out, one way is to ask control questions. This what is commonly used in tests, but designing good test questions is quite difficult. But let's say we design 12 good questions. If the student answers 9 questions correctly, we will assume that they have understood 75% of the text. Let's try a word counting approach. I've taken the first two paragraphs from Harry Potter and the Philosophical Stone. I removed one out of every four words. This a very crude approach but this is what it looks like: Mr and Mrs ___, of number four, ___ Drive, were proud __ say that they ___ perfectly normal, thank ___ very much. They __ the last people ___ expect to be ___ in anything strange ___ mysterious, because they ___ didn't hold with ___ nonsense. Mr Dursley ___ the director of ___ firm called Grunnings, ___ made drills. He ___ a big, beefy ___ with hardly any ___, although he did ___ a very large ___. Mrs Dursley was ___ and blonde and ___ nearly twice the ___ amount of neck, ___ came in very ___ as she spent ___ much of her ___ craning over garden ___, spying on the ___. The Dursleys had ___ small son called ___ and in their ___ there was no ___ boy anywhere. Is this what 75% comprehension looks like? I'm not really sure what to make of this. The interesting thing is that a native speaker of English could probably fill in many of those blanks by trial and error or just plain deduction. But our learner is probably unable to do this. In my opinion, if this is 75% comprehension, it's pretty worthless. Here is the complete text for comparison. Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you'd expect to be involved in anything strange or mysterious, because they just didn't hold with such nonsense. Mr Dursley was the director of a firm called Grunnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large moustache. Mrs Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere. Edited by s_allard on 20 August 2014 at 6:53am 1 person has voted this message useful | |||||||||
| Iversen Super Polyglot Moderator Denmark berejst.dk Joined 6705 days ago 9078 posts - 16473 votes Speaks: Danish*, French, English, German, Italian, Spanish, Portuguese, Dutch, Swedish, Esperanto, Romanian, Catalan Studies: Afrikaans, Greek, Norwegian, Russian, Serbian, Icelandic, Latin, Irish, Lowland Scots, Indonesian, Polish, Croatian Personal Language Map | Message 109 of 211 20 August 2014 at 11:17am | IP Logged | ||||||||
|
The method used by Hu and Nation and others is different in one respect: they systematically inserted nonsense instead of the rarest words until they had changed a given percentage of all the words, not just every fourth word, and the effect of this would be that the text became less messy and weird, but also much less informative. For instance all the proper names would be gone, and also words like "moustache", "drills", "opinion" and "nonsense" - just to mention a few of the possible candidates.
A native speaker or advanced learner would not have a chance to guess that one of the missing words was "drills" if every occurence of this rare word was removed. On the other hand it would be possible to guess that the missing word between "proud" and "say" in the mangled version above is "to". Edited by Iversen on 20 August 2014 at 11:24am 2 persons have voted this message useful | |||||||||
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 110 of 211 20 August 2014 at 12:27pm | IP Logged | ||||||||
|
As usual, some people start with an interesting thought but later start being a broken cd player when others just don't agree and consider the person to be a genious halfgod :-D It is even funnier because sallard actually never speaks of his real experience, never writes a log, never comments on threads actually discussing a language and not just metodology or assessment and never showed anything proving whether and to what extent he actually knows his languages ;-) That is a totally different situation than when some other people speak, such as Iversen.
As Iversent points out, one of the common troubles is the kind of the questions. The comprehension parts of exams tend to have two basic types of questions: -open ones, requiring a reaction, usually a short answer or rewording of the statement. This one is probably better but may pose a problem for people with much lower active skills. -multiple choice. And that is where wording chosen by the test author means a lot of difference. I've seen tests meant for native czechs that were confusing and there were more correct answers than the author meant. I've seen such tests in foreign language examination or even in totally unrelated fields. The point about not every word having the same value for overall comprehension was an important one. If you do not understand a rare word, it means quite nothing. If you do not understand a basic one used several times per page, it is a different story. A funny memory: I was asked by a guy in a tram whether I understand my book in English. Of course I said I understood and it was no lie. So he looked at the page, pointed to the one of the really rare word used and asked what it meant. It was a rare word I had seen a few times so I explained briefly that in this context it was something like this or that which was a correct assumption. However, the guy told me "See? You cannot understand!" because I didn't give an automatic and dictionary like answer. Some of the word count arguments and measurements remind me of that. Comprehension and knowing all the vocabulary in a dictionary like measure are two different things. I think Iversen summarised the purpose of the % as metaphores really well. As we have seen in the thread, we quite share the understanding of these methaphores. We just don't want to write lengthy descriptions saying the same thing as the number, adding no value to the information and sounding just as stupid as "I am nearly fluent" kinds of statements. 2 persons have voted this message useful | |||||||||
 emk Diglot Moderator United States Joined 5534 days ago 2615 posts - 8806 votes Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 111 of 211 20 August 2014 at 2:23pm | IP Logged | ||||||||
Cavesa, I know you're frustrated, but I would appreciate it if you would dial back the personal remarks. Thank you.
That's a pretty silly example, because you're obscuring both proper names and extremely common words. Obviously this has no bearing on how any learner would ever perceive a text. So what does partial comprehension feel like in practice? Here's an example from 2012 in my log:
As usual, I'm using a three-part breakdown between "(semi-)automatic/decipherable/opaque." But here I counted sentences, not words, because my comprehension was still low enough that I'd lose whole sentences of the text. Today, I usually count words in French, because I take the grammar as a given, and only a tiny handful of words completely escape my comprehension—normally 0.1% to 2% by my count, depending on the genre and subject of the text. Now, when I use a percentage, it's often based on a quick mental calculation: "Hmm, I understand 3 out every four sentences pretty easily, so 75%" or "I can follow a bit less than half the dialog in this TV show, so I can follow 40%" or "I actually had to look up 10 words out of the last 40 pages, and this book runs roughly 350 words/page, so that's 10 word in 14,000, or less than 1 word in 1,000, so that's less than 0.1% 'opaque'." I convert everything to percentages, because surprisingly many people have trouble telling whether 3/5ths is bigger than 3/4ths, but everybody knows that 60% is less than 75%. It's just a notational convenience: I convert everything to a single decimal scale for the reader, because not everybody likes doing fractions in their head. There's a reason everybody but the US uses the metric system. Of course, I don't necessarily assume that percentages are terribly precise: "75%" means "about 3/4ths, but it's probably rounded or an imprecise measurement" when I read it, but "75.2%" means "I actually measured this and got the equivalent of 752/1000." Similarly, "60%" means "about 6 out of 10," and interpreting it "600 out of 1,000" is actually inappropriate. "600 out of 1000" is written "60.0%". In fact, in the United States, they actually teach this stuff in some math and science programs. I've had teachers who would actually mark students down for writing "61.27%" when they should have written "about 60%." We were actually instructed in how to count significant digits, and keep track of them through a longer calculation, and we were expected to round our answers accordingly. So if I write "I understood about 60%", you can safely interpret that "as more than half, but probably less than 2/3rds, using whatever counting methodology I just described." And that can be a useful thing to communicate, because there really was a stage of my learning where I could understand roughly that fraction of the sentences in Le Tour du Monde without making an effort—and I enjoyed the book greatly. One of the reasons I've kept such a detailed log is because I wanted to communicate to other first-time students that it's possible to progress from understanding roughly 3/5ths of the sentences to understanding all but the occasional obscure word, and to do so within 10 to 40 books worth of reading (at least for an English speaker learning a Romance language). It's much harder to explain that using a scale like "none/some/all." If you look back at my quote about Le Tour du Monde, I think that communicates a much more vivid and useful idea than merely saying "some." Edited by emk on 20 August 2014 at 2:27pm 1 person has voted this message useful | |||||||||
| tarvos Super Polyglot Winner TAC 2012 Senior Member China likeapolyglot.wordpr Joined 4709 days ago 5310 posts - 9399 votes Speaks: Dutch*, English, Swedish, French, Russian, German, Italian, Norwegian, Mandarin, Romanian, Afrikaans Studies: Greek, Modern Hebrew, Spanish, Portuguese, Czech, Korean, Esperanto, Finnish | Message 112 of 211 20 August 2014 at 2:32pm | IP Logged | ||||||||
|
Significant digits is also taught in Dutch schools. Actually I think it's part and parcel
of most science curricula across the board. Edited by tarvos on 20 August 2014 at 3:34pm 1 person has voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
