|
 |
Is comprehension measurable? | ||
Share with:
| ||
 Language Learning Forum : General discussion |  | |
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes  Speaks: French*, English, Spanish Studies: Polish | Message 81 of 211 17 August 2014 at 5:43am | IP Logged | ||
I've read quite a lot of Nation's work and nowhere do I see a blanket statement to the effect that assessing active knowledge is more difficult than assessing passive or receptive knowledge. As to every other book on the subject saying the same thing, I would be curious to see what books shapd has been reading. The very simple reason that assessing productive or active knowledge is easier is that it is "tangible." In a writing test, the test taker writes a document that can be circulated among various evaluators. The same thing for speaking. Oral and written comprehension are more complicated because there is little tangible output. There's no output. All we have is some answers to a questionnaire. Now, what Nation and all vocabulary specialists do say is that it is very difficult to study a speaker's actual or real vocabulary, i.e. where you actually count all the words that the speaker actually uses in certain contexts. This would be extremely tedious and time-consuming. What everybody uses is some form of sampling technique. Even here at HTLAL, where the word counters abound, to my knowledge only one person, Iversen, has ever made the deliberate effort to count the words he has used actually in his posts. When I tried to do it, I quickly gave up because it was too much work. But the fact remains that if anybody wants to assess my writing proficiency in English, there is a lot of tangible material to see. What about assessing my reading comprehension of English? There is no material to work with . 1 person has voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 82 of 211 17 August 2014 at 7:19am | IP Logged | ||
Why there is nothing for 81-90%? Those are my exemples of statements that could more or less correspond to the percentages. However, people are obviously using the percentages because they want a whole scale which allows you to be something in between the statements. The numbers are more a representation of this need for a whole scale (allowing you to express that you feel your skill rose by a few % even though you are not at the level corresponding to a different worded statement). If htlalers in general would be more comfortable with colors than numbers, which is quite unlikely, we could as well use the rainbow and progress through various shades while having the basic colors as the huge reference points. For our htlal comunication uses, there would be no difference. If we all used the rainbow scale, I could write a nice post to my log sharing the joy that my comprehension improved from sky blue to turquois during last week. I don't think the trouble concerning the faulty thought 30% of words known = 30% of text understood is that bad on htlal. From what I remember, the word counters tend to be interested just in that. In the word count as a reflection of their progress. And I can see nothing bad about that because it makes sense for someone working hard to improve their vocabulary to seek a way to measure it. The problem with interpretation of the statements is exactly the reason why we use the % "scale". Because all kinds of such statements are even more vague than our % estimates. Because these worded statements could lead to the same discussions as the statements "upper-intermediate", "fluent", "proficient", "advanced-beginner" and so on. So I think the only way to communicate about this is just to try to understand each other. Of course my guess of 80% may be different from someone elses. But I believe the difference isn't going to be huge enough to damage our shared communication goals: to give each other advice and support, to share our progress and strugless with others. If we keep doing that, as we already have been, it doesn't matter much whether we like more the descriptions, percents or the colors of the rainbow. Last point: What is the value of small bits here and there? Well, it depends on what bits are they and where are you finding them. If you intend to read a novel, than they are quite useless, I agree. But I've found such bits here and there to be very useful when traveling and trying to find some simple key info in a wall of foreign text/speech (informational leaflets or spoken information about directions to the hotel or a delayed train are just the first exemples I can remember now). And I was totally happy about such a success at that moment, even though I was really far from reading a novel or non fiction. :-) 3 persons have voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 83 of 211 17 August 2014 at 7:40am | IP Logged | ||
|
A practical exemple: my progress in listening comprehension during my July stay in Spain
word description: When I arrived, I could understand quite a lot to slower speaking people, even though by far not as much as later during the month. I could understand very little in the fast situations, in the groups or to the southerners. After a week or so, thanks to my previous tv series preparation, I could understand not only the main message but as well a lot of details to most speakers from northern Spain around me. At the end of the month, I could understand quite everything, I missed a word or a small nuance here and there, had a little trouble with a few harder northerners and understood much less to the southerners in comparison with them, especially those from Malaga. percent scale: When I arrived, I could understand 10-15% in most situations, even though 50-60% with a slower, careful speaker who didn't steer too far away from the topics at hand. After a few days, I was up to 60-80% with the northerners, depending on the person and situation. At the end of the month, I could understand 95 % or so when speaking with vast majority of northerners around me (one or two were harder, it appears to me that older men tend to be especially complicated for me) but only 50-60% to the southerners, especially those from Malaga. Really, the communication with them at the end of the month was reminding me a lot of the situation with northerners a month earlier. rainbow scale: When I arrived, my comprehension oscilated around Orange, depending on the situation. In the better moments, I was closer to Canary Yellow but had some Harlequin moments that pleased me greatly. After a few days, I got to the Green part of the specter while being at Lime with the southerners and Cyan with the rest. At the end of the month, I was safely between Indigo and Violet with most local natives but only at Bright Green (or sometimes even worse) when it came to the speakers from Malaga. (I used color names from this scale: http://en.wikipedia.org/wiki/Spectral_color ;-) ) So, I tried three options here and I still like our widely guessed % scale the most :-) 2 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 84 of 211 17 August 2014 at 8:21am | IP Logged | ||
|
I have written myself that word-counting for comprehension can be useful for pedagogical purposes as a means of
measuring progress and encouraging study. So, one could understand 50% of the words of an article this week and 60% next week. For those purposes I see no problem. My contention is simply that the percentage system that Cavesa proposes has no measurable foundation. emk at least uses a specific technique of counting of words on the printed page. How can one measure understanding 10-15% in most situations? This is not measured; it's a pure guess. But it looks accurate because it is a figure. It's the same with 50-60%. Why not 45-55% or 58-68%? Because of my knowledge of French and Spanish, I have somewhat of an idea of how Italian works. Would I say that I understand 10% of Italian? Or let's say 20%. Sure this means I could probably guess my way around a railway schedule but it still means I don't understand 80% of the language. How useful is that? In my system I would say: I understand Some Italian. It's purposely vague: it's anywhere from 5 to 90% in the word counting universe. Now, if somebody says to me that their understanding of Spanish was A2 at the beginning of their stay in Spain and then B1 with northerners, etc., I could get a sense of what they mean. My own scale is much simpler but I see the value of something like a CEFR scale. And as Arrekkusu pointed out earlier, we don't use a percentage scale when talking about speaking or writing. Would anybody say that when they arrived in a country they could speak 10-15%? Why do we not say that we could write 50% of Spanish? The answer is simple: it doesn't make sense. The fact that we put a percentage on our guess of understanding doesn't make it accurate. 1 person has voted this message useful | |||
| Serpent Octoglot Senior Member Russian Federation serpent-849.livejour Joined 6599 days ago 9753 posts - 15779 votes 4 sounds Speaks: Russian*, English, FinnishC1, Latin, German, Italian, Spanish, Portuguese Studies: Danish, Romanian, Polish, Belarusian, Ukrainian, Croatian, Slovenian, Catalan, Czech, Galician, Dutch, Swedish | Message 85 of 211 17 August 2014 at 10:34am | IP Logged | ||
Pleasure is subjective. The idea of not reading until you (mysteriously?) reach 98% coverage is ridiculous and harmful. 1 person has voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 86 of 211 17 August 2014 at 12:33pm | IP Logged | ||
|
Yes but as I was trying to demonstrate, it is no worse way of expressing our self assessment than some alternatives. The key to usefulness here is that :
1.You know what you mean when you speak about yourself. After some time, you get to know your scale well and never forget that the primary readers of our logs are always ourselves. Therefore leaving a message about our progress to compare later is helpful. 2.Most people do not have trouble trying to understand. I cannot remember a single instance when someone would say "But I don't think you understand 85%, it's more like 78%!" Sometimes, we use the cefr levels as reference but than we hit other walls with our heads. Without actually sitting the exam, we are just guessing again and that sometimes gains us even more criticism than using %. And the official descriptions are at times even more vague than anything else we have mentioned here, including my rainbow scale. B2: "Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation." or C1: "Can understand a wide range of demanding, longer texts, and recognise implicit meaning." The cefr self assessment checklist is much less vague in the things it mentions, however, it doesn't mentioned exactly the things we are trying to cover with % at all or only without the scale many of us seem to be after. Because reading a book is just one of the many skills in the cefr, while it is an extremely important and central one for many of us. And because the book reading relevant "can do statements" aren't that much informative for our purpose of continual self-assessment. B2:"I can understand in a narrative or play the motives for the characters’ actions and their consequences for the development of the plot." C1:"I can read contemporary literary texts with ease." I'd like to highlight that we often do not use the % for overall reading comprehension or overall listening comprehension. We often use it for a specific subskill, such as reading a novel, comprehension of a tv series and so on. That is very different from the cefr, which puts all the "subskills" into a bundle. Noone is interested to test, certify and put a grade on just one subskill taken away from the others. Really, no matter that my %s are always just a guess based on my experience with learning languages and my usual patterns of progress (which may differ more than a lot from someone else's experience) and on comparison with reading in my native Czech, I am not likely to use anything else in some situations, because it is the most informative way despite all the flaws at that moment. 2 persons have voted this message useful | |||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 87 of 211 17 August 2014 at 3:08pm | IP Logged | ||
|
There is nothing preventing anyone from using any kind of scale in describing their language proficiency. Wasn't
that the problem before the CEFR became widespread, at least in Europe. In North America, where the CEFR is hardly used, we're still stuck with various systems including popular notions of "fluency." The CEFR brought about a degree of standardization that we all appreciate. Even if it is self-assessed, the CEFR scale gives us a specific idea of what the speaker can do. If you say you are B2, we can look up the definition of B2 and we all agree. But when somebody says they understand 85% of a tv series, there is no reference system. I am not questioning the person's honesty when I question this figure. It's just that I don't understand what it means. Of course, I think that the person means "I understand a lot but not everything." But why not 70% or 75% or 80% or 90%, etc. What's the difference? When an emk counts the number of words on a page into various categories and says that he automatically understands 95.4% of words on a page, I can understand that. I don't call that a measurement of comprehension, but at least I can understand the statistical method used and I can use it myself. It computes. But when someone says they understand a specific percentage of something like a tv series, I see nothing more than a number plucked out of thin air. It has an aura of precision about it, but it is a pure guess. Sure, 85% is better than 75%. But it's misleading. In reality, when we try to compare one speaker's 85% to another's 75%, there is absolutely no difference. When I look at a Spanish soap opera, I'm very aware that there are things that I understand and don't understand, but how do I go about deciding what percentage to use? I can't, so I lump it all in "I understand Some Spanish in the soap opera." Somebody could say "I understand a lot but not all." That's OK too. Before, people used to say "I can understand a French movie pretty well." Now they say, "I can understand 85% of a French movie." It just looks and sounds better but it's just as vague and meaningless. 1 person has voted this message useful | |||
| Cavesa Triglot Senior Member Czech Republic Joined 5011 days ago 3277 posts - 6779 votes Speaks: Czech*, FrenchC2, EnglishC1 Studies: Spanish, German, Italian | Message 88 of 211 17 August 2014 at 3:50pm | IP Logged | ||
As I was trying to explain, the cefr level gives as an idea about what the learner can do in the whole skill, for exemple reading comprehension. However, it may have nothing or very little to do with every and each of the subskills. Our common example here: reading books. The cefr scale doesn't care much about such a subskill on its own so a B1 learner can be "real B1" in most skills but understand contemporary novels nearly perfectly or not to understand nearly at all. I've seen exemples of both these extremes. So should such a learner say: "I am B1 at reading in general but when it comes to novels, I am B1+++ or A2++ or B2-" or whatever? I don't think it would be any easier to understand or any more objective than the %s. The key point is that % allows us to express smaller nuances we can see but which no standartized test would show. And it doesn't matter at all that the numbers are just our best guesses. It is self assessment, it obviously works for many people better than saying "I now understand a little bit more than a few months ago" so there is no reason to criticise it, in my opinion. It is not meaningless. It expresses the learner's self assessment of how much he understands and how much he or she has progressed during a period of time. And such a self assessment, no mather how vague it may seem, is a key part of self studying a language because we do not use a teacher to assess our skills for us. All of these options of expression are vague but none of them is meaningless. 75%, a little bit, B1+, whatever the learner finds appropriate. It actually reminds me of the school grading system. 1 is the best, 5 is fail. These five grades' definitions are more than vague and totally different students with different skills (and even very different level of knowledge) can get the same grade. With many teachers, you don't even know how they finally choose the grade as the tests during the semester make only a part of it, there are many other factors including bias. They are so vague but quite noone trully believes they are totally meaningless. 2 persons have voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
