|
 |
Is comprehension measurable? | ||
Share with:
| ||
 Language Learning Forum : General discussion |  | |
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes  Speaks: French*, English, Spanish Studies: Polish | Message 17 of 211 07 August 2014 at 8:14pm | IP Logged | |||||||||||||||||
This is a very interesting example here for a number of reasons. Let's look at the methodology. All the words are basically given equal weighting, Only two words are not "automatic". But here those two content words are particularly important. If you don't understand them, the sentence doesn't make much sense. But if we drop any number of function words such as les, à, de, le and even petites, it would hardly make a difference. So, one has a relatively high word count score in the automatic category but actual comprehension in terms of overall meaning from the content words is much lower. A side effect of this unweighted word counting method is, since a small number of words, especially function words and very high frequency verbs account for a large number of words in a running text, you are guaranteed to recognize a large number of words regardless of the presence of all the content words. But here is an interesting item. In the example given above, the word "clocher" has not been flagged and we can presume it is "automatically" understood. This was translated as "bell". This is totally wrong. A clocher is a church steeple or a bell tower. A bell is "une cloche". The bell did not fall down; the steeple did. Translation mistakes happen, but in this case we also see that the reader did not make the right connection between écrouler "to collapse" and clocher, thereby misinterpreting the meaning of écrouler. A bell cannot collapse; it falls, as in the mistaken translation. What is the significance of all this? What we see is that clocher was "automatically" misunderstood. And it's a very important content word. How many others are there like this? So, what do the word count statistics tell us about the reader's actual understanding of the text? I don't know, but I suspect that comprehension is lower than the figures tell us. Now, one could ask, does it really make a difference to mix up cloche and clocher? In the grand scheme of fiction it probably is not that important. In a technical report it could be crucial. What is also true is that for most people around here all this relative. 95% comprehension isn't really 95% comprehension, it's just "more than 60%" and 60% is more than 40%. The important thing is that if you say 40% it means that you have to work more on improving your reading or listening. This is why I take these measurements with a large grain of salt. They are not to be taken at face value. They have an aura of mathematical certainty but they are basically false. They do not measure comprehension, they are more learning tools to incite the learner to work harder. That's OK too. Edited by s_allard on 07 August 2014 at 8:50pm 1 person has voted this message useful | ||||||||||||||||||
| shk00design Triglot Senior Member Canada Joined 4446 days ago 747 posts - 1123 votes Speaks: Cantonese*, English, Mandarin Studies: French | Message 18 of 211 07 August 2014 at 8:41pm | IP Logged | |||||||||||||||||
|
Depending on the reading material there is a lot of variations of understanding. Like a children's book in
English vs. Shakespeare. In Chinese you can be reading an article able the latest electronic gadgets including words that are recent additions to the language: blog 博客, upload 上载, download 下载, text message 短信, etc. And you can be reading a collection of poems from the Tang Dynasty 600 years ago or sayings by Confucius over 1000 years ago. Reading a newspaper and classical literature are two completely different worlds. Edited by shk00design on 07 August 2014 at 8:42pm 1 person has voted this message useful | ||||||||||||||||||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 19 of 211 07 August 2014 at 8:44pm | IP Logged | |||||||||||||||||
I don't call this a theoretical discussion. This is a question of methodology. My starting point is a simple observation: some people, not all, make statements about their level of comprehension in terms of percentages. I'm questioning what these figures mean how the users calculate them. If we say that comprehension can "express several different things that aren't easily quantifiable" --something I believe--what are we to make of those statements with the percentages? There is a clear goal here, at least in my mind. It's certainly not to create a ruckus and irritate people. It's to sensitize people to the importance of using the right terminology so that we can have useful discussion that help us move forward. This was the same reasoning behind my tiresome complaining over the use of the word fluency and in favour of the CEFR model of language proficiency. All of this so that we can have a better understanding when we speak to each other. 1 person has voted this message useful | ||||||||||||||||||
 emk Diglot Moderator United States Joined 5534 days ago 2615 posts - 8806 votes Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 20 of 211 07 August 2014 at 10:26pm | IP Logged | |||||||||||||||||
Fortunately, I could decipher these words just by looking at them for a second or two. (For this exercise, I looked them up in the dictionary, to confirm my translation, and if my guess was wrong, I marked them as opaque instead.) See the original post for more examples.
Ah, thank you! This is why I posted the entire passage and gave a translation. As I mentioned in the original post, my translation presumably included a number of errors, and I had no way to count them myself (without pestering my wife). (Curiously, when reading French translations of my favorite English-language books, I've found some very strange translation errors. Even professional translators misunderstand cultural references and misinterpret rare idioms on occasion. Since I usually haven't read the English version for at least 6 months before reading the French version, I only notice the really blatant errors, and generally only in scenes that have stuck in my memory for some reason.)
Feel free to go read the rest of the page carefully; I bet you can find another mistake or two. I found this page to be a pleasant challenge, thanks to the geographical descriptions and "scene-setting." There are some very approachable books, such as Le Pont de la rivière Rwaï, which have fairly intimidating first pages. If you desire more accurate numbers, and you wish to volunteer to proofread my translations, I'm happy to translate several more pages for your calculating pleasure. :-)
Yes, I do try to repeat this disclaimer fairly often. But people tend to zero in on the highlights, and ignore the hedging. Again, I find it helps to do things like post an actual page, and take a public pratfall—that's the best way I know to make the weaknesses of informal estimates clear and memorable.
Yes, obviously. Anyway, all this brings me back to one of my favorite points. There's always a lot of words that are basically understandable, given the context, but which are not solidly known: - "I guess toiture probably means something like 'roofing.'" - "Is écrouler the same word we used for flowing water and the passage of time? Or am I mixing up two different words?" - I see un clocher "bell tower" and my brain reads une cloche "bell" without even noticing the mistake. This process happens on both the conscious and the unconscious levels, of course. The entire point of extensive reading is that it takes all of these words that are more-or-less decipherable, and it provides dozens of examples of how they're used. Even an error like clocher and cloche will eventually correct itself. Unfortunately, I've reached the point where many of my vocabulary problems are things like contrefort "buttress, foothill" and clocher "steeple." These words appear so infrequently that it's hard to acquire them naturally without reading millions of words. (Neither of these words appear in Routledge's frequency dictionary at all, unfortunately, so I can't estimate how often they appear in a typical corpus.) And this is why I believe that tools like Anki remain useful for advanced learners who want to work on low-frequency vocab. 2 persons have voted this message useful | ||||||||||||||||||
| Serpent Octoglot Senior Member Russian Federation serpent-849.livejour Joined 6599 days ago 9753 posts - 15779 votes 4 sounds Speaks: Russian*, English, FinnishC1, Latin, German, Italian, Spanish, Portuguese Studies: Danish, Romanian, Polish, Belarusian, Ukrainian, Croatian, Slovenian, Catalan, Czech, Galician, Dutch, Swedish | Message 21 of 211 07 August 2014 at 10:37pm | IP Logged | |||||||||||||||||
Yes, communication functions just fine otherwise :) When it doesn't, it's deliberate like in the example about milk. I can understand the noble intentions of standartization, but let's not forget the downsides too. I've already been told that this forum can feel hostile to newbies, due to numerous things that are common knowledge here but not necessarily to the outsiders. Attempting to set any objective standards for measuring comprehension would be akin to requiring a rigid format for questions, such as: LANGUAGE: LEVEL: MATERIALS AND METHODS USED: MATERIALS AND METHODS AVAILABLE: PROBLEMS: SOLUTIONS ATTEMPTED SO FAR: etc. But people generally seem to think that nobody's asked this very question before, and they may well disagree with your attempts at converting, or won't be necessarily willing to measure they comprehension in more detail than they already did. Basically, I don't think anyone would mind a more objective way of measuring comprehension, but s_allard hasn't suggested anything specific so far. And whatever he suggests is probably going to co-exist with the methods already used. 4 persons have voted this message useful | ||||||||||||||||||
| YnEoS Senior Member United States Joined 4256 days ago 472 posts - 893 votes Speaks: English* Studies: German, Russian, Cantonese, Japanese, French, Hungarian, Czech, Swedish, Mandarin, Italian, Spanish | Message 22 of 211 07 August 2014 at 10:55pm | IP Logged | |||||||||||||||||
My point is there's no "right" way to use language in absence of a specific context or goal. If there's Person A a builder, and Person B an assistant, and what matters in their communication is for Person B to act in accordance to Person A's commands. Person A shouts "Slab!" and Person B goes into the storage room finds a slab of rock and gives it to Person A. Communication was successful, because Person B understood Persona A's command and acted accordingly. He didn't need to know "a slab of what?" or "do what with the slab?" it all made sense given the specific context, and even if he doesn't know what "slab" means as an abstract context, as long as he acts accordingly in this context communication is successful. Then Persona A Shouts "5 red bricks!" Person B goes into the storage room and sees several piles of different colored bricks, one of the piles is a darkish brownish red color, the other pile is a light sort of pinkish red color. Person B, returns confused and says "What do you mean by red?" Person A clarifies "5 dark red bricks." now Person B has enough information and acts accordingly. Communication works because enough detail was given to distinguish one type of brick from another. Now in the context of this forum CEFR is much more useful than the word Fluency, because the whole forum is dedicated to learning and improving language skills. Fluency doesn't allow us to distinguish many different levels of learners, and so it is less useful on this forum, than the CEFR system which allows for a wider gradient of comparison. But sometimes CEFR might not be sufficient and more specific terms might be necessary. If someone has called them self a B2 and later posts a video of them self speaking, and someone else says "That doesn't sound like a B2 to me" then the first person might clarify "Sorry I meant B2 in reading, in speaking I'm more of an A2-B1". Fluency will be understood, but there's not a lot you can do with it given the types of conversations that happen on this forum. With "Comprehension" I'm not sure there's a useful system of communication that would work universally for every context that might come up with on this forum. If someone is keeping a personal log of their progress and they say "I read Harry Potter 1 in the language I'm learning and understood 40%, then I spent 50 hours chatting with a native speaker and read Harry Potter 2 and understood 70%" What's relevant to the discussion here is the shift from 40% to 70% comprehension, how they measure it is not necessary here since we're only trying to gauge the benefit of speaking with a native speaker. If a bunch of people are reading Le Petite Prince for a challenge and they want to compare their skill levels in the beginning and end, they might say, "Okay count how many words you don't recognize and figure out the % of words you think you understand" what's relevant in this context is that we are able to compare their perceived skills level. So It's necessary to specify the method through which they're coming up with the %. The validity of the method itself doesn't matter too much so long as everyone is measuring themselves in the same way. Now in another topic someone posts in the advice center "Help! I studied 10,000 words in Anki, and then read a book and understood 95% of the words, then I traveled to the country and people kept acting strangely towards me and whenever I talked to them it didn't seem like they understood what I meant." What's relevant in this instance is their perceived comprehension versus their actual comprehension, so it might be useful to ask, "how did you determine the 95%? Did you study any grammar? Did you practice any active speech? Did you check yourself against a translation to make sure you actually understood 95%?" People naturally ask for clarification as it's necessary in conversation. I don't think discussion on this forum is a narrow enough context to figure out a best practice for reporting % of comprehension. To come up with a system that works for every situation you would end up reporting irrelevant information which obscures what's most essential and relevant about the % of comprehension reported in each specific context. Edited by YnEoS on 07 August 2014 at 10:58pm 8 persons have voted this message useful | ||||||||||||||||||
| s_allard Triglot Senior Member Canada Joined 5432 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 23 of 211 08 August 2014 at 12:12am | IP Logged | |||||||||||||||||
|
I do not propose a system for measuring comprehension because I think it is unmeasurable; it's a simple as that.
All discussions of percentages of comprehension are doomed to failure, much as we see about fluency. What is true, and I think emk would agree with me, is that one can use word recognition as a very approximate indicator of comprehension. I don't doubt that if you recognize more words, you understand more. Just as long as we don't confuse the two. So, if I see 95.4% automatic word recognition, I am more than happy. But not with 95.4% comprehension. But hold on. It's not over. We're just talking about what I would call Level 1 comprehension, which is difficult enough. If one believes, as I do, in a Level 2, which is an implicit understanding of the underlying grammar, there is an additional difficulty of measurement. I believe both levels are closely intertwined. I believe that you can't fully understand a text, let alone speak well unless you have a very good grasp of the grammar. This is where all those function or grammar words, word endings, verb forms and idioms come into play. In the example given by emk, if you don't know the particular use of the preposition "au" in "au clocher écroulé", you will think that it means "to", which is wrong. But I won't go any further. That's a whole different chapter. 1 person has voted this message useful | ||||||||||||||||||
| Serpent Octoglot Senior Member Russian Federation serpent-849.livejour Joined 6599 days ago 9753 posts - 15779 votes 4 sounds Speaks: Russian*, English, FinnishC1, Latin, German, Italian, Spanish, Portuguese Studies: Danish, Romanian, Polish, Belarusian, Ukrainian, Croatian, Slovenian, Catalan, Czech, Galician, Dutch, Swedish | Message 24 of 211 08 August 2014 at 12:29am | IP Logged | |||||||||||||||||
|
It's not doomed just because you disagree or don't approve. The general consensus seems to be that you can say you ARE fluent in X at B1 active/B2 passive (or B2 both), and you can say you SPEAK fluently/with fluency earlier if your speech is fluid enough, even if the content doesn't match B1. Similarly, most of those who've reached fluency on their own can understand what is implied by 80% comprehension and so on, and what reading fluency or listening fluency mean.
In my experience, it's pretty rare for HTLAL members to follow through on the advice they get and fail. Failures mostly happen if motivation vanishes, or if someone discovers that the proposed methods don't work for them. Misunderstanding the self-assessment and giving bad advice seems uncommon. Edited by Serpent on 08 August 2014 at 12:29am 2 persons have voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
