|
 |
Experimenting with French word frequency | ||
Tags: Word Frequency | French | ||
Share with:
| ||
 Language Learning Forum : Specific Languages |  | |
 emk Diglot Moderator United States Joined 5320 days ago 2615 posts - 8806 votes  Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 1 of 55 04 September 2014 at 1:00pm | IP Logged | ||||||||||||||
|
While searching for French frequency data, I found the rather cool Lexique project from the Université de Savoie. They provide an open source database of over 46,947 head words and 142,694 inflected forms. It includes two sets of frequency data: frequency in books, and frequency in movie subtitles. The latter makes for a pretty reasonable spoken corpus.
One drawback of this database is that it can't connect the various parts of speech—it treats aimer, aimant and aimé as separate words. In an effort to fix that, I fed it through the Snowball stemmer, which is often used to implement computerized word search. But that produced rather weird results, including:
So, yeah, that's not going to work. Snowball may be the standard stemmer, but it's dumb. Let's stop using Snowball, because we don't want to treat tu "you" and tuer "to kill" as the same word—nor bonasse "sexy woman" and bonne "female servant" for that matter. So let's try again. Without Snowball to collapse forms like aimer, aimant and aimé, here are the top 300 words of French, based on movie subtitles:
Interestingly, it counts le, la and l' separately. And it expects us to spend some of our top 300 word slots on ah, eh and ouais "yup" (which is only fair in spoken French, I suppose). You may also notice that words like peur and mort are quite high in the list, probably thanks to action movie subtitles. But it's still better than mixing up tu and tuer, or pulling our frequency distribution from Le Monde articles. These are all terribly useful words, of course. We could probably make some very basic small talk, but we couldn't really go shopping or ask where to find the bathroom. In the next episode: Improving s_allard's recommended list of 600 words of spoken French. 4 persons have voted this message useful | |||||||||||||||
| rdearman Senior Member United Kingdom rdearman.orgRegistered users can see my Skype Name Joined 5024 days ago 881 posts - 1812 votes Speaks: English* Studies: Italian, French, Mandarin | Message 2 of 55 04 September 2014 at 1:08pm | IP Logged | ||||||||||||||
So are you using the intersection of those two lists or are you looking at them independently? Or just looking at film in order to answer/reply/investigate s_allards link? 1 person has voted this message useful | |||||||||||||||
| s_allard Triglot Senior Member Canada Joined 5218 days ago 2704 posts - 5425 votes Speaks: French*, English, Spanish Studies: Polish | Message 3 of 55 04 September 2014 at 1:42pm | IP Logged | ||||||||||||||
|
I'm not sure that movie subtitles make the best sample set for spoken French or written French. Why go to all this
trouble when there are a number of frequency lists out there that one can use. Here for example is one such list: A French word frequency list When I get a minute I'll comment on the contents of the list suggested by emk, but I question the strategy of looking at the top 300 words of frequency lists in general. What I prefer to do is to look at actual conversations and see how many words are used. When we look at vast sample of texts, we always run into that lexical aggregate problem. The issue isn't what the sum of all movie subtitles says it's more what does one movie say. Since I have a class in a few minutes, I'll come back later with results from looking at the words used in conversations in France bienvenue 2 persons have voted this message useful | |||||||||||||||
| Doitsujin Diglot Senior Member Germany Joined 5108 days ago 1256 posts - 2363 votes Speaks: German*, English | Message 4 of 55 04 September 2014 at 2:00pm | IP Logged | ||||||||||||||
Check out the dsl2mobi GitHub website. The website hosts inflections lists for 9 common languages, including French. For example, the entry for aimer looks like this:
This should be more than sufficient to reduce inflected forms to their canonical forms. 3 persons have voted this message useful | |||||||||||||||
| Jeffers Senior Member United Kingdom Joined 4697 days ago 2151 posts - 3960 votes Speaks: English* Studies: Hindi, Ancient Greek, French, Sanskrit, German | Message 5 of 55 04 September 2014 at 2:05pm | IP Logged | ||||||||||||||
The wiktionary list looks like a good idea, until you realize they didn't remove capital letters, so not only does it count "le" and "la" as separate words, it also counts "Le" and "La" separately. EDIT:
I couldn't quite figure out this objection. You can't learn the words you need for an actual conversation after having the conversation. So you learn the words most likely to come up in conversation, and then fake it, or ask for clarification, for the odd words which will turn up in any given conversation. Edited by Jeffers on 04 September 2014 at 2:12pm 4 persons have voted this message useful | |||||||||||||||
| rdearman Senior Member United Kingdom rdearman.orgRegistered users can see my Skype Name Joined 5024 days ago 881 posts - 1812 votes Speaks: English* Studies: Italian, French, Mandarin | Message 6 of 55 04 September 2014 at 2:26pm | IP Logged | ||||||||||||||
Those are easy enough to strip out, a one line command in Linux. cat FR_WORDS.txt | tr '[:upper:]' '[:lower:]' | uniq EDIT: Fixed bad grammar. Edited by rdearman on 04 September 2014 at 2:26pm 1 person has voted this message useful | |||||||||||||||
| emk Diglot Moderator United States Joined 5320 days ago 2615 posts - 8806 votes Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 7 of 55 04 September 2014 at 2:33pm | IP Logged | ||||||||||||||
Well, virtually all of the frequency lists that I can find online are based on newspaper articles and books. We could use book data as a basis for these lists, but it actually gives pretty awful results for spoken French. The movie data is much more representative of ordinary dialog, even if mort and tuer and fric are a bit too frequent. Anyway, you recently wrote:
Well, as you mentioned, this list includes no pronouns, adverbs, articles, demonstratives or other helper words. To help you improve your list, here are 350 words that aren't included in your "excellent foundation" list:
Again, there's a few duplicates here, such as aucun aucune. But it's a good starting point for improving your list. Obviously, you want to include things like je "I" and bien "well" and et "and" and un "a" and mais "but" before you claim somebody has an "excellent foundation." And if you want to go shopping, you probably want coûter "to cost". And it would help to add dix voilà voici devant enfin pardon anglais hier and a whole bunch of other stuff from this list. You could get by without vaisseau. The profanity? It's probably actually pretty useful if you're younger and you want to make casual friends. But even if we take your 600 word list as a base, we can almost certainly find 200 words on the list above that would be awfully handy even for small talk. Do you have suggestions about how to take your 600 and the 350 above, and squash them down to 300 words? Ideally, I'd like to be able to satisfy the A2 spoken interaction criteria, which seem like a pretty good standard for basic survival:
According to the Milton's study I mentioned earlier, the typical A2 French student knows about 1700 of the most frequent 5000 words. Can we actually whittle that down to a list of 300 and still meet the A2 criteria? Honestly, I just don't see how you can fit everything necessary into 300 words: something's got to give.
You make various claims about 300 words, but you don't seem to mean an actual list of 300 words that we could just go ahead and write down and make into an Anki deck. You mean an constantly-changing list of 300 words that sometimes includes vrai and sometimes includes chez and sometimes includes coûter, but the actual makeup of the list can change with every conversation. But this is sort of the whole point: Even if you try to limit yourself to A2 conversations (small talk, transportation and shopping, basically), you still need a lot of words even within that limited beginner domain. The only way I'm going to believe that 300 words will get me anywhere useful is if somebody actually writes them down on a list, and forces themselves to make the hard choices. I can't make those choices for you, because whichever list I pick, you're going to see some obvious problems: a list based on newspapers won't be conversational enough. A list based on movies won't include enough shopping. And this is more or less exactly my point: the choices are just too hard when the list is that short, and you can't really even cover the A2 basics without leaving out something important. So I'm going to put forth a challenge: If you believe that 300 words is enough to get by, please make a list. Make those hard decisions. 5 persons have voted this message useful | |||||||||||||||
| emk Diglot Moderator United States Joined 5320 days ago 2615 posts - 8806 votes Speaks: English*, FrenchB2 Studies: Spanish, Ancient Egyptian Personal Language Map | Message 8 of 55 04 September 2014 at 2:54pm | IP Logged | ||||||||||||||
The Wiktionary list is junk. It lists Belgique as the 93rd most common word in French. Seriously, the Lexique data set is awesome. It has frequency data for both movies and books, it has parts of speech, it has genders, and it has the common inflectional forms:
Like I said, the only peculiarity of this data set is that it distinguishes words by parts of speech: aimer "to love", aimé "loved" and aimant "loving" are treated as separates lexemes. It's open source, and it's far more useful than most of the garbage lists you'll find online.
Now that looks promising. At the very least, it will allow me to collapse related forms in the Lexique data set. Thank you for the link. EDIT: Drat:
No aimé, no aimant. Still, it was a good try! Edited by emk on 04 September 2014 at 3:04pm 2 persons have voted this message useful |
|  | 
Printable version
|
| Copyright 2024 FX Micheloud - All rights reserved No part of this website may be copied by any means without my written authorization. |
