frequency

Author: parmentelat

notebooks/_toc.yml

@@ -9,10 +9,12 @@ parts:
   - file: exos/beginners/EXO-countdown-nb
   # import, argparse
   - file: exos/beginners/EXO-guess-nb
-  # slicing conditional expression return/break
+  # slicing, conditional expression, return/break
   - file: exos/beginners/EXO-palindrom-nb
-  # 
+  # sorted
   - file: exos/beginners/EXO-anagrams-nb
+  # counter, regexp 
+  - file: exos/beginners/EXO-frequency-nb
 - caption: exercices basiques
   chapters:
   # str, ints, binary computations

notebooks/exos/beginners/EXO-frequency-nb.md

@@ -0,0 +1,94 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.4
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+---
+
+# fréquence des mots
+
++++
+
+## le sujet
+
+il s'agit d'écrire un programme qui lit un fichier texte, et calcule la fréquence
+des mots présents dans ce fichier
+
+par exemple ici nous allons afficher les <n> mots les plus fréquents, avec leur nombre d'apparitions  
+par exemple avec (ce fichier est fourni)  
+
+```bash
+python frequency.py -n 5 frequency-sample.txt
+the 65
+he 62
+a 52
+to 52
+it 45
+```
+
++++
+
+## un outil bien pratique
+
+pour faire ça, on va utiliser un module de la librairie standard,
+`collections.Counter`; entrainez-vous à trouver sa doc sur `docs.python.org`, et à chercher une méthode qui va nous aider à faire ce qui est demandé ici
+
+````{admonition} une méthode en particulier
+:class: dropdown
+
+et notamment la méthode `most_common()`
+````
+
++++
+
+## solutions
+
+### v0: très rustique (marche assez mal)
+
+pour montrer comment on utilise `Counter`, on va couper les mots à la serpe avec
+`split()`, et on va ignorer la ponctuation
+
+````{admonition} pour voir la v0
+:class: dropdown
+
+```{literalinclude} frequency_v0.py
+```
+````
+
++++
+
+### v1: un peu comme pour palindrom
+
+ici on va remplacer tous les caractères de ponctuation par des espaces avant de
+couper en morceaux avec `split()`; on montre 3 façons d'obtenir ces caractères
+de ponctuation (et le mieux c'est d'importer ce qu'on avait fait pour le
+palindrome)
+
+````{admonition} la v1
+:class: dropdown
+
+```{literalinclude} frequency_v1.py
+```
+````
+
++++
+
+### v2: avec les expressions régulières
+
+enfin on montre comment on ferait en vrai; c'est moins lisible car ça utilise
+les expressions régulières, un sujet disons un peu aride de prime abord, mais
+comme vous pouvez le voir ça donne une solution beaucoup plus propre, et
+globablement plus efficace aussi
+
+````{admonition} la v2
+:class: dropdown
+
+```{literalinclude} frequency_v2.py
+```
+````

notebooks/exos/beginners/frequency-sample.txt

@@ -0,0 +1,102 @@
+Chapter One
+
+The house stood on a slight rise just on the edge of the village. It stood on its own and looked out over a broad spread of West Country farmland. Not a remarkable house by any means—it was about thirty years old, squattish, squarish, made of brick, and had four windows set in the front of a size and proportion which more or less exactly failed to please the eye.
+
+The only person for whom the house was in any way special was Arthur Dent, and that was only because it happened to be the one he lived in. He had lived in it for about three years, ever since he had moved out of London because it made him nervous and irritable. He was about thirty as well, tall, dark-haired and never quite at ease with himself. The thing that used to worry him most was the fact that people always used to ask him what he was looking so worried about. He worked in local radio which he always used to tell his friends was a lot more interesting than they probably thought. It was, too—most of his friends worked in advertising.
+
+On Wednesday night it had rained very heavily, the lane was wet and muddy, but the Thursday morning sun was bright and clear as it shone on Arthur Dent’s house for what was to be the last time.
+
+It hadn’t properly registered yet with Arthur that the council wanted to knock it down and build a bypass instead.
+
+
+At eight o’clock on Thursday morning Arthur didn’t feel very good. He woke up blearily, got up, wandered blearily round his room, opened a window, saw a bulldozer, found his slippers, and stomped off to the bathroom to wash.
+
+Toothpaste on the brush—so. Scrub.
+
+Shaving mirror—pointing at the ceiling. He adjusted it. For a moment it reflected a second bulldozer through the bathroom window. Properly adjusted, it reflected Arthur Dent’s bristles. He shaved them off, washed, dried and stomped off to the kitchen to find something pleasant to put in his mouth.
+
+Kettle, plug, fridge, milk, coffee. Yawn.
+
+The word bulldozer wandered through his mind for a moment in search of something to connect with.
+
+The bulldozer outside the kitchen window was quite a big one.
+
+He stared at it.
+
+'Yellow,' he thought, and stomped off back to his bedroom to get dressed.
+
+Passing the bathroom he stopped to drink a large glass of water, and another. He began to suspect that he was hung over. Why was he hung over? Had he been drinking the night before? He supposed that he must have been. He caught a glint in the shaving mirror. “Yellow,” he thought, and stomped on to the bedroom.
+
+He stood and thought. The pub, he thought. Oh dear, the pub. He vaguely remembered being angry, angry about something that seemed important. He’d been telling people about it, telling people about it at great length, he rather suspected: his clearest visual recollection was of glazed looks on other people’s faces. Something about a new bypass he’d just found out about. It had been in the pipeline for months only no one seemed to have known about it. Ridiculous. He took a swig of water. It would sort itself out, he’ d decided, no one wanted a bypass, the council didn’t have a leg to stand on. It would sort itself out.
+
+God, what a terrible hangover it had earned him though. He looked at himself in the wardrobe mirror. He stuck out his tongue. 'Yellow,' he thought. The word yellow wandered through his mind in search of something to connect with.
+
+Fifteen seconds later he was out of the house and lying in front of a big yellow bulldozer that was advancing up his garden path.
+
+
+Mr. L. Prosser was, as they say, only human. In other words he was a carbon-based bipedal life form descended from an ape. More specifically he was forty, fat and shabby and worked for the local council. Curiously enough, though he didn’t know it, he was also a direct male-line descendant of Genghis Khan, though intervening generations and racial mixing had so juggled his genes that he had no discernible Mongoloid characteristics, and the only vestiges left in Mr. L. Prosser of his mighty ancestry were a pronounced stoutness about the tum and a predilection for little fur hats.
+
+He was by no means a great warrior; in fact he was a nervous, worried man. Today he was particularly nervous and worried because something had gone seriously wrong with his job, which was to see that Arthur Dent’s house got cleared out of the way before the day was out.
+
+“Come off it, Mr. Dent,” he said, “you can’t win, you know. You can’t lie in front of the bulldozer indefinitely.” He tried to make his eyes blaze fiercely but they just wouldn’t do it.
+
+Arthur lay in the mud and squelched at him.
+
+“I’m game,” he said, “we’ll see who rusts first.”
+
+“I’m afraid you’re going to have to accept it,” said Mr. Prosser, gripping his fur hat and rolling it round the top of his head; “this bypass has got to be built and it’s going to be built!”
+
+“First I’ve heard of it,” said Arthur, “why’s it got to be built?”
+
+Mr. Prosser shook his finger at him for a bit, then stopped and put it away again.
+
+“What do you mean, why’s it got to be built?” he said. “It’s a bypass. You’ve got to build bypasses.”
+
+Bypasses are devices that allow some people to dash from point A to point B very fast while other people dash from point B to point A very fast. People living at point C, being a point directly in between, are often given to wonder what’s so great about point A that so many people from point B are so keen to get there, and what’s so great about point B that so many people from point A are so keen to get there. They often wish that people would just once and for all work out where the hell they wanted to be.
+
+Mr. Prosser wanted to be at point D. Point D wasn’t anywhere in particular, it was just any convenient point a very long way from points A, B and C. He would have a nice little cottage at point D, with axes over the door, and spend a pleasant amount of time at point E, which would be the nearest pub to point D. His wife of course wanted climbing roses, but he wanted axes. He didn’t know why—he just liked axes. He flushed hotly under the derisive grins of the bulldozer drivers.
+
+He shifted his weight from foot to foot, but it was equally uncomfortable on each. Obviously somebody had been appallingly incompetent and he hoped to God it wasn’t him.
+
+Mr. Prosser said, “You were quite entitled to make any suggestions or protests at the appropriate time, you know.”
+
+“Appropriate time?” hooted Arthur. “Appropriate time? The first I knew about it was when a workman arrived at my home yesterday. I asked him if he’d come to clean the windows and he said no, he’d come to demolish the house. He didn’t tell me straight away of course. Oh no. First he wiped a couple of windows and charged me a fiver. Then he told me.”
+
+“But Mr. Dent, the plans have been available in the local planning office for the last nine months.”
+
+“Oh yes, well, as soon as I heard I went straight round to see them, yesterday afternoon. You hadn’t exactly gone out of your way to call attention to them, had you? I mean, like actually telling anybody or anything.”
+
+“But the plans were on display . . .”
+
+“On display? I eventually had to go down to the cellar to find them.”
+
+“That’s the display department.”
+
+“With a flashlight.”
+
+“Ah, well, the lights had probably gone.”
+
+“So had the stairs.”
+
+“But look, you found the notice, didn’t you?”
+
+“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.’”
+
+A cloud passed overhead. It cast a shadow over Arthur Dent as he lay propped up on his elbow in the cold mud. It cast a shadow over Arthur Dent’s house. Mr. Prosser frowned at it.
+
+“It’s not as if it’ s a particularly nice house,” he said.
+
+“I’m sorry, but I happen to like it.”
+
+“You’ ll like the bypass.”
+
+“Oh, shut up,” said Arthur Dent. “Shut up and go away, and take your bloody bypass with you. You haven’t got a leg to stand on and you know it.”
+
+Mr. Prosser’s mouth opened and closed a couple of times while his mind was for a moment filled with inexplicable but terribly attractive visions of Arthur Dent’ s house being consumed with fire and Arthur himself running screaming from the blazing ruin with at least three hefty spears protruding from his back. Mr. Prosser was often bothered with visions like these and they made him feel very nervous. He stuttered for a moment and then pulled himself together.
+
+“Mr. Dent,” he said.
+
+“Hello? Yes?” said Arthur.
+
+“Some factual information for you. Have you any idea how much damage that bulldozer would suffer if I just let it roll straight over you?”
+

notebooks/exos/beginners/frequency_v0.py

@@ -0,0 +1,49 @@
+"""
+un programme qui calcule le nombre d'occurrences des mots dans un fichier texte
+- on entre le nom du fichier par la ligne de commandes
+- par défaut on montre les 3 mots les plus fréquents, on peut changer ce nombre
+  également sur la ligne de commandes
+"""
+
+# dans cette v0 on ignore la ponctuation
+# le résultat est donc un peu plus "brut"
+# puisque "Hello," ne va pas être compté comme "hello"
+
+from collections import Counter
+from argparse import ArgumentParser
+
+def compute_counter(filename) -> Counter:
+    try:
+        # ici le fait de lire tout le fichier d'un coup
+        # simplifie le code
+        with open(filename) as reader:
+            text = reader.read().lower().split()
+            counter = Counter(mots)
+            return counter
+    except FileNotFoundError:
+        print(f"OOPS le fichier {filename} n'existe pas")
+        return Counter()
+
+
+def main():
+    parser = ArgumentParser()
+    # par défaut on montre les 3 mots les plus fréquents
+    parser.add_argument(
+        "-n", "--number", default=3, type=int,
+        help="the number of words to show")
+    parser.add_argument("filename")
+    args = parser.parse_args()
+
+    filename = args.filename
+    n = args.number
+
+    counter = compute_counter(filename)
+    if len(counter) == 0:
+        print("vide")
+    else:
+        for word, occurrences in counter.most_common(n):
+            print(word, occurrences)
+
+
+if __name__ == '__main__':
+    main()

notebooks/exos/beginners/frequency_v1.py

@@ -0,0 +1,66 @@
+"""
+un programme qui calcule le nombre d'occurrences des mots dans un fichier texte
+- on entre le nom du fichier par la ligne de commandes
+- par défaut on montre les 3 mots les plus fréquents, on peut changer ce nombre
+  également sur la ligne de commandes
+"""
+
+# v1: un peu mieux car on traite la ponctuation
+# mais c'est clairement sous-optimal en termes de complexité
+# (près d'un millier de caractères de ponctuation)
+
+from collections import Counter
+from argparse import ArgumentParser
+
+# plusieurs options pour calculer une chaine
+# qui contient les caractères de ponctuation
+
+# option 0: on entre à la main les caractères de ponctuation
+# dans notre texte; bon c'est sous-optimal mais en dernier recours..
+# notez l'utilisation de """ comme délimiteur
+# PONCTUATION = """,.;'"“”"""
+
+# option 1: ne marche pas bien car seulement ASCII
+# from string import punctuation
+
+# option 2: voir le palindrome
+from palindrom_v2 import unicode_punctuation
+PUNCTUATION = unicode_punctuation()
+
+def compute_counter(filename) -> Counter:
+    try:
+        with open(filename) as reader:
+            text = reader.read().lower()
+            # on remplace les caractères de ponctuation par un espace
+            for char in PUNCTUATION:
+                text = text.replace(char, " ")
+            mots = text.split()
+            counter = Counter(mots)
+            return counter
+    except FileNotFoundError:
+        print(f"OOPS le fichier {filename} n'existe pas")
+        return Counter()
+
+
+def main():
+    parser = ArgumentParser()
+    # par défaut on montre les 3 mots les plus fréquents
+    parser.add_argument(
+        "-n", "--number", default=3, type=int,
+        help="the number of words to show")
+    parser.add_argument("filename")
+    args = parser.parse_args()
+
+    filename = args.filename
+    n = args.number
+
+    counter = compute_counter(filename)
+    if len(counter) == 0:
+        print("vide")
+    else:
+        for word, occurrences in counter.most_common(n):
+            print(word, occurrences)
+
+
+if __name__ == '__main__':
+    main()

notebooks/exos/beginners/frequency_v2.py

@@ -0,0 +1,52 @@
+"""
+un programme qui calcule le nombre d'occurrences des mots dans un fichier texte
+- on entre le nom du fichier par la ligne de commandes
+- par défaut on montre les 3 mots les plus fréquents, on peut changer ce nombre
+  également sur la ligne de commandes
+"""
+
+# une deuxième version en utilisant les expressions régulières
+
+from collections import Counter
+from argparse import ArgumentParser
+
+import re
+
+def compute_counter(filename) -> Counter:
+    try:
+        # ici le fait de lire tout le fichier d'un coup
+        # simplifie le code
+        with open(filename) as reader:
+            text = reader.read().lower()
+            words_stream = re.findall(r'\w+', text)
+            counter = Counter(words_stream)
+            return counter
+            # on aurait pu raccourcir ces 4 lignes en une seule:
+            # return Counter(re.findall(r'\w+', reader.read().lower()))
+    except FileNotFoundError:
+        print(f"OOPS le fichier {filename} n'existe pas")
+        return Counter()
+
+
+def main():
+    parser = ArgumentParser()
+    # par défaut on montre les 3 mots les plus fréquents
+    parser.add_argument(
+        "-n", "--number", default=3, type=int,
+        help="the number of words to show")
+    parser.add_argument("filename")
+    args = parser.parse_args()
+
+    filename = args.filename
+    n = args.number
+
+    counter = compute_counter(filename)
+    if len(counter) == 0:
+        print("vide")
+    else:
+        for word, occurrences in counter.most_common(n):
+            print(word, occurrences)
+
+
+if __name__ == '__main__':
+    main()