+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1Wa3tpDUGzv1WR8uDFSG7II329V-MqSg6","timestamp":1742997954666}],"gpuType":"T4"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","source":["### Coding Activity: Assign probabilities to words that can follow a given sentence.\n","\n","Objective: In this notebook, we will explore the fundamentals of language modeling by predicting the next word in a given text prompt. The goal is to assign probability values to a set of candidate words based on the prompt's context and then use probabilistic sampling to generate the next word. We will understand how probabilities influence the generation of language and practice manipulating these probabilities to create sensible sentences, mimicking the basic principle behind language generation.\n","\n","\n","### Libraries\n","\n","- **JAX:** Used for efficient numerical computations and to potentially extend our work into differentiable programming.\n","- **random:** The built-in Python library for probabilistic sampling. In this exercise, we'll use `random.choices` to select the next word based on our probability distribution.\n"],"metadata":{"id":"KA9X4NTg8ev3"}},{"cell_type":"markdown","source":["**Code Type Hinting**\n","\n","[Type hinting in Python](https://docs.python.org/3/library/typing.html) helps you specify the expected data types of variables and function arguments, making your code more readable and less prone to errors.\n","\n","For example, consider the function below:\n","```\n","def greet(name: str) -> str:\n"," \"\"\"Greets the given name.\n"," \n"," Args:\n"," name: The name of the person to greet.\n"," Returns:\n"," A greeting string.\n"," \"\"\"\n"," return f\"Hello, {name}!\"\n","```\n","\n","- `name: str` indicates that the name argument is expected to be a string.\n","\n","- `-> str` specifies that the function will return a string."],"metadata":{"id":"GoI-vdHAqCgt"}},{"cell_type":"markdown","source":["#### Instructions:\n","\n","This notebook aims to deepen your understanding of how context affects language modeling and the importance of probabilistic sampling in text generation.\n","\n","\n","\n","The sentence we are trying to finish is:\n","\n","`Jide was hungry so he went looking for...`\n","\n","Such a sentence can also be called a 'prompt'.\n","\n","You are provided with a list of candidate words that can follow the given prompt: ` ['food','snacks','cookies','he','for','water','photosynthesis','pyramid'].`\n","\n","\n","\n","You will:\n","1. Assign probabilities to each candidate word reflecting their likelihood of being the appropriate next word given the prompt. Consider which words are more likely to be used in this context. Assign higher probabilities to these words and lower probabilities to less likely words.\n","\n","2. Use the `random.choices` method with the assigned probabilities to sample a candidate word.\n","3. Explore how altering the prompt (e.g., changing \"hungry\" to \"thirsty\" or modifying the subject) influences the probability distribution and the predicted next word.\n","\n"," - **Important:** The probabilities assigned to all words *must* sum up to 1. This ensures that we have a valid probability distribution. \n"," - We have added an `assert` statement to the code to verify this condition:\n"," ```python\n"," assert np.isclose(sum(your_mental_model),1) , \"Probabilities must sum to 1!\"\n"," ```\n"," - If the probabilities do not sum to 1, the assertion will fail and produce an error message.\n","\n","\n","Below is a sample prompt and completion exercise for your benefit.\n","\n","Let's get started!"],"metadata":{"id":"EvJnqcpF8yrf"}},{"cell_type":"markdown","source":["\n","Sample prompt = `Twinkle Twinkle little...`"],"metadata":{"id":"p22ADdRZjR5k"}},{"cell_type":"code","source":["import random\n","import numpy as np\n","\n","\n","candidate_words = ['star','beef','bottle']\n","\n","#Enter probabilies for each word here in the format\n","your_mental_model = [0.99,0.001,0.009]\n","\n","print('The sum of your probabilities is: ',sum(your_mental_model))\n","\n","\n","#The probabilities in your mental model must add up to 1 otherwise the following part of the code will throw an error.\n","assert np.isclose(sum(your_mental_model),1) , \"Probabilities must sum to almost 1!\"\n","\n","print(\"You've set the probabilities, now let's finish the sentence based on these!\" )\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"icYkPlVair-Q","executionInfo":{"status":"ok","timestamp":1743440063284,"user_tz":240,"elapsed":8,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"fe37981d-8e32-495b-c524-54fc88dde90b"},"execution_count":1,"outputs":[{"output_type":"stream","name":"stdout","text":["The sum of your probabilities is: 1.0\n","You've set the probabilities, now let's finish the sentence based on these!\n"]}]},{"cell_type":"markdown","source":["The random.choices function allows you to select an item from a list based on specified weights. In our case, the candidate words (e.g., cookies, food, snacks, photosynthesis, midnight, for) will have probabilities (weights) that sum to 1. When you call random.choices with your list of words and their corresponding weights, it will randomly select a word according to these probabilities. More details on random.choices [here].(https://www.geeksforgeeks.org/random-choices-method-in-python/)"],"metadata":{"id":"UJ7GJRyZm5wo"}},{"cell_type":"code","source":["chosen_word = random.choices(candidate_words,weights=your_mental_model)\n","\n","print(\"Twinkle Twinkle little \" + chosen_word[0] + \".\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"JmLQmXVIjgwe","executionInfo":{"status":"ok","timestamp":1743440065627,"user_tz":240,"elapsed":9,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"5588020b-5c70-403f-b371-ac59d1caedb1"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Twinkle Twinkle little star.\n"]}]},{"cell_type":"markdown","source":["### Your activity starts here"],"metadata":{"id":"teWlko-_jq7J"}},{"cell_type":"markdown","source":["\n","Prompt = `Jide was hungry so he went looking for...`"],"metadata":{"id":"aPqCBIUKjKYu"}},{"cell_type":"code","source":["import random\n","import jax\n","candidate_words = ['food','snacks','cookies','he','for','water','photosynthesis','pyramid']\n","\n","#Enter probabilies for each word here in the format shown in the sample above\n","\n","\n","your_mental_model = [0.161, 0.081, 0.032, 0.323, 0.242, 0.129, 0.016, 0.016]\n","\n","print('The sum of your probabilities is: ',sum(your_mental_model))\n","\n","\n","#The probabilities in your mental model must add up to 1 otherwise the following part of the code will throw an error.\n","assert jax.numpy.isclose(sum(your_mental_model),1) , \"Probabilities must sum to almost 1!\"\n","\n","print(\"You've set the probabilities, now let's finish the sentence based on these!\" )"],"metadata":{"id":"Urb2zkFM9a2a","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440194335,"user_tz":240,"elapsed":2675,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"19575065-5432-465f-db5e-a86a41161135"},"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["The sum of your probabilities is: 1.0\n","You've set the probabilities, now let's finish the sentence based on these!\n"]}]},{"cell_type":"markdown","source":["**bold text**\n","\n","*Run this cell multiple times and observe which words are chosen more frequently. Notice how the sentence changes.*"],"metadata":{"id":"gnpziVU2_lXE"}},{"cell_type":"code","source":["chosen_word = random.choices(candidate_words,weights=your_mental_model)\n","\n","print(\"Jide was hungry so he went looking for \" + chosen_word[0] + \".\")"],"metadata":{"id":"jJD4BndPAAp-","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440196447,"user_tz":240,"elapsed":8,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"bfd72923-fc36-4fdc-d5cb-160b5b8f03dd"},"execution_count":4,"outputs":[{"output_type":"stream","name":"stdout","text":["Jide was hungry so he went looking for he.\n"]}]},{"cell_type":"markdown","source":[],"metadata":{"id":"adjiILXAcxNG"}},{"cell_type":"markdown","source":["Notice that even with high probabilities assigned to certain words, the code doesn't *always* choose the word with the highest probability. This is because we are sampling from a probability distribution, not acting deterministically.\n","\n","Deterministic: A deterministic system always produces the same output for a given input. If we were being deterministic and \"greedy,\" we'd always pick the word with the absolute highest probability. Our sentences would become very repetitive.\n","\n","Stochastic: A stochastic system involves randomness. The output is not predetermined, even with the same input. The probabilities guide the outcome, but there's an element of chance. random.choices() performs stochastic sampling. We draw a random word based on the weights, meaning different words can be chosen on different runs, even if one word has a much higher probability than others. This variety is essential for generating diverse and realistic text."],"metadata":{"id":"nc4L-CMgAl32"}},{"cell_type":"markdown","source":["### Importance of Context"],"metadata":{"id":"TXb-VTdjgTxr"}},{"cell_type":"markdown","source":["Language modeling is sensitive to context. In the next part of the exercise, you will observe how a slight change in context alters the probability distribution for the next word.\n","\n","Now, consider a second prompt:\n","`Jide was thirsty so he went looking for`\n","\n","With this context, reassess the candidate words and update your probability estimates. Reflect on how being \"thirsty\" might change which words are likely to follow compared to the original prompt where Jide was \"hungry.\"\n","\n","\n","The list of words to follow is still the same: ` ['food','snacks','cookies','he','for','water','photosynthesis','pyramid'].`\n","\n","\n","How would you change the probabilities of different words now?"],"metadata":{"id":"GSboC6xUBD92"}},{"cell_type":"code","source":["candidate_words = ['food','snacks','leftovers','he','for','water','photosynthesis','pyramid']\n","\n","#Enter probabilies for each word here in the format\n","\n","your_mental_model = [0.161, 0.081, 0.032, 0.323, 0.242, 0.129, 0.016, 0.016]\n","\n","print('The sum of your probabilities is: ',sum(your_mental_model))\n","\n","\n","#The probabilities in your mental model must add up to 1 otherwise the following part of the code will throw an error.\n","assert jax.numpy.isclose(sum(your_mental_model),1) , \"Probabilities must sum to almost 1!\"\n","\n","print(\"You've set the probabilities, now let's finish the sentence based on these!\" )"],"metadata":{"id":"s8wmmLbkBXcj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440202486,"user_tz":240,"elapsed":5,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"ab59760f-445a-4bd7-f008-3cfdb1bfd440"},"execution_count":5,"outputs":[{"output_type":"stream","name":"stdout","text":["The sum of your probabilities is: 1.0\n","You've set the probabilities, now let's finish the sentence based on these!\n"]}]},{"cell_type":"code","source":["chosen_word = random.choices(candidate_words,weights=your_mental_model)\n","\n","print(\"Jide was thirsty so he went looking for \" + chosen_word[0] + \".\")"],"metadata":{"id":"1S0WN2Y2BjTD","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440205753,"user_tz":240,"elapsed":5,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"06ba27d4-b789-474d-e4ad-3d19bc5d66ba"},"execution_count":6,"outputs":[{"output_type":"stream","name":"stdout","text":["Jide was thirsty so he went looking for he.\n"]}]},{"cell_type":"markdown","source":["Finally, explore a further variation. Now the prompt is:\n","\n","`Cookie Monster was hungry so he went looking for`\n","\n","In this scenario, the subject of the sentence has changed. Re-calculate the probabilities for the candidate words. Consider how the persona of Cookie Monster (with his well-known love for cookies) might shift the distribution, possibly favoring different candidate words than before."],"metadata":{"id":"QFpEq4meluUE"}},{"cell_type":"code","source":["candidate_words = ['food','snacks','leftovers','he','for','water','photosynthesis','pyramid']\n","\n","#Enter probabilies for each word here in the format\n","\n","your_mental_model = [0.161, 0.081, 0.032, 0.323, 0.242, 0.129, 0.016, 0.016]\n","\n","print('The sum of your probabilities is: ',sum(your_mental_model))\n","\n","\n","#The probabilities in your mental model must add up to 1 otherwise the following part of the code will throw an error.\n","assert jax.numpy.isclose(sum(your_mental_model),1) , \"Probabilities must sum to almost 1!\"\n","\n","print(\"You've set the probabilities, now let's finish the sentence based on these!\" )"],"metadata":{"id":"kIKl9OYol-Fb","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440210516,"user_tz":240,"elapsed":5,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"3e361f5c-365f-493e-9e33-9e1412397649"},"execution_count":7,"outputs":[{"output_type":"stream","name":"stdout","text":["The sum of your probabilities is: 1.0\n","You've set the probabilities, now let's finish the sentence based on these!\n"]}]},{"cell_type":"code","source":["chosen_word = random.choices(candidate_words,weights=your_mental_model)\n","\n","print(\"Cookie monster was hungry so he went looking for \" + chosen_word[0] + \".\")"],"metadata":{"id":"vATbMSxnmAey","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1743440212899,"user_tz":240,"elapsed":5,"user":{"displayName":"Vishal Patel","userId":"10042835130614330420"}},"outputId":"9015973c-e825-4da7-f713-794301c4ddde"},"execution_count":8,"outputs":[{"output_type":"stream","name":"stdout","text":["Cookie monster was hungry so he went looking for for.\n"]}]}]}
0 commit comments