MPT 7B 64K Context Size Tokens Trained Open Source LLM and ChatGPT GPT4 with Code Interpreter
Full tutorial link > https://www.youtube.com/watch?v=v6TBtyO5Sxg
MosaicML has recently announced the release of #MPT-7B, a new LLM model that has been trained with over 64K+ token size. Additionally, the Code Interpreter feature of ChatGPT/GPT-4 is mind-blowing, and I have reviewed it in this video. Moreover, I show a demo speech of my hopefully upcoming deep voice cloning tutorial.
Our Discord server
https://bit.ly/SECoursesDiscord
If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰
https://www.patreon.com/SECourses
Technology & Science: News, Tips, Tutorials, Tricks, Best Applications, Guides, Reviews
https://www.youtube.com/playlist?list=PL_pbwdIyffsnkay6X91BWb9rrfLATUMr3
Playlist of StableDiffusion Tutorials, Automatic1111 and Google Colab Guides, DreamBooth, Textual Inversion / Embedding, LoRA, AI Upscaling, Pix2Pix, Img2Img
https://www.youtube.com/playlist?list=PL_pbwdIyffsmclLl0O144nQRnezKlNdx3
The Longgboi announcement tweet
https://twitter.com/NaveenGRao/status/1653053055028576256
Code Interpreter article
https://www.oneusefulthing.org/p/it-is-starting-to-get-strange
00:00:00 Introduction to Longgboi of #MosaicML and Code Interpreter of #ChatGPT
00:00:26 Announcement of Naveen Rao, who is CEO of MosaicML
00:00:38 What is context length / token size of ChatGPT
00:00:53 What does context length / token size do
00:01:30 What can you do with 64k context length
00:01:57 What is ChatGPT / GPT4 Code Interpreter
00:02:41 Example of Python code execution by ChatGPT
00:03:19 ChatGPT Code Interpreter is asked with show me something numinous using Python
00:03:41 Uploading excel file without context and asking questions to GPT4 with Code Interpreter
00:04:25 Uploading 60 mb US Census data having excel file to ChatGPT with Code Interpreter
00:05:15 GPT4 has all kind of data visualization capabilities with code interpreter
00:05:30 GPT4 with plugins and browsers
00:05:49 AI reads entire epilogue with my trained voice
MPT-7B, the latest entry in our MosaicML Foundation Series. MPT-7B is a transformer trained from scratch on 1T tokens of text and code. It is open source, available for commercial use, and matches the quality of LLaMA-7B. MPT-7B was trained on the MosaicML platform in 9.5 days with zero human intervention at a cost of ~$200k. Starting today, you can train, finetune, and deploy your own private MPT models, either starting from one of our checkpoints or training from scratch. For inspiration, we are also releasing three finetuned models in addition to the base MPT-7B: MPT-7B-Instruct, MPT-7B-Chat, and MPT-7B-StoryWriter-65k+, the last of which uses a context length of 65k tokens!
The Rise of Large Language Models and their Impact on Text Generation
Introduction:
Language models are computer programs designed to analyze, process and generate language. With the emergence of machine learning and artificial intelligence, these models have become increasingly sophisticated and capable of generating human-like language. Large language models have become particularly popular, due to their ability to process vast amounts of data, and generate coherent and natural language.
What are Large Language Models?
Large language models (LLMs) are deep learning models that are trained on massive amounts of text data. These models are designed to understand the structure of language and the context in which it is used. They work by analyzing the patterns in text data and learning to predict the likelihood of certain words or phrases appearing in a given context.
Context Size:
Context size refers to the amount of text data that a language model processes in order to generate language. The larger the context size, the more accurate and coherent the language generated by the model is likely to be. This is because larger context sizes allow the model to understand the nuances of language better, and to generate more complex and nuanced responses.
Effect of Context Size:
The effect of context size on language generation has been studied extensively, and it has been found that larger context sizes result in more accurate and coherent language generation. This is particularly important when it comes to generating longer pieces of text, such as articles or essays.
Voice Cloning:
Voice cloning is the process of creating a digital replica of a person's voice. This is achieved by training a deep learning model on samples of the person's voice, and then using the model to generate new audio that sounds like the person speaking. Voice cloning has numerous applications, including in the entertainment industry, where it can be used to create digital versions of actors or musicians.
Deep Voice Cloning:
Deep voice cloning refers to the use of deep learning models to create highly realistic and accurate voice clones. These models are trained on vast amounts of audio data, and are capable of generating audio that is almost indistinguishable from the original. Deep voice cloning has many potential applications, including in
-
00:00:00 Greetings everyone. Today I will introduce you two upcoming major events in the AI world.
-
00:00:05 Additionally, I will be demonstrating a demo of my upcoming voice cloning tutorial, where I will
-
00:00:11 have the AI read the entire epilogue generated by the new Longgboi model. The first one is a
-
00:00:18 large language model trained with 64k plus context length. This model is announced by Naveen Rao, who
-
00:00:27 is CEO of MosaicML. MosaicML has Composer PyTorch library for efficient neural network training,
-
00:00:35 so these guys are already experts in the field. When we ask the ChetGPT free edition what is its
-
00:00:42 context length, the answer is only 2k tokens. With 64k token length you can generate 32 times bigger
-
00:00:50 coherent and consistent text than ChatGPT. If you wonder what is context size? Context size refers
-
00:00:57 to the maximum number of tokens or words that a language model can take into consideration when
-
00:01:04 generating text or making predictions. In other words, it is the amount of text that the model
-
00:01:10 can use as input to make a decision or generate a response. The context size of a language model
-
00:01:16 can have significant impact on its performance and accuracy. A larger context size allows the
-
00:01:23 model to take into account more information when making predictions, which can lead to
-
00:01:29 more accurate results. With 64k context length you can input an entire project and ask it to
-
00:01:36 refactor or upload an entire document and ask any questions about it. This can dramatically improve
-
00:01:42 basically anything regarding analysis of text, data, and programming. The name of the announced
-
00:01:48 model is Longgboi and it will be open-sourced at the end of this week. Once it is released,
-
00:01:54 hopefully I will make a tutorial for how to use it. And the second big upcoming thing is ChatGPT
-
00:02:01 with code interpreter. Code interpreter feature of ChatGPT is still in alpha mode and only available
-
00:02:08 to those who are selected for alpha testing. So it is not public yet. Code interpreter is GPT-4
-
00:02:15 with 3 new capabilities. The AI can read files. You upload up to 100 megabytes which is huge. It
-
00:02:23 can let you download files and it lets the AI run its own Python code. Now this is significant. With
-
00:02:31 code interpreter, the ChatGPT, the GPT-4 will be able to run its own Python code. This is amazing.
-
00:02:39 So here an example of Python code execution. The input is like this: I am writing a blog post
-
00:02:47 about how amazing ChatGPT is at working with code right now. I would like you to create the perfect
-
00:02:54 illustration a gif using Python that represents this ability. Decide what an appropriate amazing
-
00:03:01 gif would be. Then figure out how to create it and let me download it. So the ChatGPT with code
-
00:03:08 interpreter is writing this code and then it is generating this gif image you are seeing
-
00:03:14 right now and letting author to download it. It is just amazing. But this is not all. The
-
00:03:20 first question author asked is show me something numinous using Python. And the ChatGPT with code
-
00:03:27 interpreter is saying this. Let's use Python to create a visualization of Mandelbrot set which may
-
00:03:34 evoke a sense of awe and wonder. And this is the output. This is just significant. This is huge.
-
00:03:41 And here another use case. The authors uploads an excel file without providing any context and
-
00:03:49 asks three questions. Can you do visualization and descriptive analysis to help me understand
-
00:03:54 the data? This is huge. This is significant. Can you try regressions and look for patterns? Can
-
00:04:01 you run regression diagnostics? These are the fields that requires expertise. And now with
-
00:04:08 GPT-4 and code interpreter, you won't need expert people for this. The model will be able to provide
-
00:04:15 you these informations. I will put the link of this page into description so you can check it
-
00:04:21 out if you are interested in more details. And now another major thing that GPT-4 with code
-
00:04:28 interpreter. The author uploads 60MB U.S. Census dataset and asks the AI to explore the data,
-
00:04:36 generate its own hypothesis based on the data, conduct hypothesis tests, and write a paper based
-
00:04:43 on its results. This requires huge expertise, huge amount of working time, and other things. However,
-
00:04:50 the model is able to do all of this in several seconds and give you output. You see,
-
00:04:56 it wrote an academic paper about it and here the abstract of the paper. According to the author,
-
00:05:03 it is not a stunning paper yet. Of course, we are not expecting it, but this is a stunning
-
00:05:09 work of AI model and it will only get better over time. And another thing is, it does every
-
00:05:15 data visualization that the author can think of. This is amazing. Below the author shown some of
-
00:05:21 the data visualization graphics and he didn't even provide data. He said model to generate fake data
-
00:05:28 to just demonstrate these graphs. The GPT-4 will also get plugins and browsers. However,
-
00:05:35 according to the author, they are still not very good. Plugins and browser support is still not
-
00:05:41 public, only in alpha mode and only available to those selected few people. So as a final thing,
-
00:05:49 I will make my cloned voice to read entire tweet of Naveen Rao. Hopefully, I will make
-
00:05:57 a full tutorial about how to train a voice and generate as much as possible natural sounding,
-
00:06:04 awesome quality voices for free on your computer with only 6GB VRAM having GPUs.
-
00:06:11 Large language model trained with 64000 plus context length! What could you do with that?
-
00:06:19 Prompted our model with the ENTIRE contents of The Great Gatsby and asked it to write the epilogue.
-
00:06:25 Model dropping soon to an open source repo near you. Epilogue: It seemed to me that Gatsby was
-
00:06:31 already dead when I went up to East Egg. He'd died before I could find out. I thought this, as I
-
00:06:38 climbed the old stone steps they rose up a little now, but were all the same and the grass had begun
-
00:06:45 to grow in the cracks On the steps the wind had broken the yellow tape that cordoned them off.
-
00:06:51 I tried to walk over once and had been turned back by a policeman. I had seen something like an
-
00:06:56 exclamation point in the tape when I first looked at it, but when I looked again, I couldn't see it
-
00:07:02 anymore. It had been torn away long ago, no doubt by a little boy running after his ball, but why
-
00:07:08 should I believe that he was still on the other side running after his ball? After all the years,
-
00:07:13 I would remember this little boy. Perhaps it was some cousin or something of that sort,
-
00:07:18 and I would see him running like a young deer across the grass with the sunrise flashing in
-
00:07:24 his face. It mightn't be that the boy was really there, but the sky might be brighter because
-
00:07:30 the sunlight came in from a different place. The night of Gatsby's burial came as a surprise to me.
-
00:07:36 I had made a point of coming down from my hotel early and driving out to his house in
-
00:07:42 the Ford Roadster and I had gone over the whole course of that morning in memory.
-
00:07:47 I hadn't expected there would be anything I could actually see on
-
00:07:49 the ground. It would show that the old place had once been inhabited,
-
00:07:53 but the moon was full and the white beach had a ghostly sheen. I could have seen the grass
-
00:08:00 growing and the stones standing white and unbroken forever. The cemetery was at the foot of the hill,
-
00:08:08 and when I came down the steep slope, there were white boulders piled up under my feet where they
-
00:08:13 had tumbled away from his grave in the rain in the rain. Gatsby was buried in a small stone
-
00:08:19 square at the foot of the hill. When I came out on the lawn, there was an open grave. There had
-
00:08:25 been none at first, and standing around was a crowd of people, many of them old acquaintances.
-
00:08:31 I could hear a child weeping and a woman's voice singing and weeping. A man came forward I think it
-
00:08:38 was the minister and offered me his hand. He was a strange looking man, tall and thin, but I couldn't
-
00:08:44 quite see through his spectacles. You're a good fellow nick, a real good fellow he said to me,
-
00:08:50 you know what the minister gonna say I did not know and was surprised. Mr. Carraway, that was
-
00:08:56 a good boy he repeated. He was always good to us. I am I said and I thought of my mother,
-
00:09:03 where they go for minister now said the other man, a squat man with a pale beard. He was standing at
-
00:09:09 the grave with his hand on his cane and smiling in this lot. I feel bad about all of you. He was
-
00:09:15 saying that my boy's gone and left us with so much work. But he's gone like you said, into the good
-
00:09:22 place. That's where he's at now that's where he'll stay all the rest of eternity and we can
-
00:09:27 get together there. All of us. What a thing it was when he first came out here and got so many
-
00:09:32 ideas that he forgot his own family. We had to help that boy see the truth and now he's seen it.
-
00:09:38 A woman in the crowd was getting a short, passionate wail, almost a howl over by the grave.
-
00:09:44 He's gone to another house. The woman stopped. The woman stopped. All the mourners stopped.
-
00:09:51 The minister who had seen the change in me said there was a time you know there wasn't much chance
-
00:09:58 for him, but now he's right. It's hard, but you must believe you know the saying. A young woman
-
00:10:05 began to weep then and other people took it up. They stood at the grave looking at one another,
-
00:10:11 talking and crying. And then at a signal from the minister, they began to move off in the moonlight.
-
00:10:16 I went up the hill in the fall trying to arrange in my mind the order in which I would tell people.
-
00:10:21 This is all for today. Hopefully more awesome tutorials are coming soon. Please like subscribe,
-
00:10:28 leave a comment. If you also support us on Youtube by joining or also support us on Patreon,
-
00:10:34 I would appreciate that very much. Please also join our Discord channel. The link of
-
00:10:39 the Discord channel and our Patreon will be in the description of the video and also in the comment
-
00:10:45 section of the video as a pinned comment. Also, I am open the consultation with Stable
-
00:10:50 Diffusion related stuff. If you are interested in, just support us on Patreon and contact me and
-
00:10:55 hopefully I will try to help you privately. Hopefully see you in another amazing video.