Updated to Karpathy's vidoe notes (without Info folder)

Author: prabha-git

.gitignore

@@ -6,3 +6,4 @@ scratch.md
 Templates
 Daily Notes
 Projects
+Info

assets/images/Pasted image 20250205123847.png

@@ -0,0 +1,56 @@
+
+
+# NTT data 
+
+## When was the last time you used this skill? - OpenAI
+- Developed an internal Q&A Chatbot using OpenAI's Chat models (GPT-3.5 and GPT-4) and Embedding Models, implementing a Retrieval-Augmented Generation (RAG) system for enhanced performance and accuracy.
+
+- Designed and conducted evaluations of Open Source models using OpenAI's GPT-4 as a benchmark, providing valuable insights into the performance and capabilities of various models.
+
+- Fine-tuned OpenAI's GPT-3.5 model using public datasets available on Huggingface, successfully reducing hallucinations and improving the model's overall reliability and coherence.
+
+
+## When was the last time you used this skill? - Language Model 
+
+Developed POC generative AI solutions, utilizing advanced prompt engineering techniques (Chain of Thought, ReAct) and fine-tuned models (Google's Text Bison and OpenAI 3.5) for improved performance and reduced hallucinations.
+Skilled in creating multimodal models, integrating LLMs with structured databases, and leveraging frameworks like Langchain, DSPy, Instructor, and Pydantic for building generative AI applications.
+Architected a system for generating personalized social media content using customer-specific data, and worked with in-memory and cloud vector databases for embedding management and similarity search.
+Actively contributed to open-source projects (Needle in Haystack analysis, Langchain, DSPy) and utilized DevOps platforms (Langsmith, phoenix-arize) for developing, testing, and deploying LLM applications.
+
+## When was the last time you used this skill? - Tensorflow
+
+Used keras to finetune and deploy smaller open-source model like gemma2b
+
+
+
+## Speech to Text
+
+Used Google speech-to-text service to create a transcription of videos
+
+
+## LLM 
+
+Developed POC generative AI solutions, utilizing advanced prompt engineering techniques (Chain of Thought, ReAct) and fine-tuned models (Google's Text Bison and OpenAI 3.5) for improved performance and reduced hallucinations.
+Skilled in creating multimodal models, integrating LLMs with structured databases, and leveraging frameworks like Langchain, DSPy, Instructor, and Pydantic for building generative AI applications.
+Architected a system for generating personalized social media content using customer-specific data, and worked with in-memory and cloud vector databases for embedding management and similarity search.
+Actively contributed to open-source projects (Needle in Haystack analysis, Langchain, DSPy) and utilized DevOps platforms (Langsmith, phoenix-arize) for developing, testing, and deploying LLM applications.
+
+## NTLK 
+
+Sentimental Analysis on Customer Support Emails 
+
+## AI 
+
+Worked on Churn model to predict it 2-3 months before it happens and find the leading indicator that is causing the churn
+
+
+## Vector Database
+
+Experienced in working with in-memory and cloud vector databases, such as Pinecone and Weaviate, for efficient embedding management and similarity search.
+- Utilized vector databases to support the development of LLM-based applications, enabling fast and accurate retrieval of relevant information for generating personalized content and insights.
+- Proficient in setting up schemas and leveraging advanced filtering techniques using metadata in Pinecone and Weaviate cloud databases, ensuring optimized performance and refined search results.
+- Implemented on-premise vector database solutions using Postgres with the pgvector extension for customers who prefer to keep their data in-house, adapting to their specific requirements and constraints.
+-Integrated vector databases with LLM frameworks, like Langchain and DSPy, to create end-to-end solutions that combine the power of language models with fast and accurate information retrieval.
+- Utilized Langchain Indexing for continuous embedding of documents into vector databases, enabling efficient and cost-effective embedding for Retrieval-Augmented Generation (RAG) systems, enhancing the quality and relevance of generated content.
+
+## Langchain 

docs/writing/index.md

@@ -26,7 +26,7 @@ authors:
 
 	Dataset: people names dataset in givernment website
 
-## Iteration 1:
+### Iteration 1:
 		Character level language model
 		
 		Method: Bigram (Predict next char using previous char)
@@ -68,7 +68,7 @@ print(f'{nll/n=}')
 
 To avoid infinity probability for some predictions, people do model "smoothing" (assigning very small probability to unlikely scenario)
 
-## Iteration 2: Bigram Language Model using Neural Network
+### Iteration 2: Bigram Language Model using Neural Network
 
 Need to create a dataset for training, i.e input and output char pair. (x and y).
 
@@ -126,5 +126,83 @@ We ended up with the same model , in the  NN based approach the `W` represents t
 
 
 
+## [Building makemore Part 2: MLP - YouTube](https://www.youtube.com/watch?v=TCH_1BHY58I)
 
+In this class we would build makemore to predict based on last 3 characters.
+
+#### Embedding
+As a first step, we need to build embedding for the characters, we start with 2 dimensional embedding.
+
+![[Pasted image 20250205123847.png]]
+
+```python
+h = emb.view(-1, 6) @ W1 + b1 # Hiden layer activation
+```
+
+We index on embedding matrix to get the weight / embeddings for the character. Another way to interpret is one hot encoding. indexing and one hot encoding produce similar result. in this case we think first layer as weight of neural network.
+
+```python
+logits = h @ W2 + b2
+counts = logits.exp()
+prob = counts/counts.sum(1,keepdims=True)
+prob.shape
+# torch.Size([32, 27])
+```
+
+In Final layer we get probability distribution for all 27 characters.
+
+
+```python
+# Negative Log likelihood 
+
+loss = -prob[torch.arange(32), Y].log().mean()
+loss
+```
+
+In Practice, we use mini batch for forward or backward pass. it is efficient than optimizing on the entire dataset.
+
+it is much efficient to take many steps (iteration) with low confidence in gradient
+
+#### Learning rate
+
+Learning rate is an important hyper , we need to find the reasonable range manually and we can use different techniques to search for the optimal parameter in that range.
+
+#### Dataset split
+
+Important to split dataset into three sets
+ - train split is to find model parameters 
+
+- dev split is to find hyper parameters
+
+- test split is to evaluate the model performance finally
+
+we improve the model by increasing the complexity by increasing the parameters. for example hidden layer neurons can be increased.
+
+
+In our case , bottle neck may be the embeddings, we are cramping all the character in just two dimensional space. we can increase embedding dimensions to 10 from 2.
+
+Now we get better name sounding words than before ( with just one character in context)
+
+```
+dex.
+marial.
+mekiophity.
+nevonimitta.
+nolla.
+kyman.
+arreyzyne.
+javer.
+gota.
+mic.
+jenna.
+osie.
+tedo.
+kaley.
+mess.
+suhaiaviyny.
+fobs.
+mhiriel.
+vorreys.
+dasdro.
+```