Skip to content

Commit 5e85760

Browse files
jeffxtangjeffxtang
authored
[lamini] Add lamini text2sql memory tuning tutorial (meta-llama#573)
2 parents e6b0f97 + 6c26150 commit 5e85760

23 files changed

+6091
-0
lines changed

‎.github/scripts/spellcheck_conf/wordlist.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1387,3 +1387,7 @@ LlamaChat
13871387
chatbot's
13881388
ConversationBufferWindowMemory
13891389
chatbot's
1390+
Lamini
1391+
lamini
1392+
nba
1393+
sqlite

‎recipes/3p_integrations/lamini/text2sql_memory_tuning/README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Tune Llama 3 for text-to-SQL and improve accuracy from 30% to 95%
2+
3+
This repo and notebook `meta-lamini.ipynb` demonstrate how to tune Llama 3 to generate valid SQL queries and improve accuracy from 30% to 95%.
4+
5+
In this notebook we'll be using Lamini, and more specifically, Lamini Memory Tuning.
6+
7+
Lamini is an integrated platform for LLM inference and tuning for the enterprise. Lamini Memory Tuning is a new tool you can use to embed facts into LLMs that improves factual accuracy and reduces hallucinations. Inspired by information retrieval, this method has set a new standard of accuracy for LLMs with less developer effort.
8+
9+
Learn more about Lamini Memory Tuning: https://www.lamini.ai/blog/lamini-memory-tuning
10+
11+
Please head over to https://app.lamini.ai/account to get your free api key.
12+
13+
You can authenticate by writing the following to a file `~/.lamini/configure.yaml`
14+
15+
```
16+
production:
17+
key: <YOUR-LAMINI-API-KEY>
18+
```
19+
20+
This tuning tutorial uses the `nba_roster` sqlite database to tune a Llama 3 model.
21+
22+
## Additional resources
23+
24+
▫️ Fortune 500 case study: http://www.lamini.ai/blog/llm-text-to-sql <br>
25+
▫️ Technical paper: https://github.com/lamini-ai/Lamini-Memory-Tuning/blob/main/research-paper.pdf <br>
26+
▫️ Model weights: https://huggingface.co/engineering-lamini/lamini-1-random

‎recipes/3p_integrations/lamini/text2sql_memory_tuning/assets/manual_filtering.png

1.44 MB
Loading

‎recipes/3p_integrations/lamini/text2sql_memory_tuning/assets/website.png

341 KB
Loading

‎recipes/3p_integrations/lamini/text2sql_memory_tuning/data/gold-test-set-v2.jsonl

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{"question": "Who is the pointguard for the Golden State Warriors?", "answer": "Stephen Curry, Chris Paul, and Cory Joseph", "sql": "select name from nba_roster where team='Golden State Warriors' and POS='PG';"}
2+
{"question": "What is the number of players on the Chicago Bulls who are 25 years old or younger", "answer": "10", "sql": "SELECT COUNT(*) FROM nba_roster WHERE team='Chicago Bulls' AND AGE <= 25;"}
3+
{"question": "Who is the highest-paid player on the Los Angeles Lakers", "answer": "LeBron James", "sql": "SELECT NAME, SALARY FROM nba_roster WHERE team='Los Angeles Lakers' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
4+
{"question": "Who is the highest paid player in the NBA?", "answer": "Stephen Curry", "sql": "SELECT NAME, salary FROM nba_roster WHERE SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
5+
{"question": "What team is LaMelo Ball on?", "answer": "Charlotte Hornets", "sql": "select team from nba_roster where name='LaMelo Ball';"}
6+
{"question": "How much does Lonzo Ball weigh?", "answer": "190 lbs", "sql": "select wt from nba_roster where name='Lonzo Ball';"}
7+
{"question": "What college sent the most players to the current NBA?", "answer": "Kentucky", "sql": "select college from nba_roster where college != '--' group by college order by count(*) desc limit 1;"}
8+
{"question": "How old is Lebron James?", "answer": "38", "sql": "select age from nba_roster where name='LeBron James';"}
9+
{"question": "What is the most popular jersey number in the current NBA?", "answer": "8", "sql": "select Jersey from nba_roster where Jersey != 'NA' group by Jersey order by count(*) desc limit 1;"}
10+
{"question": "Can you give me a list of all the players without college data?", "answer": "['Bogdan Bogdanovic', 'Clint Capela', 'Kristaps Porzingis', 'Darius Bazley', 'LaMelo Ball', 'Theo Maledon', 'James Nnaji', 'Frank Ntilikina', 'Marko Simonovic', 'Raul Neto', 'Ricky Rubio', 'Luka Doncic', 'Dante Exum', 'Jaden Hardy', 'Maxi Kleber', 'Vlatko Cancar', 'Nikola Jokic', 'Bojan Bogdanovic', 'Malcolm Cazalon', 'Killian Hayes', 'Ausar Thompson', 'Jonathan Kuminga', 'Dario Saric', 'Jalen Green', 'Boban Marjanovic', 'Alperen Sengun', 'Amen Thompson', 'Serge Ibaka', 'Daniel Theis', 'Nicolas Batum', 'KJ Martin', 'Kenyon Martin Jr.', 'Ivica Zubac', 'LeBron James', 'Vincent Valerio-Bodon', 'Tarik Biberovic', 'John Konchar', 'Isaiah Todd', 'Nikola Jovic', 'Giannis Antetokounmpo', 'Thanasis Antetokounmpo', 'MarJon Beauchamp', 'Goran Dragic', 'Rudy Gobert', 'Vit Krejci', 'Daishen Nix', 'Dyson Daniels', 'Willy Hernangomez', 'Jonas Valanciunas', 'Evan Fournier', 'Isaiah Hartenstein', 'Jaylen Martin', 'Mitchell Robinson', 'Davis Bertans', 'Ousmane Dieng', 'Josh Giddey', 'Vasilije Micic', 'Aleksej Pokusevski', 'Goga Bitadze', 'Joe Ingles', 'Furkan Korkmaz', 'Bismack Biyombo', 'Ibou Badji', 'Scoot Henderson', 'Jusuf Nurkic', 'Anfernee Simons', 'Sasha Vezenkov', 'Dominick Barlow', 'Sidy Cissoko', 'Cedi Osman', 'Victor Wembanyama', 'Dennis Schroder', 'Simone Fontecchio', 'Luka Samanic', 'Dennis Schroder', 'Deni Avdija', 'Bilal Coulibaly', 'Danilo Gallinari', 'Tristan Vukcevic']", "sql": "SELECT name FROM nba_roster WHERE COLLEGE IS NULL OR COLLEGE = '--';"}
11+
{"question": "What team has the smallest roster?", "answer": "Brooklyn Nets", "sql": "select team from nba_roster group by team order by count(*) asc limit 1;"}
12+
{"question": "What team has the largest roster?", "answer": "Toronto Raptors", "sql": "select team, count(*) from nba_roster group by team order by count(*) desc limit 1;"}
13+
{"question": "What team is paying its players the most in total?", "answer": "Toronto Raptors", "sql": "select team, sum(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) from nba_roster group by team order by sum(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) desc limit 1;"}
14+
{"question": "Which team is paying its players the least?", "answer": "San Antonio Spurs", "sql": "select team from nba_roster group by team order by sum(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) asc limit 1;"}
15+
{"question": "Which team is on average the tallest?","answer":"Boston Celtics", "sql": "select team, AVG(CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12) as height from nba_roster group by team order by height desc limit 1;"}
16+
{"question": "Which team is on average the shortest?", "answer": "Golden State Warriors", "sql": "select team, AVG(CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12) as height from nba_roster group by team order by height asc limit 1;"}
17+
{"question": "Who are the tallest 5 centers in the league?", "answer": "Boban Marjanovic, Kristaps Porzingis, Victor Wembanyama, Luke Kornet, Bol Bol", "sql": "SELECT name, HT FROM nba_roster WHERE POS = 'C' ORDER BY HT DESC LIMIT 5;"}
18+
{"question": "Who are the top 5 highest paid power forwards in the league?", "answer": "Kevin Durant, Giannis Antetokounmpo, Anthony Davis, Tobias Harris, Pascal Siakam", "sql": "SELECT NAME, salary FROM nba_roster WHERE POS = 'PF' AND SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 5;"}
19+
{"question": "What is the median salary in the NBA?", "answer": "6012840", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*50/100-1;"}
20+
{"question": "What is the average salary in the NBA?", "answer": "10696803", "sql": "SELECT avg(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--';"}
21+
{"question": "What is the 99th percentile salary in the NBA?", "answer": "46741590", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*99/100-1;"}
22+
{"question": "What is the 75th percentile salary in the NBA?", "answer": "13932008", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*75/100-1;"}
23+
{"question": "What is the 25th percentile salary in the NBA?", "answer": "2413304", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*25/100-1;"}
24+
{"question": "What is the median weight in the NBA?", "answer": "215", "sql": "select CAST(SUBSTR(WT, 1, INSTR(WT,' ')) as INTEGER) as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)*50/100-1;"}
25+
{"question": "What is the average weight in the NBA?", "answer": "214.98", "sql": "SELECT AVG(CAST(SUBSTR(WT, 1, INSTR(WT,' ')) as INTEGER)) FROM nba_roster;"}
26+
{"question": "What is the median height in the NBA?", "answer": "6.58333333333333", "sql": "select CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12 as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)*50/100-1;"}
27+
{"question": "What is the average height in the NBA?", "answer": "6.54986111111111", "sql": "select AVG(CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12) as height from nba_roster;"}
28+
{"question": "Can you tell me how many players are in the NBA?", "answer": "600", "sql": "select count(*) from nba_roster;"}
29+
{"question": "Would you please let me know what the highest paid players are for each position?", "answer": "The highest paid players are Nikola Jokic (C), Paul George (F), Norman Powell (G), Kevin Durant (PF), Stephen Curry (PG), LeBron James (SF), Bradley Beal (SG).", "sql": "SELECT name, pos, MAX(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as max_salary FROM nba_roster WHERE SALARY!= '--' GROUP BY POS;"}
30+
{"question": "Is Jalen Johnson 23 years old?", "answer": "No, Jalen Johnson is 21 years old", "sql" : "Select name, age from nba_roster where name='Jalen Johnson';"}
31+
{"question": "Who is the oldest player on the Brooklyn Nets?", "answer": "Spencer Dinwiddie, Dorian Finney-Smith, Royce O'Neale", "sql" : "SELECT NAME FROM nba_roster WHERE TEAM = 'Brooklyn Nets' AND AGE = (SELECT MAX(AGE) FROM nba_roster WHERE TEAM = 'Brooklyn Nets');"}
32+
{"question": "Who has the higest salary on the Memphis Grizzlies?", "answer": "Ja Morant", "sql" : "select salary, name from nba_roster where team='Memphis Grizzlies' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
33+
{"question": "Which player has the higest salary on the Cleveland Cavaliers?", "answer": "Darius Garland", "sql" : "select salary, name from nba_roster where team='Cleveland Cavaliers' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
34+
{"question": "Who is the highest paid center on the Dallas Mavericks?", "answer": "Dereck Lively II", "sql" : "select salary, name from nba_roster where team='Dallas Mavericks' and POS='C' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
35+
{"question": "How much is Marcus Smart getting paid?", "answer": "$18,833,712", "sql" : "select salary from nba_roster where name='Marcus Smart';"}
36+
{"question": "What's the average age of the Trail Blazers?", "answer": "24", "sql" : "select avg(age) from nba_roster where team='Portland Trail Blazers';"}
37+
{"question": "What's the median age of the NBA?", "answer": "25", "sql": "select CAST(AGE as INTEGER) as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)*50/100-1;"}
38+
{"question": "What's the median age of the Miami Heat?", "answer": "26", "sql": "select CAST(AGE as INTEGER) as percentile from nba_roster where team='Miami Heat' order by percentile limit 1 offset (select count(*) from nba_roster where team='Miami Heat')*50/100-1;"}
39+
{"question": "What are the 5 teams with the oldest average age in the NBA", "answer": "Golden State Warriors, Milwaukee Bucks, Miami Heat, LA Clippers, Phoenix Suns", "sql": "SELECT team, AVG(AGE) AS average_age FROM nba_roster GROUP BY team ORDER BY average_age DESC LIMIT 5;"}
40+
{"question": "What is the average salary of Power Forward players in the NBA", "answer": "$10948045", "sql": "select avg(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as average_salary from nba_roster where POS = 'PF';"}

‎recipes/3p_integrations/lamini/text2sql_memory_tuning/data/gold-test-set.jsonl

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{"question": "What is the 99th percentile salary in the NBA?", "answer": "46741590", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*99/100-1;"}
2+
{"question": "What is the 75th percentile salary in the NBA?", "answer": "13932008", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*75/100-1;"}
3+
{"question": "What is the 25th percentile salary in the NBA?", "answer": "2413304", "sql": "SELECT (CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as percentile FROM nba_roster WHERE SALARY!= '--' order by percentile limit 1 offset (select count(*) from nba_roster where SALARY != '--')*25/100-1;"}
4+
{"question": "What is the median weight in the NBA?", "answer": "215", "sql": "select CAST(SUBSTR(WT, 1, INSTR(WT,' ')) as INTEGER) as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)/2;"}
5+
{"question": "What is the average weight in the NBA?", "answer": "214.98", "sql": "SELECT AVG(CAST(SUBSTR(WT, 1, INSTR(WT,' ')) as INTEGER)) FROM nba_roster;"}
6+
{"question": "What is the median height in the NBA?", "answer": "6.58333333333333", "sql": "select CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12 as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)/2;"}
7+
{"question": "What is the average height in the NBA?", "answer": "6.54986111111111", "sql": "select AVG(CAST(SUBSTR(HT, 1, INSTR(HT,' ')-1) AS INTEGER)+ CAST(SUBSTR(HT, INSTR(HT,' ')+1) AS FLOAT)/12) as height from nba_roster;"}
8+
{"question": "Can you tell me how many players are in the NBA?", "answer": "600", "sql": "select count(*) from nba_roster;"}
9+
{"question": "Would you please let me know what the highest paid players are for each position?", "answer": "The highest paid players are Nikola Jokic (C), Paul George (F), Norman Powell (G), Kevin Durant (PF), Stephen Curry (PG), LeBron James (SF), Bradley Beal (SG).", "sql": "SELECT name, pos, MAX(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as max_salary FROM nba_roster WHERE SALARY!= '--' GROUP BY POS;"}
10+
{"question": "Is Jalen Johnson 23 years old?", "answer": "No, Jalen Johnson is 21 years old", "sql" : "Select name, age from nba_roster where name='Jalen Johnson';"}
11+
{"question": "Who is the oldest player on the Brooklyn Nets?", "answer": "Spencer Dinwiddie, Dorian Finney-Smith, Royce O'Neale", "sql" : "SELECT NAME FROM nba_roster WHERE TEAM = 'Brooklyn Nets' AND AGE = (SELECT MAX(AGE) FROM nba_roster WHERE TEAM = 'Brooklyn Nets');"}
12+
{"question": "Who has the higest salary on the Memphis Grizzlies?", "answer": "Ja Morant", "sql" : "select salary, name from nba_roster where team='Memphis Grizzlies' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
13+
{"question": "Which player has the higest salary on the Cleveland Cavaliers?", "answer": "Darius Garland", "sql" : "select salary, name from nba_roster where team='Cleveland Cavaliers' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
14+
{"question": "Who is the highest paid center on the Dallas Mavericks?", "answer": "Dereck Lively II", "sql" : "select salary, name from nba_roster where team='Dallas Mavericks' and POS='C' and SALARY!= '--' ORDER BY CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER) DESC LIMIT 1;"}
15+
{"question": "How much is Marcus Smart getting paid?", "answer": "$18,833,712", "sql" : "select salary from nba_roster where name='Marcus Smart';"}
16+
{"question": "What's the average age of the Trail Blazers?", "answer": "24", "sql" : "select avg(age) from nba_roster where team='Portland Trail Blazers';"}
17+
{"question": "What's the median age of the NBA?", "answer": "25", "sql" : "select CAST(AGE as INTEGER) as percentile from nba_roster order by percentile limit 1 offset (select count(*) from nba_roster)/2;"}
18+
{"question": "What's the median age of the Miami Heat?", "answer": "26", "sql" : "select CAST(AGE as INTEGER) as percentile from nba_roster where team='Miami Heat' order by percentile limit 1 offset (select count(*) from nba_roster where team='Miami Heat')/2;"}
19+
{"question": "What are the 5 teams with the oldest average age in the NBA", "answer": "Golden State Warriors, Milwaukee Bucks, Miami Heat, LA Clippers, Phoenix Suns", "sql": "SELECT team, AVG(AGE) AS average_age FROM nba_roster GROUP BY team ORDER BY average_age DESC LIMIT 5;"}
20+
{"question": "What is the average salary of Power Forward players in the NBA", "answer": "$10948045", "sql": "select avg(CAST(REPLACE(REPLACE(SALARY, '$', ''), ',','') AS INTEGER)) as average_salary from nba_roster where POS = 'PF';"}

0 commit comments

Comments
 (0)