sc-openai-c2-L8-vid9_1.srt

1
00:00:04,533 --> 00:00:06,266
In the previous few videos,

2
00:00:06,266 --> 00:00:12,300
users shown how to build an application
using an hour from evaluating these inputs

3
00:00:12,600 --> 00:00:15,433
to processing the inputs to then doing

4
00:00:15,433 --> 00:00:18,566
final open checking
and before you show the outputs to a user.

5
00:00:19,200 --> 00:00:20,766
Obviously built such a system.

6
00:00:20,766 --> 00:00:22,500
How do you know how is working?

7
00:00:22,500 --> 00:00:26,200
And maybe even as you deploy it
and let users use it,

8
00:00:26,500 --> 00:00:30,633
how can you track how is doing
and finding these shortcomings

9
00:00:30,633 --> 00:00:34,233
and continue to improve the quality
of the answers in the system?

10
00:00:34,766 --> 00:00:37,600
In this video, I'd like to share with you
some best practices

11
00:00:37,600 --> 00:00:40,700
for evaluating the outputs of an Elm.

12
00:00:41,100 --> 00:00:42,133
And I want to show you

13
00:00:42,133 --> 00:00:45,800
specifically what it feels like
to build one of these systems.

14
00:00:46,200 --> 00:00:49,933
One key distinction between what
you hear me talk about in this video

15
00:00:50,100 --> 00:00:54,166
and what you may have seen in more
traditional machine learning supervised

16
00:00:54,166 --> 00:00:58,133
only applications is because you can build
such an application so quickly.

17
00:00:58,433 --> 00:01:03,200
The methods evaluating it,
it tends not to start off with a test set.

18
00:01:03,233 --> 00:01:08,166
Instead, you often end up gradually
building up a set of test examples.

19
00:01:08,700 --> 00:01:10,766
Let me show you what I mean by that.

20
00:01:10,766 --> 00:01:14,600
You may remember this diagram
from the second video about how problems

21
00:01:14,600 --> 00:01:18,566
development speeds up
the core parts of multiple development

22
00:01:18,566 --> 00:01:22,300
from maybe months
to just minutes or hours.

23
00:01:22,300 --> 00:01:24,366
Like most, a very small number of days

24
00:01:25,433 --> 00:01:27,933
in the
traditional supervised learning approach.

25
00:01:28,333 --> 00:01:32,133
If you needed to collect, say, 10,000
people examples anyway,

26
00:01:32,533 --> 00:01:38,033
then the incremental costs of collecting
another 1000 test examples isn't that bad.

27
00:01:38,066 --> 00:01:41,400
So in the traditional supervised
learning setting,

28
00:01:41,400 --> 00:01:44,600
it was not unusual to click
a training site, collect

29
00:01:44,600 --> 00:01:48,400
a development set or hold our
cross-validation set in a test set,

30
00:01:48,633 --> 00:01:51,900
and then tap those at hand
throughout this development process.

31
00:01:52,533 --> 00:01:55,800
But if you're able to specify
a prompt in just minutes

32
00:01:56,000 --> 00:02:00,133
and get something working in hours,
then it would seem like a huge pain

33
00:02:00,133 --> 00:02:05,100
if you had to pause for a long time
to collect a thousand test examples

34
00:02:05,100 --> 00:02:08,633
because you can now get this working
with zero training examples.

35
00:02:09,366 --> 00:02:12,600
So when building an application
using an element,

36
00:02:12,833 --> 00:02:14,766
this is what it often feels like.

37
00:02:14,766 --> 00:02:18,866
First you would tune the prompts on
just a small handful of examples.

38
00:02:18,866 --> 00:02:21,566
Maybe one, two, 3 to 5 examples

39
00:02:21,900 --> 00:02:24,100
and try to get a prompt
that works on them.

40
00:02:24,866 --> 00:02:28,433
And then as you have the system
undergo additional testing,

41
00:02:28,733 --> 00:02:32,400
you occasionally run into a few examples
that are tricky.

42
00:02:32,400 --> 00:02:34,966
The prompt doesn't work on them
or the outcome doesn't work on them.

43
00:02:35,433 --> 00:02:38,933
And in that case, you can take these
additional one or two or three

44
00:02:38,933 --> 00:02:42,700
or five examples
and add them to the set to your testing on

45
00:02:42,700 --> 00:02:45,733
to just add additional tricky examples
opportunistically.

46
00:02:46,600 --> 00:02:49,500
Eventually,
you have enough of these examples

47
00:02:49,500 --> 00:02:54,833
you've added to your slowly
growing development set that it becomes

48
00:02:54,833 --> 00:02:58,800
a bit inconvenient to manually run
every example through the prompt.

49
00:02:58,800 --> 00:03:02,033
Every time you change the problem
and then you start to develop metrics

50
00:03:02,266 --> 00:03:03,500
to measure performance

51
00:03:03,500 --> 00:03:06,533
on just a small set of examples
such as maybe average accuracy

52
00:03:07,700 --> 00:03:09,966
and one

53
00:03:09,966 --> 00:03:14,700
interesting aspect of this process
is if you decide at any moment in time

54
00:03:14,966 --> 00:03:16,800
your system is working well enough,

55
00:03:16,800 --> 00:03:19,833
you can stop right there
and not go on to the next bullet.

56
00:03:20,400 --> 00:03:25,300
And the fact that many deploy applications
that start at maybe the first or

57
00:03:25,300 --> 00:03:29,066
the second bullet and are running actually
and they're running just fine.

58
00:03:30,266 --> 00:03:33,600
Now, if your hand-built

59
00:03:34,300 --> 00:03:38,433
development set the evaluating
the model long isn't giving you sufficient

60
00:03:38,433 --> 00:03:42,066
confidence yet in the performance
of your system, then that's when you may

61
00:03:42,066 --> 00:03:47,133
go to the next step of of collecting
a randomly sample set of examples.

62
00:03:47,633 --> 00:03:50,400
To tune the model to and this would

63
00:03:50,766 --> 00:03:53,800
continue to be a development set
through a whole lot of cross-validation.

64
00:03:53,800 --> 00:03:55,966
So that could be quite
because to be quite common

65
00:03:55,966 --> 00:03:58,800
to continue to tune your prompt to this

66
00:03:59,733 --> 00:04:03,200
and only if you need even higher fidelity

67
00:04:03,200 --> 00:04:07,133
estimate of the performance of the system,
then might you collect and

68
00:04:07,300 --> 00:04:11,100
use the holdout test sets
that you don't even look at yourself

69
00:04:11,400 --> 00:04:12,800
when you're tuning the model.

70
00:04:14,300 --> 00:04:15,166
And so

71
00:04:15,166 --> 00:04:18,366
step four tends to be more important
if, say,

72
00:04:18,600 --> 00:04:22,800
your system is getting the right answer
91% of the time.

73
00:04:22,800 --> 00:04:26,300
He wants it to that to get it
to give the right answer.

74
00:04:26,333 --> 00:04:28,366
92 or 93% of the time.

75
00:04:28,700 --> 00:04:31,666
Then you do need
the largest end of examples to measure

76
00:04:31,866 --> 00:04:36,566
those differences
between 91 and 93% performance.

77
00:04:36,566 --> 00:04:39,900
And then only if you really need
an unbiased,

78
00:04:40,033 --> 00:04:42,600
fair estimate of
how was the system doing, then

79
00:04:43,133 --> 00:04:47,133
you need to go beyond the development
set to also collect a holdout test set.

80
00:04:47,400 --> 00:04:51,066
One important caveat
I've seen a lot of applications of large

81
00:04:51,066 --> 00:04:55,600
language models
where there isn't meaningful risk of harm

82
00:04:56,333 --> 00:04:58,466
if it gives not quite the right answer,

83
00:04:58,833 --> 00:05:01,600
but obviously for any high stakes
applications,

84
00:05:01,600 --> 00:05:05,733
if there's a risk of bias
or inappropriate outputs

85
00:05:06,000 --> 00:05:09,633
causing harm to someone,
then the responsibility

86
00:05:09,633 --> 00:05:13,366
to collect a test set to rigorously
evaluate the performance of the system

87
00:05:13,566 --> 00:05:16,300
to make sure it's doing the right thing
before you use it.

88
00:05:16,566 --> 00:05:18,833
That becomes much more important.

89
00:05:18,833 --> 00:05:23,333
But if, for example,
if you are using it to summarize articles

90
00:05:23,333 --> 00:05:27,566
just for yourself to read and no one else,
then maybe the risk of harm

91
00:05:27,566 --> 00:05:30,900
is more modest
and you can stop early in this process

92
00:05:30,900 --> 00:05:35,100
without going to the expense of bullets
four and five and collecting larger

93
00:05:35,300 --> 00:05:38,100
datasets
on which to evaluate your algorithm.

94
00:05:38,100 --> 00:05:40,166
So in this example,

95
00:05:41,100 --> 00:05:45,200
let me start with the usual
helper functions.

96
00:05:45,200 --> 00:05:45,600
First,

97
00:05:48,333 --> 00:05:49,166
use a utils

98
00:05:49,166 --> 00:05:52,333
function
to get a list of products and categories.

99
00:05:53,100 --> 00:05:56,633
So in the computers in that top category,

100
00:05:57,033 --> 00:06:00,833
there's a list of computers, laptops
and smartphones and CSC category.

101
00:06:01,300 --> 00:06:03,800
So this is smartphones and systems
and so on.

102
00:06:03,800 --> 00:06:07,500
For other categories.

103
00:06:11,833 --> 00:06:18,000
Now, let's say

104
00:06:18,100 --> 00:06:24,200
this hospital, the actress is given
a user input such as what TV can they buy?

105
00:06:24,200 --> 00:06:26,000
If I'm on the budget

106
00:06:26,100 --> 00:06:28,733
to retrieve the relevant

107
00:06:28,733 --> 00:06:34,433
categories in products so that we have
the right info to answer the user's query.

108
00:06:34,933 --> 00:06:37,100
So here's a prompt for you
to pause the video

109
00:06:37,100 --> 00:06:39,366
and read through this in detail
if you wish.

110
00:06:39,366 --> 00:06:41,866
But the prompt specifies
a set of instructions

111
00:06:42,300 --> 00:06:47,033
and actually gives the language model
one example of a good output.

112
00:06:47,166 --> 00:06:50,766
This is sometimes called a few short
lets actually one side prompting because

113
00:06:50,766 --> 00:06:53,266
we're actually using a user message
in the system.

114
00:06:53,266 --> 00:06:56,733
Message
to give it one example of good output.

115
00:06:56,733 --> 00:06:58,833
This one says
I want the most expensive computer.

116
00:06:59,133 --> 00:06:59,433
You know,

117
00:06:59,433 --> 00:07:03,600
let's just return all the computers
because we don't have pricing information.

118
00:07:03,600 --> 00:07:08,133
Now let's use this front on the hospital
message.

119
00:07:08,366 --> 00:07:12,933
Which TV can I buy if I'm on a budget?

120
00:07:15,233 --> 00:07:17,300
And so we're passing in

121
00:07:17,633 --> 00:07:22,300
to this both the prompt customer,
such as zero as was the Partizan category.

122
00:07:22,300 --> 00:07:26,100
This is a information that we retrieve
a top using the UTILS function.

123
00:07:26,633 --> 00:07:30,633
And here in this cell,
the relevant information to this query,

124
00:07:30,633 --> 00:07:34,066
which is under the category
televisions and home theater systems,

125
00:07:34,066 --> 00:07:36,866
this is on this of TVs and home
theater systems.

126
00:07:36,866 --> 00:07:39,900
That's relevant
to see how well the prompt is doing.

127
00:07:39,900 --> 00:07:43,333
You may be validated on a second prompt.

128
00:07:43,500 --> 00:07:45,800
Of course, that says I mean the challenge
if I'm a smart phone,

129
00:07:47,300 --> 00:07:49,300
it looks like us correctly retrieving

130
00:07:50,700 --> 00:07:52,466
this data.

131
00:07:52,466 --> 00:07:55,666
How do we smartphones, accessories
and this rather than products?

132
00:07:56,066 --> 00:08:00,500
And here's another one.

133
00:08:00,500 --> 00:08:02,766
So what computers do you have?

134
00:08:02,766 --> 00:08:05,533
And hopefully I retrieve
a list of the computers.

135
00:08:06,400 --> 00:08:08,633
So here I have three prompts

136
00:08:08,633 --> 00:08:12,833
and if you are developing this prompt
for the first time,

137
00:08:12,833 --> 00:08:18,033
it would be quite reasonable to to have
one or two or three examples like this.

138
00:08:18,366 --> 00:08:22,500
And to keep on tuning the prompt
until it gives appropriate outputs,

139
00:08:22,866 --> 00:08:26,266
until the prompt is retrieving
the relevant products and categories

140
00:08:26,666 --> 00:08:30,133
to the customer requests
for all of your prompts.

141
00:08:30,200 --> 00:08:34,600
All three of them in this example.

142
00:08:34,600 --> 00:08:38,400
And if the prompt had been missing
some products or something,

143
00:08:38,400 --> 00:08:40,500
then what we would do
is probably go back to edit

144
00:08:40,500 --> 00:08:42,300
the prompt a few times
until it gets it right.

145
00:08:42,300 --> 00:08:46,266
On all three of these prompts.

146
00:08:46,266 --> 00:08:49,200
After
you've gotten the system to this point,

147
00:08:49,900 --> 00:08:53,233
you might then start running the system
in testing,

148
00:08:53,233 --> 00:08:57,300
maybe send it to internal test users
or try using it yourself

149
00:08:57,633 --> 00:08:59,700
and just run it for a while
to see what happens.

150
00:09:00,500 --> 00:09:05,966
And sometimes you will run across a prompt
that it fails on.

151
00:09:05,966 --> 00:09:07,600
So here's an example of a prompt

152
00:09:07,600 --> 00:09:11,100
Tell people to slice pro phone and fill
this out camera Also what TVs you have.

153
00:09:11,966 --> 00:09:14,966
So when I run it on this prompt
it looks like is outputting

154
00:09:15,366 --> 00:09:19,066
the right data,
but it also outputs a bunch of text here.

155
00:09:19,100 --> 00:09:24,700
This extra chunk makes it harder
to parse this into a python.

156
00:09:24,700 --> 00:09:29,100
This dictionaries so we don't like that
is outputting this extra chunk.

157
00:09:29,866 --> 00:09:34,733
So when you run across one example
that the system fails on,

158
00:09:34,733 --> 00:09:38,933
then common practice is to just no doubt
that this is a somewhat tricky example.

159
00:09:39,133 --> 00:09:43,166
So let's add this to our set of examples
that we're going to test the system

160
00:09:43,166 --> 00:09:44,700
on systematically.

161
00:09:44,700 --> 00:09:48,800
And if you keep on running the system
for a while longer, maybe it works.

162
00:09:48,800 --> 00:09:49,666
On those examples.

163
00:09:49,666 --> 00:09:53,366
We tested the prompt to see examples of
maybe we'll work on many examples, but

164
00:09:54,166 --> 00:09:58,133
just by chance you might run across
another example where January's an error.

165
00:09:58,800 --> 00:10:03,900
So this custom message
four also causes the system to output

166
00:10:04,033 --> 00:10:06,900
a bunch of junk text
at the end that we don't

167
00:10:06,900 --> 00:10:10,400
want.

168
00:10:10,400 --> 00:10:12,200
Try to be helpful
to get all this extra text.

169
00:10:12,200 --> 00:10:14,266
We actually don't want this
as So at this point,

170
00:10:14,266 --> 00:10:17,866
you may have run this prompt
maybe on hundreds of examples.

171
00:10:17,866 --> 00:10:21,500
Maybe your test uses,
but you would just take the examples.

172
00:10:21,500 --> 00:10:24,766
The tricky ones is doing poorly on
and now have this

173
00:10:25,266 --> 00:10:28,733
set of five examples
in text from 0 to 4 have.

174
00:10:28,733 --> 00:10:32,533
This are the five examples that you use
to further fine tune the prompts.

175
00:10:34,133 --> 00:10:37,966
And in both of these examples

176
00:10:39,200 --> 00:10:43,633
I'll put a bunch of extra junk
text at the end that we don't want.

177
00:10:44,166 --> 00:10:46,466
And after a little bit of trial and error,

178
00:10:46,866 --> 00:10:49,966
you might decide to modify
the prompts as follows.

179
00:10:51,233 --> 00:10:52,466
So here's a new prompt.

180
00:10:52,466 --> 00:10:54,433
This is called from V two.

181
00:10:54,433 --> 00:10:57,533
But what we did here was
we added to the prompt.

182
00:10:57,600 --> 00:11:01,366
Did not output any additional
text is not in Jason format just emphasize

183
00:11:01,366 --> 00:11:07,133
piece of output this Jason stuff
and add in a second example using the user

184
00:11:07,133 --> 00:11:11,800
and assistant message for future property
where the user also cheapest computer

185
00:11:12,333 --> 00:11:16,666
and in both of the few shot examples,
we're demonstrating to the system

186
00:11:16,933 --> 00:11:20,766
a response
where it gives only Jason outputs.

187
00:11:20,900 --> 00:11:23,700
So here's the extra thing that we just
added to the problem to the output.

188
00:11:23,700 --> 00:11:27,133
Any additional text
is not in Jason formats and we use future

189
00:11:27,200 --> 00:11:30,966
use the one future system
one and Fichajes to future assistant to

190
00:11:31,300 --> 00:11:33,866
to give it two of these few shot

191
00:11:34,300 --> 00:11:38,533
prompts
suddenly shift into to find that prompt

192
00:11:38,966 --> 00:11:43,200
and you have to go back and manually rerun
this prompt on all five of the examples

193
00:11:43,200 --> 00:11:47,100
of user inputs, including this one
that previously had given a broken output.

194
00:11:47,400 --> 00:11:50,133
You find that it now
gives a correct output.

195
00:11:51,000 --> 00:11:53,833
And if you were to go back and rerun
this new problem,

196
00:11:53,866 --> 00:11:58,766
this is from version
V two on that customer message example

197
00:11:59,100 --> 00:12:03,433
that results in the broken output
with extra junk after the JSON output,

198
00:12:03,933 --> 00:12:07,966
then this will generate a better output

199
00:12:08,633 --> 00:12:12,166
and I'm not going to do it here,
but I encourage you to pause the video

200
00:12:12,166 --> 00:12:16,000
and rerun it yourself on Custom Message
four as well on this prompt V

201
00:12:16,000 --> 00:12:20,966
to see if it also generates
the correct output and hopefully it will.

202
00:12:20,966 --> 00:12:24,333
I think it should.

203
00:12:24,333 --> 00:12:28,600
And of course when you modify
the prompts, it's also useful to do

204
00:12:29,433 --> 00:12:32,133
a bit of regression
testing to make sure that

205
00:12:32,866 --> 00:12:36,300
when fixing the incorrect outputs
on prompts three and four,

206
00:12:36,566 --> 00:12:40,000
it doesn't break
the outputs on from zero either.

207
00:12:41,266 --> 00:12:43,500
Now you can kind of tell that

208
00:12:43,500 --> 00:12:47,366
if I had to copy paste
five prompts customer such as zero

209
00:12:47,533 --> 00:12:50,866
one, two, three and four into my
for the notebook and run them

210
00:12:50,866 --> 00:12:52,166
and then manually look at them

211
00:12:52,166 --> 00:12:55,133
to see if the output in the right
categories and products.

212
00:12:55,400 --> 00:12:56,433
So you can kind of do it.

213
00:12:56,433 --> 00:13:00,200
I can look at this and go, Yep,
carry TV and holidays is those products.

214
00:13:00,200 --> 00:13:01,900
Yep, they'll see all of them.

215
00:13:01,900 --> 00:13:04,500
But there's actually a little bit painful
to do this manually,

216
00:13:04,500 --> 00:13:07,066
to manually inspect
or to look at this output

217
00:13:07,566 --> 00:13:11,066
to make sure with your eyes
that this is exactly the right output.

218
00:13:12,000 --> 00:13:14,600
So when the development sets,

219
00:13:14,700 --> 00:13:19,933
the tuning to becomes more
than just a small handful of examples, it

220
00:13:21,200 --> 00:13:22,866
then becomes useful

221
00:13:22,866 --> 00:13:25,633
to start to automate the testing process.

222
00:13:27,000 --> 00:13:31,466
So here is a set of ten examples

223
00:13:31,933 --> 00:13:35,500
where I'm specifying
ten customer messages.

224
00:13:35,500 --> 00:13:38,533
So here's the customer message
What TV for the biopharma budget

225
00:13:39,033 --> 00:13:40,900
as well as what's the ideal.

226
00:13:40,900 --> 00:13:45,200
So think of this
as the right answer in the test set

227
00:13:45,433 --> 00:13:48,733
or really I should say development set
because we're actually tuning to this.

228
00:13:49,000 --> 00:13:51,900
And so we've collected here ten examples

229
00:13:51,900 --> 00:13:54,600
in text from zero through nine

230
00:13:55,600 --> 00:13:57,833
where the last one is.

231
00:13:57,833 --> 00:13:58,800
If the user says,

232
00:13:58,800 --> 00:14:02,566
I would like to time as machine,
we have no relevant products to that.

233
00:14:02,566 --> 00:14:03,333
Really Sorry.

234
00:14:03,333 --> 00:14:05,466
So the idea answer is the empty set.

235
00:14:06,733 --> 00:14:10,800
And now if you want to evaluate

236
00:14:11,966 --> 00:14:14,933
automatically what the prompt is doing on

237
00:14:14,933 --> 00:14:19,400
any of these ten examples,
here is a function to do.

238
00:14:19,400 --> 00:14:23,100
So this kind of a lung function
for the pauses video and read through it

239
00:14:23,100 --> 00:14:26,933
if you wish, but let me just demonstrate
what it is actually doing.

240
00:14:27,866 --> 00:14:32,300
So let me print out the custom message
for custom messages zero.

241
00:14:33,000 --> 00:14:33,266
Right.

242
00:14:33,266 --> 00:14:35,066
So of course, customer messages.

243
00:14:35,066 --> 00:14:37,700
Which TV can I buy if I'm on a budget?

244
00:14:37,700 --> 00:14:41,700
And this also prints out the idea?

245
00:14:41,700 --> 00:14:47,000
Answer So the idea answer is all the TVs
that we want to prompt to retrieve

246
00:14:49,166 --> 00:14:51,600
and let me now call the prompt.

247
00:14:51,666 --> 00:14:54,600
This is probably to
on this customer message

248
00:14:54,833 --> 00:14:57,433
with that usual practice and calorie
information that's printed out.

249
00:14:57,800 --> 00:15:00,600
And then we'll call to give our

250
00:15:01,500 --> 00:15:04,600
we'll call
to give our responsive ideal option

251
00:15:05,233 --> 00:15:08,700
to see how well
the response matches the ideal answer.

252
00:15:09,166 --> 00:15:12,900
And in this case,
it did outputs the category that we wanted

253
00:15:12,900 --> 00:15:15,900
and it did outputs
the entire list of products.

254
00:15:16,200 --> 00:15:21,500
And so this gives it a score of 1.0.

255
00:15:21,500 --> 00:15:25,733
Just to show you one more example,
it turns out that I know it gets it

256
00:15:25,733 --> 00:15:29,733
wrong on the examples seven
So I change this from 0

257
00:15:29,733 --> 00:15:35,533
to 7 and run it.

258
00:15:35,533 --> 00:15:37,066
This is what it gets.

259
00:15:37,066 --> 00:15:39,333
And then we update this to

260
00:15:41,200 --> 00:15:42,033
seven as well.

261
00:15:42,033 --> 00:15:44,500
So on to this customer message.

262
00:15:45,033 --> 00:15:50,400
This is the idea as sir, we should all put
on the gaming consoles ancestry.

263
00:15:50,400 --> 00:15:52,433
So this is gaming consoles and Cicero's,

264
00:15:52,433 --> 00:15:55,633
but where's the response
here has the outputs.

265
00:15:56,400 --> 00:15:59,900
It actually should have had
one, two, three, four, five

266
00:16:01,033 --> 00:16:01,566
outputs.

267
00:16:01,566 --> 00:16:03,666
And so it's missing some of the products.

268
00:16:04,266 --> 00:16:06,766
So what I would do if I tuning the prompt

269
00:16:06,766 --> 00:16:10,200
now is I would then

270
00:16:10,200 --> 00:16:12,500
use a follow up to loop over

271
00:16:12,500 --> 00:16:15,933
all ten of the development set examples

272
00:16:16,400 --> 00:16:20,166
where we repeatedly put out the custom
message, get the idea.

273
00:16:20,633 --> 00:16:25,300
And so the right answer,
call the arm to get a response evaluated

274
00:16:25,566 --> 00:16:28,766
and then the accumulator and average
and then just run this

275
00:16:30,566 --> 00:16:32,033
play, right?

276
00:16:32,033 --> 00:16:34,633
So this would take a while to run,
but when it's done running,

277
00:16:34,633 --> 00:16:37,033
this is the results we're running through.

278
00:16:37,033 --> 00:16:41,033
The ten examples
it looks like example seven was wrong.

279
00:16:41,033 --> 00:16:44,433
And so the fashion
recall of ten was 90% correct.

280
00:16:45,866 --> 00:16:48,700
And so if you were to tune the prompts,

281
00:16:48,700 --> 00:16:53,000
you can rerun this to see
if the percent correct goes up or down.

282
00:16:53,433 --> 00:16:56,833
What you just saw in
this notebook is going

283
00:16:56,833 --> 00:17:00,433
through the steps one,
two and three of this bullet in this.

284
00:17:00,433 --> 00:17:04,933
And this already gives a pretty good
development sets of ten examples

285
00:17:04,933 --> 00:17:08,233
with which to tune and validate
the prompts is working.

286
00:17:08,800 --> 00:17:13,166
If you needed an additional level trigger,
then you now have the software

287
00:17:13,166 --> 00:17:17,766
needed to collect a random
sample sets of maybe 100 examples

288
00:17:17,766 --> 00:17:22,333
with their ideal outputs and maybe
even go beyond that to the rigor of hold

289
00:17:22,333 --> 00:17:25,266
our test set the you know, even
look at why you two need to prompt

290
00:17:25,600 --> 00:17:29,566
but for a lot of applications
stopping at bullet three.

291
00:17:29,566 --> 00:17:33,900
But there are also certainly applications
where you could do what you just saw me

292
00:17:33,900 --> 00:17:38,066
do in this with the notebook
and it gets a pretty performance system

293
00:17:38,066 --> 00:17:41,633
quite quickly
with again, the important caveat that

294
00:17:43,066 --> 00:17:45,000
if you're working on a safety

295
00:17:45,000 --> 00:17:47,600
critical application
or an application with this

296
00:17:48,833 --> 00:17:51,766
where there's non-trivial risk of harm,
then of course

297
00:17:51,766 --> 00:17:55,433
it would be the responsible thing
to do to actually get a much larger test

298
00:17:55,433 --> 00:17:58,700
set to really verify the performance
before you use it anywhere.

299
00:17:59,100 --> 00:18:02,033
And so that's it's
I find that the workflow

300
00:18:02,033 --> 00:18:05,866
of building applications using probes
is very different than the workflow

301
00:18:06,266 --> 00:18:09,166
of building applications
using supervised learning.

302
00:18:09,566 --> 00:18:12,566
And the pace of iteration
feels much faster.

303
00:18:12,566 --> 00:18:17,200
And if you have not yet done it before,
you might be surprised at how well

304
00:18:17,400 --> 00:18:22,233
and evaluate a method built
on just a few Hand-Curated tricky examples

305
00:18:22,233 --> 00:18:26,133
you think with ten examples
and this is not statistically valid

306
00:18:26,133 --> 00:18:29,833
for almost anything, but you might be
surprised we actually use this procedure.

307
00:18:30,100 --> 00:18:31,500
How effective?

308
00:18:31,500 --> 00:18:35,733
Adding a handful, just a handful
of tricky examples into development sets

309
00:18:36,133 --> 00:18:39,800
might be in terms of helping
you and your team get to an effective.

310
00:18:40,033 --> 00:18:41,633
So the prompt and effective system

311
00:18:43,300 --> 00:18:45,066
in this video,

312
00:18:45,066 --> 00:18:49,600
the outputs could be evaluated
quantitatively as in

313
00:18:49,866 --> 00:18:54,933
those desired outputs, and you could tell
if it gave the desired output or not.

314
00:18:55,300 --> 00:18:59,666
So the next video, let's take a look
how you can evaluate outputs

315
00:18:59,933 --> 00:19:03,700
in that setting where what is the right
answer is a bit more ambiguous.