checking in minor edits

Author: jeremymanning

paper/changes.pdf

@@ -1,7 +1,7 @@
 \documentclass[10pt]{article}
 %DIF LATEXDIFF DIFFERENCE FILE
 %DIF DEL old.tex    Mon Feb 19 07:49:49 2024
-%DIF ADD main.tex   Wed Feb 21 19:20:24 2024
+%DIF ADD main.tex   Wed Feb 21 21:04:59 2024
 %DIF 2a2
 \usepackage{amsmath} %DIF > 
 %DIF -------
@@ -776,9 +776,9 @@ \section*{Results}
 knowledge at the embedding coordinate of a }\DIFdelend \DIFaddbegin \DIFadd{question was answered correctly than when it was answered
 incorrectly. Because our knowledge estimates are computed as a weighted version
 of this same proportion-correct score (where each held-in question's weight
-reflects its embedding-space distance from the }\DIFaddend held-out \DIFdelbegin \DIFdel{question about the }\textit{\DIFdel{other}} %DIFAUXCMD
-\DIFdel{lecture (``Across-lecture''; predicting knowledge
-for }\DIFdelend \DIFaddbegin \DIFadd{question; see
+reflects its embedding-space distance from the }\DIFaddend held-out \DIFdelbegin \DIFdel{question about the
+}\textit{\DIFdel{other}} %DIFAUXCMD
+\DIFdel{lecture (``Across-lecture''; predicting knowledge for }\DIFdelend \DIFaddbegin \DIFadd{question; see
 Eqn.~\ref{eqn:prop}), if these weights are uninformative (e.g., randomly
 distributed), then we should expect to see this same inverse relationship
 between estimated knowledge and performance, on average. On the other hand, if
@@ -797,23 +797,23 @@ \section*{Results}
 \DIFadd{Before presenting our results, it is worth considering three possible
 explanations of why a participant might answer a given question correctly or
 incorrectly. One possibility is that the participant simply }\textit{\DIFadd{guessed}}
-\DIFadd{the answer. A second is that they selected an answer by mistake, despite
-``knowing'' the correct answer. In both of these scenarios, the participant's
-knowledge about the question's content should be uninformative about their
-observed response. A third possibility is that the participant's response
-reflects their }\textit{\DIFadd{actual}} \DIFadd{knowledge about the question's content. In this
-case, we }\textit{\DIFadd{might}} \DIFadd{expect to see a positive relationship between the
-participant's knowledge and their likelihood of answering the question
-correctly. However, in order to see this positive relationship, the
-participant's knowledge must be structured in a way that is reflected (at least
-partially) by the embedding space. In other words, if the participant's
-performance reflects their true knowledge, but our text embedding space does
-not sufficiently capture the structure of that knowledge, then the }\DIFaddend knowledge
-\DIFaddbegin \DIFadd{estimates we generate will not be predictive of the participant's performance.
-In the extreme, if the embedding space is completely unstructured with respect
-to the content of the quiz questions, then we would expect to see the negative
-relationship between estimated knowledge and performance that we described
-above.
+\DIFadd{the answer. A second is that they selected the incorrect answer by mistake,
+despite ``knowing'' the correct answer (or vice versa). In both of these
+scenarios, the participant's knowledge about the question's content should be
+uninformative about their observed response. A third possibility is that the
+participant's response reflects their }\textit{\DIFadd{actual}} \DIFadd{knowledge about the
+question's content. In this case, we }\textit{\DIFadd{might}} \DIFadd{expect to see a positive
+relationship between the participant's knowledge and their likelihood of
+answering the question correctly. However, in order to see this positive
+relationship, the participant's knowledge must be structured in a way that is
+reflected (at least partially) by the embedding space. In other words, if the
+participant's performance reflects their true knowledge, but our text embedding
+space does not sufficiently capture the structure of that knowledge, then the
+}\DIFaddend knowledge \DIFaddbegin \DIFadd{estimates we generate will not be predictive of the participant's
+performance. In the extreme, if the embedding space is completely unstructured
+with respect to the content of the quiz questions, then we would expect to see
+the negative relationship between estimated knowledge and performance that we
+described above.
 }
 
 \DIFadd{When we fit a GLMM to estimates of participants' knowledge for each Quiz~1
@@ -973,7 +973,7 @@ \section*{Results}
 95\%\ \textnormal{CI} = [3.033, 3.866],\ p = 0.094$). These ``prediction
 failures'' appear to come from the fact that any signal derived from
 participants' knowledge about the content of the }\textit{\DIFadd{Birth of Stars}}
-\DIFadd{lecture (prior to watching it) is swamped by the much more dramatic increase in
+\DIFadd{lecture (prior to watching it) is overwhelmed by the much more dramatic increase in
 their knowledge about the content of the }\textit{\DIFadd{Four Fundamental Forces}}
 \DIFadd{(which they watched just prior to taking Quiz~2). This is reflected in their
 Quiz~2 performance for questions about each lecture (mean proportion correct
@@ -988,15 +988,11 @@ \section*{Results}
 p = 0.017$) using responses to questions about the other lecture's content.
 Across all three versions of these analyses, our results suggest that (by and
 large) our knowledge estimates can reliably predict participants' abilities to
-answer individual quiz questions, }\DIFaddend distinguish between questions about \DIFdelbegin \DIFdel{more subtly different contentwithin the same lecture}\DIFdelend \DIFaddbegin \DIFadd{similar
+answer individual quiz questions, }\DIFaddend distinguish between questions about \DIFdelbegin \DIFdel{more subtly different contentwithin the
+same lecture}\DIFdelend \DIFaddbegin \DIFadd{similar
 content, and generalize across content areas, provided that participants' quiz
 responses reflect a minimum level of ``real'' knowledge about both content on
-which these predictions are based and that for which they are made. Our results
-also indicate some important limitations of our approach: if participants' quiz
-performance does not reflect what they know (e.g., when they ``guess''), or if
-their knowledge is not structured in a way that is reflected by the embedding
-space, then our knowledge estimates will not be predictive of their
-performance}\DIFaddend .
+which these predictions are based and that for which they are made}\DIFaddend .
 
 %DIF >  our approach works when participants have a minimal baseline level of knowledge about content predicted and used to predict
 %DIF >  our approach generalizes when knowledge of content used to predict can be assumed to be a reasonable indicator of knowledge of content predicted

paper/changes.tex

@@ -634,23 +634,23 @@ \section*{Results}
 Before presenting our results, it is worth considering three possible
 explanations of why a participant might answer a given question correctly or
 incorrectly. One possibility is that the participant simply \textit{guessed}
-the answer. A second is that they selected an answer by mistake, despite
-``knowing'' the correct answer. In both of these scenarios, the participant's
-knowledge about the question's content should be uninformative about their
-observed response. A third possibility is that the participant's response
-reflects their \textit{actual} knowledge about the question's content. In this
-case, we \textit{might} expect to see a positive relationship between the
-participant's knowledge and their likelihood of answering the question
-correctly. However, in order to see this positive relationship, the
-participant's knowledge must be structured in a way that is reflected (at least
-partially) by the embedding space. In other words, if the participant's
-performance reflects their true knowledge, but our text embedding space does
-not sufficiently capture the structure of that knowledge, then the knowledge
-estimates we generate will not be predictive of the participant's performance.
-In the extreme, if the embedding space is completely unstructured with respect
-to the content of the quiz questions, then we would expect to see the negative
-relationship between estimated knowledge and performance that we described
-above.
+the answer. A second is that they selected the incorrect answer by mistake,
+despite ``knowing'' the correct answer (or vice versa). In both of these
+scenarios, the participant's knowledge about the question's content should be
+uninformative about their observed response. A third possibility is that the
+participant's response reflects their \textit{actual} knowledge about the
+question's content. In this case, we \textit{might} expect to see a positive
+relationship between the participant's knowledge and their likelihood of
+answering the question correctly. However, in order to see this positive
+relationship, the participant's knowledge must be structured in a way that is
+reflected (at least partially) by the embedding space. In other words, if the
+participant's performance reflects their true knowledge, but our text embedding
+space does not sufficiently capture the structure of that knowledge, then the
+knowledge estimates we generate will not be predictive of the participant's
+performance. In the extreme, if the embedding space is completely unstructured
+with respect to the content of the quiz questions, then we would expect to see
+the negative relationship between estimated knowledge and performance that we
+described above.
 
 When we fit a GLMM to estimates of participants' knowledge for each Quiz~1
 question based on all other Quiz~1 questions, we observed an outcome consistent
@@ -752,7 +752,7 @@ \section*{Results}
 95\%\ \textnormal{CI} = [3.033, 3.866],\ p = 0.094$). These ``prediction
 failures'' appear to come from the fact that any signal derived from
 participants' knowledge about the content of the \textit{Birth of Stars}
-lecture (prior to watching it) is swamped by the much more dramatic increase in
+lecture (prior to watching it) is overwhelmed by the much more dramatic increase in
 their knowledge about the content of the \textit{Four Fundamental Forces}
 (which they watched just prior to taking Quiz~2). This is reflected in their
 Quiz~2 performance for questions about each lecture (mean proportion correct
@@ -770,12 +770,7 @@ \section*{Results}
 answer individual quiz questions, distinguish between questions about similar
 content, and generalize across content areas, provided that participants' quiz
 responses reflect a minimum level of ``real'' knowledge about both content on
-which these predictions are based and that for which they are made. Our results
-also indicate some important limitations of our approach: if participants' quiz
-performance does not reflect what they know (e.g., when they ``guess''), or if
-their knowledge is not structured in a way that is reflected by the embedding
-space, then our knowledge estimates will not be predictive of their
-performance.
+which these predictions are based and that for which they are made.
 
 % our approach works when participants have a minimal baseline level of knowledge about content predicted and used to predict
 % our approach generalizes when knowledge of content used to predict can be assumed to be a reasonable indicator of knowledge of content predicted