LIT 1.2 release notes.

PiperOrigin-RevId: 646672631

Author: bdu91

RELEASE.md

@@ -1,5 +1,68 @@
 # Learning Interpretability Tool Release Notes
 
+## Release 1.2
+
+This release covers clean-ups on various obsolete demos, as well as improved
+packaging and isolated dependencies on the GLUE, Penguin, Prompt Debugging with
+Sequence Salience and TyDi demos for easier launch.
+
+### New Stuff
+* Improved packaging and instructions for launching Prompt Debugging with
+Sequence Salience demo, as well as minor bug fixes -
+[08289df](https://github.com/PAIR-code/lit/commit/08289df0dd9927dee7147e5aad6e8b51bbe74f9e),
+[675ca2d](https://github.com/PAIR-code/lit/commit/675ca2de21b68dc62e4909c80a2cd57d8ee8b601),
+[15eccb1](https://github.com/PAIR-code/lit/commit/15eccb1197366c925a5beff310fb5d7d369bde0c),
+[e0e35c3](https://github.com/PAIR-code/lit/commit/e0e35c3ffcfd9ad5331d4154e7d33d0b1d0daf89),
+[c7970fb](https://github.com/PAIR-code/lit/commit/c7970fb8c51d2a8bd3647cc7eedd15cca285ac08),
+[cee3b58](https://github.com/PAIR-code/lit/commit/cee3b58baea2de27633109e6dd5b3e4211fa46ea)
+
+* Clean up of obsolete demos -
+[b16059f](https://github.com/PAIR-code/lit/commit/b16059fbd0320d411298009c0226489e1f548a69),
+[f4c0990](https://github.com/PAIR-code/lit/commit/f4c099082f0e89986aad162cc3cd0ac9bc2214c7),
+[6aa2eb6](https://github.com/PAIR-code/lit/commit/6aa2eb64eddb8ca154401bfd6a039762bc374d6d),
+[c2fb41b](https://github.com/PAIR-code/lit/commit/c2fb41b4945edb91fac973cf0ddbca48c6257511),
+[dd196e9](https://github.com/PAIR-code/lit/commit/dd196e941058a1d4246b3df3a3c37595f9791b18),
+[72fd772](https://github.com/PAIR-code/lit/commit/72fd772fa02c7445f27fb517e667987ea8ab34d7),
+[71d88fb](https://github.com/PAIR-code/lit/commit/71d88fb86eb88ffb80d665cf7571b21d7ae06bd2),
+[aa49340](https://github.com/PAIR-code/lit/commit/aa493409c454a2ed269fdedd15353404c14b4936),
+[fc7b0d0](https://github.com/PAIR-code/lit/commit/fc7b0d0624f6cc8e456ac0a1d75a4149927bef2f),
+[2475b3b](https://github.com/PAIR-code/lit/commit/2475b3bb677c8685ab9a291c490783ae2ccce5b8),
+[a59641c](https://github.com/PAIR-code/lit/commit/a59641c014b17409e8e5cfdac1cc1e6916d6da15),
+[1ed82d4](https://github.com/PAIR-code/lit/commit/1ed82d4e81ff6a6ff5146b6198e35444960d326b),
+[7d5ef58](https://github.com/PAIR-code/lit/commit/7d5ef5831427de71416c096a6dbcd46ea064457e),
+[992823b](https://github.com/PAIR-code/lit/commit/992823b027fca8c60edabe837248a508ac04da22),
+[3dad2b0](https://github.com/PAIR-code/lit/commit/3dad2b061b45cb44b1c3f9b9364660e907662069),
+[0656386](https://github.com/PAIR-code/lit/commit/0656386188d6e4b6c83dab58fb4e6569ebea217e),
+[27d7a84](https://github.com/PAIR-code/lit/commit/27d7a841cf6d514e67ebfb2af9f603398499f6e3),
+[8863019](https://github.com/PAIR-code/lit/commit/886301972ec1e7ed274040b46ec0e0c3f34c8ace),
+[71cbdba](https://github.com/PAIR-code/lit/commit/71cbdbaee0fee8e96f52cd4df7a269a0873b9259),
+[416d573](https://github.com/PAIR-code/lit/commit/416d573d79f84b9a6964d36e498b850a249ef452)
+
+* Python requirements update and isolated setup for individual demos -
+[bcc481e](https://github.com/PAIR-code/lit/commit/bcc481e44185d04268f5f8bb4ba762ec2cd35907),
+[bb29f43](https://github.com/PAIR-code/lit/commit/bb29f430ff7be55d74a82aec5dee1e54fa27bed0),
+[fbd8874](https://github.com/PAIR-code/lit/commit/fbd88746263fec0f72f2f01bcc382e88e902ab50),
+[b3c120b](https://github.com/PAIR-code/lit/commit/b3c120b22138fb03a712f11778197cf4966d0c3a),
+[5188c8c](https://github.com/PAIR-code/lit/commit/5188c8c835328efcc9dff5a0a4cf4cd79fabe099),
+[5639e3b](https://github.com/PAIR-code/lit/commit/5639e3b1b71b1c0ddf4a3c9e1bd25517fba18375)
+
+* Documentation cleanup and updates -
+[afd51fe](https://github.com/PAIR-code/lit/commit/afd51fe299c0070a19946a789984957f14a9b5bb),
+[7dda659](https://github.com/PAIR-code/lit/commit/7dda659bec4e933d187b0d7afc04d954ae262cc2),
+[79ada6e](https://github.com/PAIR-code/lit/commit/79ada6edf8b2e485ec6a6425d4c60720b4dab8d1),
+[1c8d6a0](https://github.com/PAIR-code/lit/commit/1c8d6a0269ce5637e05e79ae435f770e2a0da147),
+[2e9d267](https://github.com/PAIR-code/lit/commit/2e9d26738d9344cde0eebd66d49dfc14cd800e74)
+
+### Non-breaking Changes, Bug Fixes, and Enhancements
+* Refactor DataService reactions - [483082d](https://github.com/PAIR-code/lit/commit/483082dcb0beb39795c0fc093fe93036bb6a274c)
+* Add warm_start option to LitWidget - [a5265a4](https://github.com/PAIR-code/lit/commit/a5265a4feeb701b878986f79665d5fdf9ddc244c)
+* Pretty-printing of Model objects - [4fb3bde](https://github.com/PAIR-code/lit/commit/4fb3bde897c68fdeb3bd829f6e5a88223bc131a4)
+* Avoid equivalent shuffles in Scrambler - [0d8c0d9](https://github.com/PAIR-code/lit/commit/0d8c0d948480e0835fd3f451b95b7ec306b6409d)
+* Updated gunicorn config for demos running in Docker - [b14e3b1](https://github.com/PAIR-code/lit/commit/b14e3b1a81d7b6305063f778f46666a4d1326045)
+* Disable embeddings for TyDi - [7ff377f](https://github.com/PAIR-code/lit/commit/7ff377f92820748476e796994fd207e1b5dba1d9)
+* Cast embeddings to float32 before computing distances - [5456011](https://github.com/PAIR-code/lit/commit/5456011db8ead5d53db6f39bcdca3fc388802fbe)
+* Update colab examples to include installation of the lit-nlp package - [48b029c](https://github.com/PAIR-code/lit/commit/48b029c3a1a3f25d4d2611a9b0e94355d41078ef)
+
 ## Release 1.1.1
 
 This release covers various improvements for sequence salience, including new

docs/demos/index.html

@@ -98,6 +98,17 @@
   <div class="demo-card-copy">Analyze a tabular data model with LIT, including exploring partial dependence plots and automatically finding counterfactuals.</div>
   <div class="demo-card-cta-button"><a href="/lit/demos/penguins.html"></a></div>
 </div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/images.html" target="_blank">Image classification</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> images </span>  <span class="demo-tag"> multiclass classification </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Imagenette
+  </div>
+  <div class="demo-card-copy">Analyze an image classification model with LIT, including multiple image salience techniques.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/images.html"></a></div>
+</div>
 <div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
   <div class="demo-card-title"><a href="/lit/demos/glue.html" target="_blank">Classification and regression models</a></div>
   <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> binary classification </span>  <span class="demo-tag"> multi-class classification </span>  <span class="demo-tag"> regression </span> 
@@ -119,6 +130,50 @@
   </div>
   <div class="demo-card-copy">Use LIT directly inside a Colab notebook. Explore binary classification for sentiment analysis using SST2 from the General Language Understanding Evaluation (GLUE) benchmark suite.</div>
   <div class="demo-card-cta-button"><a href="https://colab.research.google.com/github/PAIR-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/coref.html" target="_blank">Gender bias in coreference systems</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> coreference </span>  <span class="demo-tag"> fairness </span>  <span class="demo-tag"> Winogender </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Winogender schemas
+  </div>
+  <div class="demo-card-copy">Use LIT to explore gendered associations in a coreference system, which matches pronouns to their antecedents. This demo highlights how LIT can work with structured prediction models (edge classification), and its capability for disaggregated analysis.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/coref.html"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/lm.html" target="_blank">Fill in the blanks</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> masked language model </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Stanford Sentiment Treebank, Movie Reviews
+  </div>
+  <div class="demo-card-copy">Explore a BERT-based masked-language model. See what tokens the model predicts should fill in the blank when any token from an example sentence is masked out.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/lm.html"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/t5.html" target="_blank">Text generation</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> T5 </span>  <span class="demo-tag"> generation </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    CNN / Daily Mail
+  </div>
+  <div class="demo-card-copy">Use a T5 model to summarize text. For any example of interest, quickly find similar examples from the training set, using an approximate nearest-neighbors index.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/t5.html"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/is_eval.html" target="_blank">Evaluating input salience methods</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> salience </span>  <span class="demo-tag"> evaluation </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Stanford Sentiment Treebank, Toxicity
+  </div>
+  <div class="demo-card-copy">Explore the faithfulness of input salience methods on a BERT-base model across different datasets and artificial shortcuts.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/is_eval.html"></a></div>
 </div>
   </div>
 </div>

docs/documentation/_images/attention.png

@@ -1,6 +1,6 @@
 # LIT Python API
 
-<!--* freshness: { owner: 'lit-dev' reviewed: '2024-06-24' } *-->
+<!--* freshness: { owner: 'lit-dev' reviewed: '2023-08-23' } *-->
 
 <!-- [TOC] placeholder - DO NOT REMOVE -->
 
@@ -349,7 +349,7 @@ list of scores for each token. The Integrated Gradients saliency method
 additionally requires a `TokenEmbeddings` input and corresponding output, as
 well as a label field `Target` to pin the gradient target to the same class as
 an input and corresponding output. See the
-[GLUE models class](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/glue/models.py)
+[GLUE models class](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/glue_models.py)
 for an example of these spec requirements.
 
 The core API involves implementing the `run()` method:
@@ -675,7 +675,7 @@ Each `LitType` subclass encapsulates its own semantics (see
 *   A field that appears in _both_ the model's input and output specs is assumed
     to represent the same value. This pattern is used for model-based input
     manipulation. For example, a
-    [language model](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/glue/models.py)
+    [language model](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/pretrained_lms.py)
     might output `'tokens': lit_types.Tokens(...)`, and accept as (optional)
     input `'tokens': lit_types.Tokens(required=False, ...)`. An interpretability
     component could take output from the former, swap one or more tokens (e.g.
@@ -712,9 +712,11 @@ this can cause jitter (UI modules appearing, disappearing, reordering, resizing,
 etc.) when switching between models or datasets with heterogeneous `Spec`s.
 
 When implementing your own LIT components and modules, you can use
-[`utils.find_spec_keys()`][utils-lib-py] (Python) and
-[`findSpecKeys()`][utils-lib] (TypeScript) to identify fields of interest in a
-`Spec`. These methods recognize and respect subclasses. For example,
+[`utils.find_spec_keys()`][utils-lib]
+(Python) and
+[`findSpecKeys()`][utils-lib]
+(TypeScript) to identify fields of interest in a `Spec`. These methods recognize
+and respect subclasses. For example,
 `utils.find_spec_keys(spec, Scalar)` will also match any `RegressionScore`
 fields, but `utils.find_spec_keys(spec, RegressionScore)` will not return all
 `Scalar` fields in the `Spec`.
@@ -805,13 +807,8 @@ _See the [examples](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples)
 
 ### Available types
 
-The full set of `LitType`s is defined in
-[types.py](https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/types.py). Numeric types
-such as `Integer` and `Scalar` have predefined ranges that can be overridden
-using corresponding `min_val` and `max_val` attributes as seen in
-[penguin data](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/penguin/data.py)
-`INPUT_SPEC`. The different types available in LIT are summarized in the table
-below.
+The full set of `LitType`s is defined in [types.py](https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/types.py). Numeric types such as `Integer` and `Scalar` have predefined ranges that can be overridden using corresponding `min_val` and `max_val` attributes as seen [here](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/datasets/penguin_data.py;l=19-22;rcl=574999438). The different types available in LIT are summarized
+in the table below.
 
 Note: Bracket syntax, such as `<float>[num_tokens]`, refers to the shapes of
 NumPy arrays where each element inside the brackets is an integer.
@@ -862,7 +859,7 @@ naming collisions with protected TypeScript keywords.*
 Some properties of the LIT frontend can be configured from Python as
 **arguments to `dev_server.Server()`**. These include:
 
-*   `page_title`: set a custom page title.
+*   `page_title`: set a custom page title, such as "Coreference Demo".
 *   `canonical_url`: set a "canonical" URL (such as a shortlink) that will be
     used as the base when copying links from the LIT UI.
 *   `default_layout`: set the default UI layout, by name. See `layout.ts` and
@@ -889,16 +886,22 @@ You can specify custom web app layouts from Python via the `layouts=` attribute.
 The value should be a `Mapping[str, LitCanonicalLayout]`, such as:
 
 ```python
-PENGUIN_LAYOUT = layout.LitCanonicalLayout(
+LM_LAYOUT = layout.LitCanonicalLayout(
     upper={
-        'Main': [
-            modules.DiveModule,
+        "Main": [
+            modules.EmbeddingsModule,
             modules.DataTableModule,
             modules.DatapointEditorModule,
         ]
     },
-    lower=layout.STANDARD_LAYOUT.lower,
-    description='Custom layout for the Palmer Penguins demo.',
+    lower={
+        "Predictions": [
+            modules.LanguageModelPredictionModule,
+            modules.ConfusionMatrixModule,
+        ],
+        "Counterfactuals": [modules.GeneratorModule],
+    },
+    description="Custom layout for language models.",
 )
 ```
 
@@ -909,12 +912,14 @@ lit_demo = dev_server.Server(
     models,
     datasets,
     # other args...
-    layouts=layout.DEFAULT_LAYOUTS | {'penguins': PENGUIN_LAYOUT},
-    default_layout='penguins',
+    layouts={"lm": LM_LAYOUT},
     **server_flags.get_flags())
 return lit_demo.serve()
 ```
 
+For a full example, see
+[`lm_demo.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/lm_demo.py).
+
 You can see the pre-configured layouts provided by LIT, as well as the list of
 modules that can be included in your custom layout in
 [`layout.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/layout.py). A
@@ -984,15 +989,15 @@ needing to reload the server or click the UI.
 For example, to view examples in a dataset:
 
 ```python
-from lit_nlp.examples.glue import data as glue_data
-dataset = glue_data.SST2Data('validation')
+from lit_nlp.examples.datasets import glue
+dataset = glue.SST2Data('validation')
 print(dataset.examples)  # list of records {"sentence": ..., "label": ...}
 ```
 
 And to run inference on a few of them:
 
 ```python
-from lit_nlp.examples.glue import models as glue_models
+from lit_nlp.examples.models import glue_models
 
 model = glue_models.SST2Model("/path/to/model/files")
 preds = list(model.predict(dataset.examples[:5]))
@@ -1016,19 +1021,16 @@ For a full working example in Colab, see [LIT_components_example.ipynb](https://
 <!-- Links -->
 
 [build-metadata]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/app.py
-[components-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/components.py
+[components-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/dataset.py
 [curves-interp]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/components/curves.py
 [dataset-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/dataset.py
 [grad-maps]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/components/gradient_maps.py
 [json]: https://www.json.org
 [mnli-dataset]: https://cims.nyu.edu/~sbowman/multinli/
-
 [mnli-demo]: https://pair-code.github.io/lit/demos/glue.html
-
-[model-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/model.py
+[model-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/dataset.py
 [should_display_module]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/client/core/lit_module.ts
 [types_py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/api/types.py
 [types_ts]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/client/lib/lit_types.ts
 [utils-lib]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/client/lib/utils.ts
-[utils-lib-py]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/lib/utils.py
 [word-replacer]: https://github.com/PAIR-code/lit/blob/main/lit_nlp/components/word_replacer.py