Revert "Add pinned numpy and fix build (pytorch#155129)"

This reverts commit a3098a7.

Reverted pytorch#155129 on behalf of https://github.com/malfet due to Broke test_spectral_op, looks like missing xfail, see https://hud.pytorch.org/hud/pytorch/pytorch/0db3e0cf29604dae1007a678603e4dffd1c57562/1?per_page=50&name_filter=linux-jammy-py3.9-gcc11&mergeEphemeralLF=true ([comment](pytorch#155129 (comment)))

Author: pytorchmergebot

.ci/docker/requirements-docs.txt

@@ -58,5 +58,3 @@ sphinx-copybutton==0.5.0
 sphinx-design==0.4.0
 sphinxcontrib-mermaid==1.0.0
 myst-parser==0.18.1
-
-numpy==1.26

torch/nn/attention/bias.py

@@ -36,14 +36,14 @@ class CausalVariant(IntEnum):
 
     Defines two types of causal biases:
 
-    ``UPPER_LEFT``: Represents upper-left triangular bias for standard causal attention.
+    `UPPER_LEFT`: Represents upper-left triangular bias for standard causal attention.
     The equivalent pytorch code for constructing this bias is:
 
     .. code-block:: python
 
         torch.tril(torch.ones(size, dtype=torch.bool))
 
-    For instance, with ``shape=(3,4)``, the materialized bias tensor will be:
+    For instance, with `shape=(3,4)`, the materialized bias tensor will be:
 
     .. code-block:: text
 
@@ -52,7 +52,7 @@ class CausalVariant(IntEnum):
          [1, 1, 1, 0]]
 
 
-    ``LOWER_RIGHT``: Represents lower-right triangular bias, the include values are aligned to the lower
+    `LOWER_RIGHT`: Represents lower-right triangular bias, the include values are aligned to the lower
     right corner of the matrix.
 
     The equivalent pytorch code for constructing this bias is:
@@ -65,7 +65,7 @@ class CausalVariant(IntEnum):
             diagonal=diagonal_offset,
         )
 
-    For instance, with ``shape=(3,4)``, the materialized bias tensor will be:
+    For instance, with `shape=(3,4)`, the materialized bias tensor will be:
 
     .. code-block:: text