You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -41,19 +41,19 @@ The figure above shows on the left how the contrastive loss naturally will tend
41
41
We show that our embedding scheme is able to align representations of galaxies both in-modality and cross-modality around meaningful shared semantics. Specifically, we query our embedding space with either the image or spectrum representation of a galaxy, and show that the retrieved galaxies by cosine similarity of their embeddings are extremely close to the original one. Below, we present all four retrieval types (spectrum-spectrum, image-image, spectrum-image, and image-spectrum, from left to right) for four randomly chosen query galaxies in our testing set (highlighted in red on the left).
42
42
43
43
<palign="center">
44
-
<imgsrc="/images/blog/query-retrieval.png"alt="Query and Retrieval"width="85%"style="mix-blend-mode: darken;">
44
+
<imgsrc="/images/blog/query-retrieval.png"alt="Query and Retrieval"width="770px"style="max-width:100%"style="mix-blend-mode: darken;">
45
45
</p>
46
46
47
47
As one can see, the retrieved examples are galaxies of similar types, both for in-modality retrieval (b and c) and cross-modal retrieval (d and e).
48
48
49
49
We also present a couple of examples for the retrieved spectra, for both spectra queries (in-modality) and image queries (cross-modality) below:
These results demonstrate a strong correlation between the semantic content of the query, such as the red quiescent galaxy or a blue star forming galaxy, and the semantic content of the retrieved images or spectra.
@@ -66,11 +66,11 @@ In particular, we use simple k-Nearest Neighbour (k-NN) regression of our embedd
66
66
Additionally, in-modality similarity appears to outperform cross-modality similarity as an input for the k-NN regression, indicating that, although our our contrastive training aims to connect embeddings between modalities, it has the emergent property of helping to structure the embeddings space within respective modalities. This is particularly evident for the redshift prediction (c, top panel) by similarity between spectra which is near perfect, even though redshift is not an information perfectly contained in images. This means that redshift has naturally emerged as a fundamental property which helps the spectral encoder to structure its embedding space.
Copy file name to clipboardExpand all lines: _posts/2023-10-09-mpp.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,7 @@ Our pretraining approach can be described in two steps:
37
37
2. Train a single scalable transformer model to predict the next step of a spatiotemporal series based on a small number of snapshots describing the history.
For step one, we first use a recent method from the time-series forecasting literature called [Reversible Instance Normalization](https://openreview.net/forum?id=cGDAkQo1C0p). This method unifies the scales of different datasets for ingestion into the network then re-injects the scale information back into the output. The normalized state variables are individually projected into a shared space with field-specific weights (right side of figure above).
@@ -61,14 +61,14 @@ After pretraining, our models are able to compete with or beat modern baselines
61
61
While this parity is impressive, we still expect fine-tuned, dedicated models to outperform general ones in most cases. The real question we would like to answer is whether this pretraining process actually improves the ability of the model to learn new physics. PDEBench has a natural division in the included fluid data between incompressible flow (Incompressible Navier-Stokes, Shallow Water) and compressible flow (Compressible Navier-Stokes). To explore the question, we pretrain new models without including compressible flow at all, then choose two distinct fine-tuning datasets. We call one “near” and the other “far”.
62
62
63
63
<palign="center"style="margin-bottom: 10px;">
64
-
<imgsrc="/images/blog/multiphysics_ke.png"alt="Visualizing the physics gap."width="85%">
64
+
<imgsrc="/images/blog/multiphysics_ke.png"alt="Visualizing the physics gap."width="770px"style="max-width:100%">
65
65
<!-- <figcaption style="padding-left:32px; padding-right:20px; line-height:1.3"> Looking at individual fields (density, in this case), the incompressible flow included in the training set (left) has strong resemblence to the compressible simulation with low mach number (center) with similar diffusion levels, but the high mach number flow (right) develops significantly more complex, small-scale features as a result of both lower diffusion and more compressible behavior. </figcaption> -->
66
66
</p>
67
67
68
68
Both datasets are generated by a compressible flow solver, but while "near" (center) is selected to be physically very similar to the incompressible Navier-Stokes data in the training set (left), "far" (right) is generated in a different flow regime that exhibits wildly different behavior across scales. In both cases, there are still significant differences in the solver, resolution, and boundary conditions making both challenging transfer tasks.
69
69
70
70
<palign="center"style="margin-bottom: 10px;">
71
-
<imgsrc="/images/blog/CNS_Xfer_Both.png"alt="Results of fine-tuning experiments."width="85%">
71
+
<imgsrc="/images/blog/CNS_Xfer_Both.png"alt="Results of fine-tuning experiments."width="770px"style="max-width:100%">
72
72
<!-- <figcaption style="padding-left:32px; padding-right:20px; line-height:1.3"> Normalized RMSE comparing fine-tuned and "from scratch" models over a range of available samples. </figcaption> -->
73
73
</p>
74
74
@@ -78,8 +78,8 @@ Here's an example of the long-term rollout after fine-tuning on only one-step-ah
0 commit comments