Running out of memory fast when drawing MESMER-M emulations

[veni-vidi-vici-dormivi]

Drawing emulations for MESMER-M is (at least?) 12 times more memory intensive than for MESMER. I cannot draw much more than 100 emulations on the MacBook Pro with 18 GB of RAM (the process gets killed or runs into a segmentation fault at about 60 GB of swap usage).
Something related might be #165 (2).

[veni-vidi-vici-dormivi]

mesmer/mesmer/stats/_auto_regression.py

Lines 1170 to 1177 in be8455e

	out = np.zeros([n_samples, n_ts + buffer * 12, n_gridcells])
	for t in range(1, n_ts + buffer * 12):
	month = t % 12
	out[:, t, :] = (
	intercept[month, :]
	+ slope[month, :] * out[:, t - 1, :]
	+ innovations[:, t, :]
	)

doubles the memory use...

[veni-vidi-vici-dormivi]

mesmer/mesmer/stats/_power_transformer.py

Line 44 in be8455e

transf = np.empty_like(data)

Not optimal either, even though it happens in an apply_ufunc along sample_dim so I think it deletes the doubled array after every gridcell and realisation.

Plus we have the four masks also:

mesmer/mesmer/stats/_power_transformer.py

Lines 101 to 104 in 45b6dd6

	sel_a = (data >= 0) & (np.abs(lambdas) < eps)
	sel_b = (data >= 0) & (np.abs(lambdas) >= eps)
	sel_c = (data < 0) & (np.abs(lambdas - 2) > eps)
	sel_d = (data < 0) & (np.abs(lambdas - 2) <= eps)

[veni-vidi-vici-dormivi]

For context, 1000 emulation of monthly temperature for the typical clip6 ng ~2600 land gridcells and 251 years has about 60 GB in the end.

[mathause]

This is the required RAM to create the emulations or the size of the emulations? The second case would not be possible to store in RAM no matter what we optimize.

[veni-vidi-vici-dormivi]

No that's the total size of the emulations.