From 358db883939d647fc51e31544764fc8099bf5e1c Mon Sep 17 00:00:00 2001 From: Veronica Andreo Date: Sat, 24 May 2025 17:13:28 -0300 Subject: [PATCH 1/2] Time series tutorials: Adding part 7 - Subset, import & export --- .../time_series/time_series_extraction.qmd | 279 ++++++++++++++++++ 1 file changed, 279 insertions(+) create mode 100644 content/tutorials/time_series/time_series_extraction.qmd diff --git a/content/tutorials/time_series/time_series_extraction.qmd b/content/tutorials/time_series/time_series_extraction.qmd new file mode 100644 index 0000000..4d5bb59 --- /dev/null +++ b/content/tutorials/time_series/time_series_extraction.qmd @@ -0,0 +1,279 @@ +--- +title: "Time series: Subset, import and export" +author: "Veronica Andreo" +date: 2024-08-19 +date-modified: today +lightbox: true +format: + ipynb: default + html: + toc: true + code-tools: true + code-copy: true + code-fold: false +categories: [time series, raster, intermediate, Python] +description: Tutorial on time series subsetting, import and export. +engine: jupyter +execute: + eval: false +jupyter: python3 +--- + +In this seventh time series tutorial, we will go through time +series subset, import and export. + +::: {.callout-note title="Setup"} +This tutorial can be run locally or in Google Colab. However, make sure you +install GRASS 8.4+, download the +[LST sample dataset](https://zenodo.org/doi/10.5281/zenodo.3564514) +and set up your project as explained in the +[first](time_series_management_and_visualization.qmd) time +series tutorial. +::: + +```{python} +#| echo: false + +import os +import sys +import subprocess + +# Ask GRASS where its Python packages are +sys.path.append( + subprocess.check_output(["grass", "--config", "python_path"], text=True).strip() +) +# Import the GRASS packages we need +import grass.script as gs +import grass.jupyter as gj + +path_to_project = "italy_eu_laea/italy_LST_daily" + +# Start the GRASS Session +session = gj.init(path_to_project) +``` + +## Subset + +For extracting a subset from a STRDS, we use +[t.rast.extract](https://grass.osgeo.org/grass-stable/manuals/t.rast.extract.html). +The subset is based on temporal variables like `start_time`, `start_doy`, +`end_week`, etc. +This tool outputs a new STRDS and offers the possibility to apply a +mapcalc operation on the fly. If no *r.mapcalc* expression is defined, +the selected maps are simply registered in the newly created output STRDS +to avoid data duplication. + +Let's see a couple of examples. Suppose we are only interested in northern summer +months, i.e., June, July, August. The maps matching the where condition will be +registered in the new output STRDS. + +```{python} +gs.run_command("t.rast.extract", + input="lst_daily", + output="lst_daily_summer", + where="strftime('%m',start_time)='06' or strftime('%m',start_time)='07' or strftime('%m', start_time)='08'") +``` + +If you remember, we went through several listing examples in the +[Time series management and visualization](time_series_management_and_visualization.qmd) tutorial. +To check the subset will actually give us what we want, we can run +[t.rast.list](https://grass.osgeo.org/grass-stable/manuals/t.rast.list.html) +with the same `where` condition as above. + +```{python} +#| scrolled: true +gs.run_command("t.rast.list", + input="lst_daily_summer", + columns="name,min,max") +``` + +To check whether a map is registered in different time series objects, we can use +[t.info](https://grass.osgeo.org/grass-stable/manuals/t.info.html). + +```{python} +gs.run_command("t.info", + input="lst_2018.243_avg", + type="raster") +``` + +Let's see now an example including a mapcalc operation. We still want the daily +maps of summer, but we are only interested in areas where LST was higher than +25 degrees. +Note that in this second case, we are creating a new STRDS, i.e., we modify the +original with the mapcalc expression, so we need to provide `basename` and `suffix` +for the newly created maps. + +```{python} +gs.run_command("t.rast.extract", + input="lst_daily", + output="lst_daily_summer_higher_25", + basename="lst_daily", + suffix="gran", + where="strftime('%m',start_time)='06' or strftime('%m',start_time)='07' or strftime('%m', start_time)='08'", + expression="if(lst_daily < 25.0, null(), lst_daily)") +``` + +Check that the minimum value of all extracted maps are actually above 25 degrees. + +```{python} +#| scrolled: true +gs.run_command("t.rast.list", + input="lst_daily_summer_higher_25", + columns="name,min,max") +``` + +What tools would you use now if you wanted to know how many summer days had +temperatures above 25 degrees each year and how does the maximum number of days +with LST > 25 varies regionally? + +**Small hint?** Go back to the time series +[aggregation](time_series_aggregations.qmd) tutorial. + +```{python} +# Get number of summer days with LST > 25 per year +gs.run_command("t.rast.aggregate", + input="lst_daily_summer_higher_25", + output="count_lst_daily_summer_higher_25", + basename="count_lst_daily_summer_higher_25", + suffix="gran", + method="count", + granularity="1 year") +``` + +```{python} +# Get maximum number of days with LST > 25 +gs.run_command("t.rast.series", + input="count_lst_daily_summer_higher_25", + output="max_count_summer_days_lst_higher_25", + method="maximum") +``` + +```{python} +# Mask zero values +gs.run_command("r.mask", + raster="max_count_summer_days_lst_higher_25", + maskcats=0, + flags="i") +``` + +```{python} +# Visualize the result +max_count = gj.InteractiveMap(tiles="CartoDB.DarkMatter") +max_count.add_raster("max_count_summer_days_lst_higher_25") +max_count.show() +``` + +## Export + +There are three tools to export raster time series in different formats. + +* [t.rast.export](https://grass.osgeo.org/grass-stable/manuals/t.rast.export.html) exports a strds as a tar archive containing raster maps either as GeoTIFF or GRASS binary files and several metadata files such as: timestamps, CRS info, strds and raster info. The archive can be compressed with gzip or bzip2. The output of *t.rast.export* can then be imported with [t.rast.import](https://grass.osgeo.org/grass-stable/manuals/t.rast.import.html). +* [t.rast.out.vtk](https://grass.osgeo.org/grass-stable/manuals/t.rast.out.vtk.html) exports a strds as a VTK file to be visualized with any VTK visualizer. +* [t.rast.out.xyz](https://grass.osgeo.org/grass-stable/manuals/addons/t.rast.out.xyz.html) exports a strds to a CSV file of the form x coord, y coord, value. + +Let's see some examples. + +```{python} +# Export strds as an archive +gs.run_command("t.rast.export", + input="lst_daily_summer", + output="lst_daily_summer_2014.tar.bzip2", + where="start_time < '2015-01-01'") +``` + +```{python} +!tar tjf lst_daily_summer_2014.tar.bzip2 +``` + +``` +lst_2014.152_avg.tif +lst_2014.152_avg.color +... +lst_2014.243_avg.tif +lst_2014.243_avg.color +list.txt +proj.txt +init.txt +readme.txt +metadata.txt +``` + +```{python} +# Export strds as VTK +gs.run_command("t.rast.out.vtk", + input="count_lst_daily_summer_higher_25", + directory="/tmp", + elevation="elevation") +``` + +The tool *t.rast.out.xyz* is an addon, so we first need to install it with *g.extension*. + +```{python} +# Install extension +gs.run_command("g.extension", extension="t.rast.out.xyz") +``` + +```{python} +# Export strds as xyz CSV file +gs.run_command("t.rast.out.xyz", + strds="count_lst_daily_summer_higher_25", + output="count_lst_daily_summer_higher_25.csv") +``` + +```{python} +!head count_lst_daily_summer_higher_25.csv +``` + +## Import + +There are two tools to import raster time series into GRASS GIS: + +* [t.rast.import](https://grass.osgeo.org/grass-stable/manuals/t.rast.import.html) imports strds that were exported with *t.rast.export*. It allows to create a new GRASS project with the imported data CRS and set the computational region to the raster maps imported. +* [t.rast.import.netcdf](https://grass.osgeo.org/grass-stable/manuals/addons/t.rast.import.netcdf.html) imports the content of one or more netCDF files that adhere to the CF convention into a strds. Data can be imported or linked via [r.external](https://grass.osgeo.org/grass-stable/manuals/r.external.html). + +Let's see an example with *t.rast.import*. + +```{python} +# Import the exported strds into a new GRASS project +gs.run_command("t.rast.import", + input="lst_daily_summer_2014.tar.bzip2", + output="lst_daily_summer_new", + title="Daily summer LST", + description="Daily summer LST for 2014") +``` + +```{python} +# Check the new strds was created +gs.run_command("t.list", where="NAME LIKE '%summer%'") +``` + +:::{.callout-note title="What about vector time series?"} +While not covered in this tutorial, there are also dedicated tools for subsetting, importing and exporting vector time series objects. These are: + +* [t.vect.extract](https://grass.osgeo.org/grass-stable/manuals/t.vect.extract.html) +* [t.vect.export](https://grass.osgeo.org/grass-stable/manuals/t.vect.export.html) +* [t.vect.import](https://grass.osgeo.org/grass-stable/manuals/t.vect.import.html) +::: + + +## References + +- Gebbert, S., Pebesma, E. 2014. +_TGRASS: A temporal GIS for field based environmental modeling._ +Environmental Modelling & Software 53, 1-12. +[DOI](http://dx.doi.org/10.1016/j.envsoft.2013.11.001). +- Gebbert, S., Pebesma, E. 2017. _The GRASS GIS temporal framework._ +International Journal of Geographical Information Science 31, 1273-1292. +[DOI](http://dx.doi.org/10.1080/13658816.2017.1306862). +- [Temporal data processing](https://grasswiki.osgeo.org/wiki/Temporal_data_processing) wiki page. + + +*** + +:::{.smaller} +The development of this tutorial was funded by the US +[National Science Foundation (NSF)](https://www.nsf.gov/), +award [2303651](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2303651). +::: + + From 54c34522213eb4acd7e5c4503f5c4ef534a781be Mon Sep 17 00:00:00 2001 From: Veronica Andreo Date: Sun, 25 May 2025 12:40:48 -0300 Subject: [PATCH 2/2] add thumbnail img --- .../time_series/images/export_with_change.png | Bin 0 -> 4633 bytes .../time_series/time_series_extraction.qmd | 5 +++-- 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 content/tutorials/time_series/images/export_with_change.png diff --git a/content/tutorials/time_series/images/export_with_change.png b/content/tutorials/time_series/images/export_with_change.png new file mode 100644 index 0000000000000000000000000000000000000000..407961c5be1b1e3159b6d30278a7aa21a9b2e14d GIT binary patch literal 4633 zcmeI0c~nzZ9>*U7qN37Rgo;Qis8|K!hFFwP5dp=L7AQLjvZ`##7KA`zErm8JG86$( zDO$7&AwmpW2#FG)76B13$`%$`G(y+{3EAep&_AZ9wRZlRIcMf`vfZ1P_ul=j-<$8g zaI#;uLT?2GL9329*g8XyiWEFQ`d}%Td99nn0dH#2Hb-2S0SLtYUj#JEHT$Ka1g1wnW`-t1&VXmr5w(?MpDQ6YH~2lOBaapj2ZLD%?e zz)coTk2~N*O_7v<4|J?PIV~}IkgiWeWPX{mBf%oF zA<*GqV8?Q=#4mr1@ILY7Pd3MO*B4#X9m?)%2|N`_DL*Nno1MKj!`Ql!)rr#ymlHBE z23pHDKCT$8F>9rbN@O>Dm{=;8uLADWB;cY#JjAE+r1H#s6S+2+>Q5DmhVppCnLO_$ z#BFaA6Ls+)fax8&t30*9J2c|duL?n*Em1>*!KTmtKlnFophq)c6X!eKaY@2XcSNQV zPpV3eXMhr05H0{)v+SnfC{OAV$m3Hr`I`B9T^wPmUbnqK>%J?!9L6pt7G^mCBGr#* zCjnvhANwyT5r+NU|52qx^2&TnU161ql9 z^+>~w<|44Z%g{ME9G8FaCj)$2{;g1Mi7zA0bdXPF$rt$45bIhB#U~<^96wPtH;0o) z!Ry_Y9U9RxY`9u~A5nfCUxrnM*aU@eXV+8oqd{{+r|3Q+ztqoYf&G`zby)>=L#wK8L~*;(5|T6*5r zthXq`(oU<%SN0{%w_|uo7K23j=$lTWd`2QA-@cvFr#7kZWYt!kt@}3Z`uEz zt^~Vb-Q$F{m&>fB0|FvZCQPG+q^2r_)M765B&_{5JycL->>pO_yly@DBYpjK;&~Pg zONXywhBM1%$@UJ;v^F+cD-Q6@q1{cx5y>hhS2T4S8d2B_qPMh+jDiCAM8zIqVJ6yT z*wT2P^Y8S@qPIMD;@glLU@qodQz8@fGQ7Rw1eYom@`eTt0}oLWRAGrPaTd?Er)3(W zaOktUz||p>*zH>;QuLVcwI<>ts{ueW&IEOJdIwK~ak!?LN z*Mxe0o9}f3XPDmffin&)r>Ayq&cBpmdXq*g9KN(4M8!`(0v}Rw>6Hm|pV|a-zIhY3 zY?Tznx|SoFZc4-RWV7w~zFe6oK{lev_o7fJZr<=Bx=L@kF1*GV87^H~*0Nhwrh7Nfl#EO9_UEojn#+wcaACDh(&~mn~Q&>O>;hKM>H_O`}Ga444G*FBxeA z1AK03Dh;a+v5(SUEg|~Bd_4H9&@tQ0_o__vRgvL}6cFgAi=5#sGVJQk%j@n|2fgBD z)c60nj`8D8(C*mJT$ELnGf|umD7Nc{q4@bDB^r>2wpy zKCGwm%uvHcPwG-qR`Y%lbeTj8X}w{UxOZ^&b^9Gm{td~{DkX>Y55!OQKRl1nvYLH* zNXtrlYLJYo6%CRR=anEBctIaC5zkqqCtnw;OzL~6K~mGVPL_(Mh7hAI*Z1Qmd^mIC z-4&6%WJ!8OEdM#hyTXnj8c0PZiw5ut`HO&btGO2eh;b!|J5JXvW8b)P>a?t-lXY4= z`OFfiq7~dGTWhY_f=8HXw)m~PAlB~(5@zt#Q+_<$HmE3ODm63$GkwL&`6YJWiTmez`vgmJEz zdY6a2jFO~lLQk%3`7 zK;xwil7UulAVq`;98FZs<5MC$qCr}^Ai{Mot=L5IMet} zWVHp?MW#X$k8VkX*!W|;YZY`ZHgTNJH8k_7JU#FzB~1fKn0%;#yzrvt?951W-W?Y5 z)2LU!WElNRG}I~)^brJc!daMO`P0^UElR4aQP61#&`6v7YB9Me?G-6D8wdLm1_^ie>YE;cZVH$k&7{V-a1$^oMlV6r16+8qnAo-JCl|wBf8XhT z_BmAq#v0UijP>47to@mCe7C0jZ~10VuoXlq43{mmvW(V59qaXan*Vojvw--TW z^|jL?anD1+u}!*dwpY;*seZkcqo@`2%`*pZ*jd$0I7J$>zlLyry2M*C%S=TAdvo zPoH*Zmj-_xFm%-%>;xc>ePV)HHr#58-00=O%wX{}7FLmev8`hJ&Z_7fDfKYXAtS7m z(2zVJojW=nz+Q1U&(`qRlwwox#a7#}u8v;jNf+1w4i{Z!-zC|9C+7PDrlFCs3RY78 zIcB2Kwk$7-bFR43e`h{QTPITYepD>JqP*{H2#^uh3h_Br<&GyhI^Y)fNHi!CBPV*R zZI1U=7rR5pLQkFy3p@GUiIbkR+l8BfHZ#$IYwTa{p0D)y&N7r3Da)QEP@p?#x0Qf% zLhR2S%_bSuFlAVH58p)oo9p`BBL7DRPeu7aa-G?`8yJp+jHn)p7AAmyLqa}<|KrIi aQmCsWEEl+~t;_&J=*VFw+Y%eUum1@ZxoSfI literal 0 HcmV?d00001 diff --git a/content/tutorials/time_series/time_series_extraction.qmd b/content/tutorials/time_series/time_series_extraction.qmd index 4d5bb59..2bfdc89 100644 --- a/content/tutorials/time_series/time_series_extraction.qmd +++ b/content/tutorials/time_series/time_series_extraction.qmd @@ -1,8 +1,9 @@ --- -title: "Time series: Subset, import and export" +title: "Time series subset, import and export" author: "Veronica Andreo" date: 2024-08-19 date-modified: today +image: images/export_with_change.png lightbox: true format: ipynb: default @@ -226,7 +227,7 @@ gs.run_command("t.rast.out.xyz", ## Import -There are two tools to import raster time series into GRASS GIS: +There are two tools to import raster time series into GRASS: * [t.rast.import](https://grass.osgeo.org/grass-stable/manuals/t.rast.import.html) imports strds that were exported with *t.rast.export*. It allows to create a new GRASS project with the imported data CRS and set the computational region to the raster maps imported. * [t.rast.import.netcdf](https://grass.osgeo.org/grass-stable/manuals/addons/t.rast.import.netcdf.html) imports the content of one or more netCDF files that adhere to the CF convention into a strds. Data can be imported or linked via [r.external](https://grass.osgeo.org/grass-stable/manuals/r.external.html).