rdwd: climate data from the German Weather Service
2017-11-03
Vignette Rmd source code
Interactive map vignette
Intro
The R package rdwd contains code to select, download and read weather data from measuring stations across Germany. The German Weather Service (Deutscher Wetterdienst, DWD) provides over 25 thousand datasets with weather observations through the FTP server online at
ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate.
For data interpolated onto a 1 km raster, including radar data up to the last hour, see ftp://ftp-cdc.dwd.de/pub/CDC/grids_germany/. Management of these (very large!) datasets is not yet included in rdwd. If you want it to be supported, send me a message to boost it up my priority list.
For further details, please consult the DWD FTP server documentation
Package structure
To use the observation datasets, rdwd has been designed to mainly do 3 things:
selectDWD: facilitate file selection, e.g. for certain station names (with findID), by geographical location (see map and nearbyStations), by temporal resolution (hourly, daily, monthly), variables (temperature, rain, wind, sun, clouds, etc) or observation period (historical long term records or the current year)
dataDWD: download a file (or multiple files, without getting banned by the FTP-server)
readDWD: read that data into R (including useful defaults for metadata)
selectDWD uses the result from indexFTP which recursively lists all the files on an FTP-server (using RCurl::getURL). As this is time consuming, the result is stored in the package dataset fileIndex. From this, metaIndex and geoIndex are derived.
TOC
Package installation
install.packages("rdwd")
# get the latest development version from github:
berryFunctions::instGit("brry/rdwd")
# For full usage, as needed in indexFTP and metaDWD(..., current=TRUE):
install.packages("RCurl") # is only suggested, not mandatory dependency
On Linux, instead of the last line above, use in the terminal (with lowercase rcurl):
sudo apt-get install r-cran-rcurl
If direct installation from CRAN doesn’t work, your R version might be too old. In that case it is really recommendable to update R. If you can’t update R, try installing from source (github) via instGit as mentioned above. If that’s not possible either, you might be able to source some functions from the package zip folder
Vectorize(source)(dir("path/you_unzipped_to/rdwd-master/R", full=T))
TOC
Basic usage
library(rdwd)
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
file <- dataDWD(link, read=FALSE, dir=tempdir(), quiet=TRUE)
clim <- readDWD(file)
str(clim)
## 'data.frame': 550 obs. of 19 variables:
## $ STATIONS_ID: int 3987 3987 3987 3987 3987 3987 3987 3987 3987 3987 ...
## $ MESS_DATUM : POSIXct, format: "2016-05-02" "2016-05-03" ...
## $ QN_3 : int 10 10 10 10 10 10 10 10 10 10 ...
## $ FX : num 8.5 14.8 12.2 6.1 10.5 14.7 16.8 14.4 14.3 12.2 ...
## $ FM : num 3.2 3.6 3.8 2.5 4 4.8 4.9 4.8 4.6 4.6 ...
## $ QN_4 : int 3 3 3 3 3 3 3 3 3 3 ...
## $ RSK : num 0 0.7 0 0 0 0 0 0 0 0 ...
## $ RSKF : int 0 6 6 0 0 0 0 0 0 0 ...
## $ SDK : num 13.82 8.93 3.37 14.17 14.5 ...
## $ SHK_TAG : int 0 0 0 0 0 0 0 0 0 0 ...
## $ NM : num 0.7 4.3 6.2 2.3 1.3 0.8 0.9 1.2 1.9 2.7 ...
## $ VPM : num 7.9 8.5 8.4 7.7 8.7 8.6 9.4 10.3 10.3 9.9 ...
## $ PM : num 1011 1007 1015 1015 1008 ...
## $ TMK : num 13.3 12.8 9.3 13.3 16.7 18.2 17.9 18.6 18.4 18.8 ...
## $ UPM : num 54.3 59.9 73.3 53.8 50 ...
## $ TXK : num 20.3 20.7 14.5 20.7 24.5 25.9 25.7 26 25.9 26.1 ...
## $ TNK : num 5.4 7.5 6 5.5 8.7 10.5 10.7 11.4 11.2 11.4 ...
## $ TGK : num 1.3 3.8 1.7 1.5 5 6.5 5.8 7.2 7 9.3 ...
## $ eor : Factor w/ 1 level "eor": 1 1 1 1 1 1 1 1 1 1 ...
TOC
Plotting examples
Recent temperature time series:
par(mar=c(4,4,2,0.5), mgp=c(2.7, 0.8, 0), cex=0.8)
plot(clim[,c(2,4)], type="l", xaxt="n", las=1, main="Daily temp Potsdam")
berryFunctions::monthAxis(ym=TRUE) ; abline(h=0)
mtext("Source: Deutscher Wetterdienst", adj=-0.1, line=0.5, font=3)
Long term climate graph:
link <- selectDWD("Potsdam", res="monthly", var="kl", per="h")
clim <- dataDWD(link, quiet=TRUE)
clim$month <- substr(clim$MESS_DATUM_BEGINN,5,6)
temp <- tapply(clim$MO_TT, clim$month, mean)
prec <- tapply(clim$MO_RR, clim$month, mean)
library(berryFunctions)
climateGraph(temp, prec, main="Potsdam 1893:2016")
mtext("Source: Deutscher Wetterdienst", adj=-0.05, line=2.8, font=3)
TOC
Station selection
Weather stations can be selected geographically with the interactive map. All stations within a certain radius around a given lat-long position can be obtained with nearbyStations.
The DWD station IDs can be obtained from station names with
## Potsdam
## 3987
findID("Koeln", exactmatch=FALSE)
## Warning: tools::buildVignettes -> engine$ findID: ID determined from name
## 'Koeln' has 4 elements (2665, 2666, 2667, 2968).
## Koeln-Bonn Koeln-Botanischer Garten Koeln-Porz-Eil
## 2667 2665 2666
## Koeln-Stammheim
## 2968
TOC
Available files
File selection by station name/id and folder happens with selectDWD. It needs an index of all the available files on the server. The package contains such an index (fileIndex) that is updated (at least) with each CRAN release of the package. The selectDWD documentation contains an overview of the FTP folder structure.
If you find the file index to be outdated (Error in download.file … : cannot open URL), please let me know and I will update it. Meanwhile, use current=TRUE in selectDWD:
# all files at a given path, with current file index (RCurl required):
links <- selectDWD(res="monthly", var="more_precip", per="hist", current=TRUE)
fileIndex is created with the function indexFTP used in the last section of rdwd-package.R.
# recursively list files on the FTP-server:
files <- indexFTP("hourly/sun") # use dir="some_path" to save the output elsewhere
berryFunctions::headtail(files, 5, na=TRUE)
# with other FTP servers, this should also work...
funet <- indexFTP(base="ftp.funet.fi/pub/standards/RFC/ien")
p <- RCurl::getURL("ftp.funet.fi/pub/standards/RFC/ien/",
verbose=T, ftp.use.epsv=TRUE, dirlistonly=TRUE)
TOC
File selection
selectDWD is designed to be very flexible:
# inputs can be vectorized, and period can be abbreviated:
selectDWD(c("Potsdam","Wuerzburg"), res="hourly", var="sun", per="hist")
## [[1]]
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_03987_18930101_20161231_hist.zip"
##
## [[2]]
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_05705_19510101_20161231_hist.zip"
# Time period can be doubled to get both filenames:
selectDWD("Potsdam", res="daily", var="kl", per="rh", outvec=TRUE)
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/recent/ tageswerte_KL_03987_akt.zip"
## [2] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/ tageswerte_KL_03987_18930101_20161231_hist.zip"
There may be a differing number of available files for several stations across all folders. That’s why the default outvec is FALSE.
lapply(selectDWD(id=c(3467,5116)), substr, 58, 1e4)
## Warning: tools::buildVignettes -> engine$ lapply -> selectDWD: in file
## index 'fileIndex', there are 4 files with ID 3467.
## Warning: tools::buildVignettes -> engine$ lapply -> selectDWD: in file
## index 'fileIndex', there are 2 files with ID 5116.
## [[1]]
## [1] "/daily/more_precip/historical/tageswerte_RR_03467_19930601_20161231_hist.zip"
## [2] "/daily/more_precip/recent/tageswerte_RR_03467_akt.zip"
## [3] "/monthly/more_precip/historical/monatswerte_RR_03467_19930601_20161231_hist.zip"
## [4] "/monthly/more_precip/recent/monatswerte_RR_03467_akt.zip"
##
## [[2]]
## [1] "/daily/more_precip/historical/tageswerte_RR_05116_19920701_20061231_hist.zip"
## [2] "/monthly/more_precip/historical/monatswerte_RR_05116_19920701_20061231_hist.zip"
TOC