The data is out there - web services will find it

class: center, top

![:scale 30%](/assets/images/coding_club_logo_1.png)

<!-- Adjust the presentation to the session. Focus on the challenges,
this is not a coding tutorial.

Note, to include figures, store the image in the `/docs/assets/images`
folder and use the jekyll base.url reference as done in this template
or see https://jekyllrb.com/docs/liquid/tags/#links.
  using the scale attribute ![:scale 30%](...), you can adjust the image size.
-->

# 30 JUNE 2026

## INBO coding club

Herman Teirlinck 
01.21 - Jeanne Brabants

---
class: left, top

# Reminders

1. Did we confirm the room reservation on the _roomie_?
2. Did we start the recording?

---
class: center, middle

![:scale 90%](/assets/images/20260630/20260630_badge.png)

---
class: left, top

## Web service (WS)

A **web service** "is a service offered by an electronic device to another electronic device, communicating with each other via the Internet".*

"In a web service, a web technology such as HTTP  (Hypertext Transfer Protocol) is used for transferring machine-readable file formats such as XML and JSON."*

So instead of manually download a spreadsheet from a website and open it in R, you have your R script requesting the data to the web service (the **request**), you get the data back (the **response**) and you keeps working.

No clicking, no manual downloads, no copy-pasting 😱

* Wikipedia contributors. (2026, March 25). Web service. In Wikipedia, The Free Encyclopedia. Retrieved 07:41, June 29, 2026, from https://en.wikipedia.org/w/index.php?title=Web_service&oldid=1345273563

---
class: left, top

## Web Services and APIs

We hear often speaking about APIs. Are APIs web services? Actually not, but they are sometimes used as synonyms because there is a link between them.

### The restaurant metaphor

Think of a restaurant*. You (the customer) don't walk into the kitchen and cook your own food. You look at a menu, pick something, tell the waiter, and the kitchen sends back your dish.

- The menu = the API documentation (it tells you what you're allowed to ask for)
- The waiter = the API itself (takes your request, brings back the response)
- The kitchen = the web service / server, doing the actual work behind the scenes
- You = your R script, or any program, making the request

* The restaurant metaphor is widely used in computer science. Difficult to cite the real reference :-) However, if you want to read further about APIs concepts without getting too complex, please check the blogpost [What is an API (and what do they have to do with restaurants)?](https://www.meeum.com/articles/what-is-api/)

---
class: left, top

## Wrappers

Of course web services and APIs are also commonly used for non spatial data. For example, you can retrieve biodiversity data, hidrology data or meteorological data via APIs.

It can happen that (very kind) people wrote R/Python packages built specifically as a **wrapper** around their API. What is a wrapper? A set of functions which send requests and read responses instead of you. You just program in R/Python as you are used to: behind the screen **API calls** are sent, **API responses** with data are received and data transformed for example in data.frames.

Examples:
- [{rgbif}](https://docs.ropensci.org/rgbif/) R package: biodiversity data
- [{waterinfo}](https://docs.ropensci.org/wateRinfo) R package: hidrology data
- [{pydov}](https://pydov.readthedocs.io/en/stable/) Python package: soil, subsoil and groundwater of Flanders from  the [Databank Ondergrond Vlaanderen](https://www.dov.vlaanderen.be/) (DOV)
- [{reasin}](https://guardias-eu.github.io/reasin/): R package: alien species data from the European Alien Species Information Network (EASIN)
- [{worldmet}](https://openair-project.github.io/worldmet/) R package to access data from the [NOAA Global Historical Climate Network](https://www.ncei.noaa.gov/products/global-historical-climatology-network-hourly) and the [NOAA Integrated Surface Database](https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database) (ISD)

---
class: left, top

## How does the "asking" actually work?

It's clear: no requests, no data :-)

Most APIs we'll meet work over the same protocol your browser uses to load web pages: HTTP. You send a request to a specific web address (called an **endpoint**), and you get back a response — usually as **structured** data rather than a pretty webpage.

How to build these specific web addresses? It's not magic: read the documentation, again and again! If the documentation is not clear, then it's a frustrating trial and error.

---
class: left, top

## GBIF APIs

Let's start from the GBIF API documentation: [GBIF API beginners guide](https://data-blog.gbif.org/post/gbif-api-beginners-guide/) and the complete [GBIF API Reference](https://techdocs.gbif.org/en/openapi/).

Let's take the first example from the GBIF API beginners guide:

https://api.gbif.org/v1/species/match?name=Passer%20domesticus

You should see a wall of text known as JSON.

![:scale 100%](/assets/images/20260630/20260630_basic_pattern-api_call.png)

---
class: left, top

## GBIF APIs examples

Your call can contains multiple parameter/query pairs (`&`):

https://api.gbif.org/v1/species/search?rank=SPECIES&highertaxon_key=212&limit=1000

Will give back 1000 bird species records. 1000 is the maximum you can get from the GBIF species API.

https://api.gbif.org/v1/species/search?rank=SPECIES&highertaxon_key=212&limit=1000&offset=1000

Get the next 1000 records of birds. Offset will move the window of results up by 1000. You can **page** through API results if you want more records. This technique is called **paging** (like the pages of a book).

Other GBIF APIs: the **occurrence API** to get info about occurrences, the *dataset API** to get datasets info and the **download API** to trigger and handle GBIF downloads.

- dataset API example: https://api.gbif.org/v1/dataset/7888f666-f59e-4534-8478-3a10a3bfee45
- occurrence API example: https://api.gbif.org/v1/occurrence/1229395815
- "occurrence download" API example: https://api.gbif.org/v1/occurrence/download/0073188-260519110011954

---
class: left, top

## Send requests with httr2

How to send requests via R and extract the data? Use [{httr2}](https://httr2.r-lib.org/)!

Check the ["Create a request"](https://httr2.r-lib.org/articles/httr2.html#create-a-request) section from the Get Started.

The most important concept here is the HTTP **method**: a **verb* that tells the server what you want to do. The **GET** is the most common verb, indicating that we want to get a resource. Other verbs include POST, to create a new resource (e.g. to create a new GBIF Download), PUT, to replace an existing resource, and DELETE, to delete a resource. For us mortals, PUT and DELETE are out of scope :-)

---
class: left, top

## req_url_query() vs req_headers()

They look similar but control very different parts of the request. Here's the distinction with the menu/waiter analogy still in mind.

- The `req_url_query()` function: what you're asking for. It adds parameters to the URL itself — the visible part of your restaurant order, written into the address you're calling.

- The `req_headers()` funciton: how you're asking, and who's asking. Headers are metadata sent alongside the request, but not visible in the URL. In the restaurant analogy: headers are like the things you tell the waiter quietly that aren't on the printed order ticket — "I have a peanut allergy," "I'm a member, here's my loyalty card," "I'd like that to-go, not on a plate." The kitchen needs to know it, but it's not part of the dish you ordered.

For public APIs the headers are typically not needed.

---
class: left, top

## The httr2 request

Build a request:

```
req <-
 # Basic httr2 request object
 request("https://my.beautiful.api.org/") %>%
 # Add parameters and queries pairs
 req_url_query(name = "Damiano Oldoni")
```

---
class: left, top

## Get a response with httr2

Check the ["Perform a request and fetch the response"](https://httr2.r-lib.org/articles/httr2.html#perform-a-request-and-fetch-the-response) from the Getting Started.

```
resp <- req %>% req_perform()

result <- resp %>%
 resp_body_json()
```

---
class: left, top

Load the package:

```r
library(httr2)
library(tidyverse)
```

Use `install.packages("pkgname")` where needed.

---
class: center, top

### How to get started?

Check the [Each session setup](https://inbo.github.io/coding-club/gettingstarted.html#each-session-setup) to get started.

### First time coding club?

Check the [First time setup](https://inbo.github.io/coding-club/gettingstarted.html#first-time-setup) section to setup.

---
class: left, top

![:scale 100%](/assets/images/coding_club_sticky_concept.png)

No yellow sticky notes online. We use hackmd (see next slide) but basic principle doesn't change.

---
class: center, top

### Share your code during the coding session!

Go to https://hackmd.io/DnQ53lvgSPyVlmUc9x1R1Q?both

---
class: left, top

# Download data and code
Nothing to download today!

---
background-image: url(/assets/images/background_challenge_1.png)
class: left, top

# Challenge 1 - GBIF data with httr2

Use httr2 to request the same GBIF data we visualised on our web browser.

1. https://api.gbif.org/v1/species/match?name=Passer%20domesticus. Hint: ignore the `?`. You can use the space instead of `%20`.

2. https://api.gbif.org/v1/species/search?rank=SPECIES&highertaxon_key=212&limit=1000

3. https://api.gbif.org/v1/species/search?rank=SPECIES&highertaxon_key=212&limit=1000&offset=1000

4. https://api.gbif.org/v1/dataset/7888f666-f59e-4534-8478-3a10a3bfee45

5. https://api.gbif.org/v1/occurrence/1229395815

6. https://api.gbif.org/v1/occurrence/download/0073188-260519110011954

---
class: left, top

# Intermezzo 1 - The world of geospatial web services

Geospatial web services are at INBO a relevant type of web services. There are different types, depending on the data they provide:

**Web Feature Service** (WFS) is a standard protocol for serving geospatial data as "features" (e.g., vector data) over the web.

**Web Coverage Service** (WCS) is a standard protocol for serving geospatial data as "coverages" (e.g., raster data) over the web.

**Web Map Service** (WMS) is a standard protocol for serving georeferenced map images (JPEG, PNG, ...). Developed by the Open Geospatial Consortium (OGC) in 1999.

**Web Map Tile Service** (WMTS) is a standard protocol for serving digital maps using predefined image tiles. It complements the WMS. Developed by the OGC in 2010.

---
class: left, top

# Intermezzo 1 - The world of geospatial web services

Thanks Hans for the nice overview table in ["Overview of Spatial Web Services"](https://inbo.github.io/inbospatial/articles/wfs_wcs.html#overview-of-spatial-web-services) from the documentation of the [{inbospatial}](https://inbo.github.io/inbospatial/index.html) R package. This package is another beautiful example of an R package wrapper!

![:scale 90%](/assets/images/20260630/20260630_spatial_web_services.png)

---
background-image: url(/assets/images/background_challenge_2.png)
class: left, top

# Challenge 2 - KMI's WFS

The Royal Meteorological Institute (KMI) share a lot of data openly via WFS. For this challenge, you will desperately search information in their [API Documentation Download](https://opendata.meteo.be/api_documentation_download) website :-)

1. Get the available AWS stations (`aws:aws_station`). Hint: use still the `application/json` outputFormat and pass the full request to `sf::st_read()` directly! Optional: you can then create a map using the `mapview::mapview()` function.
2. Same data, but this time try to save it on disk as a CSV file directly. Hint: check the arguments of [`req_perform()`](https://httr2.r-lib.org/reference/req_perform.html).
3. Get daily meteorological data from the station ZEEBRUGGE from 2026-01-01 and save it direclty to disk as CSV file.
4. Get hourly meteorological data from the station of Diepenbeek from 2026-01-15 for a duration of 8 days. Get the data as CSV without saving it to disk. Hint: search the right `resp_body_*()` function and read the output of it with readr::read_csv().

---
background-image: url(/assets/images/background_challenge_3.png)
class: left, top

# Challenge 3A - waterinfo & wateRinfo

The R package wateRinfo is a wrapper around the Waterinfo API (download the [PDF](https://waterinfo.vlaanderen.be/download/9f5ee0c9-dafa-46de-958b-7cac46eb8c23?dl=0) documentation), which allows us to get the data behind the the [Waterinfo](https://www.waterinfo.vlaanderen.be/) data portal.

1. Using the R package wateRinfo, **get** the list of measurement **stations** providing data about the [Penman evaporation](https://en.wikipedia.org/wiki/Penman_equation). Hint: check the [Example](https://docs.ropensci.org/wateRinfo/#example) of the homepage and the function documentation.
2. Get the ID (integer) of the timeseries of Waregem daily Penman evaporation.
3. Get the timeseries in 2 as a data.frame. Get the data for the entire 2025.*

* To get very long time series you need to ask a token via mail. See [Note on restrictions of the downloads](https://docs.ropensci.org/wateRinfo/#note-on-restrictions-of-the-downloads) section. A token is not needed to solve the challenges.

---
background-image: url(/assets/images/background_challenge_3.png)
class: left, top

# Challenge 3B - waterinfo WITHOUT wateRinfo

1. However, not everything is available via the R package wateRinfo. Example: the rainfall intensity at a specific hour obtained via composite radar data. The data returned are raster data. Check the chapter "2.5 Opvragen van beelden (meetwaarden rasterdata)".

1. Get the instantaneous rainfall intensity in Belgium at 25-06-2026, 23am local time (UTC+2) as geotiff. Save it on disk as tiff file. You can open the returned file with `terra::rast()` and visualize it with `terra::plot()`.
2. Get the total rainfall intensity in Belgium over the last 24 hours at 2026-06-28 12pm (midday) local time (UTC+2). Save it on disk as tiff file. Again, you can use terra to open and visualize the file.
3. Repeat 2 but avoid to save the tiff on disk. Hint: same strategy as in challenge 2.1

---
class: left, top

# Bonus challenge 1

## XML - KMI available datasets
The main entry point of the KMI data is returned by `getCapabilities`. And it is returned as XML. XML stays for Extended Markup Language: next JSON, XML is another popular text format used to exchange a wide variety of data on the Web. Homepage: https://www.w3.org/XML/.
Try to find the list of available datasets. Check the section about **getCapabilities**. Build the query and use the package [{xml2}](https://xml2.r-lib.org/) to get into the returned data to extract the available datasets. Hint: replace json with xml in your httr2 code.

XML is not the easiest format to read, even if it was intended so. It's also not really intuitive to extract the desired information in R.

---
class: left, top

# Bonus challenge 2

We didn't speak about DOV data yet. You can use pydov to get data from DOV. And you can use the R package [{reticulate}](https://rstudio.github.io/reticulate/) to run Python code in R! Do you have an example? Did you do it already? Share your experience in the HackMD, please 🙏

---
class: left, top

# Big thanks

A big thanks to Toon Van Daele for proposing this topic and for providing me very helpful code snippets 👏👏👏

Also thanks to Pieter Huybrechts for showing me how many web services and APIs exist in Belgium and worldwide! Sorry Pieter to have not used them for this session. Feel free to share the examples you sent me via chat in the HackMD.

---
class: left, top

# The R package of the month - geodata

[{geodata}](https://rspatial.github.io/geodata/) is an R package for downloading geographic data. This package facilitates access to climate, elevation, soil, crop, species occurrence, and administrative boundary data, and is a successor of the getData() function from the raster package.

Worth a try!

---
class: left, top

# Resources

- Comprehensive [solutions](https://github.com/inbo/coding-club/blob/main/src/20260630/20260630_challenges_solutions.R) are available on GitHub. You can opt to download the solutions automatically by using `inborutils::setup_codingclub_session("20260630")`.
- The edited [video recording](https://vimeo.com/1217429725) is available on our [vimeo channel](https://vimeo.com/user/8605285/folder/1978815).
- [{httr2}](https://httr2.r-lib.org/) R package
- [{inbospatial}](https://github.com/inbo/inbospatial/) R package
- [{rgbif}](https://docs.ropensci.org/rgbif) R package
- [{wateRinfo}](https://github.com/ropensci/wateRinfo) R package
- [{worldmet}](https://openair-project.github.io/worldmet/) R package
- [{pydov}](https://pydov.readthedocs.io/en/stable/) Python package
- [{reticulate}](https://rstudio.github.io/reticulate/) R package: a R interface to Python. Useful for running the functions of pydov.

---
class: center, middle

![:scale 30%](/assets/images/coding_club_logo_1.png)

Topic: to be decided 
Room: HT - 01.72 - Kaat Tilley 
Date: **27/08/2026** (Tuesday), van **10:00** tot **12:30
Help needed with technical setup? You are welcome from 9:45am