Pollination adapted python code

Software Support
1

Dear all

We have adapted the InVEST pollination model python script to make the species input spatially explicit: Instead of one guild table (the same relative abundance of wild-bee species for all grid points in Switzerland) we are using bioclimatic species maps (probability of occurrence 0-100%). So every grid point has its own relative abundance of wild-bee species. However after changing formula 59 the model should work exactly the same as the standard, as every grid point has its single input value for relative abundance. Furthermore Eric Lonsdorf also confirmed that InVEST is automatically standaredising input values for relative abundance to 0-1. I had a short chat with Eric Lonsdorf and he suggested to contact the software team. We have some unexpected results that the output (supply, abundance, total and for each species) are far over 1.

Is there somebody to contact directly providing some support the python code?
Kind regards
Sibylle

@jdouglass @dave @swolny @mlacayo @jesseG

2

Hi @sibylles,

This is great that you’ve been adapting the pollination model to incorporate spatially explicit data.

I would first recommend confirming that your species maps are being properly normalized to 0-1 before being passed into the model. Or, could you try manually normalizing your species maps and see if you still get the same results?

If you can confirm this, I have a few follow-up questions: How are you normalizing the relative abundance raster? What are the unexpected values you are getting? Would you be willing to share your data (an example of your spatially explicit file that you’ve created and the unexpected results you’re getting)? Are you working on this model in a repo that you can share?

Thanks

3

Hi @csimpson

Many thanks for you reply.

Yes I can conform that I have normalized now my species maps
Values between 0-1. I normalized them by the total of all the pixels, do you have any other suggestion?

image
image1265×813 88.3 KB

Here the R code
library(raster)

input_directory ← “O:/Data…” # Pfad anpassen
output_directory ← “O:…” # Pfad für normalisierte Raster
dir.create(output_directory, showWarnings = FALSE) # Sicherstellen, dass das Output-Verzeichnis existiert
normalize_raster ← function(input_path, output_path) {
raster ← raster(input_path) # Raster einlesen
normalized_raster ← raster / 100 # Normalisieren (0–100 → 0–1)
writeRaster(normalized_raster, output_path, overwrite = TRUE)
}
files ← list.files(input_directory, pattern = “\.tif$”, full.names = TRUE)
for (file in files) {
output_file ← file.path(output_directory, basename(file)) # Behalte den Dateinamen bei
cat(“Normalisiere:”, file, “->”, output_file, “\n”)
normalize_raster(file, output_file)
}

Find here the official code. We realised that the python code is normalising guild table data, but not our raster data. So that it why I normalized our raster data manually:
In the code for the official version 3.14.2 pollination model, starting on line 1332, the species abundance in the guild table is divided by the total species abundance.

Here our adapted code:
I am wondering if there is an easy way to normalize the raster data as built-in function similar to the guild tables?
NCCS-InVEST/README.rst at main · mlacayoemery/NCCS-InVEST

Kind regards
Sibylle

@Lisa

4

Hi @sibylles

Find here the official code.

Here our adapted code:

I’m not sure if you’ve tried to insert a hyperlink to your code in either of the above spots but if so, I’m unable to see it.

I am wondering if there is an easy way to normalize the raster data as built-in function similar to the guild tables?

The InVEST Pollination model does not support spatially explicit species inputs at this time, so I think your best option is to make custom adjustments required for your data, as the pollination model does not currently have built-in functionality for this. That’s great that you’ve taken a look at the source code and I would definitely recommend looking at this to ensure that you follow the way the model treats the relative species abundance in the guild table. As you mentioned, the species abundance is divided by total species abundance, so (assuming I understand your data correctly) you should be calculating total species abundance by pixel by summing your species relative abundance rasters and dividing each of your species maps by that total abundance raster.

Hope this helps!

5

The adapted code using hypterlink:
https://github.com/mlacayoemery/NCCS-InVEST

Normalization

  • Yes the normalization is done for each species (map) separately.
  • No my absolute values are probaility of species occurence from species distribution data (0-100%) that are normalized. So it sums probability data and calculated for each raster of the total sum. Does this make sense?
  • Maybe I should provide the code to the R-help to have a cross-check.
    Kind regards
    Sibylle
6

sorry: it calculates for each raster the percentage of the total sum.

7

Dear @csimpson
Hope you enjoyed some end-of-year holidays. Would you be available to continue our discussion?

Summary of our aim: We would like to make the input variable “relative_abundance” of the species guild table spatially explicit.
Technically: Instead of reading in one guild table with one column “relative abundance” (same value over the entire country) we would like to read in species raster data (probability of occurrence 0-100%, .tif files). So each raster point has its own “relative abundance”.

Adapted code: We have adapted formula 59 of the python code. See InVEST adapted python
Challenge: We realised that when reading in raster files, the automatic standardisation/normalisation function of the python code is only working for guild tables, but not for .tif files. Therefore we included a preprocessing of the maps to guarantee that the probability values are normalized between 0-1.

Question: The adapted python code works

  • supply and abundance for individual species are between 0-1.
  • however, total pollinator abundance is > 1. So there is still a bug in the code or alternatively we need to explain what this total pollinator abundance values mean?
  • It would be nice if somebody form Natural Capital may carry out a double check.

Hiere the read.txt file to install the adapted python code:

NCCS

Preferred

Prerequisites

  • Open network access for git, pip, and conda
  • Install git and register a SSH key with GitHub
  • Anaconda
  1. Clone NCCS InVEST respository

mkdir NCCS cd NCCS git clone git@github.com:mlacayoemery/NCCS-InVEST.git cd NCCS-InVEST

Versions

Branch Description
natcap InVEST 3.14.2
v1 2024-09-19 NCCS-InVEST version 1
v2 2024-11 NCCS-InVEST version 2

Pick the desired version and switch the branch. For example, the command for the v2 branch follows:

git switch v2

  1. Setup development environment

conda create -n env-invest -c conda-forge “python=3.11” git gdal “numpy<2” abseil-cpp -y conda activate env-invest python -m pip install --upgrade pip setuptools wheel pip install -r requirements-dev.txt pip install . invest list invest run --help

  1. Run test scriptNote: Species abundance rasters must be in named as pollinator_abundance_[species].tif. See pre-formatted NCCS NMDS data.
  • Copy data into NCCS/sample

python NCCS_invest_pollination_U11.py

  1. Deactivate development environment

conda deactivate

8

Additionally the link to the input files and some output files: Input_Output

9

additionally the R code for the normalization of the tif. maps.

  • As I normalized each map separately, I assume that I am missing to consider the number of species (maps). This would make sense as the aggregation of the individual species to the total species abundance is > 1.
  • However, I think a bult-in function makes more sense, could anybody help me?

library(raster)

Verzeichnisse definieren

input_directory ← “O:/Data-Work/27_Natural_Resources-RE/271_KLIM_Work/CC_Impacts/NCCS/Data/M2b_Pollination/InVEST/Species_Input/valpar_bee_presence” # Pfad anpassen
output_directory ← “O:/Data-Work/27_Natural_Resources-RE/271_KLIM_Work/CC_Impacts/NCCS/Data/M2b_Pollination/InVEST/Species_Input/valpar_bee_presence/Norm” # Pfad für normalisierte Raster
dir.create(output_directory, showWarnings = FALSE) # Sicherstellen, dass das Output-Verzeichnis existiert

Funktion zur Normalisierung eines Rasters

normalize_raster ← function(input_path, output_path) {
raster ← raster(input_path) # Raster einlesen
normalized_raster ← raster / 100 # Normalisieren (0–100 → 0–1)

Normalisierten Raster speichern

writeRaster(normalized_raster, output_path, overwrite = TRUE)
}

Alle Rasterdateien im Verzeichnis verarbeiten

files ← list.files(input_directory, pattern = “\.tif$”, full.names = TRUE)
for (file in files) {
output_file ← file.path(output_directory, basename(file)) # Behalte den Dateinamen bei
cat(“Normalisiere:”, file, “->”, output_file, “\n”)
normalize_raster(file, output_file)
}

10

Dear @csimpson
Would you be available for a skype call?

Here the python part I am referring to (standard python L. 1339 ff).
https://github.com/natcap/invest/blob/main/src/natcap/invest/pollination.py

# * species_abundance[species] (string->float)
total_relative_abundance = guild_df[_RELATIVE_SPECIES_ABUNDANCE_FIELD].sum()
result['species_abundance'] = {}
for species in result['species_list']:
    result['species_abundance'][species] = (
        guild_df[_RELATIVE_SPECIES_ABUNDANCE_FIELD][species] /
        total_relative_abundance)

I suppose that my pre-processing for the adapted python is not equal

library(raster)

Verzeichnisse definieren

input_directory ← “O:/Data-Work/27_Natural_Resources-RE/271_KLIM_Work/CC_Impacts/NCCS/Data/M2b_Pollination/InVEST/Species_Input/valpar_bee_presence” # Pfad anpassen
output_directory ← “O:/Data-Work/27_Natural_Resources-RE/271_KLIM_Work/CC_Impacts/NCCS/Data/M2b_Pollination/InVEST/Species_Input/valpar_bee_presence/Norm” # Pfad für normalisierte Raster
dir.create(output_directory, showWarnings = FALSE) # Sicherstellen, dass das Output-Verzeichnis existiert

Funktion zur Normalisierung eines Rasters

normalize_raster ← function(input_path, output_path) {
raster ← raster(input_path) # Raster einlesen
normalized_raster ← raster / 100 # Normalisieren (0–100 → 0–1)

Normalisierten Raster speichern

writeRaster(normalized_raster, output_path, overwrite = TRUE)
}

Alle Rasterdateien im Verzeichnis verarbeiten

files ← list.files(input_directory, pattern = “\.tif$”, full.names = TRUE)
for (file in files) {
output_file ← file.path(output_directory, basename(file)) # Behalte den Dateinamen bei
cat(“Normalisiere:”, file, “->”, output_file, “\n”)
normalize_raster(file, output_file)
}

O:\Data-Work\27_Natural_Resources-RE\271_KLIM_Work\CC_Impacts\NCCS\Data\M2b_Pollination\InVEST\Species_Input\valpar_bee_presence

11

@csimpson @esoth @edavis @jdouglass @dave

Dear Natural Capital Team

Do you have any idea how to adapt my normalisation pre-processing within the adapted python code?

Because I already standardized the input species maps to 0-1. What do you think about dividing the values to the number of species included?

Keep in mind that I am not able to do the same standardization as used in the standard python code: I do not have one guild table with all the relative abundance values for the species, but a species map for each species.

Kind regards
Sibylle

12

Hi @sibylles,

Thanks for your interest in modifying and adapting our models for a specific research question you’re interested in. Unfortunately the NatCap teams capacity is very limited to be able to do code reviews externally and get too into the weeds with anyone’s source code. We also are not able to provide video calls or other direct “on call” support. We just don’t have the bandwidth to provide that kind of service at large to the community.

Making guild tables, or biophysical parameters in general, spatially explicit to better allow for the type of work you’re attempting, is something we’ve heard a lot of people are interested in. We are looking into having InVEST handle this natively at some point in the future.

Around normalization, do you think you could clarify your specific question with a small toy example, in such a way that it’s a general question that may be applicable outside this specific context? That might help us better provide some help and then also allow you to pose / research your question on StackOverflow or elsewhere.

Thanks for your interest in InVEST and trying to utilize our models!

Doug

1 Like
13

Many thanks @dcdenu4
Happy to discuss the specific question about normalization of input data on the other thread.
I am very sorry to ask about code review and using @. I think there was a misunderstanding, as I got a message that natural capital is interested to review the code and as I learned to use @ in the way to specifically adress experts and not the whole group.
Kind regards
Sibylle