Seasonal Water Yield and 8 bit signed rasters

Hey y’all -

It looks like InVEST is still having problems with 8 bit signed rasters in version 3.8.9. This time we’re using Seasonal Water Yield and running into two related issues.

1/ The climate zone raster is 8 bit signed, with a NoData value of -128, which led to this error:

ValueError: The following 1 raster values [128] from “D:/Colombia/Golfo_de_Morrosquillo/output_CV_tmp\cache_dir\cz_aligned_t1.tif” do not have corresponding entries in the value_map: {11: 10.1, 12: 10.0, 14: 3.7}

  • Adding 128 to the climate zone rain events table (with rain event values of 0) got me past that error.

2/ Next came this one:

File “site-packages\natcap\invest\seasonal_water_yield\seasonal_water_yield.py”, line 1039, in _calculate_curve_number_raster
File “site-packages\pygeoprocessing\geoprocessing.py”, line 424, in raster_calculator
File “site-packages\natcap\invest\seasonal_water_yield\seasonal_water_yield.py”, line 1029, in cn_op
KeyError: 128

This seems to be caused by the soil group raster, which also is type 8 bit signed with NoData = -128. I tried adding 128 to the biophysical table, but got the same error. So converted the soil group raster to 32 bit signed, with a NoData value of -2147483647 and it ran fine.

Attached are both of the related log files, I can provide the data if it’s useful. Thanks!

InVEST-Seasonal-Water-Yield-log-2020-10-19–13_01_31.txt (9.8 KB)
InVEST-Seasonal-Water-Yield-log-2020-10-19–13_27_45.txt (11.5 KB)

~ Stacie

1 Like

Hey @swolny,

Is it possible to share this data so I can replicate and work through these bugs you’re seeing? I can’t help but feel that we recently made fixes to some of these, but obviously not!

Thanks!

Doug

Hey @swolny,

Thanks for sharing the data. One thing that I just double checked was the fact that GDAL Byte is unsigned (values 0 - 255). Looking at the soils and climate raster in QGIS, I see that they are both Byte - Eight bit unsigned integer and that nodata is not defined. pygeoprocessing shows nodata as set to -128 and QGIS shows the +128 values that should be negative nodata in the rasters. So it seems like this is where a lot of the mix up is happening. The GDAL dataset ( or datasource ) is happy to set nodata as -128, but when writing that data, it transforms it to +128 (I’m guessing).

In an interesting turn of events I ran the data on both the 3.8.9 and development 3.9 build only to find that the 3.9 dev build completes without fault. I’m going to spend a little time figuring out why that’s the case.

BUT, as a work around for the problems you’re seeing, if you define those raster to have nodata values that are in the 0 - 255 range you should be good. You can also change the raster types to int16 or another signed type.

Cheers,

Doug

1 Like

I’m also just realizing you already got to the work around part and were just informing us of the 8 bit raster type issue! So, perhaps it’s not an issue at all since GDAL Byte is unsigned? Though, I am curious how our dev 3.9 build is seemingly handling the issue.

Update: @swolny
Just to follow up, it looks like Pygeoprocessing 2.0 fixed the unsigned / signed byte issue. This will be reflected in the 3.9 release, currently 3.8.9 uses Pygeoprocessing < 2.0.