Debugging all nan in n export output

Has anyone got any tips for debugging why all my n export data are nan?

In the intermediate_outputs directory, I’ve looked at d_dn, dist_to_channel, ic_factor, load_n, ndr_n. Of these, ndr_n is all nan. Looking at the equations, this would seem to be why all the n exports are nan. But ic_factor looks valid and the right shape. The fact that valid data appears, in the right shape, in load_n, suggests the LULC raster and biophysical table is being read. Where could I look next to work out why ndr_n, and thus the exports, are nan?

ndr_n is calculated from two rasters, ic_factor and effective_retention_n. If ic_factor.tif looks normal, I’d expect that effective_retention_n.tif is the problem. Effective retention is calculated recursively upslope from drain points, so if you have a nodata gap in the wrong place, that can propagate upslope and affect a lot of the area.

If you’re interested to debug in more detail, you can check out the NDR source code. You can trace the flow of data through the model by following the add_task calls, which outline each step of the model.

Hmm, interesting. So whilst ic_factor does have nan, they’re plausibly where there’s a stream. effective_retention_n, however, looks riddled with nan. The nan look strongly associated with topography, almost like they’re on the northern slopes (but not consistently, I don’t think) or something.

Okay, so here’s what I know so far…

In my environment that has natcap-invest 3.14.0, gdal 3.6.4, and libgdal 3.7.0, and calling

    'workspace_dir': './local_data_test_1/',
    'dem_path': dem_path,
    'lulc_path': lucode_raster_path,
    'runoff_proxy_path': runoff_proxy_path,
    'watersheds_path': watersheds_path,
    'biophysical_table_path': biophysical_table_path,
    'calc_p': False,
    'calc_n': True,
    'results_suffix': f"test_1",
    'threshold_flow_accumulation': 1000,
    'k_param': 2,
    'subsurface_critical_length_n': 5,
    'subsurface_eff_n': 0.5,
    'n_workers': -1,

i.e. with n_workers set to -1 on @dcdenu4 's suggestion on this thread, I get

Warning 1: the input vector layer has a SRS, but the source raster dataset does not.
Cutline results may be incorrect.

which, with hindsight, is just a warning. I tracked this down to my runoff proxy raster missing a defined CRS. Adding that metadata, ndr.execute runs without error or warning in some 2.6 s. However, the watershed_results_ndr gpkg output has nan in the n export columns. These are the outputs that motivated this post, with ic_factor being produced but ndr_n not. On @esoth 's suggestion (above) I looked at the effective_retention_n intermediate output as well. I’ve now highlighted where these two outputs have nan:

The red boundaries are the subcatchments I’ve specified to be of interest. (The black boundary you can partially see is where I’d defined input LULC). Clearly the ic_factor has nan along the stream network, but also around the uppermost reaches around the perimeter of the watershed. The effective_retention_n intermediate, however, is riddled with nan. (NB these plots have 1 where data is nan, 0 otherwise).

Now I switch back to my environment that has natcap-invest 3.12.1, gdal 3.6.2, and libgdal 3.6.2 installed and run again. Now, instead of the warning about a source raster missing an SRS, I get

ERROR 1: ./local_data_test_1/intermediate_outputs/ic_factor_test_1.tif, band 1: Failed to compute statistics, no valid pixels found in sampling.

(NB this is when giving it input that is missing a defined SRS in an input raster).

So now “fixing” the missing SRS issue, I get the same ERROR 1 again after the model runs for 3+ minutes. Differences in output with this version include that d_dn, dist_to_channel, ic_factor, as well as ndr_n are all nan. The export columns in the watershed_results_ndr gpkg file are now zero rather than nan. The effective_retention_n intermediate output is also entirely nan now.

Whatever the issue with my inputs, they cause quite different degrees of problem between these two versions of invest. I think it’s time to beg that help from Doug and will package up my inputs, if that’s still okay. I’ve about run out of “ah ha, I think it will be … oh, it wasn’t that …”

ic_factor should not be missing data in streams either. The problem is almost certainly that one of your inputs is missing data in a critical place. If you upload your data I will look into it.

I’ll very gladly take another pair of eyes. My LULC raster should only have nan outside the region of interest, although when I create it I fill with a nodata value of -1 (and check the nodata metadata is set), but when I read it in again, I see the nodata represented as nan, I’m not sure why. But as I say, I believe this is outside the specified watershed/subcatchments.

The data are here.

Many thanks,

I believe the issue is that your DEM nodata value is nan, and an underlying pygeoprocessing function does not support nan nodata values. We’ll update this in a future release. In the mean time, you can try changing your DEM’s nodata value to a valid number. Be sure to both reclassify the nodata pixels from nan to the new value, and also update the nodata value in the raster metadata.

1 Like

Ah. That has got me further, thanks. I’d never dreamed the issue might lie in that data as I’d done a lot of experiments using delineateit.execute on exactly the same DEM file. Does delineateit not use that pygeoprocessing function that doesn’t support nan nodata values? NB This may be a red herring question; I’ve just revisited my work that used delineateit and realised this was using the earlier version of InVEST (3.10.12). I’ve discovered issues with version 3.14.0. I cannot get results out of 3.14.0. I’m working on distilling the issue down in writing and will post something about that.

But, basically, with 3.14.0 and my DEM that set nan as the nodata value, ndr.execute grinds its gears for a few minutes before spitting out that ERROR 1 about no valid pixels found.

So then resolving the nan nodata issue in the DEM (still running v. 3.14.0), it runs but produces the partial output I uploaded images for above. Also, this only runs at all if you set n_workers to -1. If you try something other than -1, you get:
ValueError: Could not open ./local_data_test_8/intermediate_outputs/aligned_runoff_proxy_test_8.tif as a gdal.OF_RASTER.

Then if I switch back to v 3.10.12, and using the DEM with finite nodata set, it not only produces the complete output I’d expect for my defined watersheds/subcatchments, but it also works for values of n_workers other than -1. Something very funky seems to be happening with v 3.14.0, at least with my data. I don’t see any issues relating to this on GitHub. I’m happy to start something, but not sure I currently have anything more useful than “it’s funky on my data”.

Hi @gtmaskall,

So then resolving the nan nodata issue in the DEM (still running v. 3.14.0 ), it runs but produces the partial output I uploaded images for above.

When I run your data via the InVEST Workbench (3.14.0) I’m seeing complete results. Could you speak more towards the “partial output” you’re seeing? I ended up defining nodata to -9999 for your shared DEM input but left everything else as is.

Left: ic_factor.tif | Right: effective_retention.tif

Also, this only runs at all if you set n_workers to -1 . If you try something other than -1 , you get:
ValueError: Could not open ./local_data_test_8/intermediate_outputs/aligned_runoff_proxy_test_8.tif as a gdal.OF_RASTER .

Thanks for reinforcing this difference and error. It looks like we did introduce a bug that will not allow NDR to be run with n_workers > 0. I made an issue for it here and we’ll patch it in the next release. Sorry about that, thanks for helping us catch it!



Let me know if it’d be helpful to share the modified DEM I used with nodata values defined as -9999 instead of nan. Like Emily mentioned above, we’ll have the handling of nan fixed in the next release.

Hi @dcdenu4 ,

So when I say “partial output”, if you look above to my figures where I showed missing values in ic_factor and effective_retention_n as 1 and 0 otherwise (so yellow is nodata), you’ll notice the effective_retention_n is riddled with missing values within the geometry (catchment) of interest, but in a way that’s clearly related to the DEM. My output n_*_export.tif file is similarly populated. In other words, output is calculated for some pixels but not others in a way that has some relationship to topography. I use VSCode these days so it’s easy to switch kernel/conda environment. Switching between versions 3.12.1 and 3.14.0 and rerunning the cell that calls ndr.execute with the same args (and deleting the workspace directory inbetween times), I get that incomplete output from version 3.14.0 but, lo, complete output with version 3.12.1.

I think I set my nodata to 9999 (it being higher than Everest) for my DEM. I’m on Linux and both those conda environments use python 3.10.12. Did you modify any pixels in the DEM or just the nodata metadata? I’ll happily run your modified DEM on my end and see if I still observe the same things.

Note, I talk about natcap-invest 3.10.12 in a few places above. I believe this is erroneous and should be 3.12.1. I was probably thinking of the Python version.

1 Like

@dcdenu4 might be able to share the dem he’s referring to, but in case you need to do this yourself, yes, you would need to both set the nodata value in the raster’s metadata and also change all of the nan pixel values to your new nodata value. If your nodata value is 9999, then you would need to do a raster calculater operation to convert all nan pixels to 9999 as well.

Yes, there aren’t any actual nodata pixels in that data anyway. My interest in trying Doug’s actual data file is that when I use a finite nodata value in the file metadata, I still only get incomplete output using 3.14.0 but complete output using 3.12.1. If I observe the same thing using the exact data file that worked for you, then this makes me wonder if the workbench does any preprocessing (I don’t use the workbench) or if some different version of an underlying library is doing something different.

The workbench makes a subprocess call to the packaged InVEST binaries for the entire model job … there is no preprocessing that’s done during model execution. So the most likely scenario here is that there’s a difference in an underlying library.

@dcdenu4 could you link to the DEM when you get a chance?

@gtmaskall I got the DEM @dcdenu4 was using, which can be downloaded here: filled_colne_dem_defined_nodata.tif - Google Drive


1 Like

Thanks for providing that, @jdouglass . I confirm I get the same problem when using your DEM as with mine. This is unique to my environment that runs natcap-invest 3.14.0.
As you can see, it’s riddled with missing output in a way that has some association with the elevation data.

I’ll add a full list of packages in this conda environment, but key packages I guess are:

  • python 3.10.12
  • pygeoprocessing 2.4.0
  • gdal 3.6.4
  • libgdal 3.7.0

Full output from conda list:

# packages in environment at /home/guy/anaconda3/envs/invest_geo_202309:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
affine                    2.4.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.5           py310h2372a71_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
annotated-types           0.5.0              pyhd8ed1ab_0    conda-forge
anyio                     4.0.0              pyhd8ed1ab_0    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.0              pyhd8ed1ab_0    conda-forge
async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
atomicwrites              1.4.1              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.7.3                he2921ad_3    conda-forge
aws-c-cal                 0.6.2                hc309b26_0    conda-forge
aws-c-common              0.9.0                hd590300_0    conda-forge
aws-c-compression         0.2.17               h4d4d85c_2    conda-forge
aws-c-event-stream        0.3.2                h2e3709c_0    conda-forge
aws-c-http                0.7.12               hc865f51_1    conda-forge
aws-c-io                  0.13.32              h1a03231_3    conda-forge
aws-c-mqtt                0.9.5                h3a0376c_1    conda-forge
aws-c-s3                  0.3.17               h1678ad6_0    conda-forge
aws-c-sdkutils            0.1.12               h4d4d85c_1    conda-forge
aws-checksums             0.1.17               h4d4d85c_1    conda-forge
aws-crt-cpp               0.23.1               hffbee3f_1    conda-forge
aws-sdk-cpp               1.11.156             h8bde0db_1    conda-forge
babel                     2.12.1             pyhd8ed1ab_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
blinker                   1.6.2                    pypi_0    pypi
blosc                     1.21.5               h0f2a231_0    conda-forge
bokeh                     3.2.2              pyhd8ed1ab_0    conda-forge
boost-cpp                 1.78.0               h6582d0a_3    conda-forge
boto3                     1.28.48            pyhd8ed1ab_0    conda-forge
botocore                  1.31.48            pyhd8ed1ab_0    conda-forge
branca                    0.6.0              pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                hd590300_0    conda-forge
brotli-bin                1.1.0                hd590300_0    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
c-blosc2                  2.10.2               hb4ffafa_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.1              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
cftime                    1.6.2           py310hde88566_1    conda-forge
chardet                   5.2.0                    pypi_0    pypi
charls                    2.4.2                h59595ed_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
contextily                1.3.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.0           py310hd41b1e2_0    conda-forge
cryptography              41.0.3          py310h75e40e8_0    conda-forge
curl                      8.3.0                hca28451_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    3.0.2                    pypi_0    pypi
cytoolz                   0.12.2          py310h2372a71_0    conda-forge
dask                      2023.9.1           pyhd8ed1ab_0    conda-forge
dask-core                 2023.9.1           pyhd8ed1ab_0    conda-forge
dask-geopandas            0.3.1              pyhd8ed1ab_1    conda-forge
dask-image                2023.8.1                 pypi_0    pypi
dask-labextension         7.0.0              pyhd8ed1ab_0    conda-forge
datacube                  1.8.9                    pypi_0    pypi
dav1d                     1.2.1                hd590300_0    conda-forge
debugpy                   1.8.0           py310hc6cd4ac_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecat                  2.1.1              pyhd8ed1ab_0    conda-forge
distributed               2023.9.1           pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
et_xmlfile                1.1.0              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fiona                     1.9.4           py310h111440e_0    conda-forge
flask                     2.3.3                    pypi_0    pypi
flask-cors                4.0.0                    pypi_0    pypi
folium                    0.14.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.42.1          py310h2372a71_0    conda-forge
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
frozenlist                1.4.0           py310h2372a71_0    conda-forge
fsspec                    2023.9.0           pyh1a96a4e_0    conda-forge
gdal                      3.6.4                    pypi_0    pypi
geoalchemy2               0.14.1             pyhd8ed1ab_0    conda-forge
geocube                   0.4.2              pyhd8ed1ab_1    conda-forge
geographiclib             1.52               pyhd8ed1ab_0    conda-forge
geopandas                 0.14.0             pyhd8ed1ab_0    conda-forge
geopandas-base            0.14.0             pyha770c72_0    conda-forge
geopy                     2.4.0              pyhd8ed1ab_0    conda-forge
geos                      3.11.2               hcb278e6_0    conda-forge
geotiff                   1.7.1               h22adcc9_11    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
greenlet                  2.0.2           py310hc6cd4ac_1    conda-forge
hdf4                      4.2.15               h501b40f_6    conda-forge
hdf5                      1.14.1          nompi_h4f84152_100    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagecodecs               2023.9.4        py310hedc1555_0    conda-forge
imageio                   2.31.1             pyh24c5eb1_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.0.1              pyhd8ed1ab_0    conda-forge
ipykernel                 6.25.2             pyh2140261_0    conda-forge
ipyleaflet                0.17.4                   pypi_0    pypi
ipython                   8.15.0             pyh0d859eb_0    conda-forge
ipywidgets                8.1.1                    pypi_0    pypi
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.19.0             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
js2py                     0.74               pyhd8ed1ab_0    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310hff52083_2    conda-forge
jsonschema                4.19.0             pyhd8ed1ab_1    conda-forge
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.19.0             pyhd8ed1ab_1    conda-forge
jupyter-lsp               2.2.0              pyhd8ed1ab_0    conda-forge
jupyter-server-proxy      4.0.0              pyhd8ed1ab_0    conda-forge
jupyter-ui-poll           0.2.2                    pypi_0    pypi
jupyter_client            8.3.1              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.1           py310hff52083_0    conda-forge
jupyter_events            0.7.0              pyhd8ed1ab_2    conda-forge
jupyter_server            2.7.3              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyterlab                4.0.6              pyhd8ed1ab_0    conda-forge
jupyterlab-widgets        3.0.9                    pypi_0    pypi
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.25.0             pyhd8ed1ab_0    conda-forge
jupyterlab_vim            4.0.3              pyhd8ed1ab_0    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
kealib                    1.5.1                h3e6883b_4    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py310hd41b1e2_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lark                      1.1.7              pyhd8ed1ab_0    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230802.0      cxx17_h59595ed_3    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libarrow                  13.0.0           h1935d02_4_cpu    conda-forge
libavif16                 1.0.1                h014f275_1    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_0    conda-forge
libbrotlidec              1.1.0                hd590300_0    conda-forge
libbrotlienc              1.1.0                hd590300_0    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.3.0                hca28451_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_0    conda-forge
libgdal                   3.7.0                h4a547c6_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_0    conda-forge
libgfortran5              13.2.0               ha4646dd_0    conda-forge
libglib                   2.78.0               hebfc3b9_0    conda-forge
libgomp                   13.2.0               h807b86a_0    conda-forge
libgoogle-cloud           2.12.0               h8d7e28b_2    conda-forge
libgrpc                   1.57.0               ha4d0f93_1    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo                 hd590300_1    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           18_linux64_openblas    conda-forge
libnetcdf                 4.9.2           nompi_he09a3a9_107    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.4                 hfc447b1_0    conda-forge
libprotobuf               4.23.4               hf27288f_6    conda-forge
librttopo                 1.1.0               h0d5128d_13    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
libspatialite             5.0.1               hca56755_27    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_0    conda-forge
libthrift                 0.19.0               h8fd135c_0    conda-forge
libtiff                   4.5.1                h8b53f26_1    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.5               h0d562d8_0    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mapclassify               2.5.0              pyhd8ed1ab_1    conda-forge
markupsafe                2.1.3           py310h2372a71_0    conda-forge
matplotlib-base           3.7.1           py310he60537e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mercantile                1.2.1              pyhd8ed1ab_0    conda-forge
mistune                   3.0.1              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
multidict                 6.0.4           py310h1fa729e_0    conda-forge
munch                     4.0.0              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
natcap-invest             3.14.0                   pypi_0    pypi
nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.8.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.4           nompi_py310h6f5dce6_101    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.92                 h1d7d5a4_0    conda-forge
numexpr                   2.8.6                    pypi_0    pypi
numpy                     1.25.2          py310ha4c1d20_0    conda-forge
odc-algo                  0.2.3                    pypi_0    pypi
odc-geo                   0.4.1              pyhd8ed1ab_0    conda-forge
odc-stac                  0.3.6                    pypi_0    pypi
odc-ui                    0.2.0a3                  pypi_0    pypi
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openpyxl                  3.1.2           py310h2372a71_0    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
orc                       1.9.0                h52d3b3c_2    conda-forge
overrides                 7.4.0              pyhd8ed1ab_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.1.0           py310hcc13569_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.0              pyhd8ed1ab_1    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.0.0          py310h582fbeb_0    conda-forge
pims                      0.6.1                    pypi_0    pypi
pint                      0.22                     pypi_0    pypi
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
planetary-computer        1.0.0              pyhd8ed1ab_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
poppler                   23.05.0              hd18248d_1    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                15.4                 h8972f4a_0    conda-forge
proj                      9.2.1                ha643af7_0    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py310h1fa729e_0    conda-forge
psycopg2                  2.9.7           py310h76c1b15_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyarrow                   13.0.0          py310hf9e7431_4_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  2.3.0              pyhd8ed1ab_0    conda-forge
pydantic-core             2.6.3           py310hcb5633a_0    conda-forge
pygeoprocessing           2.4.0                    pypi_0    pypi
pygeos                    0.14            py310h056c13c_1    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyjsparser                2.7.1              pyh8c360ce_0    conda-forge
pyogrio                   0.6.0           py310h111440e_1    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.0           py310h24ef57a_1    conda-forge
pyro4                     4.77                     pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pystac                    1.8.3              pyhd8ed1ab_0    conda-forge
pystac-client             0.7.5              pyhd8ed1ab_0    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-dotenv             1.0.0              pyhd8ed1ab_1    conda-forge
python-fastjsonschema     2.18.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1           py310h0a54255_0    conda-forge
pyyaml                    6.0.1           py310h2372a71_0    conda-forge
pyzmq                     25.1.1          py310h5bbb5d0_0    conda-forge
rasterio                  1.3.8           py310hd227816_0    conda-forge
rav1e                     0.6.6                he8a937b_2    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2             pyhd8ed1ab_0    conda-forge
regex                     2023.8.8        py310h2372a71_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
retrying                  1.3.4                    pypi_0    pypi
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rioxarray                 0.15.0             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.3          py310hcb5633a_0    conda-forge
rtree                     1.0.1           py310hbdcdc62_2    conda-forge
ruamel.yaml               0.17.32         py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
s2n                       1.3.51               h06160fa_0    conda-forge
s3transfer                0.6.2              pyhd8ed1ab_0    conda-forge
scikit-image              0.21.0          py310hc6cd4ac_0    conda-forge
scikit-learn              1.3.0           py310hf7d194e_0    conda-forge
scipy                     1.11.2          py310hb13e2d6_1    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.2              pyh41d4057_0    conda-forge
serpent                   1.41                     pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   1.8.1.post1              pypi_0    pypi
simpervisor               1.0.0              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slicerator                1.1.0                    pypi_0    pypi
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sqlalchemy                1.4.49          py310h2372a71_0    conda-forge
sqlite                    3.43.0               h2c6b66d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.0          py310h278f3c1_1    conda-forge
svt-av1                   1.7.0                h59595ed_0    conda-forge
taskgraph                 0.11.0                   pypi_0    pypi
tblib                     2.0.0              pyhd8ed1ab_0    conda-forge
terminado                 0.17.1             pyh41d4057_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tifffile                  2023.8.30          pyhd8ed1ab_0    conda-forge
tiledb                    2.13.2               hd532e3d_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py310h2372a71_0    conda-forge
traitlets                 5.10.0             pyhd8ed1ab_0    conda-forge
traittypes                0.2.1                    pypi_0    pypi
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
tzcode                    2023c                h0b41bf4_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
tzlocal                   5.0.1           py310hff52083_0    conda-forge
ucx                       1.14.1               h64cca9d_4    conda-forge
unicodedata2              15.0.0          py310h5764c6d_0    conda-forge
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
webcolors                 1.13               pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.6.3              pyhd8ed1ab_0    conda-forge
werkzeug                  2.3.7                    pypi_0    pypi
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.9                    pypi_0    pypi
wrapt                     1.15.0          py310h1fa729e_0    conda-forge
xarray                    2023.8.0           pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.4                h8d71039_2    conda-forge
xlrd                      2.0.1                    pypi_0    pypi
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xyzservices               2023.7.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.2           py310h2372a71_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zfp                       1.0.0                h27087fc_3    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zlib-ng                   2.0.7                h0b41bf4_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

I can clone the environment and upgrade (or otherwise change the version of) something. Any suggestions?


HI @gtmaskall,

Sorry for “stepping out” on this thread and thanks to James for filling in! This is pretty interesting. Tomorrow I’ll try and set up an environment close to what you shared and see if I can reproduce using the data you shared previously.

Are you capturing the model logging at all or could you capture the stdout and share it? I wonder if there is anything being reported in there that could be useful.

Could you also try downloading the InVEST Workbench and confirming your data does run okay there too?