I am successfully running the NDR model through the GUI interface. However, I would like to run it via a python script or via the command line to better facilitate batch processing and model calibration. When I run NDR on the same inputs as I use in the GUI interface (where the model runs successfully), I receive the following python error messages:
“Warning 1: A geometry of type MULTIPOLYGON is inserted into layer WARRIV_SUBW_16JUL10 of geometry type POLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.
python(88753,0x103e61e00) malloc: *** error for object 0x7ffeed5538c8: pointer being freed was not allocated
python(88753,0x103e61e00) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort python Scripts/InVEST-scripts/run-invest-NDR-batch-CJ.py
(invest3102V3) carolinejaffe@carolines-mbp UROP_Share_Folder_Dropbox % /Users/carolinejaffe/opt/anaconda3/envs/invest3102V3/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 22 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '”
Here is the code I’m using in my Python script:
output_dir = destination_folder + '/' + lulc_path.stem
dem_path = str(p) + '/Data_Inputs/NDR_INVEST_INPUTS/DEM-EPSG-26986.tif'
runoff_proxy_path = str(p) + '/Data_Inputs/NDR_INVEST_INPUTS/DEM-EPSG-26986.tif'
watersheds_path = str(p) + '/Data_Inputs/Helper_Data_Folders/Watersheds/NDR-calibration-watershed/WARRIV_SUBW_16JUL10.shp' # NDR_INVEST_INPUTS/MA-watersheds-with-cran-bogs-EPSG-26986.gpkg'
biophysical_table_path = str(p) + '/Data_Inputs/NDR_INVEST_INPUTS/biophysical_table_NDR_v3.csv'
# make sure all the paths exist
assert Path(dem_path).exists(), f'{dem_path} does not exist'
assert Path(runoff_proxy_path).exists(), f'{runoff_proxy_path} does not exist'
assert Path(watersheds_path).exists(), f'{watersheds_path} does not exist'
assert Path(biophysical_table_path).exists(), f'{biophysical_table_path} does not exist'
args['workspace_dir'] = str(output_dir)
args['results_suffix'] = str(lulc_path.stem)
args['dem_path'] = dem_path
args['lulc_path'] = str(lulc_path)
args['runoff_proxy_path'] = str(runoff_proxy_path)
args['watersheds_path'] = str(watersheds_path)
args['biophysical_table_path'] = str(biophysical_table_path)
args['calc_p'] = False
args['calc_n'] = True
args['threshold_flow_accumulation'] = 1000
args['k_param'] = 2
args['subsurface_critical_length_n'] = 150
args['subsurface_eff_n'] = .05
args['n_workers'] = -1
ndr.execute(args)
For what it’s worth, I am able to run the Carbon Sequestration model via a similar python script. I am using InVEST 3.10.2 in both the GUI and in the CLI/ python module environments, and tried running these scripts with Python 3.8.5 and Python 3.9.7. I will attach the successful GUI log file here as well.
Happy to provide more information and would be grateful if anyone had ideas about why I’m encountering this error!
Well, I’ll try getting the easy warning out of the way:
This warning can be addressed by converting your watersheds geometry to a POLYGON type, and it’s possible that this might address the malloc error as well.
These errors are about as low-level as you can get with python, and the fact that it works with the GUI interface but not with your conda python environment makes me suspect that this is an issue with the environment. Could you share the set of commands you used (if you have them handy) for how you created your conda environment and installed natcap.invest into it?
For what it’s worth, we use conda-forge for our binary build processes, and so you might also try building a new conda environment and seeing if that does the trick:
Hi James! Thank you for your response - I’m really impressed with the level of support and community you provide for InVEST users - unusual for an open source software project coming from academia!
I did convert my Multipolygon shapefile to Polygon, which got rid of that warning (great!), but didn’t fix the malloc error. I then used your suggestion to create a new conda environment and that did work - awesome! I am still getting a user warning about leaked memory objects at the end of my script, see below, but it’s not stopping the script from running.
UserWarning: resource_tracker: There appear to be 22 leaked shared_memory objects to clean up at shutdown
FWIW, I used the suggestions from the InVEST documentation to get my environment going initially, as follows:
However, it’s possible I also inadvertently did something else to alter the environment (while trying to fix it!) that caused the error. Thank you for your help and I’ll let you know if I have any other issues!
Yes, that’s a known bug that we’ll fix in an upcoming release! I’ve been seeing it myself for some time, but since it hasn’t been causing an error or anything like that, we’ve yet to resolve it.
Thanks for letting us know! We’ll take a look and see if there’s anything we can do to improve our documentation for this sort of environment setup.
I’ll mark this thread as resolved, but please feel free to write back or open a new topic if anything else comes up!
