Seasonal Water Yield running error raster

Dear all,
I am new to Invest model, I tried to install the source code to run in an IDE, but without success for now. As suggested in the github issue tracker I was able to install something and create a script for the model I am interested in: seasonal water yield.
I would like to test it and therefore to be able to debug it, is there a possibility to debug it?

At the present moment I have an error due, I think, on GDAL installation, here is the error:
ValueError: Could not open D:\development\invest-sample-data\Seasonal_Water_Yield as a gdal.OF_RASTER

The script I am trying to run is the following:

# coding=UTF-8
# -----------------------------------------------
# Generated by InVEST 3.9.0.post282+g36cc3d2e on Tue Mar  9 15:49:28 2021
# Model: Seasonal Water Yield

import logging
import sys

import natcap.invest.seasonal_water_yield.seasonal_water_yield
import natcap.invest.utils

LOGGER = logging.getLogger(__name__)
root_logger = logging.getLogger()

handler = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter(
    fmt=natcap.invest.utils.LOG_FMT,
    datefmt='%m/%d/%Y %H:%M:%S ')
handler.setFormatter(formatter)
logging.basicConfig(level=logging.INFO, handlers=[handler])

args = {
    'workspace_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\output',
#    'results_suffix': '',
    'threshold_flow_accumulation': '1000',
    'et0_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\ET0_monthly',
    'precip_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\Precipitation_monthly',
    'dem_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield',
    'lulc_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\land_use_gura.tif',
    'soil_group_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\soil_group_gura.tif',
    'aoi_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\watershed_gura.shp',
    'biophysical_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\\biophysical_table_gura_SWY.csv',
    'rain_events_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\\rain_events_gura.csv',
    'alpha_m': '1/12',
    'beta_i': '1',
    'gamma': '1',
    'user_defined_local_recharge': 'False',
#    'l_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\',
    'user_defined_climate_zones': 'True',
    'climate_zone_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\climate_zone_table_gura.csv',
    'climate_zone_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\climate_zones_gura.tif',
    'monthly_alpha': 'False',
    'monthly_alpha_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\monthly_alpha.csv',
    'n_workers': '1',
}

if __name__ == '__main__':
    natcap.invest.seasonal_water_yield.seasonal_water_yield.execute(args)

I am absolutely not used with python so I really do not understand how to fix this error. I installed the invest model with all the dependencies before. I had a problem with GDAL but it seemed to me that it was solved…

Can anyone help me solving this error and trying to debug the script?

Thanks in advance for the help.

Regards

Silvia

1 Like

Hi @silli,
Thanks for trying out the python API! Your script looks mostly good.

The error ValueError: Could not open <path> as a gdal.OF_RASTER can happen for two reasons when it tries to read a raster file:

  1. The path exists but it isn’t a valid raster file
  2. The path doesn’t exist

Your path D:\development\invest-sample-data\Seasonal_Water_Yield probably exists, but it is a directory and not a file, so it can’t be opened as a raster. It looks like your args dem_raster_path and l_path are pointing to this directory when they should point to an appropriate raster.

Another minor thing is I would remove the quotes around your numeric arg values: threshold_flow_accumulation, alpha_m, beta_i, gamma, n_workers; and your boolean arg values: user_defined_local_recharge, user_defined_climate_zones, monthly_alpha. The quotes turn them all into strings which can cause problems later: the string 'False' doesn’t behave the same as the boolean False, and you can’t do math with the string '1' like you can with the number 1. I hope that makes sense! Let me know if you still have troubles.

Thank you @esoth for your reply.
In the meanwhile I did some more steps and now I am stuck in an other part of the code. I am trying to run this script:

import logging
import sys

import natcap.invest.seasonal_water_yield.seasonal_water_yield
import natcap.invest.utils

LOGGER = logging.getLogger(__name__)
root_logger = logging.getLogger()

handler = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter(
    fmt=natcap.invest.utils.LOG_FMT,
    datefmt='%m/%d/%Y %H:%M:%S ')
handler.setFormatter(formatter)
logging.basicConfig(level=logging.INFO, handlers=[handler])

args = {
    'workspace_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\output',
    'results_suffix': '_t01',
    'threshold_flow_accumulation': 1000,
    'et0_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\ET0_monthly',
    'precip_dir': 'D:\development\invest-sample-data\Seasonal_Water_Yield\Precipitation_monthly',
    'dem_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\DEM_gura.tif',
    'lulc_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\land_use_gura.tif',
    'soil_group_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\soil_group_gura.tif',
    'aoi_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\watershed_gura.shp',
    'biophysical_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\\biophysical_table_gura_SWY.csv',
    'rain_events_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\\rain_events_gura.csv',
    'alpha_m': 1/12,
    'beta_i': 1,
    'gamma': 1,
    'user_defined_local_recharge': 'False',
    'l_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\local_recharge_gura1.tif',
    'user_defined_climate_zones': 'True',
    'climate_zone_table_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\climate_zone_table_gura.csv',
    'climate_zone_raster_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\climate_zones_gura.tif',
    'monthly_alpha': 'False',
    'monthly_alpha_path': 'D:\development\invest-sample-data\Seasonal_Water_Yield\monthly_alpha.csv',
    'n_workers': 1
}

if __name__ == '__main__':
    natcap.invest.seasonal_water_yield.seasonal_water_yield.execute(args)

I have two problems:

  • if I do not insert the l_path it gives an error looking for it even if the user_defined_local_recharge is set to False, so I did a mask of the watershed with all 1 (and also a test with 0) inside. I don’t know if this is the correct format of the data required as local recharge
  • running this script with

.\invest.exe run seasonal_water_yield

I have the following error:

Traceback (most recent call last):
File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.752.0_x64__qbz5n2kfra8p0\lib\runpy.py”, line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.752.0_x64__qbz5n2kfra8p0\lib\runpy.py”, line 87, in run_code
exec(code, run_globals)
File "C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts\invest.exe_main
.py", line 7, in
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\cli.py”, line 619, in main
model_form = getattr(module, classname)()
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\seasonal_water_yield.py”, line 9, in init
model.InVESTModel.init(
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\model.py”, line 1171, in init
self.label = label
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\model.py”, line 1485, in setattr
object.setattr(self, name, value)
TypeError: can’t apply this setattr to SeasonalWaterYield object
PS C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts> .\invest.exe run seasonal_water_yield
Traceback (most recent call last):
File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.752.0_x64__qbz5n2kfra8p0\lib\runpy.py”, line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.752.0_x64__qbz5n2kfra8p0\lib\runpy.py”, line 87, in run_code
exec(code, run_globals)
File "C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts\invest.exe_main
.py", line 7, in
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\cli.py”, line 619, in main
model_form = getattr(module, classname)()
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\seasonal_water_yield.py”, line 9, in init
model.InVESTModel.init(
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\model.py”, line 1171, in init
self.label = label
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\ui\model.py”, line 1485, in setattr
object.setattr(self, name, value)
TypeError: can’t apply this setattr to SeasonalWaterYield object

Any idea on what is the problem?

Is it possible for me at this point to debug the code? how can I do using this script in an IDE? I downloaded all the source code and have a project in LiClipse but I am not able to run/debug it from there because there isn’t any main or exec or something else to start with…

Thanks for any help…

Silvia

Hi @silli,
For your first question, you should not need to set the l_path when user_defined_local_recharge is False. It looks like you still have quotes around your False and True values. This is causing the error because 'False' is not the same as False. Since 'False' is a non-empty string, it actually behaves like True when we try to check if args['user_defined_local_recharge']... If you want to know more about why this is, check out this article. If you remove the quotes around 'False', you should not need to include the l_path.

For your second question about the TypeError: can’t apply this setattr to SeasonalWaterYield object: This is probably an incompatibility with python 3.9. We have only tested versions through 3.8. I recommend you install a 3.8 version and run it in that environment.

Let me know if you still get the error after that!

Thank you @esoth,
I did the change in the script for the quotes around True and False values and commented the l_path line and the script starts.
It seems that it does something more than before but finally it stops with this error:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “D:\development\seasonalwateryield.py”, line 48, in
natcap.invest.seasonal_water_yield.seasonal_water_yield.execute(args)
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\seasonal_water_yield\seasonal_water_yield.py”, line 377, in execute
_execute(args)
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\seasonal_water_yield\seasonal_water_yield.py”, line 786, in _execute
task_graph.join()
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\taskgraph\Task.py”, line 859, in join
timedout = not task.join(timeout)
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\taskgraph\Task.py”, line 1399, in join
raise self.exception_object
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\taskgraph\Task.py”, line 528, in _task_executor
task._call()
File “C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\taskgraph\Task.py”, line 1231, in _call
payload = result.get()
File “C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.752.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py”, line 771, in get
raise self._value
ValueError: GEOSGeom_createLinearRing_r returned a NULL pointer

What do you think?

If I will install python3.8 should I also redo the installation of all the dependencies in it and then also the installation of the Invest python package in it or can I use this version?

Sorry for my questions but I am really new in this.

Is there a way to debug this executable in an IDE?

Thanks again for the support!

Hi @silli,
That error isn’t actually the original error, it’s an incomplete stacktrace and I would need to see the entire log to debug it.

If I will install python3.8 should I also redo the installation of all the dependencies in it and then also the installation of the Invest python package in it or can I use this version?

You should not need to reinstall everything.

Sorry for my questions but I am really new in this.

No worries! It’s very confusing at first. If you haven’t already, I recommend learning to use a tool like conda, which makes it easier to keep track of your python and dependency versions.

Is there a way to debug this executable in an IDE?

You can try placing breakpoints in the source code, which you can see in the traceback is located at C:\Users\hydrologis\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\natcap\invest\. But since you’re not modifying the source code, it shouldn’t be necessary.

Dear @esoth,
first of all thanks again for the quick replay and sorry for my late reaction. I am a bit depressed on this stuff because it seems to me that I will never have something working. I would like to have installed all in “native” windows, avoiding Conda which, I know, will end in not using the tools.
I tried to run the code with Python 3.8, without doing anything else in this environment and got this error:

C:\Users\hydrologis\AppData\Local\Programs\Python\Python38\python.exe .\seasonalwateryield.py
Traceback (most recent call last):
File “.\seasonalwateryield.py”, line 9, in
import natcap.invest.seasonal_water_yield.seasonal_water_yield
ModuleNotFoundError: No module named ‘natcap’

As far as I can understand I should install all the dependencies with python38, right? but how can I choose the compiler if I can use only pip? I really can not understand…
Should I remove pyhton39 and redo all with python38 which will become the main version of python on my machine?

Which log do you need to understand the error? where is it located?
Attached you can find the complete Console log, hope this will be more clear…

Thanks again,

Silvia
invest_log_swy.txt (53.5 KB)

Dear @esoth I tried to install and use python3.8, but still I have some issues, could you please help me in understanding what is going wrong?

Attached the console log of the running with python3.8.

Any help is appreciated.

Silvia
invest_log38_swy.txt (34.4 KB)

Dear @esoth, this is just to tell you that I solved the problem and now I can run SWY from the console and also inside my IDE.

The problem was the installation of the package shapely which should have been done from the same whl package as GDAL, the version in pip isn’t compatible with the GDAL version in whl.

Thank you very much for the help.

Silvia

2 Likes

Hey @silli ,

Great to hear! If you ever get bored and want to share a synthesized how-to on how you got your setup working with an IDE I bet it would be a great community resource!

Cheers,

Doug