Hello there!
I ran the habitat quality model and all the data that was loaded is right, but when the process finishes I can saw that this show only the intermediate folder. I check the coordinate system and all the information is right, so I don’t know what else can cause the error.
I attached the Log and intermediate folder screenshot.
Thanks in advance, if you need something else, please let me know.
InVEST-Habitat-Quality-log-2022-03-22–21_22_41.txt (39.0 KB)
Thanks for providing your logfile @mfcnico ! This is an error that I haven’t seen for a long time. Could you share your inputs with me so I can try to reproduce it on my end? I bet a file sharing service like google drive or dropbox will be easiest since it looks like your files are probably pretty high-resolution.
Thanks,
James
Hi James, thanks for your answer, I changed the raster cell size and the model worked, but i would like to ask you if it is normal that it took about 6 hours?
Thanks for your help!
@mfcnico , the fact that you changed the cell size and the model worked after that makes me think that there’s a bug in our convolution that’s only going to happen on very large rasters. Would you be willing to send me your data so I can take a look at this anyways to make sure the model can run on your original set of inputs? My email is jdouglass@stanford.edu if it’s easier or preferable to send your inputs that way.
A lot of this model’s runtime is usually consumed by convolving your threat rasters within the given radius of influence, which takes a long time when pixel sizes are very small. I don’t know exactly how large your rasters are in terms of pixel dimensions, but 6 hours is not at all out of the question on large landscapes or high-resolution inputs.
One thing to note is that if any of your rasters are projected in a non-linear coordinate system (so, in degrees rather than meters), this can cause long runtimes due to a misinterpretation of units. So I’d make sure that all of your rasters are projected in meters just in case this is an issue here. I’d be very happy to take a look at your inputs to get a better idea of where speedups might be found, if any.
James
Hi James, I just sent the files to your email, thanks.
Hi @mfcnico , thanks for sharing your inputs!
In looking at your inputs and in revisiting your logfile from the original post in this topic, 6 hours of runtime sounds very reasonable for your landscape. As I mentioned before, this model’s runtime typically is consumed by the convolutions, which are decaying the influence of a stressor over a radius of MAX_DIST. So in your original run, it looks like you have pixels that are 1m x 1m in size that are decaying over large distances, MAX_DIST=5km in the case of threat amelh and MAX_DIST=10km in the case of ameind. Your landscape doesn’t cover that much area (the raster is about 7.9km by 5.5km), but because these convolutions are covering so many pixels, it’s just going to take a very long time: if the threat with 1m x 1m pixels is decaying over 5km, computing the entire raster requires visiting approximately ((2*5000)^2)*(7918 * 5574) = 4413493200000000, or about 4.413 quadrillion pixels all in all. If the threat has 100m x 100m pixels, decaying that threat would require visiting approximately ((2*500)^2)*(7918 * 5574) = 441 349 320 000 (441 billion, much more computable). There’s only so much we can do to make that run fast on desktop hardware and so increasing the size of your pixels will make your model run much more quickly, as you discovered.
One thing I would like to mention, though, is that if you do need to run Habitat Quality at higher resolutions, you can try adjusting the n_workers parameter in the InVEST settings to allow some of the convolutions to happen in parallel on separate processes. This is not a silver bullet, but can help compute certain problems more quickly.
But there is currently a limitation, as you originally posted about. That error:
2022-03-22 21:58:52,913 (pygeoprocessing.geoprocessing) geoprocessing.convolve_2d(2678) INFO starting convolve
2022-03-22 22:00:16,184 (pygeoprocessing.geoprocessing) geoprocessing.convolve_2d(2696) DEBUG start fill work queue thread
2022-03-22 22:00:16,185 (pygeoprocessing.geoprocessing) geoprocessing._fill_work_queue(2700) DEBUG fill work queue
2022-03-22 22:00:16,185 (pygeoprocessing.geoprocessing) geoprocessing.convolve_2d(2714) DEBUG start worker thread
2022-03-22 22:00:16,187 (pygeoprocessing.geoprocessing) geoprocessing.convolve_2d(2726) INFO 23206701680 sent to workers, wait for worker results
2022-03-22 22:01:16,195 (taskgraph.Task) Task.add_task(706) ERROR Something went wrong when adding task convolve_exponential_c_amelh (10), terminating taskgraph.
Traceback (most recent call last):
File "taskgraph\Task.py", line 674, in add_task
File "taskgraph\Task.py", line 1093, in _call
File "pygeoprocessing\geoprocessing.py", line 2730, in convolve_2d
File "queue.py", line 178, in get
_queue.Empty
2022-03-22 22:01:16,217 (natcap.invest.utils) utils.prepare_workspace(167) ERROR Exception while executing Habitat-Quality
Traceback (most recent call last):
File "natcap\invest\utils.py", line 165, in prepare_workspace
File "natcap\invest\ui\model.py", line 1649, in _logged_target
File "natcap\invest\habitat_quality.py", line 546, in execute
File "taskgraph\Task.py", line 674, in add_task
File "taskgraph\Task.py", line 1093, in _call
File "pygeoprocessing\geoprocessing.py", line 2730, in convolve_2d
File "queue.py", line 178, in get
_queue.Empty
This looks to be an issue with a fundamental assumption of our geoprocessing library, where it’s taking more than 60 seconds to compute statistics on the raster. This is a bug and we’ll address it in a later version of InVEST.
James
Thank you James for your time, your answer was very hepful, i hope you have a great rest of your day. 
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.