Running SDR issue

Hello,

Thanks for your product and the useful documentation available. I wonder whether you could help me out with an issue I have stumbled upon whilst running the SDR model using my data - everything worked fine with the Gura catchment sample data.

I tried to run the model on a catchment which is about 1400kmq using input rasters with 10m pixel resolution. I first attempted to run the model at the whole catchment scale having the input files stored in different folders. Please find attached the log file (CATCHMENT_Invest xxx). I thought the model was working fine until it started the “Task._call(1234) INFO 0.0% complete” at 19:51:08. Regardless of the +10 minutes between one task call and the next one, nothing seemed to happen to the progression status of the task. This made me think something was potentially wrong. Furthermore, when I clicked on the program window, it froze and when I eventually tried to close it, it crashed. I thought the problem could be due to the size of my catchment.

Therefore, I then tried to run the model on a sub-catchment which is about 200kmq (always using input rasters with 10m pixel resolution). This time, I even stored all the input data in the same folder. The model ran quicker until it hit again the “Task._call(1234) INFO 0.0% complete” and did exactly what had done before. Please find attached the log file (SUBCATCHMENT_Invest xxx).

I can’t understand whether there is a problem at all or whether it simply takes a long to complete these “Task._call(1234) INFO”. However, given the fact this didn’t occur whilst using the Gura sample data, makes me think there is actually a problem with my input data? If so, which problem/s is likely to be?

With regards to my input data, the only two things I am not sure about are the DEM and Watersheds files. It says that the DEM should extend beyond the watersheds of interest, but mine is clipped to the watershed edge. Can this be the cause of the problem? About the Watersheds, instead of using a shapefile of polygons (sub-catchments), can I simply use the shapefile of the single watershed?

Thank you very much
Looking forward to hearing from you
All the best,
Massi

CATCHMENT_InVEST-Sediment-Delivery-Ratio-Model-(SDR)-log-2021-08-30–19_32_01.txt (23.1 KB)

SUBCATCHMENT InVEST-Sediment-Delivery-Ratio-Model-(SDR)-log-2021-08-31–10_47_45.txt (7.3 KB)

Hello @Massi88 , and welcome to the forums!

I’m glad to hear SDR is running well on the Gura sample data … that’s a good base case and so the model should be able to complete as expected.

In looking through your logfiles (thank you for including those!) and your notes about your study area, it looks like your inputs are quite large … on the order of about 20 billion pixels. This might take some time to compute, and it is possible that the progress logging might not indicate much progress (though I’d still expect there to be some progress).

That being said, the fact that the model is hanging makes me wonder if there’s an infinite loop in the model’s pitfilling operation. To make sure that isn’t the case, could you try filling your DEM’s hydrological sinks? We usually recommend the Wang & Liu operation within SAGA/QGIS, but I’m sure ArcGIS has something comparable. Once the DEM has been pitfilled, could you try running the model again?

If filling sinks doesn’t fix the issue, could you provide a link to all of your input datasets so we can try running the model on your inputs and see if we can reproduce the issue?

Thanks!
James

1 Like

That worked James - thank you very much!

I had filled my DEM’s hydrological sinks before but I used the r.fill.dir tool of QGIS. I didn’t know that you guys would recommend the Wang & Liu tool within SAGA (you may want to consider specifying this in the inVEST documentation? If it’s already there, my bad! I just hadn’t seen it.). Furthermore, I had no idea that a different depit/fill sinks tool would produce such a different output which is able to cause issues to the model…

Anyway - thanks a lot again! You’ve been ace!
Massi

2 Likes

Oh awesome, that’s great to hear that filling hydrological sinks did the trick! We’ll make sure that some notes about filling hydrological sinks are in the user’s guide.

Just so that we can make sure that this issue is fixed, would you be willing to share your DEM with us?

Thanks so much!
James

For what it’s worth, there is a section of the User Guide Introduction called Working with the DEM, which lists filling sinks as a processing step, including recommending Wang & Liu. We can reword this to provide more detail if needed, and/or point to that section within the model chapters where DEMs are used? I don’t think we do that currently.

~ Stacie

2 Likes

Hi Stacie,

Thank you for pointing this out. Honestly, I hadn’t seen that before (my bad). However, I agree that you could point to that section within the model chapters where DEMs are used/mentioned or even in the tutorial? I think it’d be easier for people to spot it and avoid making the same mistake I did.

Also, as I am reading the section now, would you mind clarifying to me what exactly you mean by “Multiple runs of Fill may be needed”? I’d have thought that once you run the fill sink tool once and obtain your filled dem that’d be it. Is this not the case? Do you mean that you might have to run again the fill tool this time using as input your dem that had already been filled once from the previous step (and so on for multiple times)?

Thanks
All the best,
Massi

1 Like

Hi James,

No worries, thanks to you!
Here’s the [link] (Sign in to your account) to a shared folder where you can find:

  • the original unfilled dem: “dem_metauro_new.tif”
  • the filled dem with Wang&Liu: “dem_filledwangliu_metauro_new.tif”
  • the filled dem with r.fill.dir of QGIS: “dem_filledsinksdir_metauro_new.tif”

I hope the link works and you are able to download the files otherwise, let me know. The access to the folder is open until Tuesday 7th Sept!

About the r.fill.dir tool of QGIS (QGIS 3.16.6 Hannover), I simply ran it leaving all the parameters as default:

  • output aspect direction format (optional): grass
    -GRASS GIS 7 region extent (optional): Not set
  • GRAS GIS 7 region cellsize: 0.0

It would be good to know what’s ‘wrong’ with the r.fill.dir tool of QGIS and why SDR doesn’t like it…

Thanks again
All the best,
Massi

2 Likes

Thank you! I’ve downloaded the DEM files and we’ll take a look and see what we can find.

As for the differences between Wang&Liu and r.fill.dir, I’m afraid I don’t know the algorithms well enough to be able to compare and contrast them … but I’m also not surprised that they would perform differently. Some day it could be neat to have a comparison of the various pitfilling algorithms and in what ways they perform well, underperform, and when each should be used. If we have some time, we might be able to look into them!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.