SDR New Version 3.12 takes too long to run pit_filled_dem

What is the issue or question you have?

I am running the new version of SDR (Version 3.12). It’s taking too long (days) to finish the pit_filled_dem raster.
I am getting the following type of info: “pygeoprocessing.routing.routing INFO (fill pits): 1849600 of 73990120 pixels complete”

What do you expect to happen?

I would like to know if I should wait for this run to finish or if I can be sure that it will give me an error / wrong results.

What have you tried so far?

I tried once with a DEM without fill sinks correction, which was really fast and got me all the results. Although, I thought the results were weird concerning avoided erosion and sediment export. Too many 0 or low values and I am unable to visualize a spatial difference unless I classify the raster by 10 quartiles.
I am now trying a new run with the corrected DEM (fill sinks) but, as I said, it’s taking days.
I would like to know if the first results are probably corrected or not. Also, I would like to know if it’s normal to take this long to run with a corrected DEM.

Attach the logfile here:

I am uploaded both log files. One for the run that was complete [09_19_10.txt] and one for the run that’s happening now (incomplete) [05–09_39_08.txt].

InVEST-natcap.invest.sdr.sdr-log-2022-10-05–09_19_10.txt (321.7 KB)

InVEST-natcap.invest.sdr.sdr-log-2022-10-05–09_39_08.txt (33.3 KB)

Thank you,
Julia

1 Like

Hi Julia -

I pretty much always fill sinks in DEMs before using them in the freshwater models, and have never had this particular problem. Would you mind sending the data you’re using, including both of the DEMs that you’ve tried? You can post a link here or send it to swolny@stanford.edu.

~ Stacie

1 Like

Hello Stacie. Thank you for your reply.
I will be sending them to you by email because I can’t make this data publicly available.

Hi Julia -

Thanks for sending your data. When I bring the filled DEM into ArcGIS it appears to have a minimum value of 6e-317, which is strange (especially since this does not happen in QGIS). But then when I calculate statistics and symbolize it, the minimum value changes to 0 (just like it shows in QGIS), which is very different than the minimum of 180m in the unfilled DEM. So it’s a little confusing why there’s so much difference between the two minimum values - how did you do the Fill operation?

I’m also a bit concerned about the difference between the filled and unfilled DEM values - here’s a map of the difference, and you can see a very distinct striped pattern to it, which is suspect.

How did you resample the unfilled DEM to this coordinate system? Did you use the interpolation method of Bilinear or Cubic? If you used Nearest Neighbor, it often creates odd checkerboard patterns, so it’s better to use Bilinear or Cubic. If this isn’t the case, I’m a little concerned about the quality of the original DEM data.

Another thing that’s unusual is that your pixel size is not square in any of your inputs, including the original DEMs. This gets resampled by the model to a square cell size of 27.1m, which you can see in the output rasters. I wanted to try using the QGIS Wang & Liu Fill tool (which generally works well and we recommend) on the unfilled DEM, which also resampled the pixels to be square, and produced a filled elevation range of 188m - 911m, which seems more in line with what I’d expect (as different than a new minimum value of 0, as is in your filled DEM).

Still, if I look at the difference between the Wang & Liu filled DEM and unfilled DEM, it has that striped pattern, at least in what I’m guessing are the more flat lands:

And, it’s dramatically changing the hydrology patterns from the original DEM, which I’ve not seen happen before with this tool.

When I use this filled DEM with square pixels in SDR, it runs in about the same time as your original unfilled DEM (actually a few minutes faster). However, the defined streams are pretty bad, and take up a lot of the watershed:

Is this area very flat? Flat areas can produce this kind of stream pattern in SDR. It’s definitely hard to get good drainage patterns in very flat places, I just went through this in a large wetland in Bolivia, where I had to try 3 different DEMs to get one that worked well enough.

So I’m not sure what to say. Personally, I’d go back to the original DEM, evaluate it closely to make sure it doesn’t have any odd striped artifacts to begin with, then check how you resampled it, including the interpolation method and resampling to a square pixel size. You might want to try a different DEM too to see if that changes anything.

~ Stacie

1 Like

Dear Stacy,
Thank you very much for the help. Actually, one of my coworkers changed the original DEM to include gullies. I do not know exactly how he did it. After this, another coworker filled the sinks. I think that is probably the problem. I will retry the analysis using the original DEM dataset and will report back to you. The cell size of 27.1m is strange, since the data I used to project all others should have 30m. I will take a look at this also.
Thank you very much.

Julia

Ah, that explains a lot! Are the gullies from agriculture, or something else? The SDR model actually has a separate optional input called Drainages, which might be useful in this case. You could keep the gullies separate from the DEM, and use them as the Drainages input. The model will treat them like the stream network, routing sediment until it hits either a stream or a gully. You can read more about this in the User Guide.

Let us know how it goes.

~ Stacie

1 Like

Yes I saw that. Yes, the gullies are from agriculture but it seems that they are old and even have relatively tall trees or bushes growing inside them. I have to think if that actually makes sense to use it as a drainage. Would it mean that the drainage areas export more sediment than others? Because that’s what we wanted to incorporate in the DEM.

Another and last question: does invest still have problems with NoData values? NaN, -9999 etc? Should I export all NA as -9999 or another value, or setting it to NA works fine? I remember reading something about a bug fix for this new version but I want to check it anyway.

Thank you again very much

SDR doesn’t include erosion happening within streams/drainages themselves, if that’s what you’re asking about, it only traces overland erosion. Since the overland erosion will be considered export once it hits a stream or gully, having gullies added to the landscape will usually mean that erosion will become export sooner, since it’s likely to encounter either a stream or gully more quickly. The USLE result should stay the same per pixel, but the sed_export and related results will often be different, likely higher, with drainages.

I’m actually not sure if NoData issues have been totally resolved, but in general I find it safest to assign NoData values no matter what I’m doing. The NoData value can be any number that is not being used for actual valid data, but it should explicitly be set.

~ Stacie

Hey Stacie and Julia,

I’ll just jump in and note that all InVEST models should handle all valid NoData types including NaN. In most cases we would consider it a bug in InVEST if a GDAL supported NoData value isn’t properly handled. Having said that, explicitly setting the NoData value to something that makes sense for the data seems like a good practice to me!

Best,

Doug