I am attaching the inputs again to seek your advice which would be much appreciated as I hope we are getting closer to successfully running the model despite the continuing challenges:
@swolny - any advice much appreciated.
Dear @swolny - please let me know if there are any other files I can provide to trial the SDR model.
I am attaching the log file here too:
InVEST-Sediment-Delivery-Ratio-Model-(SDR)-log-2021-08-25–13_02_03.txt (74.6 KB)
Thank you.
Hi @ndmetherall ,
My biophyscial table has fewer fields than the example - is this ok?
Yep, this is just fine. Sometimes with sample data we’ll have more columns than needed for any given model so that they can be used for a few models that share some of those fields.
A lot of your outputs look to have min / max values of a float 64 range:
>>> numpy.finfo(numpy.float64)
finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
The intermediate directory can be a great way to debug. I’d start by looking at:
pit_filled_dem.tif,
slope.tif,
slope_threshold.tif,
flow_direction.tif,
weighted_avg_aspect_path.tif,
flow_accumulation.tif,
ls.tif.
Let us know if something stands out!
Doug
Hey @ndmetherall, I’m looking at the inputs that you posted, and here’s what stands out:
So I’m guessing that a reprojection step didn’t work properly, since these all look like degree cell sizes, not meters. ArcGIS also gives an “inconsistent extent” error when I bring them into a map. I don’t know if this is causing the specific bad results that you’re seeing, but could be, and even if not it’s something that should be remedied.
Your K values are also much higher than they should be. I know that this layer has been a bane, but it looks like you still need to re-evaluate all of the input and output units. I have several of my own work projects pressing right now, and it is unprecedented for any of us to provide the level of extremely detailed forum support that this thread already contains, so please work that out for yourself, perhaps colleagues can help?
As for filling areas of missing soil data, if it’s somewhere that looks like, say, solid rock or perhaps lakes, where you don’t expect erosion, I think it’s ok to give a value of 0. Otherwise, I advise filling those areas in with the dominant value that surrounds the missing data.
~ Stacie
Will have a go at this - thanks @dcdenu4 - will keep my biophysical table and have a look at the intermediate directory. I will let you know if anything stands out.
I may also try to limit my area of interest to a smaller area - maybe just a few catchments instead of an entire island, as I notice that the gura example is only one catchment and a few sub-basins. Maybe I should try to work at a similar scale? Do you think this would help reduce risk of errors and help me a get a result?
Hi @swolny
Thanks for pointing out the areas that stand out.
Thank you for the unprecedented amount of detail and hours of support you have all given to this thread. I hope that some of the learnings we uncover will be useful for others encountering similar challenges.
I may also try to limit my area of interest to a smaller area - maybe just a few catchments instead of an entire island, as I notice that the gura example is only one catchment and a few sub-basins. Maybe I should try to work at a similar scale? Do you think this would help reduce risk of errors and help me a get a result?
Hey @ndmetherall,
If you end up getting the inputs all set up correctly I highly recommend reducing your study area. Often times reducing the problem to a base case or simpler scope can be a great way to sanity check. If the model reports reasonable result for a single island by passing in just a vector for that islands watersheds, then it tells a lot about what might be wrong when trying to scale to other islands. Likewise, if the model is still not reporting reasonable results then it makes debugging easier because you only need to focus on the data for the smaller region.
Cheers,
Doug
The Gura sample data is intentionally small, to make it easier for people to download, and quick to run. Relative to the areas that my variety of projects have covered over the years, it’s much smaller than pretty much all of them.
Looking at your whole area of interest, across all those islands, from just a resolution/raster size point of view, my experience is that it should work fine to do the whole area in one run. However, I’ve never run SDR on only a set of disconnected islands like this. I have run it on countries that include multiple islands offshore, and it has run fine in that context, which makes me think that it should run fine for you, once all of the inputs are formatted correctly.
Given all of the complexities that you’ve run into, it could be best to get the model working well on one island, then try running the whole area. If it runs fine on one island, but is problematic over the larger AOI, then we’ll know that the model is having trouble handling the separated land masses, and that would be interesting.
~ Stacie
Thank you @dcdenu4 and @swolny
I will try to follow your collective advice over the weekend and trial over a small area while also addressing the model inputs that we’re identified as having issues.
A quick question to check if you both prefer to use This, Arcmap or ArcGIS pro when generating inputs and visualising outputs of this SDR module/model? From my understanding you have to use qgis for the soil grids in wcs.
Please let me know what your preferences are.
Kind regards
I use ArcGIS (not Pro) most of the time, except when QGIS is better for whatever reason (like filling sinks, working with WCS…) This is really just personal preference, and using the tool I have the most experience with. Others use QGIS for everything, which is great, and I kinda wish I could cut the (dysfunctional, frustrating) cord with ArcGIS, but still find it to be better for raster symbology, and its support for raster attribute tables (which QGIS still does not) is a deal-breaker for me.
~ Stacie
Thanks for this info @swolny and @dcdenu4
I am trying some new datasets to see if this helps overcome any of the errors - these include the ESRI landcover layers. I have been following up on your advice this weekend and have the following questions:
However, the results of this have included some areas with 0 values. Even after filling sinks using the QGIS Wang&Liu tool. Any advice for this?
I am also seeking to confirm that after mosaicing and reprojecting / warping my DEM using cubic reprojection / warp > I can then use Wang&Liu or one of the other tools listed here?
And what about for autofill settings? Should I select do not autofill or fill with parameter values?
Furthermore, my DEM raster pixel size seems much larger than the LULC raster size. Any thoughts on this?
For the soil WCS just triple checking that I should always select option 3? i.e. mean content for each soil input to RUSLE.
For the landcover layer I have warp reprojected to UTM to get to (10, -10) pixel size and meters units. Does this seem to be ok?
@ndmetherall, for the USGS DEM, are those values 0, or are they NoData? If they’re NoData, that’s pretty bad. For smaller holes it’s possible to use interpolation or something to fill them (see the Working with the DEM section of the User Guide), but in this case I wouldn’t do that, there’s just too much missing and you’d be making a lot up.
Yes, you can use a Fill Sinks tool after mosaicking and reprojecting.
I’m not sure about the “specified no data value”. I’d try not assigning one, and see what the tool does, hopefully it will automatically assign an appropriate one based on the data type and values. If you set it to 0, that might be a valid elevation value, so you wouldn’t want it to be used for NoData.
No idea about “Autofill mode”, I don’t know what that does and have never used it, but probably wouldn’t allow it, since that indicates to me that it would just be making up data (maybe in gaps?), which isn’t good. It would be better to see the gaps and get real data that covers them.
For the DEM raster pixel size, what size is the original SRTM data? When you do the reprojection, you get to choose the pixel size/resolution. I tend to set that to a nice round number (like 30 or 90), based on the resolution of the original data. Something to note is that, for SDR, all of your other spatial inputs will be resampled to the same resolution as your DEM, so it’s good to consider how this might affect your LULC or other layers.
Yes, take the mean value for the ISRIC sand/silt/clay soil values.
~ Stacie
