Seasonal water yield, issue with quickflow/stream map

[lukezw]

Hello all,

I’m still getting to grips with hydrological modelling in InVEST. After running the seasonal water yield model, I am getting some strange results for the quickflow and stream.tif validation file (the latter is shown below), along with a land cover map for context.

As you can see, the model is predicting large jagged areas of streams in various places. These often seem to be around where lakes exist in real life (see the land cover map for comparison), but go beyond the lake areas too. What could I do to fix this? Is it something to do with the flow accumulation threshold? So far I have just been keeping this around the recommended starting value of 1000. This seems to match the real stream distribution pretty well, until one gets to those large jagged ‘blobs’.

A whole bunch of LZW decode warning appeared as the model was running, however it still completed fine to my surprise. I am also attaching a second log file from a previous run conducted in the same region of the world but for a smaller area. This showed the same issue with the quickflow/stream.tif layers, so it doesn’t seem to be related to the LZW decode issues. The reason I’m adding its log file is because it doesn’t have hundreds of lines with LZW decode warnings, so might be easier to look through!

InVEST-Seasonal-Water-Yield-log-2020-07-27--18_15_52.txt

InVEST-Seasonal-Water-Yield-log-2020-07-27–14_02_23.txt (63.4 KB)

Any pointers would be much appreciated.

Luke

[swolny]

Hi @lukezw -

You’re seeing a common artifact of delineating streams in flat places, which happens to me quite frequently, especially with the Multiple Flow Direction (MFD) algorithm that is used by the InVEST freshwater models (on the flip side, it’s also likely to be more correct in terms of water routing in multiple directions from a pixel…) It’s generally the case that defining streams in flat places is just hard for algorithms to do, and pretty much every flow direction algorithm produces streams that are funny in some way in flat areas.

Most of the options I’ve tried so far for working around this involve manipulating the DEM to force a particular stream/lake network. One way is to try the InVEST tool RouteDEM using the D8 algorithm, or any other GIS method that defines streams in a more linear way. This could also be doing flow direction and flow accumulation manually, then selecting pixels that have a flow accumulation value of >= 1000.

Once you have a stream network that looks more to your liking, you can see what happens when you burn it into the DEM. Again, there are some tools to help with this (like ArcHydro), or you can simply subtract some relatively large value from the DEM in the places where streams occur, forcing water to flow downslope there. Doing this (especially with the very simple subtraction method) can sometimes lead to a much messier stream network output by the model, so be sure to scrutinize the results to see if it made things better or worse.

Another thing you could try is setting specifically just your lake areas to NoData in the DEM and again scrutinize the results. The model won’t create streams there, but then you’ll also have holes of missing data, where the land cover map indicates that you’ll probably have significant runoff or retention due to the lake, so you’d have to decide if that kind of result is still relevant. This is one of the problems with having water defined in both the land cover map and the DEM, when they don’t usually line up cleanly.

There are other things that you can try, like burning just lakes into the DEM so that they still get defined, and are not NoData, but valid, just altered, elevation values. Or otherwise coordinating the water defined in the land cover map with the stream network in the DEM. It’s hard to say which of these options will work best for you, since I’ve found that different DEMs behave in different ways in the same place.

That’s a bit of rambling, but hopefully something in there gave you an option or two to try.

~ Stacie

[lukezw]

Hi Stacie,

Thanks for taking the time to go through all those options for me, it helps a lot.

While I try to mitigate the issue, I just wanted to ask how far it would be acceptable to have some of these weird flow artifacts in one’s final model, seeing as it is a common problem in flat areas? I’ll certainly look into the options you suggest, but also want to make sure I don’t expend excessive time and effort on completely eliminating the issue if it is something that is unavoidable and/or acceptable to a certain extent, given the limitations of the algorithms.

Edit: I also just discovered the Fill tool in ArcGIS has actually completely flattened out the relief in some of the valley floors! I expect using the original DEM will help significantly. I understand filling the DEM is recommended, so I’m rather surprised it messed up the valley floors so much in this instance.

[swolny]

Hi @lukezw -

Good question. I’d say that the answer depends on your use of the results, and your interpretation of the areas being included as streams.

If you feel like the large jagged areas do actually correspond to where water flows (even if the defined shape isn’t quite the same), then I’d say it’s ok to leave them. But if those areas actually don’t generally show flow (maybe they’re kind of a floodplain, but really there’s agriculture being done all along there, so it should be considered agriculture and not a stream), then you might want to try altering the DEM to make that match better. In addition to aesthetics, the model treats areas defined as streams differently, so you will end up with somewhat different results if most of one of those flat areas is modeled as stream versus, say, agriculture.

Also, who will be looking at the mapping results? If just you, and/or the results will just be presented as aggregated, not mapped, it might be fine as it is. But if others will be scrutinizing the maps and wonder what the heck is going on, you might want to try altering the DEM.

We often recommend trying the QGIS tool Wang & Liu for filling DEMs. It seems to work a bit better than Arc’s tool, and may produce a better result.

Another random note: It looks like it takes around 1.5 hours to run on your data. If you’d like to do iterations a little more quickly, you could use the InVEST tool RouteDEM, which only does the hydrologic processing on the DEM, including creating streams. So you don’t have the extra overhead of the rest of the SWY model, if you’re just trying to test out different DEMs. It might save some time.

~ Stacie

[lukezw]

Hello again Stacie,

Thank you for your continuing assistance. Your response does help me in thinking about how much effort to make in correcting the stream delineation. While quickflow might be a key component of what we will present in the final maps for our project, I would be more concerned about the influence the inappropriate stream placement might have on our other results, as you note.

I have been trying your recommendation to force a different stream network. However, all the tools seem to be (inappropriately?) filling certain valleys in the DEM, including the RouteDEM tool. This makes the valley floors pretty much flat, which in my mind is contributing to the stream mapping issues, especially since these flattened out areas correspond well to the regions where the stream map is placing streams everywhere. As a result, the bunching of streamflow in certain valley floors remained even with the D8 algorithm RouteDEM method.

To show what I mean with the DEMs, I am adding screenshots of the filled DEM produced by the RouteDEM tool, and below it a picture of the original DEM for comparison. The pop-ups show the huge difference in elevation between the two for a pixel in the valley floor. I have also added a satellite image of the same area for context. Perhaps this is what the fill tools generally do, so apologies if this is a standard result. RouteDEM, Arc’s Fill tool and the Wang and Liu QGIS tool all filled and flattened this valley in a similar way.

Is this expected behaviour from these tools? I know there are some other options you recommended I could explore, but I just wanted to share this update as the RouteDEM tool sounded like one of the more promising solutions. I tried the D8 method with Arc’s flow accumulation tool on the original unfilled DEM, but those results didn’t look good, and the RouteDEM certainly seems a more promising options aside from the issues with the valley floor areas.

Sorry for all the additional comments for you to work through.

Luke

[swolny]

Hi Luke -

Interesting that they all are filling in that valley. I think you mentioned this before, but have you tried running the model with the original unfilled DEM for comparison?

Also, I wonder if it would make a difference to try larger values for the Wang & Liu “minimum slope” input, or “Z limit” in ArcGIS Fill. I haven’t tried fussing with them, but the brief documentation indicates that it might have an effect on creating flat areas.

If those don’t help, this is where I’d try burning in lakes. You’re about as far now as I’ve gone down this road, so it’s interesting to see how things do or don’t work for you.

~ Stacie

[lukezw]

Hi Stacie,

I’ve switched to prepping the Sediment delivery ratio models for a bit so I’ll let you know what happens when I return to the SWY.

I’m actually not sure myself now whether the first model run was with the filled or unfilled DEM! I will try next time with the unfilled DEM to be sure, before moving on to other solutions. The thing I’m not clear on is whether the SWY model itself performs any DEM filling as part of the modelling process? Of course, I’m aware that any issues with DEM filling will still crop up in the SDR model so remains something I’m paying attention too!

I came across the Whitebox Hydrological toolset which has a tool called breach depressions which is meant to do a better job than filling DEMs, but it produced the same result. Just mentioning this out of interest. The topography in this region really does challenging!

Luke

[swolny]

I don’t think that the freshwater models do DEM filling (and actually hope they don’t), @rich or @jdouglass can you either confirm or deny this?

~ Stacie

[jdouglass]

@swolny, yes, I confirm that NDR, SDR, SWY, RouteDEM and DelineateIt all fill pits before doing any routing.

I do think it’s worth noting, though, that our routing method is fairly conservative: it will only fill sinks as much as is needed to allow the water to continue to flow downhill or to a pour point. So we do fill pits, but only if they exist. If you’ve already ensured that your DEM has no hydrological sink, then our pitfilling routine shouldn’t change anything in your DEM.

[rich]

Just to jump on the justification of that because you seem concerned Stacie, we implemented this because at the time most of our user support issues with this model were due to pits in the DEM. We always gave the same advice: fill your DEM with the Wang&Liu pitfilling algorithm in GRASS/SATA. So what happens now is just what James says: if there are pits, they are filled with W&L, otherwise the DEM is left alone.

If for some reason W&L is not the method someone would want to use to fill their DEM then they should apply that algorithm first before running InVEST and ensure that their DEM drains. I feel like a user who is aware of this issue could comfortably do that in the first place, otherwise the other large portion of users should have their needs served by the automatic pitfilling that we’d otherwise manually recommend.

[lukezw]

Thanks @rich and @jdouglass for the extra input. Seems the challenge that I am yet to solve is that various tools, including the Wang and Liu algorithm, seem to be going way overboard in filling the DEMs for this region. See below a flow direction map produced by the Wang and Liu algorithm after it too performed some rather aggressive sink filling. This was after reducing the minimum slope threshold to 0.005 degrees. Those areas with uniform flow direction are often 50 km or more across and 100s of kms long, so it is a pretty sizable effect.

We didn’t encounter this problem when we ran the models in Rwanda where the topography is different. Unfortunately for this region, I so far have not found any DEM filling tool or technique which avoids turning these dry valley floors into streams/lakes. I’m still new to hydrological modelling, but I guess the fact that these valleys don’t always have a clear inflow or outflow point must be adding to the difficulties the DEM fill algorithms are encountering.

[rich]

That’s an interesting region you have there… From what I understand about these routed models, the science is derived from studying active runs of water through controlled slopes. Meaning the biophysical processes that SDR/NDR are modeling are caused by the active flow of water across the landscape. That’ll break down in the case of extreme slopes (SDR won’t model a waterfall) and I suspect they don’t apply when the hydrological region is so wide and flat like yours since there’s not the forcing function of gravity pulling water downhill and the moment of that water causing erosion along with it. If you want to model the erosion across those flats I suspect you’ll need to go with a different model. If you’re doing something simpler like trying to estimate total erosion from a watershed across scenarios in the drainable regions, you might be able to nodata out those flat regions and treat them like local sinks. The overall change in sediment load would be reasonable since it’s not clear what those flat regions would be contributing to the biophysical processes that SDR/NDR model. Anyway, I don’t know if that helps except to say that the InVEST routed models will not accurately model anything in near-zero slope regions.

[lukezw]

Thanks @rich for that explanation of the models and how the terrain might just be exceeding the limiting factors of the models.

As I read and experiment more, it seems more apparent to me that the key issue is not simply the flatness of the region but more the fact there are multiple valleys which are basically endorheic/closed basins, and they are being treated as huge hydrological sinks by all DEM filing algorithms I have tried, including the Wang and Liu. See the comparison below between an unfilled DEM on the left, and the filled DEM that is produced by the Wang and Liu algorithm in QGIS (which is presumably very similar to what the DEM would look like after the Invest models apply the W&L algorithm)

Today I discovered the WWF Hydroshed’s Hydrologically conditioned DEM, which seems to have been specially modified to retain natural closed basins whilst at the same time producing a sink-free DEM. It has a lower resolution than the 30 m SRTM DEM I was using previously, but the way they have handled closed basins seems to have improved things a lot, as a number of these areas are no longer being filled as sinks by the W&L algorithm (compare below left and right). However,even in the bottom left picture there is still an area that is being picked up as a sink and filled by the model towards the southwest of the region, resulting in the inaccurate streamflow.

Going out on a limb here, but I wonder if maybe including something like a checkbox to fill DEMs could be included in future updates? I understand the reasoning for automating DEM filling in the majority of cases as @rich and @jdouglass have explained, and I have no idea whether this is something that can be implemented, but thought I would just make the humble suggestion. It might be pretty beneficial in a case like this where you have a DEM like the Hydrosheds hydrologically conditioned DEM which should be sink-free, but is still being filled in places by the model. When I applied the D8 flow algorithm to the Hydrosheds DEM in ArcGis it managed to model natural looking streams across the whole region, including the area in the bottom left portion of the pictures above. In my mind, this shows that there is no hydrological sink in that region when using the Hydrosheds DEM, but because the Wang and Liu algorithm is treating that area as a sink, the Invest model’s flow mapping has gone awry. Since ArcGis seems able to handle that region fine, I’d be very interested in seeing what sort of results the model would produce if I could run it on the Hydrosheds DEM with no W&L algorithm applied!

I know this has all got rather long and convoluted but I appreciate everyone’s dedication to remain with me this long!