Coastal vulnerability memory error (and managing very large AOIs)

Hello,

I am currently trying to use the Coastal Vulnerability model its stopped saying there is a memory error. I was wondering what could be causing this? My model is looking at the global extent of mangroves, so it is a very large amount of data (several layers are global, though AOI isnt). Is there anything I can do to help ensure that it runs? At the moment I have the model resolution at 1000m, would making this courser help?

Looking at the log this was the first error message ‘ERROR Something went wrong when adding task index landmass geometry (2), terminating taskgraph.’ and then memory errors after that.

Any insight would be greatly appreciated!

Thanks

Hi @MegC,

Thanks for posting and sorry about the memory issue! Could you attach the logfile that is generated in the output workspace with each InVEST run?

Thanks,

Doug

InVEST-Coastal-Vulnerability-log-2020-10-20–18_43_07.txt (20.3 KB)

Hi Doug,

Have attached the log file, I tried running again using the landmass polygon provided by InVEST but ran into the same problem I think. Any help really appreciated!

Best,

Meg

Hi @MegC,

Thanks for that. It looks like there might be too many shore points being created. Would it be possible to share your data? It’s probably not small enough to zip and share here but you could use a cloud service like Google Drive to share a link or another cloud sharing method.

It’s not exactly the same issue as this post: Coastal Vulnerability Run time error - #4 by dave but in that post Dave mentions:

  • Use the AOI to exclude parts of the landmass layer, in order to have fewer shore points

But it looks like you already have an AOI set up.

Doug

Hi @dcdenu4,

Thank you very much for taking a look into this, I have put the data into a google drive here: https://drive.google.com/drive/folders/1s_AqMJ2D1ZOb9Ut8h74Wnx1LtVVx3V7P?usp=sharing though the mangrove extent, AOI and human population data are the only ones which differ from the data given with the model (is it okay to use the global DEM for elevation and bathymetry?). We also include geomorphology in our model, but this is currently unpublished so not included in this folder. I should also not the human population radius should be 5000m, not 500m. Please let me know if anything is missing, and thanks for your help!

Best,

Megan

Hi @MegC,

Thanks for sharing the data. It looks like the shapefile you’re using for the AOI is projected in world eckert iv. Do you have a smaller study area you’re interested in or are you interested in a global run? Could you try using a smaller more regional AOI with the corresponding regional projection and see if that runs okay?

Cheers,

Doug

Hi @dcdenu4,

I am hoping to run this globally, though centred around the tropics as its looking at mangroves. I had originally tried to run it using a mollweide projection but that wouldn’t run. Is there another projection you would suggest for a global run?

Thanks,

Megan

Hi @MegC,

I don’t think the model is currently set up to be run at a global scale on a single pass. I think people in the past have broken up the global data into smaller regional chunks with a local projection that preserves distances. They then run the model separately on all those chunks and combine the data back together. But the final index calculations need to have all of the data already combined, which might have to be a post processing step.

At the very least we should definitely look into this memory error as our models should exit much more gracefully then that.

Perhaps @dave or @rich could touch on their knowledge of the model.

Best,

Doug

Yes to what Doug said. A projection that does not distort distances is important because the model is calculating euclidean distances for things like distance from shore points to habitats, to the continental shelf, to the nearest WaveWatchIII data points. Generally these are fairly short distances (not spanning the globe), but it’s still important to consider the projection.

Maybe there is an equidistant projection that can span the tropics, as long as it doesn’t extend too far north or south? https://projectionwizard.org/

1 Like

Hi @dave and @dcdenu4,

Thank you for the information, I will try one last time to see if I can get it to work with an equidistant projection as @dave suggested, otherwise, I have been able to run the model using just Myanmar, so hopefully @dcdenu4 suggestion will work. Would you be able to give more detail on the post-processing step you mentioned for the final index calculations?

Thanks,

Meg

Hi @MegC, Thinking it through a bit more, there are a few challenges with breaking up an analysis into multiple AOIs and then getting results that can be compared from one AOI to the next.

Equation 1 is the final index equation (http://releases.naturalcapitalproject.org/invest-userguide/latest/coastal_vulnerability.html#exposure-index). In order to compare the final Exposure Index from a point in AOI_1 with a point in AOI_2, all the R_ terms data from both areas must be included the calcuation. Those R_ data are in the coastal_exposure.gpkg output. So one could combine the tables of multiple coastal_exposure.gpkg outputs and then recalculate the Exposure Index.

But before that, some of those R_ terms are relative rankings themselves. In other words, in order for an R_relief value from AOI_1 to be comparable to an R_relief from AOI_2, all of the relief data from both areas must be included in the “ranking” of R_relief. Those intermediate relief values are in intermediate_exposure.gpkg. And the same applies to R_surge, R_wind, R_wave. Table 4.1 shows how the R_ values are assigned for each of these variables - basically just splitting the raw values into 5 quantile groups.

So that’s kind of a lot of work, but this sort of thing has been done before. For example, another use-case for re-calculating the final exposure index is to compare values across different scenarios within the same AOI.

2 Likes

Hi Dave,

Thank you for following up on this, and explaining further. That makes sense and could be useful as a work-around.

Best,

Megan