ERDDAP Changes
--> - -Changes -in ERDDAP version 2.20 (released 2022-09-01)
+Changes +in ERDDAP version 2.20 (released 2022-09-30)
- New Features and Changes (for users):
-
@@ -81,8 +81,45 @@
Chan
- Things ERDDAP Administrators Need to Know and Do:
-
-
- IMPROVED: More improvements to the memory management system.
-
+ - TO DO: In [tomcat]/conf/context.xml, right before </Context>,
+ change the Resources tag (or add it if it isn't already there) to
+ increase the cacheMaxSize setting from 80000 (or whatever it was) to 200000:
+
<Resources cachingAllowed="true" cacheMaxSize="200000" /> +
This avoids numerous warnings in catalina.out that all start with +
"WARNING [main] org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/..." +
+ + - IMPROVED: We re-enabled the old memory management system (Math2.ensureMemoryAvailable) and modified
+ the new memory management system (EDStatic.shedThisRequest) to work better with it.
+
ERDDAP shouldn't ever crash or freeze up. If it does, one of the + most likely causes is insufficient memory. You can monitor memory usage + by looking at the status.html web page, which now includes a line like +
0 gc calls, 0 requests shed, and 0 dangerousMemoryEmails since last major LoadDatasets +
(those are progressively more serious events) +
and a gc Calls column in the table of statistics. + You can tell how memory-stressed your ERDDAP is by watching these numbers + in addition to the MB inUse columns. + Higher numbers indicate more stress. +-
+
- MB inUse should always be less than half of your Xmx memory setting. + Larger numbers are serious trouble. +
- gc calls indicates the number of times ERDDAP called the + garbage collector to try to alleviate high memory usage. +
- shed indicates the number of incoming requests that + were shed (with HTTP error number 503, Service Unavailable) because + memory use was already too high. Ideally, no requests should be shed. +
- dangerousMemoryEmails -
+ If memory use becomes dangerously high, ERDDAP sends an email to the email addresses listed in
+ emailEverythingToCSV (in setup.xml) with a list of the active user requests. As the email says,
+ please forward these emails to erd.data at noaa.gov so we can use the information
+ to improve future versions of ERDDAP.
+
+
- CHANGED: The default for <ipAddressMaxRequests> in datasets.xml
+ was increased from 7 to 15. It's clear that some legitimate WMS clients
+ can generate more than 7 simultaneous requests.
+
- IMPROVED: More improvements to the memory management system.
-
Chan
Changes
in ERDDAP version 2.19 (released 2022-09-01)
+- Don't use v2.19. It is flawed. But administrators still need to do the TO DO
+ items listed below when upgrading to v2.20+.
+
- New Features and Changes (for users):
- NEW: There is a new server-side function, orderByDescending, which works like
diff --git a/download/setup.html b/download/setup.html
index 6acfeba27..ea877a6ad 100644
--- a/download/setup.html
+++ b/download/setup.html
@@ -521,24 +521,17 @@
-
- - Resources Cache
-
To avoid Tomcat throwing warnings during startup about
- "org.apache.catalina.webresources.Cache.getResource Unable to add the resource
- at someFile to the cache because there was insufficient free space
- available after evicting expired cache entries -- consider increasing the
- maximum size of the cache:
-
In your [tomcat]/conf/context.xml, add this tag right before </Context> :
-
-
<Resources cachingAllowed="true" cacheMaxSize="80000" />
-
(or some other number, in KB.) For more information see
-
https://tomcat.apache.org/tomcat-8.0-doc/config/resources.html
.
-
+ - Resources Cache -
+ In tomcat/conf/context.xml, right before the </Context> tag,
+ change the Resources tag (or add it if it isn't already there) to
+ increase the cacheMaxSize setting from 80000 (or whatever it was) to 200000:
+
<Resources cachingAllowed="true" cacheMaxSize="200000" />
+
This avoids numerous warnings in catalina.out that all start with
+
"WARNING [main] org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/..."
+
+
- On Linux computers, change the Apache timeout settings
so that time-consuming user requests don't timeout
@@ -2435,6 +2428,32 @@
Things
+ - Memory Status
+
ERDDAP shouldn't ever crash or freeze up. If it does, one of the
+ most likely causes is insufficient memory. You can monitor memory usage
+ by looking at the status.html web page, which includes a line like
+
0 gc calls, 0 requests shed, and 0 dangerousMemoryEmails since last major LoadDatasets
+
(those are progressively more serious events)
+
and MB inUse and gc Calls columns in the table of statistics.
+ You can tell how memory-stressed your ERDDAP is by watching these numbers.
+ Higher numbers indicate more stress.
+
+ - MB inUse should always be less than half of the
+ -Xmx memory setting.
+ Larger numbers are serious trouble.
+
- gc calls indicates the number of times ERDDAP called the
+ garbage collector to try to alleviate high memory usage.
+
- shed indicates the number of incoming requests that
+ were shed (with HTTP error number 503, Service Unavailable) because
+ memory use was already too high. Ideally, no requests should be shed.
+
- dangerousMemoryEmails -
+ If memory use becomes dangerously high, ERDDAP sends an email to the email addresses listed in
+ emailEverythingToCSV (in setup.xml) with a list of the active user requests. As the email says,
+ please forward these emails to erd.data at noaa.gov so we can use the information
+ to improve future versions of ERDDAP.
+
+
- OutOfMemoryError
When you set up ERDDAP, you specify the maximum amount of memory that Java
@@ -2530,7 +2549,7 @@ Things
file to see what ERDDAP was doing when the error arose,
you can usually get a good clue as to the cause.
In most cases, there is a way to minimize that problem (see above),
-but sometimes you just need more memory.
+but sometimes you just need more memory and a higher -Xmx setting.
diff --git a/download/setupDatasetsXml.html b/download/setupDatasetsXml.html
index ad783439a..58c54f8af 100644
--- a/download/setupDatasetsXml.html
+++ b/download/setupDatasetsXml.html
@@ -10702,7 +10702,7 @@ Details<
requests will receive an HTTP 429 error: Too Many Requests.
The small, static files in erddap/download/ and erddap/images/
are NOT exempt from this count.
- The default is 7. The maximum allowed is 1000, which crazy high -- don't do it!
+ The default is 15. The maximum allowed is 1000, which is crazy high -- don't do it!
ERDDAP won't accept a number less than 6 because
many legitimate users (notably web browsers and WMS clients) make
up to 6 requests at a time. The ERDDAP Daily Report and the
@@ -10740,7 +10740,7 @@ Details<
have been processed.
The small, static files in erddap/download/ and erddap/images/
ARE exempt from this count and the related throttling.
- The default is 2. The maximum allowed is 100, which crazy high -- don't do it!
+ The default is 2. The maximum allowed is 100, which is crazy high -- don't do it!
You can set this to 1 to be strict, especially if you have problems
with overly aggressive or malicious users.
Users will still quickly get all the data they request (up to ipAddressMaxRequests),
- NEW: There is a new server-side function, orderByDescending, which works like
diff --git a/download/setup.html b/download/setup.html
index 6acfeba27..ea877a6ad 100644
--- a/download/setup.html
+++ b/download/setup.html
@@ -521,24 +521,17 @@
- -
- Resources Cache -
To avoid Tomcat throwing warnings during startup about - "org.apache.catalina.webresources.Cache.getResource Unable to add the resource - at someFile to the cache because there was insufficient free space - available after evicting expired cache entries -- consider increasing the - maximum size of the cache: -
In your [tomcat]/conf/context.xml, add this tag right before </Context> : - -
<Resources cachingAllowed="true" cacheMaxSize="80000" /> -
(or some other number, in KB.) For more information see -
https://tomcat.apache.org/tomcat-8.0-doc/config/resources.html.
-
+- Resources Cache - + In tomcat/conf/context.xml, right before the </Context> tag, + change the Resources tag (or add it if it isn't already there) to + increase the cacheMaxSize setting from 80000 (or whatever it was) to 200000: +
<Resources cachingAllowed="true" cacheMaxSize="200000" /> +
This avoids numerous warnings in catalina.out that all start with +
"WARNING [main] org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/..." +
+- On Linux computers, change the Apache timeout settings so that time-consuming user requests don't timeout @@ -2435,6 +2428,32 @@
Things
- Resources Cache -
ERDDAP shouldn't ever crash or freeze up. If it does, one of the + most likely causes is insufficient memory. You can monitor memory usage + by looking at the status.html web page, which includes a line like +
0 gc calls, 0 requests shed, and 0 dangerousMemoryEmails since last major LoadDatasets +
(those are progressively more serious events) +
and MB inUse and gc Calls columns in the table of statistics. + You can tell how memory-stressed your ERDDAP is by watching these numbers. + Higher numbers indicate more stress. +
-
+
- MB inUse should always be less than half of the + -Xmx memory setting. + Larger numbers are serious trouble. +
- gc calls indicates the number of times ERDDAP called the + garbage collector to try to alleviate high memory usage. +
- shed indicates the number of incoming requests that + were shed (with HTTP error number 503, Service Unavailable) because + memory use was already too high. Ideally, no requests should be shed. +
- dangerousMemoryEmails - + If memory use becomes dangerously high, ERDDAP sends an email to the email addresses listed in + emailEverythingToCSV (in setup.xml) with a list of the active user requests. As the email says, + please forward these emails to erd.data at noaa.gov so we can use the information + to improve future versions of ERDDAP. +
When you set up ERDDAP, you specify the maximum amount of memory that Java @@ -2530,7 +2549,7 @@