Merge branch 'release/v3.40.1-2'

Author: cbeyer42

CHANGELOG.md

@@ -6,6 +6,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [v3.40.1-2] - 2022-08-19
+### Added
+- Preconfigured compact blobstore task which will run every 14 days. #108
+- Preconfigured cleanup policy which, wenn added to a matching maven-snapshot repository, will mark artifacts older than 14 days for deletion. #108
+### Fixed
+- Remove orientDB credentials from log messages. #88
+
 ## [v3.40.1-1] - 2022-07-12
 ### Changed
 - Upgrade Sonatype Nexus to v3.40.1-01; #106
@@ -30,7 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [v3.34.1-4] - 2021-12-13
 ### Fixed
-- disable jndi lookup due to a vulnerability #90 (https://doc.nexusgroup.com/pages/viewpage.action?pageId=83133294)
+- Disable jndi lookup due to a vulnerability #90 (https://doc.nexusgroup.com/pages/viewpage.action?pageId=83133294)
 
 ## [v3.34.1-3] - 2021-11-02
 ### Changed
@@ -80,19 +87,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 - Update dogu-build-lib to `v1.1.1`
 - Update zalenium-build-lib to `v2.1.1`
-- toggle video recording with build parameter (#63)
+- Toggle video recording with build parameter (#63)
 
 ### Removed
-- installation of R and Helm plugins. These plugins are a built-in feature now. (#66)
+- Installation of R and Helm plugins. These plugins are a built-in feature now. (#66)
 
 ## [v3.28.1-3] - 2020-12-14
-
 ### Added
-
 - Ability to configure the `MaxRamPercentage` and `MinRamPercentage` for the Nexus process inside the container via `cesapp edit-conf`  (#61)
 
 ## [v3.28.1-2] - 2020-11-27
-
 ### Fixed
 - Remove nexus admin password from environment variable. Now, the password is passed via enviroment variable passing only to the respective tools (#59)
 
@@ -129,54 +133,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [v3.23.0-4] - 2020-09-04
 ### Changed
-- register missing appenders in logging configuration 
+- Register missing appenders in logging configuration 
 
 ## [v3.23.0-3] - 2020-07-24
 ### Changed
-- update dogu-build-lib
-- update ces-build-lib
-- update java image
-- use doguctl validation
+- Update dogu-build-lib
+- Update ces-build-lib
+- Update java image
+- Use doguctl validation
 
 ## [v3.23.0-2] - 2020-07-02
 ### Changed
-- update carp.yml.tpl to contain log-level and log-format
-- update nexus-carp to v1.0.0
+- Update carp.yml.tpl to contain log-level and log-format
+- Update nexus-carp to v1.0.0
 
 ## [v3.23.0-1] - 2020-06-18
 ### Changed
-- updated Nexus version to 3.23.0
-- enabled groovy scripting during startup in `nexus.properties`
-- update nexus-claim to v1.0.0
+- Updated Nexus version to 3.23.0
+- Enabled groovy scripting during startup in `nexus.properties`
+- Update nexus-claim to v1.0.0
 
 ## [3.19.1-2] - 2020-04-15
 ### Added
-
-A new CES registry key `logging/root` is evaluated to override the default root log level. One of these values can be set in order to increase the log verbosity: `ERROR`, `WARN`, `INFO`, `DEBUG`. These log levels are directly applied to Nexus's logback root appender configuration.
-
-Changing Nexus's log level with different settings at runtime is still supported. Please note that these settings are reset (to the root log level from above) during a restart of the Nexus dogu. (#37)
+- A new CES registry key `logging/root` is evaluated to override the default root log level. One of these values can be set in order to increase the log verbosity: `ERROR`, `WARN`, `INFO`, `DEBUG`. These log levels are directly applied to Nexus's logback root appender configuration.
+  Changing Nexus's log level with different settings at runtime is still supported. Please note that these settings are reset (to the root log level from above) during a restart of the Nexus dogu. (#37)
 
 ### Changed
-
-In order to cope with the amount of file system data the max history is set to 7 days worth of Nexus logging, capping the total log size to 10 MBytes. This is only important for Nexus's own Log viewer. Logs to the Cloudogu EcoSystem host are not subject to change though. (#37)
+- In order to cope with the amount of file system data the max history is set to 7 days worth of Nexus logging, capping the total log size to 10 MBytes. This is only important for Nexus's own Log viewer. Logs to the Cloudogu EcoSystem host are not subject to change though. (#37)
 
 ## Removed
-
-Remove unnecessary log appenders (#37).
+- Remove unnecessary log appenders (#37).
 
 ### Fixed
-
-Reduce the default root log level to WARN. Nexus's defaults to INFO which leads to an obscene amount of log entries from the underlying Felix framework. (#37)
+- Reduce the default root log level to WARN. Nexus's defaults to INFO which leads to an obscene amount of log entries from the underlying Felix framework. (#37)
 
 ## [3.19.1-1] - 2019-12-09
-
 ### Changed
-
 - Changed Nexus version from 3.18.1 to 3.19.1
 - Changed Java version in Dockerfile to 8u222-1
 
 ### Added
-
 - Added docker health check
 - Add a start-up check whether the minimum number of CPU cores is reached (#36)
    - Starting with [Nexus Repository Manager 3.17](https://issues.sonatype.org/secure/ReleaseNote.jspa?projectId=10001&version=17890) a minimum number of 4 CPU cores is enforced, otherwise the Repository Manager is no longer guaranteed to work.

Dockerfile

@@ -2,7 +2,7 @@
 FROM registry.cloudogu.com/official/java:8u302-3
 LABEL maintainer="[email protected]" \
     NAME="official/nexus" \
-    VERSION="3.40.1-1"
+    VERSION="3.40.1-2"
 
 # The version of nexus to install
 ENV NEXUS_VERSION=3.40.1-01 \

Makefile

@@ -1,5 +1,5 @@
 MAKEFILES_VERSION=6.0.3
-VERSION=3.40.1-1
+VERSION=3.40.1-2
 
 .DEFAULT_GOAL:=dogu-release
 

docs/operations/predefined_nexus_entities_de.md

@@ -0,0 +1,36 @@
+# Vorkonfigurierte Nexus Settings
+Durch die verwendung von Scripten die zum start des Nexus aufgerufen werden eine *Cleanup Policy* sowie ein *Compact Blobstore Task* angelegt.
+
+## Compact Blobstore Task
+Wird in Nexus ein Artefakt zum Löschen vorgesehen (bspw. durch den automatisch laufenden *Cleanup Service Task*) wird das
+Artefakt nur zum Löschen markiert. Das endgültige Löschen der Daten aus dem Blobstore übernimmt ein *Compact Blobstore Task,* 
+welcher aber nicht in der standardkonfiguration von Nexus konfiguriert ist.
+Dieser Task wird beim Start der Applikation von dem Skript `nexusSetupCompactBlobstoreTask.groovy` angelegt.
+Der Task löscht hierbei Daten (täglich, wenn die standard Konfiguration verwendet wird) aus dem _default_ Blobstore. Falls ein anderer Blobstore konfiguriert werden soll, kann hierfür 
+der etcd-Schlüssel `config/nexus/compact_blobstore_task/blobstore` angepasst werden. 
+Dies geht am einfachsten über den cesapp Befehl  `cesapp edit-config nexus`.
+Mehr Informationen über Tasks sind in der [offiziellen Tasks-Dokumentation](https://help.sonatype.com/repomanager3/nexus-repository-administration/tasks) zu finden.
+
+
+## Cleanup Policy
+Wie auch der oben genannte Task wird eine Policy (`ces-maven-snapshot-cleanuppolicy`) per Skript (`nexusSetupCleanupPolicies.groovy`)
+angelegt. Diese Cleanup Policy ist für maven-snapshot Repositorys gedacht. Um sie anzuwenden, muss entweder das Repository manuell konfiguriert werden oder
+in einer hcl Konfiguration per `nexus-claim` das Feld `policyName` mit einer Liste aus Policies gefüllt werden die `ces-maven-snapshot-cleanuppolicy` enthält.
+
+```
+repository "public" {
+  _state = "present"
+  online = true
+  recipeName = "maven2-hosted"
+  attributes = {
+    cleanup = {
+      policyName = ["ces-maven-snapshot-cleanuppolicy"]
+    },
+    
+    ...
+  }
+```
+
+Die Policy kann in per `cesapp edit-config nexus` konfiguriert werden. Das Standardintervall für die Policy beträgt 14 Tage.
+Mehr Informationen über Cleanup Policies können der [offiziellen Cleanup-Policy-Dokumentation](https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies) entnommen werden.
+

docs/operations/predefined_nexus_entities_en.md

@@ -0,0 +1,36 @@
+# Preconfigured Nexus settings
+By using scripts that are called to start the Nexus, a *Cleanup Policy* and a *Compact Blobstore Task* are created.
+
+## Compact Blobstore Task
+If an artifact is scheduled for deletion in Nexus (e.g. by the automatically running *Cleanup Service Task*), the artifact is only **marked** for deletion but not actually deleted.
+The final deletion of the data from the Blobstore is done by a *Compact Blobstore Task,*
+which is not configured in the default Nexus configuration.
+This task is created by the script `nexusSetupCompactBlobstoreTask.groovy` when the application is started.
+The task deletes data daily (in its standard configuration) from the _default_ blobstore. If you want to configure a different blobstore, you can do this by 
+modifying the etcd key `config/nexus/compact_blobstore_task/blobstore`.
+The easiest way to do this is to use the cesapp command `cesapp edit-config nexus`.
+
+More information about tasks can be found in the [official task-documentation](https://help.sonatype.com/repomanager3/nexus-repository-administration/tasks).
+
+
+## Cleanup Policy
+Like the task mentioned above, a policy (`ces-maven-snapshot-cleanuppolicy`) is created by script (`nexusSetupCleanupPolicies.groovy`).
+This cleanup policy is intended for maven-snapshot repositories. To apply it, either the repository has to be configured manually or
+in a hcl configuration via `nexus-claim` the field `policyName` must be filled with a list of policies containing `ces-maven-snapshot-cleanuppolicy`.
+
+```
+repository "public" {
+  _state = "present
+  online = true
+  recipeName = "maven2-hosted
+  attributes = {
+    cleanup = {
+      policyName = ["ces-maven-snapshot-cleanuppolicy"]
+    },
+    
+    ...
+  }
+```
+
+The policy can be configured via `cesapp edit-config nexus` command. The default intervall for the cleanup policy is 14 days.
+More information about cleanup policies can be found in the [official cleanup-policy-documentation](https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies).

dogu.json

@@ -1,6 +1,6 @@
 {
   "Name": "official/nexus",
-  "Version": "3.40.1-1",
+  "Version": "3.40.1-2",
   "DisplayName": "Sonatype Nexus",
   "Description": "The Nexus Repository is like the local warehouse where all of the parts and finished goods used in your software supply chain are stored and distributed.",
   "Url": "http://www.sonatype.org/nexus",
@@ -81,7 +81,7 @@
     },
     {
       "Name": "logging/root",
-      "Description": "Set the root log level to one of ERROR, WARN, INFO, DEBUG.",
+      "Description": "Set the root log level to one of ERROR, WARN, INFO, DEBUG. The default log level is set to \"WARN\"",
       "Optional": true,
       "Default": "WARN",
       "Validation": {
@@ -127,6 +127,84 @@
       "Validation": {
         "Type": "FLOAT_PERCENTAGE_HUNDRED"
       }
+    },
+    {
+      "Name": "cleanup_policy/notes",
+      "Description": "Notes to the default cleanup-policy. Default: \"Do not edit. CES will re-generate this default policy during each Nexus start-up that will overwrite your changes. You may want to create your own policies which will stay untouched during the start-up.\"",
+      "Optional": true,
+      "Default": "Do not edit. CES will re-generate this default policy during each Nexus start-up that will overwrite your changes. You may want to create your own policies which will stay untouched during the start-up."
+    },
+    {
+      "Name": "cleanup_policy/policy_format",
+      "Description": "Repository format on which the default cleanup-policy can be applied. The default is \"maven2\" which corresponds to maven repositories",
+      "Optional": true,
+      "Default": "maven2",
+      "Validation": {
+        "Type": "ONE_OF",
+        "Values": [
+          "apt",
+          "bower",
+          "cocoapods",
+          "conan",
+          "conda",
+          "docker",
+          "gitlfs",
+          "go",
+          "helm",
+          "maven2",
+          "npm",
+          "nuget",
+          "p2",
+          "pypi",
+          "r",
+          "raw",
+          "rubygems",
+          "yum"
+        ]
+      }
+    },
+    {
+      "Name": "cleanup_policy/criteria/regex",
+      "Description": "Regex which will be used to match fitting artifacts. Default: \".*SNAPSHOT\"",
+      "Optional": true,
+      "Default": ".*SNAPSHOT"
+    },
+    {
+      "Name": "cleanup_policy/criteria/release_type",
+      "Description": "The release type on which this policy will be applied. This configuration can only be applied if the 'policy_format' is of type \"maven2\", \"yum\" and \"npm\". Default: \"PRERELEASES\"",
+      "Optional": true,
+      "Default": "PRERELEASES"
+    },
+    {
+      "Name": "cleanup_policy/criteria/days_till_recognition_for_delete",
+      "Description": "The number of days after which an artifact is recognised for deletion. Default: \"14\"",
+      "Optional": true,
+      "Default": "14"
+    },
+    {
+      "Name": "compact_blobstore_task/enabled",
+      "Description": "Switch to control whether the policy should be switched on or off. Default: \"true\"",
+      "Optional": true,
+      "Default": "true",
+      "Validation": {
+        "Type": "ONE_OF",
+        "Values": [
+          "true",
+          "false"
+        ]
+      }
+    },
+    {
+      "Name": "compact_blobstore_task/blobstore",
+      "Description": "The specific blobstore instance to that the task should be applied. This configuration should only be used if the task type is 'blobstore.compact' (or other tasks that need to choose a blobstore instance). Default: \"default\"",
+      "Optional": true,
+      "Default": "default"
+    },
+    {
+      "Name": "compact_blobstore_task/cron",
+      "Description": "Cron string (in Quartz syntax) that sets the task's schedule. The default \"0 0 3 * * ?\" value will run the task daily at 3:00 AM.",
+      "Optional": true,
+      "Default": "0 0 3 * * ?"
     }
   ],
   "HealthChecks": [

resources/opt/sonatype/nexus/resources/nexusCleanupPolicies.json.tpl

@@ -0,0 +1,11 @@
+{
+  "name": "ces-default-cleanuppolicy",
+  "notes": "{{ .Config.GetOrDefault "cleanup_policy/notes" "Default policy that is generated by the ces via a script on nexus startup"}}",
+  "mode": "deletion",
+  "format": "{{ .Config.GetOrDefault "cleanup_policy/policy_format" "maven2"}}",
+  "criteria": {
+    "regex": "{{ .Config.GetOrDefault "cleanup_policy/criteria/regex" ".*SNAPSHOT"}}",
+    "criteriaReleaseType": "{{ .Config.GetOrDefault "cleanup_policy/criteria/release_type" "PRERELEASES"}}",
+    "criteriaLastBlobUpdated": "{{ .Config.GetOrDefault "cleanup_policy/criteria/days_till_recognition_for_delete" "14"}}"
+  }
+}

resources/opt/sonatype/nexus/resources/nexusCompactBlobstoreTask.json.tpl

@@ -0,0 +1,7 @@
+{
+  "type":   "blobstore.compact",
+  "enabled":  "{{ .Config.GetOrDefault "compact_blobstore_task/enabled" "true"}}",
+  "blobstore":  "{{ .Config.GetOrDefault "compact_blobstore_task/blobstore" "default"}}" ,
+  "name": "default CES compact blobstore task" ,
+  "cron": "{{ .Config.GetOrDefault "compact_blobstore_task/cron"  "0 0 3 * * ?"}}"
+}

resources/opt/sonatype/nexus/resources/nexusSetupCleanupPolicies.groovy

@@ -0,0 +1,56 @@
+import org.sonatype.nexus.cleanup.storage.CleanupPolicyStorage
+import groovy.json.JsonSlurper
+
+import static org.sonatype.nexus.repository.search.DefaultComponentMetadataProducer.IS_PRERELEASE_KEY
+import static org.sonatype.nexus.repository.search.DefaultComponentMetadataProducer.LAST_BLOB_UPDATED_KEY
+import static org.sonatype.nexus.repository.search.DefaultComponentMetadataProducer.REGEX_KEY
+
+// get parameters from payload JSON file
+def configurationParameters = new JsonSlurper().parseText(args)
+
+def Integer asSeconds(Integer days) {
+    return days * 60 * 60 * 24
+}
+
+def String asStringSeconds(String daysString) {
+    return String.valueOf(asSeconds(Integer.parseInt(daysString)))
+}
+
+/**
+ * this script is called in startup.sh and creates a cleanup
+ * @return
+ */
+def createMavenSnapshotCleanupPolicy(configurationParameters) {
+    def policyStorage = container.lookup(CleanupPolicyStorage.class.getName())
+
+    def cleanupPolicy = policyStorage.newCleanupPolicy()
+    cleanupPolicy.setName(configurationParameters.name)
+    cleanupPolicy.setNotes(configurationParameters.notes)
+    cleanupPolicy.setMode(configurationParameters.mode)
+    cleanupPolicy.setFormat(configurationParameters.format)
+
+    def criteriaMap = [:]
+    criteriaMap.put(REGEX_KEY, configurationParameters.criteria.regex) // criteriaAssetRegex
+    if (configurationParameters.criteria.criteriaReleaseType != "") {
+        // We do a additional check here as the criteriaReleaseType is not supported for every kind of repository
+        // and should therefore be only set if a value is present.
+        // see: https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies
+        criteriaMap.put(IS_PRERELEASE_KEY, "PRERELEASES".equals(configurationParameters.criteria.criteriaReleaseType).toString())
+    }
+    criteriaMap.put(LAST_BLOB_UPDATED_KEY, asStringSeconds(configurationParameters.criteria.criteriaLastBlobUpdated))
+
+    cleanupPolicy.setCriteria(criteriaMap)
+
+    deleteCleanupPolicyIfExists(configurationParameters.name)
+    policyStorage.add(cleanupPolicy)
+}
+
+def deleteCleanupPolicyIfExists(String name) {
+    def cleanupPolicyStorage = container.lookup(CleanupPolicyStorage.class.getName())
+    if (cleanupPolicyStorage.exists(name)) {
+        cleanupPolicyStorage.remove(cleanupPolicyStorage.get(name))
+    }
+}
+
+// start
+createMavenSnapshotCleanupPolicy(configurationParameters)