From 989fc84777e8461f7c0bef766a4094c9afc9e076 Mon Sep 17 00:00:00 2001
From: maggieklope <jclark@nceas.ucsb.edu>
Date: Tue, 13 Aug 2024 22:52:42 +0000
Subject: [PATCH] maggieklope published a site update

---
 reference/edit-attributelists.html | 2 +-
 reference/search_index.json        | 2 +-
 training/editing-eml.html          | 2 +-
 training/search_index.json         | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/reference/edit-attributelists.html b/reference/edit-attributelists.html
index 58276999..f4f1dc7e 100644
--- a/reference/edit-attributelists.html
+++ b/reference/edit-attributelists.html
@@ -456,7 +456,7 @@ <h3>Edit attributes<a href="edit-attributelists.html#edit-attributes" class="anc
 <li><em>ordinal</em>: ordered categories. e.g.: Low, Medium, High</li>
 <li><em>dateTime</em>: date or time values from the Gregorian calendar. e.g.: 01-01-2001</li>
 <li><em>ratio</em>: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second.</li>
-<li><em>interval</em>: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or latitude and longitude coordinates, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius.</li>
+<li><em>interval</em>: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius.</li>
 </ul></li>
 <li><strong>domain</strong>: One of: <code>textDomain</code>, <code>enumeratedDomain</code>, <code>numericDomain</code>, <code>dateTime</code>. Required.
 <ul>
diff --git a/reference/search_index.json b/reference/search_index.json
index b98bbf64..53e88305 100644
--- a/reference/search_index.json
+++ b/reference/search_index.json
@@ -1 +1 @@
-[["index.html", "NCEAS Data Team Reference Guide About", " NCEAS Data Team Reference Guide 2024-08-13 About This reference guide documents various workflows and related issues for processing data packages. "],["miscellaneous.html", "Miscellaneous ", " Miscellaneous "],["code-snippets.html", "Code Snippets", " Code Snippets Code snippets help with templating portions of code that you will be using regularly. To add your own, go to the toolbar ribbon at the top of your Rstudio screen and select: Tools &gt; Global Options... &gt; Code &gt; Edit Snippets &gt; Add these chunks to the end of the file More info can be found in this blog post by Mara Averick on how to add them: https://maraaverick.rbind.io/2017/09/custom-snippets-in-rstudio-faster-tweet-chunks-for-all/ Usual arcticdatautils ticket workflow snippet ticket library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;add your rm&quot; pkg &lt;- get_package(adc, rm) doc &lt;- EML::read_eml(getObject(adc, pkg\\$metadata)) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) eml_validate(doc) eml_path &lt;- &quot;eml.xml&quot; write_eml(doc, eml_path) #update &lt;- publish_update(adc, # metadata_pid = pkg\\$metadata, # resource_map_pid = pkg\\$resource_map, # metadata_path = eml_path, # data_pids = pkg\\$data, # public = F) #datamgmt::categorize_dataset(update\\$metadata, c(&quot;theme1&quot;), &quot;Your Name&quot;) The datapack ticket workflow snippet datapack library(dataone) library(datapack) library(digest) library(uuid) library(dataone) library(arcticdatautils) library(EML) d1c &lt;- D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) packageId &lt;- &quot;id here&quot; dp &lt;- getDataPackage(d1c, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) #get metadata id metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #edit the metadata doc &lt;- read_eml(getObject(d1c@mn, metadataId)) #add the publisher info doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) doc\\$dataset\\$project &lt;- eml_nsf_to_project(&quot;nsf id here&quot;) #check and save the metadata eml_validate(doc) eml_path &lt;- arcticdatautils::title_to_file_name(doc\\$dataset\\$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) #upload the dataset myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c, dp, public=F, accessRules=myAccessRules, quiet=FALSE) #datamgmt::categorize_dataset(&quot;doi&quot;, c(&quot;theme1&quot;), &quot;Jasmine&quot;) Quick way to give access to submitters to their datasets: snippet access library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;rm here&quot; pkg &lt;- get_package(adc, rm) set_access(adc, unlist(pkg), &quot;orcid here&quot;) Quick access to the usual code for common Solr queries: snippet solr library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) "],["file-paths.html", "File paths", " File paths This section contains some basic tips about file paths. Remote files vs. local files If you’re working on the datateam server (RStudio in an internet browser), you’re usually working in the home directory under your username. File paths generally follow this pattern: /home/dmullen/folder/file. Note: These files are not stored on your local machine, they are on a remote server. You can access them through remote SFTP software like Cyberduck. If you’re working with your local RStudio app then your file paths generally follow this pattern: /Users/datateam/folder/file. These files are stored on your local machine and you can access them with Finder. File path shortcuts Using . in a file path refers to your current working directory. You can print your working directory with the call: getwd(). If we assume that my current working directory is home/dmullen, then the following two calls are equivalent: file.exists(&#39;/home/dmullen/myFile.R&#39;) file.exists(&#39;./myFile.R&#39;) Using ~ in a file path does not always refer to your current working directory. This is a common misconception. It is actually set to the global R USER path. You can print this with the call: Sys.getenv('USER'). Let’s assume that I’m working on the arcticdatautils package which I open using a .RProj file. This will update my current working directory to: /home/dmullen/arcticdatautils. Now the following calls are equivalent: file.exists(&#39;/home/dmullen/arcticdatautils/myFile.R&#39;) file.exists(&#39;~/arcticdatautils/myFile.R&#39;) file.exists(&#39;./myFile.R&#39;) "],["metacat-version.html", "Metacat version", " Metacat version Check which version of metacat is currently deployed to - arcticdata.io here, - and to test.arcticdata.io here. "],["resources-for-r.html", "Resources for R", " Resources for R Learning The following online books are useful for expanding your R knowledge and skills: the most recent ADC training materials The cleaning and data manipulation section is useful for working with attribute tables Efficient R Programming In particular Chapter 3 Efficient Programming R for Data Science Section on Strings R Packages contributing to arcticdatatutils, datamgmt and EML Advanced R Object-oriented programming in R for S4 to understand how datapack and dataone packages are written bookdown: Authoring Books and Technical Documents with R Markdown formatting, troubleshooting and updating the training document Others Hands-On Programming with R R Programming for Data Science Exploratory Data Analysis with R Mastering Software Development in R Geocomputation with R R Markdown: The Definitive Guide The Tidyverse Style Guide The RStudio cheatsheets are also useful references for functions in tidyverse and other packages. Packages The data team uses and develops a number of R packages. Here is a listing and description of the main packages: dataone reading and writing data at DataONE member nodes http://doi.org/10.5063/F1M61H5X datapack creating and managing data packages https://github.com/ropensci/datapack EML creating and editing EML metadata documents https://ropensci.github.io/EML arcticdatautils utility functions for processing data for the Arctic Data Center https://nceas.github.io/arcticdatautils/ datamgmt data management utilities for curating, documenting, and publishing data (sandbox package) https://nceas.github.io/datamgmt/ metadig authoring MetaDIG quality checks https://github.com/NCEAS/metadig-r metajam downloading and reading data and metadata from DataONE member nodes https://nceas.github.io/metajam/ "],["edit-data-packages.html", "Edit data packages ", " Edit data packages "],["datapack-background.html", "datapack Background", " datapack Background adapted from the dataone and datapack vignettes datapack is written differently than most R packages you may have encountered in the past. This is because it uses the S4 system instead. library(dataone) library(datapack) library(uuid) Data packages Data packages are a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. Navigating data packages Nodes Using this example on arcticdata.io d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) To use the member node information, use the mn slot d1c_test@mn To access the various slots using objects created by datapack and dataone (e.g. getSystemMetadata) requires the @ which is different from what you might have seen in the past. This is because these use the S4 system. Get an existing package from the Arctic Data Center. Make sure you know as you go through this training whether you are reading or writing to test or production. We don’t want to upload any of your test datasets to production! d1c &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) dp &lt;- dataone::getDataPackage(d1c, &quot;resource_map_urn:uuid:1f9eee7e-2d03-43c4-ad7f-f300e013ab28&quot;) Data Objects You can see what slots are in an S4 object after typing the subsetting operator @, or pressing TAB with the cursor after an existing @. Try viewing the slots of the data package. dp@ Check out the objects slot dp@objects The objects slot contains a list of object PIDs that are accessed using the $ subsetting operator. Both are found within the structure of data packages in R. Get the number for data and metadata files associated with this data package: getSize(dp) Get the file names and corresponding pids getValue(dp, name=&quot;sysmeta@fileName&quot;) Get identifiers You can search by any of the sysmeta slots such as fileName and formatId and get the corresponding identifier(s): metadataId &lt;- selectMember(dp, name=&quot;sysmeta@ADD THE NAME OF THE SLOT&quot;, value=&quot;PATTERN TO SEARCH BY&quot;) Example: selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;image/tiff&quot;) selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&quot;filename.csv&quot;) Provenance View the provenance as a dataTable. We will get into detail in the Building provenance chapter. dp@relations$relations "],["create-a-new-data-package.html", "Create a new data package", " Create a new data package adapted from the dataone and datapack vingettes library(dataone) library(datapack) library(uuid) Create a new data package - data package is a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. dp &lt;- new(&quot;DataPackage&quot;) Upload new data files Create and add a metadata file In this example we will use this previously written EML metadata. Here we are getting the file path from the dataone package and saving that as the object emlFile. This is a bit of an unusual way to reference a local file path, but all this does is looks within the R package dataone and grabs the path to a metadata document stored within that package. If you print the value of emlFile you’ll see it is just a file path, but it points to a special place on the server where that package is installed. Usually you will just reference EML paths that are stored within your user file system. emlFile &lt;- system.file(&quot;extdata/strix-pacific-northwest.xml&quot;, package=&quot;dataone&quot;) Create a new DataObject for the metadata and add it to the package. metadataObj &lt;- new(&quot;DataObject&quot;, format=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;, filename=emlFile) dp &lt;- addMember(dp, metadataObj) Check the dp object to see if the metadata was added correctly. dp Add some additional data files sourceData &lt;- system.file(&quot;extdata/OwlNightj.csv&quot;, package=&quot;dataone&quot;) sourceObj &lt;- new(&quot;DataObject&quot;, format=&quot;text/csv&quot;, filename=sourceData) dp &lt;- addMember(dp, sourceObj, metadataObj) # The third argument of addMember() associates the new DataObject to the metadata that was just added. If you want to change the formatId please use updateSystemMetadata (more on this later in the book) Upload the package d1c &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Make sure to give access privileges to the ADC admins: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) Get necessary token from test.arcticdata.io to upload the dataset prior uploading the datapackage: packageId &lt;- uploadDataPackage(d1c, dp, public=TRUE, accessRules=myAccessRules, quiet=FALSE) "],["update-packages-with-datapack.html", "Update packages with datapack", " Update packages with datapack Once you have updated the data objects and saved the metadata to a file, we can update the metadata and add the new pid to the resource map using datapack::updateDataObject(). This part of the training assumes that you have an EML file created from earlier parts of the training You must use these R packages for data packages that have provenance or folder hierarchy (arcticdatatutils does not support those features) Get the package Setting the node is done slightly differently: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Get a pre-existing package: packageId &lt;- &quot;the resource map&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) Update the metadata Get the metadata identifier metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) Take note of the EML version. If it is EML 2.1.1 the value needs to be changed to -“eml://ecoinformatics.org/eml-2.1.1” Read the EML doc doc &lt;- read_eml(getObject(d1c@mn, metadataId)) Edit the EML as usual - see the documents in Edit EML for details Once you are happy with your changes, you can update your data package to include the new metadata file using replaceMember eml_path &lt;- &quot;path/to/your/saved/eml.xml&quot; write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) Modify the data files remove zip files zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) add an existing data object dataObj &lt;- getDataObject(d1c_test, id=&quot;urn:uuid: here&quot;, lazyLoad=T, limit=&quot;1TB&quot;) dp &lt;- addMember(dp, dataObj, mo=metadataId) If you want to change the formatId please use updateSystemMetadata Publish update Using a DOI If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), (a) you need to change the public argument to TRUE and generate a DOI identifier. This should only be done after the package is finalized and has been thoroughly reviewed! Updating a package with a new DOI: A Digital Object Identifier (DOI) may be assigned to the metadata DataObject, using the generateIdentifier: doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) Updating a package with a pre-issued DOI dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;your pre-issued doi previously generated&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. "],["add-a-pre-generated-identifier-to-the-eml.html", "Add a pre-generated identifier to the EML", " Add a pre-generated identifier to the EML When you pre-generate a UUID or DOI, the change is not automatically reflected in the packageId section of the EML. Use the code below to ensure that the EML lines up with the desired identifier: ## Generate DOI and add to EML # Note that you cannot generate a DOI on test nodes doiPid &lt;- generateIdentifier(mn, &quot;DOI&quot;) doc$packageId &lt;- doiPid Be sure to include the identifier= argument in your publish update command so the pre-generated identifier is applied. "],["edit-sysmeta.html", "Edit sysmeta", " Edit sysmeta To edit the sysmeta of an object (data file, EML, or resource map, etc.) with a PID, first load the sysmeta into R using the following command: sysmeta &lt;- getSystemMetadata(d1c_test@mn, pid) Then edit the sysmeta slots by using @ functionality. For example, to change the fileName use the following command: sysmeta@fileName &lt;- &#39;NewFileName.csv&#39; Note that some slots cannot be changed by simple text replace (particularly the accessPolicy). There are various helper functions for changing the accessPolicy and rightsHolder such as datapack::addAccessRule() (which takes the sysmeta as an input) or arcticdatautils::set_rights_and_access(), which only requires a PID. In general, you most frequently need to use dataone::getSystemMetadata() to change either the formatId or fileName slots (see the DataONE list of format ids) for acceptable formats. # Example of setting the formatId slot sysmeta@formatId &lt;- &quot;eml://ecoinformatics.org/eml-2.1.1&quot; After you have changed the necessary slot, you can update the system metadata using the following command: updateSystemMetadata(mn, pid, sysmeta) Identifiers and sysmeta Importantly, changing the system metadata does NOT necessitate a change in the PID of an object. This is because changes to the system metadata do not change the object itself, they are only changing the description of the object (although ideally the system metadata are accurate when an object is first published). Additional resources For a more in-depth (and technical) guide to sysmeta, check out the DataONE documentation: System Metadata Data Types in CICore "],["obsolescence-chain.html", "Obsolescence chain", " Obsolescence chain You can obsolete a dataset using the function datamgmt::obsolete_package(). Use the documentation for instructions on using the function. The following workflow explains how the functions operate. This chunk is to obsolete one data set. If there are more to add to the chain, more steps can be added. Be very careful. Make sure to fill in obsoletes and obsoletedBy slots for each one. The obsoletes and obsoletedBy fields must be NA, once they are populated they can’t be modified. # get oldest version of the file you want to be visible. Use get_all_versions and look at the latest. # urn:uuid:... # PID for data set to be obsoleted (hidden): doi:10… # adding data set to obsolete (hide) in the slot before the first version of the visible data set sysmeta1 &lt;- getSystemMetadata(mn, &quot;urn:uuid:example_pid&quot;) sysmeta1@obsoletes &lt;- &quot;doi:10.example_doi&quot; updateSystemMetadata(mn, &quot;urn:uuid:example_pid&quot;, sysmeta1) # adding first version to obsolescence chain after obsoleted (hidden) version sysmeta0 &lt;- getSystemMetadata(mn, &quot;doi:10.example_doi&quot;) sysmeta0@obsoletedBy &lt;- &quot;urn:uuid:example_pid&quot; updateSystemMetadata(mn, &quot;doi:10.example_doi&quot;, sysmeta0) The following code is equivalent to the code chunk above. This method is recommended, however it is necessary to read the function documentation first. datamgmt::obsolete_package(mn, metadata_obsolete = &quot;doi:10.example_doi&quot;, metadata_new = &quot;urn:uuid:example_pid&quot;) "],["set-dataone-nodes.html", "Set DataONE nodes", " Set DataONE nodes DataONE is a network of data repositories that is structured with coordinating nodes (CN) and member nodes (MN). The network tree looks something like this: At the top level is DataONE itself. Within DataONE there are several coordinating nodes, including nodes for both production material and testing material. Within these coordinating nodes are many member nodes, including ones for both the Arctic Data Center and the KNB. To set the environment in which you want to publish data, you need to set both the coordinating node and the member node. For example, if you are publishing to the Arctic Data Center test site, you would want to set the coordinating node to STAGING and the member node to mnTestArctic. A note on nodes - be very careful about what you publish on production nodes (PROD, or arcticdata.io). These nodes should NEVER be used to publish test or training data sets. The primary nodes we work on, and how to set them in R, are below: Staging (Test) nodes # ADC (test.arcticdata.io) cn_staging &lt;- CNode(&#39;STAGING&#39;) adc_test &lt;- getMNode(cn_staging,&#39;urn:node:mnTestARCTIC&#39;) # KNB (dev.nceas.ucsb.edu) cn_staging2 &lt;- CNode(&quot;STAGING2&quot;) knb_test &lt;- getMNode(cn_staging2, &quot;urn:node:mnTestKNB&quot;) Production nodes # ADC (arcticdata.io) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn,&#39;urn:node:ARCTIC&#39;) # KNB (knb.ecoinformatics.org) knb &lt;- getMNode(cn, &quot;urn:node:KNB&quot;) # GOA goa &lt;- getMNode(cn, &quot;urn:node:GOA&quot;) # You can also use the datamgmt::guess_membernode function to set a member node # Note: this pid looks like a URL - it&#39;s really a unique identifier dryad &lt;- datamgmt::guess_member_node(&#39;https://doi.org/10.5061/dryad.k6gf1tf/15?ver=2018-09-18T03:54:10.492+00:00&#39;) More DataONE STAGING nodes can be found here More DataONE PROD nodes can be found here "],["set-rights-and-access.html", "Set rights and access", " Set rights and access One final step when creating/updating packages is to make sure that the rights and access on all the objects that were uploaded are set correctly within the sysmeta. The function arcticdatautils::set_rights_and_access() will set both, and arcticdatautils::set_access() will just set access. There are two functions for this because a rightsHolder should always have access, but not all people who need access are rightsHolders. The rightsHolder of the data package is typically the submitter (if the data set is submitted through the web form (“editor”)), but if a data team member is publishing objects for a PI, the rightsHolder should be the main point of contact for the data set (i.e. the person who requested that we upload the data for them). To set the rights and access for all of the objects in a package, first get the ORCiD of the person to whom you are giving rights and access. You can set this manually, or grab it from one of the creators in an EML file. You can look up ORCID iDs here # Manually set ORCiD subject &lt;- &#39;http://orcid.org/PUT-YOUR-ORCD-HERE&#39; # Set ORCiD from EML creator # if only 1 creator exists subject &lt;- doc$dataset$creator$userId$userId # if more than 1 creator exists and you want the first one subject &lt;- doc$dataset$creator[[1]]$userId$userId # As a convention we use `http:` instead of `https:` in our system metadata subject &lt;- sub(&quot;^https://&quot;, &quot;http://&quot;, subject) Note, when setting metadata, the ORCiD must start with http://. ORCiDs in EML should start with https://. The sub() command above will change this formatting for you. Next, set the rights and access using the following command: set_rights_and_access(mn, pids = c(pkg$metadata, pkg$data, pkg$resource_map), subject = subject, permissions = c(&#39;read&#39;,&#39;write&#39;,&#39;changePermission&#39;)) If you ever need to remove/add public access to your package or object, you can use remove_public_read() or set_public_read(), respectively. Making files publicly readable is especially useful when downloading large amounts of files to the server in order to to use metadata helper functions that require a file path (ex: eml_get_raster_metadata() and get_ncdf4_attributes()). remove_public_read(mn, c(pkg$metadata, pkg$data, pkg$resource_map)) My profile The datasets that render under a user’s profile page like here are added if one of the following three System Metadata fields exists. The subject is the rightsHolder or the subject has one of either write or changePermission in the accessPolicy. If you ever need to remove a subject from the accessPolicy or update the rightsHolder you can use arcticdatautils::remove_access and arcticdatautils::set_rightsHolder, respectively. "],["show-indexing-status.html", "Show indexing status", " Show indexing status Sometimes it takes awhile for the website to render with the updates you’ve made in R. To check whether a PID has been indexed yet, use: show_indexing_status(mn, pid) The status bar will either show 0% (not indexed) or 100% (should be online already). "],["preserve-folder-structures.html", "Preserve folder structures", " Preserve folder structures Sometimes, researchers will upload zip files that a contain nested file and folder structure that we would like to maintain. This reference section will walk you through how to re-upload the contents of the zip file to the Arctic Data Center such that the files and folders are preserved. Note that changing the locations of the files within the package can be tricky, so take these steps with care and try to make sure it is done correctly. With that, here are the steps assuming that the PI has uploaded one zip file to their dataset. You may need to modify the steps for other scenarios, but if you are not sure, feel free to ask the data coordinator. In particular, tar files (.tgz) or rar files (.rar) are also compressed archives that might be better to unpack using command line tools. Download the zip file to datateam First, we will download the file, using R. Navigate to the dataset landing page on the Arctic Data Center Right click the “download” button next to the zip file Select “copy link address” Run the following two lines of code to set the URL variable, and extract the pid Here is an example on the test site. Note that on production, you will need to change the URL that you are substituting in the second line of code. library(magrittr) url &lt;- &quot;https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A48c0e669-fd8a-4875-acfd-e8933bb350ed&quot; pid &lt;- gsub(&quot;https://test.arcticdata.io/metacat/d1/mn/v2/object/&quot;, &quot;&quot;, url) %&gt;% gsub(&quot;%3A&quot;, &quot;:&quot;, .) Download the file Note that this will download the file to the location you specify as the second argument in writeBin. If you are organizing your scripts and data by submitter, it would look like the example below. writeBin(getObject(d1c@mn, pid), &quot;~/Submitter/example.zip&quot;) Unzip the file into your submitter directory. unzip(&quot;~/Submitter/example.zip&quot;, exdir = &quot;~/Submitter&quot;) Delete the zip file (example.zip) Now if you look at the directory, you shuold see the unzipped contents of the file in a subdirectory of ~/Submitter. The name of the directory will be the name of the folder the PI created the archive from. Right now, you should stop and examine each file in the directory closely (or each type of file). You may need to make some minor adjustments or ask for clarification from the PI. For example, we still may need to ask for csv versions of excel files, you may need to re-zip certain directories (for example: a zip which contains 5 different sets of shapefiles should be turned into 5 different zips). Evaluate the contents of the directory alongside the data coordinator. Re-upload the contents to the Arctic Data Center Once you have confirmed everything is all good, we can upload the files to the ADC while preserving the directory structure. First, get the data package loaded into your R session as usual. I recommend not attempting any EML edits while you do these steps, this update adding the files is best done on it’s own, and EML edits can be done on the next version. dp &lt;- getDataPackage(d1c, &quot;identifier&quot;) Next, we will set up two types of paths describing each of the objects. The first will be an absolute path, so we can be sure the R function finds the files. The second will be a relative path, which will be what shows up on the landing page (or in the download all result) of the data package. If you don’t know the difference between an absolute and relative path, read on. It is SUPER IMPORTANT! A path is a location of a file or folder on a computer. There are two types of paths in computing: absolute paths and relative paths. An absolute path always starts with the root of your file system and locates files from there. The absolute path to my example submitter zip is: /home/jclark/Submitter/example.zip. The generic shortcut ~/ is often used to replace the location of your home directory (/home/username) to save typing, but your path is still an absolute path if it starts with ~/. Note that a relative path will always start with either ~/ or /. Relative paths start from some location in your file system that is below the root. Relative paths are combined with the path of that location to locate files on your system. R (and some other languages like MATLAB) refer to the location where the relative path starts as our working directory. If our working directory is set to ~/Submitter, the relative path to the zip would be just example.zip. Note that a relative path will never start with either ~/ or /. Getting these paths right is very important because we don’t want submitters to download a folder of data, and have the paths look like /home/internname/ticket_27341/important_folder/important_file.csv. The first part of that path is particular to however the person processing the ticket organized the data, and is not how the submitter of the data intented to organize the data. Follow the steps below to make sure this does not happen. get a list of absolute paths for each file in the directory. NOTE The “PI_dir_name” here represents whatever directory you retrieved after running unzip in the previous step. The actual .zip file should not be in this directory. get a list of relative paths for each file in the directory. Note this is the same command, but with the argument full.names set to FALSE. abs_paths &lt;- list.files(&quot;~/Submitter&quot;, full.names = TRUE, recursive = TRUE) rel_paths &lt;- list.files(&quot;~/Submitter&quot;, full.names = FALSE, recursive = TRUE) Make sure that these paths look correct! They should contain ONLY the files that were unzipped. If you have other scripts or metadata files you might want to rearrange your directories to get the correct paths. The relative paths should start with the submitter’s directory name. In this example they will look like the below: “final_image_set/level1.png” “final_image_set/photos/level2_1.png” “final_image_set/photos/level2_2.png” Now for each of these files, we can create a dataObject for them and add them to the package using a loop. Before running this, look at the values of your abs_paths and rel_paths and make sure they look correct based on what you know about both paths and the structure of the directory. Within this loop we will also create otherEntities for each item, just putting in the bare minimum of information that will help us make sure that we know what files are what. metadataObj &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) doc &lt;- read_eml(getObject(d1c@mn, metadataObj)) doc$dataset$otherEntity &lt;- NULL oes &lt;- list() for (i in 1:length(abs_paths)){ formatId &lt;- arcticdatautils::guess_format_id(abs_paths[i]) id &lt;- generateIdentifier(d1c@mn, scheme = &quot;uuid&quot;) dataObj &lt;- new(&quot;DataObject&quot;, format = formatId, filename = abs_paths[i], targetPath = rel_paths[i], id = id) dp &lt;- addMember(dp, dataObj, metadataObj) oes[[i]] &lt;- eml$otherEntity(entityName = rel_paths[i], entityType = formatId, id = id) } doc$dataset$otherEntity &lt;- oes write_eml(doc, &quot;~/metadata.xml&quot;) dp &lt;- replaceMember(dp, metadataObj, replacement=&quot;~/metadata.xml&quot;) Once this is finished you can examine the relationships by running View(dp@relations$relations). If everything worked out well you should see rows that look like this: urn:uuid:a398312d-2c87-4b19-8380-3f11d2a1922d | http://www.w3.org/ns/prov#atLocation | figure2/figure2.tif | The targetPath (3rd column) should NOT look like urn:uuid:a398312d-2c87-4b19-8380-3f11d2a1922d | http://www.w3.org/ns/prov#atLocation | /home/jclark/ArcticSupport/Zhao/dataset1/figure2/figure2.tif | Make sure this looks correct, then update the data package with: uploadDataPackage(d1c, dp, public = FALSE) Finally, check your work. Go to the Arctic Data Center and see if the package displays correctly. Edit the package using the user interface to remove the zip file. Continue with metadata updates as normal. Summary Here is the example code all put together. Make sure you change all of the relevant bits, and check your work carefully!! url &lt;- &quot;https://arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A8fee5046-1a8f-4ccc-80f2-70c557a66338&quot; pid &lt;- gsub(&quot;https://arcticdata.io/metacat/d1/mn/v2/object/&quot;, &quot;&quot;, url) %&gt;% gsub(&quot;%3A&quot;, &quot;:&quot;, .) writeBin(getObject(d1c@mn, pid), &quot;~/Submitter/example.zip&quot;) unzip(&quot;~/Submitter/example.zip&quot;, exdir = &quot;~/Submitter&quot;) dp &lt;- getDataPackage(d1c, &quot;identifier&quot;) abs_paths &lt;- list.files(&quot;~/Submitter/PI_dir_name/&quot;, full.names = TRUE, recursive = TRUE) rel_paths &lt;- list.files(&quot;~/Submitter/PI_dir_name/&quot;, full.names = FALSE, recursive = TRUE) metadataObj &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) for (i in 1:length(abs_paths)){ formatId &lt;- arcticdatautils::guess_format_id(abs_paths[i]) dataObj &lt;- new(&quot;DataObject&quot;, format = formatId, filename = abs_paths[i], targetPath = rel_paths[i]) dp &lt;- addMember(dp, dataObj, metadataObj) } "],["update-a-package-with-a-new-data-object.html", "Update a package with a new data object", " Update a package with a new data object Once you have updated the data objects and saved the metadata to a file, we can update the metadata and use replaceMember to update the package with the new metadata. Make sure you have the package you want to update loaded into R using dataone::getDataPackage(). Publish update Now we can update your data package to include the new data object. Assuming you have updated your data package earlier something like the below: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) packageId &lt;- &quot;resource_map_urn:uuid...&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #some modification to the EML here eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) You can then upload your data package: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c_test, dp, public=FALSE, accessRules=myAccessRules, quiet=FALSE) If a package is ready to be public, you can change the public argument in the datapack::uploadDataPackage() call to TRUE. If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), you need to do this when replacing the metadata using the optional newId argument in replaceMember(). This should only be done after the package is finalized and has been thoroughly reviewed! doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If there is a pre-issued DOI (researcher requested the DOI for the publication first), please do the following: dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;doi:10.../...&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If the package has children, see how to do this using arcticdatautils::publish_update in the nesting section of the reference manual. Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. "],["update-an-object.html", "Update an object", " Update an object To update a data file associated with a data package, you need to do three things: update the object itself, update the resource map (which affiliates the object with the metadata), and update the metadata that describes that object The datapack::replaceMember function takes care of the first two of these tasks. First you need to get the pid of the file you want to replace by using datapack::selectMember # Select the EML PID dataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;text/csv&quot;) Then use replaceMember: # Replace the old file with the new dp &lt;- replaceMember(dp, dataId, replacement=file_path) If you want to remove some files from the data package we can use datapack::removeMember. If we wanted to remove all the zip files associated with this data package, we can use datapack::removeMember: zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) You will need to be explicit about your format_id here based on the file type. A list of format IDs can be found here on the DataONE website. Use line 2 (Id:) exactly, character for character. To accomplish the second task, you will need to update the metadata using the EML package. This is covered in Chapter 4. After you update a file, you will always need to update the metadata because parts of the physical section (such as the file size, checksum) will be different, and it may also require different attribute information. Once you have updated your metadata and saved it, you can update the package itself. "],["explore-eml.html", "Explore EML ", " Explore EML "],["access-specific-elements.html", "Access specific elements", " Access specific elements The eml_get() function is a powerful tool for exploring EML (more on that here ). It takes any chunk of EML and returns all instances of the element you specify. Note: you’ll have to specify the element of interest exactly, according to the spelling/capitalization conventions used in EML. Here are some examples: doc &lt;- read_eml(system.file(&quot;example-eml.xml&quot;, package = &quot;arcticdatautils&quot;)) eml_get(doc, &quot;creator&quot;) individualName: givenName: Bryce surName: Mecum organizationName: National Center for Ecological Analysis and Synthesis eml_get(doc, &quot;boundingCoordinates&quot;) eastBoundingCoordinate: &#39;-134&#39; northBoundingCoordinate: &#39;59&#39; southBoundingCoordinate: &#39;57&#39; westBoundingCoordinate: &#39;-135&#39; eml_get(doc, &quot;url&quot;) &#39;&#39;: function: download url: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 &#39;&#39;: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 eml_get_simple() is a simplified alternative to eml_get() that produces a list of the desired EML element. eml_get_simple(doc$dataset$otherEntity, &quot;entityName&quot;) To find an eml element you can use either a combination of which_in_emlfrom the arcticdatautils package or eml_get_simple and which to find the index in an EML list. Use which ever workflow you see fit. An example question you may have: Which creators have a surName “Mecum”? Example using which_in_eml: n &lt;- which_in_eml(doc$dataset$creator, &quot;surName&quot;, &quot;Mecum&quot;) # Answer: doc$dataset$creator[[n]] Example using eml_get_simple and which: ent_names &lt;- eml_get_simple(doc$dataset$creator, &quot;surName&quot;) i &lt;- which(ent_names == &quot;Mecum&quot;) # Answer: doc$dataset$creator[[i]] "],["navigate-through-eml.html", "Navigate through EML", " Navigate through EML The first task when editing an EML file is navigating the EML file. An EML file is organized in a structure that contains many lists nested within other lists. The function View allows you to get a crude view of an EML file in the viewer. It can be useful for exploring the file. # Need to be in this member node to explore file d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) doc &lt;- read_eml(getObject(d1c_test@mn, &quot;urn:uuid:558eabf1-1e91-4881-8ba3-ef8684d8f6a1&quot;)) View(doc) The complex EML document is represented in R as as series of named, nested lists. We use lists all the time in R! A data.frame is one example of a special kind of list that we use all the time. You may be familiar with the syntax dataframe$column_name which allows us to select a particular column of a data.frame. Under the hood, a data.frame is a named list of vectors with the same length. You select one of those vectors using the $ operator, which is called the “list selector operator.” Just like you navigate in a data.frame, you can use the $ operator to navigate through the EML structure. The $ operator allows you to go deeper into the EML structure and to see what elements are nested within other elements. However, you have to tell R where you want to go in the structure when you use the $ symbol. For example, if you want to view the dataset element of your EML you would use the command doc$dataset. If you want to view the creators of your data set you would use doc$dataset$creator. Note here that creator is contained within dataset. If you aren’t sure where you want to go, hit the tab button on your keyboard after typing $ and a list of available elements in the structure will appear (e.g., doc$&lt;TAB&gt;): Note that if you hit tab, and nothing pops up, this most likely implies that you are trying to go into an EML element that can take a series items. For example doc$dataset$creator$&lt;TAB&gt; will not show a pop-up menu. This is because creator is a series-type object (i.e. you can have multiple creators). If you want to go deeper into creator, you first must tell R which creator you are interested in. Do this by writing [[i]] first where i is the index of the creator you are concerned with. For example, if you want to look at the first creator i = 1. Now doc$dataset$creator[[1]]$&lt;TAB&gt; will give you many more options. Note, an empty autocomplete result sometimes means you have reached the end of a branch in the EML structure. Below is the structure of doc$dataset. There are a series of multiple creators, which can be accessed individually by index: doc$dataset$creator[[#]]. At this point stop and take a deep breath. The key takeaway is that EML is a hierarchical tree structure. The best way to get familiar with it is to explore the structure. Try entering doc$dataset into your console, and print it. Now make the search more specific, for instance: doc$dataset$abstract. "],["understand-the-eml-schema.html", "Understand the EML schema", " Understand the EML schema Another great resource for navigating the EML structure is looking at the schema which defines the structure. The schema diagrams on this page are interactive. Further explanations of the symbology can be found here. The schema is complicated and may take some time to get familiar with before you will be able to fully understand it. Use your browser’s “search in page” function (usually CTRL-F or Command-F) to navigate the EML schema page quickly. For example, let’s take a look at eml-party. To start off, notice that some elements have bolded lines leading to them. A bold line indicates that the element is required if the element above it (to the left in the schema) is used, otherwise the element is optional. Notice also that next to the givenName element it says “0..infinity”. This means that the element is unbounded — a single party can have many given names and there is no limit on how many you can add. However, this text does not appear for the surName element — a party can have only one surname. You will also see icons linking the EML slots together, which indicate the ordering of subsequent slots. These can indicate either a “sequence” or a “choice”. In our example from eml-party, a “choice” icon indicates that either an individualName, organizationName, or positionName is required, but you do not need all three. However, the “sequence” icon tells us that if you use an individualName, you must include the surName as a child element. If you include the optional child elements salutation and givenName, they must be written in the order presented in the schema. The eml schema sections you may find particularly helpful include eml-party, eml-attribute and eml-physical. For a more detailed description of the EML schema, see the reference section on exploring EML. "],["edit-eml.html", "Edit EML ", " Edit EML "],["edit-an-eml-element.html", "Edit an EML element", " Edit an EML element There are multiple ways to edit an EML element. Edit EML with strings The most basic way to edit an EML element would be to navigate to the element and replace it with something else. Easy! For example, to change the title one could use the following command: doc$dataset$title &lt;- &quot;New Title&quot; If the element you are editing allows for multiple values, you can pass it a list of character strings. Since a dataset can have multiple titles, we can do this: doc$dataset$title &lt;- list(&quot;New Title&quot;, &quot;Second New Title&quot;) However, this isn’t always the best method to edit the EML, particularly if the element has sub-elements. Adding directly to doc without a helper function can overwrite these parts of the doc that we need. Edit EML with the “EML” package To edit a section where you are not 100% sure of the sub-elements, using the eml$elementName() helper functions from the EML package will pre-populate the options for you if you utilize the RStudio autocomplete functionality. The arguments in these functions show the available slots for any given EML element. For example, typing doc$dataset$abstract &lt;- eml$abstract()&lt;TAB&gt; will show you that the abstract element can take either the section or para sub-elements. doc$dataset$abstract &lt;- eml$abstract(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) This inserts the abstract with a para element in our dataset, which we know from the EML schema is valid. Note that the above is equivalent to the following generic construction: doc$dataset$abstract &lt;- list(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) The eml() family of functions provides the sub-elements as arguments, which is extremely helpful, but functionally all it is doing is creating a named list, which you can also do using the list function. Edit EML with objects A final way to edit an EML element would be to build a new object to replace the old object. To begin, you might create an object using an eml helper function. Let’s take keywords as an example. Sometimes keyword lists in a metadata record will come from different thesauruses, which you can then add in series (similar to the way we added multiple titles) to the element keywordSet. We start by creating our first set of keywords and saving it to an object. kw_list_1 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER controlled vocabulary&quot;, keyword = list(&quot;bacteria&quot;, &quot;carnivorous plants&quot;, &quot;genetics&quot;, &quot;thresholds&quot;)) Which returns: $keyword $keyword[[1]] [1] &quot;bacteria&quot; $keyword[[2]] [1] &quot;carnivorous plants&quot; $keyword[[3]] [1] &quot;genetics&quot; $keyword[[4]] [1] &quot;thresholds&quot; $keywordThesaurus [1] &quot;LTER controlled vocabulary&quot; We create the second keyword list similarly: kw_list_2 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER core area&quot;, keyword = list(&quot;populations&quot;, &quot;inorganic nutrients&quot;, &quot;disturbance&quot;)) Finally, we can insert our two keyword lists into our EML document just like we did with the title example above, but rather than passing character strings into list(), we will pass our two keyword set objects. doc$dataset$keywordSet &lt;- list(kw_list_1, kw_list_2) Note that you must use the function list here and not the c() function. The reasons for this are complex, and due to some technical subtlety in R - but the gist of the issue is that the c() function can behave in unexpected ways with nested lists, and frequently will collapse the nesting into a single level, resulting in invalid EML. "],["edit-attributelists.html", "Edit attributeLists", " Edit attributeLists Attributes are descriptions of variables, typically columns or column names in tabular data. Attributes are stored in an attributeList. When editing attributes in R, we convert the attribute list information to data frame (table) format so that it is easier to edit. When editing attributes you will need to create one to three data frame objects: A data.frame of attributes A data.frame of custom units (if applicable) A data.frame of factors (if applicable) The attributeList is an element within one of 4 different types of entity objects. An entity corresponds to a file, typically. Multiple entities (files) can exist within a dataset. The 4 different entity types are dataTable (most common for us), spatialVector, spatialRaster, and otherEntity. Please note that submitting attribute information through the website will store them in an otherEntity object by default. We prefer to store them in a dataTable object for tabular data or a spatialVector or spatialRaster object for spatial data. To edit or examine an existing attribute list already in an EML file, you can use the following commands, where i represents the index of the series element you are interested in. Note that if there is only one item in the series (ie there is only one dataTable), you should just call doc$dataset$dataTable, as in this case doc$dataset$dataTable[[1]] will return the first sub-element of the dataTable (the entityName) # If they are stored in an otherEntity (submitted from the website by default) attribute_tables &lt;- EML::get_attributes(doc$dataset$otherEntity[[i]]$attributeList) # Or if they are stored in a dataTable (usually created by a datateam member) attribute_tables &lt;- EML::get_attributes(doc$dataset$dataTable[[i]]$attributeList) The get_attributes() function returns the attribute_tables object, which is a list of the three data frames mentioned above. The data frame with the attributes is called attribute_tables$attributes. attribute_tables$attributes print(attribute_tables$attributes) Edit attributes Attribute information should be stored in a data.frame with the following columns: attributeName: The name of the attribute as listed in the csv. Required. e.g.: “c_temp” attributeLabel: A descriptive label that can be used to display the name of an attribute. It is not constrained by system limitations on length or special characters. Optional. e.g.: “Temperature (Celsius)” attributeDefinition: Longer description of the attribute, including the required context for interpreting the attributeName. Required. e.g.: “The near shore water temperature in the upper inter-tidal zone, measured in degrees Celsius.” measurementScale: One of: nominal, ordinal, dateTime, ratio, interval. Required. nominal: unordered categories or text. e.g.: (Male, Female) or (Yukon River, Kuskokwim River) ordinal: ordered categories. e.g.: Low, Medium, High dateTime: date or time values from the Gregorian calendar. e.g.: 01-01-2001 ratio: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second. interval: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or latitude and longitude coordinates, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius. domain: One of: textDomain, enumeratedDomain, numericDomain, dateTime. Required. textDomain: text that is free-form, or matches a pattern enumeratedDomain: text that belongs to a defined list of codes and definitions. e.g.: CASC = Cascade Lake, HEAR = Heart Lake dateTimeDomain: dateTime attributes numericDomain: attributes that are numbers (either ratio or interval) formatString: Required for dateTime, NA otherwise. Format string for dates, e.g. “DD/MM/YYYY”. definition: Required for textDomain, NA otherwise. Defines a format for attributes that are a character string. e.g.: “Any text” or “7-digit alphanumeric code” unit: Required for numericDomain, NA otherwise. Unit string. If the unit is not a standard unit, a warning will appear when you create the attribute list, saying that it has been forced into a custom unit. Use caution here to make sure the unit really needs to be a custom unit. A list of standard units can be found using: standardUnits &lt;- EML::get_unitList() then running View(standardUnits$units). numberType: Required for numericDomain, NA otherwise. Options are real, natural, whole, and integer. real: positive and negative fractions and integers (…-1,-0.25,0,0.25,1…) natural: non-zero positive integers (1,2,3…) whole: positive integers and zero (0,1,2,3…) integer: positive and negative integers and zero (…-2,-1,0,1,2…) missingValueCode: Code for missing values (e.g.: ‘-999’, ‘NA’, ‘NaN’). NA otherwise. Note that an NA missing value code should be a string, ‘NA’, and numbers should also be strings, ‘-999.’ missingValueCodeExplanation: Explanation for missing values, NA if no missing value code exists. You can create attributes manually by typing them out in R following a workflow similar to the one below: attributes &lt;- data.frame( attributeName = c(&#39;Date&#39;, &#39;Location&#39;, &#39;Region&#39;,&#39;Sample_No&#39;, &#39;Sample_vol&#39;, &#39;Salinity&#39;, &#39;Temperature&#39;, &#39;sampling_comments&#39;), attributeDefinition = c(&#39;Date sample was taken on&#39;, &#39;Location code representing location where sample was taken&#39;,&#39;Region where sample was taken&#39;, &#39;Sample number&#39;, &#39;Sample volume&#39;, &#39;Salinity of sample in PSU&#39;, &#39;Temperature of sample&#39;, &#39;comments about sampling process&#39;), measurementScale = c(&#39;dateTime&#39;, &#39;nominal&#39;,&#39;nominal&#39;, &#39;nominal&#39;, &#39;ratio&#39;, &#39;ratio&#39;, &#39;interval&#39;, &#39;nominal&#39;), domain = c(&#39;dateTimeDomain&#39;, &#39;enumeratedDomain&#39;,&#39;enumeratedDomain&#39;, &#39;textDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;textDomain&#39;), formatString = c(&#39;MM-DD-YYYY&#39;, NA,NA,NA,NA,NA,NA,NA), definition = c(NA,NA,NA,&#39;Six-digit code&#39;, NA, NA, NA, &#39;Any text&#39;), unit = c(NA, NA, NA, NA,&#39;milliliter&#39;, &#39;dimensionless&#39;, &#39;celsius&#39;, NA), numberType = c(NA, NA, NA,NA, &#39;real&#39;, &#39;real&#39;, &#39;real&#39;, NA), missingValueCode = c(NA, NA, NA,NA, NA, NA, NA, &#39;NA&#39;), missingValueCodeExplanation = c(NA, NA, NA,NA, NA, NA, NA, &#39;no sampling comments&#39;)) However, typing this out in R can be a major pain. Luckily, there’s a Shiny app that you can use to build attribute information. You can use the app to build attributes from a data file loaded into R (recommended as the app will auto-fill some fields for you) to edit an existing attribute table, or to create attributes from scratch. Use the following commands to create or modify attributes. Use the following commands to create or modify attributes. These commands will launch a “Shiny” app in your web browser. #first download the CSV in your data package from Exercise #2 data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;.csv&quot;) data &lt;- read.csv(text=rawToChar(getObject(d1c_test@mn, data_pid))) # From data (recommended) attribute_tables &lt;- EML::shiny_attributes(data = data) # From scratch attribute_tables &lt;- EML::shiny_attributes() # From an existing attribute list attribute_tables &lt;- get_attributes(doc$dataset$dataTable[[i]]$attributeList) attribute_tables &lt;- EML::shiny_attributes(attributes = attribute_tables$attributes) Once you are done editing a table in the browser app, quit the app by pressing the red “Quit App” button in the top right corner of the page. If you close the Shiny app tab in your browser instead of using the “Quit App” button, your work will not be saved, R will think that the Shiny app is still open, and you will not be able to run other code. You can tell if R is confused if you have closed the Shiny app and the bottom line in the console still says Listening on http://.... If this happens, press the red stop sign button on the right hand side of the console window in order to interrupt R. The tables you constructed in the app will be assigned to the attribute_tables variable as a list of data frames (one for attributes, factors, and units). Be careful to not overwrite your completed attribute_tables object when trying to make edits. The last line of code can be used in order to make edits to an existing attribute_tables object. Alternatively, each table can be to exported to a csv file by clicking the Download button. If you downloaded the table, read the table back into your R session and assign it to a variable in your script (e.g. attributes &lt;- data.frame(...)), or just use the variable that shiny_attributes returned. For simple attribute corrections, datamgmt::edit_attribute() allows you to edit the slots of a single attribute within an attribute list. To use this function, pass an attribute through datamgmt::edit_attribute() and fill out the parameters you wish to edit/update. An example is provided below where we are changing attributeName, domain, and measurementScale in the first attribute of a dataset. After completing the edits, insert the new version of the attribute back into the EML document. new_attribute &lt;- datamgmt::edit_attribute(doc$dataset$dataTable[[1]]$attributeList$attribute[[1]], attributeName = &#39;date_and_time&#39;, domain = &#39;dateTimeDomain&#39;, measurementScale = &#39;dateTime&#39;) doc$dataset$dataTable[[1]]$attributeList$attribute[[1]] &lt;- new_attribute Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) Search the units list for your unit before attempting to create a custom unit. You can search part of the unit you can look up part of the unit ie meters in the table to see if there are any matches. If you have units that are not in the standard EML unit list, you will need to build a custom unit list. Attribute tables with custom units listed will generate a warning indicating that a custom unit will need to be described. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”). The parentSI does not need to be part of the unitList. multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;siemensPerMeter&#39;, &#39;decibar&#39;), unitType = c(&#39;resistivity&#39;, &#39;pressure&#39;), parentSI = c(&#39;ohmMeter&#39;, &#39;pascal&#39;), multiplierToSI = c(&#39;1&#39;,&#39;10000&#39;), abbreviation = c(&#39;S/m&#39;,&#39;decibar&#39;), description = c(&#39;siemens per meter&#39;, &#39;decibar&#39;)) Custom units can also be created in the shiny app, under the “units” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() custom_units &lt;- attribute_tables$units Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- list(metadata = list(unitList = unitlist)) Edit factors For attributes that are enumeratedDomains, a table is needed with three columns: attributeName, code, and definition. attributeName should be the same as the attributeName within the attribute table and repeated for all codes belonging to a common attribute. code should contain all unique values of the given attributeName that exist within the actual data. definition should contain a plain text definition that describes each code. To build factors by hand, you use the named character vectors and then convert them to a data.frame as shown in the example below. In this example, there are two enumerated domains in the attribute list - “Location” and “Region”. Location &lt;- c(CASC = &#39;Cascade Lake&#39;, CHIK = &#39;Chikumunik Lake&#39;, HEAR = &#39;Heart Lake&#39;, NISH = &#39;Nishlik Lake&#39; ) Region &lt;- c(W_MTN = &#39;West region, locations West of Eagle Mountain&#39;, E_MTN = &#39;East region, locations East of Eagle Mountain&#39;) The definitions are then written into a data.frame using the names of the named character vectors and their definitions. factors &lt;- rbind(data.frame(attributeName = &#39;Location&#39;, code = names(Location), definition = unname(Location)), data.frame(attributeName = &#39;Region&#39;, code = names(Region), definition = unname(Region))) Factors can also be created in the shiny app, under the “factors” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() attribute_tables$factors Finalize attributeList Once you have built your attributes, factors, and custom units, you can add them to EML objects. Attributes and factors are combined to form an attributeList using set_attributes(): # Create an attributeList object attributeList &lt;- EML::set_attributes(attributes = attribute_tables$attributes, factors = attribute_tables$factors) This attributeList object can then be checked for errors and added to a dataTable in the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList "],["edit-custom-units-1.html", "Edit custom units", " Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) If you have units that are not in the standard EML unit list, you will need to build a custom unit list. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”) multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;partsPerThousand&#39;, &#39;decibar&#39;, &#39;wattsPerSquareMeter&#39;, &#39;micromolesPerGram&#39;, &#39;practicalSalinityUnit&#39;), unitType = c(&#39;dimensionless&#39;, &#39;pressure&#39;, &#39;power&#39;, &#39;amountOfSubstanceWeight&#39;， &#39;dimensionless&#39;), parentSI = c(NA, &#39;pascal&#39;, &#39;watt&#39;, &#39;molesPerKilogram&#39;， NA), multiplierToSI = c(NA, &#39;10000&#39;, &#39;1&#39;, &#39;1000000000&#39;, NA), abbreviation = c(&#39;ppt&#39;, &#39;decibar&#39;, &#39;W/m^2&#39;, &#39;umol/g&#39;, &#39;PSU&#39;), description = c(&#39;parts per thousand&#39;, &#39;decibar&#39;, &#39;watts per square meter&#39;, &#39;micro moles per gram&#39;, &#39;used to describe the concentration of dissolved salts in water, the UNESCO Practical Salinity Scale of 1978 (PSS78) defines salinity in terms of a conductivity ratio&#39;)) Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- unitlist If units that should be standardUnit are added as customUnit, you can use the following code to fix this issue: # add standard unit doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$measurementScale$ratio$unit$standardUnit &lt;- &quot;your standard unit&quot; # get rid of custom unit doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$measurementScale$ratio$unit$customUnit &lt;- NULL If you want to find all of the positions of a certain custom unit that should be standard, try this code: ls &lt;- purrr::map(doc$dataset$dataTable[[i]]$attributeList$attribute, ~str_detect(.x$measurementScale$ratio$unit$customUnit, &quot;your standard unit&quot;)) which(ls == TRUE) "],["edit-datatables.html", "Edit dataTables", " Edit dataTables Entities that are dataTables require an attribute list. To edit a dataTable, first edit/create an attributeList and set the physical. Then create a new dataTable using the eml$dataTable() helper function as below: dataTable &lt;- eml$dataTable(entityName = &quot;A descriptive name for the data (does not need to be the same as the data file)&quot;, entityDescription = &quot;A description of the data&quot;, physical = physical, attributeList = attributeList) The dataTable must then be added to the EML. How exactly you do this will depend on whether there are dataTable elements in your EML, and how many there are. To replace whatever dataTable elements already exist, you could write: doc$dataset$dataTable &lt;- dataTable If there is only one dataTable in your dataset, the EML package will usually “unpack” these, so that it is not contained within a list of length 1 - this means that to add a second dataTable, you cannot use the syntax doc$dataset$dataTable[[2]], since when unpacked this will contain the entityDescription as opposed to pointing to the second in a series of dataTable elements. Confusing - I know. Not to fear though - this syntax will get you on your way, should you be trying to add a second dataTable. doc$dataset$dataTable &lt;- list(doc$dataset$dataTable, dataTable) If there is more than one dataTable in your dataset, you can return to the more straightforward construction of: doc$dataset$dataTable[[i]] &lt;- dataTable Where i is the index that you wish insert your dataTable into. To add a list of dataTables to avoid the unpacking problem above you will need to create a list of dataTables dts &lt;- list() # create an empty list for(i in seq_along(tables_you_need)){ # your code modifying/creating the dataTable here dataTable &lt;- eml$dataTable(entityName = dataTable$entityName, entityDescription = dataTable$entityDescription, physical = physical, attributeList = attributeList) dts[[i]] &lt;- dataTable # add to the list } After getting a list of dataTables, assign the resulting list to dataTable EML doc$dataset$dataTable &lt;- dts By default, the online submission form adds all entities as otherEntity, even when most should probably be dataTable. You can use eml_otherEntity_to_dataTable to easily move items in otherEntity over to dataTable, and delete the old otherEntity. Most tabular data or data that contain variables should be listed as a dataTable. Data that do not contain variables (eg: plain text readme files, pdfs, jpegs) should be listed as otherEntity. eml_otherEntity_to_dataTable(doc, 1, # Indexes of otherEntities you want to convert, for multiple use 1:5 or c(1,3,5..) validate_eml = F) # set this to False if the physical or attributes are not added "],["edit-otherentities.html", "Edit otherEntities", " Edit otherEntities Remove otherEntities To remove an otherEntity use the following command. This may be useful if a data object is originally listed as an otherEntity and then transferred to a dataTable. doc$dataset$otherEntity[[i]] &lt;- NULL Create otherEntities If you need to create/update an otherEntity, make sure to publish or update your data object first (if it is not already on the DataONE MN). Then build your otherEntity. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, pkg$data[[i]]) Alternatively, you can build the otherEntity of a data object not in your package by simply inputting the data PID. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, &quot;your_data_pid&quot;, entityType = &quot;otherEntity&quot;, entityName = &quot;Entity Name&quot;, entityDescription = &quot;Description about entity&quot;) The otherEntity must then be set to the EML, like so: doc$dataset$otherEntity &lt;- otherEntity If you have more than one otherEntity object in the EML already, you can add the new one like this: doc$dataset$otherEntity[[i]] &lt;- otherEntity Where i is set to the number of existing entities plus one. Remember the warning from the last section, however. If you only have one otherEntity, and you are trying to add another, you have to run: doc$dataset$otherEntity &lt;- list(otherEntity, doc$dataset$otherEntity) "],["semantic-annotations.html", "Semantic annotations", " Semantic annotations For a brief overview of what a semantic annotation is, and why we use them check out this video. Even more information on how to add semantic annotations to EML 2.2.0 can be found here. There are several elements in the EML 2.2.0 schema that can be annotated: dataset entity (eg: otherEntity or dataTable) attribute Attribute annotations can be edited in R and also on the website. Dataset and entity annotations are only done in R. How annotations are used This is a dataset that has semantic annotations included. On the website you can see annotations in each of the attributes. You can click on any one of them to search for more datasets with that same annotation. Semantic attribute annotations can be applied to spatialRasters, spatialVectors and dataTables Attribute-level annotations on the website editor The website has a searchable list of attribute annotations that are grouped by category and specificity. Open your dataset on the test website from earlier and enter the attribute editor. Look through all of the available annotations. Adding attribute annotations using the website is the easiest way, however adding them using R and/or the Shiny app may be quicker with very larger datasets. Attribute-level annotations in R To manually add annotations to the attributeList in R you will need information about the propertyURI and valueURI. Annotations are essentially composed of a sentence, which contains a subject (the attribute), predicate (propertyURI), and object (valueURI). Because of the way our search interface is built, for now we will be using attribute annotations that have a propertyURI label of “contains measurements of type”. Here is what an annotation for an attribute looks like in R. Note that both the propertyURI and valueURI have both a label, and the URI itself. doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$annotation $id [1] &quot;ODBcOyaTsg&quot; $propertyURI $propertyURI$label [1] &quot;contains measurements of type&quot; $propertyURI$propertyURI [1] &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot; $valueURI $valueURI$label [1] &quot;Distributed Biological Observatory region identifier&quot; $valueURI$valueURI [1] &quot;http://purl.dataone.org/odo/ECSO_00002617&quot; How to add an annotation 1. Decide which variable to annotate The goal for the datateam is to start annotating every dataset that comes in. Please make sure to add semantic annotations to spatial and temporal features such as latitude, longitude, site name and date and aim to annotate as many attributes as possible. 2. Find an appropriate valueURI The next step is to find an appropriate value to fill in the blank of the sentence: “this attribute contains measurements of _____.” There are several ontologies to search in. In order of most to least likely to be relevant to the Arctic Data Center they are: The Ecosystem Ontology (ECSO) this was developed at NCEAS, and has many terms that are relevant to ecosystem processes, especially those involving carbon and nutrient cycling The Environment Ontology (EnVO) this is an ontology for the concise, controlled description of environments National Center for Biotechnology Information (NCBI) Organismal Classification (NCBITAXON) The NCBI Taxonomy Database is a curated classification and nomenclature for all of the organisms in the public sequence databases. Information Artifact Ontology (IAO) this ontology contains terms related to information entities (eg: journals, articles, datasets, identifiers) To search, navigate through the “classes” until you find an appropriate term. When we are picking terms, it is important that we not just pick a similar term or a term that seems close - we want a term that is 100% accurate. For example, if you have an attribute for carbon tetroxide flux and an ontology with a class hierarchy like this: – carbon flux |—- carbon dioxide flux Our exact attribute, carbon tetroxide flux, is not listed. In this case, we should pick “carbon flux” as it’s completely correct/accurate and not “carbon dioxide flux” because it’s more specific/precise but not quite right. For general attributes (such as ones named depth or length), it is important to be as specific as possible about what is being measured. e.g. selecting the lake area annotation for the area attribute in this dataset 3. Build the annotation in R Manually Annotating this method is great for when you are inserting 1 annotation, fixing an existing annotation or programmatically updating annotations for multiple attributeLists First you need to figure out the index of the attribute you want to annotate. eml_get_simple(doc$dataset$dataTable[[3]]$attributeList, &quot;attributeName&quot;) [1] &quot;prdM&quot; &quot;t090C&quot; &quot;t190C&quot; &quot;c0mS/cm&quot; &quot;c1mS/cm&quot; &quot;sal00&quot; &quot;sal11&quot; &quot;sbeox0V&quot; &quot;flECO-AFL&quot; [10] &quot;CStarTr0&quot; &quot;cpar&quot; &quot;v0&quot; &quot;v4&quot; &quot;v6&quot; &quot;v7&quot; &quot;svCM&quot; &quot;altM&quot; &quot;depSM&quot; [19] &quot;scan&quot; &quot;sbeox0ML/L&quot; &quot;sbeox0dOV/dT&quot; &quot;flag&quot; Next, assign an id to the attribute. It should be unique within the document, and it’s nice if it is human readable and related to the attribute it is describing. One format you could use is entity_x_attribute_y which should be unique in scope, and is nice and descriptive. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$id &lt;- &quot;entity_ctd_attribute_salinity&quot; Now, assign the propertyURI information. This will be the same for every annotation you build. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$propertyURI &lt;- list(label = &quot;contains measurements of type&quot;, propertyURI = &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot;) Finally, add the valueURI information from your search. You should see an ID on the Bioportal page that looks like a URL - this is the valueURI. Use the value to populate the label element. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$valueURI &lt;- list(label = &quot;Water Salinity&quot;, valueURI = &quot;http://purl.dataone.org/odo/ECSO_00001164&quot;) Shiny Attributes this method is great for when you are updating many attributes On the far right of the table of shiny_attributes there are 4 columns: id, propertyURI, propertyLabel, valueURI, valueLabel that can be filled out. Dataset Annotations Dataset annotations can only be made using R. There are several helper functions that assist with making dataset annotations. Data Sensitivity Sensitive datasets that might cover protected characteristics (human subjects data, endangered species locations, etc) should be annotated using the data sensitivity ontology: https://bioportal.bioontology.org/ontologies/SENSO/?p=classes&amp;conceptid=root. Dataset Discipline As a final step in the data processing pipeline, we will categorize the dataset. We are trying to categorize datasets so we can have a general idea of what kinds of data we have at the Arctic Data Center. Datasets will be categorized using the Academic Ontology. These annotations will be seen at the top of the landing page, and can be thought of as “themes” for the dataset. In reality, they are dataset-level annotations. Be sure to ask your peers in the #datateam slack channel whether they agree with the themes you think best fit your dataset. Once there is consensus, use the following line of code: doc &lt;- arcticdatautils::eml_categorize_dataset(doc, c(&quot;Soil Science&quot;, &quot;Plant Science&quot;, &quot;Ecology&quot;)) Be careful not to duplicate dataset annotations. The above code does not existing prior dataset annotations. Duplicate annotations can be removed by setting them to NULL. doc$dataset$annotation[[i]] &lt;- NULL "],["edit-spatial-data.html", "Edit spatial data", " Edit spatial data Occasionally, you may encounter a third type of data object: spatialVector and spatialRaster. These objects contains spatial data (ie maps), such as a shapefile or geodatabase. Editing a spatialVector or spatialRaster is similar to editing a dataTable or an otherEntity. A physical and attributeList should be present. We will focus on how to get the information unique to spatialData and how to create the spatialVector/spatialRaster File types File extensions to look for that might be spatial data: kml, geoJSON, geoTIFF, .dbf, .shp, and .shx Additionally, spatial data that involve multiple files should typically be archived within a .zip file to ensure all related and interdependent files stay together (ie . a geodatabase). This is one of the exceptions to our rule regarding .zip files. For example, a spatial dataset for a shapefile should, at a minimum, consist of separate .dbf, .shp, and .shx files with the same prefix in the same directory. All these files are required in order to use the data. Also note that shapefiles limit attribute names to 10 characters, so attribute names in the metadata may not match exactly to attribute names in the data. Some spatial raster data come as standalone files (.tiff or .nc) and some come as a group of files. If you aren’t sure whether to unzip a file, ask Jasmine or Jeanette. There are specific formatIds for these kinds of zipped files: application/vnd.shp+zip image/geotiff+zip. Remember to check that the files have the correct formatId Reading Spatial Files Read in the files to (1) help you in creating your attributes table and (2) sometimes also figure out the coordinate reference system. library(sf) spatial_file &lt;- sf::read_sf(&quot;example.kml&quot;) When you read kml files, read_sf() sometimes shows additional columns that aren’t in the actual file. Always open kml files in text editor to check if the columns actually exist. If it is a zipped shapefile there is a handy function you can use arcticdatautils::read_zip_shapefile(mn, pid) Coordinate Systems The coordinate system allow to work with spatial data using the same frame of reference (a Datum). A common coordinate system is “GCS_WGS_1984 (used in Google Maps!) which is suitable for plotting points distributed globally. There are many others that may be better suited for certain areas in the world. All latitudes and longitude coordinates should have a coordinate system (like a frame of reference). There are horizontal coordinate systems (earth’s surface) and vertical coordinate systems (depth). More information can be found here. To find the horizCoordSysName you can use: sf::st_crs(spatial_file) Take the Datum and add GCS (Geographic Coordinate System) in front. For example: “GCS_WGS_1984” spatialVector Adding Geometry One important difference is that a spatialVector object should also have a geometry slot that describes the geometry features of the data. The possible values include one or more (in a list) of ‘Point’, ‘LineString’, ‘LinearRing’, ‘Polygon’, ‘MultiPoint’, ‘MultiLineString’, ‘MultiPolygon’, or ‘MultiGeometry’. You will likely have to open the file itself within QGIS or R (ie . the sf package) to get the correct geometry value. To add just a geometry slot use: doc$dataset$spatialVector[[1]]$geometry &lt;- &quot;Polygon&quot; To add it using the data pid: 1. Get the geometry and spatialReference 2. Use pid_to_eml_entity() to generate the spatialVector spatialVector &lt;- pid_to_eml_entity(adc, pkg$data[n], entity_type = &quot;spatialVector&quot;, entityName = &quot;filename.kml&quot;, entityDescription = &quot;some desciption&quot;, attributeList = attributeList, geometry = &quot;Point&quot;, spatialReference = list(horizCoordSysName = &quot;GCS_WGS_1984&quot;)) Add the spatialVector to the doc doc$dataset$spatialVector[[1]] &lt;- spatialVector spatialRasters Most often these come in GeoTiff or Tiff files. The data is presented as a grid of “pixels”. For more information ESRI has a indepth article here. To use the helper function get: the path of your raster file an attribute table a coordinate system To get a coordinate system name, you can use the output of the function on your first try (which will print the coordinate reference system, if it is defined). You can use the return value of get_coord_list() (a large data.frame) to find the correct coordinate system name. Another way to get the coordinate system name is using rgdal::GDALinfo(path). This function can provide many details for your GeoTiff or Tiff files including the coordinate system name. More information can be found here here. rgdal::GDALinfo(path) eml_get_raster_metadata(path, coord_name, attributes) "],["format-text-in-eml.html", "Format text in EML", " Format text in EML Currently, only certain fields (abstracts, methods) support text formatting in EML. Check out this demo for a full example. Additional info is also available here. Many of these formatting functions only work when enclosed by &lt;para&gt;&lt;/para&gt; You can insert these tags directly into the EML document using syntax that looks like this: doc$dataset$abstract &lt;- eml$abstract(para = &quot;Some abstracts require subscripts like CO&lt;subscript&gt;2&lt;/subscript&gt;&quot;) Type-setting Subscripts, superscripts, and italics: &lt;subscript&gt;You can do subscripts&lt;/subscript&gt; &lt;superscript&gt;or superscipts&lt;/superscript&gt; &lt;emphasis&gt;or even italics.&lt;/emphasis&gt; Links Be sure to include the “https://” before the link or it will redirect incorrectly. Also, always check that your links go through to the correct page. Please be aware that most links are inherently unstable so always default to archiving files over pointing to websites when possible and appropriate. &lt;ulink url=&quot;https://some_url.com&quot;&gt; &lt;citetitle&gt;some text&lt;/citetitle&gt; &lt;/ulink&gt; Lists Unordered (bulleted) lists: &lt;itemizedlist&gt; &lt;listitem&gt; &lt;para&gt;Paragraphs&lt;/para&gt; &lt;/listitem&gt; &lt;listitem&gt; &lt;para&gt;Sections w/ subsections (w/ titles)&lt;/para&gt; &lt;/listitem&gt; &lt;/itemizedlist&gt; Ordered lists (1, 2, 3)… &lt;orderedlist&gt; &lt;listitem&gt; &lt;para&gt;something&lt;/para&gt; &lt;/listitem&gt; &lt;listitem&gt; &lt;para&gt;something else&lt;/para&gt; &lt;/listitem&gt; &lt;/orderedlist&gt; "],["series-identifier-sid.html", "Series Identifier (SID)", " Series Identifier (SID) A series identifier is a system metadata field that represents a single identifier across multiple versions of a data package, a feature often requested by submitters. These are useful for maintaining a single identifier when a data package is expected to receive updates (usually additional data) in the future. For example, this dataset has a series id: https://doi.org/10.18739/A2154DQ22 (the doi and urn:uuid are both present at the top of the dataset). Adding a SID to a package Adding a series identifier should be the last step. In most cases the SID will be in DOI format. Once the data package is complete (peer-reviewd and approved by the PI) the series identifier can be added by updating the seriesId field of the system metadata of the metadata object. # The metadata identifier we&#39;re assigning an SID to metadata_pid &lt;- &#39;metadata_pid&#39; sys &lt;- getSystemMetadata(mn, metadata_pid) sys@seriesId &lt;- generateIdentifier(mn, scheme = &#39;DOI&#39;) #update the scheme argument if it should not be a DOI updateSystemMetadata(mn, metadata_pid, sys) Updating the child packages of a parent package with a SID When updating a parent package with a series identifier you need to use update_resource_map rather than publish_update. In this case our goal is add nested data packages to an existing parent package. # Call get_package on the parent package that has a `seriesId` in its system metadata parent_package &lt;- get_package(mn, &#39;resource_map_pid&#39;) # The resource map of the package we want to add to the &#39;child_pids&#39; of the parent resource_map_new_child &lt;- &#39;&#39; update_resource_map(mn, parent_package$resource_map, parent_package$metadata, parent_package$data, c(parent_package$child_packages, resource_map_new_child)) Updating the metadata of a package with a SID When updating the metadata (xml) of a package with a series identifier use publish_update. It’s important that you pass the metadata pid to the metadata_pid argument rather than seriesId. The metadata pid will usually have the text “version:” ahead of it, however it’s best to use get_package first to avoid mistakes. pkg &lt;- get_package(mn, &#39;resource_map_pid&#39;) pkg &lt;- publish_update(mn, pkg$metadata, pkg$resource_map, pkg$data, child_pids = pkg$child_pids) # + any additional arguments to &quot;publish_update&quot; "],["set-coverages.html", "Set coverages", " Set coverages Sometimes EML documents may lack coverage information describing the temporal, geographic, or taxonomic coverage of a data set. This example shows how to create coverage information from scratch, or replace an existing coverage element with an updated one. You can view the current coverage (if it exists) by entering doc$dataset$coverage into the console. Here the coverage, including temporal, taxonomic, and geographic coverages, is defined using set_coverage(). coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, sci_names = c(&#39;exampleGenus exampleSpecies1&#39;, &#39;exampleGenus ExampleSpecies2&#39;), geographicDescription = &quot;The geographic region covers the lake region near Eagle Mountain, Alaska.&quot;, west = -154.6192, east = -154.5753, north = 68.3831, south = 68.3619) doc$dataset$coverage &lt;- coverage Set multiple coverages You can also set multiple geographic (or temporal) coverages. Here is an example of how you might set two geographic coverages. Note that we use nested eml function helpers in this construction. geocov1 &lt;- eml$geographicCoverage(geographicDescription = &quot;The geographich region covers area 1&quot;, boundingCoordinates = eml$boundingCoordinates( northBoundingCoordinate = 68, eastBoundingCoordinate = -154, southBoundingCoordinate = 67, westBoundingCoordinate = -155)) geocov2 &lt;- eml$geographicCoverage(geographicDescription = &quot;The geographich region covers area 2&quot;, boundingCoordinates = eml$boundingCoordinates( northBoundingCoordinate = 65, eastBoundingCoordinate = -151, southBoundingCoordinate = 62, westBoundingCoordinate = -153)) coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, sci_names = list(&#39;exampleGenus exampleSpecies1&#39;, &#39;exampleGenus ExampleSpecies2&#39;)) doc$dataset$coverage$geographicCoverage &lt;- list(geocov1, geocov2) Special coverages Arctic Circle For arctic circle geographic coverage, we only have the starting vertical line of the circle shown in the projection. Here is an example with arctic circle geographic coverage. Geologic dates Example dataset with geologic coverages set using the following: geo_time_start &lt;- EML::eml$alternativeTimeScale(timeScaleName = &quot;Absolute&quot;, timeScaleAgeEstimate = &quot;7.5 Myr&quot;) coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, doc$dataset$coverage &lt;- coverage "],["set-methods.html", "Set methods", " Set methods The methods tree in the EML section has many different options, visible in the schema. You can create new elements in the methods tree by following the schema and using the eml helpers. Another simple and potentially useful way to add methods to an EML that has no methods at all is by adding them via a MS Word document. An example is shown below: methods1 &lt;- set_methods(&#39;methods_doc.docx&#39;) doc$dataset$methods &lt;- methods1 If you want to make minor changes to existing methods information that has a lot of nested elements, your best bet may be to edit the EML manually in a text editor (or in RStudio), otherwise there is a risk of accidentally overwriting nested elements with blank object classes, therefore losing methods information. Adding sampling info to methods section # add method steps as new variables step1 &lt;- eml$methodStep(description = &quot;text describing the methods used&quot;) stEx &lt;- eml$studyExtent(description = &quot;study extent description&quot;) samp &lt;- eml$sampling(studyExtent = stEx, samplingDescription = &quot;sampling description text&quot;) # combine all methods steps and sampling info methods1 &lt;- eml$methods(methodStep = step1, sampling = samp) doc$dataset$methods &lt;- methods1 "],["set-parties.html", "Set parties", " Set parties The address, creator, contact, and associatedParty classes can easily be created using functions from the EML package. However it is often easier to just edit this through the webform. To add people, with their addresses, you need to add addresses as their own object class, which you then add to the contact, creator, or associatedParty classes. NCEASadd &lt;- eml$address( deliveryPoint = &quot;735 State St #300&quot;, city = &quot;Santa Barbara&quot;, administrativeArea = &#39;CA&#39;, postalCode = &#39;93101&#39;) Here, we use eml_creator() to set our data set creator. JC_creator &lt;- eml$creator(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jclark@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, address = NCEASadd) doc$dataset$creator &lt;- JC_creator Similarly, we can set a single contact or multiple. In this case, there are two, so we set doc$dataset$contact as a list containing both of them. JC_contact &lt;- eml$contact(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jclark@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;,id = eml$userId(directory = eml$dir) address = NCEASadd) JG_contact &lt;- eml$contact(individualName = list(givenName = &quot;Jesse&quot;, surName = &quot;Goldstein&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jgoldstein@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, address = NCEASadd) doc$dataset$contact &lt;- list(JC_contact, JG_contact) Finally, the associatedPartys are set. Note that associatedPartys MUST have a role defined, unlike creator or contact. JG_ap &lt;- eml$associatedParty(individualName = list(givenName = &quot;Jesse&quot;, surName = &quot;Goldstein&quot;), organization = &quot;NCEAS&quot;, email = &quot;jgoldstein@nceas.ucsb.edu&quot;, electronicMailAddress = &quot;123-456-7890&quot;, address = NCEASadd, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, role = &quot;metadataProvider&quot;) doc$dataset$associatedParty &lt;- JG_ap "],["set-physical.html", "Set physical", " Set physical To set the physical aspects of a data object, use the following commands to build a physical object from a data PID that exists in your package. Remember to set the member node to test.arcticdata.io! Every entity that we upload needs a physical description added. When replacing files, the physical must be replaced as well. The word ‘physical’ derives from database systems, which distinguish the ‘logical’ model (e.g., what attributes are in a table, etc) from the physical model (how the data are written to a physical hard disk (basically, the serialization)). so, we grouped metadata about the file (eg. dataformat, file size, file name) as written to disk in physical. It includes info like the file size. For CSV files, the physical describes the number of header lines and the attribute orientation. # Get the PID of a file data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;your_file_name.csv&quot;) # Get the physical info and store it in an object physical &lt;- arcticdatautils::pid_to_eml_physical(mn, data_pid) The physical object can then be checked for errors and added to the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$physical &lt;- physical Note that the above workflow only works if your data object already exists on the member node. Physicals can be seen in the website representation of the EML below the entity description. "],["set-the-project-section.html", "Set the project section", " Set the project section The project section in an EML document is automatically filled out by the metacatUI editor. It sets the project title and project personnel to the submission’s title and creators. Most of the time at least some of this information is incorrect and we need to update it. Start by searching for the funding information using NSF’s award search. This will give us the project title, abstract, and personnel - along with some additional metadata. Using this information we will set the title, personnel, and funding number. For NSF funded projects prepend the funding number with “NSF”. If there are multiple awards associated with one dataset then additional funding, title, and personnel elements should be added to reflect the additional awards. doc$dataset$project$funding$para[[1]] &lt;- &#39;NSF 1503846&#39; doc$dataset$project$title[[1]] &lt;- &#39;Collaborative Research: Reconciling conflicting Arctic temperature and fire reconstructions using multi-proxy records from lake sediments north of the Brooks Range, Alaska doc$dataset$project$personnel[[1]] &lt;- eml$personnel(individualName = eml$individualName(givenName = &#39;Yongsong&#39;, surName = &#39;Huang&#39;), role = &#39;Principal Investigator&#39;) doc$dataset$project$personnel[[2]] &lt;- eml$personnel(individualName = eml$individualName(givenName = &#39;James&#39;, surName = &#39;Russell&#39;), role = &#39;co Principal Investigator&#39;) There is also a helper function eml_nsf_to_project() that can help do the searching for you. Just verify that the information retrieved is correct. # update NSF awards data awards &lt;- c(&quot;1311655&quot;, &quot;1417987&quot;, &quot;1417993&quot;) # list of award numbers proj &lt;- eml_nsf_to_project(awards) #helper function doc$dataset$project &lt;- proj On the web form right now only fills in the funding field. We need instructions on how to convert that to the new awards formatting until the form is updated to include these fields: project$award$funderName # required project$award$title # required project$award$awardNumber project$award$funderIdentifier Information can be found in using the Open Funder Registry. If a award title cannot be found you can use the dataset title. "],["use-references.html", "Use references", " Use references References are a way to avoid repeating the same information multiple times in the same EML record. There are a few benefits to doing this, including: Making it clear that two things are the same (e.g., the creator is the same person as the contact, two entities have the exact same attributes) Reducing the size on disk of EML records with highly redundant information Faster read/write/validate with the R EML package You may want to use EML references if you have the following scenarios (not exhaustive): One person has multiple roles in the dataset (creator, contact, etc) One or more entities shares all or some attributes Example with parties Do not use references for creators as it is used for the citation information. The creators will not show up on the top of the dataset if it is a reference. Until this issue is resolved in NCEAS/metacat#926 we will need to keep this in account. It’s very common to see the contact and creator referring to the same person with XML like this: &lt;eml packageId=&quot;my_test_doc&quot; system=&quot;my_system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1 eml.xsd&quot;&gt; &lt;dataset&gt; &lt;creator&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml&gt; So you see those two times Bryce Mecum is referenced there? If you mean to state that Bryce Mecum is the creator and contact for the dataset, this is a good start. But with just a name, there’s some ambiguity as to whether the creator and contact are truly the same person. Using references, we can remove all doubt. doc$dataset$creator[[1]]$id &lt;- &quot;reference_id&quot; doc$dataset$contact &lt;- list(references = &quot;reference_id&quot;) print(doc) &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; &lt;eml:eml xmlns:eml=&quot;eml://ecoinformatics.org/eml-2.1.1&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns:stmml=&quot;http://www.xml-cml.org/schema/stmml-1.1&quot; packageId=&quot;id&quot; system=&quot;system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1/ eml.xsd&quot;&gt; &lt;dataset&gt; &lt;title&gt;A Minimal Valid EML Dataset&lt;/title&gt; &lt;creator id=&quot;reference_id&quot;&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;references&gt;reference_id&lt;/references&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml:eml&gt; The reference id needs to be unique within the EML record but doesn’t need to have meaning outside of that. Example with attributes To use references with attributes: Add an attribute list to a data table Add a reference id for that attribute list Use references to add that information into the attributeLists of the other data tables For example, if all the data tables in our data package have the same attributes, we can set the attribute list for the first one, and use references for the rest: doc$dataset$dataTable[[1]]$attributeList &lt;- attribute_list doc$dataset$dataTable[[1]]$attributeList$id &lt;- &quot;shared_attributes&quot; # use any unique name for your id for (i in 2:length(doc$dataset$dataTable)) { doc$dataset$dataTable[[i]]$attributeList &lt;- list(references = &quot;shared_attributes&quot;) # use the id you set above } "],["pi-correspondence.html", "PI correspondence ", " PI correspondence "],["email-templates.html", "Email templates", " Email templates This section covers new data packages submitted. For other inquiries see the PI FAQ templates Please think critically when using these canned replies rather than just blindly sending them. Typically, content should be adjusted/ customized for each response to be as relevant, complete, and precise as possible. In your first few months, please run email drafts by the #datateam Slack and get approval before sending. Remember to consult the submission guidelines for details of what is expected. Quick reference: Initial email template Final email templates Additional email template Initial email template Hello [NAME OF REQUESTOR], Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: [COMMENTS HERE] After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, [YOUR NAME] Comment templates based on what is missing Portals Multiple datasets under the same project - suggest data portal feature I would like to highlight the Data Portals feature that would enhance the ability of your datasets to be discovered together. It also enables some custom branding for the group or project. Here is an example page that we created for the Distributed Biological Observatory: https://arcticdata.io/catalog/portals/DBO. We highly suggest considering creating a portal for your datasets, you can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/. Data portals can be set up at any point of your project. If they ask to nest the dataset We are no longer supporting new nested datasets. We recommend to create a data portal instead. Portals will allow more control and has the same functionality as a nested dataset. You can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/, but as always we are here to help over email. Dataset citations If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be done at any point. Title Provides the what, where, and when of the data We would like to add some more context to your data package title. I would like to suggest: ‘OUR SUGGESTION HERE, WHERE, WHEN’. Does not use acronyms We wanted to clarify a couple of abbreviations. Could you help us in defining some of these: [LIST OF ACRONYMS TO DEFINE HERE] Abstract Describes DATA in package (ideally &gt; 100 words) We would like to add some additional context to your abstract. We hope to add the following: [ADJUST THE DEPENDING ON WHAT IS MISSING] The motivation of the study Where and when the research took place At least one sentence summarizing general methodologies All acronyms are defined At least 100 words long Offer this if submitter is reluctant to change: If you prefer and it is appropriate, we could add language from the abstract in the NSF Award found here: [NSF AWARD URL]. Keywords We noticed that there were no keywords for this dataset. Adding keywords will help your dataset be discovered by others. Data Sensitive Data We will need to ask these questions manually until the fields are added to the webform. Data sensitivity categories Once we have the ontology this question can be asked: Based on our Data sensitivity categories, which of the 3 does your dataset align with most: Non-sensitive data - None of the data includes sensitive or protected information. Some or all data is sensitive with minimal risk - Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Some or all data is sensitive with significant risk - The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Ethical research proceedures We were wondering if you could also address this question specifically on Ethical Research Procedures: Describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Insitutional Review Board approvals, consent waivers, procedures for co-production, data sovreignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. We can help add your answers to the question to the metadata. Adding provenance Is the [mention the file names here] files related? If so we can add provenance to show the relationship between the files. Here is an example of how that is displayed: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Af00c4d71-0242-4e9d-9745-8999776fa2f2 At least one data file We noticed that no data files were submitted. With the exception of sensitive social science data, we seek to include all data products prior to publication. We wanted to check if additional files will be submitted before we move forward with the submission process. Open formats Example using xlsx. Tailor this reponse to the format in question. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. The data files can be replaced by going to the green Edit button &gt; Click the black triangle by the Describe button for the data file &gt; Select Replace (attached is also a screenshot on how to get there). Zip files Except for very specific file types, we do not recommend that data are archived in zip format. Data that are zipped together can present a barrier to reuse since it is difficult to accurately document each file within the zip file in a machine readable way. File contents and relationships among files are clear Could you provide a short description of the files submitted? Information about how each file was generated (what software, source files, etc.) will help us create more robust metadata for long term use. Data layout Would you be able to clarify how the data in your files is laid out? Specifically, what do the rows and columns represent? We try not to prescribe a way the researchers must format their data as long as reasonable. However, in extreme cases (for example Excel spreadsheets with data and charts all in one sheet) we will want to kindly ask them to reformat. We would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. [DESCRIBE WHAT MAY NEED TO BE CHANGED IN THE DATA SET]. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. Attributes Identify which attributes need additional information. If they are common attributes like date and time we do not need further clarification. Checklist for the datateam in reviewing attributes (NetCDF, CSV, shapefiles, or any other tabular datasets): A name (often the column or row header in the file). A complete definition. Any missing value codes along with explanations for those codes. For all numeric data, unit information is needed. For all date-time data, a date-time format is needed (e.g. “DD-MM-YYYY”). For text data, full descriptions for all patterns or code/definition pairs are needed if the text is constrained to a list of patterns or codes. Helpful templates: &gt; We would like your help in defining some of the attributes. Could you write a short description or units for the attributes listed? [Provide a the attribute names in list form] &gt; Could you describe ____? &gt; Please define “XYZ”, including the unit of measure. &gt; What are the units of measurement for the columns labeled “ABC” and “XYZ”? Missing value codes What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. This section is not yet available on our webform so we will add that information on your behalf. We noticed that the data files contain [blank cells - replace with missing values found]. What do these represent? Funding All NSF funded datasets need a funding number. Non-NSF funded datasets might not have funding numbers, depending on the funding organization. We noticed that your dataset does not appear to contain a funding number. The field accepts NSF funding numbers as well as other numbers by different organizations. Methods We noticed that methods were missing from the submission. Submissions should include the following: provide instrument names (if applicable) specify how sampling locations were chosen if citations for sampling methods are used, please provide a brief summary of the methods referenced any software used to process the data Note - this includes software submissions as well (see https://arcticdata.io/submit/#metadata-guidelines-for-software) Your methods section appears to be missing some information. [ADJUST THIS DEPENDING ON WHAT IS MISSING - Users should be able to understand how the data were collected, how to interpret the values, and potentially how to use the data in the case of specialized files.] Comprehensive methods information should be included directly in the metadata record. Pointers or URLs to other sites are unstable. A full example - New Submission: methods, excel to csv, and attributes Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be added at any point. We would like to add some more context to your data package title. I would like to suggest: Holocene lake-based Arctic glacier and ice cap records. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. We also would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. If the data was in a long rather than wide format, it would be easier to us to document. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. We noticed that methods were missing from the submission. Submissions should: - provide instrument names (if applicable) - specify how sampling locations were chosen - provide citations for sampling methods that are not explained in detail - any software used to process the data After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, Name Final email templates Asking for approval Hi [submitter], I have updated your data package and you can view it here after logging in: [URL] Please review and approve it for publishing or let us know if you would like anything else changed. For your convenience, if we do not hear from you within a week we will proceed with publishing with a DOI. After publishing with a DOI, any further changes to the dataset will result in a new DOI. However, any previous DOIs will still resolve and point the user to the newest version. Please let us know if you have any questions. DOI and data package finalization comments Replying to questions about DOIs We attribute DOIs to data packages as one might give a DOI to a citable publication. Thus, a DOI is permanently associated with a unique and immutable version of a data package. If the data package changes, a new DOI will be created and the old DOI will be preserved with the original version. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to that latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Clarification of updating with a DOI and version control We definitely support updating a data package that has already been assigned a DOI, but when we do so we mark it as a new revision that replaces the original and give it its own DOI. We do that so that any citations of the original version of the data package remain valid (i.e.: after the update, people still know exactly which data were used in the work citing it). Resolve the ticket Sending finalized URL and dataset citation before resolving ticket [NOTE: the URL format is very specific here, please try to follow it exactly (but substitute in the actual DOI of interest)] Here is the link and citation to your finalized data package: https://doi.org/10.18739/A20X0X First Last, et al. 2021. Title. Arctic Data Center. doi:10.18739/A20X0X. If in the future there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. Please let us know if you need any further assistance. Additional email templates Deadlines If the PI is checking about dates/timing: &gt; [give rough estimate of time it might take] &gt; Are you facing any deadlines? If so, we may be able to expedite publication of your submission. Pre-assigned DOI If the PI needs a DOI right away: We can provide you with a pre-assigned DOI that you can reference in your paper, as long as your submission is not facing a deadline from NSF for your final report. However, please note that it will not become active until after we have finished processing your submission and the package is published. Once you have your dataset published, we would appreciate it if you could register the DOI of your published paper with us by using the citations button beside the orange lock icon. We are working to build our catalog of dataset citations in the Arctic Data Center. Sensitive Data Which of the following categories best describes the level of sensitivity of your data? A. Non-sensitive data None of the data includes sensitive or protected information. Proceed with uploading data. B. Some or all data is sensitive but has been made safe for open distribution Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Proceed with uploading data, but ensure that only data that are safe for public distribution are uploaded. Address questions about anonymization, aggregation, de-identification, and data embargoes with the data curation support team before uploading data. Describe these approaches in the Methods section. C. Some or all data is sensitive and should not be distributed The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Do NOT upload sensitive data. You should still upload a metadata description of your dataset that omits all sensitive information to inform the community of the dataset’s existence. Contact the data curation support team about possible alternative approaches to safely preserve sensitive or protected data. Ethical Research Procedures. Please describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Institutional Review Board approvals, consent waivers, procedures for co-production, data sovereignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. Asking for dataset access As a security measure we ask that we get the approval from the original submitter of the dataset prior to granting edit permissions to all datasets. No response from the researcher Please email them before resolving a ticket like this: We are resolving this ticket for bookkeeping purposes, if you would like to follow up please feel free to respond to this email. Recovering Dataset submissions To recover dataset submissions that were not successful please do the following: Go to https://arcticdata.io/catalog/drafts Find your dataset and download the corresponding file Send us the file in an email Custom Search Link You could also use a permalink like this to direct users to the datasets: https://arcticdata.io/catalog/data/query=\"your search query here” for example: https://arcticdata.io/catalog/data/query=Beaufort%20Lagoon%20Ecosystems%20LTER Adding metadata via R KNB does not support direct uploading of EML metadata files through the website (we have a webform that creates metadata), but you can upload your data and metadata through R. Here are some training materials we have that use both the EML and datapack packages. It explains how to set your authentication token, build a package from metadata and data files, and publish the package to one of our test sites. I definitely recommend practicing on a test site prior to publishing to the production site your first time through. You can point to the KNB test node (dev.nceas.ucsb.edu) using this command: d1c &lt;- D1Client(\"STAGING2\", \"urn:node:mnTestKNB\") If you prefer, there are Java, Python, MATLAB, and Bash/cURL clients as well. Finding multiple data packages If linking to multiple data packages, you can send a link to the profile associated with the submitter’s ORCID iD and it will display all their data packages. e.g.: https://arcticdata.io/catalog/profile/http://orcid.org/0000-0002-2604-4533 NSF ARC data submission policy Please find an overview of our submission guidelines here: https://arcticdata.io/submit/, and NSF Office of Polar Programs policy information here: https://www.nsf.gov/pubs/2016/nsf16055/nsf16055.jsp. Investigators should upload their data to the Arctic Data Center (https://arcticdata.io), or, where appropriate, to another community endorsed data archive that ensures the longevity, interpretation, public accessibility, and preservation of the data (e.g., GenBank, NCEI). Local and university web pages generally are not sufficient as an archive. Data preservation should be part of the institutional mission and data must remain accessible even if funding for the archive wanes (i.e., succession plans are in place). We would be happy to discuss the suitability of various archival locations with you further. In order to provide a central location for discovery of ARC-funded data, a metadata record must always be uploaded to the Arctic Data Center even when another community archive is used. Linking ORCiD and LDAP accounts First create an account at orcid.org/register if you have not already. After that account registration is complete, login to the KNB with your ORCID iD here: https://knb.ecoinformatics.org/#share. Next, hover over the icon on the top right and choose “My Profile”. Then, click the “Settings” tab and scroll down to “Add Another Account”. Enter your name or username from your Morpho account and select yourself (your name should populate as an option). Click the “+”. You will then need to log out of knb.ecoinformatics.org and then log back in with your old LDAP account (click “have an existing account”, and enter your Morpho credentials with the organization set to “unaffiliated”) to finalize the linkage between the two accounts. Navigate to “My Profile” and “Settings” to confirm the linkage. After completing this, all of your previously submitted data pacakges should show up on your KNB “My Profile” page, whether you are logged in using your ORCiD or Morpho account, and you will be able to submit data either using Morpho or our web interface. Or, try reversing my instructions - log in first using your Morpho account (by clicking the “existing account” button and selecting organization “unaffiliated”), look for your ORCiD account, then log out and back in with ORCiD to confirm the linkage. "],["pi-faq-email-templates.html", "PI FAQ email templates", " PI FAQ email templates Data Q: I would like to / have already uploaded data to ____ repository Review the repository, does it issue persistent identifiers (ie. DOIs)? Do they say how long they will be commiting to storing the data? If the answer is no to either questions, we will need to document and store any files in our own system. If any of this is unclear from what you can find online, you can message Matt to discuss with NSF if a particular repository will be allowed. Possible response for a generalist repository If your data doesn’t require features that only ____ provides, we would encourage you to consider publishing both the data and the metadata at the Arctic Data Center. Some benefits to submitting to us includes: We provide a DOI that can be cited in your published papers We track where datasets are being cited By submitting both the data and metadata to us, you can fulfill your NSF reporting requirements at the same time. This will reduce the work needed on your part to get any additional metadata requirements for NSF. Q: Can I replace data that have already been uploaded and keep the DOI? A: Once you have published your data with the Arctic Data Center, it can still be updated by providing an additional version which can replace the original, while still preserving the original and making it available to anyone who might have cited it. To update your data, return to the data submission tool used to submit it, and provide an update. Any update to a data package qualifies as a new version and therefore requires a new DOI. This is because each DOI represents a unique, immutable version, just like for a journal article. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to the latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Datateam: please review this dataset again to make sure it follows current standards Q: Why don’t I see my data package on the ADC? Possible Answer #1: The data package is still private because we are processing it or awaiting your approval to publish it. Please login with your ORCID iD to view private data packages. Possible Answer #2: The data package is still private and you do not have access because you were not the submitter. If you need access please have the submitter send us a message from his/her email address confirming this, along with your ORCID iD. Once we receive that confirmation we will be happy to grant you permission to view and edit the data package. Possible Answer #3: The data package is still private and we accidentally failed to grant you access. We apologize for the mistake. We have since updated the access policy. Please let us know if you are still having trouble viewing the data package here: [URL]. Remember to login with your ORCID iD. Possible Answer #4: The URL was garbled by their email server (eg. URL Defense or the url was split into two lines). You can navigate to your dataset by going to your name in the top right and selecting my datasets. Issue: MANY files to upload (100s or 1000s) or large cumulative size. A: Datateam - Prior to accepting large uploads, take the time to make sure there is enough space available and uploading all the files is necessary. If it is not a NSF dataset see the section on Scope. We have a secure FTP you can access. Details are available here: https://help.nceas.ucsb.edu/remote_file_access . Please access our server at datateam.nceas.ucsb.edu with the username “visitor”. Let us know if you would like to use our SFTP and we will send you the password and the path to which directory to upload to. If you have files to transfer to us that total several terabytes it may be best to arrange a shipment of an external hard drive. Q: Can I add another data file to an existing submission without having to fill out another metadata form? A: Yes. Navigate to the data package after being sure to login. Then click the green “Edit” button. The form will populate with the already existing metadata so there is no need to fill it out again. Click “Add Files” and browse to the file you wish to add. Be aware that the DOI will change after you add this file (or make any changes to a data package) as, just like for a journal article, a DOI represents a unique and immutable version. The original URL and DOI will remain functional and valid, but clearly display a message at the top of that page stating that “A newer version of this dataset exists” with a link to the latest version. Only the newest version wil be discoverable via a search. We can also copy the metadata from existing dataset. Please send us the URL of the dataset you would like to copy and the title of the new dataset and we will be able to do that on your behalf. Q: I want to organize the order of the files/metadata entities A: We can reorganize entities within their entities. This is a limitation with our tools (EML). While re-ordering the groups of entities is possible manually, it isn’t possible the way write_eml and publish_update is written. We do not recommended reordering entities this way. Q: Can we submit data as an Excel file? A: While the Arctic Data Center supports the upload of any data file format, sharing data can be greatly enhanced if you use ubiquitous, easy-to-read formats. For instance, while Microsoft Excel files are commonplace, it is better to export these spreadsheets to Comma Separated Values (CSV) text files, which can be read on any computer without needing to have Microsoft products installed. So, yes, you are free to submit an Excel workbook, however we strongly recommend converting each sheet to a CSV. The goal is not only for users to be able to read data files, but to be able to analyze them with software, such as R Studio. Typically, we would extract any plots and include them as separate image files. [ONLY SAY THIS NEXT PART IF THE REQUESTOR CONTINUES TO INSIST and then USE PROV TO POINT FROM THE XLS TO THE CSVs] I understand that having the plots in the same file as the data they are built from simplifies organization. If you definitely prefer to have the Excel workbook included, we ask that you allow us to document the data in both formats and include a note in the metadata clarifying that the data are indeed duplicated (but in different formats). Q: Can we submit model output? A: Yes you can submit your model output to the Arctic Data Center. We also recommend including model source code directly in the dataset as it is critical to understand the model output dataset and will allow other researchers to re-produce your output. If the model is complex or it’s important to maintain file structure then it can submitted as a zip file. We also recommend included a detailed readme file that documents: version information, licensing information, a list of software/hardware used in development, a list of software/hardware dependencies needed to run the software, information detailing source data for models, any mathematical/physical explanations needed to understand models, any methods used to evaluate models, and instructions how to run the model if not including in the source code as comments. Please read our software submission guidelines for more detailed information: https://arcticdata.io/submit/#metadata-guidelines-for-software Access Q: May another person (e.g. my student) submit data using my ORCID iD so that it is linked to me? A: We recommend instead that the student set up their own ORCiD account at https://ORCiD.org/register and submit data packages from that account. Submissions are processed by our team and, at that point, we can grant you full rights to the metadata and data files even though another person submitted them. Issue: Web form not cooperating. A: To help us diagnose the problem, could you let us know the following: Which operating system (including the version) and browser (with version #) are you using? At which exact step did the issue arise? What error message did you receive? Do you have any reason to believe that you may be using a slow internet connection? Please provide us with any relevant screenshots and we will troubleshoot from there. Issue: Can’t log in to the Arctic Data Center A: Please accept our apologies that you are experiencing difficulty logging in. We suggest trying these troubleshooting steps: Try a different internet browser (Chrome, Firefox, Internet Explorer etc.) Try clearing your cache and then re-logging in with orcid Id Privacy blockers (like EFF’s privacy badger https://www.eff.org/privacybadger) block the cookies needed for the login to work. To enable the cookies you need to disable your privacy tracker for arcticdata.io. If the problem persists please let us know what step the error occured or if there were error messages you have recieved. Scope Q: Can I submit a non-NSF funded dataset? Sample text 1: The Arctic Data Center is open to all arctic-related data. For larger data packages, we would likely need to charge a one-time archival fee which amortizes the long-term costs of preservation in a single payment. Please let us know the size of your dataset in terms of total number and total size of all files. We look forward to receiving your submission! Sample text 2: Yes, you can submit non-NSF-funded Arctic data if you are willing to submit under the licensing terms of the Arctic Data Center (CC-0 or CC-BY), the data are moderately sized (with exact limits open to discussion), and a lot of support time to curate the submission is not required (i.e., you submit a complete metadata record and well formatted, open format data files). For larger data packages, we would likely need to charge a one-time archival fee which amortizes the long-term costs of preservation in a single payment. Also, please note that NSF-funded projects take priority when it comes to processing. Information on best practices for data and metadata organization is available here: https://arcticdata.io/submit/#organizing-your-data. Q: Is ______ considered within the arctic We don’t have a precise geographic cutoff. We definitely will preserve any dataset that is funded by the Arctic section of the National Science Foundation, which includes many sub-Arctic studies. So, as long as the dataset is related to Arctic research defined broadly, we would consider it. We will prioritize NSF-funded projects because that is our main mandate, but we will also preserve non-NSF data about the Arctic as time allows. If you send more details about the study, its subject, location, and funding, we could be more definitive about whether we could archive it. "],["good-examples-of-data-packages.html", "Good Examples of data packages", " Good Examples of data packages Modeling Model with output: https://doi.org/10.18739/A24J09X55 Model code archived as a zip if preverving directory structure is important https://doi.org/10.18739/A2JS9H795 Model with provenance: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Ae6390181-85e9-46a2-b2ba-e352ece51cc6 Code and output: Note: We can only archive the code if the submitter has the rights to redistribute it (like if they wrote it themselves, or if the code has a license that allows for that) https://doi.org/10.18739/A2XS5JH4N Guidelines for large outputs: https://arcticdata.io/submit/#guidelines-for-large-models Spatial data Vector https://doi.org/10.18739/A2TB0XV89 Raster https://doi.org/10.18739/A2GT5FG0B Portals https://arcticdata.io/catalog/portals/DBO https://arcticdata.io/catalog/portals/CALM "],["final-checklist.html", "Final Checklist", " Final Checklist You can click on the assessment report on the website to for a general check. Fix anything you see there. Send the link over slack for peer review by your fellow datateam members. Usually we look for the following (the list is not exhaustive): Special Datasets Please refer to the dedicated pages for instructions to handle these cases: MOSAiC DBO System Metadata the format ids are correct General EML Ethical Research Practice section is completed Publisher and system information have been added: doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) Title Include geographic and temporal coverage (when and where data was collected) Abbreviations are removed or defined Abstract longer than 100 words Abbreviations are removed or defined. No garbled text Tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting DataTable / OtherEntity / SpatialVectors In the correct one: DataTable / OtherEntity / SpatialVector / SpatialRaster for the file type entityDescription - longer than 5 words and unique physical present and up-to-date and format is correct Attribute Table Complete attributeDefinitions longer than 3 words Variables match what is in the file Measurement domain - if appropirate (ie dateTime correct) Missing Value Code - accounted for if applicable Semantic Annotation - appropriate semantic annotations added, especially for spatial and temporal variables: lat, lon, date etc. Custom units were created if necessary People Complete information for each person in each section includes ORCID and e-mail address for all contacts people repeated across sections have consistent information Geographic region the map looks correct and matches the geographic description check if negatives (-) are missing Project If it is an NSF award you can use the helper function: doc$dataset$project &lt;- eml_nsf_to_project(awards) for other awards that need to be set manually, see the set project page Methods Present No garbled text Acronyms and units are defined the first time they are used Check EML Version Currently using: eml-2.2.0 (as of July 30 2020) Review to see if the EML version is set correctly by reviewing the doc$`@context` that it is indeed 2.2.0 under eml Re-run your code again and have the lineemld::eml_version(\"eml-2.2.0\") at the top Make sure the system metadata is also 2.2.0 Access If necessary, granted access to PI using set_rights_and_access() make sure it is http:// (no s) note if it is a part of portals there might be specific access requirements for it to be visible using set_access() SFTP Files If there are files transferred to us via SFTP, delete those files once they have been added to the dataset and when the ticket is resolved Updated datasets All the above applies. These are some areas to do a closer check when users update with a new file: New data was added Temporal Coverage and Title Confirm if new data used same methods or if methods need updating Files were replaced update physical and entityName double-check attributes are the same check for any new missing value codes that should be accounted for Was the dataset published before 2021? update project info, annotations Glance over entire page for any small mistakes (ie. repeated additionalMetadata, any missed &amp;amps, typos) "],["initial-review-checklist.html", "Initial review checklist", " Initial review checklist Before responding to a new submission use this checklist to review the submission. When your are ready to respond use the initial email template and insert comments and modify as needed. Sensitive Data If any of the below is in the dataset, please alert the #arctica team know before proceeding. Check if there is any sensitive information or personal identifying information in the data (eg. Names) Can the data be disaggregated and de-anonymized? (eg. a small sample size and individuals could be easily identified by their answers) Dryad Human Subject data guidelines can be a good place to start Common Cases: Social Science: Any dataset involving human subjects (may include awards awarded by ASSP and topics such as COVID-19) Archaeology: archaeological site location information, which is protected from public access by law Biology: protected species location coordinates Data citations If the dataset appears to be in a publication please (might be in the abstract) make sure that those citations are registered. Title WHAT, WHERE, and WHEN: Is descriptive of the work (provides enough information to understand the contents at a general scientific level), AND includes temporal coverage Provides a location of the work from the local to state or country level Provides a time frame of the work NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out Abstract Describes the DATA as well as: The motivation (purpose) of the study Where and when the research took place At least one sentence summarizing general methodologies NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out At least 100 words total tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting Any citations to papers can be registered with us Keywords Some keywords are included Data Data is normalized (if not suggest to convert the data if possible) At least one data file, or an identifier to the files at another approved archive, unless funded by ASSP (Arctic Social Sciences Program) No xls/xlsx files (or other proprietary files) File contents and relationships among files are clear Each file is well NAMED and DESCRIBED and clearly differentiated from all others All attributes in EML match attribute names in respective data files EXACTLY, are clearly defined, have appropriate units, and are in the same order as in the file. Quality control all dimensionless units. Missing value codes are explained (WHY are the data absent?) If it is a .rar file -&gt; scan the file If there is the unit tons make sure to ask if it is metric tons or imperical tons if not clarified already People &amp; Parties At least one contact and one creator with a name, email address, and ORCID iD Coverages Includes coverages that make sense Temporal coverage - Start date BEFORE end date Geologic time scales are added if mentioned in metadata (e.g. 8 Million Years or a name of a time period like Jurassic) Spatial coverage matches geographic description (check hemispheres) Geographic description is from the local to state or country level, at the least Taxonomic coverage if appropriate Project Information At least one FUNDING number Title, personnel, and abstract match information from the AWARD (not from the data package) Methods This section is REQUIRED for ALL NSF-FUNDED data packages Enough detail is provided such that a reasonable scientist could interpret the study and data for reuse without needing to consult the researchers, nor any other resources Portals If there are multiple submissions from the same people/project let them know about the portals feature If this is part of a portal make sure this dataset can be found there. Additional steps might be needed to get that to work. Please consult Jeanette and see the data portals section. "],["large-file-transfer.html", "Large file transfer", " Large file transfer Often if the researcher has many files to upload or the files are large in size, we will need to provide alternative methods of upload. We currently have two options available for large file transfer where sFTP is easier to set up but slower while GLOBUS takes a bit more setup but faster. Before providing access, try to get a sense of how big the transfer will be. df -h on Arctic server and datateam server - look at current space available sFTP As a team member for login info. After you login, you can add a folder with your last name to add your files. If you need more detailed access instructions, I have included the link here for your convenience: https://help.nceas.ucsb.edu/NCEAS/help/remote_file_access Option 1: SSH via the Terminal https://pages.github.nceas.ucsb.edu/NCEAS/help/connecting_to_linux_using_ssh.html Option 2: Cyberduck instructions To use Cyberduck to transfer local files onto the Datateam server: Open Cyberduck. Check that you have the latest version (Cyberduck -&gt; Check for Update…). If not, download and install the latest (you may need Jeanette to enter a password). Click “Open Connection”. From the drop-down, choose “SFTP (Secure File Transfer Protocol)”. Enter “datateam.nceas.ucsb.edu” for Server. Enter your username and password. Connect. From here, you can drag and drop files to and from the server. Globus Endpoint arctic-data-center #arctic-data-center login using your credentials on the datateam server Remember to remove the files on the datateam server after you finish a ticket to free up space. "],["navigate-rt.html", "Navigate RT", " Navigate RT The RT ticketing system is how we communicate with folks interacting with the Arctic Data Center. We use it for managing submissions, accessing issues, etc. It consists of three separate interfaces: Front Page All Tickets Ticket Page Front page This is what you see first Home - brings you to this homepage Tickets - to search for tickets (also see number 5) Tools - not needed New Ticket - create a new ticket Search - Type in the ticket number to quickly navigate to a ticket Queue - Lists all of the tickets currently in a particular queue (such as ‘arcticdata’) and their statuses New = unopened tickets that require attention Open = tickets currently open and under investigation and/or being processed by a support team member Stalled = tickets awaiting responses from the PI/ submitter Tickets I Own - These are the current open tickets that are claimed by me Unowned Tickets - Newest tickets awaiting claim Ticket Status - Status and how long ago it was created Take - claim the ticket as yours All tickets This is the queue interface from number 6 of the Front page 1. Ticket number and title 2. Ticket status 3. Owner - who has claimed the ticket Example ticket Title - Include the PI’s name for reference Display - homepage of the ticket History - Comment/Email history, see bottom of Display page Basics - edit the title, status, and ownership here People - option to add more people to the watch list for a given ticket conversation. Note that user/ PI/ submitter email addresses should be listed as “Requestors”. Requestors are only emailed on “Replys”, not “Comments”. Ensure your ticket has a Requestor before attempting to contact users/ PIs/ submitters Links - option to “Merge into” another ticket number if this is part of a larger conversation. Also option to add a reference to another ticket number Verify that this is indeed the two tickets you want to merge. It is non-reversible. Actions Reply - message the submitter/ PI/ all watchers Comment - attach internal message (no submitters, only Data Teamers) Open It - Open the ticket Stall - submitter has not responded in greater than 1 month Resolve - ticket completed History - message history and option to reply (to submitter and beyond) or comment (internal message) New data submission When notified by Arcticbot about a new data submission, here are the typical steps: Update the Requestor under the People section based on the email given in the submission (usually the user/ PI/ submitter). You may have to google for the e-mail address if the PI did not include it in the metadata record. Take the ticket (Actions &gt; Take) Review the submission based on the checklist Draft an email using the template and let others review it via Slack Send your reply via Actions "],["replicate-datasets.html", "Replicate Datasets", " Replicate Datasets Sometimes we get requests to have their datasets replicated to ADC. Usually it is a dataset on EDI where you can ge the Digital Object Identifier: which serves as the resource map on DataOne If a link isn’t given, you can use the package ID (i.e. knb-lter-arc.20129.1) and add it to the end of this link: https://portal.edirepository.org/nis/mapbrowse?packageid= If we are unsure about the identifer you can try querying for it on the CN: cn &lt;- CNode(&#39;PROD&#39;) #find the DOIs result &lt;- query(cn, list(q = paste0(&quot;(id:*10.6073/pasta/d4f567844673857239eec0cb61c6f543&quot;,&quot;* *:* NOT obsoletedBy:*)&quot;), fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded, authoritativeMN, replicaMN&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) Cloning the packages Try cloning to the test node first to see how it will look. Set up your nodes from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:LTER&quot;) to &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) For LTER datasets on EDI, there are two possibilities for the mn: “urn:node:LTER” and “urn:node:EDI”. Try the other one if one isn’t working Use clone_package to copy it over to the test node clone_package(&quot;doi:10.6073/pasta/d4f567844673857239eec0cb61c6f543&quot;, #example doi to replicate from = from, to = to, add_access_to = &quot;http://orcid.org/0000-0001-8888-547X&quot;, change_auth_node = F, new_pid = T) Once you have verified that it clones properly to the test node, do the same in production but with a couple of minor modifications: from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:LTER&quot;) to &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) clone_package(&quot;doi:10.6073/pasta/0af82d3c3d9d1710775cf9b1464ce70b&quot;, from = from, to = to, add_access_to = &quot;http://orcid.org/0000-0001-8888-547X&quot;, change_auth_node = F, new_pid = F) #uses the same pid make sure the new_pid argument is set to F when you publish to production "],["miscellaneous-file-types.html", "Miscellaneous file types ", " Miscellaneous file types "],["convert-excel-file-to-csv.html", "Convert Excel file to CSV", " Convert Excel file to CSV This uses the workflow in here: https://readxl.tidyverse.org/articles/articles/readxl-workflows.html This is preferred to converting Excel files by hand to avoid human error. If files are in the visitor folder you might need to be complete the following while logged into visitor Load the necessary packages library(magrittr) library(readxl) library(readr) Get a list of files that we want to convert xlsx &lt;- dir(&quot;directory/of/files/here&quot;, pattern = &quot;*xls&quot;, recursive = T, full.names = T) Create a function that iterates through the excel files and saves it as a csv in the same folder the excel file was in read_then_csv &lt;- function(sheet, path) { #remove the file extension pathbase &lt;- path %&gt;% tools::file_path_sans_ext() #change file extension to csv and save with the sheet name for(i in 1:length(sheet)){ path %&gt;% read_excel(sheet = i) %&gt;% write_csv(paste0(pathbase, &quot;_&quot;, sheet[i], &quot;.csv&quot;)) } } Get the sheet name(s) sheets &lt;- xlsx %&gt;% map(excel_sheets) %&gt;% set_names() Iterate over all the xlsx files for each of the sheets df &lt;- map2(sheets, xlsx, ~read_then_csv(.x, path = .y)) Check the files afterwards to make sure everything was converted properly csv &lt;- dir(&quot;directory/of/files/here&quot;, pattern = &quot;*csv&quot;, recursive = T, full.names = T) Check to see if there are any empty tables and get a list of files that were empty files &lt;- map(csv, read_csv) n &lt;- which(map(files, nrow) == 0) Remove those empty table files lapply(csv[n], file.remove) Remove xlsx files if everything looks in order lapply(xlsx, file.remove) "],["datalogger-files.html", "Datalogger files", " Datalogger files .hobo files are raw data files offloaded from a sensor, and can include real-time graphs of measured variables, such as temperature and soil moisture conditions. .hobo files can be opened using HOBOware, which can be downloaded from here. "],["netcdfs.html", "NetCDFs", " NetCDFs This section is for dealing with NetCDF (.nc) files. These files require data tables but since they can not be simply opened on the computer using a text editor or Excel, you can use Panoply to explore them or the R commands. However, notice that not all NetCDF files have the attribute table information. library(ncdf4) # gets attribute info atts &lt;- arcticdatautils::get_ncdf4_attributes(&#39;filepath&#39;) # preview of View(atts) atts[1:10,] The “long_name” can be used to build a definition. The “units” can be used for either units or the enumerated domain codes. The “_FillValue” is a missing value code it looks like. # returns the actual values for a specified attribute t &lt;- ncdf4::nc_open(filepath) test &lt;- ncdf4::ncvar_get(t, &#39;attributeName&#39;) # preview of View(test) test[1:10] The formatId in the sysmeta will most likely be netCDF-4, but could be netCDF-3. "],["reorganize-files-on-the-server.html", "Reorganize files on the server", " Reorganize files on the server Useful commands to use when reorganizing large datasets to be prepped to be uploaded using datapack This is run in the terminal Basics Moving around cd directory_name Go up one directory using .. cd .. Get the contents of the current directory ls Get the directory structure for this folder -d flag shows only the directories tree tree -d Rename file based on file path For example you have some files organized like this: ../../visitor/Lastname └── RU_ALN_TR1_FL007R │ └── rgb_images | └──image.png └── RU_ALN_TR1_FL008B └── micasense_preflight_calibration_images └──image.png We want to rename the files. Find all the files with the same file name find . -name &#39;image.png&#39; First extract the file name in parts ((.*)/(.*)/(.*)/(.*)) by using the. / as a separator. Each part is represented by $#. If you want the last part of the file name:$4. Rearrange this section as needed:$2/$3/$2-$3-$4 Set the flag as -n while testing to see the result of the renaming first before committing to it using -v rename -n -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; rename -v -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; Putting it together by taking the results from find and renaming those files | the pipe is like the %&gt;% in R find . -name &#39;image.png&#39; | rename -v -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; Unzip files unzip &#39;*zip&#39; Zip shapefiles For all the files in a directory, create a folder and move those files for file in *; do dir=$(echo $file | cut -d. -f1); mkdir -p $dir; mv $file $dir; done Zip the folders for file in *;do zip -r $file $file; done Remove the directories after zipping -r means to recursively remove files rm -r */ "],["scan-rar-files.html", "Scan rar files", " Scan rar files This section is for rar files which support data compression and file scanning. Here is how to scan rar files: Download the rar file to datateam In the terminal run the following on your files: mkdir to-scan &amp;&amp; mv &lt;file.rar&gt; to-scan &amp;&amp; cd to-scan unrar e &lt;file.rar&gt; That should scan the compressed file and each individual file it contains: clamscan * Clean up by deleting the to-scan folder - ’rm -r to-scan`. "],["spatial-data.html", "Spatial data", " Spatial data Shapefiles and their associated files can be kept in a single grouping using *.zip (or another similar format). To extract metadata, simply run: library(sf) pid &lt;- &quot;pid of a zipped shapefile on the ADC&quot; spatial_obj &lt;- arcticdatautils::read_zip_shapefile(adc, pid) st_geometry(spatial_obj) This will return the shapefile as an sf style data.frame so that you can examine the attributes, and print the projection and geometry information. Note that the column geometry should not be considered an attribute for documentation purposes. This information is represented in the spatialVector$geometry element. "],["wrangle-data.html", "Wrangle data ", " Wrangle data "],["clean-column-names.html", "Clean column names", " Clean column names You might have read that column names should not include spaces or special characters. Inevitably, you’ll encounter data that are not so tidy. For example: ## First Name Last Name date of birth ## 1 Homer Simpson 1/1/1960 ## 2 Marge Simpson 10/20/1965 ## 3 Ned Flanders 3/22/1961 To tidy it up, you can use the clean_names() function from the janitor package: janitor::clean_names(df) "],["fix-excel-dates.html", "Fix Excel dates", " Fix Excel dates Do you see something that looks like 43134, 43135, 43136 even though the column header is Dates? You may have encountered an Excel date/time problem. To fix it, the janitor package has a handy function: janitor::excel_numeric_to_date(c(43134, 43135, 43136)) Make sure you check that the dates make sense! "],["solr-queries.html", "Solr queries", " Solr queries Solr is what’s known as an index, which helps us keep track of the data, metadata, and resource map objects stored in our metadata catalog, Metacat. Solr allows you to quickly search through any coordinating or member node. Some more common use-cases include: Recovering lost PIDs - if you publish new data objects (publish_object) but forget to save the PIDs, you can recover them with a query. For example, you can search for all data objects that are not associated with a resource map and are published using your ORCiD (q=-resourceMap:*+AND+submitter:*XXXX-YYYY-ZZZZ-WWWW) Working with groups of packages - if you want to find all data, metadata, and resource maps associated with a PI, you can use their last name (q=origin:*SURNAME*) or perhaps their ORCiD (q=rightsHolder:*XXXX-YYYY-ZZZZ-WWWW) or both (q=origin:*SURNAME*+OR+rightsHolder:*XXXX-YYYY-ZZZZ-WWWW) to retrieve information about packages associated with them Once you understand the logic of queries, it becomes a flexible and useful tool that you can integrate into your R workflow. You can use queries to answer a variety of interesting questions, for example: What are the most recently updated data sets? What metadata and data objects are in a given data package? What is the total size (in terms of disk space) of all objects stored in Metacat? Querying Solr is possible by adding a query onto the end of a base URL or through the dataone::query() function in R. For now, we’ll just cover the basics of Solr queries in R. "],["construct-a-query.html", "Construct a query", " Construct a query Each Solr query is comprised of a number of parameters. These are like arguments to a function in R, but they are entered as parts of a URL. The most common parameters are: q: The query. This is like subset() or dplyr::filter() in R. fl: What fields are returned for the documents that match your query (q). If not set, all fields are returned. rows: The maximum number of documents to return. Solr will truncate your result if the result size is greater than rows. sort: Sorts the result by the values in the given Solr field (e.g., sort by date uploaded). The query (q) parameter uses a syntax that looks like field:value, where field is one of the Solr fields and value is an expression. The expression can match a specific value exactly, e.g., q=identifier:arctic-data.7747.1 or q=identifier:\"doi:10.5065/D60P0X4S\", which finds the Solr document for a specific Object by PID (identifier). In the second example, the DOI PID is surrounded in double quotes. This is because Solr has reserved characters, of which : is one, so we have to help Solr by surrounding values with reserved characters in them in quotes or by escaping them. To view the list of query-able parameters on the Arctic Data Center and their descriptions, you can visit https://arcticdata.io/metacat/d1/mn/v2/query/solr. "],["query-solr-via-a-browser.html", "Query Solr via a browser", " Query Solr via a browser Solr is queried via what’s called an HTTP API (Application Program Interface). Practically, what this means it that you can execute a query in your browser by tacking a query onto a base URL. This is similar to the way Google handles your searches. If I search “soil science” in Google, for example, the URL becomes: https://www.google.com/search?q=soil+science&amp;oq=soil+science&amp;aqs=chrome.0.69i59.1350j0j1&amp;sourceid=chrome&amp;ie=UTF-8 If I break it down into pieces, I get: the base URL - https://www.google.com/search ?, after which the query parameters are listed the query - q=soil+science other parameters, which are separated by &amp; - oq=soil+science&amp;aqs=chrome.0.69i59.1350j0j1&amp;sourceid=chrome&amp;ie=UTF-8 Most of the time, you’ll query either the Arctic Data Center member node or the PROD coordinating node, which have the following base URLs: Arctic Data Center member node: https://arcticdata.io/metacat/d1/mn/v2/query/solr PROD coordinating node: https://cn.dataone.org/cn/v2/query/solr You can then append your query parameters to your base URL: https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q={QUERY}&amp;fl={FIELDS}&amp;rows={ROWS} Visit the base URL to see a list of fields Solr is storing for the objects it indexes. There is a large set of queryable fields, though not all types of objects will have values set for all of the possible fields because some fields do not make sense for some objects (e.g., title for a CSV). "],["query-solr-via-r.html", "Query Solr via R", " Query Solr via R Though it’s possible to query Solr directly through its HTTP API, we typically run our queries through R, for two main reasons: The result is returned in a more useful way to R without extra work on your part We can more easily pass our authentication token with the query Why does #2 matter? Well by default, all of those URLs above only return publicly-readable Solr documents. If a private document matches any of those queries, Solr won’t tell you that. It will act like the non-public-readable documents don’t exist. So we must pass an authentication token to access non-public-readable content. This bit is crucial for working with the ADC, so you’ll very often want to use R instead of visiting those URLs in a web browser. cn &lt;- CNode(&quot;PROD&quot;) adc &lt;- getMNode(cn, &quot;urn:node:ARCTIC&quot;) # Set your token if you need/want! #string your parameters together like this: dataone::query(mn, &quot;q=title:*soil*&amp;fl=title&amp;rows=10&quot;) #or alternatively, list them out: query(adc, list(q=&quot;title:*soil*&quot;, fl=&quot;title&quot;, rows=&quot;10&quot;)) By default, query returns the result as a list, but a data.frame can be a more useful way to work with the result. To get a data.frame instead, just set the as argument to ‘data.frame’ to get a data.frame: query(adc, list(q=&quot;title:*soil*&quot;, fl=&quot;title&quot;, rows=&quot;10&quot;), as = &quot;data.frame&quot;) "],["use-facets.html", "Use facets", " Use facets We can also summarize what’s in Solr with faceting, which lets us group Solr documents together and count them. This is like table in R. Faceting can do a query within a query, but more commonly it’s used to summarize unique values in a field. For example, we can find the unique formatIds on data objects: https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q=:&amp;fq=formatType:DATA&amp;facet=true&amp;facet.field=formatId&amp;rows=0 To facet, we usually do a few things: Add the parameter facet=true Add the parameter facet.field={FIELD} with the field we want to facet (group) on Set rows=0 because we don’t care about the matched Solr documents Optionally specify fq={expression} which filters out Solr documents before faceting. In the above example, we have to do this to count only data objects. Without it, the facet result would include formatIds for metadata and resource maps, which we don’t want. Currently, the dataone::query() function does not support faceting, so you’ll have to run your queries as a URL. For additional ways to use faceting (such as pivot faceting), check out the Solr documentation. "],["use-stats.html", "Use stats", " Use stats With stats, we can have Solr calculate statistics on numerical values (such as size). https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q=formatType:DATA&amp;stats=true&amp;stats.field=size&amp;rows=0 This query calculates a set of summary statistics for the size field on data objects that Solr has indexed. In this case, Solr’s size field indexes the size field in the system metadata for each object in Metacat. "],["example-solr-queries.html", "Example Solr queries", " Example Solr queries For additional examples and explanations, check out the Apache Lucene Query Parser Syntax page. Find everything result &lt;- query(adc, list(q=&quot;*:*&quot;, fl=&quot;*&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) Query a wildcard expression #find any id that starts with arctic-data.6 result &lt;- query(adc, list(q=&quot;id:arctic-data.6*&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query multiple fields result &lt;- query(adc, list(q=&quot;title:soil*+AND+origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) result &lt;- query(adc, list(q=&quot;title:soil* OR origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) You can use either spaces or + to separate query parameters. When typing queries in R, it’s often easier to read if you use spaces. However, when using the browser, you may want to use + to keep the query clean. (The browser will replace spaces with %20.) Query by formatType Objects are queryable via their formatType which is one of DATA, METADATA, or RESOURCE. result &lt;- query(adc, list(q = &#39;formatType:RESOURCE AND submitter:&quot;http://orcid.org/0000-0002-2561-5840&quot;&#39;, fl = &#39;identifier,submitter,fileName&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;10&#39;), as = &quot;data.frame&quot;) result &lt;- query(adc, list(q = &#39;formatType:METADATA AND title:*Toolik*&#39;, fl = &#39;identifier,submitter,fileName&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;10&#39;), as = &quot;data.frame&quot;) Query pids by a specific submitter result &lt;- query(adc, list(q = &#39;submitter:&quot;http://orcid.org/0000-0003-4703-1974&quot;&#39;, fl = &#39;identifier,submitter,fileName, size&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;1000&#39;), as = &quot;data.frame&quot;) Query pids with special characters # Wrap the pid with special characters with escaped backslashes dataone::query(adc, list(q = paste0(&#39;id:&#39;, &#39;\\&quot;&#39;, &#39;doi:10.18739/A20R9M36V&#39;, &#39;\\&quot;&#39;), fl = &#39;dateUploaded AND identifier&#39;, rows = 5000), as = &quot;data.frame&quot;) Query multiple conditions within one field result &lt;- query(adc, list(q=&quot;title:(soil* AND carbo*)&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query for latest versions only result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId&quot;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) Use NOT in a query Just add - before a query parameter! result &lt;- query(adc, list(q=&quot;title:(soil AND -carbon)&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query a coordinating node result &lt;- query(cn, list(q=&quot;title:soil* AND origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query for EMLs that document a specific data pid # Wrap the pid in escaped quotation marks if it contains special characters query(adc, list(q = paste0(&#39;documents:&#39;, &#39;\\&quot;&#39;, &#39;urn:uuid:f551460b-ce36-4dd3-aaa6-3a6c6e338ec9&#39;, &#39;\\&quot;&#39;), fl = &quot;identifier&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) # Alternatively we can use a wildcard expression in place of &quot;urn:uuid:&quot; # This way we don&#39;t need to escape the colon special characters query(adc, list(q = paste0(&quot;documents:*f551460b-ce36-4dd3-aaa6-3a6c6e338ec9&quot;), fl = &quot;identifier&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) Query for files uploaded during a specific time query(adc, list(q=&quot;dateUploaded:[2020-05-06T00:00:00Z TO NOW]&quot;, fl=&quot;title,identifier,resourceMap,dateUploaded,dateModified&quot;, sort = &quot;dateUploaded+desc&quot;, rows=&quot;200&quot;), as = &quot;data.frame&quot;) ## identifier ## 1 resource_map_doi:10.18739/A2R785R0C ## 2 doi:10.18739/A2R785R0C ## 3 resource_map_doi:10.18739/A2W08WJ3N ## 4 doi:10.18739/A2W08WJ3N ## 5 resource_map_doi:10.18739/A20P0WS61 ## 6 doi:10.18739/A20P0WS61 ## 7 resource_map_doi:10.18739/A24F1MK9P ## 8 doi:10.18739/A24F1MK9P ## 9 resource_map_doi:10.18739/A2862BD3F ## 10 doi:10.18739/A2862BD3F ## 11 urn:uuid:c3363bfa-2f6c-47c4-a8df-d533ffe1e9bb ## 12 resource_map_doi:10.18739/A2HQ3S124 ## 13 doi:10.18739/A2HQ3S124 ## 14 resource_map_urn:uuid:f0b520a4-f549-4bef-b2a7-eca9145c76d5 ## 15 urn:uuid:85869ebc-1139-49a9-a454-e020956b49d4 ## 16 resource_map_doi:10.18739/A2NG4GT6V ## 17 doi:10.18739/A2NG4GT6V ## 18 resource_map_doi:10.18739/A2S756M9H ## 19 doi:10.18739/A2S756M9H ## 20 resource_map_doi:10.18739/A2WW7720N ## 21 doi:10.18739/A2WW7720N ## 22 resource_map_urn:uuid:39f2123a-5c64-4e42-a2df-27f5777291f8 ## 23 urn:uuid:39f2123a-5c64-4e42-a2df-27f5777291f8 ## 24 urn:uuid:8529ef01-5ed3-4200-aa7a-0fb599be9126 ## 25 resource_map_doi:10.18739/A21N7XP4R ## 26 doi:10.18739/A21N7XP4R ## 27 urn:uuid:1d746797-58f7-4340-9879-b423d3765df3 ## 28 resource_map_urn:uuid:7590a8d7-8b3e-4b8f-a055-a08d10bc8d79 ## 29 urn:uuid:5a38c2f3-e3eb-4cfe-9806-a949f9063543 ## 30 urn:uuid:3b08037e-3daa-4d74-816f-1900792bcb48 ## 31 urn:uuid:1d3d8eee-b9e9-4ed0-81df-a840770ebdb3 ## 32 resource_map_doi:10.18739/A25D8NG8W ## 33 doi:10.18739/A25D8NG8W ## 34 resource_map_doi:10.18739/A2959C92N ## 35 doi:10.18739/A2959C92N ## 36 resource_map_urn:uuid:2e2cb0e0-a346-459f-a7cd-3f1a2cda882a ## 37 urn:uuid:954d62e9-b105-4c80-bb81-4599e3270bdf ## 38 resource_map_urn:uuid:e96a5179-6b35-440a-b2fc-2ef84fdbc63a ## 39 urn:uuid:d2bc3531-df72-4d32-9f10-64d648769af1 ## 40 resource_map_doi:10.18739/A2DZ0337H ## 41 doi:10.18739/A2DZ0337H ## 42 resource_map_urn:uuid:4c064889-096c-41e0-8715-81d6a6673846 ## 43 urn:uuid:4c064889-096c-41e0-8715-81d6a6673846 ## 44 resource_map_urn:uuid:4a198a6b-0d2c-42e8-9a55-69d09f168f62 ## 45 urn:uuid:925dda2a-f7cc-404b-b3e1-26f07b9f19e6 ## 46 resource_map_doi:10.18739/A2JQ0SX1G ## 47 doi:10.18739/A2JQ0SX1G ## 48 resource_map_urn:uuid:a42fe946-61e7-4f3f-9042-d27bd1b5c059 ## 49 urn:uuid:a65f8eb5-ca78-49fc-ba72-7e3837cca0f3 ## 50 resource_map_urn:uuid:6e93e759-d6b8-4141-8bed-818e7c1da076 ## 51 urn:uuid:3d1553dd-6ccb-4f5f-8608-57f7aacd2698 ## 52 resource_map_urn:uuid:dbb03275-f1d6-46f6-b73b-49f9624e35e1 ## 53 urn:uuid:e85eb97d-3a57-4aa9-8b95-edec63df1e52 ## 54 resource_map_urn:uuid:4085ac9c-81b4-4193-bfcc-fdc33fe2e2a7 ## 55 urn:uuid:1a334a5e-3a7c-4707-9b61-2b0fa7ee11b7 ## 56 resource_map_urn:uuid:dfcd5365-39f1-4a80-9b23-594ae7a516e7 ## 57 urn:uuid:4aa92ad5-2ae7-4dfd-af53-74fad3c31732 ## 58 resource_map_urn:uuid:7eb3c850-8262-4a5a-a521-35c2ee655733 ## 59 urn:uuid:7eb3c850-8262-4a5a-a521-35c2ee655733 ## 60 urn:uuid:e1250286-b19c-40c4-a39b-4f24a81d84b0 ## 61 urn:uuid:d593a60e-bbf5-4d4a-9900-535f568a11fe ## 62 resource_map_doi:10.18739/A2T43J47X ## 63 doi:10.18739/A2T43J47X ## 64 resource_map_urn:uuid:a0999bb2-62e0-4bcc-8ec0-b058321b050b ## 65 urn:uuid:f3312ad0-4d3e-4b07-9cdd-84152a45384a ## 66 urn:uuid:ac3e7c66-b1db-4ad4-a899-0fbcb19755d6 ## 67 resource_map_doi:10.18739/A2XW47Z0G ## 68 doi:10.18739/A2XW47Z0G ## 69 resource_map_urn:uuid:0dd6b926-4f0d-474e-8144-df0b727849e7 ## 70 urn:uuid:01c91431-96d6-4194-bb55-96cdb19853a9 ## 71 resource_map_urn:uuid:5c4625df-b750-4928-8653-cddbb507dfb2 ## 72 urn:uuid:5eedffbb-cba9-4ed4-ad96-7e69fd385d8c ## 73 resource_map_urn:uuid:f6260076-30be-4b28-bf85-6b6e70edd647 ## 74 urn:uuid:f6260076-30be-4b28-bf85-6b6e70edd647 ## 75 resource_map_urn:uuid:bf846180-b410-4c08-8264-a87a631c115b ## 76 urn:uuid:6c1ecfa2-ca5b-403f-9686-4880378ac8e8 ## 77 resource_map_urn:uuid:612cf386-b7ab-4fbd-95f4-3459ade37516 ## 78 urn:uuid:b4dfd84e-366f-42b6-a000-77a83ebfe511 ## 79 resource_map_urn:uuid:24559da1-c111-4797-8169-0d1600448f96 ## 80 urn:uuid:24559da1-c111-4797-8169-0d1600448f96 ## 81 resource_map_urn:uuid:b613e260-bcff-473d-b50b-125beea8106d ## 82 urn:uuid:b613e260-bcff-473d-b50b-125beea8106d ## 83 resource_map_urn:uuid:ac6b284d-c9b5-43b4-9c9a-9a0619cf6742 ## 84 urn:uuid:ac6b284d-c9b5-43b4-9c9a-9a0619cf6742 ## 85 resource_map_urn:uuid:a7bda9ee-28d6-40f6-8c00-6bcd1c693e0c ## 86 urn:uuid:7c686dcc-46e9-4c9e-a4ff-67319f2b9196 ## 87 resource_map_urn:uuid:be7fc51a-21ca-408e-9f7b-087e13dbb347 ## 88 urn:uuid:be7fc51a-21ca-408e-9f7b-087e13dbb347 ## 89 urn:uuid:56c3ab8e-951b-4c4a-abad-6174f83780fd ## 90 resource_map_urn:uuid:d7cf28c2-4f84-434e-afe1-06a7cfcf03fa ## 91 urn:uuid:32cfbc35-8eaa-4944-99b4-001d69ddc6ef ## 92 resource_map_urn:uuid:755c7d65-6382-4c3c-bde6-d44dd33b30f2 ## 93 urn:uuid:50ca6864-180b-478a-9bfe-fbe3b57f38e7 ## 94 urn:uuid:429acf20-74c4-4d83-9883-ffb9a4c3e89f ## 95 resource_map_urn:uuid:9d264156-43d8-470c-b5d5-a26965fddfed ## 96 urn:uuid:9d264156-43d8-470c-b5d5-a26965fddfed ## 97 resource_map_urn:uuid:9b0f11de-849b-49ed-b1e9-df9c4310ad0e ## 98 urn:uuid:9b0f11de-849b-49ed-b1e9-df9c4310ad0e ## 99 resource_map_urn:uuid:8720c79f-2f12-4b04-a613-81dbc22255be ## 100 urn:uuid:432db1b6-b283-45da-a9f7-5a8a933c1902 ## 101 resource_map_urn:uuid:0b2936f2-7c2a-4802-9ab2-2ee5dc75c2e6 ## 102 urn:uuid:42597b8f-92f9-4016-8e58-1baf8d519c96 ## 103 resource_map_doi:10.18739/A22N4ZK5G ## 104 doi:10.18739/A22N4ZK5G ## 105 resource_map_urn:uuid:fb99a1a2-26b7-430f-8a96-dae482007fbc ## 106 urn:uuid:7f016267-18b2-49f1-ab57-1f32d1a8621a ## 107 resource_map_doi:10.18739/A26D5PC9M ## 108 doi:10.18739/A26D5PC9M ## 109 resource_map_doi:10.18739/A2B56D64V ## 110 doi:10.18739/A2B56D64V ## 111 resource_map_doi:10.18739/A2FX7408P ## 112 doi:10.18739/A2FX7408P ## 113 resource_map_urn:uuid:36786f14-6838-432b-be3e-3d65e33bcc5e ## 114 urn:uuid:542e0f82-8c6d-4a3e-b7fb-48835f438914 ## 115 resource_map_doi:10.18739/A2KP7TT15 ## 116 doi:10.18739/A2KP7TT15 ## 117 urn:uuid:a194fb25-faa7-444b-ba56-9a73db8f3a73 ## 118 urn:uuid:a2ffcf35-6716-441a-b893-94eb8bc321fb ## 119 resource_map_doi:10.18739/A2QB9V73G ## 120 doi:10.18739/A2QB9V73G ## 121 resource_map_urn:uuid:ec28eadc-cf17-4abc-b16e-69cf87f0bebd ## 122 urn:uuid:430922f6-6b67-4491-8faf-29eb47c18a24 ## 123 resource_map_doi:10.18739/A2V40K16P ## 124 doi:10.18739/A2V40K16P ## 125 urn:uuid:bccebda1-cb04-457b-bc91-410945b0dc87 ## 126 resource_map_urn:uuid:65705b6d-13f6-4dcf-a44f-5f5ddd640509 ## 127 urn:uuid:ee566c8c-ca10-43df-812a-a9df59b3667e ## 128 urn:uuid:140e2233-fb9d-4ae2-a742-5c913c3f7724 ## 129 urn:uuid:0bfa3290-4c23-4cc6-b05d-563c32a1e725 ## 130 urn:uuid:8a89ade9-6166-4423-9b44-30a29d416cd9 ## 131 urn:uuid:7a589ae5-fd5f-421c-acda-c9b29f569686 ## 132 urn:uuid:55b3f360-976e-4bdb-8047-584f9114f2d4 ## 133 urn:uuid:5cb95853-e19a-4d9c-94dc-95254aed167d ## 134 urn:uuid:f76ce231-0fb2-4598-b06e-1d457ecec01d ## 135 urn:uuid:80169551-04de-4b1b-ae74-0c11d65ee407 ## 136 urn:uuid:1ca4c0df-c368-4c7c-8d1d-cf3cc36ccf8d ## 137 urn:uuid:24561338-dc5f-4207-a6ff-9b8bbdcb61fa ## 138 urn:uuid:23e8bbf5-1a6c-47c3-8840-bca2f86c6d5f ## 139 urn:uuid:c7946427-5ce0-4e18-a30a-3cc424f36dd0 ## 140 urn:uuid:48c0f101-16a2-4e22-a8c6-3b7acfe0b34d ## 141 urn:uuid:2189fd1c-5b6e-48f9-a7ab-4fdeca81ea87 ## 142 urn:uuid:017e2f63-3b35-432e-a3d0-20cea9d877da ## 143 urn:uuid:9135878f-6cd3-4362-8769-0dc0e4abef3e ## 144 urn:uuid:2e318964-e575-4a32-bc6f-81b93803f6e1 ## 145 urn:uuid:09cb6c47-ca6f-4973-99f4-3f2336d664fb ## 146 urn:uuid:4deb9ada-b814-4258-932b-6b8e97486d40 ## 147 urn:uuid:653b66c3-a839-45fb-a2b8-4f17f099b2dc ## 148 urn:uuid:aa9d7dc7-7c97-4fed-9cea-94e8995c3906 ## 149 urn:uuid:ea334339-7c25-4715-8edb-34f2bef57200 ## 150 urn:uuid:7d104a59-b544-4167-81b8-1c56dd556602 ## 151 urn:uuid:b296ec63-41eb-46a7-bc22-40a04bd86fb6 ## 152 urn:uuid:e26ea363-aa7d-45d3-8c3e-65afe8547fa1 ## 153 urn:uuid:4f6c3e58-d940-4a04-972e-1b9334d20ef1 ## 154 urn:uuid:8b66a954-88e7-4006-b1cf-a80a08ce02d5 ## 155 urn:uuid:2e11562a-ce15-4122-b969-984b865bef84 ## 156 urn:uuid:5adaf24a-9486-44dd-9fea-238c4fa712e2 ## 157 urn:uuid:f8d58047-10bc-4032-a324-367e4d2ea602 ## 158 resource_map_urn:uuid:353bd1c1-f691-42fe-afb7-eada331f6c04 ## 159 urn:uuid:990f7311-dfcc-4d33-a5a9-56a5e610d076 ## 160 resource_map_doi:10.18739/A2ZW18V0Q ## 161 doi:10.18739/A2ZW18V0Q ## 162 resource_map_urn:uuid:4fa2d26d-f449-48dc-9686-6307f883380f ## 163 urn:uuid:e3114753-7241-409d-86dc-42f42e5e9bba ## 164 resource_map_doi:10.18739/A23N20G53 ## 165 doi:10.18739/A23N20G53 ## 166 resource_map_urn:uuid:3ceed3d3-0697-4615-9d3d-eb16879a1c96 ## 167 urn:uuid:0efbfc54-1f2d-43fc-b26c-3af433054e61 ## 168 resource_map_urn:uuid:43ccbf5e-95f2-4398-9f02-eb24672f4509 ## 169 urn:uuid:31514230-fd0a-436a-9447-291d24da03ee ## 170 resource_map_doi:10.18739/A27D2Q90Z ## 171 doi:10.18739/A27D2Q90Z ## 172 resource_map_urn:uuid:c5e3d6ab-51ce-4369-8843-8e347b73dd41 ## 173 urn:uuid:f71379b3-6585-4a56-9f8f-4b2298a5db7c ## 174 resource_map_urn:uuid:3ec88ccd-dcfd-4dc1-ba8f-e26c4d79d189 ## 175 resource_map_urn:uuid:802fd2f7-9e78-4f63-89a3-39f70707ceaf ## 176 urn:uuid:cdc8bf98-fa22-44d4-9590-64c514867495 ## 177 resource_map_urn:uuid:bb818a59-21eb-4070-abda-36c6b101297a ## 178 resource_map_urn:uuid:d4d9d7a5-b279-4b50-bc4b-beb745496a17 ## 179 urn:uuid:d5409690-e577-477c-8149-7778ba4d5f44 ## 180 urn:uuid:6ce067fa-d9e8-4e31-9b10-996a5a005580 ## 181 urn:uuid:4f16e011-07e9-4917-a94a-c841deef75b2 ## 182 resource_map_doi:10.18739/A2C53F343 ## 183 doi:10.18739/A2C53F343 ## 184 resource_map_urn:uuid:94e710ae-6f41-4b75-b0e1-4d7f495036e9 ## 185 urn:uuid:94e710ae-6f41-4b75-b0e1-4d7f495036e9 ## 186 urn:uuid:97746d45-392f-4766-aa9f-28920bb0ab2b ## 187 urn:uuid:abacb59f-9499-4f2c-b620-e8335e9e7fe1 ## 188 urn:uuid:637901af-3eba-46db-856a-9f4e233d144a ## 189 urn:uuid:3a4c8896-fa7f-49f4-9e13-352ba859822c ## 190 urn:uuid:f875afea-4fe1-4156-8eb0-9d6803ba9e90 ## 191 urn:uuid:f695424d-4430-4017-b7b6-a565eb5b88f8 ## 192 urn:uuid:339324fd-946e-48c1-a2aa-be5a41437f18 ## 193 urn:uuid:442e3761-4d75-41c3-b7d8-1d405fedbf73 ## 194 urn:uuid:471a0bb1-43e0-48e6-9b79-233f54769660 ## 195 urn:uuid:6740416a-f07a-47e9-8e79-853c52bff278 ## 196 urn:uuid:26527141-4c8a-4a73-b78c-a34268446f62 ## 197 urn:uuid:d9a64d81-6c77-4185-964f-018156ada0c6 ## 198 urn:uuid:0bc9a4d3-f53f-4e35-adb8-6c0bef5d0443 ## 199 urn:uuid:8a14d17a-c8a1-46a8-aaa3-5a0f28779ea2 ## 200 urn:uuid:13a17977-6649-4308-9c38-27e796f26b3e ## dateUploaded dateModified ## 1 2024-08-12 22:28:49 2024-08-12 22:28:50 ## 2 2024-08-12 22:28:43 2024-08-12 22:28:46 ## 3 2024-08-12 18:17:49 2024-08-12 18:17:50 ## 4 2024-08-12 18:17:41 2024-08-12 18:17:44 ## 5 2024-08-12 17:44:38 2024-08-12 17:44:38 ## 6 2024-08-12 17:44:31 2024-08-12 17:44:35 ## 7 2024-08-12 17:41:27 2024-08-12 17:41:28 ## 8 2024-08-12 17:41:20 2024-08-12 17:41:23 ## 9 2024-08-08 23:53:25 2024-08-08 23:53:26 ## 10 2024-08-08 23:53:18 2024-08-08 23:53:21 ## 11 2024-08-08 23:07:30 2024-08-08 23:07:33 ## 12 2024-08-07 20:34:59 2024-08-07 20:35:00 ## 13 2024-08-07 20:34:52 2024-08-07 20:34:56 ## 14 2024-08-07 20:33:44 2024-08-07 20:34:59 ## 15 2024-08-07 20:33:39 2024-08-07 20:34:55 ## 16 2024-08-07 20:32:49 2024-08-07 20:33:44 ## 17 2024-08-07 20:32:41 2024-08-07 20:33:42 ## 18 2024-08-07 17:44:24 2024-08-07 20:32:49 ## 19 2024-08-07 17:43:58 2024-08-07 20:32:44 ## 20 2024-08-05 21:23:27 2024-08-05 21:23:28 ## 21 2024-08-05 21:23:21 2024-08-05 21:23:24 ## 22 2024-08-05 21:21:47 2024-08-05 21:23:27 ## 23 2024-08-05 21:21:38 2024-08-05 21:23:23 ## 24 2024-08-05 16:33:16 2024-08-12 17:44:27 ## 25 2024-08-05 16:19:53 2024-08-05 16:19:54 ## 26 2024-08-05 16:19:46 2024-08-05 16:19:49 ## 27 2024-08-02 23:27:48 2024-08-07 20:34:51 ## 28 2024-08-02 21:24:41 2024-08-05 21:21:47 ## 29 2024-08-02 21:24:32 2024-08-05 21:21:43 ## 30 2024-08-02 21:24:32 2024-08-05 21:23:20 ## 31 2024-08-02 17:21:51 2024-08-05 16:19:45 ## 32 2024-08-01 23:33:31 2024-08-01 23:33:31 ## 33 2024-08-01 23:33:15 2024-08-01 23:33:26 ## 34 2024-08-01 20:50:22 2024-08-01 20:50:23 ## 35 2024-08-01 20:50:15 2024-08-01 20:50:19 ## 36 2024-08-01 20:46:20 2024-08-01 20:50:22 ## 37 2024-08-01 20:46:15 2024-08-01 20:50:18 ## 38 2024-08-01 20:44:56 2024-08-01 20:46:21 ## 39 2024-08-01 20:44:50 2024-08-01 20:46:18 ## 40 2024-08-01 20:31:55 2024-08-01 20:44:56 ## 41 2024-08-01 20:31:48 2024-08-01 20:44:54 ## 42 2024-08-01 20:29:56 2024-08-01 20:31:55 ## 43 2024-08-01 20:29:47 2024-08-01 20:31:51 ## 44 2024-08-01 20:20:05 2024-08-01 20:29:56 ## 45 2024-08-01 20:19:57 2024-08-01 20:29:50 ## 46 2024-08-01 18:00:12 2024-08-01 20:20:05 ## 47 2024-08-01 17:59:54 2024-08-01 20:20:02 ## 48 2024-08-01 17:40:30 2024-08-01 17:40:31 ## 49 2024-08-01 17:40:22 2024-08-01 17:40:28 ## 50 2024-08-01 17:25:33 2024-08-01 17:40:30 ## 51 2024-08-01 17:25:25 2024-08-01 17:40:28 ## 52 2024-08-01 17:09:51 2024-08-01 17:25:33 ## 53 2024-08-01 17:09:45 2024-08-01 17:25:31 ## 54 2024-08-01 17:09:09 2024-08-01 17:09:51 ## 55 2024-08-01 17:09:03 2024-08-01 17:09:49 ## 56 2024-08-01 17:05:57 2024-08-01 17:09:09 ## 57 2024-08-01 17:05:48 2024-08-01 17:09:07 ## 58 2024-08-01 16:55:38 2024-08-01 17:05:57 ## 59 2024-08-01 16:55:27 2024-08-01 17:05:54 ## 60 2024-08-01 13:51:23 2024-08-12 17:41:19 ## 61 2024-07-31 20:04:29 2024-08-01 17:59:52 ## 62 2024-07-31 19:44:10 2024-07-31 19:44:11 ## 63 2024-07-31 19:44:04 2024-07-31 19:44:06 ## 64 2024-07-31 19:40:44 2024-07-31 19:44:10 ## 65 2024-07-31 19:40:40 2024-07-31 19:44:05 ## 66 2024-07-31 19:39:16 2024-07-31 19:44:03 ## 67 2024-07-31 19:38:06 2024-07-31 19:40:45 ## 68 2024-07-31 19:38:01 2024-07-31 19:40:41 ## 69 2024-07-31 18:31:23 2024-08-01 16:55:38 ## 70 2024-07-31 18:31:17 2024-08-01 16:55:32 ## 71 2024-07-31 18:24:18 2024-07-31 18:31:23 ## 72 2024-07-31 18:24:13 2024-07-31 18:31:21 ## 73 2024-07-31 18:22:50 2024-07-31 18:24:18 ## 74 2024-07-31 18:22:42 2024-07-31 18:24:17 ## 75 2024-07-31 18:18:47 2024-07-31 18:22:50 ## 76 2024-07-31 18:18:40 2024-07-31 18:22:45 ## 77 2024-07-31 18:15:45 2024-07-31 18:18:47 ## 78 2024-07-31 18:15:39 2024-07-31 18:18:44 ## 79 2024-07-31 18:14:36 2024-07-31 18:15:45 ## 80 2024-07-31 18:14:26 2024-07-31 18:15:43 ## 81 2024-07-31 18:11:59 2024-07-31 18:14:36 ## 82 2024-07-31 18:11:51 2024-07-31 18:14:30 ## 83 2024-07-31 18:02:26 2024-07-31 18:11:59 ## 84 2024-07-31 18:02:15 2024-07-31 18:11:54 ## 85 2024-07-31 18:00:11 2024-07-31 18:02:26 ## 86 2024-07-31 17:59:56 2024-07-31 18:02:20 ## 87 2024-07-31 17:50:29 2024-07-31 18:00:11 ## 88 2024-07-31 17:50:17 2024-07-31 18:00:08 ## 89 2024-07-31 17:06:57 2024-07-31 19:38:00 ## 90 2024-07-31 16:26:42 2024-07-31 17:50:29 ## 91 2024-07-31 16:26:31 2024-07-31 17:50:25 ## 92 2024-07-31 02:31:57 2024-08-02 20:29:50 ## 93 2024-07-31 02:31:20 2024-07-31 02:31:55 ## 94 2024-07-31 02:19:03 2024-07-31 02:31:53 ## 95 2024-07-30 23:52:57 2024-07-31 16:26:43 ## 96 2024-07-30 23:52:50 2024-07-31 16:26:40 ## 97 2024-07-30 23:48:21 2024-07-30 23:52:57 ## 98 2024-07-30 23:48:11 2024-07-30 23:52:53 ## 99 2024-07-30 23:36:30 2024-07-30 23:48:21 ## 100 2024-07-30 23:36:26 2024-07-30 23:48:16 ## 101 2024-07-30 23:32:03 2024-07-30 23:36:30 ## 102 2024-07-30 23:31:58 2024-07-30 23:36:29 ## 103 2024-07-30 22:11:48 2024-07-30 22:11:49 ## 104 2024-07-30 22:11:40 2024-07-30 22:11:43 ## 105 2024-07-30 22:10:52 2024-07-30 22:11:48 ## 106 2024-07-30 22:10:43 2024-07-30 22:11:43 ## 107 2024-07-30 21:58:07 2024-07-30 21:58:08 ## 108 2024-07-30 21:58:03 2024-07-30 21:58:05 ## 109 2024-07-30 19:55:36 2024-07-30 19:55:36 ## 110 2024-07-30 19:55:31 2024-07-30 19:55:32 ## 111 2024-07-30 19:52:45 2024-07-30 19:52:45 ## 112 2024-07-30 19:52:36 2024-07-30 19:52:40 ## 113 2024-07-30 18:28:16 2024-07-30 23:32:04 ## 114 2024-07-30 18:28:10 2024-07-30 23:32:01 ## 115 2024-07-30 18:00:59 2024-07-30 18:01:00 ## 116 2024-07-30 18:00:53 2024-07-30 18:00:55 ## 117 2024-07-30 17:31:00 2024-07-30 18:00:52 ## 118 2024-07-30 16:54:42 2024-07-30 18:00:51 ## 119 2024-07-29 19:46:15 2024-07-30 22:10:52 ## 120 2024-07-29 19:46:06 2024-07-30 22:10:50 ## 121 2024-07-29 19:44:42 2024-07-29 19:46:15 ## 122 2024-07-29 19:44:32 2024-07-29 19:46:09 ## 123 2024-07-29 16:42:46 2024-07-29 16:42:46 ## 124 2024-07-29 16:42:38 2024-07-29 16:42:42 ## 125 2024-07-28 13:21:29 2024-07-28 13:21:37 ## 126 2024-07-27 00:47:40 2024-07-27 00:47:40 ## 127 2024-07-27 00:47:24 2024-07-31 02:19:33 ## 128 2024-07-27 00:47:08 2024-07-31 02:19:22 ## 129 2024-07-27 00:47:08 2024-07-31 02:19:25 ## 130 2024-07-27 00:47:08 2024-07-31 02:19:23 ## 131 2024-07-27 00:47:08 2024-07-31 02:19:21 ## 132 2024-07-27 00:47:07 2024-07-31 02:19:26 ## 133 2024-07-27 00:47:07 2024-07-31 02:19:35 ## 134 2024-07-27 00:47:07 2024-07-31 02:19:33 ## 135 2024-07-27 00:47:06 2024-07-31 02:19:27 ## 136 2024-07-27 00:47:06 2024-07-31 02:19:32 ## 137 2024-07-27 00:47:06 2024-07-31 02:19:36 ## 138 2024-07-27 00:47:06 2024-07-31 02:20:01 ## 139 2024-07-27 00:47:05 2024-07-31 02:19:38 ## 140 2024-07-27 00:47:05 2024-07-31 02:19:39 ## 141 2024-07-27 00:47:05 2024-07-31 02:19:41 ## 142 2024-07-27 00:47:05 2024-07-31 02:20:11 ## 143 2024-07-27 00:47:05 2024-07-31 02:19:44 ## 144 2024-07-27 00:47:04 2024-07-31 02:19:45 ## 145 2024-07-27 00:47:04 2024-07-31 02:20:16 ## 146 2024-07-27 00:47:04 2024-07-31 02:19:47 ## 147 2024-07-27 00:47:04 2024-07-31 02:19:54 ## 148 2024-07-27 00:47:04 2024-07-31 02:19:51 ## 149 2024-07-27 00:47:04 2024-07-31 02:20:05 ## 150 2024-07-27 00:47:02 2024-07-31 02:20:25 ## 151 2024-07-27 00:47:02 2024-07-31 02:19:58 ## 152 2024-07-27 00:47:02 2024-07-31 02:20:32 ## 153 2024-07-27 00:47:01 2024-07-31 02:20:40 ## 154 2024-07-27 00:47:01 2024-07-31 02:20:13 ## 155 2024-07-27 00:47:00 2024-07-31 02:20:28 ## 156 2024-07-27 00:47:00 2024-07-31 02:20:37 ## 157 2024-07-27 00:47:00 2024-07-31 02:20:20 ## 158 2024-07-26 19:16:27 2024-07-29 19:44:43 ## 159 2024-07-26 19:16:23 2024-07-29 19:44:40 ## 160 2024-07-26 19:05:50 2024-07-26 19:05:51 ## 161 2024-07-26 19:05:43 2024-07-26 19:05:47 ## 162 2024-07-26 18:40:05 2024-07-26 19:05:50 ## 163 2024-07-26 18:40:01 2024-07-26 19:05:46 ## 164 2024-07-26 17:24:44 2024-07-26 17:24:45 ## 165 2024-07-26 17:24:38 2024-07-26 17:24:40 ## 166 2024-07-26 17:17:32 2024-07-26 18:40:05 ## 167 2024-07-26 17:17:26 2024-07-26 18:40:03 ## 168 2024-07-26 17:11:07 2024-07-26 17:17:33 ## 169 2024-07-26 17:11:01 2024-07-26 17:17:29 ## 170 2024-07-26 17:04:04 2024-07-26 17:11:07 ## 171 2024-07-26 17:03:54 2024-07-26 17:11:05 ## 172 2024-07-26 16:45:29 2024-07-26 19:16:27 ## 173 2024-07-26 16:45:26 2024-07-26 19:16:25 ## 174 2024-07-26 16:41:48 2024-07-26 16:45:30 ## 175 2024-07-26 16:24:35 2024-07-26 16:41:48 ## 176 2024-07-26 16:24:31 2024-07-26 16:45:28 ## 177 2024-07-26 16:23:25 2024-07-26 16:24:35 ## 178 2024-07-26 16:17:27 2024-07-26 16:23:26 ## 179 2024-07-26 16:17:16 2024-07-26 16:24:34 ## 180 2024-07-26 14:56:50 2024-07-26 19:05:22 ## 181 2024-07-26 14:27:32 2024-07-26 19:05:38 ## 182 2024-07-25 21:54:24 2024-07-25 21:54:24 ## 183 2024-07-25 21:54:16 2024-07-25 21:54:19 ## 184 2024-07-25 20:31:53 2024-07-26 16:17:27 ## 185 2024-07-25 20:31:41 2024-07-26 16:17:25 ## 186 2024-07-25 20:18:06 2024-08-01 23:32:55 ## 187 2024-07-25 20:18:05 2024-08-01 23:32:50 ## 188 2024-07-25 20:18:04 2024-08-01 23:32:22 ## 189 2024-07-25 20:18:04 2024-08-01 23:33:00 ## 190 2024-07-25 20:18:03 2024-08-01 23:32:57 ## 191 2024-07-25 20:18:02 2024-08-01 23:33:09 ## 192 2024-07-25 20:18:01 2024-08-01 23:33:10 ## 193 2024-07-25 20:17:59 2024-08-01 23:31:44 ## 194 2024-07-25 20:17:58 2024-08-01 23:32:13 ## 195 2024-07-25 20:17:57 2024-08-01 23:32:03 ## 196 2024-07-25 20:17:56 2024-08-01 23:32:24 ## 197 2024-07-25 20:17:56 2024-08-01 23:32:59 ## 198 2024-07-25 20:17:55 2024-08-01 23:31:55 ## 199 2024-07-25 20:17:53 2024-08-01 23:32:15 ## 200 2024-07-25 20:17:52 2024-08-01 23:32:46 ## title ## 1 &lt;NA&gt; ## 2 In Situ Observations from the International Arctic Buoy Programme (IABP) 2023, Level 1 ## 3 &lt;NA&gt; ## 4 Near-Real-Time National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 2 ## 5 &lt;NA&gt; ## 6 Small mammal mark-recapture data 2018-2023 (3 sites near Utqiagvik, Alaska) ## 7 &lt;NA&gt; ## 8 Small mammal mark-recapture data 2017-2023 (3 sites near Toolik Lake, Alaska) ## 9 &lt;NA&gt; ## 10 Surface underway dissolved oxygen/nitrogen saturation ratio and oxygen concentration data collected in the Bering, Beaufort, and Chukchi Seas on research vessel Sikuliaq in 2023 ## 11 Proceedings of the Permafrost &amp; Infrastructure Symposium ## 12 &lt;NA&gt; ## 13 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 14 &lt;NA&gt; ## 15 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 16 &lt;NA&gt; ## 17 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 18 &lt;NA&gt; ## 19 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 20 &lt;NA&gt; ## 21 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 22 &lt;NA&gt; ## 23 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 24 &lt;NA&gt; ## 25 &lt;NA&gt; ## 26 Global Vegetation Greenness Normalized Difference Vegetation Index (NDVI) from Advanced Very High Resolution Radiometer (AVHRR) Global Inventory Modeling and Mapping Studies-3rd Generation (GIMMS-3G+), 1981-2022 ## 27 &lt;NA&gt; ## 28 &lt;NA&gt; ## 29 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 30 &lt;NA&gt; ## 31 &lt;NA&gt; ## 32 &lt;NA&gt; ## 33 Sea Ice Motion Estimation Algorithm Using Computer Vision for The Sea Ice Dynamic Experiment (SIDEx) Field Campaign, Alaska, 2021 ## 34 &lt;NA&gt; ## 35 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 36 &lt;NA&gt; ## 37 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 38 &lt;NA&gt; ## 39 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 40 &lt;NA&gt; ## 41 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 42 &lt;NA&gt; ## 43 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 44 &lt;NA&gt; ## 45 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 46 &lt;NA&gt; ## 47 Near-Real-Time National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 48 &lt;NA&gt; ## 49 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 50 &lt;NA&gt; ## 51 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 52 &lt;NA&gt; ## 53 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 54 &lt;NA&gt; ## 55 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 56 &lt;NA&gt; ## 57 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 58 &lt;NA&gt; ## 59 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 60 &lt;NA&gt; ## 61 &lt;NA&gt; ## 62 &lt;NA&gt; ## 63 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 64 &lt;NA&gt; ## 65 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 66 &lt;NA&gt; ## 67 &lt;NA&gt; ## 68 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 69 &lt;NA&gt; ## 70 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 71 &lt;NA&gt; ## 72 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 73 &lt;NA&gt; ## 74 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 75 &lt;NA&gt; ## 76 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 77 &lt;NA&gt; ## 78 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 79 &lt;NA&gt; ## 80 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 81 &lt;NA&gt; ## 82 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 83 &lt;NA&gt; ## 84 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 85 &lt;NA&gt; ## 86 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 87 &lt;NA&gt; ## 88 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 89 &lt;NA&gt; ## 90 &lt;NA&gt; ## 91 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 92 &lt;NA&gt; ## 93 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 94 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 95 &lt;NA&gt; ## 96 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 97 &lt;NA&gt; ## 98 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 99 &lt;NA&gt; ## 100 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 101 &lt;NA&gt; ## 102 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 103 &lt;NA&gt; ## 104 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 105 &lt;NA&gt; ## 106 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 107 &lt;NA&gt; ## 108 Sea Ice Concentrations from Nimbus-7 Scanning Multichannel Microwave Radiometer (SMMR) and Defense Meteorological Satellites Program (DMSP) Special Sensor Microwave Imager - Special Sensor Microwave Imager Sounder (SSM/I-SSMIS) Passive Microwave Data, Version 1, North/South Poles, 1978 - 2021 ## 109 &lt;NA&gt; ## 110 Near-Real-Time Defense Meteorological Satellite Program (DMSP) Special Sensor Microwave Imager/Sounder (SSMIS) Daily Polar Gridded Sea Ice Concentrations, Version 1 ## 111 &lt;NA&gt; ## 112 Ice, Cloud, and land Elevation Satellite-2 (ICESat-2) L4 Monthly Gridded Sea Ice Thickness, Version 2, Arctic Ocean, 2018 - 2022 ## 113 &lt;NA&gt; ## 114 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 115 &lt;NA&gt; ## 116 Quicklook Arctic Weekly Equal-Area Scalable Earth Grid (EASE-Grid) Sea Ice Age, Version 1 ## 117 &lt;NA&gt; ## 118 &lt;NA&gt; ## 119 &lt;NA&gt; ## 120 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 121 &lt;NA&gt; ## 122 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 123 &lt;NA&gt; ## 124 Characterization of summertime carbonaceous aerosol on the North Slope of Alaska (Toolik and Utqiaġvik), 2022–2023 ## 125 Sermilik Fjord Hydrography ## 126 &lt;NA&gt; ## 127 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 128 &lt;NA&gt; ## 129 &lt;NA&gt; ## 130 &lt;NA&gt; ## 131 &lt;NA&gt; ## 132 &lt;NA&gt; ## 133 &lt;NA&gt; ## 134 &lt;NA&gt; ## 135 &lt;NA&gt; ## 136 &lt;NA&gt; ## 137 &lt;NA&gt; ## 138 &lt;NA&gt; ## 139 &lt;NA&gt; ## 140 &lt;NA&gt; ## 141 &lt;NA&gt; ## 142 &lt;NA&gt; ## 143 &lt;NA&gt; ## 144 &lt;NA&gt; ## 145 &lt;NA&gt; ## 146 &lt;NA&gt; ## 147 &lt;NA&gt; ## 148 &lt;NA&gt; ## 149 &lt;NA&gt; ## 150 &lt;NA&gt; ## 151 &lt;NA&gt; ## 152 &lt;NA&gt; ## 153 &lt;NA&gt; ## 154 &lt;NA&gt; ## 155 &lt;NA&gt; ## 156 &lt;NA&gt; ## 157 &lt;NA&gt; ## 158 &lt;NA&gt; ## 159 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 160 &lt;NA&gt; ## 161 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 162 &lt;NA&gt; ## 163 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 164 &lt;NA&gt; ## 165 Ice core methanesulfonic acid from the Begguya (Mount Hunter) plateau, Denali National Park, Alaska, 2013 ## 166 &lt;NA&gt; ## 167 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 168 &lt;NA&gt; ## 169 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 170 &lt;NA&gt; ## 171 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 172 &lt;NA&gt; ## 173 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 174 &lt;NA&gt; ## 175 &lt;NA&gt; ## 176 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 177 &lt;NA&gt; ## 178 &lt;NA&gt; ## 179 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 180 &lt;NA&gt; ## 181 &lt;NA&gt; ## 182 &lt;NA&gt; ## 183 Saline permafrost degradation below a shallow thermokarst lake in northern Alaska, 2008-2023 ## 184 &lt;NA&gt; ## 185 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 186 &lt;NA&gt; ## 187 &lt;NA&gt; ## 188 &lt;NA&gt; ## 189 &lt;NA&gt; ## 190 &lt;NA&gt; ## 191 &lt;NA&gt; ## 192 &lt;NA&gt; ## 193 &lt;NA&gt; ## 194 &lt;NA&gt; ## 195 &lt;NA&gt; ## 196 &lt;NA&gt; ## 197 &lt;NA&gt; ## 198 &lt;NA&gt; ## 199 &lt;NA&gt; ## 200 &lt;NA&gt; ## resourceMap ## 1 ## 2 resource.... ## 3 ## 4 resource.... ## 5 ## 6 resource.... ## 7 ## 8 resource.... ## 9 ## 10 resource.... ## 11 ## 12 ## 13 resource.... ## 14 ## 15 resource.... ## 16 ## 17 resource.... ## 18 ## 19 resource.... ## 20 ## 21 resource.... ## 22 ## 23 resource.... ## 24 resource.... ## 25 ## 26 resource.... ## 27 resource.... ## 28 ## 29 resource.... ## 30 resource.... ## 31 resource.... ## 32 ## 33 resource.... ## 34 ## 35 resource.... ## 36 ## 37 resource.... ## 38 ## 39 resource.... ## 40 ## 41 resource.... ## 42 ## 43 resource.... ## 44 ## 45 resource.... ## 46 ## 47 resource.... ## 48 ## 49 resource.... ## 50 ## 51 resource.... ## 52 ## 53 resource.... ## 54 ## 55 resource.... ## 56 ## 57 resource.... ## 58 ## 59 resource.... ## 60 resource.... ## 61 resource.... ## 62 ## 63 resource.... ## 64 ## 65 resource.... ## 66 resource.... ## 67 ## 68 resource.... ## 69 ## 70 resource.... ## 71 ## 72 resource.... ## 73 ## 74 resource.... ## 75 ## 76 resource.... ## 77 ## 78 resource.... ## 79 ## 80 resource.... ## 81 ## 82 resource.... ## 83 ## 84 resource.... ## 85 ## 86 resource.... ## 87 ## 88 resource.... ## 89 resource.... ## 90 ## 91 resource.... ## 92 ## 93 resource.... ## 94 ## 95 ## 96 resource.... ## 97 ## 98 resource.... ## 99 ## 100 resource.... ## 101 ## 102 resource.... ## 103 ## 104 resource.... ## 105 ## 106 resource.... ## 107 ## 108 resource.... ## 109 ## 110 resource.... ## 111 ## 112 resource.... ## 113 ## 114 resource.... ## 115 ## 116 resource.... ## 117 resource.... ## 118 resource.... ## 119 ## 120 resource.... ## 121 ## 122 resource.... ## 123 ## 124 resource.... ## 125 ## 126 ## 127 resource.... ## 128 resource.... ## 129 resource.... ## 130 resource.... ## 131 resource.... ## 132 resource.... ## 133 resource.... ## 134 resource.... ## 135 resource.... ## 136 resource.... ## 137 resource.... ## 138 resource.... ## 139 resource.... ## 140 resource.... ## 141 resource.... ## 142 resource.... ## 143 resource.... ## 144 resource.... ## 145 resource.... ## 146 resource.... ## 147 resource.... ## 148 resource.... ## 149 resource.... ## 150 resource.... ## 151 resource.... ## 152 resource.... ## 153 resource.... ## 154 resource.... ## 155 resource.... ## 156 resource.... ## 157 resource.... ## 158 ## 159 resource.... ## 160 ## 161 resource.... ## 162 ## 163 resource.... ## 164 ## 165 resource.... ## 166 ## 167 resource.... ## 168 ## 169 resource.... ## 170 ## 171 resource.... ## 172 ## 173 resource.... ## 174 ## 175 ## 176 resource.... ## 177 ## 178 ## 179 resource.... ## 180 resource.... ## 181 resource.... ## 182 ## 183 resource.... ## 184 ## 185 resource.... ## 186 resource.... ## 187 resource.... ## 188 resource.... ## 189 resource.... ## 190 resource.... ## 191 resource.... ## 192 resource.... ## 193 resource.... ## 194 resource.... ## 195 resource.... ## 196 resource.... ## 197 resource.... ## 198 resource.... ## 199 resource.... ## 200 resource.... Use facets All resource maps with &gt; 100 data objects that are not on the Arctic Data Center: https://cn.dataone.org/cn/v2/query/solr/?q=resourceMap:*+AND+-datasource:*ARCTIC*&amp;rows=0&amp;facet=true&amp;facet.field=resourceMap&amp;facet.mincount=100 "],["more-resources.html", "More resources", " More resources Solr’s The Standard Query Parser docs (high level of detail) Another quick reference: https://wiki.apache.org/solr/SolrQuerySyntax http://www.solrtutorial.com/ "],["adc-web-submissions.html", "ADC web submissions ", " ADC web submissions "],["add-physicals-to-submissions.html", "Add physicals to submissions", " Add physicals to submissions New submissions made through the web editor will not have any physical sections within the otherEntitys. Add them to the EML with the following script: for (i in seq_along(doc$dataset$otherEntity)) { otherEntity &lt;- doc$dataset$otherEntity[[i]] id &lt;- otherEntity$id if (!grepl(&quot;urn-uuid-&quot;, id)) { warning(&quot;otherEntity &quot;, i, &quot; is not a pid&quot;) } else { id &lt;- gsub(&quot;urn-uuid-&quot;, &quot;urn:uuid:&quot;, id) physical &lt;- arcticdatautils::pid_to_eml_physical(mn, id) doc$dataset$otherEntity[[i]]$physical &lt;- physical } } As you can see from code above, we use a for loop here to add physical sections. The for loop is a very useful tool to iterate over a list of elements. With for loop, you can repeat a specific block of code without copying and pasting the code over and over again. When processing datasets in Arctic Data Center, there are many places where for loop can be used, such as publishing a bunch of objects with pids, updating formatID for pkg$data, adding physical section like above code, etc. A loop is composed of two parts: the sequence and the body. The sequence usually generates indices to locate elements and the body contains the code that you want to iterate for each element. Here is an example of adding the same attributeList for all the dataTables in the metadata using for loop. attributes &lt;- read.csv(&#39;attributes.csv&#39;) # attribute table in csv format attributeList &lt;- EML::set_attributes(attributes = attributes) for (i in 1:length(doc$dataset$dataTable)) { # sequence part doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList # body part } "],["assess-attributes.html", "Assess attributes", " Assess attributes New submissions made through the web editor should have attributes created by the submitter. If there are no attributes, or if they are incomplete, please use the email template to correspond with the submitter to ask for attributes. Additionally, the web editor will not allow for the creation of custom units. submitters should select other/none in the editor unit dropdown if they cannot find their desired unit. In the EML document, these will result in dimensionless units. Accordingly, new submissions should be checked for dimensionless units. This can be done with the following code. Note that this only applies if there is more than one otherEntity. dim_units &lt;- sapply(doc$dataset$otherEntity, function(x) { i &lt;- datamgmt::which_in_eml(x$attributeList$attribute, &quot;standardUnit&quot;, as(&quot;dimensionless&quot;, &quot;standardUnit&quot;)) out &lt;- sapply(i, function(j){ list(entityName = x$entityName, attributeName = x$attributeList$attribute[[j]]$attributeName, attributeLabel = x$attributeList$attribute[[j]]$attributeLabel, attributeDefinition = x$attributeList$attribute[[j]]$attributeDefinition) }) return(out) }) dim_units &lt;- data.frame(t(do.call(cbind, dim_units))) dim_units If dimensionless units are found, first check the attributeDefinition, attributeLabel, and attributeName to see if dimensionless seems to be an appropriate unit for the attribute. If there is any question about whether or not dimensionless seems appropriate, ask another member of the data team. If there is still any question about what the unit should be, reach out to the submitter and ask for clarification. "],["list-submissions-to-the-arctic-data-center.html", "List submissions to the Arctic Data Center", " List submissions to the Arctic Data Center If the arctic bot fails to pick up a new submission, due to an invalid token or another error, we can check for submissions using a solr query. The list_submissions() function in the arcticdatautils package is a convenience wrapper around the solr query. If the function doesn’t specifically fit your needs it’s fairly straight-forward to modify the source code. Note: the function can be applied to any node, although it filters out results if the submitter is in the Arctic Data Center Admins group. Refer to the function documentation for more information. View submissions to the Arctic Data Center from 10/1/18 to 10/7/18 cn &lt;- dataone::CNode(&#39;PROD&#39;) adc &lt;- dataone::getMNode(cn,&#39;urn:node:ARCTIC&#39;) View(arcticdatautils::list_submissions(adc, &#39;2018-10-01&#39;, &#39;2018-10-07&#39;)) The following code shows the underlying solr query q = paste0(&#39;dateUploaded:[&quot;2018-10-01T00:00:00Z&quot; TO &quot;2018-10-08T00:00:00Z&quot;] AND formatType:*&#39;) results &lt;- dataone::query(adc, list(q = q, fl = &quot;identifier AND submitter AND dateUploaded AND formatType AND fileName&quot;, rows = 10000), as = &quot;data.frame&quot;) "],["recover-failed-submissions.html", "Recover failed submissions", " Recover failed submissions A submission may result in an error when the EML cannot be validated through EML::eml_validate, preventing it from being written to an indicated path. This usually occurs when the metacatUI detects a submission error and uploads the EML as a data object instead of text. When this happens, arcticdatautils::recover_failed_submission can be used to retrieve the original EML file. The function removes the error message to get a valid EML document. For the example below, pid is the metadata pid containing the failed submission. The path parameter indicates where you want to write the recovered EML version to on your server’s local working directory. path &lt;- &quot;path/to/save/eml.xml&quot; pid &lt;- &quot;your_metadata_pid&quot; arcticdatautils::recover_failed_submission(adc, pid, path) The output of this function is a valid EML file written to your chosen path. After recovering the document, make sure to set the rights and access to the correct submitter. Note that arcticdatautils::recover_failed_submission may not always work. Its functionality depends on the error message since they can vary. "],["data-portals.html", "Data Portals ", " Data Portals "],["customizing-data-portals.html", "Customizing Data Portals", " Customizing Data Portals Create a custom, branded portal for your research topic or lab group that spans datasets in the Arctic Data Center – available for all users. Branded portals provide a convenient, readily customized way to communicate your science, your team, your data, and related data from within the Arctic Data Center. The portals feature makes highlighting and sharing related datasets easy for researchers. General information on data portals can be found on the main page: https://arcticdata.io/data-portals/ My portals My portals allows members of the Arctic Data Center Team to view all the portals currently created. This can be found in Settings under the My Portals section. Creating Data Filters Useful search terms to use: NSF award number Geographic Region Keywords Access permissions ### Set Access Permissions For some portals such as PerCS portal, will need to be added by running set_access(). Please see the specific portal for the exact arguments to pass. Advanced portal customizations These requires the manual update of the portal document for features that are not available on the website yet. Please ask Jeanette for more assistance. For an example of a portal with all the features, see the SASAP Portal. Custom Filters Most os the time copy and pasting desired from the SASAP portal is a good place to start Tips - try to collapse the specific section and copy that to make sure you copy everything is needed Filter Groups Grouping multiple filters under one tab Possible fontawesome icons &lt;filterGroup&gt; &lt;label&gt;MOSAiC Terms&lt;/label&gt; &lt;description&gt;Terms in the MOSAiC ontology&lt;/description&gt; &lt;icon&gt;book&lt;/icon&gt; &lt;choiceFilter&gt;...&lt;/choiceFilter&gt; &lt;/filterGroup&gt; Individual Filters &lt;choiceFilter&gt; &lt;label&gt;Campaign&lt;/label&gt; &lt;!-- ====== equivalent to the fields available in the solr query ====== --&gt; &lt;field&gt;text&lt;/field&gt; &lt;operator&gt;OR&lt;/operator&gt; &lt;filterOptions&gt; &lt;placeholder&gt;Choose a campaign&lt;/placeholder&gt;&lt;icon&gt;sitemap&lt;/icon&gt; &lt;description&gt;The campaign number&lt;/description&gt; &lt;/filterOptions&gt; &lt;!-- ====== change these for different options ====== --&gt; &lt;choice&gt;&lt;label&gt;AF-MOSAiC-1&lt;/label&gt;&lt;value&gt;https://purl.dataone.org/odo/MOSAIC_00000020&lt;/value&gt;&lt;/choice&gt; &lt;choice&gt;&lt;label&gt;AT-MOSAiC-1&lt;/label&gt;&lt;value&gt;https://purl.dataone.org/odo/MOSAIC_00011201&lt;/value&gt;&lt;/choice&gt; &lt;chooseMultiple&gt;true&lt;/chooseMultiple&gt; &lt;/choiceFilter&gt; Custom Colour schemes Look for this section in the xml and replace the hex codes: &lt;option&gt; &lt;optionName&gt;primaryColor&lt;/optionName&gt; &lt;optionValue&gt;#3774b9&lt;/optionValue&gt; &lt;/option&gt; &lt;option&gt; &lt;optionName&gt;secondaryColor&lt;/optionName&gt; &lt;optionValue&gt;#82cdf5&lt;/optionValue&gt; &lt;/option&gt; &lt;option&gt; &lt;optionName&gt;accentColor&lt;/optionName&gt; &lt;optionValue&gt;#EB5638&lt;/optionValue&gt; &lt;/option&gt; People page also a formatted page - see the SASAP portal for details Updating portals For portals on DataOne - use the ucsb node. Portals made on the ADC use the same node as the datasets. library(dataone) library(EML) cn &lt;- CNode(&quot;PROD&quot;) ucsb &lt;- getMNode(cn, &quot;urn:node:mnUCSB1&quot;) Look for portals using query()so you can get the appropriate seriesID df &lt;- query(ucsb, list(q =&quot;formatId:*purl.dataone.org/portals-1.0.0 AND (*:* NOT obsoletedBy:*)&quot;, rows = &quot;300&quot;, fl = &quot;identifier,dateUploaded,formatId,obsoletedBy,origin,formatType,title, seriesId&quot;, sort = &#39;dateUploaded+desc&#39;), as = &quot;data.frame&quot;) Download the file and open to edit and add the filters and other options. id &lt;- df[df$title == &quot;Multidisciplinary drifting Observatory for the Study of Arctic Climate expedition&quot;, 2] #series ID writeBin(getObject(ucsb, id), &quot;file_mosaic.xml&quot;) #download Make whatever edits you need and upload the file. The identifier should be updated if successful but if there is an issue with the xml document go back and fix the issue before re-uploading again. sys &lt;- getSystemMetadata(ucsb, id) arcticdatautils::update_object(ucsb, sys@identifier, path = &quot;file_mosaic.xml&quot;, format_id = sys@formatId, sid = sys@seriesId) "],["distributed-biological-observatory-dbo-submissions.html", "Distributed Biological Observatory (DBO) submissions", " Distributed Biological Observatory (DBO) submissions DBO packages are organized under a project page (viewable here). In order to make sure that DBO datasets appear in the special DBO data search correctly, there are a few additional metadata requirements. give read and write access to the entire data package (using set_access) to the group: \"CN=DBO,DC=dataone,DC=org\" eg: `set_access(mn, pkg, “CN=DBO,DC=dataone,DC=org”, permissions = c(“read”, “write”, “changePermission”)) ensure that the name of the ship the data was collected from appears in the metadata record. This will likely be one of the following ship names. If the ship name is not in that list, let Jeanette or Dominic know Annika Marie Fairweather Healy Kromov Mirai Oscar Dyson Oshoro-Maru Sir Wilfrid Laurier Westward Wind Xue Long the geographic coverage should be one coverage per DBO transect sampled. The geographicDescription and bounding coordinates should be from the controlled vocabulary given in the code snippet below, which will create a data.frame with these values in your R environment geo_locs &lt;- dplyr::tribble(~siteNew, ~westBoundingCoordinate, ~eastBoundingCoordinate, ~northBoundingCoordinate,~ southBoundingCoordinate, &quot;DBO 1 - South of St. Lawrence Island, Bering Sea&quot;, -176.147, -172.187, 63.769, 61.847, &quot;DBO 2 - Chirikov Basin, Northern Bering Sea&quot;, -170.492, -167.86, 65.111, 64.482, &quot;DBO 3 - Southern Chukchi Sea&quot;, -171.419, -166.481, 68.609, 66.752, &quot;DBO 4 - Northeast Chukchi Sea&quot;, -164.553, -160.507, 71.867, 70.682, &quot;DBO 5 - Barrow Canyon, Chukchi Sea&quot;, -158.848, -155.931, 71.808, 71.111, &quot;DBO 6 - Beaufort Sea - 152 W&quot;, -153.865, -150.976, 72.119, 70.89) one of the following key phrases referencing the data type should exist in the dataset. If the dataset doesn’t seem to fit in one of these categories, ask Jeanette or Dominic ADCP benthic macroinfaunal chlorophyll CTD CPUE (referencing fish catch per unit effort) marine mammal nutrient sediment zooplankton After you have updated your dataset, navigate to https://arcticdata.io/catalog/projects/DBO/data and make sure that the different filters work as expected. Your dataset should be discoverable in some way through all of the filters. "],["mosaic.html", "MOSAiC", " MOSAiC Look out for datasets that are part of the MOSAiC expedition from 2019 -2020. There are a couple of special steps for datasets from this project. When working on MOSAiC datasets: Find out the EventId from the researcher to annotate (Campaign can be derived from the eventLabel). Ask for it in your email: We would like to ask for the event label associated with this dataset (see https://www.pangaea.de/expeditions/events/PS122%2F4). Often times researchers do not have a full list of event labels, but instrument names are provided in the dataset metadata. In these cases, try to find the correct event labels by searching through the event lists and finding matching instrument names and dates. Each MOSAiC campaign has its own event list. Here is an example of an event list for the first cruise by the Polarstern research vessel. Find the appropriate dataset and attribute level annotations There are functions in arcticdatautils to help with annotating: mosaic_annotate_dataset and mosaic_annotate_attribute The custom ontology for the datasets are currently in the sem-prov-ontologies repository but you won’t need to use it directly if the function works. Use EML::shiny_attributes only when no annotations have been added. The function does not work well with multiple annotations Please continue to annotate the measurement types using containst measurements of type for all other attributes The portal is on DataONE. Make sure the dataset(s) show up in the portal after you add the annotations. Here are two fully annotated datasets for reference: Simple example Example with MOSAiC and ECSO annotations Example with non-polarstern vessels The following shows how to add the annotations using arcticdatautils and manually in the case the functions fail to work: Dataset Level Annotations There are 5 main campaigns in the MOSAiC expedition. The main campaigns follow the pattern PS122/#. The full campaign list is easiest to view on the PANGAEA website. The first two letters of each campaign name correspond with the ship or station name (ex: PS = Polarstern). arcticdatautils doc$dataset$id &lt;- &quot;your id here&quot; doc$dataset$annotation &lt;- NULL #make sure it is empty doc$dataset$annotation &lt;- mosaic_annotate_dataset(c(&quot;PS122/1&quot;, &quot;PS122/2&quot;)) Manual Majority of the annotations will be the same across all of the datasets. Pay attention to what needs to be changed for the hasBasis and isPartOfCampaign valueURIs. More than one Campaign and Basis might be needed in rare cases. doc$dataset$id &lt;- &quot;your id here&quot; doc$dataset$annotation &lt;- NULL #make sure it is empty here doc$dataset$annotation &lt;- list( #Basis list( propertyURI = list(label = &quot;hasBasis&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000034&quot;), valueURI = list(label = &quot;Polarstern&quot;, # this depends on your event ID, most of them should be Polarstern valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000030&quot;) ), #Project list( propertyURI = list(label = &quot;hasProjectLabel&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000025&quot;), valueURI = list(label = &quot;MOSAiC20192020&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000023&quot;) ), #Campaign list( propertyURI = list(label = &quot;isPartOfCampaign&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000032&quot;), valueURI = list(label = &quot;PS122/2&quot;, #*** this changes depending on the campaign valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000018&quot;) ) #*** this changes depending on the campaign ) Attribute Level Annotations Apply these to the attribute level of the dataset. Event IDs should have already be identified earlier on in the dataset review. Check the ticket overview for details. arcticdatautils doc$dataset$dataTable[[1]]$attributeList$attribute[[1]]$id &lt;- &quot;some id here&quot; doc$dataset$dataTable[[1]]$attributeList$attribute[[1]]$annotation &lt;- mosaic_annotate_attribute(&quot;PS122/2_14-270&quot;) Manual event_annotation &lt;- list(#Event Label list( propertyURI = list(label = &quot;wasGeneratedBy&quot;, propertyURI = &quot;http://www.w3.org/ns/prov#wasGeneratedBy&quot;), valueURI = list(label = &quot;PS122/2_14-270&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00004550&quot;) ), #Method/Device - Use the long name! list( propertyURI = list(label = &quot;deployedSystem&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00002201&quot;), valueURI = list(label = &quot;Ultra-Wideband Software-defined Microwave Radiometer (0.5-2GHZ)&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00001163&quot;) )) doc$dataset$dataTable$attributeList$attribute[[1]]$id &lt;- &quot;eventId&quot; doc$dataset$dataTable$attributeList$attribute[[1]]$annotation &lt;- event_annotate Adding annotations If there are already annotations applied (ie. using ECSO), the MOSAiC annotations can be added as so: get the index of all of the attributes we want to add this annotation n &lt;- which(stringr::str_detect(eml_get_simple(doc$dataset$dataTable$attributeList, &quot;attributeName&quot;),&quot;Brightness&quot;)) loop through and add the annotation(s) for(i in n) { doc$dataset$dataTable$attributeList$attribute[[i]]$annotation &lt;- append( list( doc$dataset$dataTable$attributeList$attribute[[i]]$annotation ), event_annotation ) } Portal updates The filters are based on the MOSAiC ontology. The Campaign and Basis filters should remain relatively the same. However there are over 500 Method and Devices. So the decision was made to only show the ones we have annotated. We can query for all of them and output them formatted as a choice: mosaic_portal_filter(&quot;Method/Device&quot;) Then update the portal document using the steps outlines in the advanced data portal customizations "],["using-arcticdatautils.html", "Using arcticdatautils", " Using arcticdatautils Please use datapack and dataone for regular data processing. This information is kept for troubleshooting and legacy purposes. "],["create-a-resource-map-using-arcticdatautils.html", "Create a resource map using arcticdatautils", " Create a resource map using arcticdatautils If you are creating a new data package, you must create a resource map. Resource maps provide information about the resources in the data package (i.e. which data files are associated with a particular metadata file (xml)). Before creating a new resource map, it is best to ensure that all the objects of interest have already been published to the node. Therefore, if you’ve only run publish_object() on the xml (in section 2.4 above), you will also need to publish all the data objects you wish to associate with the xml. The resource map you are about to create controls these linkages. Once all the objects that ought to belong to the package have been published, create a resource map using their respective PIDs (or the variables you saved the PIDs to) via the arcticdatautils::create_resource_map() function, like so: resource_map_pid &lt;- create_resource_map(adc_test, metadata_pid = metadata_pid, data_pids = data_pids) "],["get-package-and-eml.html", "Get package and EML", " Get package and EML Before we look more in depth at EML, we first need to load your data package into R. After setting your node, use the following commands to load the package: rm_pid &lt;- &quot;your_resource_map_pid&quot; pkg &lt;- get_package(adc_test, rm_pid, file_names = TRUE) Keeping track of PIDs In all subsequent sections, if they ask for the pid of the metadata, reasource map or data you can refer to the object from get_package(). For example, if you assigned the value from get_package() as above as pkg then you can refer to the following by using the corresponding commands: metadata_pid &lt;- pkg$metadata data_pid &lt;- pkg$data resource_pid &lt;- pkg$resource_map After loading the package, you can also load the EML file into R using the following command: doc &lt;- read_eml(getObject(adc_test, pkg$metadata)) Note that we named the object doc. This is a good generic name to use for EML documents. The generic name eml should not be used - as the EML package ships with an eml function, which can cause namespace issues in your environment if you have an object also called eml. Tip to always have the most recent resource map. When editing data packages, you always want to be working with the most recent update. To ensure you have the most recent resource map, you can use the following commands: rm_pid_original &lt;- &quot;your_original_resource_map_pid&quot; all_rm_versions &lt;- get_all_versions(adc_test, rm_pid_original) rm_pid &lt;- all_rm_versions[length(all_rm_versions)] print(rm_pid) "],["publish-an-object.html", "Publish an object", " Publish an object Objects (data files, xml metadata files) can be published to a DataONE node using the function publish_object() from the arcticdatautils R Package. To publish an object, you must first get the formatId of the object you want to publish. A few common formatIds are listed below. # .csv file formatId &lt;- &quot;text/csv&quot; # .txt file formatId &lt;- &quot;text/plain&quot; # metadata file formatId &lt;- &quot;https://eml.ecoinformatics.org/eml-2.2.0&quot; # OR formatId &lt;- format_eml(&quot;2.2.0&quot;) Most objects have registered formatIds that can be found on the DataONE Object Format List here. Always use the “Id:” (2nd line) from the DataONE Object Format List and ensure it is copied EXACTLY. Metadata files (as shown above) use a special function to set the formatId. If the formatId is not listed at the DataONE website, you can set formatId &lt;- \"application/octet-stream\". If you want to change the formatId please use updateSystemMetadata instead. Once you know the appropriate formatId you can publish an object using these commands: path &lt;- &quot;path/to/your/file&quot; formatId &lt;- &quot;your/formatId&quot; pid &lt;- publish_object(adc_test, path = path, format_id = formatId) # note that the output of this function is the PID of the newly published object After publishing the object, the PID will need to be added to a resource map by updating or creating a resource map. Additionally, the rights and access for the object must be set. However, you only need to give other people rights and access to objects to objects that are not yours, for the training you don’t need to do this. "],["reorder-entities.html", "Reorder entities", " Reorder entities This is easier to accomplish using arcticdatautils An example reorganizing alphabetically Metadata doc$dataset$otherEntity &lt;- doc$dataset$otherEntity[order(entity_names)] Data files and PIDs retrieved with get_package pkg &lt;- get_package(adc, rm, file_names = T) doc &lt;- EML::read_eml(getObject(adc, pkg$metadata)) reordered &lt;- arcticdatautils::reorder_pids(pkg$data, doc) "],["nesting-data.html", "Nesting Data ", " Nesting Data "],["nesting-a-data-package.html", "Nesting a data package", " Nesting a data package Data packages on member nodes can exist as independent packages or in groups (nested data packages). Much like we can group multiple data files together with a common metadata file, we can group related data packages together with a common “parent” data package. The structure of nested data packages resembles a pyramid. There is one top level, or “parent”, with one or more data packages, or “child packages”, nested beneath it. There is no limit to how many nested levels can be created, but packages do not generally exceed 3 levels. This “grandparent” has 5 child packages (nested datasets), all of which have child packages of their own. Here are some common uses for nesting: collected data vary by year an NSF award funds several related projects data collection is still ongoing data files exceed the 1000 file limit per data package "],["add-children-to-an-existing-parent.html", "Add children to an existing parent", " Add children to an existing parent A new package is published with a DOI and needs to be nested underneath a pre-existing parent. Nest the new child using the child_pids argument in publish_update(). resource_map_child_new &lt;- &quot;some_child_resource_map_pid&quot; pkg_parent &lt;- get_package(mn, &#39;resource_map_parent&#39;) publish_update(mn, resource_map_pid = pkg_parent$resource_map, metadata_pid = pkg_parent$metadata, data_pids = pkg_parent$data_pids, # parents usually don&#39;t contain data, but better to be safe child_pids = c(pkg_parent$child_packages, resource_map_child_new)) # include the resource map PIDs of ALL the childs* in the `child_pids` argument, otherwise the nesting relationships between any omitted childs and the parent will be deleted Check through all arguments carefully before you publish to production! Do you need to update the metadata? Does the parent include data objects? Does the parent have a parent? Parents can be tricky to fix and work with (especially if they have serial identifiers (SIDs)), so if you’re not sure how something works, try it on a test node. "],["create-a-new-parent-package.html", "Create a new parent package", " Create a new parent package In some cases, a parent package already exists. Search the ADC for the NSF award number to see if there are already exisiting packages. Parents usually have a UUID rather than a DOI and often start with a title like “Collaborative research:”, but not always. More typically, you will need to create a new parent by editing the existing metadata. The parent package should contain a generalized summary for the metadata of each of its childs. To create a new parent, you will need to: Create parent metadata. It’s often easiest to start with a child’s metadata and generalize them. Abstract/title: Remove dates and other details that are specific to the child package. Sometimes the NSF award abstract/ title will work. Data tables/other entities: Generally, top-level parents do not include data objects, so these sections can be removed. Geographic coverage: Expand to include geographic coverage of all childs, if needed. Temporal coverage: Expand to include temporal ranges of all childs, if needed. If the study is ongoing, include the most recent end date; the parent can be updated when additional childs are added. Methods: Often not needed, but may be included if all childs use the same methods. Publish the parent metadata to the member node (ADC) using publish_object(). Create a resource map to link the parent and childs together using create_resource_map() and the child_pids argument. Make sure you use the childs’ resource map PIDs when you create the resource map! If you forgot to do so, consult Jeanette for help fixing it. "],["example.html", "Example", " Example We can start by creating two data packages on the test node to nest beneath a parent. These data packages contain measurements taken from Lake E1 in Alaska in 2013 and 2014. First, load the Arctic Data Center Test Node and libraries. library(dataone) library(arcticdatautils) library(EML) cn_staging &lt;- CNode(&#39;STAGING&#39;) adc_test &lt;- getMNode(cn_staging,&#39;urn:node:mnTestARCTIC&#39;) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) We will re-create the following parent package: https://arcticdata.io/catalog/#view/urn:uuid:799b7a86-cb1c-497c-a05a-d73492915cad on the test node with two of its children. First we will copy two of the children to the test node, make sure your token for the test node is not expired. from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) to &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) child_pkg_1 &lt;- datamgmt::clone_package(&#39;resource_map_doi:10.18739/A2KS1R&#39;, from = from, to = to, add_access_to = arcticdatautils:::get_token_subject(), change_auth_node = TRUE, new_pid = TRUE) child_pkg_2 &lt;- datamgmt::clone_package(&#39;resource_map_doi:10.18739/A2QK29&#39;, from = from, to = to, add_access_to = arcticdatautils:::get_token_subject(), change_auth_node = TRUE, new_pid = TRUE) These two packages correspond to data from the same study, varying only by year; however, they currently exist on the test node as independent entities. We will associate them with each other by nesting them underneath a parent. Now, let’s create a parent metadata file. Read in one of the childs’ metadata files (EML). We can download object from a node in binary format using dataone::getObject(). Once it’s downloaded we just need to convert to it to the proper format: in this case to EML format using EML::read_eml(). doc_parent &lt;- read_eml(getObject(adc_test, child_pkg_1$metadata)) ## View the title doc_parent$dataset$title The title of this child contains “2012-2013”. This is too specific for the parent, as the temporal range of both childs is 2012-2014. The parent should encompass this larger time range. doc_parent$dataset$title &lt;- &#39;Time series of water temperature, specific conductance, and oxygen from Lake E1, North Slope, Alaska, 2012-2014&#39; Like the title, the temporal coverage elements in this EML need to be adjusted. new_end_date &lt;- &quot;2014-09-20&quot; doc_parent$dataset$coverage$temporalCoverage$rangeOfDates$endDate$calendarDate &lt;- new_end_date Remove dataTables and otherEntitys from the metadata. If you recall from previous chapters, dataTables contain metadata associated with data files (generally CSVs) and otherEntitys contain metadata about any other files in the data package (for instance a README or coding script). Because the parent does not contain any data objects, we want to remove dataTables and otherEntitys from the metdata file. In this instance, the E1 2013 metadata only contain dataTables. We can remove these by setting the dataTable element in the EML to a new blank object. doc_parent$dataset$dataTable &lt;- NULL In this case, the abstract, contacts, creators, geographicDescription, and methods are already generalized and do not require changes. Before writing your parent EML make sure that it validates. This is just a check to make sure everything is in the correct format. eml_validate(doc_parent) After your EML validates we need to save, or “write”, it as a new file. Write your parent EML to a directory in your home folder. You can view this process like using “Save as” in Microsoft Word. We opened a file (“E1_2013.xml”), made some changes, and “saved it as” a new file called “doc_parent.xml”. # We can save the eml in a temporary file eml_path &lt;- file.path(tempdir(), &#39;science_metadata.xml&#39;) write_eml(doc_parent, path) Next, we will publish the parent metadata to the test node. metadata_parent &lt;- publish_object(adc_test, path = eml_path, format_id = format_eml()) Finally, we create a resource map for the parent package. We nest the two child data packages using the child_pids argument in create_resource_map(). Note that these child_pids are PIDs for the resource maps of the child packages, NOT the metadata PIDs. resource_map_parent &lt;- create_resource_map(adc_test, metadata_pid = metadata_parent, child_pids = c(child_pkg_1$resource_map, child_pkg_2$resource_map)) The child packages are now nested underneath the parent. "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]
+[["index.html", "NCEAS Data Team Reference Guide About", " NCEAS Data Team Reference Guide 2024-08-13 About This reference guide documents various workflows and related issues for processing data packages. "],["miscellaneous.html", "Miscellaneous ", " Miscellaneous "],["code-snippets.html", "Code Snippets", " Code Snippets Code snippets help with templating portions of code that you will be using regularly. To add your own, go to the toolbar ribbon at the top of your Rstudio screen and select: Tools &gt; Global Options... &gt; Code &gt; Edit Snippets &gt; Add these chunks to the end of the file More info can be found in this blog post by Mara Averick on how to add them: https://maraaverick.rbind.io/2017/09/custom-snippets-in-rstudio-faster-tweet-chunks-for-all/ Usual arcticdatautils ticket workflow snippet ticket library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;add your rm&quot; pkg &lt;- get_package(adc, rm) doc &lt;- EML::read_eml(getObject(adc, pkg\\$metadata)) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) eml_validate(doc) eml_path &lt;- &quot;eml.xml&quot; write_eml(doc, eml_path) #update &lt;- publish_update(adc, # metadata_pid = pkg\\$metadata, # resource_map_pid = pkg\\$resource_map, # metadata_path = eml_path, # data_pids = pkg\\$data, # public = F) #datamgmt::categorize_dataset(update\\$metadata, c(&quot;theme1&quot;), &quot;Your Name&quot;) The datapack ticket workflow snippet datapack library(dataone) library(datapack) library(digest) library(uuid) library(dataone) library(arcticdatautils) library(EML) d1c &lt;- D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) packageId &lt;- &quot;id here&quot; dp &lt;- getDataPackage(d1c, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) #get metadata id metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #edit the metadata doc &lt;- read_eml(getObject(d1c@mn, metadataId)) #add the publisher info doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) doc\\$dataset\\$project &lt;- eml_nsf_to_project(&quot;nsf id here&quot;) #check and save the metadata eml_validate(doc) eml_path &lt;- arcticdatautils::title_to_file_name(doc\\$dataset\\$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) #upload the dataset myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c, dp, public=F, accessRules=myAccessRules, quiet=FALSE) #datamgmt::categorize_dataset(&quot;doi&quot;, c(&quot;theme1&quot;), &quot;Jasmine&quot;) Quick way to give access to submitters to their datasets: snippet access library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;rm here&quot; pkg &lt;- get_package(adc, rm) set_access(adc, unlist(pkg), &quot;orcid here&quot;) Quick access to the usual code for common Solr queries: snippet solr library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) "],["file-paths.html", "File paths", " File paths This section contains some basic tips about file paths. Remote files vs. local files If you’re working on the datateam server (RStudio in an internet browser), you’re usually working in the home directory under your username. File paths generally follow this pattern: /home/dmullen/folder/file. Note: These files are not stored on your local machine, they are on a remote server. You can access them through remote SFTP software like Cyberduck. If you’re working with your local RStudio app then your file paths generally follow this pattern: /Users/datateam/folder/file. These files are stored on your local machine and you can access them with Finder. File path shortcuts Using . in a file path refers to your current working directory. You can print your working directory with the call: getwd(). If we assume that my current working directory is home/dmullen, then the following two calls are equivalent: file.exists(&#39;/home/dmullen/myFile.R&#39;) file.exists(&#39;./myFile.R&#39;) Using ~ in a file path does not always refer to your current working directory. This is a common misconception. It is actually set to the global R USER path. You can print this with the call: Sys.getenv('USER'). Let’s assume that I’m working on the arcticdatautils package which I open using a .RProj file. This will update my current working directory to: /home/dmullen/arcticdatautils. Now the following calls are equivalent: file.exists(&#39;/home/dmullen/arcticdatautils/myFile.R&#39;) file.exists(&#39;~/arcticdatautils/myFile.R&#39;) file.exists(&#39;./myFile.R&#39;) "],["metacat-version.html", "Metacat version", " Metacat version Check which version of metacat is currently deployed to - arcticdata.io here, - and to test.arcticdata.io here. "],["resources-for-r.html", "Resources for R", " Resources for R Learning The following online books are useful for expanding your R knowledge and skills: the most recent ADC training materials The cleaning and data manipulation section is useful for working with attribute tables Efficient R Programming In particular Chapter 3 Efficient Programming R for Data Science Section on Strings R Packages contributing to arcticdatatutils, datamgmt and EML Advanced R Object-oriented programming in R for S4 to understand how datapack and dataone packages are written bookdown: Authoring Books and Technical Documents with R Markdown formatting, troubleshooting and updating the training document Others Hands-On Programming with R R Programming for Data Science Exploratory Data Analysis with R Mastering Software Development in R Geocomputation with R R Markdown: The Definitive Guide The Tidyverse Style Guide The RStudio cheatsheets are also useful references for functions in tidyverse and other packages. Packages The data team uses and develops a number of R packages. Here is a listing and description of the main packages: dataone reading and writing data at DataONE member nodes http://doi.org/10.5063/F1M61H5X datapack creating and managing data packages https://github.com/ropensci/datapack EML creating and editing EML metadata documents https://ropensci.github.io/EML arcticdatautils utility functions for processing data for the Arctic Data Center https://nceas.github.io/arcticdatautils/ datamgmt data management utilities for curating, documenting, and publishing data (sandbox package) https://nceas.github.io/datamgmt/ metadig authoring MetaDIG quality checks https://github.com/NCEAS/metadig-r metajam downloading and reading data and metadata from DataONE member nodes https://nceas.github.io/metajam/ "],["edit-data-packages.html", "Edit data packages ", " Edit data packages "],["datapack-background.html", "datapack Background", " datapack Background adapted from the dataone and datapack vignettes datapack is written differently than most R packages you may have encountered in the past. This is because it uses the S4 system instead. library(dataone) library(datapack) library(uuid) Data packages Data packages are a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. Navigating data packages Nodes Using this example on arcticdata.io d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) To use the member node information, use the mn slot d1c_test@mn To access the various slots using objects created by datapack and dataone (e.g. getSystemMetadata) requires the @ which is different from what you might have seen in the past. This is because these use the S4 system. Get an existing package from the Arctic Data Center. Make sure you know as you go through this training whether you are reading or writing to test or production. We don’t want to upload any of your test datasets to production! d1c &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) dp &lt;- dataone::getDataPackage(d1c, &quot;resource_map_urn:uuid:1f9eee7e-2d03-43c4-ad7f-f300e013ab28&quot;) Data Objects You can see what slots are in an S4 object after typing the subsetting operator @, or pressing TAB with the cursor after an existing @. Try viewing the slots of the data package. dp@ Check out the objects slot dp@objects The objects slot contains a list of object PIDs that are accessed using the $ subsetting operator. Both are found within the structure of data packages in R. Get the number for data and metadata files associated with this data package: getSize(dp) Get the file names and corresponding pids getValue(dp, name=&quot;sysmeta@fileName&quot;) Get identifiers You can search by any of the sysmeta slots such as fileName and formatId and get the corresponding identifier(s): metadataId &lt;- selectMember(dp, name=&quot;sysmeta@ADD THE NAME OF THE SLOT&quot;, value=&quot;PATTERN TO SEARCH BY&quot;) Example: selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;image/tiff&quot;) selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&quot;filename.csv&quot;) Provenance View the provenance as a dataTable. We will get into detail in the Building provenance chapter. dp@relations$relations "],["create-a-new-data-package.html", "Create a new data package", " Create a new data package adapted from the dataone and datapack vingettes library(dataone) library(datapack) library(uuid) Create a new data package - data package is a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. dp &lt;- new(&quot;DataPackage&quot;) Upload new data files Create and add a metadata file In this example we will use this previously written EML metadata. Here we are getting the file path from the dataone package and saving that as the object emlFile. This is a bit of an unusual way to reference a local file path, but all this does is looks within the R package dataone and grabs the path to a metadata document stored within that package. If you print the value of emlFile you’ll see it is just a file path, but it points to a special place on the server where that package is installed. Usually you will just reference EML paths that are stored within your user file system. emlFile &lt;- system.file(&quot;extdata/strix-pacific-northwest.xml&quot;, package=&quot;dataone&quot;) Create a new DataObject for the metadata and add it to the package. metadataObj &lt;- new(&quot;DataObject&quot;, format=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;, filename=emlFile) dp &lt;- addMember(dp, metadataObj) Check the dp object to see if the metadata was added correctly. dp Add some additional data files sourceData &lt;- system.file(&quot;extdata/OwlNightj.csv&quot;, package=&quot;dataone&quot;) sourceObj &lt;- new(&quot;DataObject&quot;, format=&quot;text/csv&quot;, filename=sourceData) dp &lt;- addMember(dp, sourceObj, metadataObj) # The third argument of addMember() associates the new DataObject to the metadata that was just added. If you want to change the formatId please use updateSystemMetadata (more on this later in the book) Upload the package d1c &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Make sure to give access privileges to the ADC admins: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) Get necessary token from test.arcticdata.io to upload the dataset prior uploading the datapackage: packageId &lt;- uploadDataPackage(d1c, dp, public=TRUE, accessRules=myAccessRules, quiet=FALSE) "],["update-packages-with-datapack.html", "Update packages with datapack", " Update packages with datapack Once you have updated the data objects and saved the metadata to a file, we can update the metadata and add the new pid to the resource map using datapack::updateDataObject(). This part of the training assumes that you have an EML file created from earlier parts of the training You must use these R packages for data packages that have provenance or folder hierarchy (arcticdatatutils does not support those features) Get the package Setting the node is done slightly differently: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Get a pre-existing package: packageId &lt;- &quot;the resource map&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) Update the metadata Get the metadata identifier metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) Take note of the EML version. If it is EML 2.1.1 the value needs to be changed to -“eml://ecoinformatics.org/eml-2.1.1” Read the EML doc doc &lt;- read_eml(getObject(d1c@mn, metadataId)) Edit the EML as usual - see the documents in Edit EML for details Once you are happy with your changes, you can update your data package to include the new metadata file using replaceMember eml_path &lt;- &quot;path/to/your/saved/eml.xml&quot; write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) Modify the data files remove zip files zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) add an existing data object dataObj &lt;- getDataObject(d1c_test, id=&quot;urn:uuid: here&quot;, lazyLoad=T, limit=&quot;1TB&quot;) dp &lt;- addMember(dp, dataObj, mo=metadataId) If you want to change the formatId please use updateSystemMetadata Publish update Using a DOI If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), (a) you need to change the public argument to TRUE and generate a DOI identifier. This should only be done after the package is finalized and has been thoroughly reviewed! Updating a package with a new DOI: A Digital Object Identifier (DOI) may be assigned to the metadata DataObject, using the generateIdentifier: doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) Updating a package with a pre-issued DOI dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;your pre-issued doi previously generated&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. "],["add-a-pre-generated-identifier-to-the-eml.html", "Add a pre-generated identifier to the EML", " Add a pre-generated identifier to the EML When you pre-generate a UUID or DOI, the change is not automatically reflected in the packageId section of the EML. Use the code below to ensure that the EML lines up with the desired identifier: ## Generate DOI and add to EML # Note that you cannot generate a DOI on test nodes doiPid &lt;- generateIdentifier(mn, &quot;DOI&quot;) doc$packageId &lt;- doiPid Be sure to include the identifier= argument in your publish update command so the pre-generated identifier is applied. "],["edit-sysmeta.html", "Edit sysmeta", " Edit sysmeta To edit the sysmeta of an object (data file, EML, or resource map, etc.) with a PID, first load the sysmeta into R using the following command: sysmeta &lt;- getSystemMetadata(d1c_test@mn, pid) Then edit the sysmeta slots by using @ functionality. For example, to change the fileName use the following command: sysmeta@fileName &lt;- &#39;NewFileName.csv&#39; Note that some slots cannot be changed by simple text replace (particularly the accessPolicy). There are various helper functions for changing the accessPolicy and rightsHolder such as datapack::addAccessRule() (which takes the sysmeta as an input) or arcticdatautils::set_rights_and_access(), which only requires a PID. In general, you most frequently need to use dataone::getSystemMetadata() to change either the formatId or fileName slots (see the DataONE list of format ids) for acceptable formats. # Example of setting the formatId slot sysmeta@formatId &lt;- &quot;eml://ecoinformatics.org/eml-2.1.1&quot; After you have changed the necessary slot, you can update the system metadata using the following command: updateSystemMetadata(mn, pid, sysmeta) Identifiers and sysmeta Importantly, changing the system metadata does NOT necessitate a change in the PID of an object. This is because changes to the system metadata do not change the object itself, they are only changing the description of the object (although ideally the system metadata are accurate when an object is first published). Additional resources For a more in-depth (and technical) guide to sysmeta, check out the DataONE documentation: System Metadata Data Types in CICore "],["obsolescence-chain.html", "Obsolescence chain", " Obsolescence chain You can obsolete a dataset using the function datamgmt::obsolete_package(). Use the documentation for instructions on using the function. The following workflow explains how the functions operate. This chunk is to obsolete one data set. If there are more to add to the chain, more steps can be added. Be very careful. Make sure to fill in obsoletes and obsoletedBy slots for each one. The obsoletes and obsoletedBy fields must be NA, once they are populated they can’t be modified. # get oldest version of the file you want to be visible. Use get_all_versions and look at the latest. # urn:uuid:... # PID for data set to be obsoleted (hidden): doi:10… # adding data set to obsolete (hide) in the slot before the first version of the visible data set sysmeta1 &lt;- getSystemMetadata(mn, &quot;urn:uuid:example_pid&quot;) sysmeta1@obsoletes &lt;- &quot;doi:10.example_doi&quot; updateSystemMetadata(mn, &quot;urn:uuid:example_pid&quot;, sysmeta1) # adding first version to obsolescence chain after obsoleted (hidden) version sysmeta0 &lt;- getSystemMetadata(mn, &quot;doi:10.example_doi&quot;) sysmeta0@obsoletedBy &lt;- &quot;urn:uuid:example_pid&quot; updateSystemMetadata(mn, &quot;doi:10.example_doi&quot;, sysmeta0) The following code is equivalent to the code chunk above. This method is recommended, however it is necessary to read the function documentation first. datamgmt::obsolete_package(mn, metadata_obsolete = &quot;doi:10.example_doi&quot;, metadata_new = &quot;urn:uuid:example_pid&quot;) "],["set-dataone-nodes.html", "Set DataONE nodes", " Set DataONE nodes DataONE is a network of data repositories that is structured with coordinating nodes (CN) and member nodes (MN). The network tree looks something like this: At the top level is DataONE itself. Within DataONE there are several coordinating nodes, including nodes for both production material and testing material. Within these coordinating nodes are many member nodes, including ones for both the Arctic Data Center and the KNB. To set the environment in which you want to publish data, you need to set both the coordinating node and the member node. For example, if you are publishing to the Arctic Data Center test site, you would want to set the coordinating node to STAGING and the member node to mnTestArctic. A note on nodes - be very careful about what you publish on production nodes (PROD, or arcticdata.io). These nodes should NEVER be used to publish test or training data sets. The primary nodes we work on, and how to set them in R, are below: Staging (Test) nodes # ADC (test.arcticdata.io) cn_staging &lt;- CNode(&#39;STAGING&#39;) adc_test &lt;- getMNode(cn_staging,&#39;urn:node:mnTestARCTIC&#39;) # KNB (dev.nceas.ucsb.edu) cn_staging2 &lt;- CNode(&quot;STAGING2&quot;) knb_test &lt;- getMNode(cn_staging2, &quot;urn:node:mnTestKNB&quot;) Production nodes # ADC (arcticdata.io) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn,&#39;urn:node:ARCTIC&#39;) # KNB (knb.ecoinformatics.org) knb &lt;- getMNode(cn, &quot;urn:node:KNB&quot;) # GOA goa &lt;- getMNode(cn, &quot;urn:node:GOA&quot;) # You can also use the datamgmt::guess_membernode function to set a member node # Note: this pid looks like a URL - it&#39;s really a unique identifier dryad &lt;- datamgmt::guess_member_node(&#39;https://doi.org/10.5061/dryad.k6gf1tf/15?ver=2018-09-18T03:54:10.492+00:00&#39;) More DataONE STAGING nodes can be found here More DataONE PROD nodes can be found here "],["set-rights-and-access.html", "Set rights and access", " Set rights and access One final step when creating/updating packages is to make sure that the rights and access on all the objects that were uploaded are set correctly within the sysmeta. The function arcticdatautils::set_rights_and_access() will set both, and arcticdatautils::set_access() will just set access. There are two functions for this because a rightsHolder should always have access, but not all people who need access are rightsHolders. The rightsHolder of the data package is typically the submitter (if the data set is submitted through the web form (“editor”)), but if a data team member is publishing objects for a PI, the rightsHolder should be the main point of contact for the data set (i.e. the person who requested that we upload the data for them). To set the rights and access for all of the objects in a package, first get the ORCiD of the person to whom you are giving rights and access. You can set this manually, or grab it from one of the creators in an EML file. You can look up ORCID iDs here # Manually set ORCiD subject &lt;- &#39;http://orcid.org/PUT-YOUR-ORCD-HERE&#39; # Set ORCiD from EML creator # if only 1 creator exists subject &lt;- doc$dataset$creator$userId$userId # if more than 1 creator exists and you want the first one subject &lt;- doc$dataset$creator[[1]]$userId$userId # As a convention we use `http:` instead of `https:` in our system metadata subject &lt;- sub(&quot;^https://&quot;, &quot;http://&quot;, subject) Note, when setting metadata, the ORCiD must start with http://. ORCiDs in EML should start with https://. The sub() command above will change this formatting for you. Next, set the rights and access using the following command: set_rights_and_access(mn, pids = c(pkg$metadata, pkg$data, pkg$resource_map), subject = subject, permissions = c(&#39;read&#39;,&#39;write&#39;,&#39;changePermission&#39;)) If you ever need to remove/add public access to your package or object, you can use remove_public_read() or set_public_read(), respectively. Making files publicly readable is especially useful when downloading large amounts of files to the server in order to to use metadata helper functions that require a file path (ex: eml_get_raster_metadata() and get_ncdf4_attributes()). remove_public_read(mn, c(pkg$metadata, pkg$data, pkg$resource_map)) My profile The datasets that render under a user’s profile page like here are added if one of the following three System Metadata fields exists. The subject is the rightsHolder or the subject has one of either write or changePermission in the accessPolicy. If you ever need to remove a subject from the accessPolicy or update the rightsHolder you can use arcticdatautils::remove_access and arcticdatautils::set_rightsHolder, respectively. "],["show-indexing-status.html", "Show indexing status", " Show indexing status Sometimes it takes awhile for the website to render with the updates you’ve made in R. To check whether a PID has been indexed yet, use: show_indexing_status(mn, pid) The status bar will either show 0% (not indexed) or 100% (should be online already). "],["preserve-folder-structures.html", "Preserve folder structures", " Preserve folder structures Sometimes, researchers will upload zip files that a contain nested file and folder structure that we would like to maintain. This reference section will walk you through how to re-upload the contents of the zip file to the Arctic Data Center such that the files and folders are preserved. Note that changing the locations of the files within the package can be tricky, so take these steps with care and try to make sure it is done correctly. With that, here are the steps assuming that the PI has uploaded one zip file to their dataset. You may need to modify the steps for other scenarios, but if you are not sure, feel free to ask the data coordinator. In particular, tar files (.tgz) or rar files (.rar) are also compressed archives that might be better to unpack using command line tools. Download the zip file to datateam First, we will download the file, using R. Navigate to the dataset landing page on the Arctic Data Center Right click the “download” button next to the zip file Select “copy link address” Run the following two lines of code to set the URL variable, and extract the pid Here is an example on the test site. Note that on production, you will need to change the URL that you are substituting in the second line of code. library(magrittr) url &lt;- &quot;https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A48c0e669-fd8a-4875-acfd-e8933bb350ed&quot; pid &lt;- gsub(&quot;https://test.arcticdata.io/metacat/d1/mn/v2/object/&quot;, &quot;&quot;, url) %&gt;% gsub(&quot;%3A&quot;, &quot;:&quot;, .) Download the file Note that this will download the file to the location you specify as the second argument in writeBin. If you are organizing your scripts and data by submitter, it would look like the example below. writeBin(getObject(d1c@mn, pid), &quot;~/Submitter/example.zip&quot;) Unzip the file into your submitter directory. unzip(&quot;~/Submitter/example.zip&quot;, exdir = &quot;~/Submitter&quot;) Delete the zip file (example.zip) Now if you look at the directory, you shuold see the unzipped contents of the file in a subdirectory of ~/Submitter. The name of the directory will be the name of the folder the PI created the archive from. Right now, you should stop and examine each file in the directory closely (or each type of file). You may need to make some minor adjustments or ask for clarification from the PI. For example, we still may need to ask for csv versions of excel files, you may need to re-zip certain directories (for example: a zip which contains 5 different sets of shapefiles should be turned into 5 different zips). Evaluate the contents of the directory alongside the data coordinator. Re-upload the contents to the Arctic Data Center Once you have confirmed everything is all good, we can upload the files to the ADC while preserving the directory structure. First, get the data package loaded into your R session as usual. I recommend not attempting any EML edits while you do these steps, this update adding the files is best done on it’s own, and EML edits can be done on the next version. dp &lt;- getDataPackage(d1c, &quot;identifier&quot;) Next, we will set up two types of paths describing each of the objects. The first will be an absolute path, so we can be sure the R function finds the files. The second will be a relative path, which will be what shows up on the landing page (or in the download all result) of the data package. If you don’t know the difference between an absolute and relative path, read on. It is SUPER IMPORTANT! A path is a location of a file or folder on a computer. There are two types of paths in computing: absolute paths and relative paths. An absolute path always starts with the root of your file system and locates files from there. The absolute path to my example submitter zip is: /home/jclark/Submitter/example.zip. The generic shortcut ~/ is often used to replace the location of your home directory (/home/username) to save typing, but your path is still an absolute path if it starts with ~/. Note that a relative path will always start with either ~/ or /. Relative paths start from some location in your file system that is below the root. Relative paths are combined with the path of that location to locate files on your system. R (and some other languages like MATLAB) refer to the location where the relative path starts as our working directory. If our working directory is set to ~/Submitter, the relative path to the zip would be just example.zip. Note that a relative path will never start with either ~/ or /. Getting these paths right is very important because we don’t want submitters to download a folder of data, and have the paths look like /home/internname/ticket_27341/important_folder/important_file.csv. The first part of that path is particular to however the person processing the ticket organized the data, and is not how the submitter of the data intented to organize the data. Follow the steps below to make sure this does not happen. get a list of absolute paths for each file in the directory. NOTE The “PI_dir_name” here represents whatever directory you retrieved after running unzip in the previous step. The actual .zip file should not be in this directory. get a list of relative paths for each file in the directory. Note this is the same command, but with the argument full.names set to FALSE. abs_paths &lt;- list.files(&quot;~/Submitter&quot;, full.names = TRUE, recursive = TRUE) rel_paths &lt;- list.files(&quot;~/Submitter&quot;, full.names = FALSE, recursive = TRUE) Make sure that these paths look correct! They should contain ONLY the files that were unzipped. If you have other scripts or metadata files you might want to rearrange your directories to get the correct paths. The relative paths should start with the submitter’s directory name. In this example they will look like the below: “final_image_set/level1.png” “final_image_set/photos/level2_1.png” “final_image_set/photos/level2_2.png” Now for each of these files, we can create a dataObject for them and add them to the package using a loop. Before running this, look at the values of your abs_paths and rel_paths and make sure they look correct based on what you know about both paths and the structure of the directory. Within this loop we will also create otherEntities for each item, just putting in the bare minimum of information that will help us make sure that we know what files are what. metadataObj &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) doc &lt;- read_eml(getObject(d1c@mn, metadataObj)) doc$dataset$otherEntity &lt;- NULL oes &lt;- list() for (i in 1:length(abs_paths)){ formatId &lt;- arcticdatautils::guess_format_id(abs_paths[i]) id &lt;- generateIdentifier(d1c@mn, scheme = &quot;uuid&quot;) dataObj &lt;- new(&quot;DataObject&quot;, format = formatId, filename = abs_paths[i], targetPath = rel_paths[i], id = id) dp &lt;- addMember(dp, dataObj, metadataObj) oes[[i]] &lt;- eml$otherEntity(entityName = rel_paths[i], entityType = formatId, id = id) } doc$dataset$otherEntity &lt;- oes write_eml(doc, &quot;~/metadata.xml&quot;) dp &lt;- replaceMember(dp, metadataObj, replacement=&quot;~/metadata.xml&quot;) Once this is finished you can examine the relationships by running View(dp@relations$relations). If everything worked out well you should see rows that look like this: urn:uuid:a398312d-2c87-4b19-8380-3f11d2a1922d | http://www.w3.org/ns/prov#atLocation | figure2/figure2.tif | The targetPath (3rd column) should NOT look like urn:uuid:a398312d-2c87-4b19-8380-3f11d2a1922d | http://www.w3.org/ns/prov#atLocation | /home/jclark/ArcticSupport/Zhao/dataset1/figure2/figure2.tif | Make sure this looks correct, then update the data package with: uploadDataPackage(d1c, dp, public = FALSE) Finally, check your work. Go to the Arctic Data Center and see if the package displays correctly. Edit the package using the user interface to remove the zip file. Continue with metadata updates as normal. Summary Here is the example code all put together. Make sure you change all of the relevant bits, and check your work carefully!! url &lt;- &quot;https://arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A8fee5046-1a8f-4ccc-80f2-70c557a66338&quot; pid &lt;- gsub(&quot;https://arcticdata.io/metacat/d1/mn/v2/object/&quot;, &quot;&quot;, url) %&gt;% gsub(&quot;%3A&quot;, &quot;:&quot;, .) writeBin(getObject(d1c@mn, pid), &quot;~/Submitter/example.zip&quot;) unzip(&quot;~/Submitter/example.zip&quot;, exdir = &quot;~/Submitter&quot;) dp &lt;- getDataPackage(d1c, &quot;identifier&quot;) abs_paths &lt;- list.files(&quot;~/Submitter/PI_dir_name/&quot;, full.names = TRUE, recursive = TRUE) rel_paths &lt;- list.files(&quot;~/Submitter/PI_dir_name/&quot;, full.names = FALSE, recursive = TRUE) metadataObj &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) for (i in 1:length(abs_paths)){ formatId &lt;- arcticdatautils::guess_format_id(abs_paths[i]) dataObj &lt;- new(&quot;DataObject&quot;, format = formatId, filename = abs_paths[i], targetPath = rel_paths[i]) dp &lt;- addMember(dp, dataObj, metadataObj) } "],["update-a-package-with-a-new-data-object.html", "Update a package with a new data object", " Update a package with a new data object Once you have updated the data objects and saved the metadata to a file, we can update the metadata and use replaceMember to update the package with the new metadata. Make sure you have the package you want to update loaded into R using dataone::getDataPackage(). Publish update Now we can update your data package to include the new data object. Assuming you have updated your data package earlier something like the below: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) packageId &lt;- &quot;resource_map_urn:uuid...&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #some modification to the EML here eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) You can then upload your data package: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c_test, dp, public=FALSE, accessRules=myAccessRules, quiet=FALSE) If a package is ready to be public, you can change the public argument in the datapack::uploadDataPackage() call to TRUE. If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), you need to do this when replacing the metadata using the optional newId argument in replaceMember(). This should only be done after the package is finalized and has been thoroughly reviewed! doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If there is a pre-issued DOI (researcher requested the DOI for the publication first), please do the following: dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;doi:10.../...&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If the package has children, see how to do this using arcticdatautils::publish_update in the nesting section of the reference manual. Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. "],["update-an-object.html", "Update an object", " Update an object To update a data file associated with a data package, you need to do three things: update the object itself, update the resource map (which affiliates the object with the metadata), and update the metadata that describes that object The datapack::replaceMember function takes care of the first two of these tasks. First you need to get the pid of the file you want to replace by using datapack::selectMember # Select the EML PID dataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;text/csv&quot;) Then use replaceMember: # Replace the old file with the new dp &lt;- replaceMember(dp, dataId, replacement=file_path) If you want to remove some files from the data package we can use datapack::removeMember. If we wanted to remove all the zip files associated with this data package, we can use datapack::removeMember: zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) You will need to be explicit about your format_id here based on the file type. A list of format IDs can be found here on the DataONE website. Use line 2 (Id:) exactly, character for character. To accomplish the second task, you will need to update the metadata using the EML package. This is covered in Chapter 4. After you update a file, you will always need to update the metadata because parts of the physical section (such as the file size, checksum) will be different, and it may also require different attribute information. Once you have updated your metadata and saved it, you can update the package itself. "],["explore-eml.html", "Explore EML ", " Explore EML "],["access-specific-elements.html", "Access specific elements", " Access specific elements The eml_get() function is a powerful tool for exploring EML (more on that here ). It takes any chunk of EML and returns all instances of the element you specify. Note: you’ll have to specify the element of interest exactly, according to the spelling/capitalization conventions used in EML. Here are some examples: doc &lt;- read_eml(system.file(&quot;example-eml.xml&quot;, package = &quot;arcticdatautils&quot;)) eml_get(doc, &quot;creator&quot;) individualName: givenName: Bryce surName: Mecum organizationName: National Center for Ecological Analysis and Synthesis eml_get(doc, &quot;boundingCoordinates&quot;) eastBoundingCoordinate: &#39;-134&#39; northBoundingCoordinate: &#39;59&#39; southBoundingCoordinate: &#39;57&#39; westBoundingCoordinate: &#39;-135&#39; eml_get(doc, &quot;url&quot;) &#39;&#39;: function: download url: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 &#39;&#39;: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 eml_get_simple() is a simplified alternative to eml_get() that produces a list of the desired EML element. eml_get_simple(doc$dataset$otherEntity, &quot;entityName&quot;) To find an eml element you can use either a combination of which_in_emlfrom the arcticdatautils package or eml_get_simple and which to find the index in an EML list. Use which ever workflow you see fit. An example question you may have: Which creators have a surName “Mecum”? Example using which_in_eml: n &lt;- which_in_eml(doc$dataset$creator, &quot;surName&quot;, &quot;Mecum&quot;) # Answer: doc$dataset$creator[[n]] Example using eml_get_simple and which: ent_names &lt;- eml_get_simple(doc$dataset$creator, &quot;surName&quot;) i &lt;- which(ent_names == &quot;Mecum&quot;) # Answer: doc$dataset$creator[[i]] "],["navigate-through-eml.html", "Navigate through EML", " Navigate through EML The first task when editing an EML file is navigating the EML file. An EML file is organized in a structure that contains many lists nested within other lists. The function View allows you to get a crude view of an EML file in the viewer. It can be useful for exploring the file. # Need to be in this member node to explore file d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) doc &lt;- read_eml(getObject(d1c_test@mn, &quot;urn:uuid:558eabf1-1e91-4881-8ba3-ef8684d8f6a1&quot;)) View(doc) The complex EML document is represented in R as as series of named, nested lists. We use lists all the time in R! A data.frame is one example of a special kind of list that we use all the time. You may be familiar with the syntax dataframe$column_name which allows us to select a particular column of a data.frame. Under the hood, a data.frame is a named list of vectors with the same length. You select one of those vectors using the $ operator, which is called the “list selector operator.” Just like you navigate in a data.frame, you can use the $ operator to navigate through the EML structure. The $ operator allows you to go deeper into the EML structure and to see what elements are nested within other elements. However, you have to tell R where you want to go in the structure when you use the $ symbol. For example, if you want to view the dataset element of your EML you would use the command doc$dataset. If you want to view the creators of your data set you would use doc$dataset$creator. Note here that creator is contained within dataset. If you aren’t sure where you want to go, hit the tab button on your keyboard after typing $ and a list of available elements in the structure will appear (e.g., doc$&lt;TAB&gt;): Note that if you hit tab, and nothing pops up, this most likely implies that you are trying to go into an EML element that can take a series items. For example doc$dataset$creator$&lt;TAB&gt; will not show a pop-up menu. This is because creator is a series-type object (i.e. you can have multiple creators). If you want to go deeper into creator, you first must tell R which creator you are interested in. Do this by writing [[i]] first where i is the index of the creator you are concerned with. For example, if you want to look at the first creator i = 1. Now doc$dataset$creator[[1]]$&lt;TAB&gt; will give you many more options. Note, an empty autocomplete result sometimes means you have reached the end of a branch in the EML structure. Below is the structure of doc$dataset. There are a series of multiple creators, which can be accessed individually by index: doc$dataset$creator[[#]]. At this point stop and take a deep breath. The key takeaway is that EML is a hierarchical tree structure. The best way to get familiar with it is to explore the structure. Try entering doc$dataset into your console, and print it. Now make the search more specific, for instance: doc$dataset$abstract. "],["understand-the-eml-schema.html", "Understand the EML schema", " Understand the EML schema Another great resource for navigating the EML structure is looking at the schema which defines the structure. The schema diagrams on this page are interactive. Further explanations of the symbology can be found here. The schema is complicated and may take some time to get familiar with before you will be able to fully understand it. Use your browser’s “search in page” function (usually CTRL-F or Command-F) to navigate the EML schema page quickly. For example, let’s take a look at eml-party. To start off, notice that some elements have bolded lines leading to them. A bold line indicates that the element is required if the element above it (to the left in the schema) is used, otherwise the element is optional. Notice also that next to the givenName element it says “0..infinity”. This means that the element is unbounded — a single party can have many given names and there is no limit on how many you can add. However, this text does not appear for the surName element — a party can have only one surname. You will also see icons linking the EML slots together, which indicate the ordering of subsequent slots. These can indicate either a “sequence” or a “choice”. In our example from eml-party, a “choice” icon indicates that either an individualName, organizationName, or positionName is required, but you do not need all three. However, the “sequence” icon tells us that if you use an individualName, you must include the surName as a child element. If you include the optional child elements salutation and givenName, they must be written in the order presented in the schema. The eml schema sections you may find particularly helpful include eml-party, eml-attribute and eml-physical. For a more detailed description of the EML schema, see the reference section on exploring EML. "],["edit-eml.html", "Edit EML ", " Edit EML "],["edit-an-eml-element.html", "Edit an EML element", " Edit an EML element There are multiple ways to edit an EML element. Edit EML with strings The most basic way to edit an EML element would be to navigate to the element and replace it with something else. Easy! For example, to change the title one could use the following command: doc$dataset$title &lt;- &quot;New Title&quot; If the element you are editing allows for multiple values, you can pass it a list of character strings. Since a dataset can have multiple titles, we can do this: doc$dataset$title &lt;- list(&quot;New Title&quot;, &quot;Second New Title&quot;) However, this isn’t always the best method to edit the EML, particularly if the element has sub-elements. Adding directly to doc without a helper function can overwrite these parts of the doc that we need. Edit EML with the “EML” package To edit a section where you are not 100% sure of the sub-elements, using the eml$elementName() helper functions from the EML package will pre-populate the options for you if you utilize the RStudio autocomplete functionality. The arguments in these functions show the available slots for any given EML element. For example, typing doc$dataset$abstract &lt;- eml$abstract()&lt;TAB&gt; will show you that the abstract element can take either the section or para sub-elements. doc$dataset$abstract &lt;- eml$abstract(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) This inserts the abstract with a para element in our dataset, which we know from the EML schema is valid. Note that the above is equivalent to the following generic construction: doc$dataset$abstract &lt;- list(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) The eml() family of functions provides the sub-elements as arguments, which is extremely helpful, but functionally all it is doing is creating a named list, which you can also do using the list function. Edit EML with objects A final way to edit an EML element would be to build a new object to replace the old object. To begin, you might create an object using an eml helper function. Let’s take keywords as an example. Sometimes keyword lists in a metadata record will come from different thesauruses, which you can then add in series (similar to the way we added multiple titles) to the element keywordSet. We start by creating our first set of keywords and saving it to an object. kw_list_1 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER controlled vocabulary&quot;, keyword = list(&quot;bacteria&quot;, &quot;carnivorous plants&quot;, &quot;genetics&quot;, &quot;thresholds&quot;)) Which returns: $keyword $keyword[[1]] [1] &quot;bacteria&quot; $keyword[[2]] [1] &quot;carnivorous plants&quot; $keyword[[3]] [1] &quot;genetics&quot; $keyword[[4]] [1] &quot;thresholds&quot; $keywordThesaurus [1] &quot;LTER controlled vocabulary&quot; We create the second keyword list similarly: kw_list_2 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER core area&quot;, keyword = list(&quot;populations&quot;, &quot;inorganic nutrients&quot;, &quot;disturbance&quot;)) Finally, we can insert our two keyword lists into our EML document just like we did with the title example above, but rather than passing character strings into list(), we will pass our two keyword set objects. doc$dataset$keywordSet &lt;- list(kw_list_1, kw_list_2) Note that you must use the function list here and not the c() function. The reasons for this are complex, and due to some technical subtlety in R - but the gist of the issue is that the c() function can behave in unexpected ways with nested lists, and frequently will collapse the nesting into a single level, resulting in invalid EML. "],["edit-attributelists.html", "Edit attributeLists", " Edit attributeLists Attributes are descriptions of variables, typically columns or column names in tabular data. Attributes are stored in an attributeList. When editing attributes in R, we convert the attribute list information to data frame (table) format so that it is easier to edit. When editing attributes you will need to create one to three data frame objects: A data.frame of attributes A data.frame of custom units (if applicable) A data.frame of factors (if applicable) The attributeList is an element within one of 4 different types of entity objects. An entity corresponds to a file, typically. Multiple entities (files) can exist within a dataset. The 4 different entity types are dataTable (most common for us), spatialVector, spatialRaster, and otherEntity. Please note that submitting attribute information through the website will store them in an otherEntity object by default. We prefer to store them in a dataTable object for tabular data or a spatialVector or spatialRaster object for spatial data. To edit or examine an existing attribute list already in an EML file, you can use the following commands, where i represents the index of the series element you are interested in. Note that if there is only one item in the series (ie there is only one dataTable), you should just call doc$dataset$dataTable, as in this case doc$dataset$dataTable[[1]] will return the first sub-element of the dataTable (the entityName) # If they are stored in an otherEntity (submitted from the website by default) attribute_tables &lt;- EML::get_attributes(doc$dataset$otherEntity[[i]]$attributeList) # Or if they are stored in a dataTable (usually created by a datateam member) attribute_tables &lt;- EML::get_attributes(doc$dataset$dataTable[[i]]$attributeList) The get_attributes() function returns the attribute_tables object, which is a list of the three data frames mentioned above. The data frame with the attributes is called attribute_tables$attributes. attribute_tables$attributes print(attribute_tables$attributes) Edit attributes Attribute information should be stored in a data.frame with the following columns: attributeName: The name of the attribute as listed in the csv. Required. e.g.: “c_temp” attributeLabel: A descriptive label that can be used to display the name of an attribute. It is not constrained by system limitations on length or special characters. Optional. e.g.: “Temperature (Celsius)” attributeDefinition: Longer description of the attribute, including the required context for interpreting the attributeName. Required. e.g.: “The near shore water temperature in the upper inter-tidal zone, measured in degrees Celsius.” measurementScale: One of: nominal, ordinal, dateTime, ratio, interval. Required. nominal: unordered categories or text. e.g.: (Male, Female) or (Yukon River, Kuskokwim River) ordinal: ordered categories. e.g.: Low, Medium, High dateTime: date or time values from the Gregorian calendar. e.g.: 01-01-2001 ratio: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second. interval: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius. domain: One of: textDomain, enumeratedDomain, numericDomain, dateTime. Required. textDomain: text that is free-form, or matches a pattern enumeratedDomain: text that belongs to a defined list of codes and definitions. e.g.: CASC = Cascade Lake, HEAR = Heart Lake dateTimeDomain: dateTime attributes numericDomain: attributes that are numbers (either ratio or interval) formatString: Required for dateTime, NA otherwise. Format string for dates, e.g. “DD/MM/YYYY”. definition: Required for textDomain, NA otherwise. Defines a format for attributes that are a character string. e.g.: “Any text” or “7-digit alphanumeric code” unit: Required for numericDomain, NA otherwise. Unit string. If the unit is not a standard unit, a warning will appear when you create the attribute list, saying that it has been forced into a custom unit. Use caution here to make sure the unit really needs to be a custom unit. A list of standard units can be found using: standardUnits &lt;- EML::get_unitList() then running View(standardUnits$units). numberType: Required for numericDomain, NA otherwise. Options are real, natural, whole, and integer. real: positive and negative fractions and integers (…-1,-0.25,0,0.25,1…) natural: non-zero positive integers (1,2,3…) whole: positive integers and zero (0,1,2,3…) integer: positive and negative integers and zero (…-2,-1,0,1,2…) missingValueCode: Code for missing values (e.g.: ‘-999’, ‘NA’, ‘NaN’). NA otherwise. Note that an NA missing value code should be a string, ‘NA’, and numbers should also be strings, ‘-999.’ missingValueCodeExplanation: Explanation for missing values, NA if no missing value code exists. You can create attributes manually by typing them out in R following a workflow similar to the one below: attributes &lt;- data.frame( attributeName = c(&#39;Date&#39;, &#39;Location&#39;, &#39;Region&#39;,&#39;Sample_No&#39;, &#39;Sample_vol&#39;, &#39;Salinity&#39;, &#39;Temperature&#39;, &#39;sampling_comments&#39;), attributeDefinition = c(&#39;Date sample was taken on&#39;, &#39;Location code representing location where sample was taken&#39;,&#39;Region where sample was taken&#39;, &#39;Sample number&#39;, &#39;Sample volume&#39;, &#39;Salinity of sample in PSU&#39;, &#39;Temperature of sample&#39;, &#39;comments about sampling process&#39;), measurementScale = c(&#39;dateTime&#39;, &#39;nominal&#39;,&#39;nominal&#39;, &#39;nominal&#39;, &#39;ratio&#39;, &#39;ratio&#39;, &#39;interval&#39;, &#39;nominal&#39;), domain = c(&#39;dateTimeDomain&#39;, &#39;enumeratedDomain&#39;,&#39;enumeratedDomain&#39;, &#39;textDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;textDomain&#39;), formatString = c(&#39;MM-DD-YYYY&#39;, NA,NA,NA,NA,NA,NA,NA), definition = c(NA,NA,NA,&#39;Six-digit code&#39;, NA, NA, NA, &#39;Any text&#39;), unit = c(NA, NA, NA, NA,&#39;milliliter&#39;, &#39;dimensionless&#39;, &#39;celsius&#39;, NA), numberType = c(NA, NA, NA,NA, &#39;real&#39;, &#39;real&#39;, &#39;real&#39;, NA), missingValueCode = c(NA, NA, NA,NA, NA, NA, NA, &#39;NA&#39;), missingValueCodeExplanation = c(NA, NA, NA,NA, NA, NA, NA, &#39;no sampling comments&#39;)) However, typing this out in R can be a major pain. Luckily, there’s a Shiny app that you can use to build attribute information. You can use the app to build attributes from a data file loaded into R (recommended as the app will auto-fill some fields for you) to edit an existing attribute table, or to create attributes from scratch. Use the following commands to create or modify attributes. Use the following commands to create or modify attributes. These commands will launch a “Shiny” app in your web browser. #first download the CSV in your data package from Exercise #2 data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;.csv&quot;) data &lt;- read.csv(text=rawToChar(getObject(d1c_test@mn, data_pid))) # From data (recommended) attribute_tables &lt;- EML::shiny_attributes(data = data) # From scratch attribute_tables &lt;- EML::shiny_attributes() # From an existing attribute list attribute_tables &lt;- get_attributes(doc$dataset$dataTable[[i]]$attributeList) attribute_tables &lt;- EML::shiny_attributes(attributes = attribute_tables$attributes) Once you are done editing a table in the browser app, quit the app by pressing the red “Quit App” button in the top right corner of the page. If you close the Shiny app tab in your browser instead of using the “Quit App” button, your work will not be saved, R will think that the Shiny app is still open, and you will not be able to run other code. You can tell if R is confused if you have closed the Shiny app and the bottom line in the console still says Listening on http://.... If this happens, press the red stop sign button on the right hand side of the console window in order to interrupt R. The tables you constructed in the app will be assigned to the attribute_tables variable as a list of data frames (one for attributes, factors, and units). Be careful to not overwrite your completed attribute_tables object when trying to make edits. The last line of code can be used in order to make edits to an existing attribute_tables object. Alternatively, each table can be to exported to a csv file by clicking the Download button. If you downloaded the table, read the table back into your R session and assign it to a variable in your script (e.g. attributes &lt;- data.frame(...)), or just use the variable that shiny_attributes returned. For simple attribute corrections, datamgmt::edit_attribute() allows you to edit the slots of a single attribute within an attribute list. To use this function, pass an attribute through datamgmt::edit_attribute() and fill out the parameters you wish to edit/update. An example is provided below where we are changing attributeName, domain, and measurementScale in the first attribute of a dataset. After completing the edits, insert the new version of the attribute back into the EML document. new_attribute &lt;- datamgmt::edit_attribute(doc$dataset$dataTable[[1]]$attributeList$attribute[[1]], attributeName = &#39;date_and_time&#39;, domain = &#39;dateTimeDomain&#39;, measurementScale = &#39;dateTime&#39;) doc$dataset$dataTable[[1]]$attributeList$attribute[[1]] &lt;- new_attribute Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) Search the units list for your unit before attempting to create a custom unit. You can search part of the unit you can look up part of the unit ie meters in the table to see if there are any matches. If you have units that are not in the standard EML unit list, you will need to build a custom unit list. Attribute tables with custom units listed will generate a warning indicating that a custom unit will need to be described. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”). The parentSI does not need to be part of the unitList. multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;siemensPerMeter&#39;, &#39;decibar&#39;), unitType = c(&#39;resistivity&#39;, &#39;pressure&#39;), parentSI = c(&#39;ohmMeter&#39;, &#39;pascal&#39;), multiplierToSI = c(&#39;1&#39;,&#39;10000&#39;), abbreviation = c(&#39;S/m&#39;,&#39;decibar&#39;), description = c(&#39;siemens per meter&#39;, &#39;decibar&#39;)) Custom units can also be created in the shiny app, under the “units” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() custom_units &lt;- attribute_tables$units Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- list(metadata = list(unitList = unitlist)) Edit factors For attributes that are enumeratedDomains, a table is needed with three columns: attributeName, code, and definition. attributeName should be the same as the attributeName within the attribute table and repeated for all codes belonging to a common attribute. code should contain all unique values of the given attributeName that exist within the actual data. definition should contain a plain text definition that describes each code. To build factors by hand, you use the named character vectors and then convert them to a data.frame as shown in the example below. In this example, there are two enumerated domains in the attribute list - “Location” and “Region”. Location &lt;- c(CASC = &#39;Cascade Lake&#39;, CHIK = &#39;Chikumunik Lake&#39;, HEAR = &#39;Heart Lake&#39;, NISH = &#39;Nishlik Lake&#39; ) Region &lt;- c(W_MTN = &#39;West region, locations West of Eagle Mountain&#39;, E_MTN = &#39;East region, locations East of Eagle Mountain&#39;) The definitions are then written into a data.frame using the names of the named character vectors and their definitions. factors &lt;- rbind(data.frame(attributeName = &#39;Location&#39;, code = names(Location), definition = unname(Location)), data.frame(attributeName = &#39;Region&#39;, code = names(Region), definition = unname(Region))) Factors can also be created in the shiny app, under the “factors” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() attribute_tables$factors Finalize attributeList Once you have built your attributes, factors, and custom units, you can add them to EML objects. Attributes and factors are combined to form an attributeList using set_attributes(): # Create an attributeList object attributeList &lt;- EML::set_attributes(attributes = attribute_tables$attributes, factors = attribute_tables$factors) This attributeList object can then be checked for errors and added to a dataTable in the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList "],["edit-custom-units-1.html", "Edit custom units", " Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) If you have units that are not in the standard EML unit list, you will need to build a custom unit list. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”) multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;partsPerThousand&#39;, &#39;decibar&#39;, &#39;wattsPerSquareMeter&#39;, &#39;micromolesPerGram&#39;, &#39;practicalSalinityUnit&#39;), unitType = c(&#39;dimensionless&#39;, &#39;pressure&#39;, &#39;power&#39;, &#39;amountOfSubstanceWeight&#39;， &#39;dimensionless&#39;), parentSI = c(NA, &#39;pascal&#39;, &#39;watt&#39;, &#39;molesPerKilogram&#39;， NA), multiplierToSI = c(NA, &#39;10000&#39;, &#39;1&#39;, &#39;1000000000&#39;, NA), abbreviation = c(&#39;ppt&#39;, &#39;decibar&#39;, &#39;W/m^2&#39;, &#39;umol/g&#39;, &#39;PSU&#39;), description = c(&#39;parts per thousand&#39;, &#39;decibar&#39;, &#39;watts per square meter&#39;, &#39;micro moles per gram&#39;, &#39;used to describe the concentration of dissolved salts in water, the UNESCO Practical Salinity Scale of 1978 (PSS78) defines salinity in terms of a conductivity ratio&#39;)) Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- unitlist If units that should be standardUnit are added as customUnit, you can use the following code to fix this issue: # add standard unit doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$measurementScale$ratio$unit$standardUnit &lt;- &quot;your standard unit&quot; # get rid of custom unit doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$measurementScale$ratio$unit$customUnit &lt;- NULL If you want to find all of the positions of a certain custom unit that should be standard, try this code: ls &lt;- purrr::map(doc$dataset$dataTable[[i]]$attributeList$attribute, ~str_detect(.x$measurementScale$ratio$unit$customUnit, &quot;your standard unit&quot;)) which(ls == TRUE) "],["edit-datatables.html", "Edit dataTables", " Edit dataTables Entities that are dataTables require an attribute list. To edit a dataTable, first edit/create an attributeList and set the physical. Then create a new dataTable using the eml$dataTable() helper function as below: dataTable &lt;- eml$dataTable(entityName = &quot;A descriptive name for the data (does not need to be the same as the data file)&quot;, entityDescription = &quot;A description of the data&quot;, physical = physical, attributeList = attributeList) The dataTable must then be added to the EML. How exactly you do this will depend on whether there are dataTable elements in your EML, and how many there are. To replace whatever dataTable elements already exist, you could write: doc$dataset$dataTable &lt;- dataTable If there is only one dataTable in your dataset, the EML package will usually “unpack” these, so that it is not contained within a list of length 1 - this means that to add a second dataTable, you cannot use the syntax doc$dataset$dataTable[[2]], since when unpacked this will contain the entityDescription as opposed to pointing to the second in a series of dataTable elements. Confusing - I know. Not to fear though - this syntax will get you on your way, should you be trying to add a second dataTable. doc$dataset$dataTable &lt;- list(doc$dataset$dataTable, dataTable) If there is more than one dataTable in your dataset, you can return to the more straightforward construction of: doc$dataset$dataTable[[i]] &lt;- dataTable Where i is the index that you wish insert your dataTable into. To add a list of dataTables to avoid the unpacking problem above you will need to create a list of dataTables dts &lt;- list() # create an empty list for(i in seq_along(tables_you_need)){ # your code modifying/creating the dataTable here dataTable &lt;- eml$dataTable(entityName = dataTable$entityName, entityDescription = dataTable$entityDescription, physical = physical, attributeList = attributeList) dts[[i]] &lt;- dataTable # add to the list } After getting a list of dataTables, assign the resulting list to dataTable EML doc$dataset$dataTable &lt;- dts By default, the online submission form adds all entities as otherEntity, even when most should probably be dataTable. You can use eml_otherEntity_to_dataTable to easily move items in otherEntity over to dataTable, and delete the old otherEntity. Most tabular data or data that contain variables should be listed as a dataTable. Data that do not contain variables (eg: plain text readme files, pdfs, jpegs) should be listed as otherEntity. eml_otherEntity_to_dataTable(doc, 1, # Indexes of otherEntities you want to convert, for multiple use 1:5 or c(1,3,5..) validate_eml = F) # set this to False if the physical or attributes are not added "],["edit-otherentities.html", "Edit otherEntities", " Edit otherEntities Remove otherEntities To remove an otherEntity use the following command. This may be useful if a data object is originally listed as an otherEntity and then transferred to a dataTable. doc$dataset$otherEntity[[i]] &lt;- NULL Create otherEntities If you need to create/update an otherEntity, make sure to publish or update your data object first (if it is not already on the DataONE MN). Then build your otherEntity. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, pkg$data[[i]]) Alternatively, you can build the otherEntity of a data object not in your package by simply inputting the data PID. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, &quot;your_data_pid&quot;, entityType = &quot;otherEntity&quot;, entityName = &quot;Entity Name&quot;, entityDescription = &quot;Description about entity&quot;) The otherEntity must then be set to the EML, like so: doc$dataset$otherEntity &lt;- otherEntity If you have more than one otherEntity object in the EML already, you can add the new one like this: doc$dataset$otherEntity[[i]] &lt;- otherEntity Where i is set to the number of existing entities plus one. Remember the warning from the last section, however. If you only have one otherEntity, and you are trying to add another, you have to run: doc$dataset$otherEntity &lt;- list(otherEntity, doc$dataset$otherEntity) "],["semantic-annotations.html", "Semantic annotations", " Semantic annotations For a brief overview of what a semantic annotation is, and why we use them check out this video. Even more information on how to add semantic annotations to EML 2.2.0 can be found here. There are several elements in the EML 2.2.0 schema that can be annotated: dataset entity (eg: otherEntity or dataTable) attribute Attribute annotations can be edited in R and also on the website. Dataset and entity annotations are only done in R. How annotations are used This is a dataset that has semantic annotations included. On the website you can see annotations in each of the attributes. You can click on any one of them to search for more datasets with that same annotation. Semantic attribute annotations can be applied to spatialRasters, spatialVectors and dataTables Attribute-level annotations on the website editor The website has a searchable list of attribute annotations that are grouped by category and specificity. Open your dataset on the test website from earlier and enter the attribute editor. Look through all of the available annotations. Adding attribute annotations using the website is the easiest way, however adding them using R and/or the Shiny app may be quicker with very larger datasets. Attribute-level annotations in R To manually add annotations to the attributeList in R you will need information about the propertyURI and valueURI. Annotations are essentially composed of a sentence, which contains a subject (the attribute), predicate (propertyURI), and object (valueURI). Because of the way our search interface is built, for now we will be using attribute annotations that have a propertyURI label of “contains measurements of type”. Here is what an annotation for an attribute looks like in R. Note that both the propertyURI and valueURI have both a label, and the URI itself. doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$annotation $id [1] &quot;ODBcOyaTsg&quot; $propertyURI $propertyURI$label [1] &quot;contains measurements of type&quot; $propertyURI$propertyURI [1] &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot; $valueURI $valueURI$label [1] &quot;Distributed Biological Observatory region identifier&quot; $valueURI$valueURI [1] &quot;http://purl.dataone.org/odo/ECSO_00002617&quot; How to add an annotation 1. Decide which variable to annotate The goal for the datateam is to start annotating every dataset that comes in. Please make sure to add semantic annotations to spatial and temporal features such as latitude, longitude, site name and date and aim to annotate as many attributes as possible. 2. Find an appropriate valueURI The next step is to find an appropriate value to fill in the blank of the sentence: “this attribute contains measurements of _____.” There are several ontologies to search in. In order of most to least likely to be relevant to the Arctic Data Center they are: The Ecosystem Ontology (ECSO) this was developed at NCEAS, and has many terms that are relevant to ecosystem processes, especially those involving carbon and nutrient cycling The Environment Ontology (EnVO) this is an ontology for the concise, controlled description of environments National Center for Biotechnology Information (NCBI) Organismal Classification (NCBITAXON) The NCBI Taxonomy Database is a curated classification and nomenclature for all of the organisms in the public sequence databases. Information Artifact Ontology (IAO) this ontology contains terms related to information entities (eg: journals, articles, datasets, identifiers) To search, navigate through the “classes” until you find an appropriate term. When we are picking terms, it is important that we not just pick a similar term or a term that seems close - we want a term that is 100% accurate. For example, if you have an attribute for carbon tetroxide flux and an ontology with a class hierarchy like this: – carbon flux |—- carbon dioxide flux Our exact attribute, carbon tetroxide flux, is not listed. In this case, we should pick “carbon flux” as it’s completely correct/accurate and not “carbon dioxide flux” because it’s more specific/precise but not quite right. For general attributes (such as ones named depth or length), it is important to be as specific as possible about what is being measured. e.g. selecting the lake area annotation for the area attribute in this dataset 3. Build the annotation in R Manually Annotating this method is great for when you are inserting 1 annotation, fixing an existing annotation or programmatically updating annotations for multiple attributeLists First you need to figure out the index of the attribute you want to annotate. eml_get_simple(doc$dataset$dataTable[[3]]$attributeList, &quot;attributeName&quot;) [1] &quot;prdM&quot; &quot;t090C&quot; &quot;t190C&quot; &quot;c0mS/cm&quot; &quot;c1mS/cm&quot; &quot;sal00&quot; &quot;sal11&quot; &quot;sbeox0V&quot; &quot;flECO-AFL&quot; [10] &quot;CStarTr0&quot; &quot;cpar&quot; &quot;v0&quot; &quot;v4&quot; &quot;v6&quot; &quot;v7&quot; &quot;svCM&quot; &quot;altM&quot; &quot;depSM&quot; [19] &quot;scan&quot; &quot;sbeox0ML/L&quot; &quot;sbeox0dOV/dT&quot; &quot;flag&quot; Next, assign an id to the attribute. It should be unique within the document, and it’s nice if it is human readable and related to the attribute it is describing. One format you could use is entity_x_attribute_y which should be unique in scope, and is nice and descriptive. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$id &lt;- &quot;entity_ctd_attribute_salinity&quot; Now, assign the propertyURI information. This will be the same for every annotation you build. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$propertyURI &lt;- list(label = &quot;contains measurements of type&quot;, propertyURI = &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot;) Finally, add the valueURI information from your search. You should see an ID on the Bioportal page that looks like a URL - this is the valueURI. Use the value to populate the label element. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$valueURI &lt;- list(label = &quot;Water Salinity&quot;, valueURI = &quot;http://purl.dataone.org/odo/ECSO_00001164&quot;) Shiny Attributes this method is great for when you are updating many attributes On the far right of the table of shiny_attributes there are 4 columns: id, propertyURI, propertyLabel, valueURI, valueLabel that can be filled out. Dataset Annotations Dataset annotations can only be made using R. There are several helper functions that assist with making dataset annotations. Data Sensitivity Sensitive datasets that might cover protected characteristics (human subjects data, endangered species locations, etc) should be annotated using the data sensitivity ontology: https://bioportal.bioontology.org/ontologies/SENSO/?p=classes&amp;conceptid=root. Dataset Discipline As a final step in the data processing pipeline, we will categorize the dataset. We are trying to categorize datasets so we can have a general idea of what kinds of data we have at the Arctic Data Center. Datasets will be categorized using the Academic Ontology. These annotations will be seen at the top of the landing page, and can be thought of as “themes” for the dataset. In reality, they are dataset-level annotations. Be sure to ask your peers in the #datateam slack channel whether they agree with the themes you think best fit your dataset. Once there is consensus, use the following line of code: doc &lt;- arcticdatautils::eml_categorize_dataset(doc, c(&quot;Soil Science&quot;, &quot;Plant Science&quot;, &quot;Ecology&quot;)) Be careful not to duplicate dataset annotations. The above code does not existing prior dataset annotations. Duplicate annotations can be removed by setting them to NULL. doc$dataset$annotation[[i]] &lt;- NULL "],["edit-spatial-data.html", "Edit spatial data", " Edit spatial data Occasionally, you may encounter a third type of data object: spatialVector and spatialRaster. These objects contains spatial data (ie maps), such as a shapefile or geodatabase. Editing a spatialVector or spatialRaster is similar to editing a dataTable or an otherEntity. A physical and attributeList should be present. We will focus on how to get the information unique to spatialData and how to create the spatialVector/spatialRaster File types File extensions to look for that might be spatial data: kml, geoJSON, geoTIFF, .dbf, .shp, and .shx Additionally, spatial data that involve multiple files should typically be archived within a .zip file to ensure all related and interdependent files stay together (ie . a geodatabase). This is one of the exceptions to our rule regarding .zip files. For example, a spatial dataset for a shapefile should, at a minimum, consist of separate .dbf, .shp, and .shx files with the same prefix in the same directory. All these files are required in order to use the data. Also note that shapefiles limit attribute names to 10 characters, so attribute names in the metadata may not match exactly to attribute names in the data. Some spatial raster data come as standalone files (.tiff or .nc) and some come as a group of files. If you aren’t sure whether to unzip a file, ask Jasmine or Jeanette. There are specific formatIds for these kinds of zipped files: application/vnd.shp+zip image/geotiff+zip. Remember to check that the files have the correct formatId Reading Spatial Files Read in the files to (1) help you in creating your attributes table and (2) sometimes also figure out the coordinate reference system. library(sf) spatial_file &lt;- sf::read_sf(&quot;example.kml&quot;) When you read kml files, read_sf() sometimes shows additional columns that aren’t in the actual file. Always open kml files in text editor to check if the columns actually exist. If it is a zipped shapefile there is a handy function you can use arcticdatautils::read_zip_shapefile(mn, pid) Coordinate Systems The coordinate system allow to work with spatial data using the same frame of reference (a Datum). A common coordinate system is “GCS_WGS_1984 (used in Google Maps!) which is suitable for plotting points distributed globally. There are many others that may be better suited for certain areas in the world. All latitudes and longitude coordinates should have a coordinate system (like a frame of reference). There are horizontal coordinate systems (earth’s surface) and vertical coordinate systems (depth). More information can be found here. To find the horizCoordSysName you can use: sf::st_crs(spatial_file) Take the Datum and add GCS (Geographic Coordinate System) in front. For example: “GCS_WGS_1984” spatialVector Adding Geometry One important difference is that a spatialVector object should also have a geometry slot that describes the geometry features of the data. The possible values include one or more (in a list) of ‘Point’, ‘LineString’, ‘LinearRing’, ‘Polygon’, ‘MultiPoint’, ‘MultiLineString’, ‘MultiPolygon’, or ‘MultiGeometry’. You will likely have to open the file itself within QGIS or R (ie . the sf package) to get the correct geometry value. To add just a geometry slot use: doc$dataset$spatialVector[[1]]$geometry &lt;- &quot;Polygon&quot; To add it using the data pid: 1. Get the geometry and spatialReference 2. Use pid_to_eml_entity() to generate the spatialVector spatialVector &lt;- pid_to_eml_entity(adc, pkg$data[n], entity_type = &quot;spatialVector&quot;, entityName = &quot;filename.kml&quot;, entityDescription = &quot;some desciption&quot;, attributeList = attributeList, geometry = &quot;Point&quot;, spatialReference = list(horizCoordSysName = &quot;GCS_WGS_1984&quot;)) Add the spatialVector to the doc doc$dataset$spatialVector[[1]] &lt;- spatialVector spatialRasters Most often these come in GeoTiff or Tiff files. The data is presented as a grid of “pixels”. For more information ESRI has a indepth article here. To use the helper function get: the path of your raster file an attribute table a coordinate system To get a coordinate system name, you can use the output of the function on your first try (which will print the coordinate reference system, if it is defined). You can use the return value of get_coord_list() (a large data.frame) to find the correct coordinate system name. Another way to get the coordinate system name is using rgdal::GDALinfo(path). This function can provide many details for your GeoTiff or Tiff files including the coordinate system name. More information can be found here here. rgdal::GDALinfo(path) eml_get_raster_metadata(path, coord_name, attributes) "],["format-text-in-eml.html", "Format text in EML", " Format text in EML Currently, only certain fields (abstracts, methods) support text formatting in EML. Check out this demo for a full example. Additional info is also available here. Many of these formatting functions only work when enclosed by &lt;para&gt;&lt;/para&gt; You can insert these tags directly into the EML document using syntax that looks like this: doc$dataset$abstract &lt;- eml$abstract(para = &quot;Some abstracts require subscripts like CO&lt;subscript&gt;2&lt;/subscript&gt;&quot;) Type-setting Subscripts, superscripts, and italics: &lt;subscript&gt;You can do subscripts&lt;/subscript&gt; &lt;superscript&gt;or superscipts&lt;/superscript&gt; &lt;emphasis&gt;or even italics.&lt;/emphasis&gt; Links Be sure to include the “https://” before the link or it will redirect incorrectly. Also, always check that your links go through to the correct page. Please be aware that most links are inherently unstable so always default to archiving files over pointing to websites when possible and appropriate. &lt;ulink url=&quot;https://some_url.com&quot;&gt; &lt;citetitle&gt;some text&lt;/citetitle&gt; &lt;/ulink&gt; Lists Unordered (bulleted) lists: &lt;itemizedlist&gt; &lt;listitem&gt; &lt;para&gt;Paragraphs&lt;/para&gt; &lt;/listitem&gt; &lt;listitem&gt; &lt;para&gt;Sections w/ subsections (w/ titles)&lt;/para&gt; &lt;/listitem&gt; &lt;/itemizedlist&gt; Ordered lists (1, 2, 3)… &lt;orderedlist&gt; &lt;listitem&gt; &lt;para&gt;something&lt;/para&gt; &lt;/listitem&gt; &lt;listitem&gt; &lt;para&gt;something else&lt;/para&gt; &lt;/listitem&gt; &lt;/orderedlist&gt; "],["series-identifier-sid.html", "Series Identifier (SID)", " Series Identifier (SID) A series identifier is a system metadata field that represents a single identifier across multiple versions of a data package, a feature often requested by submitters. These are useful for maintaining a single identifier when a data package is expected to receive updates (usually additional data) in the future. For example, this dataset has a series id: https://doi.org/10.18739/A2154DQ22 (the doi and urn:uuid are both present at the top of the dataset). Adding a SID to a package Adding a series identifier should be the last step. In most cases the SID will be in DOI format. Once the data package is complete (peer-reviewd and approved by the PI) the series identifier can be added by updating the seriesId field of the system metadata of the metadata object. # The metadata identifier we&#39;re assigning an SID to metadata_pid &lt;- &#39;metadata_pid&#39; sys &lt;- getSystemMetadata(mn, metadata_pid) sys@seriesId &lt;- generateIdentifier(mn, scheme = &#39;DOI&#39;) #update the scheme argument if it should not be a DOI updateSystemMetadata(mn, metadata_pid, sys) Updating the child packages of a parent package with a SID When updating a parent package with a series identifier you need to use update_resource_map rather than publish_update. In this case our goal is add nested data packages to an existing parent package. # Call get_package on the parent package that has a `seriesId` in its system metadata parent_package &lt;- get_package(mn, &#39;resource_map_pid&#39;) # The resource map of the package we want to add to the &#39;child_pids&#39; of the parent resource_map_new_child &lt;- &#39;&#39; update_resource_map(mn, parent_package$resource_map, parent_package$metadata, parent_package$data, c(parent_package$child_packages, resource_map_new_child)) Updating the metadata of a package with a SID When updating the metadata (xml) of a package with a series identifier use publish_update. It’s important that you pass the metadata pid to the metadata_pid argument rather than seriesId. The metadata pid will usually have the text “version:” ahead of it, however it’s best to use get_package first to avoid mistakes. pkg &lt;- get_package(mn, &#39;resource_map_pid&#39;) pkg &lt;- publish_update(mn, pkg$metadata, pkg$resource_map, pkg$data, child_pids = pkg$child_pids) # + any additional arguments to &quot;publish_update&quot; "],["set-coverages.html", "Set coverages", " Set coverages Sometimes EML documents may lack coverage information describing the temporal, geographic, or taxonomic coverage of a data set. This example shows how to create coverage information from scratch, or replace an existing coverage element with an updated one. You can view the current coverage (if it exists) by entering doc$dataset$coverage into the console. Here the coverage, including temporal, taxonomic, and geographic coverages, is defined using set_coverage(). coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, sci_names = c(&#39;exampleGenus exampleSpecies1&#39;, &#39;exampleGenus ExampleSpecies2&#39;), geographicDescription = &quot;The geographic region covers the lake region near Eagle Mountain, Alaska.&quot;, west = -154.6192, east = -154.5753, north = 68.3831, south = 68.3619) doc$dataset$coverage &lt;- coverage Set multiple coverages You can also set multiple geographic (or temporal) coverages. Here is an example of how you might set two geographic coverages. Note that we use nested eml function helpers in this construction. geocov1 &lt;- eml$geographicCoverage(geographicDescription = &quot;The geographich region covers area 1&quot;, boundingCoordinates = eml$boundingCoordinates( northBoundingCoordinate = 68, eastBoundingCoordinate = -154, southBoundingCoordinate = 67, westBoundingCoordinate = -155)) geocov2 &lt;- eml$geographicCoverage(geographicDescription = &quot;The geographich region covers area 2&quot;, boundingCoordinates = eml$boundingCoordinates( northBoundingCoordinate = 65, eastBoundingCoordinate = -151, southBoundingCoordinate = 62, westBoundingCoordinate = -153)) coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, sci_names = list(&#39;exampleGenus exampleSpecies1&#39;, &#39;exampleGenus ExampleSpecies2&#39;)) doc$dataset$coverage$geographicCoverage &lt;- list(geocov1, geocov2) Special coverages Arctic Circle For arctic circle geographic coverage, we only have the starting vertical line of the circle shown in the projection. Here is an example with arctic circle geographic coverage. Geologic dates Example dataset with geologic coverages set using the following: geo_time_start &lt;- EML::eml$alternativeTimeScale(timeScaleName = &quot;Absolute&quot;, timeScaleAgeEstimate = &quot;7.5 Myr&quot;) coverage &lt;- EML::set_coverage(beginDate = &#39;2012-01-01&#39;, endDate = &#39;2012-01-10&#39;, doc$dataset$coverage &lt;- coverage "],["set-methods.html", "Set methods", " Set methods The methods tree in the EML section has many different options, visible in the schema. You can create new elements in the methods tree by following the schema and using the eml helpers. Another simple and potentially useful way to add methods to an EML that has no methods at all is by adding them via a MS Word document. An example is shown below: methods1 &lt;- set_methods(&#39;methods_doc.docx&#39;) doc$dataset$methods &lt;- methods1 If you want to make minor changes to existing methods information that has a lot of nested elements, your best bet may be to edit the EML manually in a text editor (or in RStudio), otherwise there is a risk of accidentally overwriting nested elements with blank object classes, therefore losing methods information. Adding sampling info to methods section # add method steps as new variables step1 &lt;- eml$methodStep(description = &quot;text describing the methods used&quot;) stEx &lt;- eml$studyExtent(description = &quot;study extent description&quot;) samp &lt;- eml$sampling(studyExtent = stEx, samplingDescription = &quot;sampling description text&quot;) # combine all methods steps and sampling info methods1 &lt;- eml$methods(methodStep = step1, sampling = samp) doc$dataset$methods &lt;- methods1 "],["set-parties.html", "Set parties", " Set parties The address, creator, contact, and associatedParty classes can easily be created using functions from the EML package. However it is often easier to just edit this through the webform. To add people, with their addresses, you need to add addresses as their own object class, which you then add to the contact, creator, or associatedParty classes. NCEASadd &lt;- eml$address( deliveryPoint = &quot;735 State St #300&quot;, city = &quot;Santa Barbara&quot;, administrativeArea = &#39;CA&#39;, postalCode = &#39;93101&#39;) Here, we use eml_creator() to set our data set creator. JC_creator &lt;- eml$creator(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jclark@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, address = NCEASadd) doc$dataset$creator &lt;- JC_creator Similarly, we can set a single contact or multiple. In this case, there are two, so we set doc$dataset$contact as a list containing both of them. JC_contact &lt;- eml$contact(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jclark@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;,id = eml$userId(directory = eml$dir) address = NCEASadd) JG_contact &lt;- eml$contact(individualName = list(givenName = &quot;Jesse&quot;, surName = &quot;Goldstein&quot;), organization = &quot;NCEAS&quot;, electronicMailAddress = &quot;jgoldstein@nceas.ucsb.edu&quot;, phone = &quot;123-456-7890&quot;, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, address = NCEASadd) doc$dataset$contact &lt;- list(JC_contact, JG_contact) Finally, the associatedPartys are set. Note that associatedPartys MUST have a role defined, unlike creator or contact. JG_ap &lt;- eml$associatedParty(individualName = list(givenName = &quot;Jesse&quot;, surName = &quot;Goldstein&quot;), organization = &quot;NCEAS&quot;, email = &quot;jgoldstein@nceas.ucsb.edu&quot;, electronicMailAddress = &quot;123-456-7890&quot;, address = NCEASadd, userId = &quot;https://orcid.org/WWWW-XXXX-YYYY-ZZZZ&quot;, role = &quot;metadataProvider&quot;) doc$dataset$associatedParty &lt;- JG_ap "],["set-physical.html", "Set physical", " Set physical To set the physical aspects of a data object, use the following commands to build a physical object from a data PID that exists in your package. Remember to set the member node to test.arcticdata.io! Every entity that we upload needs a physical description added. When replacing files, the physical must be replaced as well. The word ‘physical’ derives from database systems, which distinguish the ‘logical’ model (e.g., what attributes are in a table, etc) from the physical model (how the data are written to a physical hard disk (basically, the serialization)). so, we grouped metadata about the file (eg. dataformat, file size, file name) as written to disk in physical. It includes info like the file size. For CSV files, the physical describes the number of header lines and the attribute orientation. # Get the PID of a file data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;your_file_name.csv&quot;) # Get the physical info and store it in an object physical &lt;- arcticdatautils::pid_to_eml_physical(mn, data_pid) The physical object can then be checked for errors and added to the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$physical &lt;- physical Note that the above workflow only works if your data object already exists on the member node. Physicals can be seen in the website representation of the EML below the entity description. "],["set-the-project-section.html", "Set the project section", " Set the project section The project section in an EML document is automatically filled out by the metacatUI editor. It sets the project title and project personnel to the submission’s title and creators. Most of the time at least some of this information is incorrect and we need to update it. Start by searching for the funding information using NSF’s award search. This will give us the project title, abstract, and personnel - along with some additional metadata. Using this information we will set the title, personnel, and funding number. For NSF funded projects prepend the funding number with “NSF”. If there are multiple awards associated with one dataset then additional funding, title, and personnel elements should be added to reflect the additional awards. doc$dataset$project$funding$para[[1]] &lt;- &#39;NSF 1503846&#39; doc$dataset$project$title[[1]] &lt;- &#39;Collaborative Research: Reconciling conflicting Arctic temperature and fire reconstructions using multi-proxy records from lake sediments north of the Brooks Range, Alaska doc$dataset$project$personnel[[1]] &lt;- eml$personnel(individualName = eml$individualName(givenName = &#39;Yongsong&#39;, surName = &#39;Huang&#39;), role = &#39;Principal Investigator&#39;) doc$dataset$project$personnel[[2]] &lt;- eml$personnel(individualName = eml$individualName(givenName = &#39;James&#39;, surName = &#39;Russell&#39;), role = &#39;co Principal Investigator&#39;) There is also a helper function eml_nsf_to_project() that can help do the searching for you. Just verify that the information retrieved is correct. # update NSF awards data awards &lt;- c(&quot;1311655&quot;, &quot;1417987&quot;, &quot;1417993&quot;) # list of award numbers proj &lt;- eml_nsf_to_project(awards) #helper function doc$dataset$project &lt;- proj On the web form right now only fills in the funding field. We need instructions on how to convert that to the new awards formatting until the form is updated to include these fields: project$award$funderName # required project$award$title # required project$award$awardNumber project$award$funderIdentifier Information can be found in using the Open Funder Registry. If a award title cannot be found you can use the dataset title. "],["use-references.html", "Use references", " Use references References are a way to avoid repeating the same information multiple times in the same EML record. There are a few benefits to doing this, including: Making it clear that two things are the same (e.g., the creator is the same person as the contact, two entities have the exact same attributes) Reducing the size on disk of EML records with highly redundant information Faster read/write/validate with the R EML package You may want to use EML references if you have the following scenarios (not exhaustive): One person has multiple roles in the dataset (creator, contact, etc) One or more entities shares all or some attributes Example with parties Do not use references for creators as it is used for the citation information. The creators will not show up on the top of the dataset if it is a reference. Until this issue is resolved in NCEAS/metacat#926 we will need to keep this in account. It’s very common to see the contact and creator referring to the same person with XML like this: &lt;eml packageId=&quot;my_test_doc&quot; system=&quot;my_system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1 eml.xsd&quot;&gt; &lt;dataset&gt; &lt;creator&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml&gt; So you see those two times Bryce Mecum is referenced there? If you mean to state that Bryce Mecum is the creator and contact for the dataset, this is a good start. But with just a name, there’s some ambiguity as to whether the creator and contact are truly the same person. Using references, we can remove all doubt. doc$dataset$creator[[1]]$id &lt;- &quot;reference_id&quot; doc$dataset$contact &lt;- list(references = &quot;reference_id&quot;) print(doc) &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; &lt;eml:eml xmlns:eml=&quot;eml://ecoinformatics.org/eml-2.1.1&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns:stmml=&quot;http://www.xml-cml.org/schema/stmml-1.1&quot; packageId=&quot;id&quot; system=&quot;system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1/ eml.xsd&quot;&gt; &lt;dataset&gt; &lt;title&gt;A Minimal Valid EML Dataset&lt;/title&gt; &lt;creator id=&quot;reference_id&quot;&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;references&gt;reference_id&lt;/references&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml:eml&gt; The reference id needs to be unique within the EML record but doesn’t need to have meaning outside of that. Example with attributes To use references with attributes: Add an attribute list to a data table Add a reference id for that attribute list Use references to add that information into the attributeLists of the other data tables For example, if all the data tables in our data package have the same attributes, we can set the attribute list for the first one, and use references for the rest: doc$dataset$dataTable[[1]]$attributeList &lt;- attribute_list doc$dataset$dataTable[[1]]$attributeList$id &lt;- &quot;shared_attributes&quot; # use any unique name for your id for (i in 2:length(doc$dataset$dataTable)) { doc$dataset$dataTable[[i]]$attributeList &lt;- list(references = &quot;shared_attributes&quot;) # use the id you set above } "],["pi-correspondence.html", "PI correspondence ", " PI correspondence "],["email-templates.html", "Email templates", " Email templates This section covers new data packages submitted. For other inquiries see the PI FAQ templates Please think critically when using these canned replies rather than just blindly sending them. Typically, content should be adjusted/ customized for each response to be as relevant, complete, and precise as possible. In your first few months, please run email drafts by the #datateam Slack and get approval before sending. Remember to consult the submission guidelines for details of what is expected. Quick reference: Initial email template Final email templates Additional email template Initial email template Hello [NAME OF REQUESTOR], Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: [COMMENTS HERE] After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, [YOUR NAME] Comment templates based on what is missing Portals Multiple datasets under the same project - suggest data portal feature I would like to highlight the Data Portals feature that would enhance the ability of your datasets to be discovered together. It also enables some custom branding for the group or project. Here is an example page that we created for the Distributed Biological Observatory: https://arcticdata.io/catalog/portals/DBO. We highly suggest considering creating a portal for your datasets, you can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/. Data portals can be set up at any point of your project. If they ask to nest the dataset We are no longer supporting new nested datasets. We recommend to create a data portal instead. Portals will allow more control and has the same functionality as a nested dataset. You can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/, but as always we are here to help over email. Dataset citations If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be done at any point. Title Provides the what, where, and when of the data We would like to add some more context to your data package title. I would like to suggest: ‘OUR SUGGESTION HERE, WHERE, WHEN’. Does not use acronyms We wanted to clarify a couple of abbreviations. Could you help us in defining some of these: [LIST OF ACRONYMS TO DEFINE HERE] Abstract Describes DATA in package (ideally &gt; 100 words) We would like to add some additional context to your abstract. We hope to add the following: [ADJUST THE DEPENDING ON WHAT IS MISSING] The motivation of the study Where and when the research took place At least one sentence summarizing general methodologies All acronyms are defined At least 100 words long Offer this if submitter is reluctant to change: If you prefer and it is appropriate, we could add language from the abstract in the NSF Award found here: [NSF AWARD URL]. Keywords We noticed that there were no keywords for this dataset. Adding keywords will help your dataset be discovered by others. Data Sensitive Data We will need to ask these questions manually until the fields are added to the webform. Data sensitivity categories Once we have the ontology this question can be asked: Based on our Data sensitivity categories, which of the 3 does your dataset align with most: Non-sensitive data - None of the data includes sensitive or protected information. Some or all data is sensitive with minimal risk - Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Some or all data is sensitive with significant risk - The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Ethical research proceedures We were wondering if you could also address this question specifically on Ethical Research Procedures: Describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Insitutional Review Board approvals, consent waivers, procedures for co-production, data sovreignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. We can help add your answers to the question to the metadata. Adding provenance Is the [mention the file names here] files related? If so we can add provenance to show the relationship between the files. Here is an example of how that is displayed: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Af00c4d71-0242-4e9d-9745-8999776fa2f2 At least one data file We noticed that no data files were submitted. With the exception of sensitive social science data, we seek to include all data products prior to publication. We wanted to check if additional files will be submitted before we move forward with the submission process. Open formats Example using xlsx. Tailor this reponse to the format in question. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. The data files can be replaced by going to the green Edit button &gt; Click the black triangle by the Describe button for the data file &gt; Select Replace (attached is also a screenshot on how to get there). Zip files Except for very specific file types, we do not recommend that data are archived in zip format. Data that are zipped together can present a barrier to reuse since it is difficult to accurately document each file within the zip file in a machine readable way. File contents and relationships among files are clear Could you provide a short description of the files submitted? Information about how each file was generated (what software, source files, etc.) will help us create more robust metadata for long term use. Data layout Would you be able to clarify how the data in your files is laid out? Specifically, what do the rows and columns represent? We try not to prescribe a way the researchers must format their data as long as reasonable. However, in extreme cases (for example Excel spreadsheets with data and charts all in one sheet) we will want to kindly ask them to reformat. We would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. [DESCRIBE WHAT MAY NEED TO BE CHANGED IN THE DATA SET]. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. Attributes Identify which attributes need additional information. If they are common attributes like date and time we do not need further clarification. Checklist for the datateam in reviewing attributes (NetCDF, CSV, shapefiles, or any other tabular datasets): A name (often the column or row header in the file). A complete definition. Any missing value codes along with explanations for those codes. For all numeric data, unit information is needed. For all date-time data, a date-time format is needed (e.g. “DD-MM-YYYY”). For text data, full descriptions for all patterns or code/definition pairs are needed if the text is constrained to a list of patterns or codes. Helpful templates: &gt; We would like your help in defining some of the attributes. Could you write a short description or units for the attributes listed? [Provide a the attribute names in list form] &gt; Could you describe ____? &gt; Please define “XYZ”, including the unit of measure. &gt; What are the units of measurement for the columns labeled “ABC” and “XYZ”? Missing value codes What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. This section is not yet available on our webform so we will add that information on your behalf. We noticed that the data files contain [blank cells - replace with missing values found]. What do these represent? Funding All NSF funded datasets need a funding number. Non-NSF funded datasets might not have funding numbers, depending on the funding organization. We noticed that your dataset does not appear to contain a funding number. The field accepts NSF funding numbers as well as other numbers by different organizations. Methods We noticed that methods were missing from the submission. Submissions should include the following: provide instrument names (if applicable) specify how sampling locations were chosen if citations for sampling methods are used, please provide a brief summary of the methods referenced any software used to process the data Note - this includes software submissions as well (see https://arcticdata.io/submit/#metadata-guidelines-for-software) Your methods section appears to be missing some information. [ADJUST THIS DEPENDING ON WHAT IS MISSING - Users should be able to understand how the data were collected, how to interpret the values, and potentially how to use the data in the case of specialized files.] Comprehensive methods information should be included directly in the metadata record. Pointers or URLs to other sites are unstable. A full example - New Submission: methods, excel to csv, and attributes Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be added at any point. We would like to add some more context to your data package title. I would like to suggest: Holocene lake-based Arctic glacier and ice cap records. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. We also would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. If the data was in a long rather than wide format, it would be easier to us to document. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. We noticed that methods were missing from the submission. Submissions should: - provide instrument names (if applicable) - specify how sampling locations were chosen - provide citations for sampling methods that are not explained in detail - any software used to process the data After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, Name Final email templates Asking for approval Hi [submitter], I have updated your data package and you can view it here after logging in: [URL] Please review and approve it for publishing or let us know if you would like anything else changed. For your convenience, if we do not hear from you within a week we will proceed with publishing with a DOI. After publishing with a DOI, any further changes to the dataset will result in a new DOI. However, any previous DOIs will still resolve and point the user to the newest version. Please let us know if you have any questions. DOI and data package finalization comments Replying to questions about DOIs We attribute DOIs to data packages as one might give a DOI to a citable publication. Thus, a DOI is permanently associated with a unique and immutable version of a data package. If the data package changes, a new DOI will be created and the old DOI will be preserved with the original version. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to that latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Clarification of updating with a DOI and version control We definitely support updating a data package that has already been assigned a DOI, but when we do so we mark it as a new revision that replaces the original and give it its own DOI. We do that so that any citations of the original version of the data package remain valid (i.e.: after the update, people still know exactly which data were used in the work citing it). Resolve the ticket Sending finalized URL and dataset citation before resolving ticket [NOTE: the URL format is very specific here, please try to follow it exactly (but substitute in the actual DOI of interest)] Here is the link and citation to your finalized data package: https://doi.org/10.18739/A20X0X First Last, et al. 2021. Title. Arctic Data Center. doi:10.18739/A20X0X. If in the future there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. Please let us know if you need any further assistance. Additional email templates Deadlines If the PI is checking about dates/timing: &gt; [give rough estimate of time it might take] &gt; Are you facing any deadlines? If so, we may be able to expedite publication of your submission. Pre-assigned DOI If the PI needs a DOI right away: We can provide you with a pre-assigned DOI that you can reference in your paper, as long as your submission is not facing a deadline from NSF for your final report. However, please note that it will not become active until after we have finished processing your submission and the package is published. Once you have your dataset published, we would appreciate it if you could register the DOI of your published paper with us by using the citations button beside the orange lock icon. We are working to build our catalog of dataset citations in the Arctic Data Center. Sensitive Data Which of the following categories best describes the level of sensitivity of your data? A. Non-sensitive data None of the data includes sensitive or protected information. Proceed with uploading data. B. Some or all data is sensitive but has been made safe for open distribution Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Proceed with uploading data, but ensure that only data that are safe for public distribution are uploaded. Address questions about anonymization, aggregation, de-identification, and data embargoes with the data curation support team before uploading data. Describe these approaches in the Methods section. C. Some or all data is sensitive and should not be distributed The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Do NOT upload sensitive data. You should still upload a metadata description of your dataset that omits all sensitive information to inform the community of the dataset’s existence. Contact the data curation support team about possible alternative approaches to safely preserve sensitive or protected data. Ethical Research Procedures. Please describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Institutional Review Board approvals, consent waivers, procedures for co-production, data sovereignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. Asking for dataset access As a security measure we ask that we get the approval from the original submitter of the dataset prior to granting edit permissions to all datasets. No response from the researcher Please email them before resolving a ticket like this: We are resolving this ticket for bookkeeping purposes, if you would like to follow up please feel free to respond to this email. Recovering Dataset submissions To recover dataset submissions that were not successful please do the following: Go to https://arcticdata.io/catalog/drafts Find your dataset and download the corresponding file Send us the file in an email Custom Search Link You could also use a permalink like this to direct users to the datasets: https://arcticdata.io/catalog/data/query=\"your search query here” for example: https://arcticdata.io/catalog/data/query=Beaufort%20Lagoon%20Ecosystems%20LTER Adding metadata via R KNB does not support direct uploading of EML metadata files through the website (we have a webform that creates metadata), but you can upload your data and metadata through R. Here are some training materials we have that use both the EML and datapack packages. It explains how to set your authentication token, build a package from metadata and data files, and publish the package to one of our test sites. I definitely recommend practicing on a test site prior to publishing to the production site your first time through. You can point to the KNB test node (dev.nceas.ucsb.edu) using this command: d1c &lt;- D1Client(\"STAGING2\", \"urn:node:mnTestKNB\") If you prefer, there are Java, Python, MATLAB, and Bash/cURL clients as well. Finding multiple data packages If linking to multiple data packages, you can send a link to the profile associated with the submitter’s ORCID iD and it will display all their data packages. e.g.: https://arcticdata.io/catalog/profile/http://orcid.org/0000-0002-2604-4533 NSF ARC data submission policy Please find an overview of our submission guidelines here: https://arcticdata.io/submit/, and NSF Office of Polar Programs policy information here: https://www.nsf.gov/pubs/2016/nsf16055/nsf16055.jsp. Investigators should upload their data to the Arctic Data Center (https://arcticdata.io), or, where appropriate, to another community endorsed data archive that ensures the longevity, interpretation, public accessibility, and preservation of the data (e.g., GenBank, NCEI). Local and university web pages generally are not sufficient as an archive. Data preservation should be part of the institutional mission and data must remain accessible even if funding for the archive wanes (i.e., succession plans are in place). We would be happy to discuss the suitability of various archival locations with you further. In order to provide a central location for discovery of ARC-funded data, a metadata record must always be uploaded to the Arctic Data Center even when another community archive is used. Linking ORCiD and LDAP accounts First create an account at orcid.org/register if you have not already. After that account registration is complete, login to the KNB with your ORCID iD here: https://knb.ecoinformatics.org/#share. Next, hover over the icon on the top right and choose “My Profile”. Then, click the “Settings” tab and scroll down to “Add Another Account”. Enter your name or username from your Morpho account and select yourself (your name should populate as an option). Click the “+”. You will then need to log out of knb.ecoinformatics.org and then log back in with your old LDAP account (click “have an existing account”, and enter your Morpho credentials with the organization set to “unaffiliated”) to finalize the linkage between the two accounts. Navigate to “My Profile” and “Settings” to confirm the linkage. After completing this, all of your previously submitted data pacakges should show up on your KNB “My Profile” page, whether you are logged in using your ORCiD or Morpho account, and you will be able to submit data either using Morpho or our web interface. Or, try reversing my instructions - log in first using your Morpho account (by clicking the “existing account” button and selecting organization “unaffiliated”), look for your ORCiD account, then log out and back in with ORCiD to confirm the linkage. "],["pi-faq-email-templates.html", "PI FAQ email templates", " PI FAQ email templates Data Q: I would like to / have already uploaded data to ____ repository Review the repository, does it issue persistent identifiers (ie. DOIs)? Do they say how long they will be commiting to storing the data? If the answer is no to either questions, we will need to document and store any files in our own system. If any of this is unclear from what you can find online, you can message Matt to discuss with NSF if a particular repository will be allowed. Possible response for a generalist repository If your data doesn’t require features that only ____ provides, we would encourage you to consider publishing both the data and the metadata at the Arctic Data Center. Some benefits to submitting to us includes: We provide a DOI that can be cited in your published papers We track where datasets are being cited By submitting both the data and metadata to us, you can fulfill your NSF reporting requirements at the same time. This will reduce the work needed on your part to get any additional metadata requirements for NSF. Q: Can I replace data that have already been uploaded and keep the DOI? A: Once you have published your data with the Arctic Data Center, it can still be updated by providing an additional version which can replace the original, while still preserving the original and making it available to anyone who might have cited it. To update your data, return to the data submission tool used to submit it, and provide an update. Any update to a data package qualifies as a new version and therefore requires a new DOI. This is because each DOI represents a unique, immutable version, just like for a journal article. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to the latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Datateam: please review this dataset again to make sure it follows current standards Q: Why don’t I see my data package on the ADC? Possible Answer #1: The data package is still private because we are processing it or awaiting your approval to publish it. Please login with your ORCID iD to view private data packages. Possible Answer #2: The data package is still private and you do not have access because you were not the submitter. If you need access please have the submitter send us a message from his/her email address confirming this, along with your ORCID iD. Once we receive that confirmation we will be happy to grant you permission to view and edit the data package. Possible Answer #3: The data package is still private and we accidentally failed to grant you access. We apologize for the mistake. We have since updated the access policy. Please let us know if you are still having trouble viewing the data package here: [URL]. Remember to login with your ORCID iD. Possible Answer #4: The URL was garbled by their email server (eg. URL Defense or the url was split into two lines). You can navigate to your dataset by going to your name in the top right and selecting my datasets. Issue: MANY files to upload (100s or 1000s) or large cumulative size. A: Datateam - Prior to accepting large uploads, take the time to make sure there is enough space available and uploading all the files is necessary. If it is not a NSF dataset see the section on Scope. We have a secure FTP you can access. Details are available here: https://help.nceas.ucsb.edu/remote_file_access . Please access our server at datateam.nceas.ucsb.edu with the username “visitor”. Let us know if you would like to use our SFTP and we will send you the password and the path to which directory to upload to. If you have files to transfer to us that total several terabytes it may be best to arrange a shipment of an external hard drive. Q: Can I add another data file to an existing submission without having to fill out another metadata form? A: Yes. Navigate to the data package after being sure to login. Then click the green “Edit” button. The form will populate with the already existing metadata so there is no need to fill it out again. Click “Add Files” and browse to the file you wish to add. Be aware that the DOI will change after you add this file (or make any changes to a data package) as, just like for a journal article, a DOI represents a unique and immutable version. The original URL and DOI will remain functional and valid, but clearly display a message at the top of that page stating that “A newer version of this dataset exists” with a link to the latest version. Only the newest version wil be discoverable via a search. We can also copy the metadata from existing dataset. Please send us the URL of the dataset you would like to copy and the title of the new dataset and we will be able to do that on your behalf. Q: I want to organize the order of the files/metadata entities A: We can reorganize entities within their entities. This is a limitation with our tools (EML). While re-ordering the groups of entities is possible manually, it isn’t possible the way write_eml and publish_update is written. We do not recommended reordering entities this way. Q: Can we submit data as an Excel file? A: While the Arctic Data Center supports the upload of any data file format, sharing data can be greatly enhanced if you use ubiquitous, easy-to-read formats. For instance, while Microsoft Excel files are commonplace, it is better to export these spreadsheets to Comma Separated Values (CSV) text files, which can be read on any computer without needing to have Microsoft products installed. So, yes, you are free to submit an Excel workbook, however we strongly recommend converting each sheet to a CSV. The goal is not only for users to be able to read data files, but to be able to analyze them with software, such as R Studio. Typically, we would extract any plots and include them as separate image files. [ONLY SAY THIS NEXT PART IF THE REQUESTOR CONTINUES TO INSIST and then USE PROV TO POINT FROM THE XLS TO THE CSVs] I understand that having the plots in the same file as the data they are built from simplifies organization. If you definitely prefer to have the Excel workbook included, we ask that you allow us to document the data in both formats and include a note in the metadata clarifying that the data are indeed duplicated (but in different formats). Q: Can we submit model output? A: Yes you can submit your model output to the Arctic Data Center. We also recommend including model source code directly in the dataset as it is critical to understand the model output dataset and will allow other researchers to re-produce your output. If the model is complex or it’s important to maintain file structure then it can submitted as a zip file. We also recommend included a detailed readme file that documents: version information, licensing information, a list of software/hardware used in development, a list of software/hardware dependencies needed to run the software, information detailing source data for models, any mathematical/physical explanations needed to understand models, any methods used to evaluate models, and instructions how to run the model if not including in the source code as comments. Please read our software submission guidelines for more detailed information: https://arcticdata.io/submit/#metadata-guidelines-for-software Access Q: May another person (e.g. my student) submit data using my ORCID iD so that it is linked to me? A: We recommend instead that the student set up their own ORCiD account at https://ORCiD.org/register and submit data packages from that account. Submissions are processed by our team and, at that point, we can grant you full rights to the metadata and data files even though another person submitted them. Issue: Web form not cooperating. A: To help us diagnose the problem, could you let us know the following: Which operating system (including the version) and browser (with version #) are you using? At which exact step did the issue arise? What error message did you receive? Do you have any reason to believe that you may be using a slow internet connection? Please provide us with any relevant screenshots and we will troubleshoot from there. Issue: Can’t log in to the Arctic Data Center A: Please accept our apologies that you are experiencing difficulty logging in. We suggest trying these troubleshooting steps: Try a different internet browser (Chrome, Firefox, Internet Explorer etc.) Try clearing your cache and then re-logging in with orcid Id Privacy blockers (like EFF’s privacy badger https://www.eff.org/privacybadger) block the cookies needed for the login to work. To enable the cookies you need to disable your privacy tracker for arcticdata.io. If the problem persists please let us know what step the error occured or if there were error messages you have recieved. Scope Q: Can I submit a non-NSF funded dataset? Sample text 1: The Arctic Data Center is open to all arctic-related data. For larger data packages, we would likely need to charge a one-time archival fee which amortizes the long-term costs of preservation in a single payment. Please let us know the size of your dataset in terms of total number and total size of all files. We look forward to receiving your submission! Sample text 2: Yes, you can submit non-NSF-funded Arctic data if you are willing to submit under the licensing terms of the Arctic Data Center (CC-0 or CC-BY), the data are moderately sized (with exact limits open to discussion), and a lot of support time to curate the submission is not required (i.e., you submit a complete metadata record and well formatted, open format data files). For larger data packages, we would likely need to charge a one-time archival fee which amortizes the long-term costs of preservation in a single payment. Also, please note that NSF-funded projects take priority when it comes to processing. Information on best practices for data and metadata organization is available here: https://arcticdata.io/submit/#organizing-your-data. Q: Is ______ considered within the arctic We don’t have a precise geographic cutoff. We definitely will preserve any dataset that is funded by the Arctic section of the National Science Foundation, which includes many sub-Arctic studies. So, as long as the dataset is related to Arctic research defined broadly, we would consider it. We will prioritize NSF-funded projects because that is our main mandate, but we will also preserve non-NSF data about the Arctic as time allows. If you send more details about the study, its subject, location, and funding, we could be more definitive about whether we could archive it. "],["good-examples-of-data-packages.html", "Good Examples of data packages", " Good Examples of data packages Modeling Model with output: https://doi.org/10.18739/A24J09X55 Model code archived as a zip if preverving directory structure is important https://doi.org/10.18739/A2JS9H795 Model with provenance: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Ae6390181-85e9-46a2-b2ba-e352ece51cc6 Code and output: Note: We can only archive the code if the submitter has the rights to redistribute it (like if they wrote it themselves, or if the code has a license that allows for that) https://doi.org/10.18739/A2XS5JH4N Guidelines for large outputs: https://arcticdata.io/submit/#guidelines-for-large-models Spatial data Vector https://doi.org/10.18739/A2TB0XV89 Raster https://doi.org/10.18739/A2GT5FG0B Portals https://arcticdata.io/catalog/portals/DBO https://arcticdata.io/catalog/portals/CALM "],["final-checklist.html", "Final Checklist", " Final Checklist You can click on the assessment report on the website to for a general check. Fix anything you see there. Send the link over slack for peer review by your fellow datateam members. Usually we look for the following (the list is not exhaustive): Special Datasets Please refer to the dedicated pages for instructions to handle these cases: MOSAiC DBO System Metadata the format ids are correct General EML Ethical Research Practice section is completed Publisher and system information have been added: doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) Title Include geographic and temporal coverage (when and where data was collected) Abbreviations are removed or defined Abstract longer than 100 words Abbreviations are removed or defined. No garbled text Tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting DataTable / OtherEntity / SpatialVectors In the correct one: DataTable / OtherEntity / SpatialVector / SpatialRaster for the file type entityDescription - longer than 5 words and unique physical present and up-to-date and format is correct Attribute Table Complete attributeDefinitions longer than 3 words Variables match what is in the file Measurement domain - if appropirate (ie dateTime correct) Missing Value Code - accounted for if applicable Semantic Annotation - appropriate semantic annotations added, especially for spatial and temporal variables: lat, lon, date etc. Custom units were created if necessary People Complete information for each person in each section includes ORCID and e-mail address for all contacts people repeated across sections have consistent information Geographic region the map looks correct and matches the geographic description check if negatives (-) are missing Project If it is an NSF award you can use the helper function: doc$dataset$project &lt;- eml_nsf_to_project(awards) for other awards that need to be set manually, see the set project page Methods Present No garbled text Acronyms and units are defined the first time they are used Check EML Version Currently using: eml-2.2.0 (as of July 30 2020) Review to see if the EML version is set correctly by reviewing the doc$`@context` that it is indeed 2.2.0 under eml Re-run your code again and have the lineemld::eml_version(\"eml-2.2.0\") at the top Make sure the system metadata is also 2.2.0 Access If necessary, granted access to PI using set_rights_and_access() make sure it is http:// (no s) note if it is a part of portals there might be specific access requirements for it to be visible using set_access() SFTP Files If there are files transferred to us via SFTP, delete those files once they have been added to the dataset and when the ticket is resolved Updated datasets All the above applies. These are some areas to do a closer check when users update with a new file: New data was added Temporal Coverage and Title Confirm if new data used same methods or if methods need updating Files were replaced update physical and entityName double-check attributes are the same check for any new missing value codes that should be accounted for Was the dataset published before 2021? update project info, annotations Glance over entire page for any small mistakes (ie. repeated additionalMetadata, any missed &amp;amps, typos) "],["initial-review-checklist.html", "Initial review checklist", " Initial review checklist Before responding to a new submission use this checklist to review the submission. When your are ready to respond use the initial email template and insert comments and modify as needed. Sensitive Data If any of the below is in the dataset, please alert the #arctica team know before proceeding. Check if there is any sensitive information or personal identifying information in the data (eg. Names) Can the data be disaggregated and de-anonymized? (eg. a small sample size and individuals could be easily identified by their answers) Dryad Human Subject data guidelines can be a good place to start Common Cases: Social Science: Any dataset involving human subjects (may include awards awarded by ASSP and topics such as COVID-19) Archaeology: archaeological site location information, which is protected from public access by law Biology: protected species location coordinates Data citations If the dataset appears to be in a publication please (might be in the abstract) make sure that those citations are registered. Title WHAT, WHERE, and WHEN: Is descriptive of the work (provides enough information to understand the contents at a general scientific level), AND includes temporal coverage Provides a location of the work from the local to state or country level Provides a time frame of the work NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out Abstract Describes the DATA as well as: The motivation (purpose) of the study Where and when the research took place At least one sentence summarizing general methodologies NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out At least 100 words total tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting Any citations to papers can be registered with us Keywords Some keywords are included Data Data is normalized (if not suggest to convert the data if possible) At least one data file, or an identifier to the files at another approved archive, unless funded by ASSP (Arctic Social Sciences Program) No xls/xlsx files (or other proprietary files) File contents and relationships among files are clear Each file is well NAMED and DESCRIBED and clearly differentiated from all others All attributes in EML match attribute names in respective data files EXACTLY, are clearly defined, have appropriate units, and are in the same order as in the file. Quality control all dimensionless units. Missing value codes are explained (WHY are the data absent?) If it is a .rar file -&gt; scan the file If there is the unit tons make sure to ask if it is metric tons or imperical tons if not clarified already People &amp; Parties At least one contact and one creator with a name, email address, and ORCID iD Coverages Includes coverages that make sense Temporal coverage - Start date BEFORE end date Geologic time scales are added if mentioned in metadata (e.g. 8 Million Years or a name of a time period like Jurassic) Spatial coverage matches geographic description (check hemispheres) Geographic description is from the local to state or country level, at the least Taxonomic coverage if appropriate Project Information At least one FUNDING number Title, personnel, and abstract match information from the AWARD (not from the data package) Methods This section is REQUIRED for ALL NSF-FUNDED data packages Enough detail is provided such that a reasonable scientist could interpret the study and data for reuse without needing to consult the researchers, nor any other resources Portals If there are multiple submissions from the same people/project let them know about the portals feature If this is part of a portal make sure this dataset can be found there. Additional steps might be needed to get that to work. Please consult Jeanette and see the data portals section. "],["large-file-transfer.html", "Large file transfer", " Large file transfer Often if the researcher has many files to upload or the files are large in size, we will need to provide alternative methods of upload. We currently have two options available for large file transfer where sFTP is easier to set up but slower while GLOBUS takes a bit more setup but faster. Before providing access, try to get a sense of how big the transfer will be. df -h on Arctic server and datateam server - look at current space available sFTP As a team member for login info. After you login, you can add a folder with your last name to add your files. If you need more detailed access instructions, I have included the link here for your convenience: https://help.nceas.ucsb.edu/NCEAS/help/remote_file_access Option 1: SSH via the Terminal https://pages.github.nceas.ucsb.edu/NCEAS/help/connecting_to_linux_using_ssh.html Option 2: Cyberduck instructions To use Cyberduck to transfer local files onto the Datateam server: Open Cyberduck. Check that you have the latest version (Cyberduck -&gt; Check for Update…). If not, download and install the latest (you may need Jeanette to enter a password). Click “Open Connection”. From the drop-down, choose “SFTP (Secure File Transfer Protocol)”. Enter “datateam.nceas.ucsb.edu” for Server. Enter your username and password. Connect. From here, you can drag and drop files to and from the server. Globus Endpoint arctic-data-center #arctic-data-center login using your credentials on the datateam server Remember to remove the files on the datateam server after you finish a ticket to free up space. "],["navigate-rt.html", "Navigate RT", " Navigate RT The RT ticketing system is how we communicate with folks interacting with the Arctic Data Center. We use it for managing submissions, accessing issues, etc. It consists of three separate interfaces: Front Page All Tickets Ticket Page Front page This is what you see first Home - brings you to this homepage Tickets - to search for tickets (also see number 5) Tools - not needed New Ticket - create a new ticket Search - Type in the ticket number to quickly navigate to a ticket Queue - Lists all of the tickets currently in a particular queue (such as ‘arcticdata’) and their statuses New = unopened tickets that require attention Open = tickets currently open and under investigation and/or being processed by a support team member Stalled = tickets awaiting responses from the PI/ submitter Tickets I Own - These are the current open tickets that are claimed by me Unowned Tickets - Newest tickets awaiting claim Ticket Status - Status and how long ago it was created Take - claim the ticket as yours All tickets This is the queue interface from number 6 of the Front page 1. Ticket number and title 2. Ticket status 3. Owner - who has claimed the ticket Example ticket Title - Include the PI’s name for reference Display - homepage of the ticket History - Comment/Email history, see bottom of Display page Basics - edit the title, status, and ownership here People - option to add more people to the watch list for a given ticket conversation. Note that user/ PI/ submitter email addresses should be listed as “Requestors”. Requestors are only emailed on “Replys”, not “Comments”. Ensure your ticket has a Requestor before attempting to contact users/ PIs/ submitters Links - option to “Merge into” another ticket number if this is part of a larger conversation. Also option to add a reference to another ticket number Verify that this is indeed the two tickets you want to merge. It is non-reversible. Actions Reply - message the submitter/ PI/ all watchers Comment - attach internal message (no submitters, only Data Teamers) Open It - Open the ticket Stall - submitter has not responded in greater than 1 month Resolve - ticket completed History - message history and option to reply (to submitter and beyond) or comment (internal message) New data submission When notified by Arcticbot about a new data submission, here are the typical steps: Update the Requestor under the People section based on the email given in the submission (usually the user/ PI/ submitter). You may have to google for the e-mail address if the PI did not include it in the metadata record. Take the ticket (Actions &gt; Take) Review the submission based on the checklist Draft an email using the template and let others review it via Slack Send your reply via Actions "],["replicate-datasets.html", "Replicate Datasets", " Replicate Datasets Sometimes we get requests to have their datasets replicated to ADC. Usually it is a dataset on EDI where you can ge the Digital Object Identifier: which serves as the resource map on DataOne If a link isn’t given, you can use the package ID (i.e. knb-lter-arc.20129.1) and add it to the end of this link: https://portal.edirepository.org/nis/mapbrowse?packageid= If we are unsure about the identifer you can try querying for it on the CN: cn &lt;- CNode(&#39;PROD&#39;) #find the DOIs result &lt;- query(cn, list(q = paste0(&quot;(id:*10.6073/pasta/d4f567844673857239eec0cb61c6f543&quot;,&quot;* *:* NOT obsoletedBy:*)&quot;), fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded, authoritativeMN, replicaMN&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) Cloning the packages Try cloning to the test node first to see how it will look. Set up your nodes from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:LTER&quot;) to &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) For LTER datasets on EDI, there are two possibilities for the mn: “urn:node:LTER” and “urn:node:EDI”. Try the other one if one isn’t working Use clone_package to copy it over to the test node clone_package(&quot;doi:10.6073/pasta/d4f567844673857239eec0cb61c6f543&quot;, #example doi to replicate from = from, to = to, add_access_to = &quot;http://orcid.org/0000-0001-8888-547X&quot;, change_auth_node = F, new_pid = T) Once you have verified that it clones properly to the test node, do the same in production but with a couple of minor modifications: from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:LTER&quot;) to &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) clone_package(&quot;doi:10.6073/pasta/0af82d3c3d9d1710775cf9b1464ce70b&quot;, from = from, to = to, add_access_to = &quot;http://orcid.org/0000-0001-8888-547X&quot;, change_auth_node = F, new_pid = F) #uses the same pid make sure the new_pid argument is set to F when you publish to production "],["miscellaneous-file-types.html", "Miscellaneous file types ", " Miscellaneous file types "],["convert-excel-file-to-csv.html", "Convert Excel file to CSV", " Convert Excel file to CSV This uses the workflow in here: https://readxl.tidyverse.org/articles/articles/readxl-workflows.html This is preferred to converting Excel files by hand to avoid human error. If files are in the visitor folder you might need to be complete the following while logged into visitor Load the necessary packages library(magrittr) library(readxl) library(readr) Get a list of files that we want to convert xlsx &lt;- dir(&quot;directory/of/files/here&quot;, pattern = &quot;*xls&quot;, recursive = T, full.names = T) Create a function that iterates through the excel files and saves it as a csv in the same folder the excel file was in read_then_csv &lt;- function(sheet, path) { #remove the file extension pathbase &lt;- path %&gt;% tools::file_path_sans_ext() #change file extension to csv and save with the sheet name for(i in 1:length(sheet)){ path %&gt;% read_excel(sheet = i) %&gt;% write_csv(paste0(pathbase, &quot;_&quot;, sheet[i], &quot;.csv&quot;)) } } Get the sheet name(s) sheets &lt;- xlsx %&gt;% map(excel_sheets) %&gt;% set_names() Iterate over all the xlsx files for each of the sheets df &lt;- map2(sheets, xlsx, ~read_then_csv(.x, path = .y)) Check the files afterwards to make sure everything was converted properly csv &lt;- dir(&quot;directory/of/files/here&quot;, pattern = &quot;*csv&quot;, recursive = T, full.names = T) Check to see if there are any empty tables and get a list of files that were empty files &lt;- map(csv, read_csv) n &lt;- which(map(files, nrow) == 0) Remove those empty table files lapply(csv[n], file.remove) Remove xlsx files if everything looks in order lapply(xlsx, file.remove) "],["datalogger-files.html", "Datalogger files", " Datalogger files .hobo files are raw data files offloaded from a sensor, and can include real-time graphs of measured variables, such as temperature and soil moisture conditions. .hobo files can be opened using HOBOware, which can be downloaded from here. "],["netcdfs.html", "NetCDFs", " NetCDFs This section is for dealing with NetCDF (.nc) files. These files require data tables but since they can not be simply opened on the computer using a text editor or Excel, you can use Panoply to explore them or the R commands. However, notice that not all NetCDF files have the attribute table information. library(ncdf4) # gets attribute info atts &lt;- arcticdatautils::get_ncdf4_attributes(&#39;filepath&#39;) # preview of View(atts) atts[1:10,] The “long_name” can be used to build a definition. The “units” can be used for either units or the enumerated domain codes. The “_FillValue” is a missing value code it looks like. # returns the actual values for a specified attribute t &lt;- ncdf4::nc_open(filepath) test &lt;- ncdf4::ncvar_get(t, &#39;attributeName&#39;) # preview of View(test) test[1:10] The formatId in the sysmeta will most likely be netCDF-4, but could be netCDF-3. "],["reorganize-files-on-the-server.html", "Reorganize files on the server", " Reorganize files on the server Useful commands to use when reorganizing large datasets to be prepped to be uploaded using datapack This is run in the terminal Basics Moving around cd directory_name Go up one directory using .. cd .. Get the contents of the current directory ls Get the directory structure for this folder -d flag shows only the directories tree tree -d Rename file based on file path For example you have some files organized like this: ../../visitor/Lastname └── RU_ALN_TR1_FL007R │ └── rgb_images | └──image.png └── RU_ALN_TR1_FL008B └── micasense_preflight_calibration_images └──image.png We want to rename the files. Find all the files with the same file name find . -name &#39;image.png&#39; First extract the file name in parts ((.*)/(.*)/(.*)/(.*)) by using the. / as a separator. Each part is represented by $#. If you want the last part of the file name:$4. Rearrange this section as needed:$2/$3/$2-$3-$4 Set the flag as -n while testing to see the result of the renaming first before committing to it using -v rename -n -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; rename -v -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; Putting it together by taking the results from find and renaming those files | the pipe is like the %&gt;% in R find . -name &#39;image.png&#39; | rename -v -- &#39;s|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|&#39; Unzip files unzip &#39;*zip&#39; Zip shapefiles For all the files in a directory, create a folder and move those files for file in *; do dir=$(echo $file | cut -d. -f1); mkdir -p $dir; mv $file $dir; done Zip the folders for file in *;do zip -r $file $file; done Remove the directories after zipping -r means to recursively remove files rm -r */ "],["scan-rar-files.html", "Scan rar files", " Scan rar files This section is for rar files which support data compression and file scanning. Here is how to scan rar files: Download the rar file to datateam In the terminal run the following on your files: mkdir to-scan &amp;&amp; mv &lt;file.rar&gt; to-scan &amp;&amp; cd to-scan unrar e &lt;file.rar&gt; That should scan the compressed file and each individual file it contains: clamscan * Clean up by deleting the to-scan folder - ’rm -r to-scan`. "],["spatial-data.html", "Spatial data", " Spatial data Shapefiles and their associated files can be kept in a single grouping using *.zip (or another similar format). To extract metadata, simply run: library(sf) pid &lt;- &quot;pid of a zipped shapefile on the ADC&quot; spatial_obj &lt;- arcticdatautils::read_zip_shapefile(adc, pid) st_geometry(spatial_obj) This will return the shapefile as an sf style data.frame so that you can examine the attributes, and print the projection and geometry information. Note that the column geometry should not be considered an attribute for documentation purposes. This information is represented in the spatialVector$geometry element. "],["wrangle-data.html", "Wrangle data ", " Wrangle data "],["clean-column-names.html", "Clean column names", " Clean column names You might have read that column names should not include spaces or special characters. Inevitably, you’ll encounter data that are not so tidy. For example: ## First Name Last Name date of birth ## 1 Homer Simpson 1/1/1960 ## 2 Marge Simpson 10/20/1965 ## 3 Ned Flanders 3/22/1961 To tidy it up, you can use the clean_names() function from the janitor package: janitor::clean_names(df) "],["fix-excel-dates.html", "Fix Excel dates", " Fix Excel dates Do you see something that looks like 43134, 43135, 43136 even though the column header is Dates? You may have encountered an Excel date/time problem. To fix it, the janitor package has a handy function: janitor::excel_numeric_to_date(c(43134, 43135, 43136)) Make sure you check that the dates make sense! "],["solr-queries.html", "Solr queries", " Solr queries Solr is what’s known as an index, which helps us keep track of the data, metadata, and resource map objects stored in our metadata catalog, Metacat. Solr allows you to quickly search through any coordinating or member node. Some more common use-cases include: Recovering lost PIDs - if you publish new data objects (publish_object) but forget to save the PIDs, you can recover them with a query. For example, you can search for all data objects that are not associated with a resource map and are published using your ORCiD (q=-resourceMap:*+AND+submitter:*XXXX-YYYY-ZZZZ-WWWW) Working with groups of packages - if you want to find all data, metadata, and resource maps associated with a PI, you can use their last name (q=origin:*SURNAME*) or perhaps their ORCiD (q=rightsHolder:*XXXX-YYYY-ZZZZ-WWWW) or both (q=origin:*SURNAME*+OR+rightsHolder:*XXXX-YYYY-ZZZZ-WWWW) to retrieve information about packages associated with them Once you understand the logic of queries, it becomes a flexible and useful tool that you can integrate into your R workflow. You can use queries to answer a variety of interesting questions, for example: What are the most recently updated data sets? What metadata and data objects are in a given data package? What is the total size (in terms of disk space) of all objects stored in Metacat? Querying Solr is possible by adding a query onto the end of a base URL or through the dataone::query() function in R. For now, we’ll just cover the basics of Solr queries in R. "],["construct-a-query.html", "Construct a query", " Construct a query Each Solr query is comprised of a number of parameters. These are like arguments to a function in R, but they are entered as parts of a URL. The most common parameters are: q: The query. This is like subset() or dplyr::filter() in R. fl: What fields are returned for the documents that match your query (q). If not set, all fields are returned. rows: The maximum number of documents to return. Solr will truncate your result if the result size is greater than rows. sort: Sorts the result by the values in the given Solr field (e.g., sort by date uploaded). The query (q) parameter uses a syntax that looks like field:value, where field is one of the Solr fields and value is an expression. The expression can match a specific value exactly, e.g., q=identifier:arctic-data.7747.1 or q=identifier:\"doi:10.5065/D60P0X4S\", which finds the Solr document for a specific Object by PID (identifier). In the second example, the DOI PID is surrounded in double quotes. This is because Solr has reserved characters, of which : is one, so we have to help Solr by surrounding values with reserved characters in them in quotes or by escaping them. To view the list of query-able parameters on the Arctic Data Center and their descriptions, you can visit https://arcticdata.io/metacat/d1/mn/v2/query/solr. "],["query-solr-via-a-browser.html", "Query Solr via a browser", " Query Solr via a browser Solr is queried via what’s called an HTTP API (Application Program Interface). Practically, what this means it that you can execute a query in your browser by tacking a query onto a base URL. This is similar to the way Google handles your searches. If I search “soil science” in Google, for example, the URL becomes: https://www.google.com/search?q=soil+science&amp;oq=soil+science&amp;aqs=chrome.0.69i59.1350j0j1&amp;sourceid=chrome&amp;ie=UTF-8 If I break it down into pieces, I get: the base URL - https://www.google.com/search ?, after which the query parameters are listed the query - q=soil+science other parameters, which are separated by &amp; - oq=soil+science&amp;aqs=chrome.0.69i59.1350j0j1&amp;sourceid=chrome&amp;ie=UTF-8 Most of the time, you’ll query either the Arctic Data Center member node or the PROD coordinating node, which have the following base URLs: Arctic Data Center member node: https://arcticdata.io/metacat/d1/mn/v2/query/solr PROD coordinating node: https://cn.dataone.org/cn/v2/query/solr You can then append your query parameters to your base URL: https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q={QUERY}&amp;fl={FIELDS}&amp;rows={ROWS} Visit the base URL to see a list of fields Solr is storing for the objects it indexes. There is a large set of queryable fields, though not all types of objects will have values set for all of the possible fields because some fields do not make sense for some objects (e.g., title for a CSV). "],["query-solr-via-r.html", "Query Solr via R", " Query Solr via R Though it’s possible to query Solr directly through its HTTP API, we typically run our queries through R, for two main reasons: The result is returned in a more useful way to R without extra work on your part We can more easily pass our authentication token with the query Why does #2 matter? Well by default, all of those URLs above only return publicly-readable Solr documents. If a private document matches any of those queries, Solr won’t tell you that. It will act like the non-public-readable documents don’t exist. So we must pass an authentication token to access non-public-readable content. This bit is crucial for working with the ADC, so you’ll very often want to use R instead of visiting those URLs in a web browser. cn &lt;- CNode(&quot;PROD&quot;) adc &lt;- getMNode(cn, &quot;urn:node:ARCTIC&quot;) # Set your token if you need/want! #string your parameters together like this: dataone::query(mn, &quot;q=title:*soil*&amp;fl=title&amp;rows=10&quot;) #or alternatively, list them out: query(adc, list(q=&quot;title:*soil*&quot;, fl=&quot;title&quot;, rows=&quot;10&quot;)) By default, query returns the result as a list, but a data.frame can be a more useful way to work with the result. To get a data.frame instead, just set the as argument to ‘data.frame’ to get a data.frame: query(adc, list(q=&quot;title:*soil*&quot;, fl=&quot;title&quot;, rows=&quot;10&quot;), as = &quot;data.frame&quot;) "],["use-facets.html", "Use facets", " Use facets We can also summarize what’s in Solr with faceting, which lets us group Solr documents together and count them. This is like table in R. Faceting can do a query within a query, but more commonly it’s used to summarize unique values in a field. For example, we can find the unique formatIds on data objects: https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q=:&amp;fq=formatType:DATA&amp;facet=true&amp;facet.field=formatId&amp;rows=0 To facet, we usually do a few things: Add the parameter facet=true Add the parameter facet.field={FIELD} with the field we want to facet (group) on Set rows=0 because we don’t care about the matched Solr documents Optionally specify fq={expression} which filters out Solr documents before faceting. In the above example, we have to do this to count only data objects. Without it, the facet result would include formatIds for metadata and resource maps, which we don’t want. Currently, the dataone::query() function does not support faceting, so you’ll have to run your queries as a URL. For additional ways to use faceting (such as pivot faceting), check out the Solr documentation. "],["use-stats.html", "Use stats", " Use stats With stats, we can have Solr calculate statistics on numerical values (such as size). https://arcticdata.io/metacat/d1/mn/v2/query/solr/?q=formatType:DATA&amp;stats=true&amp;stats.field=size&amp;rows=0 This query calculates a set of summary statistics for the size field on data objects that Solr has indexed. In this case, Solr’s size field indexes the size field in the system metadata for each object in Metacat. "],["example-solr-queries.html", "Example Solr queries", " Example Solr queries For additional examples and explanations, check out the Apache Lucene Query Parser Syntax page. Find everything result &lt;- query(adc, list(q=&quot;*:*&quot;, fl=&quot;*&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) Query a wildcard expression #find any id that starts with arctic-data.6 result &lt;- query(adc, list(q=&quot;id:arctic-data.6*&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query multiple fields result &lt;- query(adc, list(q=&quot;title:soil*+AND+origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) result &lt;- query(adc, list(q=&quot;title:soil* OR origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) You can use either spaces or + to separate query parameters. When typing queries in R, it’s often easier to read if you use spaces. However, when using the browser, you may want to use + to keep the query clean. (The browser will replace spaces with %20.) Query by formatType Objects are queryable via their formatType which is one of DATA, METADATA, or RESOURCE. result &lt;- query(adc, list(q = &#39;formatType:RESOURCE AND submitter:&quot;http://orcid.org/0000-0002-2561-5840&quot;&#39;, fl = &#39;identifier,submitter,fileName&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;10&#39;), as = &quot;data.frame&quot;) result &lt;- query(adc, list(q = &#39;formatType:METADATA AND title:*Toolik*&#39;, fl = &#39;identifier,submitter,fileName&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;10&#39;), as = &quot;data.frame&quot;) Query pids by a specific submitter result &lt;- query(adc, list(q = &#39;submitter:&quot;http://orcid.org/0000-0003-4703-1974&quot;&#39;, fl = &#39;identifier,submitter,fileName, size&#39;, sort = &#39;dateUploaded+desc&#39;, rows=&#39;1000&#39;), as = &quot;data.frame&quot;) Query pids with special characters # Wrap the pid with special characters with escaped backslashes dataone::query(adc, list(q = paste0(&#39;id:&#39;, &#39;\\&quot;&#39;, &#39;doi:10.18739/A20R9M36V&#39;, &#39;\\&quot;&#39;), fl = &#39;dateUploaded AND identifier&#39;, rows = 5000), as = &quot;data.frame&quot;) Query multiple conditions within one field result &lt;- query(adc, list(q=&quot;title:(soil* AND carbo*)&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query for latest versions only result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId&quot;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) Use NOT in a query Just add - before a query parameter! result &lt;- query(adc, list(q=&quot;title:(soil AND -carbon)&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query a coordinating node result &lt;- query(cn, list(q=&quot;title:soil* AND origin:Ludwig&quot;, rows=&quot;5000&quot;), as = &quot;data.frame&quot;) Query for EMLs that document a specific data pid # Wrap the pid in escaped quotation marks if it contains special characters query(adc, list(q = paste0(&#39;documents:&#39;, &#39;\\&quot;&#39;, &#39;urn:uuid:f551460b-ce36-4dd3-aaa6-3a6c6e338ec9&#39;, &#39;\\&quot;&#39;), fl = &quot;identifier&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) # Alternatively we can use a wildcard expression in place of &quot;urn:uuid:&quot; # This way we don&#39;t need to escape the colon special characters query(adc, list(q = paste0(&quot;documents:*f551460b-ce36-4dd3-aaa6-3a6c6e338ec9&quot;), fl = &quot;identifier&quot;, rows=&quot;20&quot;), as = &quot;data.frame&quot;) Query for files uploaded during a specific time query(adc, list(q=&quot;dateUploaded:[2020-05-06T00:00:00Z TO NOW]&quot;, fl=&quot;title,identifier,resourceMap,dateUploaded,dateModified&quot;, sort = &quot;dateUploaded+desc&quot;, rows=&quot;200&quot;), as = &quot;data.frame&quot;) ## identifier ## 1 resource_map_doi:10.18739/A2R785R0C ## 2 doi:10.18739/A2R785R0C ## 3 resource_map_doi:10.18739/A2W08WJ3N ## 4 doi:10.18739/A2W08WJ3N ## 5 resource_map_doi:10.18739/A20P0WS61 ## 6 doi:10.18739/A20P0WS61 ## 7 resource_map_doi:10.18739/A24F1MK9P ## 8 doi:10.18739/A24F1MK9P ## 9 resource_map_doi:10.18739/A2862BD3F ## 10 doi:10.18739/A2862BD3F ## 11 urn:uuid:c3363bfa-2f6c-47c4-a8df-d533ffe1e9bb ## 12 resource_map_doi:10.18739/A2HQ3S124 ## 13 doi:10.18739/A2HQ3S124 ## 14 resource_map_urn:uuid:f0b520a4-f549-4bef-b2a7-eca9145c76d5 ## 15 urn:uuid:85869ebc-1139-49a9-a454-e020956b49d4 ## 16 resource_map_doi:10.18739/A2NG4GT6V ## 17 doi:10.18739/A2NG4GT6V ## 18 resource_map_doi:10.18739/A2S756M9H ## 19 doi:10.18739/A2S756M9H ## 20 resource_map_doi:10.18739/A2WW7720N ## 21 doi:10.18739/A2WW7720N ## 22 resource_map_urn:uuid:39f2123a-5c64-4e42-a2df-27f5777291f8 ## 23 urn:uuid:39f2123a-5c64-4e42-a2df-27f5777291f8 ## 24 urn:uuid:8529ef01-5ed3-4200-aa7a-0fb599be9126 ## 25 resource_map_doi:10.18739/A21N7XP4R ## 26 doi:10.18739/A21N7XP4R ## 27 urn:uuid:1d746797-58f7-4340-9879-b423d3765df3 ## 28 resource_map_urn:uuid:7590a8d7-8b3e-4b8f-a055-a08d10bc8d79 ## 29 urn:uuid:5a38c2f3-e3eb-4cfe-9806-a949f9063543 ## 30 urn:uuid:3b08037e-3daa-4d74-816f-1900792bcb48 ## 31 urn:uuid:1d3d8eee-b9e9-4ed0-81df-a840770ebdb3 ## 32 resource_map_doi:10.18739/A25D8NG8W ## 33 doi:10.18739/A25D8NG8W ## 34 resource_map_doi:10.18739/A2959C92N ## 35 doi:10.18739/A2959C92N ## 36 resource_map_urn:uuid:2e2cb0e0-a346-459f-a7cd-3f1a2cda882a ## 37 urn:uuid:954d62e9-b105-4c80-bb81-4599e3270bdf ## 38 resource_map_urn:uuid:e96a5179-6b35-440a-b2fc-2ef84fdbc63a ## 39 urn:uuid:d2bc3531-df72-4d32-9f10-64d648769af1 ## 40 resource_map_doi:10.18739/A2DZ0337H ## 41 doi:10.18739/A2DZ0337H ## 42 resource_map_urn:uuid:4c064889-096c-41e0-8715-81d6a6673846 ## 43 urn:uuid:4c064889-096c-41e0-8715-81d6a6673846 ## 44 resource_map_urn:uuid:4a198a6b-0d2c-42e8-9a55-69d09f168f62 ## 45 urn:uuid:925dda2a-f7cc-404b-b3e1-26f07b9f19e6 ## 46 resource_map_doi:10.18739/A2JQ0SX1G ## 47 doi:10.18739/A2JQ0SX1G ## 48 resource_map_urn:uuid:a42fe946-61e7-4f3f-9042-d27bd1b5c059 ## 49 urn:uuid:a65f8eb5-ca78-49fc-ba72-7e3837cca0f3 ## 50 resource_map_urn:uuid:6e93e759-d6b8-4141-8bed-818e7c1da076 ## 51 urn:uuid:3d1553dd-6ccb-4f5f-8608-57f7aacd2698 ## 52 resource_map_urn:uuid:dbb03275-f1d6-46f6-b73b-49f9624e35e1 ## 53 urn:uuid:e85eb97d-3a57-4aa9-8b95-edec63df1e52 ## 54 resource_map_urn:uuid:4085ac9c-81b4-4193-bfcc-fdc33fe2e2a7 ## 55 urn:uuid:1a334a5e-3a7c-4707-9b61-2b0fa7ee11b7 ## 56 resource_map_urn:uuid:dfcd5365-39f1-4a80-9b23-594ae7a516e7 ## 57 urn:uuid:4aa92ad5-2ae7-4dfd-af53-74fad3c31732 ## 58 resource_map_urn:uuid:7eb3c850-8262-4a5a-a521-35c2ee655733 ## 59 urn:uuid:7eb3c850-8262-4a5a-a521-35c2ee655733 ## 60 urn:uuid:e1250286-b19c-40c4-a39b-4f24a81d84b0 ## 61 urn:uuid:d593a60e-bbf5-4d4a-9900-535f568a11fe ## 62 resource_map_doi:10.18739/A2T43J47X ## 63 doi:10.18739/A2T43J47X ## 64 resource_map_urn:uuid:a0999bb2-62e0-4bcc-8ec0-b058321b050b ## 65 urn:uuid:f3312ad0-4d3e-4b07-9cdd-84152a45384a ## 66 urn:uuid:ac3e7c66-b1db-4ad4-a899-0fbcb19755d6 ## 67 resource_map_doi:10.18739/A2XW47Z0G ## 68 doi:10.18739/A2XW47Z0G ## 69 resource_map_urn:uuid:0dd6b926-4f0d-474e-8144-df0b727849e7 ## 70 urn:uuid:01c91431-96d6-4194-bb55-96cdb19853a9 ## 71 resource_map_urn:uuid:5c4625df-b750-4928-8653-cddbb507dfb2 ## 72 urn:uuid:5eedffbb-cba9-4ed4-ad96-7e69fd385d8c ## 73 resource_map_urn:uuid:f6260076-30be-4b28-bf85-6b6e70edd647 ## 74 urn:uuid:f6260076-30be-4b28-bf85-6b6e70edd647 ## 75 resource_map_urn:uuid:bf846180-b410-4c08-8264-a87a631c115b ## 76 urn:uuid:6c1ecfa2-ca5b-403f-9686-4880378ac8e8 ## 77 resource_map_urn:uuid:612cf386-b7ab-4fbd-95f4-3459ade37516 ## 78 urn:uuid:b4dfd84e-366f-42b6-a000-77a83ebfe511 ## 79 resource_map_urn:uuid:24559da1-c111-4797-8169-0d1600448f96 ## 80 urn:uuid:24559da1-c111-4797-8169-0d1600448f96 ## 81 resource_map_urn:uuid:b613e260-bcff-473d-b50b-125beea8106d ## 82 urn:uuid:b613e260-bcff-473d-b50b-125beea8106d ## 83 resource_map_urn:uuid:ac6b284d-c9b5-43b4-9c9a-9a0619cf6742 ## 84 urn:uuid:ac6b284d-c9b5-43b4-9c9a-9a0619cf6742 ## 85 resource_map_urn:uuid:a7bda9ee-28d6-40f6-8c00-6bcd1c693e0c ## 86 urn:uuid:7c686dcc-46e9-4c9e-a4ff-67319f2b9196 ## 87 resource_map_urn:uuid:be7fc51a-21ca-408e-9f7b-087e13dbb347 ## 88 urn:uuid:be7fc51a-21ca-408e-9f7b-087e13dbb347 ## 89 urn:uuid:56c3ab8e-951b-4c4a-abad-6174f83780fd ## 90 resource_map_urn:uuid:d7cf28c2-4f84-434e-afe1-06a7cfcf03fa ## 91 urn:uuid:32cfbc35-8eaa-4944-99b4-001d69ddc6ef ## 92 resource_map_urn:uuid:755c7d65-6382-4c3c-bde6-d44dd33b30f2 ## 93 urn:uuid:50ca6864-180b-478a-9bfe-fbe3b57f38e7 ## 94 urn:uuid:429acf20-74c4-4d83-9883-ffb9a4c3e89f ## 95 resource_map_urn:uuid:9d264156-43d8-470c-b5d5-a26965fddfed ## 96 urn:uuid:9d264156-43d8-470c-b5d5-a26965fddfed ## 97 resource_map_urn:uuid:9b0f11de-849b-49ed-b1e9-df9c4310ad0e ## 98 urn:uuid:9b0f11de-849b-49ed-b1e9-df9c4310ad0e ## 99 resource_map_urn:uuid:8720c79f-2f12-4b04-a613-81dbc22255be ## 100 urn:uuid:432db1b6-b283-45da-a9f7-5a8a933c1902 ## 101 resource_map_urn:uuid:0b2936f2-7c2a-4802-9ab2-2ee5dc75c2e6 ## 102 urn:uuid:42597b8f-92f9-4016-8e58-1baf8d519c96 ## 103 resource_map_doi:10.18739/A22N4ZK5G ## 104 doi:10.18739/A22N4ZK5G ## 105 resource_map_urn:uuid:fb99a1a2-26b7-430f-8a96-dae482007fbc ## 106 urn:uuid:7f016267-18b2-49f1-ab57-1f32d1a8621a ## 107 resource_map_doi:10.18739/A26D5PC9M ## 108 doi:10.18739/A26D5PC9M ## 109 resource_map_doi:10.18739/A2B56D64V ## 110 doi:10.18739/A2B56D64V ## 111 resource_map_doi:10.18739/A2FX7408P ## 112 doi:10.18739/A2FX7408P ## 113 resource_map_urn:uuid:36786f14-6838-432b-be3e-3d65e33bcc5e ## 114 urn:uuid:542e0f82-8c6d-4a3e-b7fb-48835f438914 ## 115 resource_map_doi:10.18739/A2KP7TT15 ## 116 doi:10.18739/A2KP7TT15 ## 117 urn:uuid:a194fb25-faa7-444b-ba56-9a73db8f3a73 ## 118 urn:uuid:a2ffcf35-6716-441a-b893-94eb8bc321fb ## 119 resource_map_doi:10.18739/A2QB9V73G ## 120 doi:10.18739/A2QB9V73G ## 121 resource_map_urn:uuid:ec28eadc-cf17-4abc-b16e-69cf87f0bebd ## 122 urn:uuid:430922f6-6b67-4491-8faf-29eb47c18a24 ## 123 resource_map_doi:10.18739/A2V40K16P ## 124 doi:10.18739/A2V40K16P ## 125 urn:uuid:bccebda1-cb04-457b-bc91-410945b0dc87 ## 126 resource_map_urn:uuid:65705b6d-13f6-4dcf-a44f-5f5ddd640509 ## 127 urn:uuid:ee566c8c-ca10-43df-812a-a9df59b3667e ## 128 urn:uuid:140e2233-fb9d-4ae2-a742-5c913c3f7724 ## 129 urn:uuid:0bfa3290-4c23-4cc6-b05d-563c32a1e725 ## 130 urn:uuid:8a89ade9-6166-4423-9b44-30a29d416cd9 ## 131 urn:uuid:7a589ae5-fd5f-421c-acda-c9b29f569686 ## 132 urn:uuid:55b3f360-976e-4bdb-8047-584f9114f2d4 ## 133 urn:uuid:5cb95853-e19a-4d9c-94dc-95254aed167d ## 134 urn:uuid:f76ce231-0fb2-4598-b06e-1d457ecec01d ## 135 urn:uuid:80169551-04de-4b1b-ae74-0c11d65ee407 ## 136 urn:uuid:1ca4c0df-c368-4c7c-8d1d-cf3cc36ccf8d ## 137 urn:uuid:24561338-dc5f-4207-a6ff-9b8bbdcb61fa ## 138 urn:uuid:23e8bbf5-1a6c-47c3-8840-bca2f86c6d5f ## 139 urn:uuid:c7946427-5ce0-4e18-a30a-3cc424f36dd0 ## 140 urn:uuid:48c0f101-16a2-4e22-a8c6-3b7acfe0b34d ## 141 urn:uuid:2189fd1c-5b6e-48f9-a7ab-4fdeca81ea87 ## 142 urn:uuid:017e2f63-3b35-432e-a3d0-20cea9d877da ## 143 urn:uuid:9135878f-6cd3-4362-8769-0dc0e4abef3e ## 144 urn:uuid:2e318964-e575-4a32-bc6f-81b93803f6e1 ## 145 urn:uuid:09cb6c47-ca6f-4973-99f4-3f2336d664fb ## 146 urn:uuid:4deb9ada-b814-4258-932b-6b8e97486d40 ## 147 urn:uuid:653b66c3-a839-45fb-a2b8-4f17f099b2dc ## 148 urn:uuid:aa9d7dc7-7c97-4fed-9cea-94e8995c3906 ## 149 urn:uuid:ea334339-7c25-4715-8edb-34f2bef57200 ## 150 urn:uuid:7d104a59-b544-4167-81b8-1c56dd556602 ## 151 urn:uuid:b296ec63-41eb-46a7-bc22-40a04bd86fb6 ## 152 urn:uuid:e26ea363-aa7d-45d3-8c3e-65afe8547fa1 ## 153 urn:uuid:4f6c3e58-d940-4a04-972e-1b9334d20ef1 ## 154 urn:uuid:8b66a954-88e7-4006-b1cf-a80a08ce02d5 ## 155 urn:uuid:2e11562a-ce15-4122-b969-984b865bef84 ## 156 urn:uuid:5adaf24a-9486-44dd-9fea-238c4fa712e2 ## 157 urn:uuid:f8d58047-10bc-4032-a324-367e4d2ea602 ## 158 resource_map_urn:uuid:353bd1c1-f691-42fe-afb7-eada331f6c04 ## 159 urn:uuid:990f7311-dfcc-4d33-a5a9-56a5e610d076 ## 160 resource_map_doi:10.18739/A2ZW18V0Q ## 161 doi:10.18739/A2ZW18V0Q ## 162 resource_map_urn:uuid:4fa2d26d-f449-48dc-9686-6307f883380f ## 163 urn:uuid:e3114753-7241-409d-86dc-42f42e5e9bba ## 164 resource_map_doi:10.18739/A23N20G53 ## 165 doi:10.18739/A23N20G53 ## 166 resource_map_urn:uuid:3ceed3d3-0697-4615-9d3d-eb16879a1c96 ## 167 urn:uuid:0efbfc54-1f2d-43fc-b26c-3af433054e61 ## 168 resource_map_urn:uuid:43ccbf5e-95f2-4398-9f02-eb24672f4509 ## 169 urn:uuid:31514230-fd0a-436a-9447-291d24da03ee ## 170 resource_map_doi:10.18739/A27D2Q90Z ## 171 doi:10.18739/A27D2Q90Z ## 172 resource_map_urn:uuid:c5e3d6ab-51ce-4369-8843-8e347b73dd41 ## 173 urn:uuid:f71379b3-6585-4a56-9f8f-4b2298a5db7c ## 174 resource_map_urn:uuid:3ec88ccd-dcfd-4dc1-ba8f-e26c4d79d189 ## 175 resource_map_urn:uuid:802fd2f7-9e78-4f63-89a3-39f70707ceaf ## 176 urn:uuid:cdc8bf98-fa22-44d4-9590-64c514867495 ## 177 resource_map_urn:uuid:bb818a59-21eb-4070-abda-36c6b101297a ## 178 resource_map_urn:uuid:d4d9d7a5-b279-4b50-bc4b-beb745496a17 ## 179 urn:uuid:d5409690-e577-477c-8149-7778ba4d5f44 ## 180 urn:uuid:6ce067fa-d9e8-4e31-9b10-996a5a005580 ## 181 urn:uuid:4f16e011-07e9-4917-a94a-c841deef75b2 ## 182 resource_map_doi:10.18739/A2C53F343 ## 183 doi:10.18739/A2C53F343 ## 184 resource_map_urn:uuid:94e710ae-6f41-4b75-b0e1-4d7f495036e9 ## 185 urn:uuid:94e710ae-6f41-4b75-b0e1-4d7f495036e9 ## 186 urn:uuid:97746d45-392f-4766-aa9f-28920bb0ab2b ## 187 urn:uuid:abacb59f-9499-4f2c-b620-e8335e9e7fe1 ## 188 urn:uuid:637901af-3eba-46db-856a-9f4e233d144a ## 189 urn:uuid:3a4c8896-fa7f-49f4-9e13-352ba859822c ## 190 urn:uuid:f875afea-4fe1-4156-8eb0-9d6803ba9e90 ## 191 urn:uuid:f695424d-4430-4017-b7b6-a565eb5b88f8 ## 192 urn:uuid:339324fd-946e-48c1-a2aa-be5a41437f18 ## 193 urn:uuid:442e3761-4d75-41c3-b7d8-1d405fedbf73 ## 194 urn:uuid:471a0bb1-43e0-48e6-9b79-233f54769660 ## 195 urn:uuid:6740416a-f07a-47e9-8e79-853c52bff278 ## 196 urn:uuid:26527141-4c8a-4a73-b78c-a34268446f62 ## 197 urn:uuid:d9a64d81-6c77-4185-964f-018156ada0c6 ## 198 urn:uuid:0bc9a4d3-f53f-4e35-adb8-6c0bef5d0443 ## 199 urn:uuid:8a14d17a-c8a1-46a8-aaa3-5a0f28779ea2 ## 200 urn:uuid:13a17977-6649-4308-9c38-27e796f26b3e ## dateUploaded dateModified ## 1 2024-08-12 22:28:49 2024-08-12 22:28:50 ## 2 2024-08-12 22:28:43 2024-08-12 22:28:46 ## 3 2024-08-12 18:17:49 2024-08-12 18:17:50 ## 4 2024-08-12 18:17:41 2024-08-12 18:17:44 ## 5 2024-08-12 17:44:38 2024-08-12 17:44:38 ## 6 2024-08-12 17:44:31 2024-08-12 17:44:35 ## 7 2024-08-12 17:41:27 2024-08-12 17:41:28 ## 8 2024-08-12 17:41:20 2024-08-12 17:41:23 ## 9 2024-08-08 23:53:25 2024-08-08 23:53:26 ## 10 2024-08-08 23:53:18 2024-08-08 23:53:21 ## 11 2024-08-08 23:07:30 2024-08-08 23:07:33 ## 12 2024-08-07 20:34:59 2024-08-07 20:35:00 ## 13 2024-08-07 20:34:52 2024-08-07 20:34:56 ## 14 2024-08-07 20:33:44 2024-08-07 20:34:59 ## 15 2024-08-07 20:33:39 2024-08-07 20:34:55 ## 16 2024-08-07 20:32:49 2024-08-07 20:33:44 ## 17 2024-08-07 20:32:41 2024-08-07 20:33:42 ## 18 2024-08-07 17:44:24 2024-08-07 20:32:49 ## 19 2024-08-07 17:43:58 2024-08-07 20:32:44 ## 20 2024-08-05 21:23:27 2024-08-05 21:23:28 ## 21 2024-08-05 21:23:21 2024-08-05 21:23:24 ## 22 2024-08-05 21:21:47 2024-08-05 21:23:27 ## 23 2024-08-05 21:21:38 2024-08-05 21:23:23 ## 24 2024-08-05 16:33:16 2024-08-12 17:44:27 ## 25 2024-08-05 16:19:53 2024-08-05 16:19:54 ## 26 2024-08-05 16:19:46 2024-08-05 16:19:49 ## 27 2024-08-02 23:27:48 2024-08-07 20:34:51 ## 28 2024-08-02 21:24:41 2024-08-05 21:21:47 ## 29 2024-08-02 21:24:32 2024-08-05 21:21:43 ## 30 2024-08-02 21:24:32 2024-08-05 21:23:20 ## 31 2024-08-02 17:21:51 2024-08-05 16:19:45 ## 32 2024-08-01 23:33:31 2024-08-01 23:33:31 ## 33 2024-08-01 23:33:15 2024-08-01 23:33:26 ## 34 2024-08-01 20:50:22 2024-08-01 20:50:23 ## 35 2024-08-01 20:50:15 2024-08-01 20:50:19 ## 36 2024-08-01 20:46:20 2024-08-01 20:50:22 ## 37 2024-08-01 20:46:15 2024-08-01 20:50:18 ## 38 2024-08-01 20:44:56 2024-08-01 20:46:21 ## 39 2024-08-01 20:44:50 2024-08-01 20:46:18 ## 40 2024-08-01 20:31:55 2024-08-01 20:44:56 ## 41 2024-08-01 20:31:48 2024-08-01 20:44:54 ## 42 2024-08-01 20:29:56 2024-08-01 20:31:55 ## 43 2024-08-01 20:29:47 2024-08-01 20:31:51 ## 44 2024-08-01 20:20:05 2024-08-01 20:29:56 ## 45 2024-08-01 20:19:57 2024-08-01 20:29:50 ## 46 2024-08-01 18:00:12 2024-08-01 20:20:05 ## 47 2024-08-01 17:59:54 2024-08-01 20:20:02 ## 48 2024-08-01 17:40:30 2024-08-01 17:40:31 ## 49 2024-08-01 17:40:22 2024-08-01 17:40:28 ## 50 2024-08-01 17:25:33 2024-08-01 17:40:30 ## 51 2024-08-01 17:25:25 2024-08-01 17:40:28 ## 52 2024-08-01 17:09:51 2024-08-01 17:25:33 ## 53 2024-08-01 17:09:45 2024-08-01 17:25:31 ## 54 2024-08-01 17:09:09 2024-08-01 17:09:51 ## 55 2024-08-01 17:09:03 2024-08-01 17:09:49 ## 56 2024-08-01 17:05:57 2024-08-01 17:09:09 ## 57 2024-08-01 17:05:48 2024-08-01 17:09:07 ## 58 2024-08-01 16:55:38 2024-08-01 17:05:57 ## 59 2024-08-01 16:55:27 2024-08-01 17:05:54 ## 60 2024-08-01 13:51:23 2024-08-12 17:41:19 ## 61 2024-07-31 20:04:29 2024-08-01 17:59:52 ## 62 2024-07-31 19:44:10 2024-07-31 19:44:11 ## 63 2024-07-31 19:44:04 2024-07-31 19:44:06 ## 64 2024-07-31 19:40:44 2024-07-31 19:44:10 ## 65 2024-07-31 19:40:40 2024-07-31 19:44:05 ## 66 2024-07-31 19:39:16 2024-07-31 19:44:03 ## 67 2024-07-31 19:38:06 2024-07-31 19:40:45 ## 68 2024-07-31 19:38:01 2024-07-31 19:40:41 ## 69 2024-07-31 18:31:23 2024-08-01 16:55:38 ## 70 2024-07-31 18:31:17 2024-08-01 16:55:32 ## 71 2024-07-31 18:24:18 2024-07-31 18:31:23 ## 72 2024-07-31 18:24:13 2024-07-31 18:31:21 ## 73 2024-07-31 18:22:50 2024-07-31 18:24:18 ## 74 2024-07-31 18:22:42 2024-07-31 18:24:17 ## 75 2024-07-31 18:18:47 2024-07-31 18:22:50 ## 76 2024-07-31 18:18:40 2024-07-31 18:22:45 ## 77 2024-07-31 18:15:45 2024-07-31 18:18:47 ## 78 2024-07-31 18:15:39 2024-07-31 18:18:44 ## 79 2024-07-31 18:14:36 2024-07-31 18:15:45 ## 80 2024-07-31 18:14:26 2024-07-31 18:15:43 ## 81 2024-07-31 18:11:59 2024-07-31 18:14:36 ## 82 2024-07-31 18:11:51 2024-07-31 18:14:30 ## 83 2024-07-31 18:02:26 2024-07-31 18:11:59 ## 84 2024-07-31 18:02:15 2024-07-31 18:11:54 ## 85 2024-07-31 18:00:11 2024-07-31 18:02:26 ## 86 2024-07-31 17:59:56 2024-07-31 18:02:20 ## 87 2024-07-31 17:50:29 2024-07-31 18:00:11 ## 88 2024-07-31 17:50:17 2024-07-31 18:00:08 ## 89 2024-07-31 17:06:57 2024-07-31 19:38:00 ## 90 2024-07-31 16:26:42 2024-07-31 17:50:29 ## 91 2024-07-31 16:26:31 2024-07-31 17:50:25 ## 92 2024-07-31 02:31:57 2024-08-02 20:29:50 ## 93 2024-07-31 02:31:20 2024-07-31 02:31:55 ## 94 2024-07-31 02:19:03 2024-07-31 02:31:53 ## 95 2024-07-30 23:52:57 2024-07-31 16:26:43 ## 96 2024-07-30 23:52:50 2024-07-31 16:26:40 ## 97 2024-07-30 23:48:21 2024-07-30 23:52:57 ## 98 2024-07-30 23:48:11 2024-07-30 23:52:53 ## 99 2024-07-30 23:36:30 2024-07-30 23:48:21 ## 100 2024-07-30 23:36:26 2024-07-30 23:48:16 ## 101 2024-07-30 23:32:03 2024-07-30 23:36:30 ## 102 2024-07-30 23:31:58 2024-07-30 23:36:29 ## 103 2024-07-30 22:11:48 2024-07-30 22:11:49 ## 104 2024-07-30 22:11:40 2024-07-30 22:11:43 ## 105 2024-07-30 22:10:52 2024-07-30 22:11:48 ## 106 2024-07-30 22:10:43 2024-07-30 22:11:43 ## 107 2024-07-30 21:58:07 2024-07-30 21:58:08 ## 108 2024-07-30 21:58:03 2024-07-30 21:58:05 ## 109 2024-07-30 19:55:36 2024-07-30 19:55:36 ## 110 2024-07-30 19:55:31 2024-07-30 19:55:32 ## 111 2024-07-30 19:52:45 2024-07-30 19:52:45 ## 112 2024-07-30 19:52:36 2024-07-30 19:52:40 ## 113 2024-07-30 18:28:16 2024-07-30 23:32:04 ## 114 2024-07-30 18:28:10 2024-07-30 23:32:01 ## 115 2024-07-30 18:00:59 2024-07-30 18:01:00 ## 116 2024-07-30 18:00:53 2024-07-30 18:00:55 ## 117 2024-07-30 17:31:00 2024-07-30 18:00:52 ## 118 2024-07-30 16:54:42 2024-07-30 18:00:51 ## 119 2024-07-29 19:46:15 2024-07-30 22:10:52 ## 120 2024-07-29 19:46:06 2024-07-30 22:10:50 ## 121 2024-07-29 19:44:42 2024-07-29 19:46:15 ## 122 2024-07-29 19:44:32 2024-07-29 19:46:09 ## 123 2024-07-29 16:42:46 2024-07-29 16:42:46 ## 124 2024-07-29 16:42:38 2024-07-29 16:42:42 ## 125 2024-07-28 13:21:29 2024-07-28 13:21:37 ## 126 2024-07-27 00:47:40 2024-07-27 00:47:40 ## 127 2024-07-27 00:47:24 2024-07-31 02:19:33 ## 128 2024-07-27 00:47:08 2024-07-31 02:19:22 ## 129 2024-07-27 00:47:08 2024-07-31 02:19:25 ## 130 2024-07-27 00:47:08 2024-07-31 02:19:23 ## 131 2024-07-27 00:47:08 2024-07-31 02:19:21 ## 132 2024-07-27 00:47:07 2024-07-31 02:19:26 ## 133 2024-07-27 00:47:07 2024-07-31 02:19:35 ## 134 2024-07-27 00:47:07 2024-07-31 02:19:33 ## 135 2024-07-27 00:47:06 2024-07-31 02:19:27 ## 136 2024-07-27 00:47:06 2024-07-31 02:19:32 ## 137 2024-07-27 00:47:06 2024-07-31 02:19:36 ## 138 2024-07-27 00:47:06 2024-07-31 02:20:01 ## 139 2024-07-27 00:47:05 2024-07-31 02:19:38 ## 140 2024-07-27 00:47:05 2024-07-31 02:19:39 ## 141 2024-07-27 00:47:05 2024-07-31 02:19:41 ## 142 2024-07-27 00:47:05 2024-07-31 02:20:11 ## 143 2024-07-27 00:47:05 2024-07-31 02:19:44 ## 144 2024-07-27 00:47:04 2024-07-31 02:19:45 ## 145 2024-07-27 00:47:04 2024-07-31 02:20:16 ## 146 2024-07-27 00:47:04 2024-07-31 02:19:47 ## 147 2024-07-27 00:47:04 2024-07-31 02:19:54 ## 148 2024-07-27 00:47:04 2024-07-31 02:19:51 ## 149 2024-07-27 00:47:04 2024-07-31 02:20:05 ## 150 2024-07-27 00:47:02 2024-07-31 02:20:25 ## 151 2024-07-27 00:47:02 2024-07-31 02:19:58 ## 152 2024-07-27 00:47:02 2024-07-31 02:20:32 ## 153 2024-07-27 00:47:01 2024-07-31 02:20:40 ## 154 2024-07-27 00:47:01 2024-07-31 02:20:13 ## 155 2024-07-27 00:47:00 2024-07-31 02:20:28 ## 156 2024-07-27 00:47:00 2024-07-31 02:20:37 ## 157 2024-07-27 00:47:00 2024-07-31 02:20:20 ## 158 2024-07-26 19:16:27 2024-07-29 19:44:43 ## 159 2024-07-26 19:16:23 2024-07-29 19:44:40 ## 160 2024-07-26 19:05:50 2024-07-26 19:05:51 ## 161 2024-07-26 19:05:43 2024-07-26 19:05:47 ## 162 2024-07-26 18:40:05 2024-07-26 19:05:50 ## 163 2024-07-26 18:40:01 2024-07-26 19:05:46 ## 164 2024-07-26 17:24:44 2024-07-26 17:24:45 ## 165 2024-07-26 17:24:38 2024-07-26 17:24:40 ## 166 2024-07-26 17:17:32 2024-07-26 18:40:05 ## 167 2024-07-26 17:17:26 2024-07-26 18:40:03 ## 168 2024-07-26 17:11:07 2024-07-26 17:17:33 ## 169 2024-07-26 17:11:01 2024-07-26 17:17:29 ## 170 2024-07-26 17:04:04 2024-07-26 17:11:07 ## 171 2024-07-26 17:03:54 2024-07-26 17:11:05 ## 172 2024-07-26 16:45:29 2024-07-26 19:16:27 ## 173 2024-07-26 16:45:26 2024-07-26 19:16:25 ## 174 2024-07-26 16:41:48 2024-07-26 16:45:30 ## 175 2024-07-26 16:24:35 2024-07-26 16:41:48 ## 176 2024-07-26 16:24:31 2024-07-26 16:45:28 ## 177 2024-07-26 16:23:25 2024-07-26 16:24:35 ## 178 2024-07-26 16:17:27 2024-07-26 16:23:26 ## 179 2024-07-26 16:17:16 2024-07-26 16:24:34 ## 180 2024-07-26 14:56:50 2024-07-26 19:05:22 ## 181 2024-07-26 14:27:32 2024-07-26 19:05:38 ## 182 2024-07-25 21:54:24 2024-07-25 21:54:24 ## 183 2024-07-25 21:54:16 2024-07-25 21:54:19 ## 184 2024-07-25 20:31:53 2024-07-26 16:17:27 ## 185 2024-07-25 20:31:41 2024-07-26 16:17:25 ## 186 2024-07-25 20:18:06 2024-08-01 23:32:55 ## 187 2024-07-25 20:18:05 2024-08-01 23:32:50 ## 188 2024-07-25 20:18:04 2024-08-01 23:32:22 ## 189 2024-07-25 20:18:04 2024-08-01 23:33:00 ## 190 2024-07-25 20:18:03 2024-08-01 23:32:57 ## 191 2024-07-25 20:18:02 2024-08-01 23:33:09 ## 192 2024-07-25 20:18:01 2024-08-01 23:33:10 ## 193 2024-07-25 20:17:59 2024-08-01 23:31:44 ## 194 2024-07-25 20:17:58 2024-08-01 23:32:13 ## 195 2024-07-25 20:17:57 2024-08-01 23:32:03 ## 196 2024-07-25 20:17:56 2024-08-01 23:32:24 ## 197 2024-07-25 20:17:56 2024-08-01 23:32:59 ## 198 2024-07-25 20:17:55 2024-08-01 23:31:55 ## 199 2024-07-25 20:17:53 2024-08-01 23:32:15 ## 200 2024-07-25 20:17:52 2024-08-01 23:32:46 ## title ## 1 &lt;NA&gt; ## 2 In Situ Observations from the International Arctic Buoy Programme (IABP) 2023, Level 1 ## 3 &lt;NA&gt; ## 4 Near-Real-Time National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 2 ## 5 &lt;NA&gt; ## 6 Small mammal mark-recapture data 2018-2023 (3 sites near Utqiagvik, Alaska) ## 7 &lt;NA&gt; ## 8 Small mammal mark-recapture data 2017-2023 (3 sites near Toolik Lake, Alaska) ## 9 &lt;NA&gt; ## 10 Surface underway dissolved oxygen/nitrogen saturation ratio and oxygen concentration data collected in the Bering, Beaufort, and Chukchi Seas on research vessel Sikuliaq in 2023 ## 11 Proceedings of the Permafrost &amp; Infrastructure Symposium ## 12 &lt;NA&gt; ## 13 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 14 &lt;NA&gt; ## 15 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 16 &lt;NA&gt; ## 17 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 18 &lt;NA&gt; ## 19 Metal Concentrations in Tundra Seep, Tributary and River Waters of the Western Brooks Range, Alaska, 2022-2023 ## 20 &lt;NA&gt; ## 21 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 22 &lt;NA&gt; ## 23 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 24 &lt;NA&gt; ## 25 &lt;NA&gt; ## 26 Global Vegetation Greenness Normalized Difference Vegetation Index (NDVI) from Advanced Very High Resolution Radiometer (AVHRR) Global Inventory Modeling and Mapping Studies-3rd Generation (GIMMS-3G+), 1981-2022 ## 27 &lt;NA&gt; ## 28 &lt;NA&gt; ## 29 Mercury content in floodplain sediments from the Yukon River Basin, Alaska, 2022 ## 30 &lt;NA&gt; ## 31 &lt;NA&gt; ## 32 &lt;NA&gt; ## 33 Sea Ice Motion Estimation Algorithm Using Computer Vision for The Sea Ice Dynamic Experiment (SIDEx) Field Campaign, Alaska, 2021 ## 34 &lt;NA&gt; ## 35 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 36 &lt;NA&gt; ## 37 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 38 &lt;NA&gt; ## 39 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 40 &lt;NA&gt; ## 41 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 42 &lt;NA&gt; ## 43 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 44 &lt;NA&gt; ## 45 National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 46 &lt;NA&gt; ## 47 Near-Real-Time National Oceanic and Atmospheric Administration/ National Snow and Ice Data Center (NOAA/NSIDC) Climate Data Record of Passive Microwave Sea Ice Concentration, Version 4 ## 48 &lt;NA&gt; ## 49 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 50 &lt;NA&gt; ## 51 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 52 &lt;NA&gt; ## 53 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 54 &lt;NA&gt; ## 55 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 56 &lt;NA&gt; ## 57 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 58 &lt;NA&gt; ## 59 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 60 &lt;NA&gt; ## 61 &lt;NA&gt; ## 62 &lt;NA&gt; ## 63 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 64 &lt;NA&gt; ## 65 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 66 &lt;NA&gt; ## 67 &lt;NA&gt; ## 68 Soil Moisture and Ocean Salinity (SMOS)/CryoSat-2 Sea Ice Thickness/Volume Data ## 69 &lt;NA&gt; ## 70 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 71 &lt;NA&gt; ## 72 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 73 &lt;NA&gt; ## 74 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 75 &lt;NA&gt; ## 76 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 77 &lt;NA&gt; ## 78 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 79 &lt;NA&gt; ## 80 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 81 &lt;NA&gt; ## 82 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 83 &lt;NA&gt; ## 84 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 85 &lt;NA&gt; ## 86 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 87 &lt;NA&gt; ## 88 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 89 &lt;NA&gt; ## 90 &lt;NA&gt; ## 91 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 92 &lt;NA&gt; ## 93 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 94 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 95 &lt;NA&gt; ## 96 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 97 &lt;NA&gt; ## 98 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 99 &lt;NA&gt; ## 100 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 101 &lt;NA&gt; ## 102 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 103 &lt;NA&gt; ## 104 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 105 &lt;NA&gt; ## 106 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 107 &lt;NA&gt; ## 108 Sea Ice Concentrations from Nimbus-7 Scanning Multichannel Microwave Radiometer (SMMR) and Defense Meteorological Satellites Program (DMSP) Special Sensor Microwave Imager - Special Sensor Microwave Imager Sounder (SSM/I-SSMIS) Passive Microwave Data, Version 1, North/South Poles, 1978 - 2021 ## 109 &lt;NA&gt; ## 110 Near-Real-Time Defense Meteorological Satellite Program (DMSP) Special Sensor Microwave Imager/Sounder (SSMIS) Daily Polar Gridded Sea Ice Concentrations, Version 1 ## 111 &lt;NA&gt; ## 112 Ice, Cloud, and land Elevation Satellite-2 (ICESat-2) L4 Monthly Gridded Sea Ice Thickness, Version 2, Arctic Ocean, 2018 - 2022 ## 113 &lt;NA&gt; ## 114 The Onset of Recrystallization in Polar Firn, Summit, Greenland, June 2017 ## 115 &lt;NA&gt; ## 116 Quicklook Arctic Weekly Equal-Area Scalable Earth Grid (EASE-Grid) Sea Ice Age, Version 1 ## 117 &lt;NA&gt; ## 118 &lt;NA&gt; ## 119 &lt;NA&gt; ## 120 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 121 &lt;NA&gt; ## 122 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 123 &lt;NA&gt; ## 124 Characterization of summertime carbonaceous aerosol on the North Slope of Alaska (Toolik and Utqiaġvik), 2022–2023 ## 125 Sermilik Fjord Hydrography ## 126 &lt;NA&gt; ## 127 Flow data from Hot Springs Creek, Imuruk Basin, Alaska, 2012 ## 128 &lt;NA&gt; ## 129 &lt;NA&gt; ## 130 &lt;NA&gt; ## 131 &lt;NA&gt; ## 132 &lt;NA&gt; ## 133 &lt;NA&gt; ## 134 &lt;NA&gt; ## 135 &lt;NA&gt; ## 136 &lt;NA&gt; ## 137 &lt;NA&gt; ## 138 &lt;NA&gt; ## 139 &lt;NA&gt; ## 140 &lt;NA&gt; ## 141 &lt;NA&gt; ## 142 &lt;NA&gt; ## 143 &lt;NA&gt; ## 144 &lt;NA&gt; ## 145 &lt;NA&gt; ## 146 &lt;NA&gt; ## 147 &lt;NA&gt; ## 148 &lt;NA&gt; ## 149 &lt;NA&gt; ## 150 &lt;NA&gt; ## 151 &lt;NA&gt; ## 152 &lt;NA&gt; ## 153 &lt;NA&gt; ## 154 &lt;NA&gt; ## 155 &lt;NA&gt; ## 156 &lt;NA&gt; ## 157 &lt;NA&gt; ## 158 &lt;NA&gt; ## 159 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 160 &lt;NA&gt; ## 161 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 162 &lt;NA&gt; ## 163 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 164 &lt;NA&gt; ## 165 Ice core methanesulfonic acid from the Begguya (Mount Hunter) plateau, Denali National Park, Alaska, 2013 ## 166 &lt;NA&gt; ## 167 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 168 &lt;NA&gt; ## 169 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 170 &lt;NA&gt; ## 171 An updatable and comprehensive global cargo maritime network and strategic seaborne cargo routing model for global containerized and bulk vessel flow estimation 2021 ## 172 &lt;NA&gt; ## 173 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 174 &lt;NA&gt; ## 175 &lt;NA&gt; ## 176 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 177 &lt;NA&gt; ## 178 &lt;NA&gt; ## 179 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 180 &lt;NA&gt; ## 181 &lt;NA&gt; ## 182 &lt;NA&gt; ## 183 Saline permafrost degradation below a shallow thermokarst lake in northern Alaska, 2008-2023 ## 184 &lt;NA&gt; ## 185 Arc2kmTM: Arctic 2 kilometer Tide Model, 2021 ## 186 &lt;NA&gt; ## 187 &lt;NA&gt; ## 188 &lt;NA&gt; ## 189 &lt;NA&gt; ## 190 &lt;NA&gt; ## 191 &lt;NA&gt; ## 192 &lt;NA&gt; ## 193 &lt;NA&gt; ## 194 &lt;NA&gt; ## 195 &lt;NA&gt; ## 196 &lt;NA&gt; ## 197 &lt;NA&gt; ## 198 &lt;NA&gt; ## 199 &lt;NA&gt; ## 200 &lt;NA&gt; ## resourceMap ## 1 ## 2 resource.... ## 3 ## 4 resource.... ## 5 ## 6 resource.... ## 7 ## 8 resource.... ## 9 ## 10 resource.... ## 11 ## 12 ## 13 resource.... ## 14 ## 15 resource.... ## 16 ## 17 resource.... ## 18 ## 19 resource.... ## 20 ## 21 resource.... ## 22 ## 23 resource.... ## 24 resource.... ## 25 ## 26 resource.... ## 27 resource.... ## 28 ## 29 resource.... ## 30 resource.... ## 31 resource.... ## 32 ## 33 resource.... ## 34 ## 35 resource.... ## 36 ## 37 resource.... ## 38 ## 39 resource.... ## 40 ## 41 resource.... ## 42 ## 43 resource.... ## 44 ## 45 resource.... ## 46 ## 47 resource.... ## 48 ## 49 resource.... ## 50 ## 51 resource.... ## 52 ## 53 resource.... ## 54 ## 55 resource.... ## 56 ## 57 resource.... ## 58 ## 59 resource.... ## 60 resource.... ## 61 resource.... ## 62 ## 63 resource.... ## 64 ## 65 resource.... ## 66 resource.... ## 67 ## 68 resource.... ## 69 ## 70 resource.... ## 71 ## 72 resource.... ## 73 ## 74 resource.... ## 75 ## 76 resource.... ## 77 ## 78 resource.... ## 79 ## 80 resource.... ## 81 ## 82 resource.... ## 83 ## 84 resource.... ## 85 ## 86 resource.... ## 87 ## 88 resource.... ## 89 resource.... ## 90 ## 91 resource.... ## 92 ## 93 resource.... ## 94 ## 95 ## 96 resource.... ## 97 ## 98 resource.... ## 99 ## 100 resource.... ## 101 ## 102 resource.... ## 103 ## 104 resource.... ## 105 ## 106 resource.... ## 107 ## 108 resource.... ## 109 ## 110 resource.... ## 111 ## 112 resource.... ## 113 ## 114 resource.... ## 115 ## 116 resource.... ## 117 resource.... ## 118 resource.... ## 119 ## 120 resource.... ## 121 ## 122 resource.... ## 123 ## 124 resource.... ## 125 ## 126 ## 127 resource.... ## 128 resource.... ## 129 resource.... ## 130 resource.... ## 131 resource.... ## 132 resource.... ## 133 resource.... ## 134 resource.... ## 135 resource.... ## 136 resource.... ## 137 resource.... ## 138 resource.... ## 139 resource.... ## 140 resource.... ## 141 resource.... ## 142 resource.... ## 143 resource.... ## 144 resource.... ## 145 resource.... ## 146 resource.... ## 147 resource.... ## 148 resource.... ## 149 resource.... ## 150 resource.... ## 151 resource.... ## 152 resource.... ## 153 resource.... ## 154 resource.... ## 155 resource.... ## 156 resource.... ## 157 resource.... ## 158 ## 159 resource.... ## 160 ## 161 resource.... ## 162 ## 163 resource.... ## 164 ## 165 resource.... ## 166 ## 167 resource.... ## 168 ## 169 resource.... ## 170 ## 171 resource.... ## 172 ## 173 resource.... ## 174 ## 175 ## 176 resource.... ## 177 ## 178 ## 179 resource.... ## 180 resource.... ## 181 resource.... ## 182 ## 183 resource.... ## 184 ## 185 resource.... ## 186 resource.... ## 187 resource.... ## 188 resource.... ## 189 resource.... ## 190 resource.... ## 191 resource.... ## 192 resource.... ## 193 resource.... ## 194 resource.... ## 195 resource.... ## 196 resource.... ## 197 resource.... ## 198 resource.... ## 199 resource.... ## 200 resource.... Use facets All resource maps with &gt; 100 data objects that are not on the Arctic Data Center: https://cn.dataone.org/cn/v2/query/solr/?q=resourceMap:*+AND+-datasource:*ARCTIC*&amp;rows=0&amp;facet=true&amp;facet.field=resourceMap&amp;facet.mincount=100 "],["more-resources.html", "More resources", " More resources Solr’s The Standard Query Parser docs (high level of detail) Another quick reference: https://wiki.apache.org/solr/SolrQuerySyntax http://www.solrtutorial.com/ "],["adc-web-submissions.html", "ADC web submissions ", " ADC web submissions "],["add-physicals-to-submissions.html", "Add physicals to submissions", " Add physicals to submissions New submissions made through the web editor will not have any physical sections within the otherEntitys. Add them to the EML with the following script: for (i in seq_along(doc$dataset$otherEntity)) { otherEntity &lt;- doc$dataset$otherEntity[[i]] id &lt;- otherEntity$id if (!grepl(&quot;urn-uuid-&quot;, id)) { warning(&quot;otherEntity &quot;, i, &quot; is not a pid&quot;) } else { id &lt;- gsub(&quot;urn-uuid-&quot;, &quot;urn:uuid:&quot;, id) physical &lt;- arcticdatautils::pid_to_eml_physical(mn, id) doc$dataset$otherEntity[[i]]$physical &lt;- physical } } As you can see from code above, we use a for loop here to add physical sections. The for loop is a very useful tool to iterate over a list of elements. With for loop, you can repeat a specific block of code without copying and pasting the code over and over again. When processing datasets in Arctic Data Center, there are many places where for loop can be used, such as publishing a bunch of objects with pids, updating formatID for pkg$data, adding physical section like above code, etc. A loop is composed of two parts: the sequence and the body. The sequence usually generates indices to locate elements and the body contains the code that you want to iterate for each element. Here is an example of adding the same attributeList for all the dataTables in the metadata using for loop. attributes &lt;- read.csv(&#39;attributes.csv&#39;) # attribute table in csv format attributeList &lt;- EML::set_attributes(attributes = attributes) for (i in 1:length(doc$dataset$dataTable)) { # sequence part doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList # body part } "],["assess-attributes.html", "Assess attributes", " Assess attributes New submissions made through the web editor should have attributes created by the submitter. If there are no attributes, or if they are incomplete, please use the email template to correspond with the submitter to ask for attributes. Additionally, the web editor will not allow for the creation of custom units. submitters should select other/none in the editor unit dropdown if they cannot find their desired unit. In the EML document, these will result in dimensionless units. Accordingly, new submissions should be checked for dimensionless units. This can be done with the following code. Note that this only applies if there is more than one otherEntity. dim_units &lt;- sapply(doc$dataset$otherEntity, function(x) { i &lt;- datamgmt::which_in_eml(x$attributeList$attribute, &quot;standardUnit&quot;, as(&quot;dimensionless&quot;, &quot;standardUnit&quot;)) out &lt;- sapply(i, function(j){ list(entityName = x$entityName, attributeName = x$attributeList$attribute[[j]]$attributeName, attributeLabel = x$attributeList$attribute[[j]]$attributeLabel, attributeDefinition = x$attributeList$attribute[[j]]$attributeDefinition) }) return(out) }) dim_units &lt;- data.frame(t(do.call(cbind, dim_units))) dim_units If dimensionless units are found, first check the attributeDefinition, attributeLabel, and attributeName to see if dimensionless seems to be an appropriate unit for the attribute. If there is any question about whether or not dimensionless seems appropriate, ask another member of the data team. If there is still any question about what the unit should be, reach out to the submitter and ask for clarification. "],["list-submissions-to-the-arctic-data-center.html", "List submissions to the Arctic Data Center", " List submissions to the Arctic Data Center If the arctic bot fails to pick up a new submission, due to an invalid token or another error, we can check for submissions using a solr query. The list_submissions() function in the arcticdatautils package is a convenience wrapper around the solr query. If the function doesn’t specifically fit your needs it’s fairly straight-forward to modify the source code. Note: the function can be applied to any node, although it filters out results if the submitter is in the Arctic Data Center Admins group. Refer to the function documentation for more information. View submissions to the Arctic Data Center from 10/1/18 to 10/7/18 cn &lt;- dataone::CNode(&#39;PROD&#39;) adc &lt;- dataone::getMNode(cn,&#39;urn:node:ARCTIC&#39;) View(arcticdatautils::list_submissions(adc, &#39;2018-10-01&#39;, &#39;2018-10-07&#39;)) The following code shows the underlying solr query q = paste0(&#39;dateUploaded:[&quot;2018-10-01T00:00:00Z&quot; TO &quot;2018-10-08T00:00:00Z&quot;] AND formatType:*&#39;) results &lt;- dataone::query(adc, list(q = q, fl = &quot;identifier AND submitter AND dateUploaded AND formatType AND fileName&quot;, rows = 10000), as = &quot;data.frame&quot;) "],["recover-failed-submissions.html", "Recover failed submissions", " Recover failed submissions A submission may result in an error when the EML cannot be validated through EML::eml_validate, preventing it from being written to an indicated path. This usually occurs when the metacatUI detects a submission error and uploads the EML as a data object instead of text. When this happens, arcticdatautils::recover_failed_submission can be used to retrieve the original EML file. The function removes the error message to get a valid EML document. For the example below, pid is the metadata pid containing the failed submission. The path parameter indicates where you want to write the recovered EML version to on your server’s local working directory. path &lt;- &quot;path/to/save/eml.xml&quot; pid &lt;- &quot;your_metadata_pid&quot; arcticdatautils::recover_failed_submission(adc, pid, path) The output of this function is a valid EML file written to your chosen path. After recovering the document, make sure to set the rights and access to the correct submitter. Note that arcticdatautils::recover_failed_submission may not always work. Its functionality depends on the error message since they can vary. "],["data-portals.html", "Data Portals ", " Data Portals "],["customizing-data-portals.html", "Customizing Data Portals", " Customizing Data Portals Create a custom, branded portal for your research topic or lab group that spans datasets in the Arctic Data Center – available for all users. Branded portals provide a convenient, readily customized way to communicate your science, your team, your data, and related data from within the Arctic Data Center. The portals feature makes highlighting and sharing related datasets easy for researchers. General information on data portals can be found on the main page: https://arcticdata.io/data-portals/ My portals My portals allows members of the Arctic Data Center Team to view all the portals currently created. This can be found in Settings under the My Portals section. Creating Data Filters Useful search terms to use: NSF award number Geographic Region Keywords Access permissions ### Set Access Permissions For some portals such as PerCS portal, will need to be added by running set_access(). Please see the specific portal for the exact arguments to pass. Advanced portal customizations These requires the manual update of the portal document for features that are not available on the website yet. Please ask Jeanette for more assistance. For an example of a portal with all the features, see the SASAP Portal. Custom Filters Most os the time copy and pasting desired from the SASAP portal is a good place to start Tips - try to collapse the specific section and copy that to make sure you copy everything is needed Filter Groups Grouping multiple filters under one tab Possible fontawesome icons &lt;filterGroup&gt; &lt;label&gt;MOSAiC Terms&lt;/label&gt; &lt;description&gt;Terms in the MOSAiC ontology&lt;/description&gt; &lt;icon&gt;book&lt;/icon&gt; &lt;choiceFilter&gt;...&lt;/choiceFilter&gt; &lt;/filterGroup&gt; Individual Filters &lt;choiceFilter&gt; &lt;label&gt;Campaign&lt;/label&gt; &lt;!-- ====== equivalent to the fields available in the solr query ====== --&gt; &lt;field&gt;text&lt;/field&gt; &lt;operator&gt;OR&lt;/operator&gt; &lt;filterOptions&gt; &lt;placeholder&gt;Choose a campaign&lt;/placeholder&gt;&lt;icon&gt;sitemap&lt;/icon&gt; &lt;description&gt;The campaign number&lt;/description&gt; &lt;/filterOptions&gt; &lt;!-- ====== change these for different options ====== --&gt; &lt;choice&gt;&lt;label&gt;AF-MOSAiC-1&lt;/label&gt;&lt;value&gt;https://purl.dataone.org/odo/MOSAIC_00000020&lt;/value&gt;&lt;/choice&gt; &lt;choice&gt;&lt;label&gt;AT-MOSAiC-1&lt;/label&gt;&lt;value&gt;https://purl.dataone.org/odo/MOSAIC_00011201&lt;/value&gt;&lt;/choice&gt; &lt;chooseMultiple&gt;true&lt;/chooseMultiple&gt; &lt;/choiceFilter&gt; Custom Colour schemes Look for this section in the xml and replace the hex codes: &lt;option&gt; &lt;optionName&gt;primaryColor&lt;/optionName&gt; &lt;optionValue&gt;#3774b9&lt;/optionValue&gt; &lt;/option&gt; &lt;option&gt; &lt;optionName&gt;secondaryColor&lt;/optionName&gt; &lt;optionValue&gt;#82cdf5&lt;/optionValue&gt; &lt;/option&gt; &lt;option&gt; &lt;optionName&gt;accentColor&lt;/optionName&gt; &lt;optionValue&gt;#EB5638&lt;/optionValue&gt; &lt;/option&gt; People page also a formatted page - see the SASAP portal for details Updating portals For portals on DataOne - use the ucsb node. Portals made on the ADC use the same node as the datasets. library(dataone) library(EML) cn &lt;- CNode(&quot;PROD&quot;) ucsb &lt;- getMNode(cn, &quot;urn:node:mnUCSB1&quot;) Look for portals using query()so you can get the appropriate seriesID df &lt;- query(ucsb, list(q =&quot;formatId:*purl.dataone.org/portals-1.0.0 AND (*:* NOT obsoletedBy:*)&quot;, rows = &quot;300&quot;, fl = &quot;identifier,dateUploaded,formatId,obsoletedBy,origin,formatType,title, seriesId&quot;, sort = &#39;dateUploaded+desc&#39;), as = &quot;data.frame&quot;) Download the file and open to edit and add the filters and other options. id &lt;- df[df$title == &quot;Multidisciplinary drifting Observatory for the Study of Arctic Climate expedition&quot;, 2] #series ID writeBin(getObject(ucsb, id), &quot;file_mosaic.xml&quot;) #download Make whatever edits you need and upload the file. The identifier should be updated if successful but if there is an issue with the xml document go back and fix the issue before re-uploading again. sys &lt;- getSystemMetadata(ucsb, id) arcticdatautils::update_object(ucsb, sys@identifier, path = &quot;file_mosaic.xml&quot;, format_id = sys@formatId, sid = sys@seriesId) "],["distributed-biological-observatory-dbo-submissions.html", "Distributed Biological Observatory (DBO) submissions", " Distributed Biological Observatory (DBO) submissions DBO packages are organized under a project page (viewable here). In order to make sure that DBO datasets appear in the special DBO data search correctly, there are a few additional metadata requirements. give read and write access to the entire data package (using set_access) to the group: \"CN=DBO,DC=dataone,DC=org\" eg: `set_access(mn, pkg, “CN=DBO,DC=dataone,DC=org”, permissions = c(“read”, “write”, “changePermission”)) ensure that the name of the ship the data was collected from appears in the metadata record. This will likely be one of the following ship names. If the ship name is not in that list, let Jeanette or Dominic know Annika Marie Fairweather Healy Kromov Mirai Oscar Dyson Oshoro-Maru Sir Wilfrid Laurier Westward Wind Xue Long the geographic coverage should be one coverage per DBO transect sampled. The geographicDescription and bounding coordinates should be from the controlled vocabulary given in the code snippet below, which will create a data.frame with these values in your R environment geo_locs &lt;- dplyr::tribble(~siteNew, ~westBoundingCoordinate, ~eastBoundingCoordinate, ~northBoundingCoordinate,~ southBoundingCoordinate, &quot;DBO 1 - South of St. Lawrence Island, Bering Sea&quot;, -176.147, -172.187, 63.769, 61.847, &quot;DBO 2 - Chirikov Basin, Northern Bering Sea&quot;, -170.492, -167.86, 65.111, 64.482, &quot;DBO 3 - Southern Chukchi Sea&quot;, -171.419, -166.481, 68.609, 66.752, &quot;DBO 4 - Northeast Chukchi Sea&quot;, -164.553, -160.507, 71.867, 70.682, &quot;DBO 5 - Barrow Canyon, Chukchi Sea&quot;, -158.848, -155.931, 71.808, 71.111, &quot;DBO 6 - Beaufort Sea - 152 W&quot;, -153.865, -150.976, 72.119, 70.89) one of the following key phrases referencing the data type should exist in the dataset. If the dataset doesn’t seem to fit in one of these categories, ask Jeanette or Dominic ADCP benthic macroinfaunal chlorophyll CTD CPUE (referencing fish catch per unit effort) marine mammal nutrient sediment zooplankton After you have updated your dataset, navigate to https://arcticdata.io/catalog/projects/DBO/data and make sure that the different filters work as expected. Your dataset should be discoverable in some way through all of the filters. "],["mosaic.html", "MOSAiC", " MOSAiC Look out for datasets that are part of the MOSAiC expedition from 2019 -2020. There are a couple of special steps for datasets from this project. When working on MOSAiC datasets: Find out the EventId from the researcher to annotate (Campaign can be derived from the eventLabel). Ask for it in your email: We would like to ask for the event label associated with this dataset (see https://www.pangaea.de/expeditions/events/PS122%2F4). Often times researchers do not have a full list of event labels, but instrument names are provided in the dataset metadata. In these cases, try to find the correct event labels by searching through the event lists and finding matching instrument names and dates. Each MOSAiC campaign has its own event list. Here is an example of an event list for the first cruise by the Polarstern research vessel. Find the appropriate dataset and attribute level annotations There are functions in arcticdatautils to help with annotating: mosaic_annotate_dataset and mosaic_annotate_attribute The custom ontology for the datasets are currently in the sem-prov-ontologies repository but you won’t need to use it directly if the function works. Use EML::shiny_attributes only when no annotations have been added. The function does not work well with multiple annotations Please continue to annotate the measurement types using containst measurements of type for all other attributes The portal is on DataONE. Make sure the dataset(s) show up in the portal after you add the annotations. Here are two fully annotated datasets for reference: Simple example Example with MOSAiC and ECSO annotations Example with non-polarstern vessels The following shows how to add the annotations using arcticdatautils and manually in the case the functions fail to work: Dataset Level Annotations There are 5 main campaigns in the MOSAiC expedition. The main campaigns follow the pattern PS122/#. The full campaign list is easiest to view on the PANGAEA website. The first two letters of each campaign name correspond with the ship or station name (ex: PS = Polarstern). arcticdatautils doc$dataset$id &lt;- &quot;your id here&quot; doc$dataset$annotation &lt;- NULL #make sure it is empty doc$dataset$annotation &lt;- mosaic_annotate_dataset(c(&quot;PS122/1&quot;, &quot;PS122/2&quot;)) Manual Majority of the annotations will be the same across all of the datasets. Pay attention to what needs to be changed for the hasBasis and isPartOfCampaign valueURIs. More than one Campaign and Basis might be needed in rare cases. doc$dataset$id &lt;- &quot;your id here&quot; doc$dataset$annotation &lt;- NULL #make sure it is empty here doc$dataset$annotation &lt;- list( #Basis list( propertyURI = list(label = &quot;hasBasis&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000034&quot;), valueURI = list(label = &quot;Polarstern&quot;, # this depends on your event ID, most of them should be Polarstern valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000030&quot;) ), #Project list( propertyURI = list(label = &quot;hasProjectLabel&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000025&quot;), valueURI = list(label = &quot;MOSAiC20192020&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000023&quot;) ), #Campaign list( propertyURI = list(label = &quot;isPartOfCampaign&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000032&quot;), valueURI = list(label = &quot;PS122/2&quot;, #*** this changes depending on the campaign valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00000018&quot;) ) #*** this changes depending on the campaign ) Attribute Level Annotations Apply these to the attribute level of the dataset. Event IDs should have already be identified earlier on in the dataset review. Check the ticket overview for details. arcticdatautils doc$dataset$dataTable[[1]]$attributeList$attribute[[1]]$id &lt;- &quot;some id here&quot; doc$dataset$dataTable[[1]]$attributeList$attribute[[1]]$annotation &lt;- mosaic_annotate_attribute(&quot;PS122/2_14-270&quot;) Manual event_annotation &lt;- list(#Event Label list( propertyURI = list(label = &quot;wasGeneratedBy&quot;, propertyURI = &quot;http://www.w3.org/ns/prov#wasGeneratedBy&quot;), valueURI = list(label = &quot;PS122/2_14-270&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00004550&quot;) ), #Method/Device - Use the long name! list( propertyURI = list(label = &quot;deployedSystem&quot;, propertyURI = &quot;https://purl.dataone.org/odo/MOSAIC_00002201&quot;), valueURI = list(label = &quot;Ultra-Wideband Software-defined Microwave Radiometer (0.5-2GHZ)&quot;, valueURI = &quot;https://purl.dataone.org/odo/MOSAIC_00001163&quot;) )) doc$dataset$dataTable$attributeList$attribute[[1]]$id &lt;- &quot;eventId&quot; doc$dataset$dataTable$attributeList$attribute[[1]]$annotation &lt;- event_annotate Adding annotations If there are already annotations applied (ie. using ECSO), the MOSAiC annotations can be added as so: get the index of all of the attributes we want to add this annotation n &lt;- which(stringr::str_detect(eml_get_simple(doc$dataset$dataTable$attributeList, &quot;attributeName&quot;),&quot;Brightness&quot;)) loop through and add the annotation(s) for(i in n) { doc$dataset$dataTable$attributeList$attribute[[i]]$annotation &lt;- append( list( doc$dataset$dataTable$attributeList$attribute[[i]]$annotation ), event_annotation ) } Portal updates The filters are based on the MOSAiC ontology. The Campaign and Basis filters should remain relatively the same. However there are over 500 Method and Devices. So the decision was made to only show the ones we have annotated. We can query for all of them and output them formatted as a choice: mosaic_portal_filter(&quot;Method/Device&quot;) Then update the portal document using the steps outlines in the advanced data portal customizations "],["using-arcticdatautils.html", "Using arcticdatautils", " Using arcticdatautils Please use datapack and dataone for regular data processing. This information is kept for troubleshooting and legacy purposes. "],["create-a-resource-map-using-arcticdatautils.html", "Create a resource map using arcticdatautils", " Create a resource map using arcticdatautils If you are creating a new data package, you must create a resource map. Resource maps provide information about the resources in the data package (i.e. which data files are associated with a particular metadata file (xml)). Before creating a new resource map, it is best to ensure that all the objects of interest have already been published to the node. Therefore, if you’ve only run publish_object() on the xml (in section 2.4 above), you will also need to publish all the data objects you wish to associate with the xml. The resource map you are about to create controls these linkages. Once all the objects that ought to belong to the package have been published, create a resource map using their respective PIDs (or the variables you saved the PIDs to) via the arcticdatautils::create_resource_map() function, like so: resource_map_pid &lt;- create_resource_map(adc_test, metadata_pid = metadata_pid, data_pids = data_pids) "],["get-package-and-eml.html", "Get package and EML", " Get package and EML Before we look more in depth at EML, we first need to load your data package into R. After setting your node, use the following commands to load the package: rm_pid &lt;- &quot;your_resource_map_pid&quot; pkg &lt;- get_package(adc_test, rm_pid, file_names = TRUE) Keeping track of PIDs In all subsequent sections, if they ask for the pid of the metadata, reasource map or data you can refer to the object from get_package(). For example, if you assigned the value from get_package() as above as pkg then you can refer to the following by using the corresponding commands: metadata_pid &lt;- pkg$metadata data_pid &lt;- pkg$data resource_pid &lt;- pkg$resource_map After loading the package, you can also load the EML file into R using the following command: doc &lt;- read_eml(getObject(adc_test, pkg$metadata)) Note that we named the object doc. This is a good generic name to use for EML documents. The generic name eml should not be used - as the EML package ships with an eml function, which can cause namespace issues in your environment if you have an object also called eml. Tip to always have the most recent resource map. When editing data packages, you always want to be working with the most recent update. To ensure you have the most recent resource map, you can use the following commands: rm_pid_original &lt;- &quot;your_original_resource_map_pid&quot; all_rm_versions &lt;- get_all_versions(adc_test, rm_pid_original) rm_pid &lt;- all_rm_versions[length(all_rm_versions)] print(rm_pid) "],["publish-an-object.html", "Publish an object", " Publish an object Objects (data files, xml metadata files) can be published to a DataONE node using the function publish_object() from the arcticdatautils R Package. To publish an object, you must first get the formatId of the object you want to publish. A few common formatIds are listed below. # .csv file formatId &lt;- &quot;text/csv&quot; # .txt file formatId &lt;- &quot;text/plain&quot; # metadata file formatId &lt;- &quot;https://eml.ecoinformatics.org/eml-2.2.0&quot; # OR formatId &lt;- format_eml(&quot;2.2.0&quot;) Most objects have registered formatIds that can be found on the DataONE Object Format List here. Always use the “Id:” (2nd line) from the DataONE Object Format List and ensure it is copied EXACTLY. Metadata files (as shown above) use a special function to set the formatId. If the formatId is not listed at the DataONE website, you can set formatId &lt;- \"application/octet-stream\". If you want to change the formatId please use updateSystemMetadata instead. Once you know the appropriate formatId you can publish an object using these commands: path &lt;- &quot;path/to/your/file&quot; formatId &lt;- &quot;your/formatId&quot; pid &lt;- publish_object(adc_test, path = path, format_id = formatId) # note that the output of this function is the PID of the newly published object After publishing the object, the PID will need to be added to a resource map by updating or creating a resource map. Additionally, the rights and access for the object must be set. However, you only need to give other people rights and access to objects to objects that are not yours, for the training you don’t need to do this. "],["reorder-entities.html", "Reorder entities", " Reorder entities This is easier to accomplish using arcticdatautils An example reorganizing alphabetically Metadata doc$dataset$otherEntity &lt;- doc$dataset$otherEntity[order(entity_names)] Data files and PIDs retrieved with get_package pkg &lt;- get_package(adc, rm, file_names = T) doc &lt;- EML::read_eml(getObject(adc, pkg$metadata)) reordered &lt;- arcticdatautils::reorder_pids(pkg$data, doc) "],["nesting-data.html", "Nesting Data ", " Nesting Data "],["nesting-a-data-package.html", "Nesting a data package", " Nesting a data package Data packages on member nodes can exist as independent packages or in groups (nested data packages). Much like we can group multiple data files together with a common metadata file, we can group related data packages together with a common “parent” data package. The structure of nested data packages resembles a pyramid. There is one top level, or “parent”, with one or more data packages, or “child packages”, nested beneath it. There is no limit to how many nested levels can be created, but packages do not generally exceed 3 levels. This “grandparent” has 5 child packages (nested datasets), all of which have child packages of their own. Here are some common uses for nesting: collected data vary by year an NSF award funds several related projects data collection is still ongoing data files exceed the 1000 file limit per data package "],["add-children-to-an-existing-parent.html", "Add children to an existing parent", " Add children to an existing parent A new package is published with a DOI and needs to be nested underneath a pre-existing parent. Nest the new child using the child_pids argument in publish_update(). resource_map_child_new &lt;- &quot;some_child_resource_map_pid&quot; pkg_parent &lt;- get_package(mn, &#39;resource_map_parent&#39;) publish_update(mn, resource_map_pid = pkg_parent$resource_map, metadata_pid = pkg_parent$metadata, data_pids = pkg_parent$data_pids, # parents usually don&#39;t contain data, but better to be safe child_pids = c(pkg_parent$child_packages, resource_map_child_new)) # include the resource map PIDs of ALL the childs* in the `child_pids` argument, otherwise the nesting relationships between any omitted childs and the parent will be deleted Check through all arguments carefully before you publish to production! Do you need to update the metadata? Does the parent include data objects? Does the parent have a parent? Parents can be tricky to fix and work with (especially if they have serial identifiers (SIDs)), so if you’re not sure how something works, try it on a test node. "],["create-a-new-parent-package.html", "Create a new parent package", " Create a new parent package In some cases, a parent package already exists. Search the ADC for the NSF award number to see if there are already exisiting packages. Parents usually have a UUID rather than a DOI and often start with a title like “Collaborative research:”, but not always. More typically, you will need to create a new parent by editing the existing metadata. The parent package should contain a generalized summary for the metadata of each of its childs. To create a new parent, you will need to: Create parent metadata. It’s often easiest to start with a child’s metadata and generalize them. Abstract/title: Remove dates and other details that are specific to the child package. Sometimes the NSF award abstract/ title will work. Data tables/other entities: Generally, top-level parents do not include data objects, so these sections can be removed. Geographic coverage: Expand to include geographic coverage of all childs, if needed. Temporal coverage: Expand to include temporal ranges of all childs, if needed. If the study is ongoing, include the most recent end date; the parent can be updated when additional childs are added. Methods: Often not needed, but may be included if all childs use the same methods. Publish the parent metadata to the member node (ADC) using publish_object(). Create a resource map to link the parent and childs together using create_resource_map() and the child_pids argument. Make sure you use the childs’ resource map PIDs when you create the resource map! If you forgot to do so, consult Jeanette for help fixing it. "],["example.html", "Example", " Example We can start by creating two data packages on the test node to nest beneath a parent. These data packages contain measurements taken from Lake E1 in Alaska in 2013 and 2014. First, load the Arctic Data Center Test Node and libraries. library(dataone) library(arcticdatautils) library(EML) cn_staging &lt;- CNode(&#39;STAGING&#39;) adc_test &lt;- getMNode(cn_staging,&#39;urn:node:mnTestARCTIC&#39;) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) We will re-create the following parent package: https://arcticdata.io/catalog/#view/urn:uuid:799b7a86-cb1c-497c-a05a-d73492915cad on the test node with two of its children. First we will copy two of the children to the test node, make sure your token for the test node is not expired. from &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) to &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) child_pkg_1 &lt;- datamgmt::clone_package(&#39;resource_map_doi:10.18739/A2KS1R&#39;, from = from, to = to, add_access_to = arcticdatautils:::get_token_subject(), change_auth_node = TRUE, new_pid = TRUE) child_pkg_2 &lt;- datamgmt::clone_package(&#39;resource_map_doi:10.18739/A2QK29&#39;, from = from, to = to, add_access_to = arcticdatautils:::get_token_subject(), change_auth_node = TRUE, new_pid = TRUE) These two packages correspond to data from the same study, varying only by year; however, they currently exist on the test node as independent entities. We will associate them with each other by nesting them underneath a parent. Now, let’s create a parent metadata file. Read in one of the childs’ metadata files (EML). We can download object from a node in binary format using dataone::getObject(). Once it’s downloaded we just need to convert to it to the proper format: in this case to EML format using EML::read_eml(). doc_parent &lt;- read_eml(getObject(adc_test, child_pkg_1$metadata)) ## View the title doc_parent$dataset$title The title of this child contains “2012-2013”. This is too specific for the parent, as the temporal range of both childs is 2012-2014. The parent should encompass this larger time range. doc_parent$dataset$title &lt;- &#39;Time series of water temperature, specific conductance, and oxygen from Lake E1, North Slope, Alaska, 2012-2014&#39; Like the title, the temporal coverage elements in this EML need to be adjusted. new_end_date &lt;- &quot;2014-09-20&quot; doc_parent$dataset$coverage$temporalCoverage$rangeOfDates$endDate$calendarDate &lt;- new_end_date Remove dataTables and otherEntitys from the metadata. If you recall from previous chapters, dataTables contain metadata associated with data files (generally CSVs) and otherEntitys contain metadata about any other files in the data package (for instance a README or coding script). Because the parent does not contain any data objects, we want to remove dataTables and otherEntitys from the metdata file. In this instance, the E1 2013 metadata only contain dataTables. We can remove these by setting the dataTable element in the EML to a new blank object. doc_parent$dataset$dataTable &lt;- NULL In this case, the abstract, contacts, creators, geographicDescription, and methods are already generalized and do not require changes. Before writing your parent EML make sure that it validates. This is just a check to make sure everything is in the correct format. eml_validate(doc_parent) After your EML validates we need to save, or “write”, it as a new file. Write your parent EML to a directory in your home folder. You can view this process like using “Save as” in Microsoft Word. We opened a file (“E1_2013.xml”), made some changes, and “saved it as” a new file called “doc_parent.xml”. # We can save the eml in a temporary file eml_path &lt;- file.path(tempdir(), &#39;science_metadata.xml&#39;) write_eml(doc_parent, path) Next, we will publish the parent metadata to the test node. metadata_parent &lt;- publish_object(adc_test, path = eml_path, format_id = format_eml()) Finally, we create a resource map for the parent package. We nest the two child data packages using the child_pids argument in create_resource_map(). Note that these child_pids are PIDs for the resource maps of the child packages, NOT the metadata PIDs. resource_map_parent &lt;- create_resource_map(adc_test, metadata_pid = metadata_parent, child_pids = c(child_pkg_1$resource_map, child_pkg_2$resource_map)) The child packages are now nested underneath the parent. "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]
diff --git a/training/editing-eml.html b/training/editing-eml.html
index 155e16fb..da451679 100644
--- a/training/editing-eml.html
+++ b/training/editing-eml.html
@@ -480,7 +480,7 @@ <h3><span class="header-section-number">4.3.1</span> Edit attributes<a href="edi
 <li><em>ordinal</em>: ordered categories. e.g.: Low, Medium, High</li>
 <li><em>dateTime</em>: date or time values from the Gregorian calendar. e.g.: 01-01-2001</li>
 <li><em>ratio</em>: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second.</li>
-<li><em>interval</em>: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or latitude and longitude coordinates, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius.</li>
+<li><em>interval</em>: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius.</li>
 </ul></li>
 <li><strong>domain</strong>: One of: <code>textDomain</code>, <code>enumeratedDomain</code>, <code>numericDomain</code>, <code>dateTime</code>. Required.
 <ul>
diff --git a/training/search_index.json b/training/search_index.json
index 1c7c943a..29fea17c 100644
--- a/training/search_index.json
+++ b/training/search_index.json
@@ -1 +1 @@
-[["index.html", "NCEAS Data Team Training Welcome to NCEAS! First day to-dos Account information NCEAS events NCEAS Community Meetings/ Groups Internship Expectations Code of Conduct Citation Information", " NCEAS Data Team Training Jesse Goldstein, Dominic Mullen, Jeanette Clark, Irene Steves, Mitchell Maier, Bryce Mecum, Peter Slaughter, Emily O’Dean, Jasmine Lai, Daphne Virlar-Knight, Maggie Klope 2024-08-13 Welcome to NCEAS! First day to-dos Get a tour of the office from Ginger Fill out required paperwork you received via email Set up the remainder of your accounts Account information LDAP - this should be set up prior to your start date in order to help get other accounts set. This account and password control your access to: arcticdata RT queue GitHub Enterprise - arctic-data Datateam server - follow instructions in email from Nick at NCEAS to reset your datateam password in the terminal ORCiD - create an account Login to test.arcticdata.io with your ORCID iD NCEAS Slack - get an invite from slack.nceas.ucsb.edu Channels to join: #arctica, #arcticbot, #computing, #datateam, #devteam Introduce yourself in #datateam Arctic Data Center Team - after creation of ORCiD and sign-in to both arcticdata.io and test.arcticdata.io, request addition to the admin group from Justin or Maggie in Slack GitHub - if you do not have a public GitHub account already, please register for one here If you are an intern, fill out anticipated quarterly schedule on the intern google calendar shared with you. Electronic Timekeeping - make sure you can log on to electronic timekeeping via your UCSBNetID and password (may not be accessible on the first day, if you continue to have issues please let Ana know). If you are an hourly employee, log your hours for your first day! Under today’s date select ‘Hours Worked’ under the Pay Code column, enter the amount of hours under the Amount column, and finally click the ‘Save’ button in the top right. At the end of every two-week pay period you will also need to click the ‘Approve Timecard’ button in order to submit your timecard. UCPath - you can set up your paycheck preferences here, including direct deposit and income tax withholding. Detailed Instructions Let Justin or Maggie know what email you would like to use for general NCEAS updates from all@nceas.ucsb.edu NCEAS events NCEAS hosts a number of events that you are encouraged to attend. Keep an eye on your email but the recurring events are: Roundtable presentation and discussion of research by a visiting or local scientist first or second Thursdays of the month at 3:30 in the lounge and via zoom followed by happy hour at 4:30 on the terrace Coffee Klatch coffee, socializing, and news updates for NCEAS Tuesdays at 10:30 in the lounge NCEAS Community Meetings/ Groups Check out their individual calendar entries and channels for more information Early Career Researcher Community Forum - #ecr_community Hacky Hours - #hackyhour Data Science Chats - #data-science-chats NCEAS Diversity Team NCEAS Book Club - #bookclub NCEAS Social - #social Internship Expectations As an intern with the data team, there are a few expectations that the Project Coordinators have of you. Overall, we expect you to be communicative and proactive. We want you to learn and grow in this position, but we don’t want you spinning your wheels going nowhere fast! If you’ve spent 10-15 minutes on an issue and you’re not making any progress, reach out to us and your peers for help in the #datateam slack channel. The #datateam slack channel is our main form of communication, and we expect all interns to become comfortable communicating in this space. By posting your questions and code in the #datateam channel (instead of sending direct messages), multiple people will be able to help at once, and we all can learn from the problems that our peers encounter. Additionally, we expect interns to work within the standard business hours of 8am - 5pm (pacific time). We ask that you mark your expected work hours on the shared “Intern” Google Calendar. This is so that the Project Coordinators are aware of who’s working day-to-day and can plan their days accordingly. We also use this to verify time sheets when they are submitted. Ideally, interns would input their proposed hours on the calendar at least one week in advance. During exams and other unusually busy weeks at school, we understand you may need to shift your hours or reduce your workload. When this occurs, please make sure to email either Daphne or Jeanette so that we know not to expect you during your usual schedule. Code of Conduct Take a moment to review the diversity and inclusion page and code of conduct at NCEAS so that we can foster an environment that is safe, welcoming and inclusive for everyone. Citation Information This work is licensed under a Creative Commons Attribution 4.0 International License. Clark, Jeanette, Jesse Goldstein, Dominic Mullen, Irene Steves, Mitchell Maier, Stephanie Freund, Sharis Ochs, Bryce Mecum, Peter Slaughter, Emily O’Dean, Jasmine Lai, Daphne Virlar-Knight. 2022 Training Materials for the Arctic Data Center Curation Team. Arctic Data Center. doi:10.18739/A20G3GX8W. "],["introduction-to-open-science.html", "Chapter 1 Introduction to open science 1.1 Background reading 1.2 Effective data management 1.3 Using DataONE 1.4 Working on a remote server 1.5 A note on paths 1.6 A note on R 1.7 A note on effective troubleshooting in R 1.8 A note on Exercises 1.9 Exercise 1", " Chapter 1 Introduction to open science These materials are meant to introduce you to the principles of open science, effective data management, and data archival with the DataONE data repository. It also provides an overview on the tools we will be using (remote servers, Rstudio, R, Troubleshooting, Exercises) throughout the training. This document is meant to take multiple days to complete, depending on your previous knowledge. We believe in allowing employees the space to fully grasp concepts during training, even if it means taking longer than expected. Quality learning is our priority, and there’s no pressure to finish within a specific timeframe. You may find it helpful to take notes on important concepts, and you will always be able to refer back to this training during your time at NCEAS. If you see anything that needs fixing, submit a issue in the github issues 1.1 Background reading Read the content on the Arctic Data Center (ADC) webpage to learn more about data submission, preservation, and the history of the ADC. We encourage you to follow the links within these pages to gain a deeper understanding. about submission preservation history 1.2 Effective data management Read Matt Jones et al.’s paper on effective data management to learn how we will be organizing datasets prior to archival. (Please note that while the tips outlined in this article are best practices, we often do not reformat data files submitted to our repositories unless necessary. It is best to be conservative and not alter other people’s data without good reason.) You may also want to explore the DataONE education resources related to data management. 1.3 Using DataONE Data Observation Network for Earth (DataONE) is a community driven initiative that provides access to data across multiple member repositories, supporting enhanced search and discovery of Earth and environmental data. Read more about what DataONE is here and about DataONE member node (MN) guidelines here. Please feel free to ask your supervisor any questions you have about DataONE. We will be applying these concepts in the next chapter. 1.4 Working on a remote server All of the work that we do at NCEAS is done on our remote server, datateam.nceas.ucsb.edu. If you have never worked on a remote server before, you can think of it like working on a different computer via the internet. We access RStudio on our server through this link. This is the same as your desktop version of RStudio with one main difference is that files are on the server. Please do all your work here, and bookmark this link. Do not use RStudio on your local computer. By only using the RStudio server, it is easier to share your code with the rest of us. 1.4.1 Check your understanding Open a new tab in your browser and try logging into the remote server using your SSH credentials. If your R session is frozen and unresponsive check out the guide on how to fix it. 1.5 A note on paths On the servers, paths to files in your folder always start with /home/yourusername/.... Note - if you are a more advanced user, you may use the method you prefer as long as it is evident where your file is from. When you write scripts, try to avoid writing relative paths (which rely on what you have set your working directory ~/ to) as much as possible. Instead, write out the entire path as shown above, so that if another data team member needs to run your script, it is not dependent on a working directory. 1.6 A note on R This training assumes basic knowledge of R and RStudio. Spend at least 15 minutes walking through Jenny Bryan’s excellent materials here for a refresher. Throughout this training we will occasionally use the namespace syntax package_name::function_name() when writing a function. This syntax denotes which package a function came from. For example dataone::getSystemMetadata selects the getSystemMetadata function from the dataone R package. More detailed information on namespaces can be found here. 1.7 A note on effective troubleshooting in R One of the advantages with using the R programming language is the extensive documentation that is available for R packages. The R help operator ? can be used to learn more about functions from all of the R packages we use. You can put the operator before the name of any function to view its documentation in RStudio: ?function When asking for help in the #datateam channel in Slack, we suggest using a combination of minimal reproducible examples (MRE) and the package reprex to create reproducible examples. This will allow others to better help you if we can run the code on our own computers. A MRE is stripping down your code to only the parts that cause the bug. When troubleshooting errors over Slack, send the code that returned an error and the error message itself. How to generate a reprex: copy the code you want to ask about call reprex() fix until everything runs smoothly copy the result to ask your question When copy and paste code slack message or github issues, use three backticks for code blocks and two backticks for a small piece of code will prevent issues with slack formats quotation. For more information and examples check out more of Jenny Bryan’s slides or watch the video starting at about the 10 min mark. Note for EML related MREs: - Generating a reprex for these situations (ie. tokens) might be complicated but you can should still follow the MRE principles even if the reprex won’t render fully - You can include a minimal EML to avoid some get_package issues: me &lt;- list(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;)) attributes &lt;- data.frame(attributeName = &#39;length_1&#39;, attributeDefinition = &#39;def1&#39;, measurementScale = &#39;ratio&#39;, domain = &#39;numericDomain&#39;, unit = &#39;meter&#39;, numberType = &#39;real&#39;) att_list &lt;- set_attributes(attributes) doc_ex &lt;- list(packageId = &quot;id&quot;, system = &quot;system&quot;, dataset = list(title = &quot;A Minimal Valid EML Dataset&quot;, creator = me, contact = me, dataTable = list(entityName = &quot;data table&quot;, attributeList = att_list)) ) 1.8 A note on Exercises The rest of the training has a series of exercises. These are meant to take you through the process as someone submitting a dataset from scratch. This is slightly different than the usual workflow but important in understanding the underlying system behind the Arctic Data Center. Please note that you will be completing everything on the site for the training. In the future if you are unsure about doing anything with a dataset. The test site is a good place to try things out! 1.9 Exercise 1 This part of the exercise walks you through submitting data through the web form on “test.arcticdata.io”. In addition to learning to use the webform, this exercise will also help you practice sleuthing for information in order to provide complete metadata. Most datasets do not come with all contextual information, so you will need to skim cited literature and search google for definitions of discipline-specific jargon. Don’t be afraid to use the internet as a resource! 1.9.1 Part 1 Download the csv of Table 1 from this paper. Reformat the table to meet the guidelines outlined in the journal article on effective data management (this might be easier to do in an interactive environment like Excel). Hint: This table is in wide format and can be made longer. Note: we usually don’t edit the content in data submissions so don’t stress over this part too much 1.9.2 Part 2 Go to “test.arcticdata.io” and submit your reformatted file with appropriate metadata that you derive from the text of the paper: List yourself as the first ‘Creator’ so your test submission can easily be found. For the purposes of this training exercise, not every single author needs to be listed with full contact details, listing the first two authors is fine. Directly copying and pasting sections from the paper (abstract, methods, etc.) is also fine. Attributes (column names) should be defined, including correct units and missing value codes. Click “describe” to the right of the file name in order to add file-specific information. The title and description can be edited in the “Overview” tab, while attributes are defined in the “Attributes” tab. Submit the dataset Post a message to the #datateam Slack channel with a link to your package. "],["creating-a-data-package.html", "Chapter 2 Creating a data package 2.1 What is in a package? 2.2 Packages on the Website 2.3 About identifiers 2.4 Upload a package 2.5 datapack Background 2.6 Exercise 2a 2.7 Create a new data package 2.8 Exercise 2b", " Chapter 2 Creating a data package This chapter will teach you how to create and submit a data package to a DataONE MN via R. But first, please read this paper on the value of structured metadata, namely the Ecological Metadata Language (EML). 2.1 What is in a package? A data package generally consists of at least 3 components. Metadata: One object is the metadata file itself. In case you are unfamiliar with metadata, metadata are information that describe data (e.g. who made the data, how were the data made, etc.). The metadata file will be in an XML format, and have the extension .xml (extensible markup language). We often refer to this file as the EML (Ecological Metadata Language), which is the metadata standard that it uses. Each dataset page in the Arctic Data Center is a direct representation of an EML document, made to look prettier for the web. Data: Other objects in a package are the data files themselves. Most commonly these are data tables (.csv), but they can also be audio files, NetCDF files, plain text files, PDF documents, image files, etc. Resource Map: The final object is the resource map. This object is a plain text file with the extension .rdf (Resource Description Framework) that defines the relationships between all of the other objects in the data package. You can think of it like a “basket” that holds the metadata file and all data files together. It says things like “this metadata file describes this data file,” and is critical to making a data package render correctly on the website. Fortunately, we rarely, if ever, have to actually look at the contents of resource maps; they are generated for us using tools in R. From the DataOne Community Meeting (Session 7) 2.2 Packages on the Website All of the package information is represented when you go to the landing page for a dataset. In the previous section, you uploaded a data file and made edits to the metadata using the web editor. When you make changes to the metadata and data files through R, those published changes will also be reflected here. 2.3 About identifiers Each object (metadata files, data files, resource maps) on the ADC or the KNB (another repo) has a unique identifier, also sometimes called a “PID” (persistent identifier). When you look at the landing page for a dataset, for example here, you can find the resource map identifier listed under the title in the gray bar after the words “Files in this dataset Package:” (resource_map_doi:10.18739/A2836Z), the metadata identifier in the “General &gt; Identifier” section of the metadata record or after the title with blue font (doi:10.18739/A2836Z), and the data identifier by clicking the “more info” link next to the data object, and looking at the “Online Distribution Info” section (arctic-data.9546.1). Note, all datasets submitted are given a preliminary identifier (usually starting with urn:uuid:). When the dataset is finalized, a doi will be issued. Different versions of a package are linked together by what we call the “version chain” or “obsolescence chain”. Making an update to a data package, such as replacing a data file, changing a metadata record, etc, will result in a new identifier for the new version of the updated object. When making changes to a package, always use datapack::uploadDataPackage() for updating the entire package on the latest versions of all objects to ensure that the version chain is maintained. 2.4 Upload a package We will be using R to connect to the NSF Arctic Data Center (ADC) data repository to push and pull edits in actual datasets. To identify yourself as an admin you will need to pass a ‘token’ into R. Do this by signing in to the ADC with your ORCid and password, then hovering over your name in the top right corner and clicking on “My profile”, then navigating to “Settings” and “Authentication Token”, copying the “Token for DataONE R”, and finally pasting and running it in your R console. The console is the bottom left window in RStudio. This token is your identity on these sites, please treat it as you would a password (i.e. don’t paste into scripts that will be shared). The easiest way to do this is to always run the token in the console. There’s no need to keep it in your script since it’s temporary anyway. You will need to retrieve a new one after it either expires or you quit your R session. Setting the token does not produce any output in the console. If the token is not set or is set incorrectly, you will know when an error is produced after trying to load a private dataset. Sometimes you’ll see a placeholder in scripts to remind users to get their token, such as: options(dataone_test_token = &quot;...&quot;) Since we will be working on the test site and not the production site, please remember to get your token from test.arcticdata.io Next, please be sure these packages are loaded for the training (these should already exist if you are working on the server): library(devtools) library(dataone) library(datapack) library(EML) library(remotes) library(XML) library(uuid) If any package could not be loaded, use the following command (replacing package_name with the actual package name) to install the package, then load them. install.packages(&quot;package_name&quot;) Now you’ll install the arcticdatautils and datamgmt packages with the code below. If prompted to update packages during the installation process, skip the updates. Now, run the following code to install and load the libraries. remotes::install_github(&quot;nceas/arcticdatautils&quot;) library(arcticdatautils) remotes::install_github(&quot;nceas/datamgmt&quot;) library(datamgmt) When you are usually working with data packages you will only need the following: library(dataone) library(datapack) library(EML) library(arcticdatautils) For this training, we will be working exclusively on the Arctic test site, or “node.” In many of the functions you will use this will be the first argument. It is often referred to in documentation as mn, short for member node. Different repositories use different member nodes. More information on the other nodes can be found in the reference section under Set DataONE nodes Set DataONE nodes For example, if we are using the test site, set the node to the test Arctic node: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Once all set up you can first publish an object (data) If you are curious how everything magically works, here is a handy diagram: From the DataOne Community Meeting (Session 7) 2.5 datapack Background adapted from the dataone and datapack vignettes datapack is written differently than most R packages you may have encountered in the past. This is because it uses the S4 system instead. library(dataone) library(datapack) library(uuid) Data packages Data packages are a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. 2.5.1 Navigating data packages Nodes Using this example on arcticdata.io d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) To use the member node information, use the mn slot d1c_test@mn To access the various slots using objects created by datapack and dataone (e.g. getSystemMetadata) requires the @ which is different from what you might have seen in the past. This is because these use the S4 system. Get an existing package from the Arctic Data Center. Make sure you know as you go through this training whether you are reading or writing to test or production. We don’t want to upload any of your test datasets to production! d1c &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) dp &lt;- dataone::getDataPackage(d1c, &quot;resource_map_urn:uuid:1f9eee7e-2d03-43c4-ad7f-f300e013ab28&quot;) 2.5.2 Data Objects You can see what slots are in an S4 object after typing the subsetting operator @, or pressing TAB with the cursor after an existing @. Try viewing the slots of the data package. dp@ Check out the objects slot dp@objects The objects slot contains a list of object PIDs that are accessed using the $ subsetting operator. Both are found within the structure of data packages in R. Get the number for data and metadata files associated with this data package: getSize(dp) Get the file names and corresponding pids getValue(dp, name=&quot;sysmeta@fileName&quot;) Get identifiers You can search by any of the sysmeta slots such as fileName and formatId and get the corresponding identifier(s): metadataId &lt;- selectMember(dp, name=&quot;sysmeta@ADD THE NAME OF THE SLOT&quot;, value=&quot;PATTERN TO SEARCH BY&quot;) Example: selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;image/tiff&quot;) selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&quot;filename.csv&quot;) 2.5.3 Provenance View the provenance as a dataTable. We will get into detail in the Building provenance chapter. dp@relations$relations 2.6 Exercise 2a Select a dataset from the catalog on the Arctic Data Center. Observe the number of data files in the dataset. Try to find identifiers for the metadata file and resource map on the landing page for the dataset based on the screenshot shown above. 2.7 Create a new data package adapted from the dataone and datapack vingettes library(dataone) library(datapack) library(uuid) Create a new data package - data package is a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. dp &lt;- new(&quot;DataPackage&quot;) 2.7.1 Upload new data files 2.7.1.1 Create and add a metadata file In this example we will use this previously written EML metadata. Here we are getting the file path from the dataone package and saving that as the object emlFile. This is a bit of an unusual way to reference a local file path, but all this does is looks within the R package dataone and grabs the path to a metadata document stored within that package. If you print the value of emlFile you’ll see it is just a file path, but it points to a special place on the server where that package is installed. Usually you will just reference EML paths that are stored within your user file system. emlFile &lt;- system.file(&quot;extdata/strix-pacific-northwest.xml&quot;, package=&quot;dataone&quot;) Create a new DataObject for the metadata and add it to the package. metadataObj &lt;- new(&quot;DataObject&quot;, format=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;, filename=emlFile) dp &lt;- addMember(dp, metadataObj) Check the dp object to see if the metadata was added correctly. dp 2.7.1.2 Add some additional data files sourceData &lt;- system.file(&quot;extdata/OwlNightj.csv&quot;, package=&quot;dataone&quot;) sourceObj &lt;- new(&quot;DataObject&quot;, format=&quot;text/csv&quot;, filename=sourceData) dp &lt;- addMember(dp, sourceObj, metadataObj) # The third argument of addMember() associates the new DataObject to the metadata that was just added. If you want to change the formatId please use updateSystemMetadata (more on this later in the book) 2.7.2 Upload the package d1c &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Make sure to give access privileges to the ADC admins: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) Get necessary token from test.arcticdata.io to upload the dataset prior uploading the datapackage: packageId &lt;- uploadDataPackage(d1c, dp, public=TRUE, accessRules=myAccessRules, quiet=FALSE) 2.8 Exercise 2b This exercise will take you through how to do the submission process through R instead of the webform (exercise 1). 2.8.1 Part 1 - Gather your data files For our convenience, we will be grabbing the metadata and data files from the file we published earlier: Locate the data package you published in Exercise 1 by navigating to the “My Profile &gt; My Data” section on test.arcticdata.io. Download the metadata and data files and transfer them to the Datateam server. 2.8.2 Part 2 - Working in R Now we want to publish the metadata and data files we downloaded again to test.arcticdata.io Obtain a token and please note that for this exercise please make sure you grab the token from the arcticdata test site Publish your metadata and data file to the site. #set the node d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) dp &lt;- new(&quot;DataPackage&quot;) #add your metadata metadataObj &lt;- new(...) dp &lt;- addMember(...) #add your data files sourceObj &lt;- new(...) dp &lt;- addMember(...) #upload your package myAccessRules &lt;- data.frame(...) packageId &lt;- uploadDataPackage(...) View your new data set by appending the metadata PID to the end of the URL test.arcticdata.io/view/… If you are successful it should look the same as the dataset you created in exercise 1 Send a message to #datateam with the exercise number and a link to your new package. "],["exploring-eml.html", "Chapter 3 Exploring EML 3.1 Navigate through EML 3.2 Understand the EML schema 3.3 Access specific elements", " Chapter 3 Exploring EML We use the Ecological Metadata Language (EML) to store structured metadata for all datasets submitted to the Arctic Data Center. EML is written in XML (extensible markup language) and functions for building and editing EML are in the EML R package. Currently the Arctic Data Center website supports editing EML version 2.2.0. There are still some metadata in 2.1.1 that will be converted eventually. For additional background on EML and principles for metadata creation, check out this paper. If you aren’t too familiar with lists and how to navigate them yet take a look at the relevant sections in the Stat 545 class. 3.1 Navigate through EML The first task when editing an EML file is navigating the EML file. An EML file is organized in a structure that contains many lists nested within other lists. The function View allows you to get a crude view of an EML file in the viewer. It can be useful for exploring the file. # Need to be in this member node to explore file d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) doc &lt;- read_eml(getObject(d1c_test@mn, &quot;urn:uuid:558eabf1-1e91-4881-8ba3-ef8684d8f6a1&quot;)) View(doc) The complex EML document is represented in R as as series of named, nested lists. We use lists all the time in R! A data.frame is one example of a special kind of list that we use all the time. You may be familiar with the syntax dataframe$column_name which allows us to select a particular column of a data.frame. Under the hood, a data.frame is a named list of vectors with the same length. You select one of those vectors using the $ operator, which is called the “list selector operator.” Just like you navigate in a data.frame, you can use the $ operator to navigate through the EML structure. The $ operator allows you to go deeper into the EML structure and to see what elements are nested within other elements. However, you have to tell R where you want to go in the structure when you use the $ symbol. For example, if you want to view the dataset element of your EML you would use the command doc$dataset. If you want to view the creators of your data set you would use doc$dataset$creator. Note here that creator is contained within dataset. If you aren’t sure where you want to go, hit the tab button on your keyboard after typing $ and a list of available elements in the structure will appear (e.g., doc$&lt;TAB&gt;): Note that if you hit tab, and nothing pops up, this most likely implies that you are trying to go into an EML element that can take a series items. For example doc$dataset$creator$&lt;TAB&gt; will not show a pop-up menu. This is because creator is a series-type object (i.e. you can have multiple creators). If you want to go deeper into creator, you first must tell R which creator you are interested in. Do this by writing [[i]] first where i is the index of the creator you are concerned with. For example, if you want to look at the first creator i = 1. Now doc$dataset$creator[[1]]$&lt;TAB&gt; will give you many more options. Note, an empty autocomplete result sometimes means you have reached the end of a branch in the EML structure. Below is the structure of doc$dataset. There are a series of multiple creators, which can be accessed individually by index: doc$dataset$creator[[#]]. At this point stop and take a deep breath. The key takeaway is that EML is a hierarchical tree structure. The best way to get familiar with it is to explore the structure. Try entering doc$dataset into your console, and print it. Now make the search more specific, for instance: doc$dataset$abstract. 3.2 Understand the EML schema Another great resource for navigating the EML structure is looking at the schema which defines the structure. The schema diagrams on this page are interactive. Further explanations of the symbology can be found here. The schema is complicated and may take some time to get familiar with before you will be able to fully understand it. Use your browser’s “search in page” function (usually CTRL-F or Command-F) to navigate the EML schema page quickly. For example, let’s take a look at eml-party. To start off, notice that some elements have bolded lines leading to them. A bold line indicates that the element is required if the element above it (to the left in the schema) is used, otherwise the element is optional. Notice also that next to the givenName element it says “0..infinity”. This means that the element is unbounded — a single party can have many given names and there is no limit on how many you can add. However, this text does not appear for the surName element — a party can have only one surname. You will also see icons linking the EML slots together, which indicate the ordering of subsequent slots. These can indicate either a “sequence” or a “choice”. In our example from eml-party, a “choice” icon indicates that either an individualName, organizationName, or positionName is required, but you do not need all three. However, the “sequence” icon tells us that if you use an individualName, you must include the surName as a child element. If you include the optional child elements salutation and givenName, they must be written in the order presented in the schema. The eml schema sections you may find particularly helpful include eml-party, eml-attribute and eml-physical. For a more detailed description of the EML schema, see the reference section on exploring EML. 3.2.1 Check your understanding Find otherEntity within the EML schema. Which elements are required? Can otherEntity be a series object? Answer otherEntity requires entityType and entityName children, or alternatively will accept only references. It is a series object, so there can be multiple otherEntities. Along with otherEntity and creator, dataTable and attribute can also be series objects. 3.3 Access specific elements The eml_get() function is a powerful tool for exploring EML (more on that here ). It takes any chunk of EML and returns all instances of the element you specify. Note: you’ll have to specify the element of interest exactly, according to the spelling/capitalization conventions used in EML. Here are some examples: doc &lt;- read_eml(system.file(&quot;example-eml.xml&quot;, package = &quot;arcticdatautils&quot;)) eml_get(doc, &quot;creator&quot;) individualName: givenName: Bryce surName: Mecum organizationName: National Center for Ecological Analysis and Synthesis eml_get(doc, &quot;boundingCoordinates&quot;) eastBoundingCoordinate: &#39;-134&#39; northBoundingCoordinate: &#39;59&#39; southBoundingCoordinate: &#39;57&#39; westBoundingCoordinate: &#39;-135&#39; eml_get(doc, &quot;url&quot;) &#39;&#39;: function: download url: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 &#39;&#39;: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 eml_get_simple() is a simplified alternative to eml_get() that produces a list of the desired EML element. eml_get_simple(doc$dataset$otherEntity, &quot;entityName&quot;) To find an eml element you can use either a combination of which_in_emlfrom the arcticdatautils package or eml_get_simple and which to find the index in an EML list. Use which ever workflow you see fit. An example question you may have: Which creators have a surName “Mecum”? Example using which_in_eml: n &lt;- which_in_eml(doc$dataset$creator, &quot;surName&quot;, &quot;Mecum&quot;) # Answer: doc$dataset$creator[[n]] Example using eml_get_simple and which: ent_names &lt;- eml_get_simple(doc$dataset$creator, &quot;surName&quot;) i &lt;- which(ent_names == &quot;Mecum&quot;) # Answer: doc$dataset$creator[[i]] "],["editing-eml.html", "Chapter 4 Editing EML 4.1 Edit an EML element 4.2 FAIR data practices 4.3 Edit attributeLists 4.4 Set physical 4.5 Edit dataTables 4.6 Edit otherEntities 4.7 Semantic annotations 4.8 Exercise 3a 4.9 Validate EML and update package 4.10 Exercise 3b", " Chapter 4 Editing EML This chapter is a practical tutorial for using R to read, edit, write, and validate EML documents. Much of the information here can also be found in the vignettes for the R packages used in this section (e.g. the EML package). Most of the functions you will see in this chapter will use the arcticdatautils and EML packages. This chapter will be longest of all the sections! This is a reminder to take frequent breaks when completing this section. If you struggle with getting a piece of code to work more than 10 minutes, reach out to your supervisor for help. When using R to edit EML documents, run each line individually by highlighting the line and using CTRL+ENTER). Many EML functions only need to be ran once, and will either produce errors or make the EML invalid if run multiple times. 4.1 Edit an EML element There are multiple ways to edit an EML element. 4.1.1 Edit EML with strings The most basic way to edit an EML element would be to navigate to the element and replace it with something else. Easy! For example, to change the title one could use the following command: doc$dataset$title &lt;- &quot;New Title&quot; If the element you are editing allows for multiple values, you can pass it a list of character strings. Since a dataset can have multiple titles, we can do this: doc$dataset$title &lt;- list(&quot;New Title&quot;, &quot;Second New Title&quot;) However, this isn’t always the best method to edit the EML, particularly if the element has sub-elements. Adding directly to doc without a helper function can overwrite these parts of the doc that we need. 4.1.2 Edit EML with the “EML” package To edit a section where you are not 100% sure of the sub-elements, using the eml$elementName() helper functions from the EML package will pre-populate the options for you if you utilize the RStudio autocomplete functionality. The arguments in these functions show the available slots for any given EML element. For example, typing doc$dataset$abstract &lt;- eml$abstract()&lt;TAB&gt; will show you that the abstract element can take either the section or para sub-elements. doc$dataset$abstract &lt;- eml$abstract(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) This inserts the abstract with a para element in our dataset, which we know from the EML schema is valid. Note that the above is equivalent to the following generic construction: doc$dataset$abstract &lt;- list(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) The eml() family of functions provides the sub-elements as arguments, which is extremely helpful, but functionally all it is doing is creating a named list, which you can also do using the list function. 4.1.3 Edit EML with objects A final way to edit an EML element would be to build a new object to replace the old object. To begin, you might create an object using an eml helper function. Let’s take keywords as an example. Sometimes keyword lists in a metadata record will come from different thesauruses, which you can then add in series (similar to the way we added multiple titles) to the element keywordSet. We start by creating our first set of keywords and saving it to an object. kw_list_1 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER controlled vocabulary&quot;, keyword = list(&quot;bacteria&quot;, &quot;carnivorous plants&quot;, &quot;genetics&quot;, &quot;thresholds&quot;)) Which returns: $keyword $keyword[[1]] [1] &quot;bacteria&quot; $keyword[[2]] [1] &quot;carnivorous plants&quot; $keyword[[3]] [1] &quot;genetics&quot; $keyword[[4]] [1] &quot;thresholds&quot; $keywordThesaurus [1] &quot;LTER controlled vocabulary&quot; We create the second keyword list similarly: kw_list_2 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER core area&quot;, keyword = list(&quot;populations&quot;, &quot;inorganic nutrients&quot;, &quot;disturbance&quot;)) Finally, we can insert our two keyword lists into our EML document just like we did with the title example above, but rather than passing character strings into list(), we will pass our two keyword set objects. doc$dataset$keywordSet &lt;- list(kw_list_1, kw_list_2) Note that you must use the function list here and not the c() function. The reasons for this are complex, and due to some technical subtlety in R - but the gist of the issue is that the c() function can behave in unexpected ways with nested lists, and frequently will collapse the nesting into a single level, resulting in invalid EML. 4.2 FAIR data practices The result of these function calls won’t show up on the webpage but they will add a publisher element to the dataset element and a system to all of the entities based on what the PID is. This will help make our metadata more FAIR (Findable, Accessible, Interoperable, Reusable). These two functions come from the arcticatautils package, an R package we wrote to help with some very specific data processing tasks. Add these function calls to all of your EML processing scripts. library(arcticdatautils) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) 4.3 Edit attributeLists Attributes are descriptions of variables, typically columns or column names in tabular data. Attributes are stored in an attributeList. When editing attributes in R, we convert the attribute list information to data frame (table) format so that it is easier to edit. When editing attributes you will need to create one to three data frame objects: A data.frame of attributes A data.frame of custom units (if applicable) A data.frame of factors (if applicable) The attributeList is an element within one of 4 different types of entity objects. An entity corresponds to a file, typically. Multiple entities (files) can exist within a dataset. The 4 different entity types are dataTable (most common for us), spatialVector, spatialRaster, and otherEntity. Please note that submitting attribute information through the website will store them in an otherEntity object by default. We prefer to store them in a dataTable object for tabular data or a spatialVector or spatialRaster object for spatial data. To edit or examine an existing attribute list already in an EML file, you can use the following commands, where i represents the index of the series element you are interested in. Note that if there is only one item in the series (ie there is only one dataTable), you should just call doc$dataset$dataTable, as in this case doc$dataset$dataTable[[1]] will return the first sub-element of the dataTable (the entityName) # If they are stored in an otherEntity (submitted from the website by default) attribute_tables &lt;- EML::get_attributes(doc$dataset$otherEntity[[i]]$attributeList) # Or if they are stored in a dataTable (usually created by a datateam member) attribute_tables &lt;- EML::get_attributes(doc$dataset$dataTable[[i]]$attributeList) The get_attributes() function returns the attribute_tables object, which is a list of the three data frames mentioned above. The data frame with the attributes is called attribute_tables$attributes. attribute_tables$attributes print(attribute_tables$attributes) 4.3.1 Edit attributes Attribute information should be stored in a data.frame with the following columns: attributeName: The name of the attribute as listed in the csv. Required. e.g.: “c_temp” attributeLabel: A descriptive label that can be used to display the name of an attribute. It is not constrained by system limitations on length or special characters. Optional. e.g.: “Temperature (Celsius)” attributeDefinition: Longer description of the attribute, including the required context for interpreting the attributeName. Required. e.g.: “The near shore water temperature in the upper inter-tidal zone, measured in degrees Celsius.” measurementScale: One of: nominal, ordinal, dateTime, ratio, interval. Required. nominal: unordered categories or text. e.g.: (Male, Female) or (Yukon River, Kuskokwim River) ordinal: ordered categories. e.g.: Low, Medium, High dateTime: date or time values from the Gregorian calendar. e.g.: 01-01-2001 ratio: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second. interval: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or latitude and longitude coordinates, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius. domain: One of: textDomain, enumeratedDomain, numericDomain, dateTime. Required. textDomain: text that is free-form, or matches a pattern enumeratedDomain: text that belongs to a defined list of codes and definitions. e.g.: CASC = Cascade Lake, HEAR = Heart Lake dateTimeDomain: dateTime attributes numericDomain: attributes that are numbers (either ratio or interval) formatString: Required for dateTime, NA otherwise. Format string for dates, e.g. “DD/MM/YYYY”. definition: Required for textDomain, NA otherwise. Defines a format for attributes that are a character string. e.g.: “Any text” or “7-digit alphanumeric code” unit: Required for numericDomain, NA otherwise. Unit string. If the unit is not a standard unit, a warning will appear when you create the attribute list, saying that it has been forced into a custom unit. Use caution here to make sure the unit really needs to be a custom unit. A list of standard units can be found using: standardUnits &lt;- EML::get_unitList() then running View(standardUnits$units). numberType: Required for numericDomain, NA otherwise. Options are real, natural, whole, and integer. real: positive and negative fractions and integers (…-1,-0.25,0,0.25,1…) natural: non-zero positive integers (1,2,3…) whole: positive integers and zero (0,1,2,3…) integer: positive and negative integers and zero (…-2,-1,0,1,2…) missingValueCode: Code for missing values (e.g.: ‘-999’, ‘NA’, ‘NaN’). NA otherwise. Note that an NA missing value code should be a string, ‘NA’, and numbers should also be strings, ‘-999.’ missingValueCodeExplanation: Explanation for missing values, NA if no missing value code exists. You can create attributes manually by typing them out in R following a workflow similar to the one below: attributes &lt;- data.frame( attributeName = c(&#39;Date&#39;, &#39;Location&#39;, &#39;Region&#39;,&#39;Sample_No&#39;, &#39;Sample_vol&#39;, &#39;Salinity&#39;, &#39;Temperature&#39;, &#39;sampling_comments&#39;), attributeDefinition = c(&#39;Date sample was taken on&#39;, &#39;Location code representing location where sample was taken&#39;,&#39;Region where sample was taken&#39;, &#39;Sample number&#39;, &#39;Sample volume&#39;, &#39;Salinity of sample in PSU&#39;, &#39;Temperature of sample&#39;, &#39;comments about sampling process&#39;), measurementScale = c(&#39;dateTime&#39;, &#39;nominal&#39;,&#39;nominal&#39;, &#39;nominal&#39;, &#39;ratio&#39;, &#39;ratio&#39;, &#39;interval&#39;, &#39;nominal&#39;), domain = c(&#39;dateTimeDomain&#39;, &#39;enumeratedDomain&#39;,&#39;enumeratedDomain&#39;, &#39;textDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;textDomain&#39;), formatString = c(&#39;MM-DD-YYYY&#39;, NA,NA,NA,NA,NA,NA,NA), definition = c(NA,NA,NA,&#39;Six-digit code&#39;, NA, NA, NA, &#39;Any text&#39;), unit = c(NA, NA, NA, NA,&#39;milliliter&#39;, &#39;dimensionless&#39;, &#39;celsius&#39;, NA), numberType = c(NA, NA, NA,NA, &#39;real&#39;, &#39;real&#39;, &#39;real&#39;, NA), missingValueCode = c(NA, NA, NA,NA, NA, NA, NA, &#39;NA&#39;), missingValueCodeExplanation = c(NA, NA, NA,NA, NA, NA, NA, &#39;no sampling comments&#39;)) However, typing this out in R can be a major pain. Luckily, there’s a Shiny app that you can use to build attribute information. You can use the app to build attributes from a data file loaded into R (recommended as the app will auto-fill some fields for you) to edit an existing attribute table, or to create attributes from scratch. Use the following commands to create or modify attributes. Use the following commands to create or modify attributes. These commands will launch a “Shiny” app in your web browser. #first download the CSV in your data package from Exercise #2 data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;.csv&quot;) data &lt;- read.csv(text=rawToChar(getObject(d1c_test@mn, data_pid))) # From data (recommended) attribute_tables &lt;- EML::shiny_attributes(data = data) # From scratch attribute_tables &lt;- EML::shiny_attributes() # From an existing attribute list attribute_tables &lt;- get_attributes(doc$dataset$dataTable[[i]]$attributeList) attribute_tables &lt;- EML::shiny_attributes(attributes = attribute_tables$attributes) Once you are done editing a table in the browser app, quit the app by pressing the red “Quit App” button in the top right corner of the page. If you close the Shiny app tab in your browser instead of using the “Quit App” button, your work will not be saved, R will think that the Shiny app is still open, and you will not be able to run other code. You can tell if R is confused if you have closed the Shiny app and the bottom line in the console still says Listening on http://.... If this happens, press the red stop sign button on the right hand side of the console window in order to interrupt R. The tables you constructed in the app will be assigned to the attribute_tables variable as a list of data frames (one for attributes, factors, and units). Be careful to not overwrite your completed attribute_tables object when trying to make edits. The last line of code can be used in order to make edits to an existing attribute_tables object. Alternatively, each table can be to exported to a csv file by clicking the Download button. If you downloaded the table, read the table back into your R session and assign it to a variable in your script (e.g. attributes &lt;- data.frame(...)), or just use the variable that shiny_attributes returned. For simple attribute corrections, datamgmt::edit_attribute() allows you to edit the slots of a single attribute within an attribute list. To use this function, pass an attribute through datamgmt::edit_attribute() and fill out the parameters you wish to edit/update. An example is provided below where we are changing attributeName, domain, and measurementScale in the first attribute of a dataset. After completing the edits, insert the new version of the attribute back into the EML document. new_attribute &lt;- datamgmt::edit_attribute(doc$dataset$dataTable[[1]]$attributeList$attribute[[1]], attributeName = &#39;date_and_time&#39;, domain = &#39;dateTimeDomain&#39;, measurementScale = &#39;dateTime&#39;) doc$dataset$dataTable[[1]]$attributeList$attribute[[1]] &lt;- new_attribute 4.3.2 Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) Search the units list for your unit before attempting to create a custom unit. You can search part of the unit you can look up part of the unit ie meters in the table to see if there are any matches. If you have units that are not in the standard EML unit list, you will need to build a custom unit list. Attribute tables with custom units listed will generate a warning indicating that a custom unit will need to be described. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”). The parentSI does not need to be part of the unitList. multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;siemensPerMeter&#39;, &#39;decibar&#39;), unitType = c(&#39;resistivity&#39;, &#39;pressure&#39;), parentSI = c(&#39;ohmMeter&#39;, &#39;pascal&#39;), multiplierToSI = c(&#39;1&#39;,&#39;10000&#39;), abbreviation = c(&#39;S/m&#39;,&#39;decibar&#39;), description = c(&#39;siemens per meter&#39;, &#39;decibar&#39;)) Custom units can also be created in the shiny app, under the “units” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() custom_units &lt;- attribute_tables$units Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- list(metadata = list(unitList = unitlist)) 4.3.3 Edit factors For attributes that are enumeratedDomains, a table is needed with three columns: attributeName, code, and definition. attributeName should be the same as the attributeName within the attribute table and repeated for all codes belonging to a common attribute. code should contain all unique values of the given attributeName that exist within the actual data. definition should contain a plain text definition that describes each code. To build factors by hand, you use the named character vectors and then convert them to a data.frame as shown in the example below. In this example, there are two enumerated domains in the attribute list - “Location” and “Region”. Location &lt;- c(CASC = &#39;Cascade Lake&#39;, CHIK = &#39;Chikumunik Lake&#39;, HEAR = &#39;Heart Lake&#39;, NISH = &#39;Nishlik Lake&#39; ) Region &lt;- c(W_MTN = &#39;West region, locations West of Eagle Mountain&#39;, E_MTN = &#39;East region, locations East of Eagle Mountain&#39;) The definitions are then written into a data.frame using the names of the named character vectors and their definitions. factors &lt;- rbind(data.frame(attributeName = &#39;Location&#39;, code = names(Location), definition = unname(Location)), data.frame(attributeName = &#39;Region&#39;, code = names(Region), definition = unname(Region))) Factors can also be created in the shiny app, under the “factors” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() attribute_tables$factors 4.3.4 Finalize attributeList Once you have built your attributes, factors, and custom units, you can add them to EML objects. Attributes and factors are combined to form an attributeList using set_attributes(): # Create an attributeList object attributeList &lt;- EML::set_attributes(attributes = attribute_tables$attributes, factors = attribute_tables$factors) This attributeList object can then be checked for errors and added to a dataTable in the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList Remember to use: d1c_test &lt;- dataone::D1Client(“STAGING”, “urn:node:mnTestARCTIC”) d1c_test@mn 4.4 Set physical To set the physical aspects of a data object, use the following commands to build a physical object from a data PID that exists in your package. Remember to set the member node to test.arcticdata.io! Every entity that we upload needs a physical description added. When replacing files, the physical must be replaced as well. The word ‘physical’ derives from database systems, which distinguish the ‘logical’ model (e.g., what attributes are in a table, etc) from the physical model (how the data are written to a physical hard disk (basically, the serialization)). so, we grouped metadata about the file (eg. dataformat, file size, file name) as written to disk in physical. It includes info like the file size. For CSV files, the physical describes the number of header lines and the attribute orientation. # Get the PID of a file data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;your_file_name.csv&quot;) # Get the physical info and store it in an object physical &lt;- arcticdatautils::pid_to_eml_physical(mn, data_pid) The physical object can then be checked for errors and added to the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$physical &lt;- physical Note that the above workflow only works if your data object already exists on the member node. Physicals can be seen in the website representation of the EML below the entity description. 4.5 Edit dataTables Entities that are dataTables require an attribute list. To edit a dataTable, first edit/create an attributeList and set the physical. Then create a new dataTable using the eml$dataTable() helper function as below: dataTable &lt;- eml$dataTable(entityName = &quot;A descriptive name for the data (does not need to be the same as the data file)&quot;, entityDescription = &quot;A description of the data&quot;, physical = physical, attributeList = attributeList) The dataTable must then be added to the EML. How exactly you do this will depend on whether there are dataTable elements in your EML, and how many there are. To replace whatever dataTable elements already exist, you could write: doc$dataset$dataTable &lt;- dataTable If there is only one dataTable in your dataset, the EML package will usually “unpack” these, so that it is not contained within a list of length 1 - this means that to add a second dataTable, you cannot use the syntax doc$dataset$dataTable[[2]], since when unpacked this will contain the entityDescription as opposed to pointing to the second in a series of dataTable elements. Confusing - I know. Not to fear though - this syntax will get you on your way, should you be trying to add a second dataTable. doc$dataset$dataTable &lt;- list(doc$dataset$dataTable, dataTable) If there is more than one dataTable in your dataset, you can return to the more straightforward construction of: doc$dataset$dataTable[[i]] &lt;- dataTable Where i is the index that you wish insert your dataTable into. To add a list of dataTables to avoid the unpacking problem above you will need to create a list of dataTables dts &lt;- list() # create an empty list for(i in seq_along(tables_you_need)){ # your code modifying/creating the dataTable here dataTable &lt;- eml$dataTable(entityName = dataTable$entityName, entityDescription = dataTable$entityDescription, physical = physical, attributeList = attributeList) dts[[i]] &lt;- dataTable # add to the list } After getting a list of dataTables, assign the resulting list to dataTable EML doc$dataset$dataTable &lt;- dts By default, the online submission form adds all entities as otherEntity, even when most should probably be dataTable. You can use eml_otherEntity_to_dataTable to easily move items in otherEntity over to dataTable, and delete the old otherEntity. Most tabular data or data that contain variables should be listed as a dataTable. Data that do not contain variables (eg: plain text readme files, pdfs, jpegs) should be listed as otherEntity. eml_otherEntity_to_dataTable(doc, 1, # Indexes of otherEntities you want to convert, for multiple use 1:5 or c(1,3,5..) validate_eml = F) # set this to False if the physical or attributes are not added 4.6 Edit otherEntities 4.6.1 Remove otherEntities To remove an otherEntity use the following command. This may be useful if a data object is originally listed as an otherEntity and then transferred to a dataTable. doc$dataset$otherEntity[[i]] &lt;- NULL 4.6.2 Create otherEntities If you need to create/update an otherEntity, make sure to publish or update your data object first (if it is not already on the DataONE MN). Then build your otherEntity. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, pkg$data[[i]]) Alternatively, you can build the otherEntity of a data object not in your package by simply inputting the data PID. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, &quot;your_data_pid&quot;, entityType = &quot;otherEntity&quot;, entityName = &quot;Entity Name&quot;, entityDescription = &quot;Description about entity&quot;) The otherEntity must then be set to the EML, like so: doc$dataset$otherEntity &lt;- otherEntity If you have more than one otherEntity object in the EML already, you can add the new one like this: doc$dataset$otherEntity[[i]] &lt;- otherEntity Where i is set to the number of existing entities plus one. Remember the warning from the last section, however. If you only have one otherEntity, and you are trying to add another, you have to run: doc$dataset$otherEntity &lt;- list(otherEntity, doc$dataset$otherEntity) 4.7 Semantic annotations For a brief overview of what a semantic annotation is, and why we use them check out this video. Even more information on how to add semantic annotations to EML 2.2.0 can be found here. There are several elements in the EML 2.2.0 schema that can be annotated: dataset entity (eg: otherEntity or dataTable) attribute Attribute annotations can be edited in R and also on the website. Dataset and entity annotations are only done in R. 4.7.1 How annotations are used This is a dataset that has semantic annotations included. On the website you can see annotations in each of the attributes. You can click on any one of them to search for more datasets with that same annotation. Semantic attribute annotations can be applied to spatialRasters, spatialVectors and dataTables 4.7.1.1 Attribute-level annotations on the website editor The website has a searchable list of attribute annotations that are grouped by category and specificity. Open your dataset on the test website from earlier and enter the attribute editor. Look through all of the available annotations. Adding attribute annotations using the website is the easiest way, however adding them using R and/or the Shiny app may be quicker with very larger datasets. 4.7.1.2 Attribute-level annotations in R To manually add annotations to the attributeList in R you will need information about the propertyURI and valueURI. Annotations are essentially composed of a sentence, which contains a subject (the attribute), predicate (propertyURI), and object (valueURI). Because of the way our search interface is built, for now we will be using attribute annotations that have a propertyURI label of “contains measurements of type”. Here is what an annotation for an attribute looks like in R. Note that both the propertyURI and valueURI have both a label, and the URI itself. doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$annotation $id [1] &quot;ODBcOyaTsg&quot; $propertyURI $propertyURI$label [1] &quot;contains measurements of type&quot; $propertyURI$propertyURI [1] &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot; $valueURI $valueURI$label [1] &quot;Distributed Biological Observatory region identifier&quot; $valueURI$valueURI [1] &quot;http://purl.dataone.org/odo/ECSO_00002617&quot; 4.7.2 How to add an annotation 1. Decide which variable to annotate The goal for the datateam is to start annotating every dataset that comes in. Please make sure to add semantic annotations to spatial and temporal features such as latitude, longitude, site name and date and aim to annotate as many attributes as possible. 2. Find an appropriate valueURI The next step is to find an appropriate value to fill in the blank of the sentence: “this attribute contains measurements of _____.” There are several ontologies to search in. In order of most to least likely to be relevant to the Arctic Data Center they are: The Ecosystem Ontology (ECSO) this was developed at NCEAS, and has many terms that are relevant to ecosystem processes, especially those involving carbon and nutrient cycling The Environment Ontology (EnVO) this is an ontology for the concise, controlled description of environments National Center for Biotechnology Information (NCBI) Organismal Classification (NCBITAXON) The NCBI Taxonomy Database is a curated classification and nomenclature for all of the organisms in the public sequence databases. Information Artifact Ontology (IAO) this ontology contains terms related to information entities (eg: journals, articles, datasets, identifiers) To search, navigate through the “classes” until you find an appropriate term. When we are picking terms, it is important that we not just pick a similar term or a term that seems close - we want a term that is 100% accurate. For example, if you have an attribute for carbon tetroxide flux and an ontology with a class hierarchy like this: – carbon flux |—- carbon dioxide flux Our exact attribute, carbon tetroxide flux, is not listed. In this case, we should pick “carbon flux” as it’s completely correct/accurate and not “carbon dioxide flux” because it’s more specific/precise but not quite right. For general attributes (such as ones named depth or length), it is important to be as specific as possible about what is being measured. e.g. selecting the lake area annotation for the area attribute in this dataset 3. Build the annotation in R 4.7.2.1 Manually Annotating this method is great for when you are inserting 1 annotation, fixing an existing annotation or programmatically updating annotations for multiple attributeLists First you need to figure out the index of the attribute you want to annotate. eml_get_simple(doc$dataset$dataTable[[3]]$attributeList, &quot;attributeName&quot;) [1] &quot;prdM&quot; &quot;t090C&quot; &quot;t190C&quot; &quot;c0mS/cm&quot; &quot;c1mS/cm&quot; &quot;sal00&quot; &quot;sal11&quot; &quot;sbeox0V&quot; &quot;flECO-AFL&quot; [10] &quot;CStarTr0&quot; &quot;cpar&quot; &quot;v0&quot; &quot;v4&quot; &quot;v6&quot; &quot;v7&quot; &quot;svCM&quot; &quot;altM&quot; &quot;depSM&quot; [19] &quot;scan&quot; &quot;sbeox0ML/L&quot; &quot;sbeox0dOV/dT&quot; &quot;flag&quot; Next, assign an id to the attribute. It should be unique within the document, and it’s nice if it is human readable and related to the attribute it is describing. One format you could use is entity_x_attribute_y which should be unique in scope, and is nice and descriptive. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$id &lt;- &quot;entity_ctd_attribute_salinity&quot; Now, assign the propertyURI information. This will be the same for every annotation you build. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$propertyURI &lt;- list(label = &quot;contains measurements of type&quot;, propertyURI = &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot;) Finally, add the valueURI information from your search. You should see an ID on the Bioportal page that looks like a URL - this is the valueURI. Use the value to populate the label element. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$valueURI &lt;- list(label = &quot;Water Salinity&quot;, valueURI = &quot;http://purl.dataone.org/odo/ECSO_00001164&quot;) 4.7.2.2 Shiny Attributes this method is great for when you are updating many attributes On the far right of the table of shiny_attributes there are 4 columns: id, propertyURI, propertyLabel, valueURI, valueLabel that can be filled out. 4.7.3 Dataset Annotations Dataset annotations can only be made using R. There are several helper functions that assist with making dataset annotations. 4.7.3.1 Data Sensitivity Sensitive datasets that might cover protected characteristics (human subjects data, endangered species locations, etc) should be annotated using the data sensitivity ontology: https://bioportal.bioontology.org/ontologies/SENSO/?p=classes&amp;conceptid=root. 4.7.3.2 Dataset Discipline As a final step in the data processing pipeline, we will categorize the dataset. We are trying to categorize datasets so we can have a general idea of what kinds of data we have at the Arctic Data Center. Datasets will be categorized using the Academic Ontology. These annotations will be seen at the top of the landing page, and can be thought of as “themes” for the dataset. In reality, they are dataset-level annotations. Be sure to ask your peers in the #datateam slack channel whether they agree with the themes you think best fit your dataset. Once there is consensus, use the following line of code: doc &lt;- arcticdatautils::eml_categorize_dataset(doc, c(&quot;Soil Science&quot;, &quot;Plant Science&quot;, &quot;Ecology&quot;)) Be careful not to duplicate dataset annotations. The above code does not existing prior dataset annotations. Duplicate annotations can be removed by setting them to NULL. doc$dataset$annotation[[i]] &lt;- NULL 4.8 Exercise 3a The metadata for the dataset created earlier in Exercise 2 was not very complete. Here we will add a attribute and physical to our entity (the csv file). Make sure your package from before is loaded into R. Convert otherEntity into dataTable. Replace the existing dataTable with a new dataTable object with an attributelist you write in R using the above commands. We will continue using the objects created and updated in this exercise in 3b. Below is some pseudo-code for how to accomplish the above steps. Fill in the dots according to the above sections to complete the exercise. # get the latest version of the resource map identifier from your dataset on the arctic data center resource_map_pid &lt;- ... dp &lt;- getDataPackage(d1c_test, identifier=resource_map_pid, lazyLoad=TRUE, quiet=FALSE) # get metadata pid metadataId &lt;- selectMember(...) # read in EML doc &lt;- read_eml(getObject(...)) # convert otherEntity to dataTable doc &lt;- eml_otherEntity_to_dataTable(...) # write an attribute list using shiny_attributes based on the data in your file ex_data &lt;- read.csv(...) atts &lt;- shiny_attributes(data = ex_data) # set the attributeList doc$dataset$dataTable$attributeList &lt;- set_attributes(...) 4.9 Validate EML and update package To make sure that your edited EML is valid against the EML schema, run eml_validate() on your EML. Fix any errors that you see. eml_validate(doc) You should see something like if everything passes: &gt;[1] TRUE &gt;attr(,“errors”) &gt;character(0) When troubleshooting EML errors, it is helpful to run eml_validate() after every edit to the EML document in order to pinpoint the problematic code. Then save your EML to a path of your choice or a temp file. You will later pass this path as an argument to update the package. # Create a standardized EML name from the dataset title eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) 4.10 Exercise 3b Make sure you have everything from before in R. After adding more metadata, we want to publish the dataset onto test.arcticdata.io. Before we publish updates we need to do a couple checks before doing so. Validate your metadata using eml_validate. Use the checklist to review your submission. Make edits where necessary (e.g. physicals) Once eml_validate returns TRUE go ahead and run write_eml, replaceMember, and uploadDataPackage. There might be a small lag for your changes to appear on the website. This part of the workflow will look roughly like this: # validate and write the EML eml_validate(...) write_eml(...) # replace the old metadata file with the new one in the local package dp &lt;- replaceMember(dp, metadataId, replacement = eml_path) # upload the data package packageId &lt;- uploadDataPackage(...) "],["updating-a-data-package.html", "Chapter 5 Updating a data package 5.1 Update an object 5.2 Update a package with a new data object 5.3 Exercise 4", " Chapter 5 Updating a data package This chapter will teach you how to edit and update an existing data package in R. Earlier, we updated metadata. In this section we will learn how to update a data file, and how to update a package by adding an additional data file. 5.1 Update an object To update a data file associated with a data package, you need to do three things: update the object itself, update the resource map (which affiliates the object with the metadata), and update the metadata that describes that object The datapack::replaceMember function takes care of the first two of these tasks. First you need to get the pid of the file you want to replace by using datapack::selectMember # Select the EML PID dataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;text/csv&quot;) Then use replaceMember: # Replace the old file with the new dp &lt;- replaceMember(dp, dataId, replacement=file_path) If you want to remove some files from the data package we can use datapack::removeMember. If we wanted to remove all the zip files associated with this data package, we can use datapack::removeMember: zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) You will need to be explicit about your format_id here based on the file type. A list of format IDs can be found here on the DataONE website. Use line 2 (Id:) exactly, character for character. To accomplish the second task, you will need to update the metadata using the EML package. This is covered in Chapter 4. After you update a file, you will always need to update the metadata because parts of the physical section (such as the file size, checksum) will be different, and it may also require different attribute information. Once you have updated your metadata and saved it, you can update the package itself. 5.2 Update a package with a new data object Once you have updated the data objects and saved the metadata to a file, we can update the metadata and use replaceMember to update the package with the new metadata. Make sure you have the package you want to update loaded into R using dataone::getDataPackage(). 5.2.1 Publish update Now we can update your data package to include the new data object. Assuming you have updated your data package earlier something like the below: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) packageId &lt;- &quot;resource_map_urn:uuid...&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #some modification to the EML here eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) You can then upload your data package: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c_test, dp, public=FALSE, accessRules=myAccessRules, quiet=FALSE) If a package is ready to be public, you can change the public argument in the datapack::uploadDataPackage() call to TRUE. If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), you need to do this when replacing the metadata using the optional newId argument in replaceMember(). This should only be done after the package is finalized and has been thoroughly reviewed! doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If there is a pre-issued DOI (researcher requested the DOI for the publication first), please do the following: dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;doi:10.../...&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If the package has children, see how to do this using arcticdatautils::publish_update in the nesting section of the reference manual. Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. 5.3 Exercise 4 What if the researcher notices that some information needed to be updated in the data file? We can use replaceMember to do just that! If you haven’t already: Locate the data package you published in the previous exercise by navigating to the URL test.arcticdata.io/#view/… Load the package and EML into R using the above commands. Make a slightly different data file to upload to test.arcticdata.io for this exercise: Load the data file associated with the package into R as a data.frame. (Hint: use read.csv() to upload the data file from your computer/the server.) Make an edit to the data in R (e.g. change one of the colnames to \"TEST\"). Save the edited data. (Hint: use write.csv(data, row.names = FALSE).) Upload the new csv file, get a new pid and publish those updates: Update the data file in the package with the edited data file using replaceMember. Update your package using uploadDataPackage. "],["editing-system-metadata.html", "Chapter 6 Editing system metadata 6.1 Edit sysmeta 6.2 Exercise 5", " Chapter 6 Editing system metadata Every object on the ADC (or the KNB) has “system metadata”. An object’s system metadata have information about the file itself, such as the name of the file (fileName), the format (formatId), who the rightsHolder is, what the accessPolicy is, and more. Sometimes we will need to edit system metadata in order to make sure that things on the webpage display correctly, or to ensure a file downloads from the website with the correct file name and extension. Although the majority of system metadata changes that need to be made are done automatically, sometimes we need to change aspects of the system metadata (or ‘sysmeta’ for short) manually. 6.1 Edit sysmeta To edit the sysmeta of an object (data file, EML, or resource map, etc.) with a PID, first load the sysmeta into R using the following command: sysmeta &lt;- getSystemMetadata(d1c_test@mn, pid) Then edit the sysmeta slots by using @ functionality. For example, to change the fileName use the following command: sysmeta@fileName &lt;- &#39;NewFileName.csv&#39; Note that some slots cannot be changed by simple text replace (particularly the accessPolicy). There are various helper functions for changing the accessPolicy and rightsHolder such as datapack::addAccessRule() (which takes the sysmeta as an input) or arcticdatautils::set_rights_and_access(), which only requires a PID. In general, you most frequently need to use dataone::getSystemMetadata() to change either the formatId or fileName slots (see the DataONE list of format ids) for acceptable formats. # Example of setting the formatId slot sysmeta@formatId &lt;- &quot;eml://ecoinformatics.org/eml-2.1.1&quot; After you have changed the necessary slot, you can update the system metadata using the following command: updateSystemMetadata(mn, pid, sysmeta) 6.1.1 Identifiers and sysmeta Importantly, changing the system metadata does NOT necessitate a change in the PID of an object. This is because changes to the system metadata do not change the object itself, they are only changing the description of the object (although ideally the system metadata are accurate when an object is first published). 6.1.2 Additional resources For a more in-depth (and technical) guide to sysmeta, check out the DataONE documentation: System Metadata Data Types in CICore 6.2 Exercise 5 Sometimes the system doesn’t recognize the file types properly. For example you have a csv file but the File type on the website says Microsoft Excel Read the system metadata in from the data file you uploaded previously. Check to make sure the fileName and formatId are set correctly (the extension in fileName should match the formatId). Update the system metadata if necessary. CSVs have the formatId “text/csv”. "],["building-provenance.html", "Chapter 7 Building provenance 7.1 Using the prov editor 7.2 Understanding resource maps 7.3 Using the datapack package 7.4 References", " Chapter 7 Building provenance Note - It is rare for a dataset to have provenance - though we would like to see that change by encouraging researchers to submit scripts whenever it is reasonable. When processing datasets if you notice that provenance is needed let Daphne or Jeanette know. The provenance chain describes the origin and processing history of data. Provenance (or “prov”) can exist on a continuum, ranging from prose descriptions of the history, to formal provenance traces, to fully executable environments. In this section we will describe how to build provenance using formal provenance traces in DataONE. Provenance is becoming increasingly important in the face of what is being called a reproducibility crisis in science. J. P. A. Ioannidis (2005) wrote that “Most Research Findings Are False for Most Research Designs and for Most Fields”. Ioannidis outlined ways in which the research process has lead to inflated effect sizes and hypothesis tests that codify existing biases. The first step towards addressing these issues is to be able to evaluate the data, analyses, and models on which conclusions are drawn. Under current practice, this can be difficult because data are typically unavailable, the method sections of papers do not detail the computational approaches used, and analyses and models are often conducted in graphical programs, or, when scripted analyses are employed, the code is not available. And yet, this is easily remedied. Researchers can achieve computational reproducibility through open science approaches, including straightforward steps for archiving data and code openly along with the scientific workflows describing the provenance of scientific results (e.g., Hampton et al. (2015), Munafò et al. (2017)). At NCEAS and in the datateam, not only do we archive data and code openly, but we also describe the workflows that involve that data and code using provenance, formalizing the provenance trace for a workflow that might look like this into an easily understandable trace including archived data objects, such as what is shown here. There are two ways that we add provenance in the datateam - the prov editor and the R datapack package. 7.1 Using the prov editor Provenance can easily be added to production Arctic Data Center packages using the provenance editor on the Arctic Data Center. On the landing page of a data package within beta, in the dataTable or otherEntity section where you would like to add a provenance relationship, you can choose to add either a “source” or a “derivation”, to the left or right of the object pane, respectively. To add a source data file, click on the circle with the “+ add” text. Similarly, a source script would be added by selecting the arrow. Selecting the circle to add a source file pulls up the following screen, where you can select the source from other data objects within the same data package. A data package with an object that has multiple sources added will look like this. For simple packages on the Arctic Data Center, adding prov through the prov editor is super easy! 7.2 Understanding resource maps Before we dive further into constructing prov in R, we need to talk more about resource maps (or “resmaps”). All data packages have a single resource map. But what is a resource map and how do we use one to find out what objects are in a particular data package? This document is a short introduction but a more complete guide can be found here. A resource map is a special kind of XML document that describes (among other things) an “aggregation”. The aggregation describes the members of a data package (metadata and data, usually). We can use the dataone R package to download a resource map if we know its PID: library(dataone) d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) pid &lt;- &quot;urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot; # A resource map PID path &lt;- tempfile(fileext = &quot;.xml&quot;) # We&#39;re saving to a temporary file but you can save elsewhere writeLines(rawToChar(getObject(d1c_test@mn, pid)), path) # Write the object to `path` If we open that file up in a text editor, we see this: &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt; &lt;rdf:RDF xmlns:cito=&quot;http://purl.org/spar/cito/&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot; xmlns:dcterms=&quot;http://purl.org/dc/terms/&quot; xmlns:foaf=&quot;http://xmlns.com/foaf/0.1/&quot; xmlns:ore=&quot;http://www.openarchives.org/ore/terms/&quot; xmlns:prov=&quot;http://www.w3.org/ns/prov#&quot; xmlns:provone=&quot;http://purl.dataone.org/provone/2015/01/15/ontology#&quot; xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:rdfs=&quot;http://www.w3.org/2000/01/rdf-schema#&quot; xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema#&quot;&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;cito:isDocumentedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;rdf:type rdf:resource=&quot;http://www.openarchives.org/ore/terms/Aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:isDocumentedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:documents rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:documents rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;ore:isAggregatedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;dc:title&gt;DataONE Aggregation&lt;/dc:title&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;ore:describes rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;ore:isAggregatedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;rdf:type rdf:resource=&quot;http://www.openarchives.org/ore/terms/ResourceMap&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;/rdf:RDF&gt; Whoa! What is this thing and how do you read it to find the members of the data package? The short answer is to look for lines like this: &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; This line says “The aggregation aggregates urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5” so that means urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5 is in our data package! The key bit is the &lt;rdf:Description rdf:about=\"...#aggregation part. If you look for another similar statement, you’ll also see that urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27 is part of our data package. Now we know which objects are in our data package but we don’t know which one contains metadata and which one contains data. For that, we need to get a copy of the system metadata for each object: getSystemMetadata(d1c_test@mn, &quot;urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5&quot;)@formatId getSystemMetadata(d1c_test@mn, &quot;urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;)@formatId From the formatIds, we can see the first PID is the EML (formatId: [1]\"eml://ecoinformatics.org/eml-2.1.1\") and the second PID is a data object (formatId: [1]\"application/octet-stream\"). Now we know enough to know what’s in the data package: Resource map: urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556 Metadata: urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5 Data: urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27 Now that you’ve actually seen a resource map, we can dive further into prov. 7.3 Using the datapack package For packages not on the ADC, or packages that are extremely complicated, it may be best to upload prov relationships using R. The datapack package has several functions which help add relationships in a very simple way. These relationships are stored in the resource map. When you update a package to only add prov, the package will not be assigned any new identifiers with the exception of the resource map. First, we set the environment, in a similar, but slightly different way than what you may be used to. Here the function D1Client() sets the DataONE client with the coordinating node instance as the first argument, and member node as the second argument. library(dataone) library(datapack) d1c &lt;- D1Client(&quot;STAGING2&quot;, &quot;urn:node:mnTestKNB&quot;) Next, get the PID of the resource map of the data package you are adding prov to, and load that package into R using the getDataPackage() function. packageId &lt;- &quot;urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b&quot; dp &lt;- getDataPackage(d1c, id=packageId, lazyLoad=TRUE, limit=&quot;0MB&quot;, quiet=FALSE) Printing pkg in your console shows you the contents of the data package, including all of the objects and their names: &gt; pkg Members: filename format mediaType size identifier modified local esc...er.R application/R NA 888 knb.92049.1 n n PWS....csv text/csv NA 1871469 knb.92050.1 n n PWS....csv text/csv NA 1508128 knb.92051.1 n n NA eml:/...-2.1.1 NA 15658 urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b n y Package identifier: resource_map_urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b RightsHolder: http://orcid.org/0000-0002-2192-403X It will also show the existing relationships in the resource map, which in this case are mostly the “documents” relationships that specify that the metadata record is describing all of these data files. Relationships: subject predicate object 2 esc_reformatting_PWSweirTower.R cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 4 PWS_weirTower.csv cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 1 PWS_Weir_Tower_export.csv cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 3 urn:uuid:8f501606-...4-b22d-050a4176a97b dcterms:creator _r1515542097r415842r1 5 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents esc_reformatting_PWSweirTower.R 6 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents PWS_weirTower.csv 7 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents PWS_Weir_Tower_export.csv 8 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents urn:uuid:8f501606-...4-b22d-050a4176a97b 9 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b In this example above, the data package has two .csv files, with an R script that converts one to the other. To create our provenance trace, first we need to select the source object, and save the PID to a variable. We do this using the selectMember() function, and we can query part of the system metadata to select the file that we want. This function takes the data package (pkg), the name of the sysmeta field to query (in this case we use the fileName), and the value that you want to match that field to (in this case, ‘PWS_Weir_Tower_export.csv’). sourceObjId &lt;- selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&#39;PWS_Weir_Tower_export.csv&#39;) This returns a list of the source object PIDs that match the query (in this case only one object matches). &gt; sourceObjId [1] &quot;knb.92051.1&quot; Now we need to select our output object. Here, we use the selectMember() function again, and save the result to a new variable. outputObjId &lt;- selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&#39;PWS_weirTower.csv&#39;) Now we query for the R script. In this case, we query based on the value of the formatId as opposed to the fileName. This can be useful if you wish to select a large list of PIDs that are all similar. programObjId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/R&quot;) Next, you use these lists of PIDs and a function called describeWorkflow() to add these relationships to the data package. Note that if you do not have a program in the workflow, or a source file, you can simply leave those arguments blank. dp &lt;- describeWorkflow(dp, sources=sourceObjId, program=programObjId, derivations=outputObjId) Viewing pkg again confirms that these relationships have been inserted into the data package, as shown by the wasDerivedFrom and wasGeneratedBy statements. It is always a good idea to print pkg to confirm that your PID selection process worked as expected, and your prov relationships make sense. Relationships (updated): subject predicate object 15 _1db49d06-ae98-4...9101-39f7c0b45a95 rdf:type prov:Association 14 _1db49d06-ae98-4...9101-39f7c0b45a95 prov:hadPlan esc_reformatting_PWSweirTower.R 1 esc_reformatting_PWSweirTower.R cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 16 esc_reformatting_PWSweirTower.R rdf:type provone:Program 8 PWS_weirTower.csv cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 11 PWS_weirTower.csv rdf:type provone:Data 20 PWS_weirTower.csv prov:wasDerivedFrom PWS_Weir_Tower_export.csv 19 PWS_weirTower.csv prov:wasGeneratedBy urn:uuid:3dd59b0...bc38-3b5d8fa644ac 6 PWS_Weir_Tower_export.csv cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 10 PWS_Weir_Tower_export.csv rdf:type provone:Data 9 _r1515544826r415842r1 foaf:name DataONE R Client 17 urn:uuid:3dd59b0...bc38-3b5d8fa644ac dcterms:identifier urn:uuid:3dd59b0...bc38-3b5d8fa644ac 13 urn:uuid:3dd59b0...bc38-3b5d8fa644ac rdf:type provone:Execution 12 urn:uuid:3dd59b0...bc38-3b5d8fa644ac prov:qualifiedAssociation _1db49d06-ae98-4...9101-39f7c0b45a95 18 urn:uuid:3dd59b0...bc38-3b5d8fa644ac prov:used PWS_Weir_Tower_export.csv 5 urn:uuid:8f50160...b22d-050a4176a97b cito:documents esc_reformatting_PWSweirTower.R 4 urn:uuid:8f50160...b22d-050a4176a97b cito:documents PWS_weirTower.csv 3 urn:uuid:8f50160...b22d-050a4176a97b cito:documents PWS_Weir_Tower_export.csv 2 urn:uuid:8f50160...b22d-050a4176a97b cito:documents urn:uuid:8f50160...b22d-050a4176a97b 7 urn:uuid:8f50160...b22d-050a4176a97b cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b Finally, you can upload the data package using the uploadDataPackage() function, which takes the DataONE client d1c we set in the beginning, the updated pkg variable, some options for public read, and whether or not informational messages are printed during the upload process. First, you will need to run a token obtained from https://dev.nceas.ucsb.edu/ to publish to that node. resmapId_new &lt;- uploadDataPackage(d1c, dp, public = TRUE, quiet = FALSE) If successful you should be able to navigate to the landing page of your dataset, and icons should show up where the sources and derivations are, such as in this example: 7.3.1 Fixing mistakes If you messed up updating a data package using datapack, there unfortunately isn’t a great way to undo your work, as the describeWorkflow() only adds prov relationships, it does not replace them. If you messed up, the best course of action is to update the resource map with a clean version that does not have prov using update_resource_map(), and then go through the steps outlined above again. Note: this has not been thoroughly tested, and more extreme actions may be necessary to fully nuke the prov relationships. Consult Peter or Jeanette if things do not work as expected. 7.4 References Ioannidis, John P A. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124. Hampton, Stephanie E, Sean Anderson, Sarah C Bagby, Corinna Gries, Xueying Han, Edmund Hart, Matthew B Jones, et al. 2015. “The Tao of Open Science for Ecology.” Ecosphere 6 (July). https://doi.org/10.1890/ES14-00402.1. Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. “A Manifesto for Reproducible Science.” Nature Human Behaviour 1 (1): 0021. https://doi.org/10.1038/s41562-016-0021. "],["using-git.html", "Chapter 8 Using Git 8.1 Usage in the datateam 8.2 What is Git? 8.3 Setting up Git 8.4 GitHub Authentication 8.5 Working with the repository 8.6 My Git tab disappeared", " Chapter 8 Using Git 8.1 Usage in the datateam We use git and Github to manage our packages (ie datamgmt, arcticdatautils) and even this document! 8.2 What is Git? Git is a distributed version control system. The NCEAS Github repository contains many processing scripts and templates that can help when working with datasets. Important! If you have never used Git before, or only used it a little, or have no idea what it is, check out this intro to Git put together by the ecodatascience group at UCSB. Don’t worry too much about the forking and branching sections, as we will primarily be using the basic commit-pull-push commands. After you have read through that presentation, come back to this chapter. 8.2.1 So why do I need to use this again? There are several reasons why using the arctic-data GitHub repository is helpful, both for you and for the rest of the data team. Here are a few: Versioning: Did you accidentally make a change to your code and can’t figure out why it broke? Do you wish you could go back to the version that worked? If you add your code to the GitHub repo you can do this! Reproducibility: Being able to reproduce how you accomplished something is incredibly important. We should be able to tell anyone exactly how data have been reformatted, how metadata have been altered, and how packages have been created. As a data center, this is especially important for us with data team members that stay for 6-12 months because we may need to go back and figure out how something was done after the intern or fellow who wrote the code left the team. Troubleshooting: If you are building a particularly complicated EML, or doing some other advanced task, it is much easier for others to help you troubleshoot your code if it is on the GitHub repo. We can view, troubleshoot, and fix bugs very easily when code is on the GitHub repo, with the added bonus of being able to go back a version if something should break. Solve future problems: Some of the issues we see in ADC submissions come up over and over again. When all of our code is on GitHub, we can easily reference code built for other submissions, instead of trying to solve the same problems over and over again from scratch. 8.3 Setting up Git Before using git, you need to tell it who you are. The only way to do this is through the command line. When you open RStudio, you should see a Terminal tab located to the right of the Console tab. If the Terminal tab is not visible, you can open it by selecting Tools -&gt; Terminal -&gt; New Terminal at the top of your RStudio window. To tell git who you are, you’re going to set the global options. This includes setting your name, email address, PAT (Personal Access Token), and caching the PAT. Make sure you run the following commands one at a time. Type the following command in the Terminal window, with your actual name, and press enter. If you do this correctly, it will look as though nothing happened. git config --global user.name &quot;Your Name&quot; Next, enter the following line, with the email address associated with your GitHub account. git config --global user.email MyEmail@domain.com After running these commands, the Terminal window should look like this: Next, we will tell Git to store your PAT. We must do this because of the way our server operating system handles credentials. If you don’t run the next line, your PAT will expire immediately on the server, even though we will set it up on GitHub to be valid for 90 days. git config --global credential.helper &#39;cache -- timeout=10000000&#39; Finally, check to make sure everything looks correct by entering the following command, which will return the options that you have set. git config --global --list 8.4 GitHub Authentication GitHub recently deprecated password authentication for accessing repositories, so we need to set up a secure way to authenticate. That’s where the PAT comes in. In the tabs at the bottom of your RStudio window, make sure to switch from the Terminal to the Console. Now that, you’ve done that, follow these steps: Run usethis::create_github_token() In the browser window that pops up, log into GitHub In the “Expiration” drop down menu, select “90 days” Click “Generate token” in the green box at the bottom of the page On the next page, copy the token from the green box Back in RStudio, run credentials::set_github_pat() Paste your token into the dialog box that pops up You’ve now successfully set your PAT in RStudio, and connected R, Git, and GitHub. Great job! Now, you’re going to clone the repository we work in so that you can version control your code. 8.4.1 Cloning the arctic-data repo To clone the arctic-data repository, navigate to the repository on GitHub: https://github.nceas.ucsb.edu/KNB/arctic-data/. If you are not already logged on, this will be the enterprise account you use to log in when you go to RT. If you have trouble with this, send a message to Nick Outin (@Nick) in the datateam slack channel. Once you are logged on and can view the above link, click on the green “Code” button, and copy the URL in the gray box. In your RStudio window, click File -&gt; New Project -&gt; Version Control -&gt; Git. Paste the URL you just copied into the “Repository URL” box, and press tab on your keyboard. This should autofill the “Project directory name.” If it does not, then type in arctic-data. Fill it out as shown in the image below to clone the arctic-data repository into the top level of your home directory. Click “Create Project,” and now you should have a directory called arctic-data in your RStudio files window. If you are prompted to save your workspace during this process, make sure all of your work is saved, and you don’t need anything in your environment, and then click ‘Don’t Save’. 8.5 Working with the repository 8.5.1 Adding a new script If you have been working on a script that you want to put in the arctic-data GitHub repo, you first need to save it somewhere in the arctic-data folder you cloned to your account on the Datateam server (/home/username/…). You can do this by either moving your script into the folder or using the save-as functionality. Note that Git will try and version anything that you save in this folder, so you should be careful about what you save here. For our purposes, things that probably shouldn’t be saved in this folder include: Tokens: Any token file or script with a token in it should NOT be saved in the repository. Others could steal your login credentials if you put a token in GitHub. Data files: Git does not version data files very well. You shouldn’t save any .csv files or any other data files (including metadata). Workspaces/.RData: If you are in the habit of saving your R workspace, you shouldn’t save it in this directory. Plots/Graphics: For the same reasons as data files. Note: Do not EVER make a commit that you don’t understand. If something unexpected (like a file you have never worked on) shows up in your Git tab, ask for help before committing. After you save your script in the appropriate place within the arctic-data folder, it will show up in your Git tab looking like this: Before you commit your changes, you need to click the little box under “Staged”. Do not stage or commit any .Rproj file. After clicking the box for your file, click “Commit” to commit your changes. In the window that pops up (you may need to force the browser to allow pop-ups), write your commit message. Always include a commit message. Remember that the commit message should be a concise description of the changes that were made to a file. Your window should look like this: Push ‘Commit’, and your commit will be saved to your local repository (this will not push it to the remote repository, yet). Now you want to merge the commits you made with the master version of the remote repository. You do this by using the command “Push.” But before you push, you always need to pull first to avoid merge conflicts. Pulling will merge the current version of the remote repository with your local repository, on your local machine. Click “Pull” and type in your credentials. Then, assuming you don’t have a merge conflict, you can push your changes by clicking “Push”. Always remember, the order is commit-pull-push. 8.5.2 Editing a script If you want to change a script, the workflow is the same. Just open the script that was saved in the arctic-data folder on your server account, make your changes, save the changes, stage them by clicking the box, commit, pull, then push to merge your version with the main version on the website. Do NOT edit scripts using the GitHub website. It is much easier to accidentally overwrite the history of a file this way. One thing you might be wondering as you are working on a script is, how often should I be committing my changes? It might not make sense to commit-pull-push after every single tiny change - if only because it would slow you way down. Personally, I commit every time I feel that a significant change has happened and that the chunk of code I was working on is “done”. Sometimes this is an entire script, other times it is just a few lines within a script. A good sign that you are committing too infrequently might be if many of your commit messages address a wide variety of coding tasks, such as: “wrote for loop to create referenced attribute lists for tables 1:20. also created nesting structure for this package with another package. also created attribute list for data table 40”. One final note, you can make multiple commits before you push to the repo. If you are making lots of changes to the script, you might want to make several commits before pull-push. You can see how many commits you are ahead of the “origin/master” branch (i.e. what you see on the website) by looking for text in your Git tab in RStudio that looks like this: 8.5.3 Where do I commit? The default right now is to save data-processing scripts in the arctic-data/datateam/data-processing/ directory, with sub-folders listed by project. Directories can be created as needed but please ask Dom or Jesse first so we can try and maintain some semblance of order in the file structure. The Github workflow diagram from stamminator shows each step and their functions explicitly. 8.6 My Git tab disappeared Sometimes R will crash so hard it loses your project information, causing your Git tab to disappear. Most likely, RStudio has just closed your “project” and all you need to do is reopen it. If your Git tab has disappeared, in the top right of your RStudio session, you should see a little R logo with “Project: (None)” next to it. This means you do not currently have a project open. Clicking the arrow should give you a dropdown menu of recent projects, where you can select “arctic-data” or “sasap-data.” Once you have opened your project, the Git tab should reappear! This is also a convenient way to switch between projects if you are working in multiple repositories. "],["first-ticket.html", "Chapter 9 First Ticket 9.1 Navigate RT 9.2 Initial review checklist 9.3 Processing templates 9.4 Final Checklist 9.5 Email templates 9.6 Categorize datasets 9.7 Congrats!", " Chapter 9 First Ticket After completing the previous chapters, your supervisor will assign a ticket from RT. Login using your LDAP credentials got get familiarized with RT. 9.1 Navigate RT The RT ticketing system is how we communicate with folks interacting with the Arctic Data Center. We use it for managing submissions, accessing issues, etc. It consists of three separate interfaces: Front Page All Tickets Ticket Page 9.1.1 Front page This is what you see first Home - brings you to this homepage Tickets - to search for tickets (also see number 5) Tools - not needed New Ticket - create a new ticket Search - Type in the ticket number to quickly navigate to a ticket Queue - Lists all of the tickets currently in a particular queue (such as ‘arcticdata’) and their statuses New = unopened tickets that require attention Open = tickets currently open and under investigation and/or being processed by a support team member Stalled = tickets awaiting responses from the PI/ submitter Tickets I Own - These are the current open tickets that are claimed by me Unowned Tickets - Newest tickets awaiting claim Ticket Status - Status and how long ago it was created Take - claim the ticket as yours 9.1.2 All tickets This is the queue interface from number 6 of the Front page 1. Ticket number and title 2. Ticket status 3. Owner - who has claimed the ticket 9.1.3 Example ticket Title - Include the PI’s name for reference Display - homepage of the ticket History - Comment/Email history, see bottom of Display page Basics - edit the title, status, and ownership here People - option to add more people to the watch list for a given ticket conversation. Note that user/ PI/ submitter email addresses should be listed as “Requestors”. Requestors are only emailed on “Replys”, not “Comments”. Ensure your ticket has a Requestor before attempting to contact users/ PIs/ submitters Links - option to “Merge into” another ticket number if this is part of a larger conversation. Also option to add a reference to another ticket number Verify that this is indeed the two tickets you want to merge. It is non-reversible. Actions Reply - message the submitter/ PI/ all watchers Comment - attach internal message (no submitters, only Data Teamers) Open It - Open the ticket Stall - submitter has not responded in greater than 1 month Resolve - ticket completed History - message history and option to reply (to submitter and beyond) or comment (internal message) 9.1.4 New data submission When notified by Arcticbot about a new data submission, here are the typical steps: Update the Requestor under the People section based on the email given in the submission (usually the user/ PI/ submitter). You may have to google for the e-mail address if the PI did not include it in the metadata record. Take the ticket (Actions &gt; Take) Review the submission based on the checklist Draft an email using the template and let others review it via Slack Send your reply via Actions Before opening a R script first look over the initial checklist first to identify what you will need to update in the metadata. 9.2 Initial review checklist Before responding to a new submission use this checklist to review the submission. When your are ready to respond use the initial email template and insert comments and modify as needed. 9.2.0.1 Sensitive Data If any of the below is in the dataset, please alert the #arctica team know before proceeding. Check if there is any sensitive information or personal identifying information in the data (eg. Names) Can the data be disaggregated and de-anonymized? (eg. a small sample size and individuals could be easily identified by their answers) Dryad Human Subject data guidelines can be a good place to start Common Cases: Social Science: Any dataset involving human subjects (may include awards awarded by ASSP and topics such as COVID-19) Archaeology: archaeological site location information, which is protected from public access by law Biology: protected species location coordinates 9.2.0.2 Data citations If the dataset appears to be in a publication please (might be in the abstract) make sure that those citations are registered. 9.2.0.3 Title WHAT, WHERE, and WHEN: Is descriptive of the work (provides enough information to understand the contents at a general scientific level), AND includes temporal coverage Provides a location of the work from the local to state or country level Provides a time frame of the work NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out 9.2.0.4 Abstract Describes the DATA as well as: The motivation (purpose) of the study Where and when the research took place At least one sentence summarizing general methodologies NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out At least 100 words total tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting Any citations to papers can be registered with us 9.2.0.5 Keywords Some keywords are included 9.2.0.6 Data Data is normalized (if not suggest to convert the data if possible) At least one data file, or an identifier to the files at another approved archive, unless funded by ASSP (Arctic Social Sciences Program) No xls/xlsx files (or other proprietary files) File contents and relationships among files are clear Each file is well NAMED and DESCRIBED and clearly differentiated from all others All attributes in EML match attribute names in respective data files EXACTLY, are clearly defined, have appropriate units, and are in the same order as in the file. Quality control all dimensionless units. Missing value codes are explained (WHY are the data absent?) If it is a .rar file -&gt; scan the file If there is the unit tons make sure to ask if it is metric tons or imperical tons if not clarified already 9.2.0.7 People &amp; Parties At least one contact and one creator with a name, email address, and ORCID iD 9.2.0.8 Coverages Includes coverages that make sense Temporal coverage - Start date BEFORE end date Geologic time scales are added if mentioned in metadata (e.g. 8 Million Years or a name of a time period like Jurassic) Spatial coverage matches geographic description (check hemispheres) Geographic description is from the local to state or country level, at the least Taxonomic coverage if appropriate 9.2.0.9 Project Information At least one FUNDING number Title, personnel, and abstract match information from the AWARD (not from the data package) 9.2.0.10 Methods This section is REQUIRED for ALL NSF-FUNDED data packages Enough detail is provided such that a reasonable scientist could interpret the study and data for reuse without needing to consult the researchers, nor any other resources 9.2.0.11 Portals If there are multiple submissions from the same people/project let them know about the portals feature If this is part of a portal make sure this dataset can be found there. Additional steps might be needed to get that to work. Please consult Jeanette and see the data portals section. 9.3 Processing templates We have developed some partially filled R scripts to get you started on working on your first dataset. They outline common functions used in processing a dataset. However, it will differ depending on the dataset. You can use this template where you can fill in the blanks to get familiar with the functions we use and workflow at first. We also have a more minimal example A filled example as a intermediate step. You can look at the filled example if you get stuck or message the #datateam. In addition, you may find this cheat sheet of data team R functions helpful. Once you have updated the dataset to your satisfaction and reviewed the Final Checklist, post the link to the dataset on #datateam for peer review. 9.4 Final Checklist You can click on the assessment report on the website to for a general check. Fix anything you see there. Send the link over slack for peer review by your fellow datateam members. Usually we look for the following (the list is not exhaustive): 9.4.1 Special Datasets Please refer to the dedicated pages for instructions to handle these cases: MOSAiC DBO 9.4.2 System Metadata the format ids are correct 9.4.3 General EML Ethical Research Practice section is completed Publisher and system information have been added: doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) 9.4.4 Title Include geographic and temporal coverage (when and where data was collected) Abbreviations are removed or defined 9.4.5 Abstract longer than 100 words Abbreviations are removed or defined. No garbled text Tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting 9.4.6 DataTable / OtherEntity / SpatialVectors In the correct one: DataTable / OtherEntity / SpatialVector / SpatialRaster for the file type entityDescription - longer than 5 words and unique physical present and up-to-date and format is correct 9.4.6.1 Attribute Table Complete attributeDefinitions longer than 3 words Variables match what is in the file Measurement domain - if appropirate (ie dateTime correct) Missing Value Code - accounted for if applicable Semantic Annotation - appropriate semantic annotations added, especially for spatial and temporal variables: lat, lon, date etc. Custom units were created if necessary 9.4.7 People Complete information for each person in each section includes ORCID and e-mail address for all contacts people repeated across sections have consistent information 9.4.8 Geographic region the map looks correct and matches the geographic description check if negatives (-) are missing 9.4.9 Project If it is an NSF award you can use the helper function: doc$dataset$project &lt;- eml_nsf_to_project(awards) for other awards that need to be set manually, see the set project page 9.4.10 Methods Present No garbled text Acronyms and units are defined the first time they are used 9.4.11 Check EML Version Currently using: eml-2.2.0 (as of July 30 2020) Review to see if the EML version is set correctly by reviewing the doc$`@context` that it is indeed 2.2.0 under eml Re-run your code again and have the lineemld::eml_version(\"eml-2.2.0\") at the top Make sure the system metadata is also 2.2.0 9.4.12 Access If necessary, granted access to PI using set_rights_and_access() make sure it is http:// (no s) note if it is a part of portals there might be specific access requirements for it to be visible using set_access() 9.4.13 SFTP Files If there are files transferred to us via SFTP, delete those files once they have been added to the dataset and when the ticket is resolved 9.4.14 Updated datasets All the above applies. These are some areas to do a closer check when users update with a new file: New data was added Temporal Coverage and Title Confirm if new data used same methods or if methods need updating Files were replaced update physical and entityName double-check attributes are the same check for any new missing value codes that should be accounted for Was the dataset published before 2021? update project info, annotations Glance over entire page for any small mistakes (ie. repeated additionalMetadata, any missed &amp;amps, typos) After all the revisions send the link to the PI in an email through RT. Send the draft of the email to Daphne or Jeanette on Slack. 9.5 Email templates This section covers new data packages submitted. For other inquiries see the PI FAQ templates Please think critically when using these canned replies rather than just blindly sending them. Typically, content should be adjusted/ customized for each response to be as relevant, complete, and precise as possible. In your first few months, please run email drafts by the #datateam Slack and get approval before sending. Remember to consult the submission guidelines for details of what is expected. Quick reference: Initial email template Final email templates Additional email template 9.5.1 Initial email template Hello [NAME OF REQUESTOR], Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: [COMMENTS HERE] After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, [YOUR NAME] 9.5.2 Comment templates based on what is missing 9.5.2.1 Portals Multiple datasets under the same project - suggest data portal feature I would like to highlight the Data Portals feature that would enhance the ability of your datasets to be discovered together. It also enables some custom branding for the group or project. Here is an example page that we created for the Distributed Biological Observatory: https://arcticdata.io/catalog/portals/DBO. We highly suggest considering creating a portal for your datasets, you can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/. Data portals can be set up at any point of your project. If they ask to nest the dataset We are no longer supporting new nested datasets. We recommend to create a data portal instead. Portals will allow more control and has the same functionality as a nested dataset. You can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/, but as always we are here to help over email. 9.5.2.2 Dataset citations If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be done at any point. 9.5.2.3 Title Provides the what, where, and when of the data We would like to add some more context to your data package title. I would like to suggest: ‘OUR SUGGESTION HERE, WHERE, WHEN’. Does not use acronyms We wanted to clarify a couple of abbreviations. Could you help us in defining some of these: [LIST OF ACRONYMS TO DEFINE HERE] 9.5.2.4 Abstract Describes DATA in package (ideally &gt; 100 words) We would like to add some additional context to your abstract. We hope to add the following: [ADJUST THE DEPENDING ON WHAT IS MISSING] The motivation of the study Where and when the research took place At least one sentence summarizing general methodologies All acronyms are defined At least 100 words long Offer this if submitter is reluctant to change: If you prefer and it is appropriate, we could add language from the abstract in the NSF Award found here: [NSF AWARD URL]. 9.5.2.5 Keywords We noticed that there were no keywords for this dataset. Adding keywords will help your dataset be discovered by others. 9.5.2.6 Data Sensitive Data We will need to ask these questions manually until the fields are added to the webform. Data sensitivity categories Once we have the ontology this question can be asked: Based on our Data sensitivity categories, which of the 3 does your dataset align with most: Non-sensitive data - None of the data includes sensitive or protected information. Some or all data is sensitive with minimal risk - Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Some or all data is sensitive with significant risk - The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Ethical research proceedures We were wondering if you could also address this question specifically on Ethical Research Procedures: Describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Insitutional Review Board approvals, consent waivers, procedures for co-production, data sovreignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. We can help add your answers to the question to the metadata. Adding provenance Is the [mention the file names here] files related? If so we can add provenance to show the relationship between the files. Here is an example of how that is displayed: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Af00c4d71-0242-4e9d-9745-8999776fa2f2 At least one data file We noticed that no data files were submitted. With the exception of sensitive social science data, we seek to include all data products prior to publication. We wanted to check if additional files will be submitted before we move forward with the submission process. Open formats Example using xlsx. Tailor this reponse to the format in question. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. The data files can be replaced by going to the green Edit button &gt; Click the black triangle by the Describe button for the data file &gt; Select Replace (attached is also a screenshot on how to get there). Zip files Except for very specific file types, we do not recommend that data are archived in zip format. Data that are zipped together can present a barrier to reuse since it is difficult to accurately document each file within the zip file in a machine readable way. File contents and relationships among files are clear Could you provide a short description of the files submitted? Information about how each file was generated (what software, source files, etc.) will help us create more robust metadata for long term use. Data layout Would you be able to clarify how the data in your files is laid out? Specifically, what do the rows and columns represent? We try not to prescribe a way the researchers must format their data as long as reasonable. However, in extreme cases (for example Excel spreadsheets with data and charts all in one sheet) we will want to kindly ask them to reformat. We would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. [DESCRIBE WHAT MAY NEED TO BE CHANGED IN THE DATA SET]. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. 9.5.2.7 Attributes Identify which attributes need additional information. If they are common attributes like date and time we do not need further clarification. Checklist for the datateam in reviewing attributes (NetCDF, CSV, shapefiles, or any other tabular datasets): A name (often the column or row header in the file). A complete definition. Any missing value codes along with explanations for those codes. For all numeric data, unit information is needed. For all date-time data, a date-time format is needed (e.g. “DD-MM-YYYY”). For text data, full descriptions for all patterns or code/definition pairs are needed if the text is constrained to a list of patterns or codes. Helpful templates: &gt; We would like your help in defining some of the attributes. Could you write a short description or units for the attributes listed? [Provide a the attribute names in list form] &gt; Could you describe ____? &gt; Please define “XYZ”, including the unit of measure. &gt; What are the units of measurement for the columns labeled “ABC” and “XYZ”? Missing value codes What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. This section is not yet available on our webform so we will add that information on your behalf. We noticed that the data files contain [blank cells - replace with missing values found]. What do these represent? 9.5.2.8 Funding All NSF funded datasets need a funding number. Non-NSF funded datasets might not have funding numbers, depending on the funding organization. We noticed that your dataset does not appear to contain a funding number. The field accepts NSF funding numbers as well as other numbers by different organizations. 9.5.2.9 Methods We noticed that methods were missing from the submission. Submissions should include the following: provide instrument names (if applicable) specify how sampling locations were chosen if citations for sampling methods are used, please provide a brief summary of the methods referenced any software used to process the data Note - this includes software submissions as well (see https://arcticdata.io/submit/#metadata-guidelines-for-software) Your methods section appears to be missing some information. [ADJUST THIS DEPENDING ON WHAT IS MISSING - Users should be able to understand how the data were collected, how to interpret the values, and potentially how to use the data in the case of specialized files.] Comprehensive methods information should be included directly in the metadata record. Pointers or URLs to other sites are unstable. A full example - New Submission: methods, excel to csv, and attributes Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be added at any point. We would like to add some more context to your data package title. I would like to suggest: Holocene lake-based Arctic glacier and ice cap records. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. We also would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. If the data was in a long rather than wide format, it would be easier to us to document. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. We noticed that methods were missing from the submission. Submissions should: - provide instrument names (if applicable) - specify how sampling locations were chosen - provide citations for sampling methods that are not explained in detail - any software used to process the data After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, Name 9.5.3 Final email templates 9.5.3.1 Asking for approval Hi [submitter], I have updated your data package and you can view it here after logging in: [URL] Please review and approve it for publishing or let us know if you would like anything else changed. For your convenience, if we do not hear from you within a week we will proceed with publishing with a DOI. After publishing with a DOI, any further changes to the dataset will result in a new DOI. However, any previous DOIs will still resolve and point the user to the newest version. Please let us know if you have any questions. 9.5.3.2 DOI and data package finalization comments Replying to questions about DOIs We attribute DOIs to data packages as one might give a DOI to a citable publication. Thus, a DOI is permanently associated with a unique and immutable version of a data package. If the data package changes, a new DOI will be created and the old DOI will be preserved with the original version. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to that latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Clarification of updating with a DOI and version control We definitely support updating a data package that has already been assigned a DOI, but when we do so we mark it as a new revision that replaces the original and give it its own DOI. We do that so that any citations of the original version of the data package remain valid (i.e.: after the update, people still know exactly which data were used in the work citing it). 9.5.3.3 Resolve the ticket Sending finalized URL and dataset citation before resolving ticket [NOTE: the URL format is very specific here, please try to follow it exactly (but substitute in the actual DOI of interest)] Here is the link and citation to your finalized data package: https://doi.org/10.18739/A20X0X First Last, et al. 2021. Title. Arctic Data Center. doi:10.18739/A20X0X. If in the future there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. Please let us know if you need any further assistance. 9.5.4 Additional email templates 9.5.4.1 Deadlines If the PI is checking about dates/timing: &gt; [give rough estimate of time it might take] &gt; Are you facing any deadlines? If so, we may be able to expedite publication of your submission. 9.5.4.2 Pre-assigned DOI If the PI needs a DOI right away: We can provide you with a pre-assigned DOI that you can reference in your paper, as long as your submission is not facing a deadline from NSF for your final report. However, please note that it will not become active until after we have finished processing your submission and the package is published. Once you have your dataset published, we would appreciate it if you could register the DOI of your published paper with us by using the citations button beside the orange lock icon. We are working to build our catalog of dataset citations in the Arctic Data Center. 9.5.4.3 Sensitive Data Which of the following categories best describes the level of sensitivity of your data? A. Non-sensitive data None of the data includes sensitive or protected information. Proceed with uploading data. B. Some or all data is sensitive but has been made safe for open distribution Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Proceed with uploading data, but ensure that only data that are safe for public distribution are uploaded. Address questions about anonymization, aggregation, de-identification, and data embargoes with the data curation support team before uploading data. Describe these approaches in the Methods section. C. Some or all data is sensitive and should not be distributed The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Do NOT upload sensitive data. You should still upload a metadata description of your dataset that omits all sensitive information to inform the community of the dataset’s existence. Contact the data curation support team about possible alternative approaches to safely preserve sensitive or protected data. Ethical Research Procedures. Please describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Institutional Review Board approvals, consent waivers, procedures for co-production, data sovereignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. 9.5.4.4 Asking for dataset access As a security measure we ask that we get the approval from the original submitter of the dataset prior to granting edit permissions to all datasets. 9.5.4.5 No response from the researcher Please email them before resolving a ticket like this: We are resolving this ticket for bookkeeping purposes, if you would like to follow up please feel free to respond to this email. 9.5.4.6 Recovering Dataset submissions To recover dataset submissions that were not successful please do the following: Go to https://arcticdata.io/catalog/drafts Find your dataset and download the corresponding file Send us the file in an email 9.5.4.7 Custom Search Link You could also use a permalink like this to direct users to the datasets: https://arcticdata.io/catalog/data/query=\"your search query here” for example: https://arcticdata.io/catalog/data/query=Beaufort%20Lagoon%20Ecosystems%20LTER 9.5.4.8 Adding metadata via R KNB does not support direct uploading of EML metadata files through the website (we have a webform that creates metadata), but you can upload your data and metadata through R. Here are some training materials we have that use both the EML and datapack packages. It explains how to set your authentication token, build a package from metadata and data files, and publish the package to one of our test sites. I definitely recommend practicing on a test site prior to publishing to the production site your first time through. You can point to the KNB test node (dev.nceas.ucsb.edu) using this command: d1c &lt;- D1Client(\"STAGING2\", \"urn:node:mnTestKNB\") If you prefer, there are Java, Python, MATLAB, and Bash/cURL clients as well. 9.5.4.9 Finding multiple data packages If linking to multiple data packages, you can send a link to the profile associated with the submitter’s ORCID iD and it will display all their data packages. e.g.: https://arcticdata.io/catalog/profile/http://orcid.org/0000-0002-2604-4533 9.5.4.10 NSF ARC data submission policy Please find an overview of our submission guidelines here: https://arcticdata.io/submit/, and NSF Office of Polar Programs policy information here: https://www.nsf.gov/pubs/2016/nsf16055/nsf16055.jsp. Investigators should upload their data to the Arctic Data Center (https://arcticdata.io), or, where appropriate, to another community endorsed data archive that ensures the longevity, interpretation, public accessibility, and preservation of the data (e.g., GenBank, NCEI). Local and university web pages generally are not sufficient as an archive. Data preservation should be part of the institutional mission and data must remain accessible even if funding for the archive wanes (i.e., succession plans are in place). We would be happy to discuss the suitability of various archival locations with you further. In order to provide a central location for discovery of ARC-funded data, a metadata record must always be uploaded to the Arctic Data Center even when another community archive is used. 9.5.4.11 Linking ORCiD and LDAP accounts First create an account at orcid.org/register if you have not already. After that account registration is complete, login to the KNB with your ORCID iD here: https://knb.ecoinformatics.org/#share. Next, hover over the icon on the top right and choose “My Profile”. Then, click the “Settings” tab and scroll down to “Add Another Account”. Enter your name or username from your Morpho account and select yourself (your name should populate as an option). Click the “+”. You will then need to log out of knb.ecoinformatics.org and then log back in with your old LDAP account (click “have an existing account”, and enter your Morpho credentials with the organization set to “unaffiliated”) to finalize the linkage between the two accounts. Navigate to “My Profile” and “Settings” to confirm the linkage. After completing this, all of your previously submitted data pacakges should show up on your KNB “My Profile” page, whether you are logged in using your ORCiD or Morpho account, and you will be able to submit data either using Morpho or our web interface. Or, try reversing my instructions - log in first using your Morpho account (by clicking the “existing account” button and selecting organization “unaffiliated”), look for your ORCiD account, then log out and back in with ORCiD to confirm the linkage. Once the dataset is approved by the PI and there are no further changes, publish the dataset with a doi. doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) 9.6 Categorize datasets As a final step we will categorize the dataset you processed. We are trying to categorize datasets so we can have a rough idea of what kinds of datasets we have at the Arctic Data Center. We will grant you access to the google sheet that has all of the categorized datasets We will categorize each dataset into one of the predefined themes (ie. biology, ecology etc.). Definition of the themes can be found in the google sheet Run the following line with your doi and themes as a list. datamgmt::categorize_dataset(&quot;your_doi&quot;, c(&quot;list&quot;, &quot;of&quot;, &quot;themes&quot;), &quot;your name&quot;) 9.7 Congrats! Congratulations on finishing your first ticket! You can head over to the the repository, data-processing to get your ticket processing code reviewed by the team so we can learn from each other! "],["beyond-your-first-ticket.html", "Chapter 10 Beyond your first ticket 10.1 Working with large data packages 10.2 Add physicals to submissions 10.3 Use references 10.4 Annotations 10.5 Streamlining your workflow 10.6 Code Snippets 10.7 Resources for R", " Chapter 10 Beyond your first ticket This section is meant to be read after you have processed a couple of tickets and you are comfortable with the workflow with a relatively simple dataset with 1-2 files and want to expand your skills and workflows further. 10.1 Working with large data packages 10.2 Add physicals to submissions New submissions made through the web editor will not have any physical sections within the otherEntitys. Add them to the EML with the following script: for (i in seq_along(doc$dataset$otherEntity)) { otherEntity &lt;- doc$dataset$otherEntity[[i]] id &lt;- otherEntity$id if (!grepl(&quot;urn-uuid-&quot;, id)) { warning(&quot;otherEntity &quot;, i, &quot; is not a pid&quot;) } else { id &lt;- gsub(&quot;urn-uuid-&quot;, &quot;urn:uuid:&quot;, id) physical &lt;- arcticdatautils::pid_to_eml_physical(mn, id) doc$dataset$otherEntity[[i]]$physical &lt;- physical } } As you can see from code above, we use a for loop here to add physical sections. The for loop is a very useful tool to iterate over a list of elements. With for loop, you can repeat a specific block of code without copying and pasting the code over and over again. When processing datasets in Arctic Data Center, there are many places where for loop can be used, such as publishing a bunch of objects with pids, updating formatID for pkg$data, adding physical section like above code, etc. A loop is composed of two parts: the sequence and the body. The sequence usually generates indices to locate elements and the body contains the code that you want to iterate for each element. Here is an example of adding the same attributeList for all the dataTables in the metadata using for loop. attributes &lt;- read.csv(&#39;attributes.csv&#39;) # attribute table in csv format attributeList &lt;- EML::set_attributes(attributes = attributes) for (i in 1:length(doc$dataset$dataTable)) { # sequence part doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList # body part } 10.3 Use references References are a way to avoid repeating the same information multiple times in the same EML record. There are a few benefits to doing this, including: Making it clear that two things are the same (e.g., the creator is the same person as the contact, two entities have the exact same attributes) Reducing the size on disk of EML records with highly redundant information Faster read/write/validate with the R EML package You may want to use EML references if you have the following scenarios (not exhaustive): One person has multiple roles in the dataset (creator, contact, etc) One or more entities shares all or some attributes 10.3.1 Example with parties Do not use references for creators as it is used for the citation information. The creators will not show up on the top of the dataset if it is a reference. Until this issue is resolved in NCEAS/metacat#926 we will need to keep this in account. It’s very common to see the contact and creator referring to the same person with XML like this: &lt;eml packageId=&quot;my_test_doc&quot; system=&quot;my_system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1 eml.xsd&quot;&gt; &lt;dataset&gt; &lt;creator&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml&gt; So you see those two times Bryce Mecum is referenced there? If you mean to state that Bryce Mecum is the creator and contact for the dataset, this is a good start. But with just a name, there’s some ambiguity as to whether the creator and contact are truly the same person. Using references, we can remove all doubt. doc$dataset$creator[[1]]$id &lt;- &quot;reference_id&quot; doc$dataset$contact &lt;- list(references = &quot;reference_id&quot;) print(doc) &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; &lt;eml:eml xmlns:eml=&quot;eml://ecoinformatics.org/eml-2.1.1&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns:stmml=&quot;http://www.xml-cml.org/schema/stmml-1.1&quot; packageId=&quot;id&quot; system=&quot;system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1/ eml.xsd&quot;&gt; &lt;dataset&gt; &lt;title&gt;A Minimal Valid EML Dataset&lt;/title&gt; &lt;creator id=&quot;reference_id&quot;&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;references&gt;reference_id&lt;/references&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml:eml&gt; The reference id needs to be unique within the EML record but doesn’t need to have meaning outside of that. 10.3.2 Example with attributes To use references with attributes: Add an attribute list to a data table Add a reference id for that attribute list Use references to add that information into the attributeLists of the other data tables For example, if all the data tables in our data package have the same attributes, we can set the attribute list for the first one, and use references for the rest: doc$dataset$dataTable[[1]]$attributeList &lt;- attribute_list doc$dataset$dataTable[[1]]$attributeList$id &lt;- &quot;shared_attributes&quot; # use any unique name for your id for (i in 2:length(doc$dataset$dataTable)) { doc$dataset$dataTable[[i]]$attributeList &lt;- list(references = &quot;shared_attributes&quot;) # use the id you set above } 10.4 Annotations If there are multiple tables with similar annotations you can try something like this: #go through each dataTable for(i in 1:length(doc$dataset$dataTable)){ #get all the attibute names an &lt;- eml_get_simple(doc$dataset$dataTable[[i]], &quot;attributeName&quot;) #go through the attributes figure out what will match for(a in 1:length(an)){ annotation &lt;- dplyr::case_when( # the attributeName to match ~ valueLabel an[[a]] == &quot;Sample ID&quot; ~ &quot;Identity&quot;, an[[a]] == &quot;Location&quot; ~ &quot;study location name&quot;, an[[a]] == &quot;Latitude&quot; ~ &quot;latitude coordinate&quot;, an[[a]] == &quot;Longitude&quot; ~ &quot;longitude coordinate&quot;, an[[a]] == &quot;Elevation (m)&quot; ~ &quot;elevation&quot;, str_detect(an[[a]], &quot;Depth|depth&quot;) ~ &quot;Depth&quot;) #only run this code when the annotations match if(!is.na(annotation)){ #based on the entity Name create a unique id entity &lt;- str_split(doc$dataset$dataTable[[i]]$entityName, &quot;_&quot;) doc$dataset$dataTable[[i]]$attributeList$attribute[[a]]$id &lt;- paste0(entity[[1]][[1]], &quot;_&quot;, an[[a]]) #add the annotation doc$dataset$dataTable[[i]]$attributeList$attribute[[a]]$annotation &lt;- eml_ecso_annotation(annotation) } } } 10.5 Streamlining your workflow 10.6 Code Snippets Code snippets help with templating portions of code that you will be using regularly. To add your own, go to the toolbar ribbon at the top of your Rstudio screen and select: Tools &gt; Global Options... &gt; Code &gt; Edit Snippets &gt; Add these chunks to the end of the file More info can be found in this blog post by Mara Averick on how to add them: https://maraaverick.rbind.io/2017/09/custom-snippets-in-rstudio-faster-tweet-chunks-for-all/ Usual arcticdatautils ticket workflow snippet ticket library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;add your rm&quot; pkg &lt;- get_package(adc, rm) doc &lt;- EML::read_eml(getObject(adc, pkg\\$metadata)) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) eml_validate(doc) eml_path &lt;- &quot;eml.xml&quot; write_eml(doc, eml_path) #update &lt;- publish_update(adc, # metadata_pid = pkg\\$metadata, # resource_map_pid = pkg\\$resource_map, # metadata_path = eml_path, # data_pids = pkg\\$data, # public = F) #datamgmt::categorize_dataset(update\\$metadata, c(&quot;theme1&quot;), &quot;Your Name&quot;) The datapack ticket workflow snippet datapack library(dataone) library(datapack) library(digest) library(uuid) library(dataone) library(arcticdatautils) library(EML) d1c &lt;- D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) packageId &lt;- &quot;id here&quot; dp &lt;- getDataPackage(d1c, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) #get metadata id metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #edit the metadata doc &lt;- read_eml(getObject(d1c@mn, metadataId)) #add the publisher info doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) doc\\$dataset\\$project &lt;- eml_nsf_to_project(&quot;nsf id here&quot;) #check and save the metadata eml_validate(doc) eml_path &lt;- arcticdatautils::title_to_file_name(doc\\$dataset\\$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) #upload the dataset myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c, dp, public=F, accessRules=myAccessRules, quiet=FALSE) #datamgmt::categorize_dataset(&quot;doi&quot;, c(&quot;theme1&quot;), &quot;Jasmine&quot;) Quick way to give access to submitters to their datasets: snippet access library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;rm here&quot; pkg &lt;- get_package(adc, rm) set_access(adc, unlist(pkg), &quot;orcid here&quot;) Quick access to the usual code for common Solr queries: snippet solr library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) 10.7 Resources for R 10.7.1 Learning The following online books are useful for expanding your R knowledge and skills: the most recent ADC training materials The cleaning and data manipulation section is useful for working with attribute tables Efficient R Programming In particular Chapter 3 Efficient Programming R for Data Science Section on Strings R Packages contributing to arcticdatatutils, datamgmt and EML Advanced R Object-oriented programming in R for S4 to understand how datapack and dataone packages are written bookdown: Authoring Books and Technical Documents with R Markdown formatting, troubleshooting and updating the training document Others Hands-On Programming with R R Programming for Data Science Exploratory Data Analysis with R Mastering Software Development in R Geocomputation with R R Markdown: The Definitive Guide The Tidyverse Style Guide The RStudio cheatsheets are also useful references for functions in tidyverse and other packages. 10.7.2 Packages The data team uses and develops a number of R packages. Here is a listing and description of the main packages: dataone reading and writing data at DataONE member nodes http://doi.org/10.5063/F1M61H5X datapack creating and managing data packages https://github.com/ropensci/datapack EML creating and editing EML metadata documents https://ropensci.github.io/EML arcticdatautils utility functions for processing data for the Arctic Data Center https://nceas.github.io/arcticdatautils/ datamgmt data management utilities for curating, documenting, and publishing data (sandbox package) https://nceas.github.io/datamgmt/ metadig authoring MetaDIG quality checks https://github.com/NCEAS/metadig-r metajam downloading and reading data and metadata from DataONE member nodes https://nceas.github.io/metajam/ "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]
+[["index.html", "NCEAS Data Team Training Welcome to NCEAS! First day to-dos Account information NCEAS events NCEAS Community Meetings/ Groups Internship Expectations Code of Conduct Citation Information", " NCEAS Data Team Training Jesse Goldstein, Dominic Mullen, Jeanette Clark, Irene Steves, Mitchell Maier, Bryce Mecum, Peter Slaughter, Emily O’Dean, Jasmine Lai, Daphne Virlar-Knight, Maggie Klope 2024-08-13 Welcome to NCEAS! First day to-dos Get a tour of the office from Ginger Fill out required paperwork you received via email Set up the remainder of your accounts Account information LDAP - this should be set up prior to your start date in order to help get other accounts set. This account and password control your access to: arcticdata RT queue GitHub Enterprise - arctic-data Datateam server - follow instructions in email from Nick at NCEAS to reset your datateam password in the terminal ORCiD - create an account Login to test.arcticdata.io with your ORCID iD NCEAS Slack - get an invite from slack.nceas.ucsb.edu Channels to join: #arctica, #arcticbot, #computing, #datateam, #devteam Introduce yourself in #datateam Arctic Data Center Team - after creation of ORCiD and sign-in to both arcticdata.io and test.arcticdata.io, request addition to the admin group from Justin or Maggie in Slack GitHub - if you do not have a public GitHub account already, please register for one here If you are an intern, fill out anticipated quarterly schedule on the intern google calendar shared with you. Electronic Timekeeping - make sure you can log on to electronic timekeeping via your UCSBNetID and password (may not be accessible on the first day, if you continue to have issues please let Ana know). If you are an hourly employee, log your hours for your first day! Under today’s date select ‘Hours Worked’ under the Pay Code column, enter the amount of hours under the Amount column, and finally click the ‘Save’ button in the top right. At the end of every two-week pay period you will also need to click the ‘Approve Timecard’ button in order to submit your timecard. UCPath - you can set up your paycheck preferences here, including direct deposit and income tax withholding. Detailed Instructions Let Justin or Maggie know what email you would like to use for general NCEAS updates from all@nceas.ucsb.edu NCEAS events NCEAS hosts a number of events that you are encouraged to attend. Keep an eye on your email but the recurring events are: Roundtable presentation and discussion of research by a visiting or local scientist first or second Thursdays of the month at 3:30 in the lounge and via zoom followed by happy hour at 4:30 on the terrace Coffee Klatch coffee, socializing, and news updates for NCEAS Tuesdays at 10:30 in the lounge NCEAS Community Meetings/ Groups Check out their individual calendar entries and channels for more information Early Career Researcher Community Forum - #ecr_community Hacky Hours - #hackyhour Data Science Chats - #data-science-chats NCEAS Diversity Team NCEAS Book Club - #bookclub NCEAS Social - #social Internship Expectations As an intern with the data team, there are a few expectations that the Project Coordinators have of you. Overall, we expect you to be communicative and proactive. We want you to learn and grow in this position, but we don’t want you spinning your wheels going nowhere fast! If you’ve spent 10-15 minutes on an issue and you’re not making any progress, reach out to us and your peers for help in the #datateam slack channel. The #datateam slack channel is our main form of communication, and we expect all interns to become comfortable communicating in this space. By posting your questions and code in the #datateam channel (instead of sending direct messages), multiple people will be able to help at once, and we all can learn from the problems that our peers encounter. Additionally, we expect interns to work within the standard business hours of 8am - 5pm (pacific time). We ask that you mark your expected work hours on the shared “Intern” Google Calendar. This is so that the Project Coordinators are aware of who’s working day-to-day and can plan their days accordingly. We also use this to verify time sheets when they are submitted. Ideally, interns would input their proposed hours on the calendar at least one week in advance. During exams and other unusually busy weeks at school, we understand you may need to shift your hours or reduce your workload. When this occurs, please make sure to email either Daphne or Jeanette so that we know not to expect you during your usual schedule. Code of Conduct Take a moment to review the diversity and inclusion page and code of conduct at NCEAS so that we can foster an environment that is safe, welcoming and inclusive for everyone. Citation Information This work is licensed under a Creative Commons Attribution 4.0 International License. Clark, Jeanette, Jesse Goldstein, Dominic Mullen, Irene Steves, Mitchell Maier, Stephanie Freund, Sharis Ochs, Bryce Mecum, Peter Slaughter, Emily O’Dean, Jasmine Lai, Daphne Virlar-Knight. 2022 Training Materials for the Arctic Data Center Curation Team. Arctic Data Center. doi:10.18739/A20G3GX8W. "],["introduction-to-open-science.html", "Chapter 1 Introduction to open science 1.1 Background reading 1.2 Effective data management 1.3 Using DataONE 1.4 Working on a remote server 1.5 A note on paths 1.6 A note on R 1.7 A note on effective troubleshooting in R 1.8 A note on Exercises 1.9 Exercise 1", " Chapter 1 Introduction to open science These materials are meant to introduce you to the principles of open science, effective data management, and data archival with the DataONE data repository. It also provides an overview on the tools we will be using (remote servers, Rstudio, R, Troubleshooting, Exercises) throughout the training. This document is meant to take multiple days to complete, depending on your previous knowledge. We believe in allowing employees the space to fully grasp concepts during training, even if it means taking longer than expected. Quality learning is our priority, and there’s no pressure to finish within a specific timeframe. You may find it helpful to take notes on important concepts, and you will always be able to refer back to this training during your time at NCEAS. If you see anything that needs fixing, submit a issue in the github issues 1.1 Background reading Read the content on the Arctic Data Center (ADC) webpage to learn more about data submission, preservation, and the history of the ADC. We encourage you to follow the links within these pages to gain a deeper understanding. about submission preservation history 1.2 Effective data management Read Matt Jones et al.’s paper on effective data management to learn how we will be organizing datasets prior to archival. (Please note that while the tips outlined in this article are best practices, we often do not reformat data files submitted to our repositories unless necessary. It is best to be conservative and not alter other people’s data without good reason.) You may also want to explore the DataONE education resources related to data management. 1.3 Using DataONE Data Observation Network for Earth (DataONE) is a community driven initiative that provides access to data across multiple member repositories, supporting enhanced search and discovery of Earth and environmental data. Read more about what DataONE is here and about DataONE member node (MN) guidelines here. Please feel free to ask your supervisor any questions you have about DataONE. We will be applying these concepts in the next chapter. 1.4 Working on a remote server All of the work that we do at NCEAS is done on our remote server, datateam.nceas.ucsb.edu. If you have never worked on a remote server before, you can think of it like working on a different computer via the internet. We access RStudio on our server through this link. This is the same as your desktop version of RStudio with one main difference is that files are on the server. Please do all your work here, and bookmark this link. Do not use RStudio on your local computer. By only using the RStudio server, it is easier to share your code with the rest of us. 1.4.1 Check your understanding Open a new tab in your browser and try logging into the remote server using your SSH credentials. If your R session is frozen and unresponsive check out the guide on how to fix it. 1.5 A note on paths On the servers, paths to files in your folder always start with /home/yourusername/.... Note - if you are a more advanced user, you may use the method you prefer as long as it is evident where your file is from. When you write scripts, try to avoid writing relative paths (which rely on what you have set your working directory ~/ to) as much as possible. Instead, write out the entire path as shown above, so that if another data team member needs to run your script, it is not dependent on a working directory. 1.6 A note on R This training assumes basic knowledge of R and RStudio. Spend at least 15 minutes walking through Jenny Bryan’s excellent materials here for a refresher. Throughout this training we will occasionally use the namespace syntax package_name::function_name() when writing a function. This syntax denotes which package a function came from. For example dataone::getSystemMetadata selects the getSystemMetadata function from the dataone R package. More detailed information on namespaces can be found here. 1.7 A note on effective troubleshooting in R One of the advantages with using the R programming language is the extensive documentation that is available for R packages. The R help operator ? can be used to learn more about functions from all of the R packages we use. You can put the operator before the name of any function to view its documentation in RStudio: ?function When asking for help in the #datateam channel in Slack, we suggest using a combination of minimal reproducible examples (MRE) and the package reprex to create reproducible examples. This will allow others to better help you if we can run the code on our own computers. A MRE is stripping down your code to only the parts that cause the bug. When troubleshooting errors over Slack, send the code that returned an error and the error message itself. How to generate a reprex: copy the code you want to ask about call reprex() fix until everything runs smoothly copy the result to ask your question When copy and paste code slack message or github issues, use three backticks for code blocks and two backticks for a small piece of code will prevent issues with slack formats quotation. For more information and examples check out more of Jenny Bryan’s slides or watch the video starting at about the 10 min mark. Note for EML related MREs: - Generating a reprex for these situations (ie. tokens) might be complicated but you can should still follow the MRE principles even if the reprex won’t render fully - You can include a minimal EML to avoid some get_package issues: me &lt;- list(individualName = list(givenName = &quot;Jeanette&quot;, surName = &quot;Clark&quot;)) attributes &lt;- data.frame(attributeName = &#39;length_1&#39;, attributeDefinition = &#39;def1&#39;, measurementScale = &#39;ratio&#39;, domain = &#39;numericDomain&#39;, unit = &#39;meter&#39;, numberType = &#39;real&#39;) att_list &lt;- set_attributes(attributes) doc_ex &lt;- list(packageId = &quot;id&quot;, system = &quot;system&quot;, dataset = list(title = &quot;A Minimal Valid EML Dataset&quot;, creator = me, contact = me, dataTable = list(entityName = &quot;data table&quot;, attributeList = att_list)) ) 1.8 A note on Exercises The rest of the training has a series of exercises. These are meant to take you through the process as someone submitting a dataset from scratch. This is slightly different than the usual workflow but important in understanding the underlying system behind the Arctic Data Center. Please note that you will be completing everything on the site for the training. In the future if you are unsure about doing anything with a dataset. The test site is a good place to try things out! 1.9 Exercise 1 This part of the exercise walks you through submitting data through the web form on “test.arcticdata.io”. In addition to learning to use the webform, this exercise will also help you practice sleuthing for information in order to provide complete metadata. Most datasets do not come with all contextual information, so you will need to skim cited literature and search google for definitions of discipline-specific jargon. Don’t be afraid to use the internet as a resource! 1.9.1 Part 1 Download the csv of Table 1 from this paper. Reformat the table to meet the guidelines outlined in the journal article on effective data management (this might be easier to do in an interactive environment like Excel). Hint: This table is in wide format and can be made longer. Note: we usually don’t edit the content in data submissions so don’t stress over this part too much 1.9.2 Part 2 Go to “test.arcticdata.io” and submit your reformatted file with appropriate metadata that you derive from the text of the paper: List yourself as the first ‘Creator’ so your test submission can easily be found. For the purposes of this training exercise, not every single author needs to be listed with full contact details, listing the first two authors is fine. Directly copying and pasting sections from the paper (abstract, methods, etc.) is also fine. Attributes (column names) should be defined, including correct units and missing value codes. Click “describe” to the right of the file name in order to add file-specific information. The title and description can be edited in the “Overview” tab, while attributes are defined in the “Attributes” tab. Submit the dataset Post a message to the #datateam Slack channel with a link to your package. "],["creating-a-data-package.html", "Chapter 2 Creating a data package 2.1 What is in a package? 2.2 Packages on the Website 2.3 About identifiers 2.4 Upload a package 2.5 datapack Background 2.6 Exercise 2a 2.7 Create a new data package 2.8 Exercise 2b", " Chapter 2 Creating a data package This chapter will teach you how to create and submit a data package to a DataONE MN via R. But first, please read this paper on the value of structured metadata, namely the Ecological Metadata Language (EML). 2.1 What is in a package? A data package generally consists of at least 3 components. Metadata: One object is the metadata file itself. In case you are unfamiliar with metadata, metadata are information that describe data (e.g. who made the data, how were the data made, etc.). The metadata file will be in an XML format, and have the extension .xml (extensible markup language). We often refer to this file as the EML (Ecological Metadata Language), which is the metadata standard that it uses. Each dataset page in the Arctic Data Center is a direct representation of an EML document, made to look prettier for the web. Data: Other objects in a package are the data files themselves. Most commonly these are data tables (.csv), but they can also be audio files, NetCDF files, plain text files, PDF documents, image files, etc. Resource Map: The final object is the resource map. This object is a plain text file with the extension .rdf (Resource Description Framework) that defines the relationships between all of the other objects in the data package. You can think of it like a “basket” that holds the metadata file and all data files together. It says things like “this metadata file describes this data file,” and is critical to making a data package render correctly on the website. Fortunately, we rarely, if ever, have to actually look at the contents of resource maps; they are generated for us using tools in R. From the DataOne Community Meeting (Session 7) 2.2 Packages on the Website All of the package information is represented when you go to the landing page for a dataset. In the previous section, you uploaded a data file and made edits to the metadata using the web editor. When you make changes to the metadata and data files through R, those published changes will also be reflected here. 2.3 About identifiers Each object (metadata files, data files, resource maps) on the ADC or the KNB (another repo) has a unique identifier, also sometimes called a “PID” (persistent identifier). When you look at the landing page for a dataset, for example here, you can find the resource map identifier listed under the title in the gray bar after the words “Files in this dataset Package:” (resource_map_doi:10.18739/A2836Z), the metadata identifier in the “General &gt; Identifier” section of the metadata record or after the title with blue font (doi:10.18739/A2836Z), and the data identifier by clicking the “more info” link next to the data object, and looking at the “Online Distribution Info” section (arctic-data.9546.1). Note, all datasets submitted are given a preliminary identifier (usually starting with urn:uuid:). When the dataset is finalized, a doi will be issued. Different versions of a package are linked together by what we call the “version chain” or “obsolescence chain”. Making an update to a data package, such as replacing a data file, changing a metadata record, etc, will result in a new identifier for the new version of the updated object. When making changes to a package, always use datapack::uploadDataPackage() for updating the entire package on the latest versions of all objects to ensure that the version chain is maintained. 2.4 Upload a package We will be using R to connect to the NSF Arctic Data Center (ADC) data repository to push and pull edits in actual datasets. To identify yourself as an admin you will need to pass a ‘token’ into R. Do this by signing in to the ADC with your ORCid and password, then hovering over your name in the top right corner and clicking on “My profile”, then navigating to “Settings” and “Authentication Token”, copying the “Token for DataONE R”, and finally pasting and running it in your R console. The console is the bottom left window in RStudio. This token is your identity on these sites, please treat it as you would a password (i.e. don’t paste into scripts that will be shared). The easiest way to do this is to always run the token in the console. There’s no need to keep it in your script since it’s temporary anyway. You will need to retrieve a new one after it either expires or you quit your R session. Setting the token does not produce any output in the console. If the token is not set or is set incorrectly, you will know when an error is produced after trying to load a private dataset. Sometimes you’ll see a placeholder in scripts to remind users to get their token, such as: options(dataone_test_token = &quot;...&quot;) Since we will be working on the test site and not the production site, please remember to get your token from test.arcticdata.io Next, please be sure these packages are loaded for the training (these should already exist if you are working on the server): library(devtools) library(dataone) library(datapack) library(EML) library(remotes) library(XML) library(uuid) If any package could not be loaded, use the following command (replacing package_name with the actual package name) to install the package, then load them. install.packages(&quot;package_name&quot;) Now you’ll install the arcticdatautils and datamgmt packages with the code below. If prompted to update packages during the installation process, skip the updates. Now, run the following code to install and load the libraries. remotes::install_github(&quot;nceas/arcticdatautils&quot;) library(arcticdatautils) remotes::install_github(&quot;nceas/datamgmt&quot;) library(datamgmt) When you are usually working with data packages you will only need the following: library(dataone) library(datapack) library(EML) library(arcticdatautils) For this training, we will be working exclusively on the Arctic test site, or “node.” In many of the functions you will use this will be the first argument. It is often referred to in documentation as mn, short for member node. Different repositories use different member nodes. More information on the other nodes can be found in the reference section under Set DataONE nodes Set DataONE nodes For example, if we are using the test site, set the node to the test Arctic node: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Once all set up you can first publish an object (data) If you are curious how everything magically works, here is a handy diagram: From the DataOne Community Meeting (Session 7) 2.5 datapack Background adapted from the dataone and datapack vignettes datapack is written differently than most R packages you may have encountered in the past. This is because it uses the S4 system instead. library(dataone) library(datapack) library(uuid) Data packages Data packages are a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. 2.5.1 Navigating data packages Nodes Using this example on arcticdata.io d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) To use the member node information, use the mn slot d1c_test@mn To access the various slots using objects created by datapack and dataone (e.g. getSystemMetadata) requires the @ which is different from what you might have seen in the past. This is because these use the S4 system. Get an existing package from the Arctic Data Center. Make sure you know as you go through this training whether you are reading or writing to test or production. We don’t want to upload any of your test datasets to production! d1c &lt;- dataone::D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) dp &lt;- dataone::getDataPackage(d1c, &quot;resource_map_urn:uuid:1f9eee7e-2d03-43c4-ad7f-f300e013ab28&quot;) 2.5.2 Data Objects You can see what slots are in an S4 object after typing the subsetting operator @, or pressing TAB with the cursor after an existing @. Try viewing the slots of the data package. dp@ Check out the objects slot dp@objects The objects slot contains a list of object PIDs that are accessed using the $ subsetting operator. Both are found within the structure of data packages in R. Get the number for data and metadata files associated with this data package: getSize(dp) Get the file names and corresponding pids getValue(dp, name=&quot;sysmeta@fileName&quot;) Get identifiers You can search by any of the sysmeta slots such as fileName and formatId and get the corresponding identifier(s): metadataId &lt;- selectMember(dp, name=&quot;sysmeta@ADD THE NAME OF THE SLOT&quot;, value=&quot;PATTERN TO SEARCH BY&quot;) Example: selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;image/tiff&quot;) selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&quot;filename.csv&quot;) 2.5.3 Provenance View the provenance as a dataTable. We will get into detail in the Building provenance chapter. dp@relations$relations 2.6 Exercise 2a Select a dataset from the catalog on the Arctic Data Center. Observe the number of data files in the dataset. Try to find identifiers for the metadata file and resource map on the landing page for the dataset based on the screenshot shown above. 2.7 Create a new data package adapted from the dataone and datapack vingettes library(dataone) library(datapack) library(uuid) Create a new data package - data package is a class that has slots for relations (provenance), objects(the metadata and data file(s)) and systemMetadata. dp &lt;- new(&quot;DataPackage&quot;) 2.7.1 Upload new data files 2.7.1.1 Create and add a metadata file In this example we will use this previously written EML metadata. Here we are getting the file path from the dataone package and saving that as the object emlFile. This is a bit of an unusual way to reference a local file path, but all this does is looks within the R package dataone and grabs the path to a metadata document stored within that package. If you print the value of emlFile you’ll see it is just a file path, but it points to a special place on the server where that package is installed. Usually you will just reference EML paths that are stored within your user file system. emlFile &lt;- system.file(&quot;extdata/strix-pacific-northwest.xml&quot;, package=&quot;dataone&quot;) Create a new DataObject for the metadata and add it to the package. metadataObj &lt;- new(&quot;DataObject&quot;, format=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;, filename=emlFile) dp &lt;- addMember(dp, metadataObj) Check the dp object to see if the metadata was added correctly. dp 2.7.1.2 Add some additional data files sourceData &lt;- system.file(&quot;extdata/OwlNightj.csv&quot;, package=&quot;dataone&quot;) sourceObj &lt;- new(&quot;DataObject&quot;, format=&quot;text/csv&quot;, filename=sourceData) dp &lt;- addMember(dp, sourceObj, metadataObj) # The third argument of addMember() associates the new DataObject to the metadata that was just added. If you want to change the formatId please use updateSystemMetadata (more on this later in the book) 2.7.2 Upload the package d1c &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) Make sure to give access privileges to the ADC admins: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) Get necessary token from test.arcticdata.io to upload the dataset prior uploading the datapackage: packageId &lt;- uploadDataPackage(d1c, dp, public=TRUE, accessRules=myAccessRules, quiet=FALSE) 2.8 Exercise 2b This exercise will take you through how to do the submission process through R instead of the webform (exercise 1). 2.8.1 Part 1 - Gather your data files For our convenience, we will be grabbing the metadata and data files from the file we published earlier: Locate the data package you published in Exercise 1 by navigating to the “My Profile &gt; My Data” section on test.arcticdata.io. Download the metadata and data files and transfer them to the Datateam server. 2.8.2 Part 2 - Working in R Now we want to publish the metadata and data files we downloaded again to test.arcticdata.io Obtain a token and please note that for this exercise please make sure you grab the token from the arcticdata test site Publish your metadata and data file to the site. #set the node d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) dp &lt;- new(&quot;DataPackage&quot;) #add your metadata metadataObj &lt;- new(...) dp &lt;- addMember(...) #add your data files sourceObj &lt;- new(...) dp &lt;- addMember(...) #upload your package myAccessRules &lt;- data.frame(...) packageId &lt;- uploadDataPackage(...) View your new data set by appending the metadata PID to the end of the URL test.arcticdata.io/view/… If you are successful it should look the same as the dataset you created in exercise 1 Send a message to #datateam with the exercise number and a link to your new package. "],["exploring-eml.html", "Chapter 3 Exploring EML 3.1 Navigate through EML 3.2 Understand the EML schema 3.3 Access specific elements", " Chapter 3 Exploring EML We use the Ecological Metadata Language (EML) to store structured metadata for all datasets submitted to the Arctic Data Center. EML is written in XML (extensible markup language) and functions for building and editing EML are in the EML R package. Currently the Arctic Data Center website supports editing EML version 2.2.0. There are still some metadata in 2.1.1 that will be converted eventually. For additional background on EML and principles for metadata creation, check out this paper. If you aren’t too familiar with lists and how to navigate them yet take a look at the relevant sections in the Stat 545 class. 3.1 Navigate through EML The first task when editing an EML file is navigating the EML file. An EML file is organized in a structure that contains many lists nested within other lists. The function View allows you to get a crude view of an EML file in the viewer. It can be useful for exploring the file. # Need to be in this member node to explore file d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) doc &lt;- read_eml(getObject(d1c_test@mn, &quot;urn:uuid:558eabf1-1e91-4881-8ba3-ef8684d8f6a1&quot;)) View(doc) The complex EML document is represented in R as as series of named, nested lists. We use lists all the time in R! A data.frame is one example of a special kind of list that we use all the time. You may be familiar with the syntax dataframe$column_name which allows us to select a particular column of a data.frame. Under the hood, a data.frame is a named list of vectors with the same length. You select one of those vectors using the $ operator, which is called the “list selector operator.” Just like you navigate in a data.frame, you can use the $ operator to navigate through the EML structure. The $ operator allows you to go deeper into the EML structure and to see what elements are nested within other elements. However, you have to tell R where you want to go in the structure when you use the $ symbol. For example, if you want to view the dataset element of your EML you would use the command doc$dataset. If you want to view the creators of your data set you would use doc$dataset$creator. Note here that creator is contained within dataset. If you aren’t sure where you want to go, hit the tab button on your keyboard after typing $ and a list of available elements in the structure will appear (e.g., doc$&lt;TAB&gt;): Note that if you hit tab, and nothing pops up, this most likely implies that you are trying to go into an EML element that can take a series items. For example doc$dataset$creator$&lt;TAB&gt; will not show a pop-up menu. This is because creator is a series-type object (i.e. you can have multiple creators). If you want to go deeper into creator, you first must tell R which creator you are interested in. Do this by writing [[i]] first where i is the index of the creator you are concerned with. For example, if you want to look at the first creator i = 1. Now doc$dataset$creator[[1]]$&lt;TAB&gt; will give you many more options. Note, an empty autocomplete result sometimes means you have reached the end of a branch in the EML structure. Below is the structure of doc$dataset. There are a series of multiple creators, which can be accessed individually by index: doc$dataset$creator[[#]]. At this point stop and take a deep breath. The key takeaway is that EML is a hierarchical tree structure. The best way to get familiar with it is to explore the structure. Try entering doc$dataset into your console, and print it. Now make the search more specific, for instance: doc$dataset$abstract. 3.2 Understand the EML schema Another great resource for navigating the EML structure is looking at the schema which defines the structure. The schema diagrams on this page are interactive. Further explanations of the symbology can be found here. The schema is complicated and may take some time to get familiar with before you will be able to fully understand it. Use your browser’s “search in page” function (usually CTRL-F or Command-F) to navigate the EML schema page quickly. For example, let’s take a look at eml-party. To start off, notice that some elements have bolded lines leading to them. A bold line indicates that the element is required if the element above it (to the left in the schema) is used, otherwise the element is optional. Notice also that next to the givenName element it says “0..infinity”. This means that the element is unbounded — a single party can have many given names and there is no limit on how many you can add. However, this text does not appear for the surName element — a party can have only one surname. You will also see icons linking the EML slots together, which indicate the ordering of subsequent slots. These can indicate either a “sequence” or a “choice”. In our example from eml-party, a “choice” icon indicates that either an individualName, organizationName, or positionName is required, but you do not need all three. However, the “sequence” icon tells us that if you use an individualName, you must include the surName as a child element. If you include the optional child elements salutation and givenName, they must be written in the order presented in the schema. The eml schema sections you may find particularly helpful include eml-party, eml-attribute and eml-physical. For a more detailed description of the EML schema, see the reference section on exploring EML. 3.2.1 Check your understanding Find otherEntity within the EML schema. Which elements are required? Can otherEntity be a series object? Answer otherEntity requires entityType and entityName children, or alternatively will accept only references. It is a series object, so there can be multiple otherEntities. Along with otherEntity and creator, dataTable and attribute can also be series objects. 3.3 Access specific elements The eml_get() function is a powerful tool for exploring EML (more on that here ). It takes any chunk of EML and returns all instances of the element you specify. Note: you’ll have to specify the element of interest exactly, according to the spelling/capitalization conventions used in EML. Here are some examples: doc &lt;- read_eml(system.file(&quot;example-eml.xml&quot;, package = &quot;arcticdatautils&quot;)) eml_get(doc, &quot;creator&quot;) individualName: givenName: Bryce surName: Mecum organizationName: National Center for Ecological Analysis and Synthesis eml_get(doc, &quot;boundingCoordinates&quot;) eastBoundingCoordinate: &#39;-134&#39; northBoundingCoordinate: &#39;59&#39; southBoundingCoordinate: &#39;57&#39; westBoundingCoordinate: &#39;-135&#39; eml_get(doc, &quot;url&quot;) &#39;&#39;: function: download url: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 &#39;&#39;: ecogrid://knb/urn:uuid:89bec5d0-26db-48ac-ae54-e1b4c999c456 eml_get_simple() is a simplified alternative to eml_get() that produces a list of the desired EML element. eml_get_simple(doc$dataset$otherEntity, &quot;entityName&quot;) To find an eml element you can use either a combination of which_in_emlfrom the arcticdatautils package or eml_get_simple and which to find the index in an EML list. Use which ever workflow you see fit. An example question you may have: Which creators have a surName “Mecum”? Example using which_in_eml: n &lt;- which_in_eml(doc$dataset$creator, &quot;surName&quot;, &quot;Mecum&quot;) # Answer: doc$dataset$creator[[n]] Example using eml_get_simple and which: ent_names &lt;- eml_get_simple(doc$dataset$creator, &quot;surName&quot;) i &lt;- which(ent_names == &quot;Mecum&quot;) # Answer: doc$dataset$creator[[i]] "],["editing-eml.html", "Chapter 4 Editing EML 4.1 Edit an EML element 4.2 FAIR data practices 4.3 Edit attributeLists 4.4 Set physical 4.5 Edit dataTables 4.6 Edit otherEntities 4.7 Semantic annotations 4.8 Exercise 3a 4.9 Validate EML and update package 4.10 Exercise 3b", " Chapter 4 Editing EML This chapter is a practical tutorial for using R to read, edit, write, and validate EML documents. Much of the information here can also be found in the vignettes for the R packages used in this section (e.g. the EML package). Most of the functions you will see in this chapter will use the arcticdatautils and EML packages. This chapter will be longest of all the sections! This is a reminder to take frequent breaks when completing this section. If you struggle with getting a piece of code to work more than 10 minutes, reach out to your supervisor for help. When using R to edit EML documents, run each line individually by highlighting the line and using CTRL+ENTER). Many EML functions only need to be ran once, and will either produce errors or make the EML invalid if run multiple times. 4.1 Edit an EML element There are multiple ways to edit an EML element. 4.1.1 Edit EML with strings The most basic way to edit an EML element would be to navigate to the element and replace it with something else. Easy! For example, to change the title one could use the following command: doc$dataset$title &lt;- &quot;New Title&quot; If the element you are editing allows for multiple values, you can pass it a list of character strings. Since a dataset can have multiple titles, we can do this: doc$dataset$title &lt;- list(&quot;New Title&quot;, &quot;Second New Title&quot;) However, this isn’t always the best method to edit the EML, particularly if the element has sub-elements. Adding directly to doc without a helper function can overwrite these parts of the doc that we need. 4.1.2 Edit EML with the “EML” package To edit a section where you are not 100% sure of the sub-elements, using the eml$elementName() helper functions from the EML package will pre-populate the options for you if you utilize the RStudio autocomplete functionality. The arguments in these functions show the available slots for any given EML element. For example, typing doc$dataset$abstract &lt;- eml$abstract()&lt;TAB&gt; will show you that the abstract element can take either the section or para sub-elements. doc$dataset$abstract &lt;- eml$abstract(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) This inserts the abstract with a para element in our dataset, which we know from the EML schema is valid. Note that the above is equivalent to the following generic construction: doc$dataset$abstract &lt;- list(para = &quot;A concise but thorough description of the who, what, where, when, why, and how of a dataset.&quot;) The eml() family of functions provides the sub-elements as arguments, which is extremely helpful, but functionally all it is doing is creating a named list, which you can also do using the list function. 4.1.3 Edit EML with objects A final way to edit an EML element would be to build a new object to replace the old object. To begin, you might create an object using an eml helper function. Let’s take keywords as an example. Sometimes keyword lists in a metadata record will come from different thesauruses, which you can then add in series (similar to the way we added multiple titles) to the element keywordSet. We start by creating our first set of keywords and saving it to an object. kw_list_1 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER controlled vocabulary&quot;, keyword = list(&quot;bacteria&quot;, &quot;carnivorous plants&quot;, &quot;genetics&quot;, &quot;thresholds&quot;)) Which returns: $keyword $keyword[[1]] [1] &quot;bacteria&quot; $keyword[[2]] [1] &quot;carnivorous plants&quot; $keyword[[3]] [1] &quot;genetics&quot; $keyword[[4]] [1] &quot;thresholds&quot; $keywordThesaurus [1] &quot;LTER controlled vocabulary&quot; We create the second keyword list similarly: kw_list_2 &lt;- eml$keywordSet(keywordThesaurus = &quot;LTER core area&quot;, keyword = list(&quot;populations&quot;, &quot;inorganic nutrients&quot;, &quot;disturbance&quot;)) Finally, we can insert our two keyword lists into our EML document just like we did with the title example above, but rather than passing character strings into list(), we will pass our two keyword set objects. doc$dataset$keywordSet &lt;- list(kw_list_1, kw_list_2) Note that you must use the function list here and not the c() function. The reasons for this are complex, and due to some technical subtlety in R - but the gist of the issue is that the c() function can behave in unexpected ways with nested lists, and frequently will collapse the nesting into a single level, resulting in invalid EML. 4.2 FAIR data practices The result of these function calls won’t show up on the webpage but they will add a publisher element to the dataset element and a system to all of the entities based on what the PID is. This will help make our metadata more FAIR (Findable, Accessible, Interoperable, Reusable). These two functions come from the arcticatautils package, an R package we wrote to help with some very specific data processing tasks. Add these function calls to all of your EML processing scripts. library(arcticdatautils) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) 4.3 Edit attributeLists Attributes are descriptions of variables, typically columns or column names in tabular data. Attributes are stored in an attributeList. When editing attributes in R, we convert the attribute list information to data frame (table) format so that it is easier to edit. When editing attributes you will need to create one to three data frame objects: A data.frame of attributes A data.frame of custom units (if applicable) A data.frame of factors (if applicable) The attributeList is an element within one of 4 different types of entity objects. An entity corresponds to a file, typically. Multiple entities (files) can exist within a dataset. The 4 different entity types are dataTable (most common for us), spatialVector, spatialRaster, and otherEntity. Please note that submitting attribute information through the website will store them in an otherEntity object by default. We prefer to store them in a dataTable object for tabular data or a spatialVector or spatialRaster object for spatial data. To edit or examine an existing attribute list already in an EML file, you can use the following commands, where i represents the index of the series element you are interested in. Note that if there is only one item in the series (ie there is only one dataTable), you should just call doc$dataset$dataTable, as in this case doc$dataset$dataTable[[1]] will return the first sub-element of the dataTable (the entityName) # If they are stored in an otherEntity (submitted from the website by default) attribute_tables &lt;- EML::get_attributes(doc$dataset$otherEntity[[i]]$attributeList) # Or if they are stored in a dataTable (usually created by a datateam member) attribute_tables &lt;- EML::get_attributes(doc$dataset$dataTable[[i]]$attributeList) The get_attributes() function returns the attribute_tables object, which is a list of the three data frames mentioned above. The data frame with the attributes is called attribute_tables$attributes. attribute_tables$attributes print(attribute_tables$attributes) 4.3.1 Edit attributes Attribute information should be stored in a data.frame with the following columns: attributeName: The name of the attribute as listed in the csv. Required. e.g.: “c_temp” attributeLabel: A descriptive label that can be used to display the name of an attribute. It is not constrained by system limitations on length or special characters. Optional. e.g.: “Temperature (Celsius)” attributeDefinition: Longer description of the attribute, including the required context for interpreting the attributeName. Required. e.g.: “The near shore water temperature in the upper inter-tidal zone, measured in degrees Celsius.” measurementScale: One of: nominal, ordinal, dateTime, ratio, interval. Required. nominal: unordered categories or text. e.g.: (Male, Female) or (Yukon River, Kuskokwim River) ordinal: ordered categories. e.g.: Low, Medium, High dateTime: date or time values from the Gregorian calendar. e.g.: 01-01-2001 ratio: measurement scale with a meaningful zero point in nature. Ratios are proportional to the measured variable. e.g.: 0 Kelvin represents a complete absence of heat. 200 Kelvin is half as hot as 400 Kelvin. 1.2 meters per second is twice as fast as 0.6 meters per second. interval: values from a scale with equidistant points, where the zero point is arbitrary. This is usually reserved for degrees Celsius or Fahrenheit, or any other human-constructed scale. e.g.: there is still heat at 0° Celsius; 12° Celsius is NOT half as hot as 24° Celsius. domain: One of: textDomain, enumeratedDomain, numericDomain, dateTime. Required. textDomain: text that is free-form, or matches a pattern enumeratedDomain: text that belongs to a defined list of codes and definitions. e.g.: CASC = Cascade Lake, HEAR = Heart Lake dateTimeDomain: dateTime attributes numericDomain: attributes that are numbers (either ratio or interval) formatString: Required for dateTime, NA otherwise. Format string for dates, e.g. “DD/MM/YYYY”. definition: Required for textDomain, NA otherwise. Defines a format for attributes that are a character string. e.g.: “Any text” or “7-digit alphanumeric code” unit: Required for numericDomain, NA otherwise. Unit string. If the unit is not a standard unit, a warning will appear when you create the attribute list, saying that it has been forced into a custom unit. Use caution here to make sure the unit really needs to be a custom unit. A list of standard units can be found using: standardUnits &lt;- EML::get_unitList() then running View(standardUnits$units). numberType: Required for numericDomain, NA otherwise. Options are real, natural, whole, and integer. real: positive and negative fractions and integers (…-1,-0.25,0,0.25,1…) natural: non-zero positive integers (1,2,3…) whole: positive integers and zero (0,1,2,3…) integer: positive and negative integers and zero (…-2,-1,0,1,2…) missingValueCode: Code for missing values (e.g.: ‘-999’, ‘NA’, ‘NaN’). NA otherwise. Note that an NA missing value code should be a string, ‘NA’, and numbers should also be strings, ‘-999.’ missingValueCodeExplanation: Explanation for missing values, NA if no missing value code exists. You can create attributes manually by typing them out in R following a workflow similar to the one below: attributes &lt;- data.frame( attributeName = c(&#39;Date&#39;, &#39;Location&#39;, &#39;Region&#39;,&#39;Sample_No&#39;, &#39;Sample_vol&#39;, &#39;Salinity&#39;, &#39;Temperature&#39;, &#39;sampling_comments&#39;), attributeDefinition = c(&#39;Date sample was taken on&#39;, &#39;Location code representing location where sample was taken&#39;,&#39;Region where sample was taken&#39;, &#39;Sample number&#39;, &#39;Sample volume&#39;, &#39;Salinity of sample in PSU&#39;, &#39;Temperature of sample&#39;, &#39;comments about sampling process&#39;), measurementScale = c(&#39;dateTime&#39;, &#39;nominal&#39;,&#39;nominal&#39;, &#39;nominal&#39;, &#39;ratio&#39;, &#39;ratio&#39;, &#39;interval&#39;, &#39;nominal&#39;), domain = c(&#39;dateTimeDomain&#39;, &#39;enumeratedDomain&#39;,&#39;enumeratedDomain&#39;, &#39;textDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;numericDomain&#39;, &#39;textDomain&#39;), formatString = c(&#39;MM-DD-YYYY&#39;, NA,NA,NA,NA,NA,NA,NA), definition = c(NA,NA,NA,&#39;Six-digit code&#39;, NA, NA, NA, &#39;Any text&#39;), unit = c(NA, NA, NA, NA,&#39;milliliter&#39;, &#39;dimensionless&#39;, &#39;celsius&#39;, NA), numberType = c(NA, NA, NA,NA, &#39;real&#39;, &#39;real&#39;, &#39;real&#39;, NA), missingValueCode = c(NA, NA, NA,NA, NA, NA, NA, &#39;NA&#39;), missingValueCodeExplanation = c(NA, NA, NA,NA, NA, NA, NA, &#39;no sampling comments&#39;)) However, typing this out in R can be a major pain. Luckily, there’s a Shiny app that you can use to build attribute information. You can use the app to build attributes from a data file loaded into R (recommended as the app will auto-fill some fields for you) to edit an existing attribute table, or to create attributes from scratch. Use the following commands to create or modify attributes. Use the following commands to create or modify attributes. These commands will launch a “Shiny” app in your web browser. #first download the CSV in your data package from Exercise #2 data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;.csv&quot;) data &lt;- read.csv(text=rawToChar(getObject(d1c_test@mn, data_pid))) # From data (recommended) attribute_tables &lt;- EML::shiny_attributes(data = data) # From scratch attribute_tables &lt;- EML::shiny_attributes() # From an existing attribute list attribute_tables &lt;- get_attributes(doc$dataset$dataTable[[i]]$attributeList) attribute_tables &lt;- EML::shiny_attributes(attributes = attribute_tables$attributes) Once you are done editing a table in the browser app, quit the app by pressing the red “Quit App” button in the top right corner of the page. If you close the Shiny app tab in your browser instead of using the “Quit App” button, your work will not be saved, R will think that the Shiny app is still open, and you will not be able to run other code. You can tell if R is confused if you have closed the Shiny app and the bottom line in the console still says Listening on http://.... If this happens, press the red stop sign button on the right hand side of the console window in order to interrupt R. The tables you constructed in the app will be assigned to the attribute_tables variable as a list of data frames (one for attributes, factors, and units). Be careful to not overwrite your completed attribute_tables object when trying to make edits. The last line of code can be used in order to make edits to an existing attribute_tables object. Alternatively, each table can be to exported to a csv file by clicking the Download button. If you downloaded the table, read the table back into your R session and assign it to a variable in your script (e.g. attributes &lt;- data.frame(...)), or just use the variable that shiny_attributes returned. For simple attribute corrections, datamgmt::edit_attribute() allows you to edit the slots of a single attribute within an attribute list. To use this function, pass an attribute through datamgmt::edit_attribute() and fill out the parameters you wish to edit/update. An example is provided below where we are changing attributeName, domain, and measurementScale in the first attribute of a dataset. After completing the edits, insert the new version of the attribute back into the EML document. new_attribute &lt;- datamgmt::edit_attribute(doc$dataset$dataTable[[1]]$attributeList$attribute[[1]], attributeName = &#39;date_and_time&#39;, domain = &#39;dateTimeDomain&#39;, measurementScale = &#39;dateTime&#39;) doc$dataset$dataTable[[1]]$attributeList$attribute[[1]] &lt;- new_attribute 4.3.2 Edit custom units EML has a set list of units that can be added to an EML file. These can be seen by using the following code: standardUnits &lt;- EML::get_unitList() View(standardUnits$units) Search the units list for your unit before attempting to create a custom unit. You can search part of the unit you can look up part of the unit ie meters in the table to see if there are any matches. If you have units that are not in the standard EML unit list, you will need to build a custom unit list. Attribute tables with custom units listed will generate a warning indicating that a custom unit will need to be described. A unit typically consists of the following fields: id: The unit id (ids are camelCased) unitType: The unitType (run View(standardUnits$unitTypes) to see standard unitTypes) parentSI: The parentSI unit (e.g. for kilometer parentSI = “meter”). The parentSI does not need to be part of the unitList. multiplierToSI: Multiplier to the parentSI unit (e.g. for kilometer multiplierToSI = 1000) name: Unit abbreviation (e.g. for kilometer name = “km”) description: Text defining the unit (e.g. for kilometer description = “1000 meters”) To manually generate the custom units list, create a dataframe with the fields mentioned above. An example is provided below that can be used as a template: custom_units &lt;- data.frame( id = c(&#39;siemensPerMeter&#39;, &#39;decibar&#39;), unitType = c(&#39;resistivity&#39;, &#39;pressure&#39;), parentSI = c(&#39;ohmMeter&#39;, &#39;pascal&#39;), multiplierToSI = c(&#39;1&#39;,&#39;10000&#39;), abbreviation = c(&#39;S/m&#39;,&#39;decibar&#39;), description = c(&#39;siemens per meter&#39;, &#39;decibar&#39;)) Custom units can also be created in the shiny app, under the “units” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() custom_units &lt;- attribute_tables$units Using EML::get_unit_id for custom units will also generate valid EML unit ids. Custom units are then added to additionalMetadata using the following command: unitlist &lt;- set_unitList(custom_units, as_metadata = TRUE) doc$additionalMetadata &lt;- list(metadata = list(unitList = unitlist)) 4.3.3 Edit factors For attributes that are enumeratedDomains, a table is needed with three columns: attributeName, code, and definition. attributeName should be the same as the attributeName within the attribute table and repeated for all codes belonging to a common attribute. code should contain all unique values of the given attributeName that exist within the actual data. definition should contain a plain text definition that describes each code. To build factors by hand, you use the named character vectors and then convert them to a data.frame as shown in the example below. In this example, there are two enumerated domains in the attribute list - “Location” and “Region”. Location &lt;- c(CASC = &#39;Cascade Lake&#39;, CHIK = &#39;Chikumunik Lake&#39;, HEAR = &#39;Heart Lake&#39;, NISH = &#39;Nishlik Lake&#39; ) Region &lt;- c(W_MTN = &#39;West region, locations West of Eagle Mountain&#39;, E_MTN = &#39;East region, locations East of Eagle Mountain&#39;) The definitions are then written into a data.frame using the names of the named character vectors and their definitions. factors &lt;- rbind(data.frame(attributeName = &#39;Location&#39;, code = names(Location), definition = unname(Location)), data.frame(attributeName = &#39;Region&#39;, code = names(Region), definition = unname(Region))) Factors can also be created in the shiny app, under the “factors” tab. They cannot be edited again in the shiny app once created. attribute_tables &lt;- EML::shiny_attributes() attribute_tables$factors 4.3.4 Finalize attributeList Once you have built your attributes, factors, and custom units, you can add them to EML objects. Attributes and factors are combined to form an attributeList using set_attributes(): # Create an attributeList object attributeList &lt;- EML::set_attributes(attributes = attribute_tables$attributes, factors = attribute_tables$factors) This attributeList object can then be checked for errors and added to a dataTable in the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList Remember to use: d1c_test &lt;- dataone::D1Client(“STAGING”, “urn:node:mnTestARCTIC”) d1c_test@mn 4.4 Set physical To set the physical aspects of a data object, use the following commands to build a physical object from a data PID that exists in your package. Remember to set the member node to test.arcticdata.io! Every entity that we upload needs a physical description added. When replacing files, the physical must be replaced as well. The word ‘physical’ derives from database systems, which distinguish the ‘logical’ model (e.g., what attributes are in a table, etc) from the physical model (how the data are written to a physical hard disk (basically, the serialization)). so, we grouped metadata about the file (eg. dataformat, file size, file name) as written to disk in physical. It includes info like the file size. For CSV files, the physical describes the number of header lines and the attribute orientation. # Get the PID of a file data_pid &lt;- selectMember(dp, name = &quot;sysmeta@fileName&quot;, value = &quot;your_file_name.csv&quot;) # Get the physical info and store it in an object physical &lt;- arcticdatautils::pid_to_eml_physical(mn, data_pid) The physical object can then be checked for errors and added to the EML document. # Edit EML document with object doc$dataset$dataTable[[i]]$physical &lt;- physical Note that the above workflow only works if your data object already exists on the member node. Physicals can be seen in the website representation of the EML below the entity description. 4.5 Edit dataTables Entities that are dataTables require an attribute list. To edit a dataTable, first edit/create an attributeList and set the physical. Then create a new dataTable using the eml$dataTable() helper function as below: dataTable &lt;- eml$dataTable(entityName = &quot;A descriptive name for the data (does not need to be the same as the data file)&quot;, entityDescription = &quot;A description of the data&quot;, physical = physical, attributeList = attributeList) The dataTable must then be added to the EML. How exactly you do this will depend on whether there are dataTable elements in your EML, and how many there are. To replace whatever dataTable elements already exist, you could write: doc$dataset$dataTable &lt;- dataTable If there is only one dataTable in your dataset, the EML package will usually “unpack” these, so that it is not contained within a list of length 1 - this means that to add a second dataTable, you cannot use the syntax doc$dataset$dataTable[[2]], since when unpacked this will contain the entityDescription as opposed to pointing to the second in a series of dataTable elements. Confusing - I know. Not to fear though - this syntax will get you on your way, should you be trying to add a second dataTable. doc$dataset$dataTable &lt;- list(doc$dataset$dataTable, dataTable) If there is more than one dataTable in your dataset, you can return to the more straightforward construction of: doc$dataset$dataTable[[i]] &lt;- dataTable Where i is the index that you wish insert your dataTable into. To add a list of dataTables to avoid the unpacking problem above you will need to create a list of dataTables dts &lt;- list() # create an empty list for(i in seq_along(tables_you_need)){ # your code modifying/creating the dataTable here dataTable &lt;- eml$dataTable(entityName = dataTable$entityName, entityDescription = dataTable$entityDescription, physical = physical, attributeList = attributeList) dts[[i]] &lt;- dataTable # add to the list } After getting a list of dataTables, assign the resulting list to dataTable EML doc$dataset$dataTable &lt;- dts By default, the online submission form adds all entities as otherEntity, even when most should probably be dataTable. You can use eml_otherEntity_to_dataTable to easily move items in otherEntity over to dataTable, and delete the old otherEntity. Most tabular data or data that contain variables should be listed as a dataTable. Data that do not contain variables (eg: plain text readme files, pdfs, jpegs) should be listed as otherEntity. eml_otherEntity_to_dataTable(doc, 1, # Indexes of otherEntities you want to convert, for multiple use 1:5 or c(1,3,5..) validate_eml = F) # set this to False if the physical or attributes are not added 4.6 Edit otherEntities 4.6.1 Remove otherEntities To remove an otherEntity use the following command. This may be useful if a data object is originally listed as an otherEntity and then transferred to a dataTable. doc$dataset$otherEntity[[i]] &lt;- NULL 4.6.2 Create otherEntities If you need to create/update an otherEntity, make sure to publish or update your data object first (if it is not already on the DataONE MN). Then build your otherEntity. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, pkg$data[[i]]) Alternatively, you can build the otherEntity of a data object not in your package by simply inputting the data PID. otherEntity &lt;- arcticdatautils::pid_to_eml_entity(mn, &quot;your_data_pid&quot;, entityType = &quot;otherEntity&quot;, entityName = &quot;Entity Name&quot;, entityDescription = &quot;Description about entity&quot;) The otherEntity must then be set to the EML, like so: doc$dataset$otherEntity &lt;- otherEntity If you have more than one otherEntity object in the EML already, you can add the new one like this: doc$dataset$otherEntity[[i]] &lt;- otherEntity Where i is set to the number of existing entities plus one. Remember the warning from the last section, however. If you only have one otherEntity, and you are trying to add another, you have to run: doc$dataset$otherEntity &lt;- list(otherEntity, doc$dataset$otherEntity) 4.7 Semantic annotations For a brief overview of what a semantic annotation is, and why we use them check out this video. Even more information on how to add semantic annotations to EML 2.2.0 can be found here. There are several elements in the EML 2.2.0 schema that can be annotated: dataset entity (eg: otherEntity or dataTable) attribute Attribute annotations can be edited in R and also on the website. Dataset and entity annotations are only done in R. 4.7.1 How annotations are used This is a dataset that has semantic annotations included. On the website you can see annotations in each of the attributes. You can click on any one of them to search for more datasets with that same annotation. Semantic attribute annotations can be applied to spatialRasters, spatialVectors and dataTables 4.7.1.1 Attribute-level annotations on the website editor The website has a searchable list of attribute annotations that are grouped by category and specificity. Open your dataset on the test website from earlier and enter the attribute editor. Look through all of the available annotations. Adding attribute annotations using the website is the easiest way, however adding them using R and/or the Shiny app may be quicker with very larger datasets. 4.7.1.2 Attribute-level annotations in R To manually add annotations to the attributeList in R you will need information about the propertyURI and valueURI. Annotations are essentially composed of a sentence, which contains a subject (the attribute), predicate (propertyURI), and object (valueURI). Because of the way our search interface is built, for now we will be using attribute annotations that have a propertyURI label of “contains measurements of type”. Here is what an annotation for an attribute looks like in R. Note that both the propertyURI and valueURI have both a label, and the URI itself. doc$dataset$dataTable[[i]]$attributeList$attribute[[i]]$annotation $id [1] &quot;ODBcOyaTsg&quot; $propertyURI $propertyURI$label [1] &quot;contains measurements of type&quot; $propertyURI$propertyURI [1] &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot; $valueURI $valueURI$label [1] &quot;Distributed Biological Observatory region identifier&quot; $valueURI$valueURI [1] &quot;http://purl.dataone.org/odo/ECSO_00002617&quot; 4.7.2 How to add an annotation 1. Decide which variable to annotate The goal for the datateam is to start annotating every dataset that comes in. Please make sure to add semantic annotations to spatial and temporal features such as latitude, longitude, site name and date and aim to annotate as many attributes as possible. 2. Find an appropriate valueURI The next step is to find an appropriate value to fill in the blank of the sentence: “this attribute contains measurements of _____.” There are several ontologies to search in. In order of most to least likely to be relevant to the Arctic Data Center they are: The Ecosystem Ontology (ECSO) this was developed at NCEAS, and has many terms that are relevant to ecosystem processes, especially those involving carbon and nutrient cycling The Environment Ontology (EnVO) this is an ontology for the concise, controlled description of environments National Center for Biotechnology Information (NCBI) Organismal Classification (NCBITAXON) The NCBI Taxonomy Database is a curated classification and nomenclature for all of the organisms in the public sequence databases. Information Artifact Ontology (IAO) this ontology contains terms related to information entities (eg: journals, articles, datasets, identifiers) To search, navigate through the “classes” until you find an appropriate term. When we are picking terms, it is important that we not just pick a similar term or a term that seems close - we want a term that is 100% accurate. For example, if you have an attribute for carbon tetroxide flux and an ontology with a class hierarchy like this: – carbon flux |—- carbon dioxide flux Our exact attribute, carbon tetroxide flux, is not listed. In this case, we should pick “carbon flux” as it’s completely correct/accurate and not “carbon dioxide flux” because it’s more specific/precise but not quite right. For general attributes (such as ones named depth or length), it is important to be as specific as possible about what is being measured. e.g. selecting the lake area annotation for the area attribute in this dataset 3. Build the annotation in R 4.7.2.1 Manually Annotating this method is great for when you are inserting 1 annotation, fixing an existing annotation or programmatically updating annotations for multiple attributeLists First you need to figure out the index of the attribute you want to annotate. eml_get_simple(doc$dataset$dataTable[[3]]$attributeList, &quot;attributeName&quot;) [1] &quot;prdM&quot; &quot;t090C&quot; &quot;t190C&quot; &quot;c0mS/cm&quot; &quot;c1mS/cm&quot; &quot;sal00&quot; &quot;sal11&quot; &quot;sbeox0V&quot; &quot;flECO-AFL&quot; [10] &quot;CStarTr0&quot; &quot;cpar&quot; &quot;v0&quot; &quot;v4&quot; &quot;v6&quot; &quot;v7&quot; &quot;svCM&quot; &quot;altM&quot; &quot;depSM&quot; [19] &quot;scan&quot; &quot;sbeox0ML/L&quot; &quot;sbeox0dOV/dT&quot; &quot;flag&quot; Next, assign an id to the attribute. It should be unique within the document, and it’s nice if it is human readable and related to the attribute it is describing. One format you could use is entity_x_attribute_y which should be unique in scope, and is nice and descriptive. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$id &lt;- &quot;entity_ctd_attribute_salinity&quot; Now, assign the propertyURI information. This will be the same for every annotation you build. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$propertyURI &lt;- list(label = &quot;contains measurements of type&quot;, propertyURI = &quot;http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#containsMeasurementsOfType&quot;) Finally, add the valueURI information from your search. You should see an ID on the Bioportal page that looks like a URL - this is the valueURI. Use the value to populate the label element. doc$dataset$dataTable[[3]]$attributeList$attribute[[6]]$annotation$valueURI &lt;- list(label = &quot;Water Salinity&quot;, valueURI = &quot;http://purl.dataone.org/odo/ECSO_00001164&quot;) 4.7.2.2 Shiny Attributes this method is great for when you are updating many attributes On the far right of the table of shiny_attributes there are 4 columns: id, propertyURI, propertyLabel, valueURI, valueLabel that can be filled out. 4.7.3 Dataset Annotations Dataset annotations can only be made using R. There are several helper functions that assist with making dataset annotations. 4.7.3.1 Data Sensitivity Sensitive datasets that might cover protected characteristics (human subjects data, endangered species locations, etc) should be annotated using the data sensitivity ontology: https://bioportal.bioontology.org/ontologies/SENSO/?p=classes&amp;conceptid=root. 4.7.3.2 Dataset Discipline As a final step in the data processing pipeline, we will categorize the dataset. We are trying to categorize datasets so we can have a general idea of what kinds of data we have at the Arctic Data Center. Datasets will be categorized using the Academic Ontology. These annotations will be seen at the top of the landing page, and can be thought of as “themes” for the dataset. In reality, they are dataset-level annotations. Be sure to ask your peers in the #datateam slack channel whether they agree with the themes you think best fit your dataset. Once there is consensus, use the following line of code: doc &lt;- arcticdatautils::eml_categorize_dataset(doc, c(&quot;Soil Science&quot;, &quot;Plant Science&quot;, &quot;Ecology&quot;)) Be careful not to duplicate dataset annotations. The above code does not existing prior dataset annotations. Duplicate annotations can be removed by setting them to NULL. doc$dataset$annotation[[i]] &lt;- NULL 4.8 Exercise 3a The metadata for the dataset created earlier in Exercise 2 was not very complete. Here we will add a attribute and physical to our entity (the csv file). Make sure your package from before is loaded into R. Convert otherEntity into dataTable. Replace the existing dataTable with a new dataTable object with an attributelist you write in R using the above commands. We will continue using the objects created and updated in this exercise in 3b. Below is some pseudo-code for how to accomplish the above steps. Fill in the dots according to the above sections to complete the exercise. # get the latest version of the resource map identifier from your dataset on the arctic data center resource_map_pid &lt;- ... dp &lt;- getDataPackage(d1c_test, identifier=resource_map_pid, lazyLoad=TRUE, quiet=FALSE) # get metadata pid metadataId &lt;- selectMember(...) # read in EML doc &lt;- read_eml(getObject(...)) # convert otherEntity to dataTable doc &lt;- eml_otherEntity_to_dataTable(...) # write an attribute list using shiny_attributes based on the data in your file ex_data &lt;- read.csv(...) atts &lt;- shiny_attributes(data = ex_data) # set the attributeList doc$dataset$dataTable$attributeList &lt;- set_attributes(...) 4.9 Validate EML and update package To make sure that your edited EML is valid against the EML schema, run eml_validate() on your EML. Fix any errors that you see. eml_validate(doc) You should see something like if everything passes: &gt;[1] TRUE &gt;attr(,“errors”) &gt;character(0) When troubleshooting EML errors, it is helpful to run eml_validate() after every edit to the EML document in order to pinpoint the problematic code. Then save your EML to a path of your choice or a temp file. You will later pass this path as an argument to update the package. # Create a standardized EML name from the dataset title eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) 4.10 Exercise 3b Make sure you have everything from before in R. After adding more metadata, we want to publish the dataset onto test.arcticdata.io. Before we publish updates we need to do a couple checks before doing so. Validate your metadata using eml_validate. Use the checklist to review your submission. Make edits where necessary (e.g. physicals) Once eml_validate returns TRUE go ahead and run write_eml, replaceMember, and uploadDataPackage. There might be a small lag for your changes to appear on the website. This part of the workflow will look roughly like this: # validate and write the EML eml_validate(...) write_eml(...) # replace the old metadata file with the new one in the local package dp &lt;- replaceMember(dp, metadataId, replacement = eml_path) # upload the data package packageId &lt;- uploadDataPackage(...) "],["updating-a-data-package.html", "Chapter 5 Updating a data package 5.1 Update an object 5.2 Update a package with a new data object 5.3 Exercise 4", " Chapter 5 Updating a data package This chapter will teach you how to edit and update an existing data package in R. Earlier, we updated metadata. In this section we will learn how to update a data file, and how to update a package by adding an additional data file. 5.1 Update an object To update a data file associated with a data package, you need to do three things: update the object itself, update the resource map (which affiliates the object with the metadata), and update the metadata that describes that object The datapack::replaceMember function takes care of the first two of these tasks. First you need to get the pid of the file you want to replace by using datapack::selectMember # Select the EML PID dataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;text/csv&quot;) Then use replaceMember: # Replace the old file with the new dp &lt;- replaceMember(dp, dataId, replacement=file_path) If you want to remove some files from the data package we can use datapack::removeMember. If we wanted to remove all the zip files associated with this data package, we can use datapack::removeMember: zipId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/vnd.shp+zip&quot;) removeMember(dp, zipId, removeRelationships = T) You will need to be explicit about your format_id here based on the file type. A list of format IDs can be found here on the DataONE website. Use line 2 (Id:) exactly, character for character. To accomplish the second task, you will need to update the metadata using the EML package. This is covered in Chapter 4. After you update a file, you will always need to update the metadata because parts of the physical section (such as the file size, checksum) will be different, and it may also require different attribute information. Once you have updated your metadata and saved it, you can update the package itself. 5.2 Update a package with a new data object Once you have updated the data objects and saved the metadata to a file, we can update the metadata and use replaceMember to update the package with the new metadata. Make sure you have the package you want to update loaded into R using dataone::getDataPackage(). 5.2.1 Publish update Now we can update your data package to include the new data object. Assuming you have updated your data package earlier something like the below: d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) packageId &lt;- &quot;resource_map_urn:uuid...&quot; dp &lt;- getDataPackage(d1c_test, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #some modification to the EML here eml_path &lt;- arcticdatautils::title_to_file_name(doc$dataset$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) You can then upload your data package: myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c_test, dp, public=FALSE, accessRules=myAccessRules, quiet=FALSE) If a package is ready to be public, you can change the public argument in the datapack::uploadDataPackage() call to TRUE. If you want to publish with a DOI (Digital Object Identifier) instead of a UUID (Universally Unique Identifier), you need to do this when replacing the metadata using the optional newId argument in replaceMember(). This should only be done after the package is finalized and has been thoroughly reviewed! doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If there is a pre-issued DOI (researcher requested the DOI for the publication first), please do the following: dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=&quot;doi:10.../...&quot;) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) If the package has children, see how to do this using arcticdatautils::publish_update in the nesting section of the reference manual. Refresh the landing page at test.arcticdata.io/#view/… for this package and then follow the “newer version” link to view the latest. 5.3 Exercise 4 What if the researcher notices that some information needed to be updated in the data file? We can use replaceMember to do just that! If you haven’t already: Locate the data package you published in the previous exercise by navigating to the URL test.arcticdata.io/#view/… Load the package and EML into R using the above commands. Make a slightly different data file to upload to test.arcticdata.io for this exercise: Load the data file associated with the package into R as a data.frame. (Hint: use read.csv() to upload the data file from your computer/the server.) Make an edit to the data in R (e.g. change one of the colnames to \"TEST\"). Save the edited data. (Hint: use write.csv(data, row.names = FALSE).) Upload the new csv file, get a new pid and publish those updates: Update the data file in the package with the edited data file using replaceMember. Update your package using uploadDataPackage. "],["editing-system-metadata.html", "Chapter 6 Editing system metadata 6.1 Edit sysmeta 6.2 Exercise 5", " Chapter 6 Editing system metadata Every object on the ADC (or the KNB) has “system metadata”. An object’s system metadata have information about the file itself, such as the name of the file (fileName), the format (formatId), who the rightsHolder is, what the accessPolicy is, and more. Sometimes we will need to edit system metadata in order to make sure that things on the webpage display correctly, or to ensure a file downloads from the website with the correct file name and extension. Although the majority of system metadata changes that need to be made are done automatically, sometimes we need to change aspects of the system metadata (or ‘sysmeta’ for short) manually. 6.1 Edit sysmeta To edit the sysmeta of an object (data file, EML, or resource map, etc.) with a PID, first load the sysmeta into R using the following command: sysmeta &lt;- getSystemMetadata(d1c_test@mn, pid) Then edit the sysmeta slots by using @ functionality. For example, to change the fileName use the following command: sysmeta@fileName &lt;- &#39;NewFileName.csv&#39; Note that some slots cannot be changed by simple text replace (particularly the accessPolicy). There are various helper functions for changing the accessPolicy and rightsHolder such as datapack::addAccessRule() (which takes the sysmeta as an input) or arcticdatautils::set_rights_and_access(), which only requires a PID. In general, you most frequently need to use dataone::getSystemMetadata() to change either the formatId or fileName slots (see the DataONE list of format ids) for acceptable formats. # Example of setting the formatId slot sysmeta@formatId &lt;- &quot;eml://ecoinformatics.org/eml-2.1.1&quot; After you have changed the necessary slot, you can update the system metadata using the following command: updateSystemMetadata(mn, pid, sysmeta) 6.1.1 Identifiers and sysmeta Importantly, changing the system metadata does NOT necessitate a change in the PID of an object. This is because changes to the system metadata do not change the object itself, they are only changing the description of the object (although ideally the system metadata are accurate when an object is first published). 6.1.2 Additional resources For a more in-depth (and technical) guide to sysmeta, check out the DataONE documentation: System Metadata Data Types in CICore 6.2 Exercise 5 Sometimes the system doesn’t recognize the file types properly. For example you have a csv file but the File type on the website says Microsoft Excel Read the system metadata in from the data file you uploaded previously. Check to make sure the fileName and formatId are set correctly (the extension in fileName should match the formatId). Update the system metadata if necessary. CSVs have the formatId “text/csv”. "],["building-provenance.html", "Chapter 7 Building provenance 7.1 Using the prov editor 7.2 Understanding resource maps 7.3 Using the datapack package 7.4 References", " Chapter 7 Building provenance Note - It is rare for a dataset to have provenance - though we would like to see that change by encouraging researchers to submit scripts whenever it is reasonable. When processing datasets if you notice that provenance is needed let Daphne or Jeanette know. The provenance chain describes the origin and processing history of data. Provenance (or “prov”) can exist on a continuum, ranging from prose descriptions of the history, to formal provenance traces, to fully executable environments. In this section we will describe how to build provenance using formal provenance traces in DataONE. Provenance is becoming increasingly important in the face of what is being called a reproducibility crisis in science. J. P. A. Ioannidis (2005) wrote that “Most Research Findings Are False for Most Research Designs and for Most Fields”. Ioannidis outlined ways in which the research process has lead to inflated effect sizes and hypothesis tests that codify existing biases. The first step towards addressing these issues is to be able to evaluate the data, analyses, and models on which conclusions are drawn. Under current practice, this can be difficult because data are typically unavailable, the method sections of papers do not detail the computational approaches used, and analyses and models are often conducted in graphical programs, or, when scripted analyses are employed, the code is not available. And yet, this is easily remedied. Researchers can achieve computational reproducibility through open science approaches, including straightforward steps for archiving data and code openly along with the scientific workflows describing the provenance of scientific results (e.g., Hampton et al. (2015), Munafò et al. (2017)). At NCEAS and in the datateam, not only do we archive data and code openly, but we also describe the workflows that involve that data and code using provenance, formalizing the provenance trace for a workflow that might look like this into an easily understandable trace including archived data objects, such as what is shown here. There are two ways that we add provenance in the datateam - the prov editor and the R datapack package. 7.1 Using the prov editor Provenance can easily be added to production Arctic Data Center packages using the provenance editor on the Arctic Data Center. On the landing page of a data package within beta, in the dataTable or otherEntity section where you would like to add a provenance relationship, you can choose to add either a “source” or a “derivation”, to the left or right of the object pane, respectively. To add a source data file, click on the circle with the “+ add” text. Similarly, a source script would be added by selecting the arrow. Selecting the circle to add a source file pulls up the following screen, where you can select the source from other data objects within the same data package. A data package with an object that has multiple sources added will look like this. For simple packages on the Arctic Data Center, adding prov through the prov editor is super easy! 7.2 Understanding resource maps Before we dive further into constructing prov in R, we need to talk more about resource maps (or “resmaps”). All data packages have a single resource map. But what is a resource map and how do we use one to find out what objects are in a particular data package? This document is a short introduction but a more complete guide can be found here. A resource map is a special kind of XML document that describes (among other things) an “aggregation”. The aggregation describes the members of a data package (metadata and data, usually). We can use the dataone R package to download a resource map if we know its PID: library(dataone) d1c_test &lt;- dataone::D1Client(&quot;STAGING&quot;, &quot;urn:node:mnTestARCTIC&quot;) pid &lt;- &quot;urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot; # A resource map PID path &lt;- tempfile(fileext = &quot;.xml&quot;) # We&#39;re saving to a temporary file but you can save elsewhere writeLines(rawToChar(getObject(d1c_test@mn, pid)), path) # Write the object to `path` If we open that file up in a text editor, we see this: &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt; &lt;rdf:RDF xmlns:cito=&quot;http://purl.org/spar/cito/&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot; xmlns:dcterms=&quot;http://purl.org/dc/terms/&quot; xmlns:foaf=&quot;http://xmlns.com/foaf/0.1/&quot; xmlns:ore=&quot;http://www.openarchives.org/ore/terms/&quot; xmlns:prov=&quot;http://www.w3.org/ns/prov#&quot; xmlns:provone=&quot;http://purl.dataone.org/provone/2015/01/15/ontology#&quot; xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:rdfs=&quot;http://www.w3.org/2000/01/rdf-schema#&quot; xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema#&quot;&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;cito:isDocumentedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;rdf:type rdf:resource=&quot;http://www.openarchives.org/ore/terms/Aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:isDocumentedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:documents rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;cito:documents rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;ore:isAggregatedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;dc:title&gt;DataONE Aggregation&lt;/dc:title&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;ore:describes rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;ore:isAggregatedBy rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556&quot;&gt; &lt;rdf:type rdf:resource=&quot;http://www.openarchives.org/ore/terms/ResourceMap&quot;/&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;&gt; &lt;dcterms:identifier rdf:datatype=&quot;http://www.w3.org/2001/XMLSchema#string&quot;&gt;urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5&lt;/dcterms:identifier&gt; &lt;/rdf:Description&gt; &lt;/rdf:RDF&gt; Whoa! What is this thing and how do you read it to find the members of the data package? The short answer is to look for lines like this: &lt;rdf:Description rdf:about=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3A82bd7d7f-9e18-4fd2-8bda-99b1fddab556#aggregation&quot;&gt; &lt;ore:aggregates rdf:resource=&quot;https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ac59b7505-39e6-4def-bc82-b67a8d117ce5&quot;/&gt; This line says “The aggregation aggregates urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5” so that means urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5 is in our data package! The key bit is the &lt;rdf:Description rdf:about=\"...#aggregation part. If you look for another similar statement, you’ll also see that urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27 is part of our data package. Now we know which objects are in our data package but we don’t know which one contains metadata and which one contains data. For that, we need to get a copy of the system metadata for each object: getSystemMetadata(d1c_test@mn, &quot;urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5&quot;)@formatId getSystemMetadata(d1c_test@mn, &quot;urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27&quot;)@formatId From the formatIds, we can see the first PID is the EML (formatId: [1]\"eml://ecoinformatics.org/eml-2.1.1\") and the second PID is a data object (formatId: [1]\"application/octet-stream\"). Now we know enough to know what’s in the data package: Resource map: urn:uuid:82bd7d7f-9e18-4fd2-8bda-99b1fddab556 Metadata: urn:uuid:c59b7505-39e6-4def-bc82-b67a8d117ce5 Data: urn:uuid:61b48e72-ea29-4ba5-8131-4f59a9ebcd27 Now that you’ve actually seen a resource map, we can dive further into prov. 7.3 Using the datapack package For packages not on the ADC, or packages that are extremely complicated, it may be best to upload prov relationships using R. The datapack package has several functions which help add relationships in a very simple way. These relationships are stored in the resource map. When you update a package to only add prov, the package will not be assigned any new identifiers with the exception of the resource map. First, we set the environment, in a similar, but slightly different way than what you may be used to. Here the function D1Client() sets the DataONE client with the coordinating node instance as the first argument, and member node as the second argument. library(dataone) library(datapack) d1c &lt;- D1Client(&quot;STAGING2&quot;, &quot;urn:node:mnTestKNB&quot;) Next, get the PID of the resource map of the data package you are adding prov to, and load that package into R using the getDataPackage() function. packageId &lt;- &quot;urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b&quot; dp &lt;- getDataPackage(d1c, id=packageId, lazyLoad=TRUE, limit=&quot;0MB&quot;, quiet=FALSE) Printing pkg in your console shows you the contents of the data package, including all of the objects and their names: &gt; pkg Members: filename format mediaType size identifier modified local esc...er.R application/R NA 888 knb.92049.1 n n PWS....csv text/csv NA 1871469 knb.92050.1 n n PWS....csv text/csv NA 1508128 knb.92051.1 n n NA eml:/...-2.1.1 NA 15658 urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b n y Package identifier: resource_map_urn:uuid:8f501606-2c13-4454-b22d-050a4176a97b RightsHolder: http://orcid.org/0000-0002-2192-403X It will also show the existing relationships in the resource map, which in this case are mostly the “documents” relationships that specify that the metadata record is describing all of these data files. Relationships: subject predicate object 2 esc_reformatting_PWSweirTower.R cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 4 PWS_weirTower.csv cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 1 PWS_Weir_Tower_export.csv cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b 3 urn:uuid:8f501606-...4-b22d-050a4176a97b dcterms:creator _r1515542097r415842r1 5 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents esc_reformatting_PWSweirTower.R 6 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents PWS_weirTower.csv 7 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents PWS_Weir_Tower_export.csv 8 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:documents urn:uuid:8f501606-...4-b22d-050a4176a97b 9 urn:uuid:8f501606-...4-b22d-050a4176a97b cito:isDocumentedBy urn:uuid:8f501606-...4-b22d-050a4176a97b In this example above, the data package has two .csv files, with an R script that converts one to the other. To create our provenance trace, first we need to select the source object, and save the PID to a variable. We do this using the selectMember() function, and we can query part of the system metadata to select the file that we want. This function takes the data package (pkg), the name of the sysmeta field to query (in this case we use the fileName), and the value that you want to match that field to (in this case, ‘PWS_Weir_Tower_export.csv’). sourceObjId &lt;- selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&#39;PWS_Weir_Tower_export.csv&#39;) This returns a list of the source object PIDs that match the query (in this case only one object matches). &gt; sourceObjId [1] &quot;knb.92051.1&quot; Now we need to select our output object. Here, we use the selectMember() function again, and save the result to a new variable. outputObjId &lt;- selectMember(dp, name=&quot;sysmeta@fileName&quot;, value=&#39;PWS_weirTower.csv&#39;) Now we query for the R script. In this case, we query based on the value of the formatId as opposed to the fileName. This can be useful if you wish to select a large list of PIDs that are all similar. programObjId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;application/R&quot;) Next, you use these lists of PIDs and a function called describeWorkflow() to add these relationships to the data package. Note that if you do not have a program in the workflow, or a source file, you can simply leave those arguments blank. dp &lt;- describeWorkflow(dp, sources=sourceObjId, program=programObjId, derivations=outputObjId) Viewing pkg again confirms that these relationships have been inserted into the data package, as shown by the wasDerivedFrom and wasGeneratedBy statements. It is always a good idea to print pkg to confirm that your PID selection process worked as expected, and your prov relationships make sense. Relationships (updated): subject predicate object 15 _1db49d06-ae98-4...9101-39f7c0b45a95 rdf:type prov:Association 14 _1db49d06-ae98-4...9101-39f7c0b45a95 prov:hadPlan esc_reformatting_PWSweirTower.R 1 esc_reformatting_PWSweirTower.R cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 16 esc_reformatting_PWSweirTower.R rdf:type provone:Program 8 PWS_weirTower.csv cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 11 PWS_weirTower.csv rdf:type provone:Data 20 PWS_weirTower.csv prov:wasDerivedFrom PWS_Weir_Tower_export.csv 19 PWS_weirTower.csv prov:wasGeneratedBy urn:uuid:3dd59b0...bc38-3b5d8fa644ac 6 PWS_Weir_Tower_export.csv cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b 10 PWS_Weir_Tower_export.csv rdf:type provone:Data 9 _r1515544826r415842r1 foaf:name DataONE R Client 17 urn:uuid:3dd59b0...bc38-3b5d8fa644ac dcterms:identifier urn:uuid:3dd59b0...bc38-3b5d8fa644ac 13 urn:uuid:3dd59b0...bc38-3b5d8fa644ac rdf:type provone:Execution 12 urn:uuid:3dd59b0...bc38-3b5d8fa644ac prov:qualifiedAssociation _1db49d06-ae98-4...9101-39f7c0b45a95 18 urn:uuid:3dd59b0...bc38-3b5d8fa644ac prov:used PWS_Weir_Tower_export.csv 5 urn:uuid:8f50160...b22d-050a4176a97b cito:documents esc_reformatting_PWSweirTower.R 4 urn:uuid:8f50160...b22d-050a4176a97b cito:documents PWS_weirTower.csv 3 urn:uuid:8f50160...b22d-050a4176a97b cito:documents PWS_Weir_Tower_export.csv 2 urn:uuid:8f50160...b22d-050a4176a97b cito:documents urn:uuid:8f50160...b22d-050a4176a97b 7 urn:uuid:8f50160...b22d-050a4176a97b cito:isDocumentedBy urn:uuid:8f50160...b22d-050a4176a97b Finally, you can upload the data package using the uploadDataPackage() function, which takes the DataONE client d1c we set in the beginning, the updated pkg variable, some options for public read, and whether or not informational messages are printed during the upload process. First, you will need to run a token obtained from https://dev.nceas.ucsb.edu/ to publish to that node. resmapId_new &lt;- uploadDataPackage(d1c, dp, public = TRUE, quiet = FALSE) If successful you should be able to navigate to the landing page of your dataset, and icons should show up where the sources and derivations are, such as in this example: 7.3.1 Fixing mistakes If you messed up updating a data package using datapack, there unfortunately isn’t a great way to undo your work, as the describeWorkflow() only adds prov relationships, it does not replace them. If you messed up, the best course of action is to update the resource map with a clean version that does not have prov using update_resource_map(), and then go through the steps outlined above again. Note: this has not been thoroughly tested, and more extreme actions may be necessary to fully nuke the prov relationships. Consult Peter or Jeanette if things do not work as expected. 7.4 References Ioannidis, John P A. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124. Hampton, Stephanie E, Sean Anderson, Sarah C Bagby, Corinna Gries, Xueying Han, Edmund Hart, Matthew B Jones, et al. 2015. “The Tao of Open Science for Ecology.” Ecosphere 6 (July). https://doi.org/10.1890/ES14-00402.1. Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. “A Manifesto for Reproducible Science.” Nature Human Behaviour 1 (1): 0021. https://doi.org/10.1038/s41562-016-0021. "],["using-git.html", "Chapter 8 Using Git 8.1 Usage in the datateam 8.2 What is Git? 8.3 Setting up Git 8.4 GitHub Authentication 8.5 Working with the repository 8.6 My Git tab disappeared", " Chapter 8 Using Git 8.1 Usage in the datateam We use git and Github to manage our packages (ie datamgmt, arcticdatautils) and even this document! 8.2 What is Git? Git is a distributed version control system. The NCEAS Github repository contains many processing scripts and templates that can help when working with datasets. Important! If you have never used Git before, or only used it a little, or have no idea what it is, check out this intro to Git put together by the ecodatascience group at UCSB. Don’t worry too much about the forking and branching sections, as we will primarily be using the basic commit-pull-push commands. After you have read through that presentation, come back to this chapter. 8.2.1 So why do I need to use this again? There are several reasons why using the arctic-data GitHub repository is helpful, both for you and for the rest of the data team. Here are a few: Versioning: Did you accidentally make a change to your code and can’t figure out why it broke? Do you wish you could go back to the version that worked? If you add your code to the GitHub repo you can do this! Reproducibility: Being able to reproduce how you accomplished something is incredibly important. We should be able to tell anyone exactly how data have been reformatted, how metadata have been altered, and how packages have been created. As a data center, this is especially important for us with data team members that stay for 6-12 months because we may need to go back and figure out how something was done after the intern or fellow who wrote the code left the team. Troubleshooting: If you are building a particularly complicated EML, or doing some other advanced task, it is much easier for others to help you troubleshoot your code if it is on the GitHub repo. We can view, troubleshoot, and fix bugs very easily when code is on the GitHub repo, with the added bonus of being able to go back a version if something should break. Solve future problems: Some of the issues we see in ADC submissions come up over and over again. When all of our code is on GitHub, we can easily reference code built for other submissions, instead of trying to solve the same problems over and over again from scratch. 8.3 Setting up Git Before using git, you need to tell it who you are. The only way to do this is through the command line. When you open RStudio, you should see a Terminal tab located to the right of the Console tab. If the Terminal tab is not visible, you can open it by selecting Tools -&gt; Terminal -&gt; New Terminal at the top of your RStudio window. To tell git who you are, you’re going to set the global options. This includes setting your name, email address, PAT (Personal Access Token), and caching the PAT. Make sure you run the following commands one at a time. Type the following command in the Terminal window, with your actual name, and press enter. If you do this correctly, it will look as though nothing happened. git config --global user.name &quot;Your Name&quot; Next, enter the following line, with the email address associated with your GitHub account. git config --global user.email MyEmail@domain.com After running these commands, the Terminal window should look like this: Next, we will tell Git to store your PAT. We must do this because of the way our server operating system handles credentials. If you don’t run the next line, your PAT will expire immediately on the server, even though we will set it up on GitHub to be valid for 90 days. git config --global credential.helper &#39;cache -- timeout=10000000&#39; Finally, check to make sure everything looks correct by entering the following command, which will return the options that you have set. git config --global --list 8.4 GitHub Authentication GitHub recently deprecated password authentication for accessing repositories, so we need to set up a secure way to authenticate. That’s where the PAT comes in. In the tabs at the bottom of your RStudio window, make sure to switch from the Terminal to the Console. Now that, you’ve done that, follow these steps: Run usethis::create_github_token() In the browser window that pops up, log into GitHub In the “Expiration” drop down menu, select “90 days” Click “Generate token” in the green box at the bottom of the page On the next page, copy the token from the green box Back in RStudio, run credentials::set_github_pat() Paste your token into the dialog box that pops up You’ve now successfully set your PAT in RStudio, and connected R, Git, and GitHub. Great job! Now, you’re going to clone the repository we work in so that you can version control your code. 8.4.1 Cloning the arctic-data repo To clone the arctic-data repository, navigate to the repository on GitHub: https://github.nceas.ucsb.edu/KNB/arctic-data/. If you are not already logged on, this will be the enterprise account you use to log in when you go to RT. If you have trouble with this, send a message to Nick Outin (@Nick) in the datateam slack channel. Once you are logged on and can view the above link, click on the green “Code” button, and copy the URL in the gray box. In your RStudio window, click File -&gt; New Project -&gt; Version Control -&gt; Git. Paste the URL you just copied into the “Repository URL” box, and press tab on your keyboard. This should autofill the “Project directory name.” If it does not, then type in arctic-data. Fill it out as shown in the image below to clone the arctic-data repository into the top level of your home directory. Click “Create Project,” and now you should have a directory called arctic-data in your RStudio files window. If you are prompted to save your workspace during this process, make sure all of your work is saved, and you don’t need anything in your environment, and then click ‘Don’t Save’. 8.5 Working with the repository 8.5.1 Adding a new script If you have been working on a script that you want to put in the arctic-data GitHub repo, you first need to save it somewhere in the arctic-data folder you cloned to your account on the Datateam server (/home/username/…). You can do this by either moving your script into the folder or using the save-as functionality. Note that Git will try and version anything that you save in this folder, so you should be careful about what you save here. For our purposes, things that probably shouldn’t be saved in this folder include: Tokens: Any token file or script with a token in it should NOT be saved in the repository. Others could steal your login credentials if you put a token in GitHub. Data files: Git does not version data files very well. You shouldn’t save any .csv files or any other data files (including metadata). Workspaces/.RData: If you are in the habit of saving your R workspace, you shouldn’t save it in this directory. Plots/Graphics: For the same reasons as data files. Note: Do not EVER make a commit that you don’t understand. If something unexpected (like a file you have never worked on) shows up in your Git tab, ask for help before committing. After you save your script in the appropriate place within the arctic-data folder, it will show up in your Git tab looking like this: Before you commit your changes, you need to click the little box under “Staged”. Do not stage or commit any .Rproj file. After clicking the box for your file, click “Commit” to commit your changes. In the window that pops up (you may need to force the browser to allow pop-ups), write your commit message. Always include a commit message. Remember that the commit message should be a concise description of the changes that were made to a file. Your window should look like this: Push ‘Commit’, and your commit will be saved to your local repository (this will not push it to the remote repository, yet). Now you want to merge the commits you made with the master version of the remote repository. You do this by using the command “Push.” But before you push, you always need to pull first to avoid merge conflicts. Pulling will merge the current version of the remote repository with your local repository, on your local machine. Click “Pull” and type in your credentials. Then, assuming you don’t have a merge conflict, you can push your changes by clicking “Push”. Always remember, the order is commit-pull-push. 8.5.2 Editing a script If you want to change a script, the workflow is the same. Just open the script that was saved in the arctic-data folder on your server account, make your changes, save the changes, stage them by clicking the box, commit, pull, then push to merge your version with the main version on the website. Do NOT edit scripts using the GitHub website. It is much easier to accidentally overwrite the history of a file this way. One thing you might be wondering as you are working on a script is, how often should I be committing my changes? It might not make sense to commit-pull-push after every single tiny change - if only because it would slow you way down. Personally, I commit every time I feel that a significant change has happened and that the chunk of code I was working on is “done”. Sometimes this is an entire script, other times it is just a few lines within a script. A good sign that you are committing too infrequently might be if many of your commit messages address a wide variety of coding tasks, such as: “wrote for loop to create referenced attribute lists for tables 1:20. also created nesting structure for this package with another package. also created attribute list for data table 40”. One final note, you can make multiple commits before you push to the repo. If you are making lots of changes to the script, you might want to make several commits before pull-push. You can see how many commits you are ahead of the “origin/master” branch (i.e. what you see on the website) by looking for text in your Git tab in RStudio that looks like this: 8.5.3 Where do I commit? The default right now is to save data-processing scripts in the arctic-data/datateam/data-processing/ directory, with sub-folders listed by project. Directories can be created as needed but please ask Dom or Jesse first so we can try and maintain some semblance of order in the file structure. The Github workflow diagram from stamminator shows each step and their functions explicitly. 8.6 My Git tab disappeared Sometimes R will crash so hard it loses your project information, causing your Git tab to disappear. Most likely, RStudio has just closed your “project” and all you need to do is reopen it. If your Git tab has disappeared, in the top right of your RStudio session, you should see a little R logo with “Project: (None)” next to it. This means you do not currently have a project open. Clicking the arrow should give you a dropdown menu of recent projects, where you can select “arctic-data” or “sasap-data.” Once you have opened your project, the Git tab should reappear! This is also a convenient way to switch between projects if you are working in multiple repositories. "],["first-ticket.html", "Chapter 9 First Ticket 9.1 Navigate RT 9.2 Initial review checklist 9.3 Processing templates 9.4 Final Checklist 9.5 Email templates 9.6 Categorize datasets 9.7 Congrats!", " Chapter 9 First Ticket After completing the previous chapters, your supervisor will assign a ticket from RT. Login using your LDAP credentials got get familiarized with RT. 9.1 Navigate RT The RT ticketing system is how we communicate with folks interacting with the Arctic Data Center. We use it for managing submissions, accessing issues, etc. It consists of three separate interfaces: Front Page All Tickets Ticket Page 9.1.1 Front page This is what you see first Home - brings you to this homepage Tickets - to search for tickets (also see number 5) Tools - not needed New Ticket - create a new ticket Search - Type in the ticket number to quickly navigate to a ticket Queue - Lists all of the tickets currently in a particular queue (such as ‘arcticdata’) and their statuses New = unopened tickets that require attention Open = tickets currently open and under investigation and/or being processed by a support team member Stalled = tickets awaiting responses from the PI/ submitter Tickets I Own - These are the current open tickets that are claimed by me Unowned Tickets - Newest tickets awaiting claim Ticket Status - Status and how long ago it was created Take - claim the ticket as yours 9.1.2 All tickets This is the queue interface from number 6 of the Front page 1. Ticket number and title 2. Ticket status 3. Owner - who has claimed the ticket 9.1.3 Example ticket Title - Include the PI’s name for reference Display - homepage of the ticket History - Comment/Email history, see bottom of Display page Basics - edit the title, status, and ownership here People - option to add more people to the watch list for a given ticket conversation. Note that user/ PI/ submitter email addresses should be listed as “Requestors”. Requestors are only emailed on “Replys”, not “Comments”. Ensure your ticket has a Requestor before attempting to contact users/ PIs/ submitters Links - option to “Merge into” another ticket number if this is part of a larger conversation. Also option to add a reference to another ticket number Verify that this is indeed the two tickets you want to merge. It is non-reversible. Actions Reply - message the submitter/ PI/ all watchers Comment - attach internal message (no submitters, only Data Teamers) Open It - Open the ticket Stall - submitter has not responded in greater than 1 month Resolve - ticket completed History - message history and option to reply (to submitter and beyond) or comment (internal message) 9.1.4 New data submission When notified by Arcticbot about a new data submission, here are the typical steps: Update the Requestor under the People section based on the email given in the submission (usually the user/ PI/ submitter). You may have to google for the e-mail address if the PI did not include it in the metadata record. Take the ticket (Actions &gt; Take) Review the submission based on the checklist Draft an email using the template and let others review it via Slack Send your reply via Actions Before opening a R script first look over the initial checklist first to identify what you will need to update in the metadata. 9.2 Initial review checklist Before responding to a new submission use this checklist to review the submission. When your are ready to respond use the initial email template and insert comments and modify as needed. 9.2.0.1 Sensitive Data If any of the below is in the dataset, please alert the #arctica team know before proceeding. Check if there is any sensitive information or personal identifying information in the data (eg. Names) Can the data be disaggregated and de-anonymized? (eg. a small sample size and individuals could be easily identified by their answers) Dryad Human Subject data guidelines can be a good place to start Common Cases: Social Science: Any dataset involving human subjects (may include awards awarded by ASSP and topics such as COVID-19) Archaeology: archaeological site location information, which is protected from public access by law Biology: protected species location coordinates 9.2.0.2 Data citations If the dataset appears to be in a publication please (might be in the abstract) make sure that those citations are registered. 9.2.0.3 Title WHAT, WHERE, and WHEN: Is descriptive of the work (provides enough information to understand the contents at a general scientific level), AND includes temporal coverage Provides a location of the work from the local to state or country level Provides a time frame of the work NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out 9.2.0.4 Abstract Describes the DATA as well as: The motivation (purpose) of the study Where and when the research took place At least one sentence summarizing general methodologies NO UNDEFINED ACRONYMS, ABBREVIATIONS, nor INITIALISMS unless approved of as being more widely-known in that form than spelled out At least 100 words total tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting Any citations to papers can be registered with us 9.2.0.5 Keywords Some keywords are included 9.2.0.6 Data Data is normalized (if not suggest to convert the data if possible) At least one data file, or an identifier to the files at another approved archive, unless funded by ASSP (Arctic Social Sciences Program) No xls/xlsx files (or other proprietary files) File contents and relationships among files are clear Each file is well NAMED and DESCRIBED and clearly differentiated from all others All attributes in EML match attribute names in respective data files EXACTLY, are clearly defined, have appropriate units, and are in the same order as in the file. Quality control all dimensionless units. Missing value codes are explained (WHY are the data absent?) If it is a .rar file -&gt; scan the file If there is the unit tons make sure to ask if it is metric tons or imperical tons if not clarified already 9.2.0.7 People &amp; Parties At least one contact and one creator with a name, email address, and ORCID iD 9.2.0.8 Coverages Includes coverages that make sense Temporal coverage - Start date BEFORE end date Geologic time scales are added if mentioned in metadata (e.g. 8 Million Years or a name of a time period like Jurassic) Spatial coverage matches geographic description (check hemispheres) Geographic description is from the local to state or country level, at the least Taxonomic coverage if appropriate 9.2.0.9 Project Information At least one FUNDING number Title, personnel, and abstract match information from the AWARD (not from the data package) 9.2.0.10 Methods This section is REQUIRED for ALL NSF-FUNDED data packages Enough detail is provided such that a reasonable scientist could interpret the study and data for reuse without needing to consult the researchers, nor any other resources 9.2.0.11 Portals If there are multiple submissions from the same people/project let them know about the portals feature If this is part of a portal make sure this dataset can be found there. Additional steps might be needed to get that to work. Please consult Jeanette and see the data portals section. 9.3 Processing templates We have developed some partially filled R scripts to get you started on working on your first dataset. They outline common functions used in processing a dataset. However, it will differ depending on the dataset. You can use this template where you can fill in the blanks to get familiar with the functions we use and workflow at first. We also have a more minimal example A filled example as a intermediate step. You can look at the filled example if you get stuck or message the #datateam. In addition, you may find this cheat sheet of data team R functions helpful. Once you have updated the dataset to your satisfaction and reviewed the Final Checklist, post the link to the dataset on #datateam for peer review. 9.4 Final Checklist You can click on the assessment report on the website to for a general check. Fix anything you see there. Send the link over slack for peer review by your fellow datateam members. Usually we look for the following (the list is not exhaustive): 9.4.1 Special Datasets Please refer to the dedicated pages for instructions to handle these cases: MOSAiC DBO 9.4.2 System Metadata the format ids are correct 9.4.3 General EML Ethical Research Practice section is completed Publisher and system information have been added: doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) 9.4.4 Title Include geographic and temporal coverage (when and where data was collected) Abbreviations are removed or defined 9.4.5 Abstract longer than 100 words Abbreviations are removed or defined. No garbled text Tags such as &lt;superscript&gt;2&lt;/superscript&gt; and &lt;subscript&gt;2&lt;/subscript&gt; can be used for nicer formatting 9.4.6 DataTable / OtherEntity / SpatialVectors In the correct one: DataTable / OtherEntity / SpatialVector / SpatialRaster for the file type entityDescription - longer than 5 words and unique physical present and up-to-date and format is correct 9.4.6.1 Attribute Table Complete attributeDefinitions longer than 3 words Variables match what is in the file Measurement domain - if appropirate (ie dateTime correct) Missing Value Code - accounted for if applicable Semantic Annotation - appropriate semantic annotations added, especially for spatial and temporal variables: lat, lon, date etc. Custom units were created if necessary 9.4.7 People Complete information for each person in each section includes ORCID and e-mail address for all contacts people repeated across sections have consistent information 9.4.8 Geographic region the map looks correct and matches the geographic description check if negatives (-) are missing 9.4.9 Project If it is an NSF award you can use the helper function: doc$dataset$project &lt;- eml_nsf_to_project(awards) for other awards that need to be set manually, see the set project page 9.4.10 Methods Present No garbled text Acronyms and units are defined the first time they are used 9.4.11 Check EML Version Currently using: eml-2.2.0 (as of July 30 2020) Review to see if the EML version is set correctly by reviewing the doc$`@context` that it is indeed 2.2.0 under eml Re-run your code again and have the lineemld::eml_version(\"eml-2.2.0\") at the top Make sure the system metadata is also 2.2.0 9.4.12 Access If necessary, granted access to PI using set_rights_and_access() make sure it is http:// (no s) note if it is a part of portals there might be specific access requirements for it to be visible using set_access() 9.4.13 SFTP Files If there are files transferred to us via SFTP, delete those files once they have been added to the dataset and when the ticket is resolved 9.4.14 Updated datasets All the above applies. These are some areas to do a closer check when users update with a new file: New data was added Temporal Coverage and Title Confirm if new data used same methods or if methods need updating Files were replaced update physical and entityName double-check attributes are the same check for any new missing value codes that should be accounted for Was the dataset published before 2021? update project info, annotations Glance over entire page for any small mistakes (ie. repeated additionalMetadata, any missed &amp;amps, typos) After all the revisions send the link to the PI in an email through RT. Send the draft of the email to Daphne or Jeanette on Slack. 9.5 Email templates This section covers new data packages submitted. For other inquiries see the PI FAQ templates Please think critically when using these canned replies rather than just blindly sending them. Typically, content should be adjusted/ customized for each response to be as relevant, complete, and precise as possible. In your first few months, please run email drafts by the #datateam Slack and get approval before sending. Remember to consult the submission guidelines for details of what is expected. Quick reference: Initial email template Final email templates Additional email template 9.5.1 Initial email template Hello [NAME OF REQUESTOR], Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: [COMMENTS HERE] After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, [YOUR NAME] 9.5.2 Comment templates based on what is missing 9.5.2.1 Portals Multiple datasets under the same project - suggest data portal feature I would like to highlight the Data Portals feature that would enhance the ability of your datasets to be discovered together. It also enables some custom branding for the group or project. Here is an example page that we created for the Distributed Biological Observatory: https://arcticdata.io/catalog/portals/DBO. We highly suggest considering creating a portal for your datasets, you can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/. Data portals can be set up at any point of your project. If they ask to nest the dataset We are no longer supporting new nested datasets. We recommend to create a data portal instead. Portals will allow more control and has the same functionality as a nested dataset. You can get one started here: https://arcticdata.io/catalog/edit/portals/new/Settings. More information on how to set up can be found here: https://arcticdata.io/data-portals/, but as always we are here to help over email. 9.5.2.2 Dataset citations If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be done at any point. 9.5.2.3 Title Provides the what, where, and when of the data We would like to add some more context to your data package title. I would like to suggest: ‘OUR SUGGESTION HERE, WHERE, WHEN’. Does not use acronyms We wanted to clarify a couple of abbreviations. Could you help us in defining some of these: [LIST OF ACRONYMS TO DEFINE HERE] 9.5.2.4 Abstract Describes DATA in package (ideally &gt; 100 words) We would like to add some additional context to your abstract. We hope to add the following: [ADJUST THE DEPENDING ON WHAT IS MISSING] The motivation of the study Where and when the research took place At least one sentence summarizing general methodologies All acronyms are defined At least 100 words long Offer this if submitter is reluctant to change: If you prefer and it is appropriate, we could add language from the abstract in the NSF Award found here: [NSF AWARD URL]. 9.5.2.5 Keywords We noticed that there were no keywords for this dataset. Adding keywords will help your dataset be discovered by others. 9.5.2.6 Data Sensitive Data We will need to ask these questions manually until the fields are added to the webform. Data sensitivity categories Once we have the ontology this question can be asked: Based on our Data sensitivity categories, which of the 3 does your dataset align with most: Non-sensitive data - None of the data includes sensitive or protected information. Some or all data is sensitive with minimal risk - Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Some or all data is sensitive with significant risk - The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Ethical research proceedures We were wondering if you could also address this question specifically on Ethical Research Procedures: Describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Insitutional Review Board approvals, consent waivers, procedures for co-production, data sovreignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. We can help add your answers to the question to the metadata. Adding provenance Is the [mention the file names here] files related? If so we can add provenance to show the relationship between the files. Here is an example of how that is displayed: https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2WS8HM6C#urn%3Auuid%3Af00c4d71-0242-4e9d-9745-8999776fa2f2 At least one data file We noticed that no data files were submitted. With the exception of sensitive social science data, we seek to include all data products prior to publication. We wanted to check if additional files will be submitted before we move forward with the submission process. Open formats Example using xlsx. Tailor this reponse to the format in question. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. The data files can be replaced by going to the green Edit button &gt; Click the black triangle by the Describe button for the data file &gt; Select Replace (attached is also a screenshot on how to get there). Zip files Except for very specific file types, we do not recommend that data are archived in zip format. Data that are zipped together can present a barrier to reuse since it is difficult to accurately document each file within the zip file in a machine readable way. File contents and relationships among files are clear Could you provide a short description of the files submitted? Information about how each file was generated (what software, source files, etc.) will help us create more robust metadata for long term use. Data layout Would you be able to clarify how the data in your files is laid out? Specifically, what do the rows and columns represent? We try not to prescribe a way the researchers must format their data as long as reasonable. However, in extreme cases (for example Excel spreadsheets with data and charts all in one sheet) we will want to kindly ask them to reformat. We would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. [DESCRIBE WHAT MAY NEED TO BE CHANGED IN THE DATA SET]. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. 9.5.2.7 Attributes Identify which attributes need additional information. If they are common attributes like date and time we do not need further clarification. Checklist for the datateam in reviewing attributes (NetCDF, CSV, shapefiles, or any other tabular datasets): A name (often the column or row header in the file). A complete definition. Any missing value codes along with explanations for those codes. For all numeric data, unit information is needed. For all date-time data, a date-time format is needed (e.g. “DD-MM-YYYY”). For text data, full descriptions for all patterns or code/definition pairs are needed if the text is constrained to a list of patterns or codes. Helpful templates: &gt; We would like your help in defining some of the attributes. Could you write a short description or units for the attributes listed? [Provide a the attribute names in list form] &gt; Could you describe ____? &gt; Please define “XYZ”, including the unit of measure. &gt; What are the units of measurement for the columns labeled “ABC” and “XYZ”? Missing value codes What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. This section is not yet available on our webform so we will add that information on your behalf. We noticed that the data files contain [blank cells - replace with missing values found]. What do these represent? 9.5.2.8 Funding All NSF funded datasets need a funding number. Non-NSF funded datasets might not have funding numbers, depending on the funding organization. We noticed that your dataset does not appear to contain a funding number. The field accepts NSF funding numbers as well as other numbers by different organizations. 9.5.2.9 Methods We noticed that methods were missing from the submission. Submissions should include the following: provide instrument names (if applicable) specify how sampling locations were chosen if citations for sampling methods are used, please provide a brief summary of the methods referenced any software used to process the data Note - this includes software submissions as well (see https://arcticdata.io/submit/#metadata-guidelines-for-software) Your methods section appears to be missing some information. [ADJUST THIS DEPENDING ON WHAT IS MISSING - Users should be able to understand how the data were collected, how to interpret the values, and potentially how to use the data in the case of specialized files.] Comprehensive methods information should be included directly in the metadata record. Pointers or URLs to other sites are unstable. A full example - New Submission: methods, excel to csv, and attributes Thank you for your recent submission to the NSF Arctic Data Center! From my preliminary examination of your submission I have noticed a few items that I would like to bring to your attention. We are here to help you publish your submission, but your continued assistance is needed to do so. See comments below: If there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. This can be added at any point. We would like to add some more context to your data package title. I would like to suggest: Holocene lake-based Arctic glacier and ice cap records. We noticed that the submitted data files are in xlsx format. Please convert your files to a plain text/csv (or other open format); this helps ensure your data are usable in the long-term. We also would like to suggest a couple of modifications to the structure of your data. This will others to re-use it most effectively. If the data was in a long rather than wide format, it would be easier to us to document. Our data submission guidelines page (https://arcticdata.io/submit/) outlines what are best practices for data submissions to the Arctic Data Center. Let us know if you have any questions or if we can be of any help. What do the missing values in your measurements represent? A short description of the reason why the values are missing (instrument failure, site not found, etc.) will suffice. We noticed that methods were missing from the submission. Submissions should: - provide instrument names (if applicable) - specify how sampling locations were chosen - provide citations for sampling methods that are not explained in detail - any software used to process the data After we receive your responses, we can make the edits on your behalf, or you are welcome to make them yourself using our user interface. Best, Name 9.5.3 Final email templates 9.5.3.1 Asking for approval Hi [submitter], I have updated your data package and you can view it here after logging in: [URL] Please review and approve it for publishing or let us know if you would like anything else changed. For your convenience, if we do not hear from you within a week we will proceed with publishing with a DOI. After publishing with a DOI, any further changes to the dataset will result in a new DOI. However, any previous DOIs will still resolve and point the user to the newest version. Please let us know if you have any questions. 9.5.3.2 DOI and data package finalization comments Replying to questions about DOIs We attribute DOIs to data packages as one might give a DOI to a citable publication. Thus, a DOI is permanently associated with a unique and immutable version of a data package. If the data package changes, a new DOI will be created and the old DOI will be preserved with the original version. DOIs and URLs for previous versions of data packages remain active on the Arctic Data Center (will continue to resolve to the data package landing page for the specific version they are associated with), but a clear message will appear at the top of the page stating that “A newer version of this dataset exists” with a hyperlink to that latest version. With this approach, any past uses of a DOI (such as in a publication) will remain functional and will reference the specific version of the data package that was cited, while pointing users to the newest version if one exists. Clarification of updating with a DOI and version control We definitely support updating a data package that has already been assigned a DOI, but when we do so we mark it as a new revision that replaces the original and give it its own DOI. We do that so that any citations of the original version of the data package remain valid (i.e.: after the update, people still know exactly which data were used in the work citing it). 9.5.3.3 Resolve the ticket Sending finalized URL and dataset citation before resolving ticket [NOTE: the URL format is very specific here, please try to follow it exactly (but substitute in the actual DOI of interest)] Here is the link and citation to your finalized data package: https://doi.org/10.18739/A20X0X First Last, et al. 2021. Title. Arctic Data Center. doi:10.18739/A20X0X. If in the future there is a publication associated with this dataset, we would appreciate it if you could register the DOI of your published paper with us by using the Citations button right below the title at the dataset landing page. We are working to build our catalog of dataset citations in the Arctic Data Center. Please let us know if you need any further assistance. 9.5.4 Additional email templates 9.5.4.1 Deadlines If the PI is checking about dates/timing: &gt; [give rough estimate of time it might take] &gt; Are you facing any deadlines? If so, we may be able to expedite publication of your submission. 9.5.4.2 Pre-assigned DOI If the PI needs a DOI right away: We can provide you with a pre-assigned DOI that you can reference in your paper, as long as your submission is not facing a deadline from NSF for your final report. However, please note that it will not become active until after we have finished processing your submission and the package is published. Once you have your dataset published, we would appreciate it if you could register the DOI of your published paper with us by using the citations button beside the orange lock icon. We are working to build our catalog of dataset citations in the Arctic Data Center. 9.5.4.3 Sensitive Data Which of the following categories best describes the level of sensitivity of your data? A. Non-sensitive data None of the data includes sensitive or protected information. Proceed with uploading data. B. Some or all data is sensitive but has been made safe for open distribution Sensitive data has been de-identified, anonymized, aggregated, or summarized to remove sensitivities and enable safe data distribution. Examples include ensuring that human subjects data, protected species data, archaeological site locations and personally identifiable information have been properly anonymized, aggregated and summarized. Proceed with uploading data, but ensure that only data that are safe for public distribution are uploaded. Address questions about anonymization, aggregation, de-identification, and data embargoes with the data curation support team before uploading data. Describe these approaches in the Methods section. C. Some or all data is sensitive and should not be distributed The data contains human subjects data or other sensitive data. Release of the data could cause harm or violate statutes, and must remain confidential following restrictions from an Institutional Review Board (IRB) or similar body. Do NOT upload sensitive data. You should still upload a metadata description of your dataset that omits all sensitive information to inform the community of the dataset’s existence. Contact the data curation support team about possible alternative approaches to safely preserve sensitive or protected data. Ethical Research Procedures. Please describe how and the extent to which data collection procedures followed community standards for ethical research practices (e.g., CARE Principles). Be explicit about Institutional Review Board approvals, consent waivers, procedures for co-production, data sovereignty, and other issues addressing responsible and ethical research. Include any steps to anonymize, aggregate or de-identify the dataset, or to otherwise create a version for public distribution. 9.5.4.4 Asking for dataset access As a security measure we ask that we get the approval from the original submitter of the dataset prior to granting edit permissions to all datasets. 9.5.4.5 No response from the researcher Please email them before resolving a ticket like this: We are resolving this ticket for bookkeeping purposes, if you would like to follow up please feel free to respond to this email. 9.5.4.6 Recovering Dataset submissions To recover dataset submissions that were not successful please do the following: Go to https://arcticdata.io/catalog/drafts Find your dataset and download the corresponding file Send us the file in an email 9.5.4.7 Custom Search Link You could also use a permalink like this to direct users to the datasets: https://arcticdata.io/catalog/data/query=\"your search query here” for example: https://arcticdata.io/catalog/data/query=Beaufort%20Lagoon%20Ecosystems%20LTER 9.5.4.8 Adding metadata via R KNB does not support direct uploading of EML metadata files through the website (we have a webform that creates metadata), but you can upload your data and metadata through R. Here are some training materials we have that use both the EML and datapack packages. It explains how to set your authentication token, build a package from metadata and data files, and publish the package to one of our test sites. I definitely recommend practicing on a test site prior to publishing to the production site your first time through. You can point to the KNB test node (dev.nceas.ucsb.edu) using this command: d1c &lt;- D1Client(\"STAGING2\", \"urn:node:mnTestKNB\") If you prefer, there are Java, Python, MATLAB, and Bash/cURL clients as well. 9.5.4.9 Finding multiple data packages If linking to multiple data packages, you can send a link to the profile associated with the submitter’s ORCID iD and it will display all their data packages. e.g.: https://arcticdata.io/catalog/profile/http://orcid.org/0000-0002-2604-4533 9.5.4.10 NSF ARC data submission policy Please find an overview of our submission guidelines here: https://arcticdata.io/submit/, and NSF Office of Polar Programs policy information here: https://www.nsf.gov/pubs/2016/nsf16055/nsf16055.jsp. Investigators should upload their data to the Arctic Data Center (https://arcticdata.io), or, where appropriate, to another community endorsed data archive that ensures the longevity, interpretation, public accessibility, and preservation of the data (e.g., GenBank, NCEI). Local and university web pages generally are not sufficient as an archive. Data preservation should be part of the institutional mission and data must remain accessible even if funding for the archive wanes (i.e., succession plans are in place). We would be happy to discuss the suitability of various archival locations with you further. In order to provide a central location for discovery of ARC-funded data, a metadata record must always be uploaded to the Arctic Data Center even when another community archive is used. 9.5.4.11 Linking ORCiD and LDAP accounts First create an account at orcid.org/register if you have not already. After that account registration is complete, login to the KNB with your ORCID iD here: https://knb.ecoinformatics.org/#share. Next, hover over the icon on the top right and choose “My Profile”. Then, click the “Settings” tab and scroll down to “Add Another Account”. Enter your name or username from your Morpho account and select yourself (your name should populate as an option). Click the “+”. You will then need to log out of knb.ecoinformatics.org and then log back in with your old LDAP account (click “have an existing account”, and enter your Morpho credentials with the organization set to “unaffiliated”) to finalize the linkage between the two accounts. Navigate to “My Profile” and “Settings” to confirm the linkage. After completing this, all of your previously submitted data pacakges should show up on your KNB “My Profile” page, whether you are logged in using your ORCiD or Morpho account, and you will be able to submit data either using Morpho or our web interface. Or, try reversing my instructions - log in first using your Morpho account (by clicking the “existing account” button and selecting organization “unaffiliated”), look for your ORCiD account, then log out and back in with ORCiD to confirm the linkage. Once the dataset is approved by the PI and there are no further changes, publish the dataset with a doi. doi &lt;- dataone::generateIdentifier(d1c_test@mn, &quot;DOI&quot;) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path, newId=doi) newPackageId &lt;- uploadDataPackage(d1c_test, dp, public=TRUE, quiet=FALSE) 9.6 Categorize datasets As a final step we will categorize the dataset you processed. We are trying to categorize datasets so we can have a rough idea of what kinds of datasets we have at the Arctic Data Center. We will grant you access to the google sheet that has all of the categorized datasets We will categorize each dataset into one of the predefined themes (ie. biology, ecology etc.). Definition of the themes can be found in the google sheet Run the following line with your doi and themes as a list. datamgmt::categorize_dataset(&quot;your_doi&quot;, c(&quot;list&quot;, &quot;of&quot;, &quot;themes&quot;), &quot;your name&quot;) 9.7 Congrats! Congratulations on finishing your first ticket! You can head over to the the repository, data-processing to get your ticket processing code reviewed by the team so we can learn from each other! "],["beyond-your-first-ticket.html", "Chapter 10 Beyond your first ticket 10.1 Working with large data packages 10.2 Add physicals to submissions 10.3 Use references 10.4 Annotations 10.5 Streamlining your workflow 10.6 Code Snippets 10.7 Resources for R", " Chapter 10 Beyond your first ticket This section is meant to be read after you have processed a couple of tickets and you are comfortable with the workflow with a relatively simple dataset with 1-2 files and want to expand your skills and workflows further. 10.1 Working with large data packages 10.2 Add physicals to submissions New submissions made through the web editor will not have any physical sections within the otherEntitys. Add them to the EML with the following script: for (i in seq_along(doc$dataset$otherEntity)) { otherEntity &lt;- doc$dataset$otherEntity[[i]] id &lt;- otherEntity$id if (!grepl(&quot;urn-uuid-&quot;, id)) { warning(&quot;otherEntity &quot;, i, &quot; is not a pid&quot;) } else { id &lt;- gsub(&quot;urn-uuid-&quot;, &quot;urn:uuid:&quot;, id) physical &lt;- arcticdatautils::pid_to_eml_physical(mn, id) doc$dataset$otherEntity[[i]]$physical &lt;- physical } } As you can see from code above, we use a for loop here to add physical sections. The for loop is a very useful tool to iterate over a list of elements. With for loop, you can repeat a specific block of code without copying and pasting the code over and over again. When processing datasets in Arctic Data Center, there are many places where for loop can be used, such as publishing a bunch of objects with pids, updating formatID for pkg$data, adding physical section like above code, etc. A loop is composed of two parts: the sequence and the body. The sequence usually generates indices to locate elements and the body contains the code that you want to iterate for each element. Here is an example of adding the same attributeList for all the dataTables in the metadata using for loop. attributes &lt;- read.csv(&#39;attributes.csv&#39;) # attribute table in csv format attributeList &lt;- EML::set_attributes(attributes = attributes) for (i in 1:length(doc$dataset$dataTable)) { # sequence part doc$dataset$dataTable[[i]]$attributeList &lt;- attributeList # body part } 10.3 Use references References are a way to avoid repeating the same information multiple times in the same EML record. There are a few benefits to doing this, including: Making it clear that two things are the same (e.g., the creator is the same person as the contact, two entities have the exact same attributes) Reducing the size on disk of EML records with highly redundant information Faster read/write/validate with the R EML package You may want to use EML references if you have the following scenarios (not exhaustive): One person has multiple roles in the dataset (creator, contact, etc) One or more entities shares all or some attributes 10.3.1 Example with parties Do not use references for creators as it is used for the citation information. The creators will not show up on the top of the dataset if it is a reference. Until this issue is resolved in NCEAS/metacat#926 we will need to keep this in account. It’s very common to see the contact and creator referring to the same person with XML like this: &lt;eml packageId=&quot;my_test_doc&quot; system=&quot;my_system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1 eml.xsd&quot;&gt; &lt;dataset&gt; &lt;creator&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml&gt; So you see those two times Bryce Mecum is referenced there? If you mean to state that Bryce Mecum is the creator and contact for the dataset, this is a good start. But with just a name, there’s some ambiguity as to whether the creator and contact are truly the same person. Using references, we can remove all doubt. doc$dataset$creator[[1]]$id &lt;- &quot;reference_id&quot; doc$dataset$contact &lt;- list(references = &quot;reference_id&quot;) print(doc) &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; &lt;eml:eml xmlns:eml=&quot;eml://ecoinformatics.org/eml-2.1.1&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns:stmml=&quot;http://www.xml-cml.org/schema/stmml-1.1&quot; packageId=&quot;id&quot; system=&quot;system&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1/ eml.xsd&quot;&gt; &lt;dataset&gt; &lt;title&gt;A Minimal Valid EML Dataset&lt;/title&gt; &lt;creator id=&quot;reference_id&quot;&gt; &lt;individualName&gt; &lt;givenName&gt;Bryce&lt;/givenName&gt; &lt;surName&gt;Mecum&lt;/surName&gt; &lt;/individualName&gt; &lt;/creator&gt; &lt;contact&gt; &lt;references&gt;reference_id&lt;/references&gt; &lt;/contact&gt; &lt;/dataset&gt; &lt;/eml:eml&gt; The reference id needs to be unique within the EML record but doesn’t need to have meaning outside of that. 10.3.2 Example with attributes To use references with attributes: Add an attribute list to a data table Add a reference id for that attribute list Use references to add that information into the attributeLists of the other data tables For example, if all the data tables in our data package have the same attributes, we can set the attribute list for the first one, and use references for the rest: doc$dataset$dataTable[[1]]$attributeList &lt;- attribute_list doc$dataset$dataTable[[1]]$attributeList$id &lt;- &quot;shared_attributes&quot; # use any unique name for your id for (i in 2:length(doc$dataset$dataTable)) { doc$dataset$dataTable[[i]]$attributeList &lt;- list(references = &quot;shared_attributes&quot;) # use the id you set above } 10.4 Annotations If there are multiple tables with similar annotations you can try something like this: #go through each dataTable for(i in 1:length(doc$dataset$dataTable)){ #get all the attibute names an &lt;- eml_get_simple(doc$dataset$dataTable[[i]], &quot;attributeName&quot;) #go through the attributes figure out what will match for(a in 1:length(an)){ annotation &lt;- dplyr::case_when( # the attributeName to match ~ valueLabel an[[a]] == &quot;Sample ID&quot; ~ &quot;Identity&quot;, an[[a]] == &quot;Location&quot; ~ &quot;study location name&quot;, an[[a]] == &quot;Latitude&quot; ~ &quot;latitude coordinate&quot;, an[[a]] == &quot;Longitude&quot; ~ &quot;longitude coordinate&quot;, an[[a]] == &quot;Elevation (m)&quot; ~ &quot;elevation&quot;, str_detect(an[[a]], &quot;Depth|depth&quot;) ~ &quot;Depth&quot;) #only run this code when the annotations match if(!is.na(annotation)){ #based on the entity Name create a unique id entity &lt;- str_split(doc$dataset$dataTable[[i]]$entityName, &quot;_&quot;) doc$dataset$dataTable[[i]]$attributeList$attribute[[a]]$id &lt;- paste0(entity[[1]][[1]], &quot;_&quot;, an[[a]]) #add the annotation doc$dataset$dataTable[[i]]$attributeList$attribute[[a]]$annotation &lt;- eml_ecso_annotation(annotation) } } } 10.5 Streamlining your workflow 10.6 Code Snippets Code snippets help with templating portions of code that you will be using regularly. To add your own, go to the toolbar ribbon at the top of your Rstudio screen and select: Tools &gt; Global Options... &gt; Code &gt; Edit Snippets &gt; Add these chunks to the end of the file More info can be found in this blog post by Mara Averick on how to add them: https://maraaverick.rbind.io/2017/09/custom-snippets-in-rstudio-faster-tweet-chunks-for-all/ Usual arcticdatautils ticket workflow snippet ticket library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;add your rm&quot; pkg &lt;- get_package(adc, rm) doc &lt;- EML::read_eml(getObject(adc, pkg\\$metadata)) doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) eml_validate(doc) eml_path &lt;- &quot;eml.xml&quot; write_eml(doc, eml_path) #update &lt;- publish_update(adc, # metadata_pid = pkg\\$metadata, # resource_map_pid = pkg\\$resource_map, # metadata_path = eml_path, # data_pids = pkg\\$data, # public = F) #datamgmt::categorize_dataset(update\\$metadata, c(&quot;theme1&quot;), &quot;Your Name&quot;) The datapack ticket workflow snippet datapack library(dataone) library(datapack) library(digest) library(uuid) library(dataone) library(arcticdatautils) library(EML) d1c &lt;- D1Client(&quot;PROD&quot;, &quot;urn:node:ARCTIC&quot;) packageId &lt;- &quot;id here&quot; dp &lt;- getDataPackage(d1c, identifier=packageId, lazyLoad=TRUE, quiet=FALSE) #get metadata id metadataId &lt;- selectMember(dp, name=&quot;sysmeta@formatId&quot;, value=&quot;https://eml.ecoinformatics.org/eml-2.2.0&quot;) #edit the metadata doc &lt;- read_eml(getObject(d1c@mn, metadataId)) #add the publisher info doc &lt;- eml_add_publisher(doc) doc &lt;- eml_add_entity_system(doc) doc\\$dataset\\$project &lt;- eml_nsf_to_project(&quot;nsf id here&quot;) #check and save the metadata eml_validate(doc) eml_path &lt;- arcticdatautils::title_to_file_name(doc\\$dataset\\$title) write_eml(doc, eml_path) dp &lt;- replaceMember(dp, metadataId, replacement=eml_path) #upload the dataset myAccessRules &lt;- data.frame(subject=&quot;CN=arctic-data-admins,DC=dataone,DC=org&quot;, permission=&quot;changePermission&quot;) packageId &lt;- uploadDataPackage(d1c, dp, public=F, accessRules=myAccessRules, quiet=FALSE) #datamgmt::categorize_dataset(&quot;doi&quot;, c(&quot;theme1&quot;), &quot;Jasmine&quot;) Quick way to give access to submitters to their datasets: snippet access library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) rm &lt;- &quot;rm here&quot; pkg &lt;- get_package(adc, rm) set_access(adc, unlist(pkg), &quot;orcid here&quot;) Quick access to the usual code for common Solr queries: snippet solr library(dataone) library(arcticdatautils) library(EML) cn &lt;- CNode(&#39;PROD&#39;) adc &lt;- getMNode(cn, &#39;urn:node:ARCTIC&#39;) result &lt;- query(adc, list(q = &quot;rightsHolder:*orcid.org/0000-000X-XXXX-XXXX* AND (*:* NOT obsoletedBy:*)&quot;, fl = &quot;identifier,rightsHolder,formatId, fileName, dateUploaded&quot;, sort = &#39;dateUploaded+desc&#39;, start =&quot;0&quot;, rows = &quot;1500&quot;), as=&quot;data.frame&quot;) 10.7 Resources for R 10.7.1 Learning The following online books are useful for expanding your R knowledge and skills: the most recent ADC training materials The cleaning and data manipulation section is useful for working with attribute tables Efficient R Programming In particular Chapter 3 Efficient Programming R for Data Science Section on Strings R Packages contributing to arcticdatatutils, datamgmt and EML Advanced R Object-oriented programming in R for S4 to understand how datapack and dataone packages are written bookdown: Authoring Books and Technical Documents with R Markdown formatting, troubleshooting and updating the training document Others Hands-On Programming with R R Programming for Data Science Exploratory Data Analysis with R Mastering Software Development in R Geocomputation with R R Markdown: The Definitive Guide The Tidyverse Style Guide The RStudio cheatsheets are also useful references for functions in tidyverse and other packages. 10.7.2 Packages The data team uses and develops a number of R packages. Here is a listing and description of the main packages: dataone reading and writing data at DataONE member nodes http://doi.org/10.5063/F1M61H5X datapack creating and managing data packages https://github.com/ropensci/datapack EML creating and editing EML metadata documents https://ropensci.github.io/EML arcticdatautils utility functions for processing data for the Arctic Data Center https://nceas.github.io/arcticdatautils/ datamgmt data management utilities for curating, documenting, and publishing data (sandbox package) https://nceas.github.io/datamgmt/ metadig authoring MetaDIG quality checks https://github.com/NCEAS/metadig-r metajam downloading and reading data and metadata from DataONE member nodes https://nceas.github.io/metajam/ "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]