dataprep_jack_1st.Rmd

---
title: "project"
author: "jack"
date: "20/07/2021"
output: html_document
---
data from ALA https://doi.org/10.26197/ala.3bf1b394-8646-4e3e-99d9-0ef7a5258d35 with filters for: spatially suspect records, records with additional spatial quality issues, records with unresolved user annotations, absence records, records based on scientific name quality issues, duplicate records, records based on location uncertainty, records that are environmental outliers, environmental DNA and fossil specimen records and records that are pre 1700.


```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```


```{r setup}
library (galah)
library (stringr)
library (rgdal)
library(sf)
library(maps)
library(mapdata)
library(dplyr)
library(tidyr)
library(ggplot2)

```

data from ALA for Eucalypts

```{r pressure, echo=FALSE}
library(readr)
records.2021.07.21 <- read.csv("~/Documents/R/bigdata/project/records-2021-07-21/records-2021-07-21.csv")
View(records.2021.07.21)
```


```{r}
#plot data on a map to get initial view
library(ggplot2)
wm <- borders("world", colour="slategray1", fill="slategray1")
ggplot()+ coord_fixed()+ wm +
  geom_point(data = records.2021.07.21, aes(x = decimalLongitude, y = decimalLatitude),
             colour = "green", size = 0.5)+
  theme_bw()
```



```{r} 
#now clean with coordinate cleaner 
library(countrycode)
library(CoordinateCleaner)
library(rgbif)
library(sp)
library(dplyr)
```

```{r}
#columns of interest
records_filter_1 <- records.2021.07.21 %>%
  dplyr::select(species, decimalLongitude, decimalLatitude, countryCode, individualCount,
         family, taxonRank, coordinateUncertaintyInMeters, year,
         basisOfRecord, institutionCode, datasetName)

```

```{r}
#filter records to only Australia
records_filter_au <- records.2021.07.21 %>%
  filter(countryCode == "AU")
```

```{r}
#plot data on a map to get  view
library(ggplot2)
wm <- borders("world", colour="slategray1", fill="slategray1")
ggplot()+ coord_fixed()+ wm +
  geom_point(data = records_filter_au , aes(x = decimalLongitude, y = decimalLatitude),
             colour = "green", size = 0.5)+
  theme_bw()
```
#still some cleaning to do ^ but many points from North America and Asia are removed

```{r}
# get rid of records without coordinates as ALA did not get rid of all records without coordinates
records_filter_au_1  <- records_filter_au %>%
  filter(!is.na(decimalLongitude))%>%
  filter(!is.na(decimalLatitude))
```
#5334 records were removed ^


```{r}
#automatic cleaning algorithm of CoordinateCleaner
library(rnaturalearth)
#conversion of country code from ISO2c to ISO3c
records_filter_au_1$countryCode <-  countrycode(records_filter_au_1$countryCode, origin =  'iso2c', destination = 'iso3c')


#flag problems (won't include 'country' test as it flags all records)
records_filter_au_1 <- data.frame(records_filter_au_1)
flags <- clean_coordinates(x = records_filter_au_1 ,
                           lon = "decimalLongitude",
                           lat = "decimalLatitude",
                           countries = "countryCode",
                           species = "species",
                          tests = c("capitals", "centroids", "equal","gbif", "institutions",
                                    "zeros" )) 
summary(flags)
plot(flags, lon = "decimalLongitude", lat = "decimalLatitude")
```
#^  Flagged 1709 of 754717 records, but there is still cleaning to do 

```{r}
#now remove problematic records
dat_cl_aus <- records_filter_au_1[flags$.summary,]

#The flagged records
dat_fl_aus <- records_filter_au_1[!flags$.summary,]
```



```{r}
#plot flagged records

wm <- borders("world", colour="slategray1", fill="slategray1")
ggplot()+ coord_fixed()+ wm +
  geom_point(data = dat_fl_aus , aes(x = decimalLongitude, y = decimalLatitude),
             colour = "green", size = 0.5)+
  theme_bw()
```
#flagged records ^ like the one in the middle of the ocean above New Zealand are now removed 

```{r}
#Remove records with low coordinate precision 
#remove all records with a precision below 100 km
hist(dat_cl_aus$coordinateUncertaintyInMeters / 1000, breaks = 20)

dat_cl_aus <- dat_cl_aus %>%
  filter(coordinateUncertaintyInMeters / 1000 <= 100 | is.na(coordinateUncertaintyInMeters))

#Remove unsuitable data sources, 
table(dat_cl_aus$basisOfRecord)



dat_cl_aus <- filter(dat_cl_aus, basisOfRecord == "HUMAN_OBSERVATION" |
                         basisOfRecord == "OBSERVATION" |
                         basisOfRecord == "PRESERVED_SPECIMEN")
```
```{r}
#remove records with suspicious individual counts
table(dat_cl_aus$individualCount)
```

```{r} 
#remove records of absence i.e (individual count = 0) 
dat_cl_aus <- dat_cl_aus%>%
  filter(individualCount > 0 | is.na(individualCount))%>%
  filter(individualCount < 99 | is.na(individualCount)) 
```

```{r}
#exclude based on fixed longitude and latitude
#i.e remove all points above Cape York, the most North point in Australia 
dat_cl_aus <- filter(dat_cl_aus, decimalLatitude < 10.689167)
```

```{r}
#plot flagged records

wm <- borders("world", colour="slategray1", fill="slategray1")
ggplot()+ coord_fixed()+ wm +
  geom_point(data = dat_cl_aus, aes(x = decimalLongitude, y = decimalLatitude),
             colour = "green", size = 0.5)+
  theme_bw()
```
# ^ now the records in China, Japan and the Philippines that were likely incorrectly labeled are removed 

```{r}
#check for Temporal outliers
flags <- cf_age(x =  dat_cl_aus,
                lon = "decimalLongitude",
                lat = "decimalLatitude",
                taxon = "species", 
                min_age = "year", 
                max_age = "year", 
                value = "flagged")

dat_cl_aus[!flags, "year"]
dat_cl_aus<- dat_cl_aus[flags, ]
```

```{r}
write.csv(dat_cl_aus,"/Users/jackhanigan/Documents/R/bigdata/BEES3041_BCI\\File Name.csv", row.names = FALSE)

```

#now import these records into Biodiverse analysis software (http://www.biodiverse.unsw.edu.au), with analyses for phylogenetic endemism and phylogenetic diversity conducted in the software.