URISA_2023.qmd

---
title: "RURAL AREAS ARE OVERREPRESENTED IN UNSERVED AREAS AND UNDERESTIMATED IN STATISTICS"
subtitle: "Extending Broadband to Underserved Areas"
author: "Olivier Leroy & Drew Rosebush"
date: "October 18 2023"
institute: "Center On Rural Innovation"
fontsize: 20pt
format: 
  revealjs: 
    theme: custom_cori.scss
    logo: img/CORI_Logo.png
    css: logo.css
editor: visual
title-slide-attributes:
  data-background-image: img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg
  data-background-size: contain
  data-background-opacity: "0.5"
---

## Our slides are online: {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

Slides realized in [Quarto](https://quarto.org/) hosted in [GitHub](https://github.com/ruralinnovation/conf_URISA_2023)

Data stack: SQL ([PostgreSQL](https://www.postgresql.org/)), R with [DBI](https://dbi.r-dbi.org/),  [targets](https://docs.ropensci.org/targets/), [ggplot2](https://ggplot2.tidyverse.org/) and "in house" packages

![](img/prez_qr.png){height="300" fig-align="center"}

[https://ruralinnovation.github.io/conf_URISA_2023/#/title-slide](https://ruralinnovation.github.io/conf_URISA_2023/#/title-slide)

## Who are we? {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: columns
::: {.column width="50%"}

![](img/dad_fall.png){height="400"}

Olivier 🇨🇵 Senior Data Engineer
:::

::: {.column width="50%"}

![](img/dad_waterfall.png){height="400"}

Drew 🇺🇸 Lead Data Scientist
:::
:::

::: notes
@drew 
:::

## Center On Rural Innovation (CORI) {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

*Advancing economic prosperity in rural America through the creation of inclusive tech economy ecosystem that support scalable entrepreneurship and tech job creation*

<p style="text-align:right;">
Learn more at: <https://ruralinnovation.us/>
</p>

![](img/cori_triangle_w_note.png)


::: notes
Where does Broadband stand in CORI goals Here the focus is Broadband is an ingredient of tech ecosystem

[FCC staff estimates](https://www.fcc.gov/staff-block-estimates):

Why we like it: 

-   Dataset provided for **both** 2010 and 2020 US census boundaries

-   FCC provides their methodology and codes

-   Same institution that NBM and F477

-   Provides estimates for housing units  

:::

## Defining Rural {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: columns
::: {.column width="50%"}
"Rural" refers to the "**nonmetro plus**" definition: includes all nonmetro counties and all tracts classified as RUCA 4 or higher.

USDA’s Rural-Urban Community Area (RUCA) codes categorize rural areas by population density, urbanization, and daily commuting.

More nuanced rural definition, classifying rural places along a continuum.

Combining definitions: better encapsulates places that are rural in character.

:::

::: {.column width="50%"}

![](img/different_rural_defs.svg){fig-align="center"}
For more information see [Defining rural America: The consequences of how we count](https://ruralinnovation.us/blog/defining-rural-america/).

:::
:::



::: notes
@drew 
:::


## Historical and new data context {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

-   [Federal Communications Commission](https://www.fcc.gov/): FCC

-   [Form 477](https://www.fcc.gov/economics-analytics/industry-analysis-division/form-477-resources): F477

-   [National Broadband Map](https://broadbandmap.fcc.gov/home): NBM

-   **Unserved**: less than 25/3 Mbps downloads/uploads

|                      | F477             | NBM                                                                                                                                                                                                                                  |
|-------------------------|------------------------|------------------------|
| US Census Boundaries | 2010             | 2020                                                                                                                                                                                                                                 |
| Type of recording    | self declarative | self declarative                                                                                                                                                                                                                     |
| Granularity          | Census block     | [Locations](https://broadbandmap.fcc.gov/location-summary/fixed?version=dec2022&location_id=1350199199&addr1=390+MORGAN+ST&addr2=OBERLIN%2C+OH+44074&zoom=15.00&vlon=-82.231744&vlat=41.287448&br=r&speed=25_3&tech=1_2_3_4_5_6_7_8) |
| Services             | Mobile/**Fixed** | Mobile/**Fixed**  |
| Releases | 2014-2021 | 2022 - *Ongoing* |

F477: "at least one" rule

NBM's [BEAD](https://broadbandusa.ntia.doc.gov/sites/default/files/2022-05/BEAD%20NOFO.pdf) definitions for **Unserved area**: *an area in which not less than 80 percent of broadband-serviceable locations are unserved locations.*

::: notes
some BB defs here should we add an iframe of some FCC NBM localisation?

@drew is it 2014?

at least one rule: if one ISP said he was present with 25/3 in a census block then all location in that block was "Served"
:::


## What's on the table {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: columns
::: {.column width="50%"}
**Broadband Equity Access and Deployment (BEAD)** Program provides \$42.45B to expand high-speed internet access by planning, infrastructure deployment, and adoption programs

**Minimum BEAD allocations** range from \$25M (US territories) to \$100M (states and Puerto Rico)

**Remaining BEAD allocations** vary across states depending on the number and concentration of unserved locations
:::

::: {.column width="50%"}
![](img/BEAD.png)
:::
:::

::: aside
Source: [PEW](https://www.pewtrusts.org/en/research-and-analysis/issue-briefs/2023/01/what-states-need-to-know-about-federal-bead-funding-for-high-speed-internet-expansion) / [State specific funding](https://broadbandusa.ntia.doc.gov/public-notice-posting-state-and-territory-bead-and-digital-equity-plansproposals)
:::

::: notes
drew
:::


## BEAD Priorities {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: columns
::: {.column width="40%"}
**Coverage**

Primary goal is to extend broadband internet service (≥ 25Mbps/3Mbps) to all

**Technology**

Prioritizes projects designed to provide fiber connectivity directly to the end user

**Affordability**

Prioritizes projects that aim at improving affordability of service
:::

::: {.column width="60%"}

```{=html}
<iframe width="780" height="500" src="https://broadbandmap.fcc.gov/home" title="FCC National Broadband Map"></iframe>
```

:::
:::

::: aside
Source: [BEAD NOFO](https://broadbandusa.ntia.doc.gov/sites/default/files/2022-05/BEAD%20NOFO.pdf)
:::

## Workflow: From FCC to our products {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

```{mermaid}
%%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '20px', 'fontFamily': 'Bitter'}}}%%
flowchart LR
    A1[FCC Website] --> B1[Server]
    A2[TIGER Census block] --> C2
    subgraph Ingestion
    direction TB
    B1--> B2[Data Wrangling]
    B2--> B3[Populate source DB]
    end
    subgraph Transform
    direction TB
    C1[Remove Sat.]--> C2[Count Services]
    C2 --> C3[Blocks DB]
    end
    subgraph Serving
    direction TB
    D1[Prod. Server] --> D2[Analytic Tables]
    end
    Ingestion --> Transform
    C3 --> D1
    D1--> E1[Tools: <Br> Public CH <BR> Internal BCAT]
    D2--> E1
    D2 --> E4[Data team <BR> Broadband team]
    D2 --> E5[URISA]
    click E1 "https://broadband-risk.ruralinnovation.us/" _parent
    click E5 "https://ruralinnovation.github.io/conf_URISA_2023/#/workflow-from-fcc-to-our-products" _parent
    style Ingestion fill:#fff
    style Transform fill:#fff
    style Serving fill:#fff
```

-   Two releases per year (but 7 versions of Dec. 2022 releases)

-   440 files for 11GB zipped (*Fixed*)

-   This presentation uses [FCC staff estimates](https://www.fcc.gov/staff-block-estimates)

::: aside
Tools used: Shell (Bash) with GNU tools (make, csvkit, sed) and GDAL / SQL (PostgreSQL/PostGIS) / A bit of R (ease some processes)
:::


## Results: At the US level {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: panel-tabset
### Not Reported - Unserved

![](img/imp_data_quality_bd_services.png)

### Fiber access - 100/20 MBps

![](img/imp_data_quality_gd_services.png)
:::

::: notes
@olivier
:::

## Rural areas Overrepresented in poor quality broadband! {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: panel-tabset
### NBM

![](img/rural_vs_non_rual_nbm.png)

### F477

![](img/rural_vs_non_rual_f477.png)
:::

::: notes
@olivier Observation about f477 -\> NBM in US is similar in rural but "amplifed"

Rural has a worth bb access: \* more Not Reported / Unserved. \* less coverage, especialy fiber.
:::

## Rural Areas underestimated in statistics {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

![](img/thresold.png)

Picking a 50% threshold can help close the gap!

## Conclusions {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

:::: {.columns}

::: {.column width="60%"}
![](img/rural_nonrural_broadband_gap.png){fig-align="left"}

While nonrural access to broadband has exceeded 94% since 2014, rural broadband access stood at just 65% in 2014 and only reached 92% in 2020.
:::

::: {.column width="40%"}

::: {.incremental}
- Overall great improvement in data quality and better understanding on what is happening on the ground (is it enough?)

- Rural areas are **still** overrepresented in poor quality broadband

- Rural areas are systematically underestimated by data definitions
:::

:::

::::


::: notes
@drew even if we use the tabulate 2010 2020 we will face some issues: where are the locations in % covered of x 2020 or y 2020 block
:::

## Next steps {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

What are we measuring? An improvement on coverage, an improvement on data quality or both?


::: {.incremental}
1. Getting better at counting people (use Housing Units, Microsoft building footprint, ACS)

2. Getting better at estimating locations (optimize the cost of building broadband infrastructure)

  * Individual can provides ground level recording from their location ([challenges](https://www.fcc.gov/sites/default/files/bdc-challenge-overview.pdf))

  * Fabric data
  
  * Upcoming FCC updates (API?)
  
  
3. Sharing our results: Providing a reproducible way for others to explore FCC data at scale 

:::

## Contacts {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

::: columns
::: {.column width="50%"}
**Olivier Leroy**

📧: [olivier.leroy@ruralinnovations](olivier.leroy@ruralinnovations) 

[LinkedIn](https://www.linkedin.com/in/olivier-leroy-r/)

:::

::: {.column width="50%"}
**Drew Rosebush**

📧: [drew.rosebush@ruralinnovation.us](drew.rosebush@ruralinnovation.us)

[LinkedIn](https://www.linkedin.com/in/drewrosebush/?trk=people-guest_people_search-card)

:::

::: columns

:::
:::

Website: [https://ruralinnovation.us](https://ruralinnovation.us)


[LinkedIn](https://www.linkedin.com/company/11099658) | [Twitter](https://twitter.com/Ruralinno) | [Facebook](https://www.facebook.com/RuralInno/) | [Instagram](https://www.instagram.com/ruralinno/) | [YouTube](https://www.youtube.com/channel/UCyqbtSrv4Nad9vVv4dQptcA) 


![](img/prez_qr.png){height="100" fig-align="center"}

## Check out our BEAD Climate Tool {background-image="img/URISA_GISPro2023_SpeakerPPTTemplate.pptx.svg" background-size="auto"}

![](img/bb_climate_risk_mitigation_tool_QR.png){height="200" fig-align="center"}

<p style="text-align:center;">
[https://broadband-risk.ruralinnovation.us/#about](https://broadband-risk.ruralinnovation.us/#about)
</p>