change .Rmd to .qmd throughout

.github/workflows/quarto-publish.yml

@@ -0,0 +1,37 @@
+name: quarto-publish
+on:
+  push:
+    branches: dev
+jobs:
+  build-deploy:
+    runs-on: windows-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@dev # checkout 'dev' branch
+
+#      - name: Setup pandoc
+#        uses: r-lib/actions/setup-pandoc@v2
+
+      - name: Setup R
+        uses: r-lib/actions/setup-r@v2
+        with:
+          r-version: '4.2.2'
+
+      - name: Install Dependencies
+        run: |
+          Rscript -e "install.packages('quarto')"
+
+      - name: Set up Quarto
+        uses: quarto-dev/quarto-actions/setup@v2
+        with:
+          # To install LaTeX to build PDF book
+          tinytex: true
+          # uncomment below and fill to pin a version
+          version: 1.3.353
+
+      - name: Publish to GitHub Pages (and render)
+        uses: quarto-dev/quarto-actions/publish@v2
+        with:
+          target: gh-pages
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # this secret is always available for github actions

.gitignore

@@ -2,3 +2,5 @@
 .Rhistory
 .RData
 .Ruserdata
+
+/.quarto/

Folder_structure.Rmd → Folder_structure.qmd

@@ -22,11 +22,9 @@ This was the original PIP_Data folder. It was used for testing and development o
 
 ## PIP-Data_QA
 
-This is the main output directory for the `{pipdp}` and `{pipaux}` packages. It contains all the survey and auxiliary data 
-needed to run the [Poverty Calculator](#pcpipeline) and [Table Maker](#tmpipeline) pipelines. 
+This is the main output directory for the `{pipdp}` and `{pipaux}` packages. It contains all the survey and auxiliary data needed to run the [Poverty Calculator](#pcpipeline) and [Table Maker](#tmpipeline) pipelines. 
 
-Note that the contents of this folder is for QA and production purposes. Please use the `PIP_Data_Testing` directory, 
-or your own personal testing directory, if you intend to test code changes in `{pipdp}` or `{pipaux}`. 
+Note that the contents of this folder is for QA and production purposes. Please use the `PIP_Data_Testing` directory, or your own personal testing directory, if you intend to test code changes in `{pipdp}` or `{pipaux}`. 
 
 ### _aux
 
@@ -61,27 +59,20 @@ An explanation of each subfolder is given below.
 |sna         | Special National Account cases | pipaux       | Manual |
 |weo         | World Economic Outlook (GDP)   | pipaux       | IMF, pipaux |
 
-The data in the folders `countries`, `regions`, `pl`, `cp` and `indicators` are loaded into the Poverty Calculator pipeline, 
-but that they are not used for any calculations or modified in any way, when being parsed through the pipeline.
-They are thus only listed with `{pipapi}` as their usage. 
+The data in the folders `countries`, `regions`, `pl`, `cp` and `indicators` are loaded into the Poverty Calculator pipeline, but that they are not used for any calculations or modified in any way, when being parsed through the pipeline. They are thus only listed with `{pipapi}` as their usage. 
 
-In contrast the measures CPI, GDP, PCE, POP and PPP are both used in the pre-calculations and transformed before 
-being saved as pipeline outputs. So even though these measures are also available in the PIP PC API, the files at this stage 
-only have the Poverty Calculator pipeline as their use case. 
+In contrast the measures CPI, GDP, PCE, POP and PPP are both used in the pre-calculations and transformed before being saved as pipeline outputs. So even though these measures are also available in the PIP PC API, the files at this stage only have the Poverty Calculator pipeline as their use case. 
 
 ### _inventory 
 
-The `_inventory` folder contains the PIP inventory file created by `pipload::pip_update_inventory()`. It is important to update this file if the survey data 
-has been updated. 
+The `_inventory` folder contains the PIP inventory file created by `pipload::pip_update_inventory()`. It is important to update this file if the survey data has been updated. 
 
 ### Country data
 
-The folders with country codes as names, e.g `AGO`, contain survey data for each country. This is created by `{pipdp}`.
-The folder structure within each country folder follows roughly the same convention as used by DLW. 
+The folders with country codes as names, e.g `AGO`, contain survey data for each country. This is created by `{pipdp}`. The folder structure within each country folder follows roughly the same convention as used by DLW. 
 
 There will be one data file for each survey, based on DLW's `GMD` module and the grouped data in the PovcalNet-drive. These are labelled with PC in 
-their filenames. Additionally, if there is survey data available from DLW's `ALL` module there will also be a dataset for the Table Maker, 
-labelled with TB. 
+their filenames. Additionally, if there is survey data available from DLW's `ALL` module there will also be a dataset for the Table Maker, labelled with TB. 
 
 ## PIP-Data_ExtSOL
 
@@ -105,7 +96,7 @@ Please do not touch or change any content in this folder without permission
 
 ## PIP_Data_Testing 
 
-This folder is used as a testing directory for development of the `{pipdp}` and `{pipaux}` packges. 
+This folder is used as a testing directory for development of the `{pipdp}` and `{pipaux}` packages. 
 
 ## PIP_Data_Vintage 
 
@@ -126,29 +117,22 @@ This is the main output directory for the both the [Poverty Calculator](#pcpipel
 
 #### _targets
 
-This folder contains the objects that are cached by `{targets}` when the [Poverty Calculator Pipeline](#pcpipeline) is run. It is located on the 
-shared network drive so that the latest cached objects are available to all team members. Please note however that a shared `_targets` folder
-does entail some risks. **It is very important that only one person runs the pipeline at a time.** Concurrent runs against the same `_targets` store will
-wreak havoc. And needless to say; if you are developing new features or customising the pipeline use a different `_targets` store. 
+This folder contains the objects that are cached by `{targets}` when the [Poverty Calculator Pipeline](#pcpipeline) is run. It is located on the shared network drive so that the latest cached objects are available to all team members. Please note however that a shared `_targets` folder does entail some risks. **It is very important that only one person runs the pipeline at a time.** Concurrent runs against the same `_targets` store will wreak havoc. And needless to say; if you are developing new features or customising the pipeline use a different `_targets` store. 
 ([We should probably have a custom DEV or testing folder for this, similar to PIP_Data_Testing]{style="color:red"}.) 
 
 #### cache
 
-The survey data output of `{pipdp}` cannot be directly used as input to the [Poverty Calculator Pipeline](#pcpipeline). This is because the data needs to be cleaned 
-before running the necessary pre-calculations for the PIP database. The cleaning currently includes removal of rows with missing welfare and weight values,
-standardising of grouped data, and restructuring of datasets with both income and consumption based welfare values in the same survey. 
+The survey data output of `{pipdp}` cannot be directly used as input to the [Poverty Calculator Pipeline](#pcpipeline). This is because the data needs to be cleaned  before running the necessary pre-calculations for the PIP database. The cleaning currently includes removal of rows with missing welfare and weight values, standardising of grouped data, and restructuring of datasets with both income and consumption based welfare values in the same survey. 
 Please refer to `{wbpip}` and `{pipdm}` for the latest cleaning procedures.  
 
 In order to avoid doing this cleaning every time the pipeline runs an intermediate version of the survey data is stored in the `cache/clean_survey_data/` folder. 
 Please make sure this folder is up-to-date before running the `_targets.R` pipeline. 
 
-[Note: This is not exactly true. We need to solve the issue with the double caching of survey data. But the essence of what we want to accomplish is to avoid 
-cleaning and checking the survey data in every pipeline run.]{style="color:red"}.
+[Note: This is not exactly true. We need to solve the issue with the double caching of survey data. But the essence of what we want to accomplish is to avoid cleaning and checking the survey data in every pipeline run.]{style="color:red"}.
 
 #### output 
 
-This folder contains the outputs of the [Poverty Calculator Pipeline](#pcpipeline). This is seperated into three subfolders; `aux` containing the
-auxiliary data, `estimations` containing the survey, interpolated and distributional pre-calculated estimates, and `survey_data` containing
+This folder contains the outputs of the [Poverty Calculator Pipeline](#pcpipeline). This is seperated into three subfolders; `aux` containing the auxiliary data, `estimations` containing the survey, interpolated and distributional pre-calculated estimates, and `survey_data` containing
 the cleaned survey data. 
 
 #### validation

PRIMUS.Rmd → PRIMUS.qmd

@@ -48,15 +48,18 @@ the instruction section \@ref(stata-github). Basically, you need to install
 first the [github](https://github.com/haghish/github) Stata command by [E. F.
 Haghish](https://github.com/haghish)
 
-```{stata, eval = FALSE}
-net install github, from("https://haghish.github.io/github/")
-```
+
+<!-- ```{stata, eval = FALSE} -->
+<!--  net install github, from("https://haghish.github.io/github/") -->
+<!-- ``` -->
+
 
 Now, you can install `primus` by just typing the following in Stata
 
-```{stata, eval = FALSE}
-github install worldbank/primus
-```
+<!-- ```{stata, eval = FALSE} -->
+<!-- github install worldbank/primus -->
+<!-- ``` -->
+
 
 In case this does not work, follow instructions in section \@ref(stata-github)
 for alternative methods.
@@ -92,40 +95,41 @@ DECRG, or the Poverty GP) has not approved it in the system. You can click on
 the check box *PENDING* in the PRIMUS website to see which surveys have such a
 status, or you can use the `primus` command list this,
 
-```{stata, eval= FALSE}
-qui primus query, overallstatus(PENDING)
-list transaction_id country year date_modified in 1/`=min(10, _N)'
-     +----------------------------------------------+
-     |              transaction_id   country   year |
-     |----------------------------------------------|
-  1. | TRN-000327173-EAP-IDN-QR48Q       IDN   2017 |
-  2. | TRN-000327173-ECA-DEU-YJYVZ       DEU   1995 |
-  3. | TRN-000327173-ECA-DEU-2P4DR       DEU   2002 |
-  4. | TRN-000327173-ECA-DEU-LJN8R       DEU   2003 |
-  5. | TRN-000327173-ECA-DEU-ZSN9J       DEU   2005 |
-     |----------------------------------------------|
-  6. | TRN-000327173-ECA-DEU-UBS7M       DEU   2008 |
-  7. | TRN-000327173-ECA-DEU-41TOU       DEU   2009 |
-  8. | TRN-000327173-EAP-AUS-KKZ2E       AUS   2004 |
-     +----------------------------------------------+
-```
+
+<!-- ```{stata, eval= FALSE} -->
+<!-- qui primus query, overallstatus(PENDING) -->
+<!-- list transaction_id country year date_modified in 1/`=min(10, _N)' -->
+<!--      +----------------------------------------------+ -->
+<!--      |              transaction_id   country   year | -->
+<!--      |----------------------------------------------| -->
+<!--   1. | TRN-000327173-EAP-IDN-QR48Q       IDN   2017 | -->
+<!--   2. | TRN-000327173-ECA-DEU-YJYVZ       DEU   1995 | -->
+<!--   3. | TRN-000327173-ECA-DEU-2P4DR       DEU   2002 | -->
+<!--   4. | TRN-000327173-ECA-DEU-LJN8R       DEU   2003 | -->
+<!--   5. | TRN-000327173-ECA-DEU-ZSN9J       DEU   2005 | -->
+<!--      |----------------------------------------------| -->
+<!--   6. | TRN-000327173-ECA-DEU-UBS7M       DEU   2008 | -->
+<!--   7. | TRN-000327173-ECA-DEU-41TOU       DEU   2009 | -->
+<!--   8. | TRN-000327173-EAP-AUS-KKZ2E       AUS   2004 | -->
+<!--      +----------------------------------------------+ -->
+<!-- ``` -->
 
 Notice that the overall status of a transaction is independent from survey ID.
 Thus, it is possible to find several transactions for the same country and year.
 Indonesia 2017, for instance, has three transactions, two of them rejected and
-one of them pending.
-
-```{stata, eval = FALSE}
-qui primus query, country(IDN) year(2017)
-list transaction_id overall_status date_modified in 1/`=min(10, _N)'
-     +--------------------------------------------------+
-     |              transaction_id        date_modified |
-     |--------------------------------------------------|
-  1. | TRN-000104674-EAP-IDN-8R9IF   23may2018 15:28:47 |
-  2. | TRN-000327173-EAP-IDN-TYA1A   23may2018 23:57:27 |
-  3. | TRN-000327173-EAP-IDN-QR48Q   24may2018 00:27:33 |
-     +--------------------------------------------------+
-```
+<!-- one of them pending. -->
+
+<!-- ```{stata, eval = FALSE} -->
+<!-- qui primus query, country(IDN) year(2017) -->
+<!-- list transaction_id overall_status date_modified in 1/`=min(10, _N)' -->
+<!--      +--------------------------------------------------+ -->
+<!--      |              transaction_id        date_modified | -->
+<!--      |--------------------------------------------------| -->
+<!--   1. | TRN-000104674-EAP-IDN-8R9IF   23may2018 15:28:47 | -->
+<!--   2. | TRN-000327173-EAP-IDN-TYA1A   23may2018 23:57:27 | -->
+<!--   3. | TRN-000327173-EAP-IDN-QR48Q   24may2018 00:27:33 | -->
+<!--      +--------------------------------------------------+ -->
+<!-- ``` -->
 
 A transaction is *rejected* when at least one of the three parties rejected the
 database. Finally, a transaction is *approved* only when all three parties have
@@ -149,9 +153,9 @@ the estimates of pending transactions for us and properly stores them in the
 folder `p:\01.PovcalNet\03.QA\02.PRIMUS\pending\` `r emo::ji("tada")`
 `r emo::ji("tada")` . You only need to type,
 
-```{stata, eval = FALSE}
-pcn primus pending, down(estimates)
-```
+<!-- ```{stata, eval = FALSE} -->
+<!-- pcn primus pending, down(estimates) -->
+<!-- ``` -->
 
 In addition, `pcn` checks the date for which you're downloading the estimates
 and keeps only those transactions that have been uploaded for the next spring or
@@ -167,9 +171,9 @@ July. Thus, it is likely that you will find and error saying
 
 You can load the recently-downloaded estimates by typing,
 
-```{stata, eval = FALSE}
-pcn primus pending, load(estimates)
-```
+<!-- ```{stata, eval = FALSE} -->
+<!-- pcn primus pending, load(estimates) -->
+<!-- ``` -->
 
 Now, you have to check whether the new estimates make sense. Once way to that is
 to follow this do-file,
@@ -192,65 +196,65 @@ the responsibility to approve on behalf or two of them, DECDG and DECRG. This
 process can easily done with the code below, which can be found in this file,
 `p:\01.PovcalNet\03.QA\02.PRIMUS\pending\2020_SM\approve\primus_approve.do`.
 
-```{stata, eval = FALSE}
-/*==================================================
-0: Program set up
-==================================================*/
-version 14
-drop _all
-
-*---------- Modify this
-local excl = "BRA SOM SSD" // countries to exclude 
-local excl = "" // countries to exclude 
-
-/*==================================================
-Load data
-==================================================*/
-
-primus query, overalls(pending)
-//------------Cut off date
-local filtdate = "2019-12-01" // filter date (december last year)
-local filtdate = "2020-02-18" // filter date (surveys uploaded by Minh)
-keep if  date_modified >= clock("`filtdate'", "YMD")
-
-//------------Select username
-if (lower("`c(username)'") == "wb424681") {
-  local dep = "povcalnet"
-}
-else if (lower("`c(username)'") == "wb384996") {
-  local dep = "decdg"
-}
-else {
-  noi disp in red "you don't have rights to run this code"
-  break
-}
-
-tab `dep'
-keep if `dep' == "PENDING"
-
-if ("`excl'" != "") {
-  local excl: subinstr local excl " " "|", all
-  drop if regexm("`country'", "`excl'") 
-}
-
-/*=================================================
-Approve (Do NOT modify)
-==================================================*/
-
-local n = _N
-preserve 
-qui foreach i of numlist 1/`n' {
-  restore, preserve
-  local country = country[`i']
-  local year    = year[`i']
-  local id      = transaction_id[`i']
-  
-  noi disp in y "primus action, tranxid(`id') decision(approved)"
-  cap noi primus action, tranxid(`id') decision(approved)
-  if (_rc) noi disp "problem with `id'"
-}
-
-```
+<!-- ```{stata, eval = FALSE} -->
+<!-- /*================================================== -->
+<!-- 0: Program set up -->
+<!-- ==================================================*/ -->
+<!-- version 14 -->
+<!-- drop _all -->
+
+<!-- *---------- Modify this -->
+<!-- local excl = "BRA SOM SSD" // countries to exclude -->
+<!-- local excl = "" // countries to exclude -->
+
+<!-- /*================================================== -->
+<!-- Load data -->
+<!-- ==================================================*/ -->
+
+<!-- primus query, overalls(pending) -->
+<!-- //------------Cut off date -->
+<!-- local filtdate = "2019-12-01" // filter date (december last year) -->
+<!-- local filtdate = "2020-02-18" // filter date (surveys uploaded by Minh) -->
+<!-- keep if  date_modified >= clock("`filtdate'", "YMD") -->
+
+<!-- //------------Select username -->
+<!-- if (lower("`c(username)'") == "wb424681") { -->
+<!--   local dep = "povcalnet" -->
+<!-- } -->
+<!-- else if (lower("`c(username)'") == "wb384996") { -->
+<!--   local dep = "decdg" -->
+<!-- } -->
+<!-- else { -->
+<!--   noi disp in red "you don't have rights to run this code" -->
+<!--   break -->
+<!-- } -->
+
+<!-- tab `dep' -->
+<!-- keep if `dep' == "PENDING" -->
+
+<!-- if ("`excl'" != "") { -->
+<!--   local excl: subinstr local excl " " "|", all -->
+<!--   drop if regexm("`country'", "`excl'") -->
+<!-- } -->
+
+<!-- /*================================================= -->
+<!-- Approve (Do NOT modify) -->
+<!-- ==================================================*/ -->
+
+<!-- local n = _N -->
+<!-- preserve -->
+<!-- qui foreach i of numlist 1/`n' { -->
+<!--   restore, preserve -->
+<!--   local country = country[`i'] -->
+<!--   local year    = year[`i'] -->
+<!--   local id      = transaction_id[`i'] -->
+
+<!--   noi disp in y "primus action, tranxid(`id') decision(approved)" -->
+<!--   cap noi primus action, tranxid(`id') decision(approved) -->
+<!--   if (_rc) noi disp "problem with `id'" -->
+<!-- } -->
+
+<!-- ``` -->
 
 Basically, this is what you need to do with this file.