Skip to content

s3 prefix #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
camorosi merged 6 commits into from
Apr 4, 2026
Merged

s3 prefix #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
573b36b
Add prefix option for s3 boards
camorosi Jan 9, 2026
48aaa56
Bump version
camorosi Jan 16, 2026
cb5afa1
Fix prefix bug where NA was converted to character
camorosi Jan 20, 2026
ce3c6d5
Try to fix qs deprecation issue
camorosi Feb 5, 2026
23f8981
Remove qs
camorosi Feb 17, 2026
acaf499
Merge pull request #43 from amashadihossein/42-s3-prefix
camorosi Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: dpdeploy
Title: A package to manage deployment of data products
Version: 0.4.0
Version: 0.5.0
Authors@R: c(
person(given = "Afshin", family = "Mashadi-Hossein", email ="amashadihossein@gmail.com", role = c("aut", "cre")),
person(given = "Yue", family = "Jiang", email = "rivehill@gmail.com", role = "aut")
Expand Down Expand Up @@ -28,13 +28,12 @@ Imports:
yaml
Suggests:
covr,
qs,
roxygen2,
testthat (>= 3.0.0)
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.2
RoxygenNote: 7.3.3
Remotes:
github::camorosi/pinsLabkey
Config/testthat/edition: 3
4 changes: 4 additions & 0 deletions NEWS.md
View file
Open in desktop

@leslem leslem Apr 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add a comment that support for qs was removed?

@camorosi camorosi Apr 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit: will add to release notes

Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# dpdeploy 0.5.0

* Add s3 "prefix" option so that a subfolder can be used for s3 daaps

# dpdeploy 0.4.0

* Enable saving data objects in qs format via the "type" param in `dpinput_sync()`. Note that `dp_deploy()` now detects type automatically.
Expand Down
30 changes: 15 additions & 15 deletions R/dp_deploy.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,24 +45,24 @@ detect_type = function(project_path){
#' @description read the output data object, for now only rds and qs are supported
#' @noRd
object_read <- function(project_path, type){
type = rlang::arg_match0(type, c('rds', 'qs'))
type = rlang::arg_match0(type, c('rds')) #, 'qs'))
switch(type,
rds = readRDS(file = glue::glue("{project_path}/output_files/RDS_format/data_object.RDS")),
qs = read_qs(project_path)
rds = readRDS(file = glue::glue("{project_path}/output_files/RDS_format/data_object.RDS")) #,
# qs = read_qs(project_path)
)
}

#' @title Reas qs object
#' @description Read in qs object
#' @noRd
read_qs <- function(path){
rlang::check_installed("qs")
dataobj_path <- glue::glue(
"{path}/",
"output_files/qs_format/data_object.qs"
)
qs::qread(dataobj_path)
}
# #' @title Reas qs object
# #' @description Read in qs object
# #' @noRd
# read_qs <- function(path){
# rlang::check_installed("qs")
# dataobj_path <- glue::glue(
# "{path}/",
# "output_files/qs_format/data_object.qs"
# )
# qs::qread(dataobj_path)
# }


#' The dp_deploy is a wrapper around this.
Expand Down Expand Up @@ -99,7 +99,7 @@ dp_deployCore.s3_board <- function(conf, project_path, d, dlog, git_info, type,
}

board <- pins::board_s3(
prefix = "daap/",
prefix = paste0(ifelse(is.na(conf$board_params$prefix), "", conf$board_params$prefix), "daap/"),
bucket = conf$board_params$bucket_name,
region = conf$board_params$region,
access_key = aws_creds$key,
Expand Down
2 changes: 1 addition & 1 deletion R/dpinput_sync.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ init_board.s3_board <- function(conf) {
}

pins::board_s3(
prefix = "dpinput/",
prefix = paste0(ifelse(is.na(conf$board_params$prefix), "", conf$board_params$prefix), "dpinput/"),
bucket = conf$board_params$bucket_name,
region = conf$board_params$region,
access_key = aws_creds$key,
Expand Down
16 changes: 8 additions & 8 deletions tests/testthat/test-dp_deploy.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ test_that("properly checks valid repository ", {

test_that("object_read properly detects type", {

project_path = withr::local_tempfile()
path = file.path(project_path, "output_files/qs_format/")
dir.create(path, recursive = TRUE)
qs::qsave(structure(list(), class = "dp"), file = file.path(path, "data_object.qs"))
expect_equal(
detect_type(project_path),
'qs'
)
# project_path = withr::local_tempfile()
# path = file.path(project_path, "output_files/qs_format/")
# dir.create(path, recursive = TRUE)
# qs::qsave(structure(list(), class = "dp"), file = file.path(path, "data_object.qs"))
# expect_equal(
# detect_type(project_path),
# 'qs'
# )

project_path = withr::local_tempfile()
path = file.path(project_path, "output_files/RDS_format/")
Expand Down
Loading