Skip to content

Move asset hashing into a linker intercept step #3988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
568d074
Fix deduping assets
ealmloff Apr 14, 2025
0bac001
hash assets in the linker
ealmloff Apr 14, 2025
2e1292f
fix extracting assets
ealmloff Apr 16, 2025
1d4faf7
Fix the hashed asset name
ealmloff Apr 16, 2025
e3468a6
deserialize the asset back from the link section
ealmloff Apr 16, 2025
1cee44e
prevent the read from being inlined
ealmloff Apr 16, 2025
60a9da0
remove some extra logs
ealmloff Apr 16, 2025
1af064a
Fix liveview routing
ealmloff Apr 25, 2025
489e12d
Only look for direct dependencies on dioxus features
ealmloff Apr 28, 2025
f42a72b
fix wasm split test
ealmloff Apr 28, 2025
bc858bd
On wasm, create a new object file with the asset information
ealmloff Apr 29, 2025
9844b90
Fix multiple assets
ealmloff Apr 30, 2025
d232ead
add a regression test for 3467
ealmloff Apr 30, 2025
59d2ef0
fix image link
ealmloff Apr 30, 2025
dfa4526
Merge remote-tracking branch 'origin/main' into assets-are-a-linker-step
ealmloff Apr 30, 2025
8511e6a
Merge branch 'fix-liveview-ci' into assets-are-a-linker-step
ealmloff May 2, 2025
bd5dd07
fix linker out file
ealmloff May 2, 2025
adcdaa4
fix linux linker args
ealmloff May 2, 2025
abf7b12
add windows linker command
ealmloff May 2, 2025
bad852f
fix linux link args
ealmloff May 2, 2025
d9e5cd5
fix windows target triple
ealmloff May 2, 2025
a974bd2
Expose an option to build assets in the linker intercept
ealmloff May 5, 2025
06d90b9
Fix the standalone optimizer and document it in the dx help command
ealmloff May 5, 2025
584f95c
include the cli opt version in the hash
ealmloff May 6, 2025
9bfcf6d
fix clippy
ealmloff May 6, 2025
1598fb1
fix ssg port
ealmloff May 6, 2025
b37df5c
add a safety note to the read_volatile call
ealmloff May 6, 2025
22fa73a
move found assets log
ealmloff May 7, 2025
86ad4a6
Merge remote-tracking branch 'origin/main' into assets-are-a-linker-step
ealmloff May 7, 2025
9d8459d
Merge remote-tracking branch 'origin/main' into assets-are-a-linker-step
ealmloff May 7, 2025
cbb38f1
fix some merge conflicts
ealmloff May 7, 2025
65d57b7
fix build
ealmloff May 7, 2025
d3e8df6
make most env var for the linker optional
ealmloff May 7, 2025
772eb69
guess the target triple when dx acts as the linker intercept
ealmloff May 8, 2025
28ecc98
automatically find the linker for normal builds
ealmloff May 8, 2025
64ca767
use the right linker in wasm mode
ealmloff May 8, 2025
a6a2369
fix mac linker
ealmloff May 8, 2025
f51984e
look inside archive files (rlibs)
ealmloff May 8, 2025
659e9f4
Merge remote-tracking branch 'origin/main' into assets-are-a-linker-step
ealmloff May 8, 2025
a5f1afa
remove out.txt
ealmloff May 8, 2025
942bb16
fix clippy
ealmloff May 8, 2025
95345a3
fix typo
ealmloff May 8, 2025
9b9d67c
fix clippy
ealmloff May 8, 2025
a5ac200
fix build
ealmloff May 8, 2025
7fb4155
fix dx link name in docs
ealmloff May 8, 2025
77dc4e6
ignore libcompiler_builtins warnings
ealmloff May 8, 2025
acabb40
fix playwright
ealmloff May 8, 2025
b12092e
pass typocheck
ealmloff May 8, 2025
cbe447f
Merge branch 'remote-origin' into assets-are-a-linker-step
ealmloff May 14, 2025
d5ce977
fix printing sysroot
ealmloff May 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 21 additions & 20 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/asset-resolver/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ pub fn serve_asset_from_raw_path(path: &str) -> Result<Response<Vec<u8>>, AssetS
/// - [ ] Linux (rpm)
/// - [ ] Linux (deb)
/// - [ ] Android
#[allow(unused)]
#[allow(unreachable_code)]
fn get_asset_root() -> PathBuf {
let cur_exe = std::env::current_exe().unwrap();

Expand Down
3 changes: 3 additions & 0 deletions packages/cli-opt/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,6 @@ swc_parallel = { version = "=1.0.1", default-features = false }
swc_timer = { version = "=1.0.0", default-features = false }
swc_visit = { version = "=2.0.0", default-features = false }
browserslist-rs = { version = "=0.16.0" }

[build-dependencies]
built = { version = "0.7.5", features = ["git2"] }
3 changes: 3 additions & 0 deletions packages/cli-opt/build.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
fn main() {
built::write_built_file().expect("Failed to acquire build-time information");
}
10 changes: 10 additions & 0 deletions packages/cli-opt/src/build_info.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// The file has been placed there by the build script.
include!(concat!(env!("OUT_DIR"), "/built.rs"));

pub(crate) fn version() -> String {
format!(
"{} ({})",
PKG_VERSION,
GIT_COMMIT_HASH_SHORT.unwrap_or("was built without git repository")
)
}
38 changes: 32 additions & 6 deletions packages/cli-opt/src/css.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use std::path::Path;
use std::{hash::Hasher, path::Path};

use anyhow::{anyhow, Context};
use codemap::SpanLoc;
Expand Down Expand Up @@ -146,12 +146,11 @@ pub(crate) fn minify_css(css: &str) -> anyhow::Result<String> {
Ok(res.code)
}

/// Process an scss/sass file into css.
pub(crate) fn process_scss(
/// Compile scss with grass
pub(crate) fn compile_scss(
scss_options: &CssAssetOptions,
source: &Path,
output_path: &Path,
) -> anyhow::Result<()> {
) -> anyhow::Result<String> {
let style = match scss_options.minified() {
true => OutputStyle::Compressed,
false => OutputStyle::Expanded,
Expand All @@ -162,7 +161,18 @@ pub(crate) fn process_scss(
.quiet(false)
.logger(&ScssLogger {});

let css = grass::from_path(source, &options)?;
let css = grass::from_path(source, &options)
.with_context(|| format!("Failed to compile scss file: {}", source.display()))?;
Ok(css)
}

/// Process an scss/sass file into css.
pub(crate) fn process_scss(
scss_options: &CssAssetOptions,
source: &Path,
output_path: &Path,
) -> anyhow::Result<()> {
let css = compile_scss(scss_options, source)?;
let minified = minify_css(&css)?;

std::fs::write(output_path, minified).with_context(|| {
Expand Down Expand Up @@ -199,3 +209,19 @@ impl grass::Logger for ScssLogger {
);
}
}

/// Hash the inputs to the scss file
pub(crate) fn hash_scss(
scss_options: &CssAssetOptions,
source: &Path,
hasher: &mut impl Hasher,
) -> anyhow::Result<()> {
// Grass doesn't expose the ast for us to traverse the imports in the file. Instead of parsing scss ourselves
// we just hash the expanded version of the file for now
let css = compile_scss(scss_options, source)?;

// Hash the compiled css
hasher.write(css.as_bytes());

Ok(())
}
118 changes: 75 additions & 43 deletions packages/cli-opt/src/file.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
use anyhow::Context;
use manganis::{CssModuleAssetOptions, FolderAssetOptions};
use manganis_core::{AssetOptions, CssAssetOptions, ImageAssetOptions, JsAssetOptions};
use std::path::Path;

Expand Down Expand Up @@ -33,7 +34,7 @@ pub(crate) fn process_file_to_with_options(
}
if let Some(parent) = output_path.parent() {
if !parent.exists() {
std::fs::create_dir_all(parent)?;
std::fs::create_dir_all(parent).context("Failed to create directory")?;
}
}

Expand All @@ -47,63 +48,94 @@ pub(crate) fn process_file_to_with_options(
.unwrap_or_default()
.to_string_lossy()
));
let resolved_options = resolve_asset_options(source, options);

match options {
AssetOptions::Unknown => match source.extension().map(|e| e.to_string_lossy()).as_deref() {
Some("css") => {
process_css(&CssAssetOptions::new(), source, &temp_path)?;
}
Some("scss" | "sass") => {
process_scss(&CssAssetOptions::new(), source, &temp_path)?;
}
Some("js") => {
process_js(&JsAssetOptions::new(), source, &temp_path, !in_folder)?;
}
Some("json") => {
process_json(source, &temp_path)?;
}
Some("jpg" | "jpeg" | "png" | "webp" | "avif") => {
process_image(&ImageAssetOptions::new(), source, &temp_path)?;
}
Some(_) | None => {
if source.is_dir() {
process_folder(source, &temp_path)?;
} else {
let source_file = std::fs::File::open(source)?;
let mut reader = std::io::BufReader::new(source_file);
let output_file = std::fs::File::create(&temp_path)?;
let mut writer = std::io::BufWriter::new(output_file);
std::io::copy(&mut reader, &mut writer).with_context(|| {
format!(
"Failed to write file to output location: {}",
temp_path.display()
)
})?;
}
}
},
AssetOptions::Css(options) => {
match &resolved_options {
ResolvedAssetType::Css(options) => {
process_css(options, source, &temp_path)?;
}
AssetOptions::CssModule(options) => {
ResolvedAssetType::CssModule(options) => {
process_css_module(options, source, output_path, &temp_path)?;
}
AssetOptions::Js(options) => {
ResolvedAssetType::Scss(options) => {
process_scss(options, source, &temp_path)?;
}
ResolvedAssetType::Js(options) => {
process_js(options, source, &temp_path, !in_folder)?;
}
AssetOptions::Image(options) => {
ResolvedAssetType::Image(options) => {
process_image(options, source, &temp_path)?;
}
AssetOptions::Folder(_) => {
ResolvedAssetType::Json => {
process_json(source, &temp_path)?;
}
ResolvedAssetType::Folder(_) => {
process_folder(source, &temp_path)?;
}
_ => {
tracing::warn!("Unknown asset options: {:?}", options);
ResolvedAssetType::File => {
let source_file = std::fs::File::open(source)?;
let mut reader = std::io::BufReader::new(source_file);
let output_file = std::fs::File::create(&temp_path)?;
let mut writer = std::io::BufWriter::new(output_file);
std::io::copy(&mut reader, &mut writer).with_context(|| {
format!(
"Failed to write file to output location: {}",
temp_path.display()
)
})?;
}
}

// If everything was successful, rename the temp file to the final output path
std::fs::rename(temp_path, output_path)?;
std::fs::rename(temp_path, output_path).context("Failed to rename output file")?;

Ok(())
}

pub(crate) enum ResolvedAssetType {
/// An image asset
Image(ImageAssetOptions),
/// A css asset
Css(CssAssetOptions),
/// A css module asset
CssModule(CssModuleAssetOptions),
/// A SCSS asset
Scss(CssAssetOptions),
/// A javascript asset
Js(JsAssetOptions),
/// A json asset
Json,
/// A folder asset
Folder(FolderAssetOptions),
/// A generic file
File,
}

pub(crate) fn resolve_asset_options(source: &Path, options: &AssetOptions) -> ResolvedAssetType {
match options {
AssetOptions::Image(image) => ResolvedAssetType::Image(*image),
AssetOptions::Css(css) => ResolvedAssetType::Css(*css),
AssetOptions::CssModule(css) => ResolvedAssetType::CssModule(*css),
AssetOptions::Js(js) => ResolvedAssetType::Js(*js),
AssetOptions::Folder(folder) => ResolvedAssetType::Folder(*folder),
AssetOptions::Unknown => resolve_unknown_asset_options(source),
_ => {
tracing::warn!("Unknown asset options... you may need to update the Dioxus CLI. Defaulting to a generic file: {:?}", options);
resolve_unknown_asset_options(source)
}
}
}

fn resolve_unknown_asset_options(source: &Path) -> ResolvedAssetType {
match source.extension().map(|e| e.to_string_lossy()).as_deref() {
Some("scss" | "sass") => ResolvedAssetType::Scss(CssAssetOptions::new()),
Some("css") => ResolvedAssetType::Css(CssAssetOptions::new()),
Some("js") => ResolvedAssetType::Js(JsAssetOptions::new()),
Some("json") => ResolvedAssetType::Json,
Some("jpg" | "jpeg" | "png" | "webp" | "avif") => {
ResolvedAssetType::Image(ImageAssetOptions::new())
}
_ if source.is_dir() => ResolvedAssetType::Folder(FolderAssetOptions::new()),
_ => ResolvedAssetType::File,
}
}
Loading
Loading