Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: running near_workspaces::compile_project concurrently in tests #266

Merged
merged 6 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions cargo-near-build/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ git2 = { version = "0.19", optional = true }
home = { version = "0.5.9", optional = true }
pathdiff = { version = "0.2.1", features = ["camino"], optional = true }
unix_path = { version = "1.0.1", optional = true }
tempfile = { version = "3.10.1", optional = true }
tempfile = { version = "3.10.1" }
shell-words = { version = "1.0.0", optional = true }
wasm-opt = "=0.116.1"
humantime = "2.1.0"
Expand All @@ -57,7 +57,6 @@ docker = [
"dep:home",
"dep:pathdiff",
"dep:unix_path",
"dep:tempfile",
"dep:nix",
"dep:shell-words",
]
Expand Down
34 changes: 32 additions & 2 deletions cargo-near-build/src/fs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,45 @@ use eyre::WrapErr;
/// Copy a file to a destination.
///
/// Does nothing if the destination is the same as the source to avoid truncating the file.
pub fn copy(from: &Utf8Path, to: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
let out_path = to.join(from.file_name().unwrap());
pub fn copy(from: &Utf8Path, out_dir: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
if !out_dir.is_dir() {
return Err(eyre::eyre!("`{}` should be a directory", out_dir));
}
let out_path = out_dir.join(from.file_name().unwrap());
tracing::debug!("Copying file `{}` -> `{}`", from, out_path,);
if from != out_path {
std::fs::copy(from, &out_path)
.wrap_err_with(|| format!("failed to copy `{}` to `{}`", from, out_path))?;
}
Ok(out_path)
}

/// Copy a file to a file destination.
///
/// Does nothing if the destination is the same as the source to avoid truncating the file.
pub fn copy_to_file(from: &Utf8Path, to: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
tracing::debug!("Copying file `{}` -> `{}`", from, to,);

if from != to && to.is_file() {
let from_content = std::fs::read(from)?;
let to_content = std::fs::read(to)?;

if from_content == to_content {
tracing::debug!(
"skipped copying file `{}` -> `{}` on identical contents",
from,
to,
);
return Ok(to.to_path_buf());
}
}
if from != to {
std::fs::copy(from, to)
.wrap_err_with(|| format!("failed to copy `{}` to `{}`", from, to))?;
}
Ok(to.to_path_buf())
}

/// Create the directory if it doesn't exist, and return the absolute path to it.
pub fn force_canonicalize_dir(dir: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
std::fs::create_dir_all(dir)
Expand Down
56 changes: 43 additions & 13 deletions cargo-near-build/src/near/build/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ use crate::types::near::build::buildtime_env;
use camino::Utf8PathBuf;
use colored::Colorize;
use near_abi::BuildInfo;
use tempfile::NamedTempFile;

use crate::types::near::build::output::CompilationArtifact;
use crate::types::near::build::side_effects::ArtifactMessages;
Expand Down Expand Up @@ -162,19 +163,14 @@ pub fn run(args: Opts) -> eyre::Result<CompilationArtifact> {
color,
)?;

wasm_artifact.path = crate::fs::copy(&wasm_artifact.path, &out_dir)?;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sometimes the final destination file is written with non-wasm-opt-ed content,
which might be the reason of failures in concurrent context when many tests run
near_workspaces::compile_project

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was able to easily reproduce this on this commit dj8yfo/near-sdk-rs@027ce75

    receipts: [
        ExecutionOutcome {
            transaction_hash: 61H3aajosM8kDV3jbpuzFaAnMrTCb4DL871gVbbocNag,
            block_hash: HVzmo5HTPZJySPhG5CXMPvde5kMTAGigWKgREbttsoBH,
            logs: [],
            receipt_ids: [
                Dm63UmQr9mqrF6mDiRUxHTaWo8ESKbPHr1p9taEzSzaf,
            ],
            gas_burnt: NearGas {
                inner: 888148937360,
            },
            tokens_burnt: NearToken {
                inner: 88814893736000000000,
            },
            executor_id: AccountId(
                "dev-20241216183040-18506921751143",
            ),
            status: Failure(ActionError(ActionError { index: Some(0), kind: FunctionCallError(CompilationError(PrepareError(Deserialization))) })),
        },
    ],
    status: Failure(ActionError(ActionError { index: Some(0), kind: FunctionCallError(CompilationError(PrepareError(Deserialization))) })),
}
thread 'test_storage_deposit_refunds_excessive_deposit' panicked at tests/workspaces.rs:86:5:
assertion failed: res.is_success()


failures:
    simulate_transfer_call_when_called_contract_not_registered_with_ft
    simulate_transfer_call_with_promise_and_refund
    test_close_account_empty_balance
    test_storage_deposit_refunds_excessive_deposit

test result: FAILED. 9 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.43s

Copy link
Collaborator Author

@dj8yfo dj8yfo Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding near/near-sdk-rs@ffab607 with patch, pointing to current pr, on top of previous change (near/near-sdk-rs#1277) resolved flaky failures

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


if !args.no_wasmopt {
println!();
pretty_print::handle_step(
"Running an optimize for size post-step with wasm-opt...",
|| {
wasm_opt::OptimizationOptions::new_optimize_for_size()
.run(&wasm_artifact.path, &wasm_artifact.path)?;
Ok(())
},
)?;
}
wasm_artifact.path = {
let (from_path, _maybe_tmpfile) =
maybe_wasm_opt_step(&wasm_artifact.path, args.no_wasmopt)?;
crate::fs::copy_to_file(
&from_path,
&out_dir.join(wasm_artifact.path.file_name().expect("has filename")),
)?
};

wasm_artifact.builder_version_info = Some(builder_version_info);

Expand Down Expand Up @@ -206,3 +202,37 @@ pub fn run(args: Opts) -> eyre::Result<CompilationArtifact> {
pretty_print::duration(start, "cargo near build");
Ok(wasm_artifact)
}

fn maybe_wasm_opt_step(
input_path: &Utf8PathBuf,
no_wasmopt: bool,
) -> eyre::Result<(Utf8PathBuf, Option<NamedTempFile>)> {
let result = if !no_wasmopt {
let opt_destination = tempfile::Builder::new()
.prefix("optimized-")
.suffix(".wasm")
.tempfile()?;
println!();
pretty_print::handle_step(
"Running an optimize for size post-step with wasm-opt...",
|| {
println!(
"{} -> {}",
format!("{}", input_path).cyan(),
format!("{}", opt_destination.path().to_string_lossy()).cyan()
);
wasm_opt::OptimizationOptions::new_optimize_for_size()
.run(input_path, opt_destination.path())?;
Ok(())
},
)?;

(
Utf8PathBuf::try_from(opt_destination.path().to_path_buf())?,
Some(opt_destination),
)
} else {
(input_path.clone(), None)
};
Ok(result)
}
Loading