-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] Automate preparation of release binaries for publishing (via artifacts) #12181
Closed
Closed
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| function help() { | ||
| echo "Downloads binaries created by a CircleCI job to the current working," | ||
| echo "directory and renames them according to the naming convention used" | ||
| echo "on the Solidity release page on Github." | ||
| echo | ||
| echo "WARNING: The binaries will be overwritten if they already exist." | ||
| echo | ||
| echo | ||
| echo "Usage:" | ||
| echo " ./$(basename "$0") --help" | ||
| echo " ./$(basename "$0") workflow_id" | ||
| echo | ||
| echo " workflow_id CircleCI workflow ID. Identifies the workflow that contains" | ||
| echo " jobs whose artifacts should be downloaded." | ||
| echo | ||
| echo | ||
| echo "Examples:" | ||
| echo " ./$(basename "$0") --help" | ||
| echo " ./$(basename "$0") 12345678-90ab-cdef-1234-567890abcdef" | ||
| } | ||
|
|
||
|
|
||
| # GENERAL UTILITIES | ||
|
|
||
| query_api() { | ||
| local api_endpoint="$1" | ||
|
|
||
| curl --fail --silent --show-error "$api_endpoint" | ||
| } | ||
|
|
||
| fail() { | ||
| local format="$1" | ||
|
|
||
| # shellcheck disable=SC2059 | ||
| >&2 printf "ERROR: $format\n" "${@:2}" | ||
| exit 1 | ||
| } | ||
|
|
||
| expect_single_line() { | ||
| local text="$1" | ||
|
|
||
| local line_count; line_count="$(echo "$text" | grep --count "")" | ||
| [[ $text != "" ]] || fail "Expected one line, got zero." | ||
| (( "$line_count" < 2 )) || fail "Expected one line, got %d:\n%s" "$line_count" "$text" | ||
| } | ||
|
|
||
|
|
||
| # JSON FILTERING | ||
|
|
||
| filter_artifacts_by_name() { | ||
| local artifact_name="$1" | ||
|
|
||
| jq '[ .items[] | select (.path == "'"$artifact_name"'") ]' | ||
| } | ||
|
|
||
| filter_jobs_by_name() { | ||
| local job_name="$1" | ||
|
|
||
| jq '[ .items[] | select (.name == "'"$job_name"'") ]' | ||
| } | ||
|
|
||
|
|
||
| # ENUMERATIONS | ||
|
|
||
| target_to_job_name() { | ||
| local target="$1" | ||
|
|
||
| case "$target" in | ||
| linux-amd64) echo b_ubu_static ;; | ||
| macosx-amd64) echo b_osx ;; | ||
| windows-amd64) echo b_win_release ;; | ||
| emscripten) echo b_ems ;; | ||
| *) fail "Invalid target: %s" "$target" ;; | ||
| esac | ||
| } | ||
|
|
||
| target_to_circleci_artifact_name() { | ||
| local target="$1" | ||
|
|
||
| case "$target" in | ||
| linux-amd64) echo solc ;; | ||
| macosx-amd64) echo solc ;; | ||
| windows-amd64) echo upload/bin/solc.exe ;; | ||
| emscripten) echo soljson.js ;; | ||
| *) fail "Invalid target: %s" "$target" ;; | ||
| esac | ||
| } | ||
|
|
||
| is_executable() { | ||
| local target="$1" | ||
|
|
||
| case "$target" in | ||
| linux-amd64) true ;; | ||
| macosx-amd64) true ;; | ||
| windows-amd64) false ;; | ||
| emscripten) false ;; | ||
| *) fail "Invalid target: %s" "$target" ;; | ||
| esac | ||
| } | ||
|
|
||
| target_to_binary_name() { | ||
| local target="$1" | ||
|
|
||
| case "$target" in | ||
| linux-amd64) echo solc-static-linux ;; | ||
| macosx-amd64) echo solc-macos ;; | ||
| windows-amd64) echo solc-windows.exe ;; | ||
| emscripten) echo soljson.js ;; | ||
| *) fail "Invalid target: %s" "$target" ;; | ||
| esac | ||
| } | ||
|
|
||
|
|
||
| # MAIN LOGIC | ||
|
|
||
| query_circleci_artifact_url() { | ||
| local target="$1" | ||
| local build_num="$2" | ||
|
|
||
| local artifact_name; artifact_name="$(target_to_circleci_artifact_name "$target")" | ||
| local artifact_info; artifact_info="$( | ||
| query_api "https://circleci.com/api/v2/project/gh/ethereum/solidity/${build_num}/artifacts" | | ||
| filter_artifacts_by_name "$artifact_name" | ||
| )" | ||
|
|
||
| [[ $artifact_info != "[]" ]] || fail "Artifact '%s' is missing." "$artifact_name" | ||
|
|
||
| local artifact_url; artifact_url="$(echo "$artifact_info" | jq --raw-output '.[].url')" | ||
| expect_single_line "$artifact_url" | ||
|
|
||
| echo "$artifact_url" | ||
| } | ||
|
|
||
| download_binary() { | ||
| local target="$1" | ||
| local download_url="$2" | ||
|
|
||
| local target_path; target_path="$(target_to_binary_name "$target")" | ||
|
|
||
| # If the target exists we ovewrite it. As a special case, if it's a symlink, remove it | ||
| # so that we only change link not the file it links to. | ||
| [[ ! -L "$target_path" ]] || rm "$target_path" | ||
|
|
||
| echo "Downloading release binary from ${download_url} into ${target_path}" | ||
| curl "$download_url" --output "$target_path" --location --no-progress-meter --create-dirs | ||
|
|
||
| ! is_executable "$target" || chmod +x "$target_path" | ||
| } | ||
|
|
||
| download_binary_from_circleci() { | ||
| local target="$1" | ||
| local workflow_job_info="$2" | ||
|
|
||
| local job_name; job_name="$(target_to_job_name "$target")" | ||
| local job_info; job_info="$( | ||
| echo "$workflow_job_info" | | ||
| filter_jobs_by_name "$job_name" | ||
| )" | ||
| [[ $job_info != "[]" ]] || fail "Job '%s' not found." "$job_name" | ||
|
|
||
| local build_num; build_num="$(echo "$job_info" | jq --raw-output '.[].job_number')" | ||
| expect_single_line "$build_num" | ||
|
|
||
| local artifact_url; artifact_url="$(query_circleci_artifact_url "$target" "$build_num")" | ||
| download_binary "$target" "$artifact_url" | ||
| } | ||
|
|
||
| download_binaries() { | ||
| local workflow_id="$1" | ||
|
|
||
| echo "===> DOWNLOADING BINARIES FOR WORKFLOW ${workflow_id} FROM CIRCLECI" | ||
|
|
||
| local workflow_job_info; workflow_job_info="$( | ||
| query_api "https://circleci.com/api/v2/workflow/${workflow_id}/job" | ||
| )" | ||
|
|
||
| local release_targets=( | ||
| linux-amd64 | ||
| windows-amd64 | ||
| macosx-amd64 | ||
| emscripten | ||
| ) | ||
|
|
||
| for target in "${release_targets[@]}"; do | ||
| download_binary_from_circleci "$target" "$workflow_job_info" | ||
| done | ||
| } | ||
|
|
||
| main() { | ||
| if (( $# > 0 )) && [[ $1 == --help ]]; then | ||
| help | ||
| exit 0 | ||
| fi | ||
|
|
||
| (( $# >= 1 )) || fail "Too few arguments" | ||
| (( $# <= 1 )) || fail "Too many arguments" | ||
|
|
||
| local workflow_id="${1:-$PWD}" | ||
|
|
||
| download_binaries "$workflow_id" | ||
| } | ||
|
|
||
| main "$@" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| if (( $# > 0 )) && [[ $1 == --help ]]; then | ||
| echo "Renames binaries following the naming convention from Solidity release page on Github to match" | ||
| echo "the naming convention used in solc-bin." | ||
| echo "Assumes that the binaries are present in the current working directory." | ||
| echo "Obtains version from the --version output of the Linux binary." | ||
| echo | ||
| echo "WARNING: The binaries will be overwritten if they already exist." | ||
| echo | ||
| echo | ||
| echo "Usage:" | ||
| echo " ./$(basename "$0") --help" | ||
| echo " ./$(basename "$0") [solc_bin_dir]" | ||
| echo | ||
| echo " solc_bin_dir Location of the solc-bin directory." | ||
| echo " Default: current working directory." | ||
| echo | ||
| echo | ||
| echo "Examples:" | ||
| echo " ./$(basename "$0") --help" | ||
| echo " ./$(basename "$0") ~/solc-bin/" | ||
|
|
||
| exit 0 | ||
| fi | ||
|
|
||
| (( $# <= 1 )) || { >&2 printf "ERROR: Too many arguments"; exit 1; } | ||
|
|
||
| solc_bin_dir="${1:-$PWD}" | ||
|
|
||
| full_version=$( | ||
| ./solc-static-linux --version | | ||
| sed -En 's/^Version: ([0-9.]+.*\+commit\.[0-9a-f]+(\.mod)?).*$/\1/p' | ||
| ) | ||
|
|
||
| target=linux-amd64 | ||
| mkdir -p "${solc_bin_dir}/${target}" | ||
| mv solc-static-linux "${solc_bin_dir}/${target}/solc-${target}-${full_version}" | ||
|
|
||
| target=macosx-amd64 | ||
| mkdir -p "${solc_bin_dir}/${target}" | ||
| mv solc-macos "${solc_bin_dir}/${target}/solc-${target}-${full_version}" | ||
|
|
||
| target=windows-amd64 | ||
| mkdir -p "${solc_bin_dir}/${target}" | ||
| mv solc-windows.exe "${solc_bin_dir}/${target}/solc-${target}-${full_version}.exe" | ||
|
|
||
| target=bin | ||
| mkdir -p "${solc_bin_dir}/${target}" | ||
| mv soljson.js "${solc_bin_dir}/${target}/soljson-${full_version}.js" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't it be easier, cleaner and safer to rename the binaries in the respective builds already and attach them to a common workspace similar to what we do for the bytecode run?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not necessarily easier but might be doable. The biggest issue is that I need to get all binaries in the workspace but under different paths. You get an error if a job depends on multiple jobs adding a workspace file under the same path. Doing it via artifacts does not have this problem.
When I started this PR I actually did not understand our workspace layout all that well. It's often mounted at different locations in similar jobs and not in a consistent way. It was really confusing to me and I tried to clean that up in #12450. Unfortunately this resulted in a conflict between binaries on Linux and macOS now that the paths are consistent.
But after working on it I think that the current gnarly layout would work without a conflict. So yeah, I'm going to try to redo it like that.
Another option I was thinking of was rewriting the Bash script here in Python. In #12818 I already have a Python module with helpers for CircleCI and Github API so this would replace the big script in this PR with something much smaller. But if the workspace solution works, maybe it's better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I know the problem with path conflicts in merging workspaces... but I thought since we have the same problem with the bytecode comparison runs, it may just work or at least not be too hard to achieve, but not entirely sure...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you must be right.
t_bytecode_comparedoes attach the workspace so it should work. It's just that I did not realize that at first and thought that trying to do it via workspace would be very messy and require renaming binaries. I only see now that it shouldn't :)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should really clean up the workspace layout though. Once we do this, we will have a conflict. My current idea to sidestep that is to name each binary after the platform and use symlinks in #12450.