Set up continuous integration in Forgejo #75
|
|
@ -10,5 +10,13 @@ runs:
|
|||
using: "composite"
|
||||
steps:
|
||||
|
glen marked this conversation as resolved
|
||||
- run: rustup target add wasm32-unknown-unknown
|
||||
# Forgejo Runner seems to modify the HOME variable when it's running a job
|
||||
# in a Docker container, so we have to find the home directory another way,
|
||||
|
Vectornaut marked this conversation as resolved
Outdated
glen
commented
I don't love that we are apparently using some layout detail of the docker image we are currently using (that it has a /home/circleci directory that is somehow relevant). Can we put this stuff in a controlled location, like in our tree, and then invoke cargo and or trunk later in a way that explicitly points to where we put it? I would prefer that if possible. Again, happy for pointers on how that might be done and then I could try to do. I don't love that we are apparently using some layout detail of the docker image we are currently using (that it has a /home/circleci directory that is somehow relevant). Can we put this stuff in a controlled location, like in our tree, and then invoke cargo and or trunk later in a way that explicitly points to where we put it? I would prefer that if possible. Again, happy for pointers on how that might be done and then I could try to do.
Vectornaut
commented
Good point: it would be nice to make the Good point: it would be nice to make the `setup-trunk` action more container-independent. I think I've done this in commit df2d458. There may be a cleaner way to do it, but this is the best I could come up with after about an hour of work.
glen
commented
Interesting. Does cargo not let you specify a place for it to put its stuff? That would be preferable if so. Interesting. Does cargo not let you _specify_ a place for it to put its stuff? That would be preferable if so.
glen
commented
In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained. If you concur, let me make the change so I can make sure the CI running is working for me. But I won't make any changes until we reach consensus. In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained. If you concur, let me make the change so I can make sure the CI running is working for me. But I won't make any changes until we reach consensus.
Vectornaut
commented
Could you clarify what you mean by "self-contained"? Do you mean making the CI workflow as image-independent as possible? In both of the Rust images I've looked at ( Cargo can install things in alternate locations using the > In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained.
Could you clarify what you mean by "self-contained"? Do you mean making the CI workflow as image-independent as possible?
In both of the Rust images I've looked at ([`cimg/rust:1.85-node`](https://hub.docker.com/layers/cimg/rust/1.85.1-node/images/sha256-2c9efb22d74d19d9efa65035922bbe177b153cd9d163843c917f9505a462dffa) and [`rust:1.85-slim-bookworm`](https://hub.docker.com/layers/library/rust/1.85-slim-bookworm/images/sha256-98c3b8e2721a704bc2eb62b98ffde90aa679060beab5c0261fc9fb3e53dcb96c)), Cargo already has the Rust compiler (`rustc`) and other standard tools installed in a pre-determined home location. Setting or changing `CARGO_HOME` to point to a different directory would feel weird to me, although it doesn't actually seem to break our access to the tools that are already installed.
Cargo can install things in alternate locations using the [`--root` option](https://doc.rust-lang.org/cargo/commands/cargo-install.html), but I haven't yet figured out how to do that manually.
glen
commented
To me it seems very strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly. That is what I mean by self-contained: the location of files and the operation of the project, both in ordinary use and execution as well as testing, are as little context-dependent as possible. The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be. I find it bad etiquette at best and potentially a point of failure at worst when a package installs/depends on any file outside its project tree. It's possibly that CARGO_HOME isn't the right parameter to "cargo" to accomplish this ideal -- there are a lot of configuration settings, some that are similar/possibly overlapping -- but it seems to me that "cargo" is configurable enough that we should be able to get there or almost all the way there, and then we don't need to do anything at all to try to figure out what our environment is -- we just put the stuff we need in the place where we know we will need it because of how we are invoking cargo. Is that making sense? Does CARGO_HOME seem like a reasonable approach to this form of "self-containedness"? Feel free to make alternate suggestions, but unless you feel there are aspects you definitely need to experiment with yourself, let me make the commit(s) so that I can give the CI runs some meaningful test drives. As before, I won't commit until we have reached consensus. Thanks! To me it seems _very_ strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly. That is what I mean by self-contained: the location of files and the operation of the project, both in ordinary use and execution as well as testing, are as little context-dependent as possible. The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be. I find it bad etiquette at best and potentially a point of failure at worst when a package installs/depends on any file outside its project tree.
It's possibly that CARGO_HOME isn't the right parameter to "cargo" to accomplish this ideal -- there are a lot of configuration settings, some that are similar/possibly overlapping -- but it seems to me that "cargo" is configurable enough that we should be able to get there or almost all the way there, and then we don't need to do anything at all to try to figure out what our environment is -- we just put the stuff we need in the place where we know we will need it because of how we are invoking cargo.
Is that making sense? Does CARGO_HOME seem like a reasonable approach to this form of "self-containedness"? Feel free to make alternate suggestions, but unless you feel there are aspects you definitely need to experiment with yourself, let me make the commit(s) so that I can give the CI runs some meaningful test drives. As before, I won't commit until we have reached consensus. Thanks!
Vectornaut
commented
Yes, I think the ideal would be to call
There's no guesswork involved here. Cargo's home directory is In summary, we're only relying on one or two assumptions about the environment where the workflow is running:
> The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be.
Yes, I think the ideal would be to call `cargo install trunk`, as we do during [image setup in commit `15375dc`](https://code.studioinfinity.org/StudioInfinity/dyna3/src/commit/15375dc93213c22d9b79fa728ba6f849f102eeba/tools/ci-image/Dockerfile). The manual installation I do in commit df2d458 isn't the way Cargo is intended to be used. I'm only doing it because I couldn't find a way to build Trunk within the memory constraints of the CI host. I considered installing Trunk from a binary with [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall), but building that crate seems to suck up even more RAM than building Trunk.
> To me it seems very strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly.
There's no guesswork involved here. Cargo's [home directory](https://doc.rust-lang.org/cargo/guide/cargo-home.html) is `$CARGO_HOME` if that variable is set, and `$HOME/.cargo/` otherwise. We could put that logic into the workflow, but I don't currently see a need to, since we know that `CARGO_HOME` is unset in the image that we're using; I could add a comment to that effect. The weird workaround I'm doing to effectively evaluate `HOME` is annoying, but I don't think I know Docker well enough to find a better solution.
In summary, we're only relying on one or two assumptions about the environment where the workflow is running:
- Cargo is installed. (This could become version-dependent if Cargo's home directory rules ever change.)
- The `CARGO_HOME` environment variable is unset. (This assumption is optional, as described above.)
glen
commented
@Vectornaut wrote in #75 (comment):
That's exactly what I am saying: it will be cleaner to set CARGO_HOME or otherwise control where cargo looks for these files, to put it inside the project tree. Then no weird workarounds and no dependency on files outside the project tree. Both of those things are significant upside. Is there any downside? I'm not yet understanding where the resistance to configuring cargo to look for the stuff it needs within the project tree is coming from. @Vectornaut wrote in https://code.studioinfinity.org/StudioInfinity/dyna3/pulls/75#issuecomment-2515:
> weird workaround
That's exactly what I am saying: it will be cleaner to set CARGO_HOME or otherwise control where cargo looks for these files, to put it inside the project tree. Then no weird workarounds and no dependency on files outside the project tree. Both of those things are significant upside. Is there any downside? I'm not yet understanding where the resistance to configuring cargo to look for the stuff it needs within the project tree is coming from.
Vectornaut
commented
People in the Rust ecosystem consistently seem to say that Since Trunk is a self-contained binary, we could dump it in some directory listed in the If you really want to install Trunk within the project tree, we could add its location to People in the Rust ecosystem consistently seem to say that `CARGO_HOME` is meant to be set before setting up the Rust toolchain [[1](https://stackoverflow.com/a/62801744)][[2](https://github.com/rust-lang/rustup/issues/3020)][[3](https://github.com/astral-sh/uv/issues/7250)], although the Cargo documentation doesn't make this clear. I've confirmed, in a `cimg/rust:1.85-node` container, that if I change `CARGO_HOME` and then use `cargo install` to [install a tool](https://doc.rust-lang.org/book/ch14-04-installing-binaries.html), the new tool doesn't show up on the binary search path, which still points to the Cargo home directory set at the time of installation.
Since Trunk is a self-contained binary, we could dump it in some directory listed in the `PATH` variable. Then our workflow would only depend on `PATH` being non-empty.
If you really want to install Trunk within the project tree, we could add its location to `PATH`. However, I'm leery about adding a directory within the project tree to `PATH`, because to me that would make the CI environment meaningfully different from a typical development environment.
glen
commented
Thanks for the clarifying exposition. I think it separates three issues for me: (A) Where should this project install tools needed for its development and execution? (And, related, how should we invoke them?) Addressing these in turn: (A) I am strong proponent that projects should only mess with their own project tree. They can ask/expect that prerequisites, like (B) The relevant doc of
and
If CARGO_HOME is meant to be set prior to the installation of Cargo, then I have no problem with using one of the earlier three options. If we can set the 'install.root' Cargo config value by a passive, auto-found configuration file in the repo, that might be the easiest. If that's not how Cargo config values work, it would suggest we should use one of the two earlier options. And since I think command-line arguments are a bit easier to specify portably than environment variables, in that case I'd suggest just specifying the --root option when invoking (C) These suggested principles do now beg the question of whether Since the project does not contain the source for Note that even in (ii), in the case of automated testing if we are not using Hopefully these comments have brought the issues into focus and we can resolve them relatively easily now. Thanks for the clarifying exposition. I think it separates three issues for me:
(A) Where should this project install tools needed for its development and execution? (And, related, how should we invoke them?)
(B) For those cases where it's useful/appropriate to use `cargo` for such installation, how should we convey to it our choice in (A)?
(C) Is `trunk` a _prerequisite_ for installation/development/use of dyna3, or a _tool_ that the project is itself installing?
Addressing these in turn:
(A) I am strong proponent that projects should only mess with their own project tree. They can ask/expect that prerequisites, like `cargo`, be installed in the ambient OS in some place in the PATH. Everything that the project does once the prerequisites are in place should be contained within its project tree. Hence, tools/ and/or bin/, etc., directories in project trees, some of which or some contents of which that are generated by the project from its files in the repo may be `.gitignore`d. I have no problem with routinely invoking a tool via something like `bin/ourtool arg1 arg2`, for example.
(B) The relevant doc of `cargo install` states:
> all executables are installed into the installation root’s bin folder
and
> The installation root is determined, in order of precedence:
> * --root option
> * CARGO_INSTALL_ROOT environment variable
> * install.root Cargo config value
> * CARGO_HOME environment variable
> * $HOME/.cargo
If CARGO_HOME is meant to be set prior to the installation of Cargo, then I have no problem with using one of the earlier three options. If we can set the 'install.root' Cargo config value by a passive, auto-found configuration file in the repo, that might be the easiest. If that's not how Cargo config values work, it would suggest we should use one of the two earlier options. And since I think command-line arguments are a bit easier to specify portably than environment variables, in that case I'd suggest just specifying the --root option when invoking `cargo install`.
(C) These suggested principles do now beg the question of whether `trunk` is (i) a tool "installed by the package" or (ii) should be considered, like `cargo` itself, just a prerequisite in the ambient OS that the person wishing to use the package must set up and have ready in the path prior to beginning work with the package. I am fairly agnostic on this question. I have been operating under the assumption of (i). But if we feel that (ii) is more appropriate, then we can stop worrying about where it is installed: you can put it where you like, I will uninstall it from where it is now and install it someplace I like, and in the testing we can do whatever is easy and clean, e.g. install it in say a bin directory at the top level of the project tree and either add that directory to PATH or invoke it via that explicit location in the test steps that use it.
Since the project does not contain the source for `trunk` or build it itself, I'd say a plausible criterion for (i) vs (ii) for `trunk` would be whether we are going to have (tools from) the package itself monitor the version of trunk that is installed, and make sure that it is a particular "frozen" version before executing it to accomplish the various goals of the project. (To be explicit: monitoring and freezing corresponds to option (i) and otherwise, perhaps we should pick option (ii) if that's more comfortable.)
Note that even in (ii), in the case of automated testing if we are not using `cargo install` to install `trunk` so that it would "Just Work (tm)", I still don't see any reason to either use a "workaround" to find a place in the PATH to install `trunk`, or to install it outside the project tree: we should adhere to the "self-contained" aspect in terms of "keep your hands inside your own project tree" and install it (say) in a bin directory at the top of the project tree and either add that directory to the PATH or just invoke it directly at that location.
Hopefully these comments have brought the issues into focus and we can resolve them relatively easily now.
glen
commented
In meeting, we chose (ii): In meeting, we chose (ii): `trunk` is just a prerequisite. Therefore, Glen will push a commit that in CI, (1) installs trunk in `ci-bin/` at the top level of the projext; (2) in CI, adds `ci-bin/` to the PATH, and (3) adds `ci-bin` to .gitignore.
glen
commented
Hmm, exporting PATH has no effect on any subsequent step, so far as I can see, Looking for other techniques. Hmm, exporting PATH has no effect on any subsequent step, so far as I can see, Looking for other techniques.
Vectornaut
commented
Oh, now I remember how to do this! You have to write to the Oh, now I remember how to do this! You have to write to the `GITHUB_PATH` variable. I did this in the [`setup-rust`](https://code.studioinfinity.org/Vectornaut/dyna3/src/commit/bbfdf2b87f562772aa02b0b90627f14fe60887cc/.forgejo/setup-rust/action.yaml#L10) action from commit bbfdf2b.
Vectornaut
commented
Oops, I see that you just discovered the same thing. Oops, I see that you just discovered the same thing.
glen
commented
Indeed. I now consider this thread done. Also, the CI testing ran just fine for my pushes. Please "resolve conversation" if you agree all is well on this point. Indeed. I now consider this thread done. Also, the CI testing ran just fine for my pushes. Please "resolve conversation" if you agree all is well on this point.
|
||||
# inspired by this StackOverflow answer:
|
||||
#
|
||||
# https://unix.stackexchange.com/a/247582
|
||||
#
|
||||
- id: find-home
|
||||
run: eval echo "home=~$(whoami)" >> $GITHUB_OUTPUT
|
||||
- run: curl --output - --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused --location --silent --show-error --fail 'https://github.com/trunk-rs/trunk/releases/download/v0.21.12/trunk-x86_64-unknown-linux-gnu.tar.gz' | tar --gunzip --extract --file -
|
||||
working-directory: /home/circleci/.cargo/bin
|
||||
working-directory: ${{ steps.find-home.outputs.home }}/.cargo/bin
|
||||
Reviewing this is hampered because I have no idea what
using: "composite"means, and https://forgejo.org/docs/next/user/actions/ sheds no light on this. Can you either explain or point to somewhere that has the information?The documentation of Gitea Actions and Forgejo Actions seems to rely a lot on the principle that these are "similar and mostly compatible to GitHub Actions," despite Forgejo's insistence that "they are not and will never be identical." GitHub's description of composite actions is pretty much what I'd guess from the usage here: