Set up continuous integration in Forgejo #75
14
.forgejo/setup-trunk/action.yaml
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# set up the Trunk web build system
|
||||
#
|
||||
# https://trunkrs.dev
|
||||
#
|
||||
# the `curl` call is based on David Tolnay's `rust-toolchain` action
|
||||
#
|
||||
# https://github.com/dtolnay/rust-toolchain
|
||||
#
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
|
glen marked this conversation as resolved
|
||||
- run: rustup target add wasm32-unknown-unknown
|
||||
- run: curl --output - --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused --location --silent --show-error --fail 'https://github.com/trunk-rs/trunk/releases/download/v0.21.12/trunk-x86_64-unknown-linux-gnu.tar.gz' | tar --gunzip --extract --file -
|
||||
working-directory: /usr/local/cargo/bin
|
||||
|
Vectornaut marked this conversation as resolved
Outdated
glen
commented
I don't love that we are apparently using some layout detail of the docker image we are currently using (that it has a /home/circleci directory that is somehow relevant). Can we put this stuff in a controlled location, like in our tree, and then invoke cargo and or trunk later in a way that explicitly points to where we put it? I would prefer that if possible. Again, happy for pointers on how that might be done and then I could try to do. I don't love that we are apparently using some layout detail of the docker image we are currently using (that it has a /home/circleci directory that is somehow relevant). Can we put this stuff in a controlled location, like in our tree, and then invoke cargo and or trunk later in a way that explicitly points to where we put it? I would prefer that if possible. Again, happy for pointers on how that might be done and then I could try to do.
Vectornaut
commented
Good point: it would be nice to make the Good point: it would be nice to make the `setup-trunk` action more container-independent. I think I've done this in commit df2d458. There may be a cleaner way to do it, but this is the best I could come up with after about an hour of work.
glen
commented
Interesting. Does cargo not let you specify a place for it to put its stuff? That would be preferable if so. Interesting. Does cargo not let you _specify_ a place for it to put its stuff? That would be preferable if so.
glen
commented
In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained. If you concur, let me make the change so I can make sure the CI running is working for me. But I won't make any changes until we reach consensus. In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained. If you concur, let me make the change so I can make sure the CI running is working for me. But I won't make any changes until we reach consensus.
Vectornaut
commented
Could you clarify what you mean by "self-contained"? Do you mean making the CI workflow as image-independent as possible? In both of the Rust images I've looked at ( Cargo can install things in alternate locations using the > In fact, there seems to be a CARGO_HOME environment variable. I'd rather set that to someplace in our directory tree to keep this self-contained.
Could you clarify what you mean by "self-contained"? Do you mean making the CI workflow as image-independent as possible?
In both of the Rust images I've looked at ([`cimg/rust:1.85-node`](https://hub.docker.com/layers/cimg/rust/1.85.1-node/images/sha256-2c9efb22d74d19d9efa65035922bbe177b153cd9d163843c917f9505a462dffa) and [`rust:1.85-slim-bookworm`](https://hub.docker.com/layers/library/rust/1.85-slim-bookworm/images/sha256-98c3b8e2721a704bc2eb62b98ffde90aa679060beab5c0261fc9fb3e53dcb96c)), Cargo already has the Rust compiler (`rustc`) and other standard tools installed in a pre-determined home location. Setting or changing `CARGO_HOME` to point to a different directory would feel weird to me, although it doesn't actually seem to break our access to the tools that are already installed.
Cargo can install things in alternate locations using the [`--root` option](https://doc.rust-lang.org/cargo/commands/cargo-install.html), but I haven't yet figured out how to do that manually.
glen
commented
To me it seems very strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly. That is what I mean by self-contained: the location of files and the operation of the project, both in ordinary use and execution as well as testing, are as little context-dependent as possible. The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be. I find it bad etiquette at best and potentially a point of failure at worst when a package installs/depends on any file outside its project tree. It's possibly that CARGO_HOME isn't the right parameter to "cargo" to accomplish this ideal -- there are a lot of configuration settings, some that are similar/possibly overlapping -- but it seems to me that "cargo" is configurable enough that we should be able to get there or almost all the way there, and then we don't need to do anything at all to try to figure out what our environment is -- we just put the stuff we need in the place where we know we will need it because of how we are invoking cargo. Is that making sense? Does CARGO_HOME seem like a reasonable approach to this form of "self-containedness"? Feel free to make alternate suggestions, but unless you feel there are aspects you definitely need to experiment with yourself, let me make the commit(s) so that I can give the CI runs some meaningful test drives. As before, I won't commit until we have reached consensus. Thanks! To me it seems _very_ strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly. That is what I mean by self-contained: the location of files and the operation of the project, both in ordinary use and execution as well as testing, are as little context-dependent as possible. The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be. I find it bad etiquette at best and potentially a point of failure at worst when a package installs/depends on any file outside its project tree.
It's possibly that CARGO_HOME isn't the right parameter to "cargo" to accomplish this ideal -- there are a lot of configuration settings, some that are similar/possibly overlapping -- but it seems to me that "cargo" is configurable enough that we should be able to get there or almost all the way there, and then we don't need to do anything at all to try to figure out what our environment is -- we just put the stuff we need in the place where we know we will need it because of how we are invoking cargo.
Is that making sense? Does CARGO_HOME seem like a reasonable approach to this form of "self-containedness"? Feel free to make alternate suggestions, but unless you feel there are aspects you definitely need to experiment with yourself, let me make the commit(s) so that I can give the CI runs some meaningful test drives. As before, I won't commit until we have reached consensus. Thanks!
Vectornaut
commented
Yes, I think the ideal would be to call
There's no guesswork involved here. Cargo's home directory is In summary, we're only relying on one or two assumptions about the environment where the workflow is running:
> The ideal is that as long as a "cargo" command of sufficiently high version is findable/executable when invoked simply as "cargo", then nothing else from the context/environment is needed/used, and every file used remains within the project tree where it can be examined as need be.
Yes, I think the ideal would be to call `cargo install trunk`, as we do during [image setup in commit `15375dc`](https://code.studioinfinity.org/StudioInfinity/dyna3/src/commit/15375dc93213c22d9b79fa728ba6f849f102eeba/tools/ci-image/Dockerfile). The manual installation I do in commit df2d458 isn't the way Cargo is intended to be used. I'm only doing it because I couldn't find a way to build Trunk within the memory constraints of the CI host. I considered installing Trunk from a binary with [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall), but building that crate seems to suck up even more RAM than building Trunk.
> To me it seems very strange to go about trying to figure out "hey, what qualifies as the 'home directory' for the 'user' I happen to be running under in this circumstance, and so where should I try to put this stuff, outside of the directory tree of this project, to jigger everything so it will work?" rather than specifying where to put everything, within the directory tree of this project, so that I know it will work and everything will be found properly.
There's no guesswork involved here. Cargo's [home directory](https://doc.rust-lang.org/cargo/guide/cargo-home.html) is `$CARGO_HOME` if that variable is set, and `$HOME/.cargo/` otherwise. We could put that logic into the workflow, but I don't currently see a need to, since we know that `CARGO_HOME` is unset in the image that we're using; I could add a comment to that effect. The weird workaround I'm doing to effectively evaluate `HOME` is annoying, but I don't think I know Docker well enough to find a better solution.
In summary, we're only relying on one or two assumptions about the environment where the workflow is running:
- Cargo is installed. (This could become version-dependent if Cargo's home directory rules ever change.)
- The `CARGO_HOME` environment variable is unset. (This assumption is optional, as described above.)
glen
commented
@Vectornaut wrote in #75 (comment):
That's exactly what I am saying: it will be cleaner to set CARGO_HOME or otherwise control where cargo looks for these files, to put it inside the project tree. Then no weird workarounds and no dependency on files outside the project tree. Both of those things are significant upside. Is there any downside? I'm not yet understanding where the resistance to configuring cargo to look for the stuff it needs within the project tree is coming from. @Vectornaut wrote in https://code.studioinfinity.org/StudioInfinity/dyna3/pulls/75#issuecomment-2515:
> weird workaround
That's exactly what I am saying: it will be cleaner to set CARGO_HOME or otherwise control where cargo looks for these files, to put it inside the project tree. Then no weird workarounds and no dependency on files outside the project tree. Both of those things are significant upside. Is there any downside? I'm not yet understanding where the resistance to configuring cargo to look for the stuff it needs within the project tree is coming from.
Vectornaut
commented
People in the Rust ecosystem consistently seem to say that Since Trunk is a self-contained binary, we could dump it in some directory listed in the If you really want to install Trunk within the project tree, we could add its location to People in the Rust ecosystem consistently seem to say that `CARGO_HOME` is meant to be set before setting up the Rust toolchain [[1](https://stackoverflow.com/a/62801744)][[2](https://github.com/rust-lang/rustup/issues/3020)][[3](https://github.com/astral-sh/uv/issues/7250)], although the Cargo documentation doesn't make this clear. I've confirmed, in a `cimg/rust:1.85-node` container, that if I change `CARGO_HOME` and then use `cargo install` to [install a tool](https://doc.rust-lang.org/book/ch14-04-installing-binaries.html), the new tool doesn't show up on the binary search path, which still points to the Cargo home directory set at the time of installation.
Since Trunk is a self-contained binary, we could dump it in some directory listed in the `PATH` variable. Then our workflow would only depend on `PATH` being non-empty.
If you really want to install Trunk within the project tree, we could add its location to `PATH`. However, I'm leery about adding a directory within the project tree to `PATH`, because to me that would make the CI environment meaningfully different from a typical development environment.
glen
commented
Thanks for the clarifying exposition. I think it separates three issues for me: (A) Where should this project install tools needed for its development and execution? (And, related, how should we invoke them?) Addressing these in turn: (A) I am strong proponent that projects should only mess with their own project tree. They can ask/expect that prerequisites, like (B) The relevant doc of
and
If CARGO_HOME is meant to be set prior to the installation of Cargo, then I have no problem with using one of the earlier three options. If we can set the 'install.root' Cargo config value by a passive, auto-found configuration file in the repo, that might be the easiest. If that's not how Cargo config values work, it would suggest we should use one of the two earlier options. And since I think command-line arguments are a bit easier to specify portably than environment variables, in that case I'd suggest just specifying the --root option when invoking (C) These suggested principles do now beg the question of whether Since the project does not contain the source for Note that even in (ii), in the case of automated testing if we are not using Hopefully these comments have brought the issues into focus and we can resolve them relatively easily now. Thanks for the clarifying exposition. I think it separates three issues for me:
(A) Where should this project install tools needed for its development and execution? (And, related, how should we invoke them?)
(B) For those cases where it's useful/appropriate to use `cargo` for such installation, how should we convey to it our choice in (A)?
(C) Is `trunk` a _prerequisite_ for installation/development/use of dyna3, or a _tool_ that the project is itself installing?
Addressing these in turn:
(A) I am strong proponent that projects should only mess with their own project tree. They can ask/expect that prerequisites, like `cargo`, be installed in the ambient OS in some place in the PATH. Everything that the project does once the prerequisites are in place should be contained within its project tree. Hence, tools/ and/or bin/, etc., directories in project trees, some of which or some contents of which that are generated by the project from its files in the repo may be `.gitignore`d. I have no problem with routinely invoking a tool via something like `bin/ourtool arg1 arg2`, for example.
(B) The relevant doc of `cargo install` states:
> all executables are installed into the installation root’s bin folder
and
> The installation root is determined, in order of precedence:
> * --root option
> * CARGO_INSTALL_ROOT environment variable
> * install.root Cargo config value
> * CARGO_HOME environment variable
> * $HOME/.cargo
If CARGO_HOME is meant to be set prior to the installation of Cargo, then I have no problem with using one of the earlier three options. If we can set the 'install.root' Cargo config value by a passive, auto-found configuration file in the repo, that might be the easiest. If that's not how Cargo config values work, it would suggest we should use one of the two earlier options. And since I think command-line arguments are a bit easier to specify portably than environment variables, in that case I'd suggest just specifying the --root option when invoking `cargo install`.
(C) These suggested principles do now beg the question of whether `trunk` is (i) a tool "installed by the package" or (ii) should be considered, like `cargo` itself, just a prerequisite in the ambient OS that the person wishing to use the package must set up and have ready in the path prior to beginning work with the package. I am fairly agnostic on this question. I have been operating under the assumption of (i). But if we feel that (ii) is more appropriate, then we can stop worrying about where it is installed: you can put it where you like, I will uninstall it from where it is now and install it someplace I like, and in the testing we can do whatever is easy and clean, e.g. install it in say a bin directory at the top level of the project tree and either add that directory to PATH or invoke it via that explicit location in the test steps that use it.
Since the project does not contain the source for `trunk` or build it itself, I'd say a plausible criterion for (i) vs (ii) for `trunk` would be whether we are going to have (tools from) the package itself monitor the version of trunk that is installed, and make sure that it is a particular "frozen" version before executing it to accomplish the various goals of the project. (To be explicit: monitoring and freezing corresponds to option (i) and otherwise, perhaps we should pick option (ii) if that's more comfortable.)
Note that even in (ii), in the case of automated testing if we are not using `cargo install` to install `trunk` so that it would "Just Work (tm)", I still don't see any reason to either use a "workaround" to find a place in the PATH to install `trunk`, or to install it outside the project tree: we should adhere to the "self-contained" aspect in terms of "keep your hands inside your own project tree" and install it (say) in a bin directory at the top of the project tree and either add that directory to the PATH or just invoke it directly at that location.
Hopefully these comments have brought the issues into focus and we can resolve them relatively easily now.
glen
commented
In meeting, we chose (ii): In meeting, we chose (ii): `trunk` is just a prerequisite. Therefore, Glen will push a commit that in CI, (1) installs trunk in `ci-bin/` at the top level of the projext; (2) in CI, adds `ci-bin/` to the PATH, and (3) adds `ci-bin` to .gitignore.
glen
commented
Hmm, exporting PATH has no effect on any subsequent step, so far as I can see, Looking for other techniques. Hmm, exporting PATH has no effect on any subsequent step, so far as I can see, Looking for other techniques.
Vectornaut
commented
Oh, now I remember how to do this! You have to write to the Oh, now I remember how to do this! You have to write to the `GITHUB_PATH` variable. I did this in the [`setup-rust`](https://code.studioinfinity.org/Vectornaut/dyna3/src/commit/bbfdf2b87f562772aa02b0b90627f14fe60887cc/.forgejo/setup-rust/action.yaml#L10) action from commit bbfdf2b.
Vectornaut
commented
Oops, I see that you just discovered the same thing. Oops, I see that you just discovered the same thing.
glen
commented
Indeed. I now consider this thread done. Also, the CI testing ran just fine for my pushes. Please "resolve conversation" if you agree all is well on this point. Indeed. I now consider this thread done. Also, the CI testing ran just fine for my pushes. Please "resolve conversation" if you agree all is well on this point.
|
||||
|
|
@ -3,26 +3,17 @@ on:
|
|||
push:
|
||||
branches: [main]
|
||||
jobs:
|
||||
# build and tag the Docker image that the rest of the jobs will run on
|
||||
setup-image:
|
||||
runs-on: self-hosted
|
||||
defaults:
|
||||
run:
|
||||
working-directory: tools/ci-image
|
||||
steps:
|
||||
- uses: https://code.forgejo.org/actions/checkout@v4
|
||||
- run: docker build . --tag dyna3:ci
|
||||
# build the application, reporting success if there are no errors or warnings
|
||||
build:
|
||||
runs-on: docker
|
||||
container:
|
||||
image: dyna3:ci
|
||||
needs: [setup-image]
|
||||
image: rust:1.85-bookworm
|
||||
defaults:
|
||||
run:
|
||||
working-directory: app-proto
|
||||
steps:
|
||||
- uses: https://code.forgejo.org/actions/checkout@v4
|
||||
- uses: ./.forgejo/setup-trunk
|
||||
- run: RUSTFLAGS='-D warnings' trunk build
|
||||
|
||||
# run the automated tests, reporting success if the tests pass and were built
|
||||
|
|
@ -30,8 +21,7 @@ jobs:
|
|||
test:
|
||||
|
glen marked this conversation as resolved
Outdated
glen
commented
Why isn't this path Why isn't this path `../.forgejo/setup-trunk`? That would seem to be the relative path from the working-directory to the action. I mean, I am not suggesting the file is incorrect; I am just saying the notation is unclear, and my particular concern is that when we (presumably sometime soon) remove the app-proto layer of the hierarchy, that this workflow file be reasonably resilient to that.
Vectornaut
commented
I think the label I think the label `defaults.run.working-directory` sets the default only for `run` steps, [like in GitHub Actions](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_iddefaultsrun).
glen
commented
How is the working directory for a uses step determined then? Should that be made explicit in this file somehow? How is the working directory for a uses step determined then? Should that be made explicit in this file somehow?
glen
commented
In meeting, we concluded that Aaron will push a commit that adds some mild commentary he is comfortable with elucidating how the current directory state is evolving during this workflow. In meeting, we concluded that Aaron will push a commit that adds some mild commentary he is comfortable with elucidating how the current directory state is evolving during this workflow.
Vectornaut
commented
In the In the `setup-trunk` action, you added the [comment](https://code.studioinfinity.org/StudioInfinity/dyna3/src/commit/0e38a38b5780e3e9368bb1f1a41467dabf66494b/.forgejo/setup-trunk/action.yaml#L13): "Assume we remain in the top-level directory of the checkout." It seems to me, though, that it doesn't really matter where `setup-trunk` runs: Trunk will be added to the path regardless. What were you thinking might go wrong if `setup-trunk` ran somewhere else? Or was this comment just meant to indicate that, for the sake of organization, `setup-trunk` *should* be run from the top-level directory of the checkout?
Vectornaut
commented
The GitHub Actions documentation is clearer about how the path to a local action is specified. This is consistent with the behavior I've seen in Forgejo Actions.
The GitHub Actions documentation is clearer about [how the path to a local action is specified](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#example-using-an-action-in-the-same-repository-as-the-workflow). This is consistent with the behavior I've seen in Forgejo Actions.
> The path is relative (`./`) to the default working directory (`github.workspace`, `$GITHUB_WORKSPACE`). If the action checks out the repository to a location different than the workflow, the relative path used for local actions must be updated.
glen
commented
What I meant was: it was our intention that trunk be installed in a directory ci-bin created at the top level of the project hierarchy. The script as written only accomplishes that intention if it in fact is the case that post the What I meant was: it was our intention that trunk be installed in a directory ci-bin created at the top level of the project hierarchy. The script as written only accomplishes that intention if it in fact is the case that post the `run: rustup...` action it remains that case that the current directory is the top-level directory. [Sorry about the delay in posting this, apparently I forgot to hit the Reply button last night when I wrote it, and only discovered that omission today when I started closing all of my numerous open browser windows.]
glen
commented
Ah, and now we have crossed to some extent. When you push clarifying comments as you see fit for the location of the uses action and/or what the checkout does to the current directory (I think those are the points we discussed, feel free to adjust the comment I put in to the setup-trunk action so that my intended meaning is clearer. Thanks! Ah, and now we have crossed to some extent. When you push clarifying comments as you see fit for the location of the uses action and/or what the checkout does to the current directory (I think those are the points we discussed, feel free to adjust the comment I put in to the setup-trunk action so that my intended meaning is clearer. Thanks!
Vectornaut
commented
Got it! It turns out that the
As of commit
I've made a second commit ( > What I meant was: it was our intention that trunk be installed in a directory ci-bin created at the top level of the project hierarchy.
Got it! It turns out that the `setup-trunk` action doesn't actually do that, because I was wrong during the meeting about how location control works in actions. The workflow comes with a *workspace directory*, which is the default place to run commands and place files that are shared between steps. This is mentioned in the Forgejo Actions [glossary](https://forgejo.org/docs/next/user/actions/#glossary):
> **workspace:** the directory where the files of the **job** are stored and shared between all **step**s.
As of commit efb2d39, the `setup-trunk` action creates `ci-bin` within the workspace directory, even if we tell the `checkout` action to check the repository out to somewhere else.
> I think those are the points we discussed, feel free to adjust the comment I put in to the setup-trunk action so that my intended meaning is clearer.
I've made a second commit (440b1df) that adjusts your comment and makes the location used in `setup-trunk` more explicit. Feel free to revert or modify it if you'd like.
glen
commented
@Vectornaut wrote in #75 (comment):
Agreed, but because it installs ci-bin as as a subdirectory of the workspace, and it checks out the repository so that its top-level directory is exactly the workspace, the ci-bin directory does end up at the top level of the checkout (as verified by an explicit @Vectornaut wrote in https://code.studioinfinity.org/StudioInfinity/dyna3/pulls/75#issuecomment-2550:
> It turns out that the `setup-trunk` action doesn't actually do that
Agreed, but because it installs ci-bin as as a subdirectory of the workspace, and it checks out the repository so that its top-level directory is exactly the workspace, the ci-bin directory does end up at the top level of the checkout (as verified by an explicit `ls` step I temporarily put in the workflow). As a consequence, I clarified the comments on checkout to note exactly where the top-level directory of the repo ends up, and I left `ci-bin` in `.gitignore` because in CI, that's where it ends up, so if somehow ever (A) the CI steps ran directly in the host [hard to imagine] or (B) we had a CI action that created a commit [slightly more plausible, even if unlikely], the ci-bin directory would be ignored. With those changes, I am resolving this conversation, and merging the PR. Thanks for all of your efforts on this!
|
||||
runs-on: docker
|
||||
container:
|
||||
image: dyna3:ci
|
||||
needs: [setup-image]
|
||||
image: rust:1.85-bookworm
|
||||
defaults:
|
||||
run:
|
||||
working-directory: app-proto
|
||||
|
|
@ -48,8 +38,7 @@ jobs:
|
|||
run-examples:
|
||||
runs-on: docker
|
||||
container:
|
||||
image: dyna3:ci
|
||||
needs: [setup-image]
|
||||
image: rust:1.85-bookworm
|
||||
defaults:
|
||||
run:
|
||||
working-directory: app-proto
|
||||
|
|
|
|||
|
|
@ -1,4 +0,0 @@
|
|||
FROM rust:1.85-slim-bookworm
|
||||
RUN rustup target add wasm32-unknown-unknown
|
||||
RUN cargo install trunk --jobs 1
|
||||
WORKDIR /home
|
||||
Reviewing this is hampered because I have no idea what
using: "composite"means, and https://forgejo.org/docs/next/user/actions/ sheds no light on this. Can you either explain or point to somewhere that has the information?The documentation of Gitea Actions and Forgejo Actions seems to rely a lot on the principle that these are "similar and mostly compatible to GitHub Actions," despite Forgejo's insistence that "they are not and will never be identical." GitHub's description of composite actions is pretty much what I'd guess from the usage here: