The most of #autotools based open source software is sort of anti-pattern for QA/CI because the test suite is hard-bound to the source project. This is the reason why I rarely (or almost never) run TPM2 TSS test suite.

I wonder if #rustlang continues to follow this anti-pattern or is there cargo install for the test?

It is sort of thing that has been always bad for anything with disjoint host and target system but is part of “craftmanship” because things has been done that way long enough :-)

@jarkko I don't exactly follow this. If you mean by that unit tests are included in the same package/namespace as tested code.. well, those are not tests that should be QA concern. Integration tests for the API in rust projects are usually in different directory tree next to the source, but that is just a convenience as those will import source tree as crate. So those tests could also be in entirely different repository or sub-project if you want and test against (pre-)released source crate.

@orva How do you build BuildRoot/Yocto whatever image without Rust support and deploy test suite as a package? I.e. you are testing on target and do not have the Rust environment there at all.

E.g. just because of this pattern I need to put sometimes GNU make to a test image because test frameworks tend to rely on toolchains.

@jarkko I don't think that is feasible for library code.

Test suite is "just" a binary that includes all the toolchains _and_ testable code (as a crate). So having library code testable with external application would need to export linkable symbols which is non-rust way of doing things.

Testing application code should be easy though (if doing "proper" e2e/integration testing, no shortcuts by calling application internals). Tests are just self contained binary, build it and release package.

@jarkko So the static linking only approach makes external library tests impossible/inconvenient and makes application testing easy.

@jarkko Or if you are fine that test-suite is your tests+toolchains+library, then producing that test binary as installable package is trivial.

This is how container based workflows usually go: first there is build image which contains rust toolchain which generates the static blob, which is then copied to the runner image which is probably barebones alpine image that hopefully doesn't contain anything else than the test binary.

@jarkko: it sounds like you're looking for a different KIND of test - a full-stack functional test - but are complaining that integration and unit tests aren't this other kind of test?

I don't want to bust in and tell you what you want or should want at all, but unit and integration tests are foundationally different from this third kind of test (which I don't have a GREAT name for, to be honest). This "functional" test is for full stack integration testing as packaged and deployed (where integration tests might be slightly lower level and integrated with the source code repo and tooling to ensure interaction between siloed pieces during development and unit tests...well, they test source code "units", but I am sure you know this).

I find the third kind ("full-stack" or whatever) hard to write during development, as I probably have a version installed on my machine and a version under development and the mixing of those two geta messy quickly.
@orva

@BryanBennett @orva All I'm saying that it is repeated by-design mistake not to design all layers of tests detachable. It was persistent in autotools and the same bad design is repeated e.g. by cargo.

E.g. unit test suite of kernel called kselftest is counter-example of this. It can be run in-tree but can be also packaged and deployed to the target device. So it is not a law of nature but more like bad engineering practice that is common enough to have become a tradition. Just criticising the obvious I guess...

Usually everything I do there is a cross-compiler involved, so even for unit tests there's a hard border between the world where testing happens and the host system, it is more like a law of nature.

@BryanBennett @orva Speaking of full-stack, consider some IoT device, let's say a smart light bulb with a bluetooth speaker and a webcam for the sake of example.

We can run integration tests for it designed for the product right now but essentially a major portion of all possible unit tests written to to the components that encompass its software stack have a total zero value. Sure you can check them in your development system but it has much less value than to run them with the hardware where they are deployed. If unit tests were detachable and deployable you could run (a) your integration test suite and (b) all possible tests written each individual component. That would be a measurable improvement e.g. to the product security metrics.

@BryanBennett @orva To put this in perspective all the testing methodology was invented in a world where embedded systems were essentially something like Intel 8051 with a custom piece of control code written in assembly language and no operating system whatsoever.