Conversation
Jarkko Sakkinen
jarkko
Edited yesterday
In inference I hit to Rust bottlenecks. It is an enormous effort to make all loops error check free in order to not generate boilerplate. It's really hard to optimize against icache when using Rust TBH.
I spot all the time ".context()" (anyhow) or try-statements and similar cruft. When I work around not having to do any of these checks, inference is suddenly like 1-3% faster. This is ridiculous. Most of my time is now spent on Rust performance issues.
What I will do is to create a C library with the shared primitives (like matrices, tensors, fp16 and similar), which will then be used by ReadSeek and Goosedump to implement inference engines for Qwen3-VL-2B and GPT-OSS-20B.
I'll call this library as libcrank :-)
#ffmpeg
I spot all the time ".context()" (anyhow) or try-statements and similar cruft. When I work around not having to do any of these checks, inference is suddenly like 1-3% faster. This is ridiculous. Most of my time is now spent on Rust performance issues.
What I will do is to create a C library with the shared primitives (like matrices, tensors, fp16 and similar), which will then be used by ReadSeek and Goosedump to implement inference engines for Qwen3-VL-2B and GPT-OSS-20B.
I'll call this library as libcrank :-)
#ffmpeg
0
0
1