Conversation
jarkko

Jarkko Sakkinen

The screen cast below demonstrates ReadSeek 0.8.0 with hand-optimized inference engine for Qwen3-VL-2B. Given how shitty the CPU is (AMD Ryzen 5 PRO 4650) in this laptop, the performance is not all that bad, and it will of course only improve over time :-)

ReadSeek will retain itself as being based only on CPU inference as a core design decision; ReadSeek *is* smart but *does not* get in the way. That's why hand-optimized inference is even more so important aspect of this project.

The reasoning roots allowing to retain CUDA, Metal, unified memory and VRAM under user's full control. You can run your local models and ReadSeek will always do its job as an ubiquitous partner :-)

#pi #opencode #coding #agent
1
0
0
jarkko

Jarkko Sakkinen

Reply to @jarkko
Edited 21 days ago
This work is highly inspired by the gist of Dwarf Star 4 and Colibri. I was thinking what could practical application for those ideas? Then it came to me that optimizing inference for distilled models opens a whole world of possibilities for command-line tools.
1
0
0
jarkko

Jarkko Sakkinen

Reply to @jarkko
I also think there's a lot of future innovation to be done in model streaming from e.g. NVMe media.
1
0
0
penguin42@mastodon.org.uk

penguin42

Reply to @jarkko

@jarkko I think that's why there's 'high bandwidth flash' being talked about - the same stacked die arrangement they currently do for HBM ram.

1
0
1
jarkko

Jarkko Sakkinen

Reply to @penguin42@mastodon.org.uk
@penguin42 yeah also flash prices are going high up to the skies :-)
1
0
1
jarkko

Jarkko Sakkinen

Reply to @jarkko
@penguin42 It's funny. 128 GiB of RAM in a PC used to be a lot but not unheard. E.g., this 10 year old laptop that I'm typing right now has 64 GiB of RAM. Now 128 GiB of RAM is like something that only people with white coats and ties can access: -) Soon we probably have to stop selling cars thanks to big build.
2
0
0
penguin42@mastodon.org.uk

penguin42

Reply to @jarkko

@jarkko Yeh it's crazy; I'd heard there were getting that way on some types of PCB and even passive components.

0
0
1
zygoon@fosstodon.org

Zygmunt Krynicki

Reply to @jarkko

@jarkko @penguin42 Sir, I present to you Asus PX13 - 128GB of ram in a very handy 13" laptop

1
0
1
jarkko

Jarkko Sakkinen

Reply to @zygoon@fosstodon.org
@zygoon @penguin42@mastodon.org all my external displays are ProArt :-)
0
0
0
About social.kernel.org

Terms of service

  1. Please do not use this service in violation of the Linux Kernel Code of Conduct. Doing so will result in your account suspension with the referral of the matter to the CoC committee.

  2. "Repeating"/"boosting" someone else's status on this platform will be treated as endorsement and will fall under rule #1.

  3. You are encouraged to use this platform to promote your work on the Linux Kernel, but there is no restriction on permitted topics (with the exception of anything covered by #1 above).

  4. There is no requirement to post in English, but it should be considered the primary language of communication on this platform.

Privacy notice

The admins of this service have access to all posted statuses. They aren't looking, but if it's something they shouldn't know about, then you should not post it on this platform.

Please see the Linux Foundation Privacy Policy, which applies to this platform as well.

Getting your own account

If you would like an account on this instance, please check that the following applies to you:

  • You are listed in MAINTAINERS or CREDITS

  • OR: You have a kernel.org account or email address

  • OR: You have a long and established history of involvement with the Linux Kernel

If the above is true and you agree with the Terms of Service and Privacy Notice listed above, please use these instructions to request an account: