#phonecamera #linuxphone #librem5

Librem 5 camera/kernel can do three possible resolutions, ~1024x768 @ ~24fps, ~2048x.. @ ~31 fps and ~4096x.. @ ~15fps. Debayering is actually easier and better quality if we downscale at the same time, and that allows best framerate, so we do that (2048x.. resolution).

ARM has problems with cache coherency w.r.t. DMA, and kernel solution is to simply disable cache on DMAbufs for userspace, which means accessing video data is 10x slower than it should be on the CPU. Which means debayering on GPU is attractive, and that's what we do. (gold.frag). GPU can do more image signal processing functions easily, too, so we do some of that.

Unfortunately, we hit the same uncached memory problem at the GPU output. So we use separate thread to copy. All this unfortunately does not fit on one core, so we need two threads, one controlling GPU debayer on frame n+1, while the other one copies video data from frame n. (heart.c). We save resulting RGBA data to ramdisk. This all costs maybe 80% of one core.

From there, Python scripts can pick them up: ucam.py displaying the viewfinder and mpegize.py handling the video encoding via gstreamer. There's basically 0% cpu left, but I can encode ~1024x.. video. Unfortunately that's without audio and with viewfinder at 1fps. Plus, combination of C + Python is great for prototyping, but may not be that great for performance.

Code is here: https://gitlab.com/tui/tui/-/tree/master/icam?ref_type=heads .

At this point I'd like viewfinder functionality merged into the rest of GPU processing. Ideally, I'd like to have a bitmap with GUI elements, combine it with scaled RGBA data, and rendering it to screen. I know SDL and Gtk, SDL looked like better match, but I could not get SDL and GPU debayering to work in single process (template SDL code is here https://gitlab.com/tui/debayer-gpu/-/blob/master/sdl/main.c?ref_type=heads ).

If you can integrate main.c and heart.c, that would be welcome. If you have example code that combines SDL with processing on GPU, that would be nice, too. If you know someone who can do GPU/SDL, boost would not be bad, I guess.

@datenwolf
@NekoCWD
@dcz
@martijnbraam

@pavel @datenwolf @dcz @martijnbraam

On OnePlus 6 we have faster CPU (sdm845)
Rear camera max resolution: 2560x1600

CPU debayer works fine, but for now we can't use hw accelerated video encoding.

As workaround, on video capture we're downscaling it to 640x. Than video encoding works fine (check my mastodon for examples).

I've made custom camera app with video downscaling and QR code detection via ZBar lib

https://gitlab.com/NekoCWD/eyeneko

@NekoCWD @datenwolf @dcz @martijnbraam Yes, Oneplus 6 is easier there. (Someone should get Megapixels to work there -- https://gitlab.com/megapixels-org/Megapixels -- already has QR code decoder, AF, AE). But at 640x.. , gstreamer should be plenty fast enough to do encoding. You have 8 cores, right? I guess 1024x768 should be doable in software...

@pavel @datenwolf @dcz @martijnbraam Yea, megapixels is great app, but I want to improve pipewire integration. Btw, some performance decrease is because of pipewire (bcoz of buffer copy i guess)

@pavel @datenwolf @dcz @martijnbraam @NekoCWD I played with it over the last days and I already have 526x390 30 FPS encoding with live viewfinder, perfectly synced audio, color correction, lens shading correction, tone mapping, AWB and AE - consuming about 40% CPU. Still needs chromatic shading correction and AF, and I started experimenting with enabling PDAF.

I can also make the sensor output 60 FPS and RAW10. Patches incoming ;) Still only 2 MIPI lanes though, so no 13MP 30FPS yet.

BTW. Recording 30 minutes with screen off ate about 15% of battery, so 3 hours of recording should be possible.

I think both left- and right-masked PDAF pixels are accessible in 1:2 mode with binning disabled, though I haven't done the math yet to be 100% sure. Enabling/disabling it on demand will be somewhat clunky though. I can also read calibration data from OTP, but AFAIK there are no kernel APIs to expose that to userspace :(
https://source.puri.sm/Librem5/linux/-/issues/411#note_285719

Note that the picture is shifted down-left a bit because the driver was misconfiguring the output parameters to include dummy pixels - fixed that too, along with several out-of-range clocks 👻

@dos @datenwolf @dcz @martijnbraam @NekoCWD I hope you played with it in your day job for about 8 hours a day, because otherwise I'll feel kind of bad :-). Thanks!

Do you also get constant clicking from the AF system? I thought it was gone on Mobian, but then it came back.

526x390 is easy, can you do 1052x..? Is it possible to peek at sources somewhere?

Thanks again.

@dos Woa 🐱! That's awesome

@pavel @datenwolf @dcz @martijnbraam @NekoCWD It's 526x390, but properly binned (each channel's sample consists of 4 raw pixels averaged), which reduces noise. The shader got heavy though, does only ~35 FPS at this res - but there should be room for optimization. I've been more concerned with its correctness than performance so far.

Stats counting on the CPU with NEON is fast enough for full frame with some subsampling.

I'm giving it some finishing touches and will then publish it of course 😁

@pavel @datenwolf @dcz @martijnbraam @NekoCWD I'd expect moving binning to a separate pre-pass to make it faster, we'll see.
Also, my stats are center-weighted. Millipixels annoyed me with its reluctance to overexpose the sky 😄

@dos @pavel @datenwolf @dcz @martijnbraam @NekoCWD I'm an absolute noob in all these but i have a very naive question : how come older android smartphone can do the same thing with bigger resolution on older chip ? If i compare with an old samsung galaxy s3, it did all this very easily. Is there some secret proprietary sauce to it with specialized closed-source firmware ? Or the librem 5 just has an exotic hardware ?

@lord @datenwolf @dcz @martijnbraam @NekoCWD @pavel Specialized hardware. Phones usually don't have to do image processing nor video encoding on CPU or GPU at all, they have hardware ISP and encoders. L5 does not.

On other hardware there's also a matter of whether there's driver and software support for it, so Android may be able to use that, but pmOS - not necessarily.

@dos @datenwolf @dcz @martijnbraam @NekoCWD @pavel Ok so it really is due to some hardware "missing" and not just some closed-source firmware in the case of L5. Good to know :-)

@dos @datenwolf @dcz @martijnbraam @NekoCWD I'm quite curious how the viewfinder integration looks. Even unfinished code would be interesting to look at :-).

@pavel @datenwolf @dcz @martijnbraam @NekoCWD I'm just using waylandsink at this point, but it could be passed anywhere else. That's literally the least interesting part of the thing 😂

@pavel @datenwolf @dcz @martijnbraam @NekoCWD By the way, the actual framerate limits (verified to work) are 120 FPS @ 1052x780, 60 FPS at 2104x1560 and 20 FPS (8-bit) or 15 FPS (10-bit) at 4208x3120. Full res could go up to 30 FPS, but only once we get 4 lane configuration to work.

I can even produce 120 FPS video, although it doesn't skip frames only once downscaled to 127x93 right now 😂

@dos @datenwolf @dcz @martijnbraam @NekoCWD If you have something where you can draw / handle controls on top of the viewfinder, or at least around it. I did some hacks in the past with capturing gstreamer window, but they were a) probably X11 specific and b) not really great code. If you have nice solution, I'd like to see one.

@pavel There's plenty of apps that embed GStreamer's output to look at, and you can even skip it completely and simply import the V4L buffer into SDL's GL context and don't create your own one at all. This is just gluing things together at this point.

@dos @datenwolf @dcz @martijnbraam @NekoCWD I guess that for 120 fps video, you'd want to capture a burst to available memory, and do the conversion after the capture is done.

Could we get some intermediate (maybe cropped) modes, too? For digital zoom, or maybe for cases where performance is not available to deal with full image.

@pavel Of course. The driver doesn't handle that right now, but it's just some straightforward plumbing. Good entry level task.

@dos I'm not really looking for doing anything with gstreamer output. But if you have example of doing debayering and output via SDL, or know where to find it, I'd love to see it.

@pavel Doing it via GStreamer makes buffer queue management easier, but of course it can be done either way. With SDL you get GL context already, so you just do it the way I showed you some time ago. You skip the context creation part and you're done.