ML New writing strategy – “variable buffering”

June 21st, 2013
ML New writing strategy - "variable buffering"

MLMagic Lantren (ML) is continuing optimising their Canon HDSLR RAW recording on various camera models.
a1ex is one of the leading figures in the developing team and now he decided to revise the buffering strategy to achieve buffer times that will eliminate or reduce dropped frames while shooting high resolution RAW on fast cards.
He calls the new method “variable buffering” and here is what he has to say:

If you ever looked in the comments from raw_rec.c, you have noticed that I’ve stated a little goal: 1920×1080 on 1000x cards (of course, 5D3 at 24p). Goal achieved and exceeded – even got reports of 1920×1280 continuous.

During the last few days I took a closer look at the buffering strategy. While it was near-optimal for continuous recording (large contiguous chunks = faster write speed), there was (and still is) room for improvement for those cases when you want to push the recording past the sustained write speed, and squeeze as many frames as possible.

So, I’ve designed a new buffering strategy (I’ll call it variable buffering), with the following ideas in mind:

* Write speed varies with buffer size, like this (thanks to testers who ran the benchmarks for hours on their cameras)

* Noticing the speed drop is small, it’s almost always better to start writing as soon as we have one frame captured. Therefore, the new strategy aims for 100% duty cycle of the card writing task.

* Because large buffers are faster than small ones, these are preferred. If the card is fast enough, only the largest buffers will be touched, and therefore the method is still optimal for continuous recording. Even better – adding a bunch of small buffers will not slow it down at all.

* This algorithm will use every single memory buffer that can contain at least one frame (because small buffers are no longer slowing it down).

* Another cause of stopping: when the buffer is about to overflow, it’s usually because the camera is trying to save a huge buffer (say a 32MB one), which will take a long time (say 1.5 seconds on slow SD cameras, 21MB/s). So, I’ve added a heuristic that limits buffer size – so, in this case, if we predict the buffer will overflow after only 1 second, we’ll save only 20MB out of 32, which will finish at 0.95 seconds. At that moment, the capturing task will have a 20MB free chunk that can be used for capturing more frames.

* Buffering is now done at frame level, not at chunk level. This finer granularity allows me to split buffers on the fly, in whatever configuration I believe it’s best for the situation.

* The algorithm is designed to adjust itself on the fly; for this, it does some predictions, such as when the buffer is likely to overflow. If it predicts well, it will squeeze a few more frames. If not… not

Read his full post or try the new buffering for your self and comment here: LINK

18
Leave a Reply

guest
Filter:
all
Sort by:
latest
aliev
aliev
Guest
June 21st, 2013

Rofl, my short film affected by every single update since magic lantern Raw recording cme out. Cool. This one is awesome. Love u magic lanternians

Bart van der Horst
Member
June 21st, 2013

Well by now it is completely clear that a fullframe videocamera can be build between 1000 and 2500 dollars. It is pretty sure that millions of people will buy such a camera.

Why the big companies don’t build it is because they probably don’t have the guts to do so. They are to scared.

Why? Well… it is not capitalism. It just one of those mysteries…

Derek McCabe
Guest
June 21st, 2013

American Capitalism… where the shareholders come before the customers, and the employees too.

marklondon
Guest
June 21st, 2013

I think some people who always write the same comments on various blogs are unfamiliar with the concept of capitalism.
And as for Canon hiring the ML team: that would be the worst idea in history for us. :-) They would find them other things to do, and we’d never hear from them again. If Canon were a web company this would have already happened 2 years ago.

jerimy123
jerimy123
Guest
June 21st, 2013
Reply to  marklondon

You are wrong! I’m all for capitalism. It means I get to vote with my wallet. And until Canon starts innovating again and stop using the same tech over and over, camera after camera after camera, I’ll look elsewhere.
The ML team is the only thing breathing a bit of air into these cameras atm. Even my old 5D2 got to see a bit of daylight again.

Matt Burton
Guest
June 21st, 2013

these people act like any self respecting camera company should, continually updating their products for the good of the user and not monetary gain !

Matt Burton
Guest
June 21st, 2013

these people act like any self respecting camera company should, continually updating their products for the good of the user and not monetary gain !

Angus Lion
Guest
June 21st, 2013

I wonder how much this will benefit the currently crippled lower end cameras like the 550D, 60D etc.

Angus Lion
Guest
June 21st, 2013

I wonder how much this will benefit the currently crippled lower end cameras like the 550D, 60D etc.

jerimy123
jerimy123
Guest
June 21st, 2013

Canons approach to all of this is still to just ignore it? They are fools.

jerimy123
jerimy123
Guest
June 21st, 2013
Reply to  Johnnie Behiri

They could revolutionize DSLR video…again.

Instead they try to protect their much more expensive cameras while keeping their cheaper cameras crippled.
They should hire the ML team…everybody would benefit.

Doug Gray
Doug Gray
Guest
June 21st, 2013
Reply to  jerimy123

Of course they are trying to protect their more expensive cameras. Who wouldn’t? Do you have a clue how much money they have invested in those cameras from a research/development/production standpoint? Of course they want to protect those cameras so they can recoup what they put in.

I would also strongly argue that their cheaper cameras aren’t crippled. They simply aren’t playing the superman that can be. Even without Magic Lantern, a whole bunch of their cheaper cameras are awfully valuable pieces of equipment for videographers/photographers. Could they be better straight from Canon? Absolutely. But let’s not pretend that they are 2003 point and shoot cameras either and call them crippled.

jerimy123
jerimy123
Guest
June 21st, 2013
Reply to  Doug Gray

This is really not a case where you can argue if they are crippled or not. Canon reuses the same hardware in different models and simply chooses to turn off features in the firmware. A good example is the 1DX vs. 1DC which are identical except for the firmware.
I understand the reason for doing it, but it sure sucks from a consumers view plus it’s holding back technology instead of pushing boundaries

basedgawd
basedgawd
Guest
June 21st, 2013

Can someone explain in kindergarten language what this development means

jerimy123
jerimy123
Guest
June 21st, 2013
Reply to  basedgawd

Higher resolution and less framedrops.

Filter:
all
Sort by:
latest

Take part in the CineD community experience