{"id":97,"date":"2012-07-10T07:14:53","date_gmt":"2012-07-10T07:14:53","guid":{"rendered":"http:\/\/www.vfxpdx.com\/?p=97"},"modified":"2015-04-29T09:41:01","modified_gmt":"2015-04-29T16:41:01","slug":"an-accidental-love-letter-to-dpx","status":"publish","type":"post","link":"http:\/\/www.vfxpdx.com\/?p=97","title":{"rendered":"An accidental late night sexy text to DPX"},"content":{"rendered":"

So everyone gets what VFX is.\u00a0 And PDX is a no brainer for anyone who’s ever gotten tired of writing out P-O-R-T-L-A-N-D and is fine enough with an airport acronym, and a cool one at that (what with it’s iconic X).\u00a0 But DPX?\u00a0 Who the what now?\u00a0 Have I gone and scrambled my brains at the Driftwood Room<\/a> again prior to posting?<\/p>\n

DPX is my favorite file image format.\u00a0 And oh how I missed it.<\/p>\n

You’ll have to excuse me for a sec while I geek out.\u00a0 But we’re never gonna survive unless we get a little geeky<\/a>.<\/p>\n

Pulling a DPX into Nuke used to mean you were working on a film.<\/em>\u00a0 Hell yeah.\u00a0 This is something I took for granted; that those tasty dpx’s would always be sitting there, waiting for me every morning.<\/p>\n

I’m not sure quite sure when they went away, but at some point along the way, along came EXR’s, and things got all complicated<\/em>.<\/p>\n

Now at my brand new gig, I’m the “end of the line.”\u00a0 Has a nice ring to it!\u00a0 And guess what?\u00a0 DPX is back in vogue.\u00a0 Allow me to sing it’s praises.<\/p>\n

For live action footage, there is no better.\u00a0 Nuke absolutely eats a DPX for breakfast.<\/p>\n

And I’m not talking Portland brunch<\/strong>… \u00a0 where you cruise into Tin Shed<\/a> on Alberta, put your name down, grab yourself a cuppa the “free” Joe only it’s not really free because you’ll be billed for it in the event that you wait long enough to finally sit down and your server sees that you have the cup and marks you down for 1 stumptown <\/a>special, but let’s not even go there yet because you’re still sitting there waiting but you stick it out because it’s like you brought your dog and Tin Shed<\/a> is like the only place that really allows dogs because they have the outdoor seating and all that but it’s like sunny and so it’s crowded but hey it is SUNday after all so like whatever I’ll just sit here for what time is it? is that even morning anymore?\u00a0 and wait until they call my name and there’s not really much else within biking distance except hey maybe Juniors is my backup plan but I’d have to tie the dog up out front and the food is good and all but it was kind of dirty in there last time I was there or at least my cup had spots on it and my water was kind of, I don’t know, dusty?\u00a0 so like I should probably just wait it out but there’s like 3 other 2-tops ahead of me on the list, like when did this place turn into Gravy <\/a>anyway I thought Tin Shed<\/a> was like normal but whatever it is like the best thing Alberta has so I better just suck it up.<\/p>\n

Ok, that diatribe you just went through?\u00a0 That was an EXR.\u00a0 DPX would have you out the door and smiling already while EXR is still loading in scanlines.\u00a0 If you want to talk in brunch terms, this is like Gravy <\/a>vs. Equinox <\/a>– am I right?\u00a0 Ok, maybe I should explain that ref for any non P-towners who aren’t familiar with our obsessive brunch scene.\u00a0 Gravy over on Mississippi Ave is hands down one of the best brunches in town.\u00a0 The best french toast in the city – maybe the galaxy<\/em>.\u00a0 But you wait for it.\u00a0 It’s a mob scene even on a weekday.\u00a0 In other words, it’s great but it’s the SLOW boat.\u00a0 However, right around the corner from Gravy is Equinox.\u00a0 Which is a damn good brunch – up on par with Gravy for sure.\u00a0 The difference being it’s tucked around the corner, just enough off the beaten path where you can walk right in and sit down immediately.\u00a0 You’re done and moving on with the rest of your day while the Gravy folks are still standing outside, tapping feet, shooting glances to watches, pondering how much longer before they just…\u00a0 bail.<\/p>\n

Don’t be afraid to bail on EXR when the circumstances permit.\u00a0 Proof?\u00a0 You want proof?\u00a0 Get this – for a plain Jane little 3 node comp render of 50 some odd frames in Nuke (in > blur > out) with footage shot originally with a RED\u00a0 Epic @ 4k,\u00a0 here are the stats:<\/p>\n

EXR (zip scanline) – 2 minutes 57 seconds<\/p>\n

DPX – 10 seconds.<\/p>\n

No @$#% kidding.\u00a0 10 seconds.\u00a0 I mean, I had a feeling DPX would be faster, just from working with them time and time again… \u00a0 but I’ll admit that had my head spinning.\u00a0 I’m not even sure how to explain this because it doesn’t add up – there is nothing apparently different about this machine I’m on…\u00a0 there’s no DPX equivalent of a Red Rocket card or anything, and no GPU accelerated nodes in my script.\u00a0 I’m scratching my head – how is this possible?\u00a0 This box does have an SSD but I was sure to clear the caches before rendering, so it should be apples to apples.\u00a0 I ran it twice and got similar render times.\u00a0 The sys admin wizards at Weta Digital might have some sort of caching enabled on Linux here that is somehow locally stashing the DPXs in RAM, but not the EXRs.\u00a0 Hmmm.\u00a0 That gives me more questions than answers!\u00a0 Mental note:\u00a0 never ever leave Linux.<\/p>\n

Anyway, a 2nd opinion is in order, good doctor.\u00a0 Let’s try something a bit more scientific and I’ll shoot this to the renderfarm.\u00a0 I’ve switched shots and have a 2k shot that originated on film, not that it really matters once you’ve made a DPX and an EXR.\u00a0 Kicking these off now and will wait for that juicy stats email…\u00a0 looking for total time elapsed and I’ll get you a per frame CPU time (minus batch load time).<\/p>\n

EXR (piz):\u00a0 6m 53s – 7 secs<\/p>\n

EXR (scanline): 6m 38s – 6 secs<\/p>\n

DPX: 4m 10s – 4 secs<\/p>\n

This was done on a Sunday – as low traffic and open of a farm as Weta Digital gets.\u00a0 Different versions were rendered within minutes of each other, and those Nuke renders should have had the blades all to themselves.\u00a0 The bottom line:\u00a0 33% faster rendering.<\/p>\n

You can talk all you want about how Nuke uses EXRs as it’s internal intermediary format and is floating point across the board.\u00a0 Bla bla bla.\u00a0 Look at the results.\u00a0 That adds up to a heck of a lot of saved time over the course of the day, not to mention the interactive boost while working.\u00a0 As long as your capture format is less than 16 bit, and if they are shooting digital (or even film<\/a>), it is most definitely less than 16 bit – you simply cannot beat what I’ll call “integer love.”\u00a0 The CPU just chews through it.\u00a0 For grained live action footage shot with most cameras, DPX.\u00a0 There is no substitute.<\/a><\/p>\n

The RED Epic, with it’s claimed 11 stops of RAW dynamic range?\u00a0 No problem.\u00a0 The Canon 5D and it’s 14 bit sensor?\u00a0 Easy money.\u00a0 10bit Log DPX handles as much dynamic range as 16bit linear because, kind of like an mp3 still sounds damn good (or maybe FLAC is a better comparison), log space will put all of the image goodness where it counts.\u00a0 Your highlights?\u00a0 Don’t worry, they’re unscathed and intact.<\/p>\n

File size you say?\u00a0 Network traffic?\u00a0 Also not a problem here.\u00a0 10bit log DPX was 13mb per beautiful grainy frame in this example (2k live action plate).\u00a0 EXR\/Piz could only come up with 14mb.\u00a0 That varies, but the time spent unpacking them doesn’t so often a small filesize advantage for Piz can be ignored in favor of performance.<\/p>\n

Some people will scoff at a raw DPX and it’s log encoding, saying it looks washed out and unviewable.\u00a0 First of all, who are these people and why are you letting them anywhere near a vfx pipe decision?\u00a0 Amateurs!\u00a0 Ha!\u00a0 The truth is, an EXR in linear colorspace is just as strange to view in it’s dark raw state.\u00a0 You’re never going to view either of them raw and are gonna have to chuck a LUT on it either way, so this is a non-issue.\u00a0 Worry not, Photoshop can still open it<\/a> correctly.<\/p>\n

Don’t get me wrong, DPX is not for everything.\u00a0 There’s the whole alpha channel thing.\u00a0 The dpx spec can handle it (as well as rendering up to 16 bit linear<\/em>) but many of the software, After Effects comes to mind, still seems to be stuck on the old Cineon spec.\u00a0 Cineon was what DPX was before it was dpx – .cin – and Cineon had no alpha channel support and was locked to 10 bit log.<\/p>\n

CG renders especially will use all of the extra range goodness that EXR can offer up.\u00a0 Over in EXR land, Piz compressed is the way to go for live action plates over slow networks, or large feature film projects that need oodles of storage.\u00a0 If you can afford larger filesizes, Nuke likes a good scanline zipped EXR much better.\u00a0 Single scanlines for live action, and often shops will write tiled for CG out of the renderer and then convert to ZipS as a post process.<\/p>\n

Really, the big revelation with EXR was embedding multiple passes into one file, and knocking things down to “half-float” for all of the calculation and storage advantages.\u00a0 Finally, floating point was worth the price of admission.\u00a0 Historically, Tiff could always do float precision but no one used it because the extra usable range came at such a rendering and disk space cost that it was overkill 99.997% of the time.\u00a0 In fact, at Weta prior to the EXR revolution, we were using liff (essentially a log IFF) as our primary format for CG renders, for many of the same reasons that DPX works so well.<\/p>\n

When EXR 2.0 is finally released, it will up the ante and you’re going to start seeing more standardization of the deep image formatting<\/a> for CG without holdouts or layering problems that useless per pixel Z passes always had.<\/p>\n

But for live action plates (in most cases*), viva la DPX!<\/p>\n","protected":false},"excerpt":{"rendered":"

So everyone gets what VFX is.\u00a0 And PDX is a no brainer for anyone who’s ever gotten tired of writing out P-O-R-T-L-A-N-D and is fine enough with an airport acronym, and a cool one at that (what with it’s iconic X).\u00a0 But DPX?\u00a0 Who the what now?\u00a0 Have I gone and scrambled my brains at… Continue reading →<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[5,6],"tags":[9,7,8],"class_list":["post-97","post","type-post","status-publish","format-standard","hentry","category-blog","category-tip-o-the-day","tag-color","tag-nuke","tag-pixel-nerd-stuff"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2Cfrz-1z","_links":{"self":[{"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/posts\/97","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=97"}],"version-history":[{"count":40,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/posts\/97\/revisions"}],"predecessor-version":[{"id":1705,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=\/wp\/v2\/posts\/97\/revisions\/1705"}],"wp:attachment":[{"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=97"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=97"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.vfxpdx.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=97"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}