<![CDATA[THE CHAOS WORKS - Blog]]>Mon, 16 Oct 2017 16:11:03 -0700Weebly<![CDATA[Morning shifts suck balls]]>Mon, 16 Oct 2017 17:37:00 GMThttp://thechaosworks.com/blog/morning-shifts-suck-ballsOr, more to the point, the switch over from late shift to early shift just kills me. Takes me a week or more to get the brain firing over as normally as possible. However, I was able to pass my third course of this term @ WGU, so I'm almost 50% done with my degree and I'm only into my second term. I'm assuming my progress will slow once we get into the less generalized stuff and start digging deeper into the real meat of the program.
Next and last course (depending on how long it takes me to complete) is Intro to Probability and Statistics. The title alone makes me drowsy. Whee! 
So far, I'm really enjoying my time at WGU. Their pass/fail (which they call competency based education) approach really works for me as there many aspects to several courses that I already know having been through a normal American education and having 20+ years of IT experience. So I was able to breeze through a ton of courses in just a few months. New we're getting into the taller grass. There be lions here.

Production continues apace. So far, there have been some good episodes, with the best so far being S21E02 - Put It Down.

Also, the game drops tomorrow! Very eager to see how it is received given all the effort made in addressing the complaints people had with the first one. I also can't wait to see how people respond to certain scenarios in this one. Some of these I was SURE wouldn't see final, but to Ubisoft's credit, they took it all and asked to go further. Rare to see in a publisher. I won't give anything away and no I can't give you any free copies. Buy it.

My wife needs shoes!

On the development front, haven't done much. Waiting for UE4 18.0 to officially release so I can start playing with AR in earnest. I have several ideas for how one could make compelling experiences, and am eager to try them out.

​That's all for now. 
<![CDATA[First three are down...]]>Fri, 29 Sep 2017 23:15:42 GMThttp://thechaosworks.com/blog/first-three-are-downnNow we get to do all the maintenance and clean up that needs to be done whenever you have 70 people working 120+ hours a week for 3 weeks straight. Currently I'm in the process of updating the VM our art dept. uses, mainly just minor patches, making sure all the Windows security updates (so many!) are up to date, everyone has the latest version of our toolset, etc. Also taking this opportunity to catch up on my studying. I was supposed to be 1/2 way through the reading material for the current course I'm taking and I'm only 1/3. Gots to get some reading done. Whee... Alas, Forza 7 also just released. How's a person supposed to study with all this racing that needs to be done?]]><![CDATA[Another first Tuesday]]>Wed, 13 Sep 2017 03:15:49 GMThttp://thechaosworks.com/blog/another-first-tuesdayAnd thus I begin a string of late tuesday nights. Our first episode airs tomorrow and we at least have an entire 23:00 worth of script. Alas, as of 20:15 Tuesday night we only have 13:51 of animation approved. So it's gonna be a long night of just grinding through the process. For me, that mainly means just keeping an eye on the render queue to make sure nothing gets stuck and helping people out with the occasional issue they may run into ("Where did I save this file?", "I forgot my password again.", "Can you upgrade my flash plugin please?"). 
Yup, the glamorous life of IT. On the upside, I have time to study, play games, watch TV and Movies, etc. The main challenge is in remaining engaged and conscious...
<![CDATA[Yup. Again. Deal.]]>Tue, 05 Sep 2017 17:57:55 GMThttp://thechaosworks.com/blog/yup-again-dealOnce more I'm going to give this a try. Maybe even gain some *gasp* discipline while doing so. 

That being said, it's been a busy year around these here parts. Here's what we've gotten done since Jan-2017:
  • Replaced the entire network infrastructure for the entire studio from BD-8810 chassis switchs to A Juniper based solution with multiple 40G core switches and several standard switches in a virtual chassis configuration.
  • Increased the inter-connect speed between our tow machine rooms from 20G to 160G
  • Replaced the Hitachi Frankenstein HNAS beast that was our primary file server with two Open Drives Spectre based systems. Not only are they each capable of over 2 GB/s of throughput (vs the HNAS' 400 MB/s), they are ZFS based, and have 4x40G NICs so they can handle as much net data was we can throw at them and still have bandwidth and IOPs to spare.
  • Replaced our so very sadly aging XServe video conversion farm with rack mounted 2013 Mac Pros (yes the trash can lookin things) and switched from using Shake for video conversion to Nuke. This gets us onto both software and an operating system made in this decade.
  • Finally finished our portion of the work for the next SP video game, Fractured But Whole. Finally. After fucking forever. We're very happy to be done with that thing, and I'm very eager to see peoples reactions to it. I can't wait to see the "family protection groups" give their "reviews" especially.

So, as you can see, we've been busy but productive. And now we being production on Season 21. 21... Fuck me, I'm old. And I've been here for a very very long ass time.

On that note, I've started working towards a degree in Software Development in the hopes I can leverage that on top of my production experience to shift back into making content instead of just building the machine that the content makers use. Already into my second term and I'm over 30% of the way to my degree. This is possible because I am "attending" Western Governors University. I say it like that since it's an online only university, but is still fully accredited. The neat thing about their approach is what they call "Competency Based Learning". So, there's no letter grading system. You just take tests that judge your competency level in a given subject and if you pass a certain threshold, you are considered good enough.
Since I have a couple decades of IT production experience, a lot of which has involved coding in various forms, not to mention my own self driven forays into programming, I've been able to test out of most the courses in the first term without even having to read the course materials. And the rest I was able to knock out relatively quickly. So I got done 9 courses in my first term, instead of the usual 4. That was nice. Even nicer was the fact that some of the credits I'd earned at USC, even though it was over 20 years ago, transferred over, so that gave me a extra terms worth of credits. I hopefully will test on this next course, which is really just a basic intro to HTML and CSS, most of which I already know and the rest I could guess reliably.
I'm finding the really challenging stuff to be the business informatics and political philosophies they are also trying to teach in this course. I can understand why they feel this info is necessary since most of the course participants will end up working in corporate environments, but since I have the luxury of not giving a fuck about that shit, I find it rather mind numbing. Which is true of any information I don't feel I would have need for in my life. Again, the joys of being a fussy bitch.

And that's it for today's brain dump. I promise future ones will be both more frequent, and much shorter.


<![CDATA[Terragen. In Spaaaaaaaaaace!]]>Mon, 21 Jul 2014 23:33:57 GMThttp://thechaosworks.com/blog/terragen-in-spaaaaaaaaaace
This is an example of using Terragen, a terrain (and world) generation program in a way you wouldn't normally expect. Their cloud code is truly phenominal, and some clever lad figured out how to use multiple light sources along with that cloud system to simulate a nebula. Very slick, no?
<![CDATA[Hashtastic]]>Wed, 11 Jun 2014 00:16:12 GMThttp://thechaosworks.com/blog/hashtasticOne of the more esoteric bits of our pipeline has always been the interface between our render pipeline and our editing systems. Back in the dark ages, the only efficient way to get material into an editing system was to capture it off of a video tape. Everyone did it. Even feature films. They would use a process called telecine which would basically use a video camera to take a picture of each frame of film and record it onto one frame of video. The fun part comes in when you bring in the nature of video. It's not like film, where each frame is essentially a single photograph. Due to the ingenious ways our forebears engineered their way around the technical limitations they had in order to make television transmissions work, they couldn't just send a single full frame picture for each frame of video. They had to break each frame into sub frames, which they ended up calling fields. In order to get some semblance of picture transmitted and displayed as fast as was possible back then, instead of breaking the frame into the top half and then the bottom half, or the left then the right, they broke the frame into lines and then made fields out of every other line. So, the first field would have the first line, the third, the fifth, and so on and the second field would have the second line, the fourth line, the sixth, etc.
For obvious reasons, these fields were called Even and Odd. And depending on who made the camera being used, the VCR, TV, etc. which field came first changed. So for some things, the even field was expected to be received first, and other things expected to receive the odd field first. As one would guess, this kept things very easy and simple to understand. Fast forward to the digital age and the non-linear editing system and the telecine process. To understand the true joy of telecine and video fields, one needs to understand that film usually is shot at a frame rate of 24 fps, meaning the camera takes 24 pictures every second. Video instead runs at apprx 30 frames per second, or really 60 fields per second. Why 60? Because that was the frequency that electricity flows through the country's power grid, which made it a easy way to synchronize everyone's TVs to the same approximate timing. This also means when converting a film sequence to video, you had to not only convert the 24 frames to video fields, you also had to change the number of frames per second. This is done with a process called a 3/2 pulldown. Now comes Math. I apologize in advance.
So, if film is at 24 frames per second, that would be the same as 48 fields per second. Since video is 60 field per second, we need 12 more frames. The easy way to do that is to just copy a field every once in a while. Say, every fourth field for example. For a better explanation, check out:


Which field you decide to copy is called cadence. And, it turns out, what cadence you decide to use can have a very big impact on how your material looks after its been converted. Do it wrong and things look weird and jaggy and stuttery.
So, why am I telling you any of this? Because cadence has been an issue for us almost from the very beginning. As I mentioned WAY back at the beginning, since the fastest way to get video onto an editing system back when we started was to capture it from a video tape, we had to find a way to take our CG generated frames onto video tape. Luckily, we were not the only people trying to do this kind of thing. The digital effects industry was just starting to boom so there was a lot of demand for something that could take CG frames and display them in realtime on a video system. We used something called a DDR or Digital Disk Recorder. This allowed us to render our animation at film frame rate (which is what all our animators were used to animating to) and just copy those frame files to the DDR hard drive and it would play back those frames automatically doing the 3/2 pulldown conversion from film to video. It also could pretend to be a video tape deck which our editing systems could capture from. Thus we were able to get our animation onto an editing system as quickly as possible. 
Alas, now that technology has reached the point that video fields are considered on the way out if not obsolete, we still have to deal with them as we re-produce our older episodes to HD. Thats right, re-produce. A lot of shows that were airing during the HD conversion process just took their video signal and scaled it up. Since HD is a different aspect ratio from regular video, this results in black bars to either side of the image. That sucks. Since we still have all the source files from when we made an episode the first time, we are able to reload them into our animation software and recreate the image in proper HD format. This make us rather unique in the animation industry. It also means we've had to dedicate a small team of people to this process because the change in aspect ratio creates a ton of issues. Our animators would use the framing of the shot to tell when they could stop animating a character, for example. Since a HD frame is wider then a regular video frame, this results in a lot of shots where a character would seem to just stop moving instead of walking off the screen.
Another of the issues we have to deal with is because of the cheat we did back then using that wonderful little DDR. Since we were just dumping frames onto it and letting it handle the conversion to video, things like cadence changed on a shot by shot basis. This is very very bad if for some reason you have to re-edit that material years later, maybe when changing it to use HD shots instead of regular video shots, since there's no way to match the cadence originally used unless you do it by eye. And that is why our HD editor can be a very cranky man. And why we've spent a long time tweaking and tuning our render queue to output files in different cadences for different episodes. 
Our HD team just started working on episode 6 of season 1. And yet again we've found the cadence has changed. And we also discovered that the new version of editing software they are now using is even more fussy about field cadence and frame rate then the previous version. So we've spent the last few weeks going back and forth with them trying to dial in our render queue output again. And we finally got it sorted. So, I am recording what we are using now here for posterity since I am sure I will forget as soon as I move on to the next problem. And I REALLY don't want to go through this process again. So, for the record:

Episode 106:

Field dominance = ODD
Field cadence = BC

In Shake (which we STILL use for frame to quicktime file conversion), that would look like:

FIPullUpDown(MainFramesIn, "Pulldown", 1, 0);

And, just for completeness' sake, here at the QT HASHes Shake needs to output the right format quicktime file:

Pre season 12.5 (29.97 fps episodes):

No Alpha:

With Alpha:

Post season 12.5 (23.976 fps episodes):

No Alpha:

With Alpha:

No, this information has no use to anyone other then me. What's your point? 
<![CDATA[6 months later...]]>Fri, 06 Jun 2014 17:43:40 GMThttp://thechaosworks.com/blog/6-months-laterYes, I am bad at this. Never claimed otherwise. But I am going to try again.

Quick status update. Still at the 'Park. Still doing what I do. We've overhauled just about the entire production now. All artist workstations have been update to the new Mac Pro (fuckers are tiny and FAST). We gone Mavericks across the board to mixed results. Some things are better, others are much much worse. The network throughput bug I found in 10.8.4 is gone. The Finder refresh bug is back. And we updated our LDAP/email server to Mavericks from OS X Server 10.6.8. And I must say, DO NOT EVER DO THIS. 

If you absolutely have to use the Fisher Price equivalent of a enterprise server OS, say because maybe you have so much legacy data that the idea of having to start over from scratch on a linux box makes you nauseous, do yourself a favor. Do NOT migrate your old server to the new OS. Set up a fresh new server and re-create your users from scratch. And just say good bye to any legacy data like emails and such. The problems you will inherit due to very shoddy migration scripts and processes make it actually more painful then if you'd just gone to linux after all. Ah, the joy of hindsight.

So, where does that leave us. We are up and running. Aside from the Mavericks Server ickiness, everything else has been migrated off to linux boxes, specifically CentOS 6.4 on HP Proliant servers. I must say I'm impressed with the build quality and design of those Proliants. Very slick. We've also switched our tape backup workflow from a NDMP based LTO 4 setup to 10G ethernet LTO-6 based one. And it's kinda faster. Kinda? But by dumping NDMP we can get much more granular backups and finally get back to TRUE incremental backups. Yay!

And that's it for the studio. As for me, I'm still vacillating on what I want to do for a side project. I'd love to develop a turn based strategy game, but the more I delve into what all it would take the more my brain hurts. Alas, I can't seem to find a game out there that is in need of someone with my skill set that captures my interest. Such is the price of being a fussy lil bitch I guess.
<![CDATA[And we are wrapped!]]>Thu, 12 Dec 2013 19:30:34 GMThttp://thechaosworks.com/blog/and-we-are-wrappedSo, I survived yet another season of producing South Park. And we had some damned good episodes this year. Can't wait to see how the game is received once it releases as well. Should be interesting. Now it's time for cleanup and maintenance and then a couple weeks of vacation. Can't wait!

<![CDATA[Just two more...]]>Fri, 29 Nov 2013 17:21:38 GMThttp://thechaosworks.com/blog/just-two-moreMan, even though I've been averaging a great deal less in hours per week worked, I am feeling this run even more then the last. I think the 10 month long break between runs was enough for my body to forget most of the crunch-time survival skills I learned over the years. Combine that with the extra three episodes of this run and it's like a formula for crushing my very soul. Ugh. On the down side I dropped out of the one project I found interesting due to needing to focus on Laura and life, but I'm bummed I had to. I still want a project to work on. I've been waiting for the Lads behind KoboldTouch to finish their iOS SpriteKit version but they've gone off on a whole new tangent that while promising also pushes back the day I can start working with it by another month or two. Meh. I guess I could spend this time creating an honest to goodness design doc. How crazy would that be?]]><![CDATA[25 hours and 695 shots later...]]>Wed, 20 Nov 2013 17:03:22 GMThttp://thechaosworks.com/blog/25-hours-and-695-shots-laterWe wrap episode 1708. Yeesh. I am fucking beat and we still have two more episode to deliver. Luckily I have next week off. Kinda. We are upgrading the UPS in our primary machine room on Monday and Tuesday of next week which means the week I'm supposed to have off isn't really a week off but 2 days off, one of which will be mostly spent on the road as we drive out to visit my wife's family in BFE (otherwise known as Hisperia). I will be happy when this run is over... I need a fucking vacation...]]>