Video encoding and progressive output on HDTV
I've finally upgraded my HDTV from the original Mitsubishi WS-55411 to a new HP MD5880N 58" 1080p DLP display. In doing so, I am going to be really excited to get my HTPC all hooked back up again and test out larger desktop sizes, as well as see how the XBox content from XMBC looks on it.

It has also prompted me to get back into learning more about video encoding and interlacing/deinterlacing formats. Especially concerning some of my DirecTV standard def content with massive field order flipping and non-standard GOP length encodes. Most irritating is animated source material, like South Park with obvious interlaced combing effects on computer and HD progressive displays. I'm finding that standard def content and the way they variable encode GOP and do stream/field order flipping can be devastating for some progressive display's ability to gracefully handle the deinterlacing correctly. For good reads on general video editing, check out Nicky Guides and Home Theater HiFi Progressive DVD guide.

I found a cabinet that seems like it will fit, and it has closed glass doors so that my 1 and 3 year old won't be fiddling with the components.


The HP MD5880N claims to support the following formats:

• 2 HDMI (480i/p, 720p, 1080i/p*)
  
• 2 Component (480i/p, 720p, 1080i/p**)
  
• 1 VGA (480i/p, 720p, 1080i/p***)
  
• 1 USB
  
• 1 Serial
  
• 3 Composite/Svideo
  
• 2 Cable
  
• 1 CableCard
  
• 5 Analog RCA Audio
  

* Supports 24, 30 and 60fps.
** Supports 24, and 30fps.
*** Supports 24, and 30fps but we have heard that the maximum resolution is 1280x1024p60

You've probably heard/read before, how 3:2 pull down interlacing or "telecine" conversion works. You end up with combing effects where the exclamation points are in the diagram below when converting 24fps to 30fps interlaced NTSC content (what is called 480i and has been the standard def broadcast for well over 30 years). The artifact comes from essentially having two halves of different frames combined to get an interpolative frame in between when stitched back together to make a progressive frame for display on a high definition display. In most cases, this doesn't cause too much problem, but when there is a lot of "differences" between two adjacent original frames (like during fast motion content or scene changes), you could end up with some VERY nasty displays on progressive output:




Another example of a problem with displaying telecined content on a progressive display is detailed below, where a scene from "Battlestar Galactica 2003" shows a scene transition on one of the interlaced transition frames when doing 3:2 pull-down conversion from 24fps to 30fps. The first image is the final interlaced frame. Following that are the individual half-frame A and B fields that came from two separate 24fps content frames when being telecined (essentially a merger between two very different fields across a "scene transition" from the diagram above where the exclamation points are).



What isn't obvious from all this, is if this new HP TV supports actual native 24fps display play back, or merely supports converting from 24fps FILM MPEG-2 input content to 30 or 60fps NTSC progressive content as detailed in the flow below of a scene from "The Fifth Element" (image taken from Home Theater HiFi):


A problem develops with this process when the progressive display process performs an inverse telecine back to the original 24fps and essentially drops every third field of five, then returns to the original 24fps full frame content. Then a DIFFERENT progressive 3/2/3/2 full frame duplication is made to end up with 4 frames -> 10 frames conversion (turns 24fps into 60fps). The problem with this, is it ends up making motion sensitive footage a bit "jerky", because you spend 3/10ths of the time on one frame and 2/10ths of the time on the next.

So, why can't today's HDTV's simply display the raw native 24fps original FILM content at 48fps or even 72fps and keep a more symmetrical "weighting" of each frame? What isn't clear to me, is does my new TV actually essentially DO this (and do it even on component inputs from the specs above), or does it simply accept this as an input and effectively do the conversion from 24fps to 60fps with jitter (along with upsampling to 1080p) internally, as per the diagram above?

I need to find a good DVD test pattern/encode that I can run through my DVD player (and xbox media player) and test how my DVD and TV deinterlacers work. I've also heard that some of these newer TV's with 1080p support may "cheat" and drop the B-field and simply then line double the A-field (or vice-versa), rather than deinterlace the 1080i input content to a fully progressive 1080p signal for display. If it were to do this, a simple single line resolution test pattern at 1080 horizontal lines should quickly expose this. I need to find content like this for more than just test patterns, but also various nasty DVD encode/frame rate (it should be possible to expose 24fps converted to 30fps progressive as well, since you would have jerky motion). I'm not sure if Digital Video Essentials DVD will do what I want or not...Probably not.
- posted by Jeff @ 10:54 PM

 
Update!
Well, it has been a long time since I last posted here. There are a couple things that I've been doing:

• First, I really don't like my Mitsubishi WD55411 55" HDTV for use as a HTPC display. For one thing, it only supports up to 1080i (which means the best desktop I can get is a 1920x540 display). This also has the poor side effect of needing me to run a custom display setting, and is analog RGB, which has pincushion and window size problems. And good luck getting a lot of newer control panels and dialog boxes to fit on a 540 vertical display. After upgrading my MAME Cabinet to a 1280x768 LCD display over DVI, I'm amazed at just how much better the digital interface is, and how nice having at LEAST 768 lines of desktop is. Now, I really want to buy one of the new 1080p DLP displays out from Mitsubishi or Samsung (I know...the dreaded giant...but they sure make nice stuff). Unfortunately, the Mits units don't support >720p over the HDMI interface (same with Samsung) and have no other PC interface. At least the Sammy has support for an RGB (back to analog, though...sigh) that will support 1920x1024 display. So, NEITHER supports digital 1080p input, so you are screwed in the future if there is a 1080p HDMI source!
  
• Next, my DirecTV 40-hour Tivo that I had hacked with a 160GB HDD, turned off encryption, enabled network support, extraction to PC for video editing and burning to DVD, scheduling over the web, and many other enhancements, finally went T.U. The drive was ALWAYS very noisy compared to the 40GB, and I'm not sure if the drive is bad or if a sector just went bad and needs to be reinstalled. Anyway, I put the old back up drive back in, and got the update to the new 6.2 Tivo software (much faster menus and folder support), and have been reading all the latest ways to hack it again. Got a 200GB Maxtor for $79 at Staples and am in the middle of hacking it again from 6.2. Should be able to enable HMO (Home Media Option) and MRV (Multi Room Viewing).
  
  Here is the best way to do the hack:
  
  1. BACKUP your drive or COPY to new mirror drive before doing the hack with MFSBACKUP/MFSRESTORE. In my case, this involves getting the LBA48 Linux boot CD with enhancements from PTV Upgrade. I connected original drive to Primary Slave (/dev/hdb) and new 200GB drive to Primary Master (/dev/hda). Then once you boot the LBA48 Boot Disk (BE SURE NOT TO BOOT INTO WINDOWS XP/NT/2000 or Windows will attach some data to your new virgin drives and make it UNUSEABLE for TIVO), you can check some stats on the old drive with:
     # mfsinfo /dev/hdb
     
     And then upgrade/copy the drive image in one step with:
     # mfsbackup -aqo - /dev/hdb | mfsrestore -xpi - /dev/hda
     
     
  2. Follow Gunnyman's UnGuide to hack 6.2. This assumes you already have a 6.2 kernel to start with and you've backed up stuff
     
  
  
• I also just got a Hughes HR10-250 hidef 250-hour DirecTV DVR for $199 (after $100 mail in rebate). I LOVE it for the main area TV, but it has the old 3.5 OS on it and doesn't sound like DirecTV is going to upgrade it. But most people have managed to hack it to work, so I'm planning to upgrade this unit as well to enable the HMO/MRV and all the other stuff. Saw an add for a 250GB Maxtor for $79 from CompUSA.

- posted by Jeff @ 10:24 AM

 
Useful DirecTiVo files

• endpad Allows your TIVO to gracefully "pad" recordings only if it won't cause you to miss other shows. I record a lot of shows back to back, and with 2 tuners, it works pretty well, but if there are two seperate channels that I'm recording shows on back to back, it will skip recording interleaved shows if I try to pad. This TSR gracefully adds padding if nothing else is going on.
  
• TivoWeb Plus Gives you great web based access to your tivo (including a virtual remote to control everything with).
  

- posted by Jeff @ 10:08 PM

 
Filling up drives
Well, I now have a 160GB DirecTivo drive almost full (OK, it is only recognizing 124GB of the drive), a 200GB drive on my main system COMPLETELY full, and a 160GB drive on my HTPC almost full of movies. Star Trek, ST:TNG, Friends, Seinfeld, Band of Brothers. Man, this stuff really starts adding up. I don't think I can even begin to burn these to DVD and help make space. Band of Brothers (10 episodes) is going to run me around 12GB alone (and that is pretty tight compression coming straight from DTV in MPEG2 format). I'm not happy with how poor the compression looks when played back with a player. Not sure how/why it is different (probably because I'm stretching and blowing up the leterbox version in a 4:3 display to fully fit my 16:9 widescreen TV). Anyway, I can't wait for DivX/MPEG4 and 1080i HDTV digital broadcasts to take hold (hopefully there will be a way to record those broadcasts and get them off the recorder). But, boy, if I thought I had disk space and network bandwidth problems now...that is going to be a REAL increase.

I'm still trying to figure out the best way to re-encode the DTV MPEG2 broadcast. Right now, the TY (proprietary Tivo encoded MPEG2 bastardized format) has errors when multiplexed into MPEG2 (not really a re-encode, just a header fix/change done by TYTOOLS). VirtualDub-MPEG2 and TMPGEnc has audio sync problems, Windows Media Player gets frozen/stuck on video frames (but the audio keeps playing), and SnapStream BTV player also has glitches. Seems like it plays fine on a DVD player when you mux it to VOB and burn it with TYTOOLS and PowerDVD and WinDVD both play it back just fine (as MPEG2) as well.

Anyway, I really don't like how the comercial edit/cuts are done in TYTOOLS, and wish I could stream copy it into VirutalDub (love the edit/cut functions in there). I also wish I had a DVD player that supported native MPEG4/DivX. What am I talking about? I do...my HTPC. Hum...
- posted by Jeff @ 11:04 PM

 
Transcoding the results from Tivo
I've spent a little time now, trying to get a good transcoding from the bastardized MPEG2 format that the Tivo stores the DirecTV digital stream in. They are typically .ty files on the DirecTivo, and I have to FTP them over with Tytools, which is a start to finish toolset to join the tivo files together, remultiplex the fragmented video and audio files back together, and (if you want) setup a VOB burn to DVD. Mostly, I've been spending my time, just trying to get it into a good MPEG2 format, and then possibly transcode to Divx or XVid. Problem is, most of the players (WMP, BTV, VirtualDub-MPEG2) all seem to have audio sync problems or skip/stutter with the post tytooled MPEG2 stream. PowerDVD and WinDVD both seem to play it ok, though, but the quality, when played back on my HDTV from the PVR is very blotchy (you can see very nasty dithering in the colors...probably because the desktop is at 32-bit, and Tivo must be storing at some lower color). Either that, or it is just quantitization errors that is being magnified. Hopefully upcoding to Divx or XVid will help.
- posted by Jeff @ 8:50 AM

 
The curse of becoming "mainstream"
It seems that the fate of Tivo hacking has the gloomy cloud of irony hanging over it. All the improvements, work, and efforts of the community to make it easier to hack the Tivo, while wonderful for those of us that don't have 12 hours/day to work on and figure out all the intricacies of the nuances of the Tivo, also means that more and more people are going to be able to be able to actually do this. Seems all well and good on the surface, but the very work these folks are putting into guides and tools, may actually undermine the whole tapestry. Because, as it all becomes easier and easier to do, and more people do it, there is going to be more of an effort put into shutting this sort of thing down and defeating this by the Tivo and DirecTV folks. Sad fact, but probably true.
- posted by Jeff @ 8:16 PM

 
Newbies Tivo Video Extraction FAQ
AltOrgPhpWiki - Newbies Extraction is a good site for learning how to extract the files off your TIVO onto a computer network.
- posted by Jeff @ 5:51 PM

 
Tivo Will Die!
Yahoo! News - TiVo Will Die: "It's always hard to write an obituary, especially when the subject is still alive. It's especially hard for me, because I love the little guy like a brother. But, alas, TiVo (news - web sites) will die. "

Cool article. And it does sort of parallel some of my own thoughts (at least on the idea to stay digital once you are getting digital broadcasts and not analog capture the stuff).
- posted by Jeff @ 5:51 PM

 
TMPGE utilities and how-tos
Tools has a great link of how-to and utilities for TMPGEnc.
- posted by Jeff @ 12:46 PM

 
A Beginners Guide for MPEG-2 Standard
A Beginners Guide for MPEG-2 Standard offers a VERY good indepth look at how MPEG2 is broken down, stored, and compressed.
- posted by Jeff @ 1:17 PM