What No Man Has Seen Before: Remastering Deep Space Nine to Maximum Quality

I’m done.

Most stories don’t begin at the ending, but that’s the only place to start this one. I’ve been working to remaster Deep Space Nine for the past nine months, ever since AI-based video upscaling software began to hit the market. After I saw how much improvement could be wrung out of some old MKVs, I decided I’d start over, using the original, superior, Deep Space Nine DVD source. Nine months later, I’ve accomplished what I set out to do: Create a method of remastering and upscaling Deep Space Nine that didn’t rely on hand-combing episodes to fine-tune deinterlacing algorithms while compromising on image quality to the smallest extent reasonably possible. I’ll be demonstrating the results all throughout this article.

I have created a 23.976 progressive version of Deep Space Nine, codenamed Rio Grande. While I have not yet checked the method against the entire show, it worked well on test episodes from S2, S4, S5, and S6. Special thanks to Cyril Niderprim, who found the hilariously simple solution below, and a pox on the small mountain of more complicated scripts I’d written. While Rio Grande is capable of throwing errors — I have included an example of such — there’s an alternative if it tangles up an episode in a way you don’t like.

I have also achieved an identical-quality 59.94 fps conversion, codenamed Orinoco. Orinoco preserves motion correctly in all scenes and is included here as an insurance policy. While it matches Rio Grande on quality, it packs 2.5x the frames and takes 2.5x as long to process and upscale. Rio Grande should work — but if it doesn’t, Orinoco will. If either of these methods proves insufficient to deal with an episode, we’ll come up with a custom way of handling it and I’ll wind up writing something about it and/or updating this article to keep track of special cases.

I’ve done a couple renders of the credits to showcase two different AI upscaling models Topaz offers: Theia Detail (at default settings) and Gaia-CG. Gaia-CG is below:

This article is not a step-by-step tutorial on how to perform this process suitable for literally anyone to follow. That will be its own project. I will, however, provide enough information that anyone with a passing knowledge of AviSynth should be able to recreate both approaches.

Theia-Detail credits below:

There’s a missing bit of frame for several seconds when the Bajoran freighter is panning around the station. This is a unique error — I’ve never seen it anywhere else on the show — and it pops up a lot, nearly regardless of what filters you use to adjust an episode. It’s always unique to the credits; you don’t need to worry about random blocks of content missing from the sides of your video.

Maximum Warp

Rio Grande and Orinoco make as few compromises on image quality as is reasonably possible. Every aspect of this process is designed to preserve detail at the deliberate cost of hard drive space until the final encode, at which point you can be as lossy as you like.

All of the clips you’ll see below were encoded in 0.0 H.264 until losslessly upscaled frame-by-frame. I encoded these clips in H.265 at a CRF of 6, which turned out to be ludicrous overkill. Live and learn. There’s a freeze-frame, just-noticeable quality difference between CRF 14 and CRF 20, while 25 is pretty ugly.

Upscaler Models: Avoid Artemis, 200 Percent Upscales

I’ve encoded at least one example of every upscaler mode Topaz offers except for Artemis-LQ and Artemis-HQ. These algorithms are completely unsuited to Deep Space Nine, and while they no longer produce literal garbage output when run against the show, they do not react well to its content. Consider them fundamentally unsuited to the task. I don’t care for Gaia-HQ’s output against DS9 very much, though I included one example below. That leaves the two Theia models and Gaia-CG.

I’ve always preferred Gaia-CG, so that’s what the majority of the encodes are in, though I’ve also showcased the Theia algorithm family. The Theia models offer tunable parameters for noise, sharpness, and detail recovery, and these can be very useful if you’re attempting to denoise an ugly patch of wall or carpet.

The 200 percent upscale option should be avoided. It creates errors that were not originally present in the source in some cases and it offers meaningfully reduced image quality compared to the 400 percent version. Be advised that the repair process has a habit of creating a seam down the left or right side of the image frame — you can crop it out with no trouble, but it’s something to be considered.

Topaz offers the option to upscale losslessly by image or via lossy MP4. All of the upscale videos you’ll see here are lossless PNG conversions.

How Much Additional Detail Did I Recover?

The idea that sent me haring off on this mad quest was that I could extract additional visual data if I returned to the source DVD rather than relying on old backup files. Here’s an example of my best upscale in January:

That was then.

While some of these improvements come from low-level improvements to the AI upscalers themselves, the upscaler has to be able to lock on to enough information to upscale an image without making hash of it. Imagine a screenshot where everything was rendered at DVD resolution except for Odo’s hair, which was done in 4K. It would look odd, to say the least. The more detailer the upscaler can get its hands on, the better the chance it creates something gorgeous on the back end. Going back to the DVDs gave the upscaler more to work with in the final process.

This is now.

As a reminder of how far we’ve come: The following three screenshots are the Defiant as it appears on the native DVD, the best upscale result I could achieve in early February, and my current best result today:

The Defiant in all her DVD glory. At least it beats Netflix. Let’s see what we can do.

February, 2020: It’s a big improvement, but I believed better was possible.

That’s more like it. Image from Orinoco encode.

Here’s my favorite Defiant shot from the entire show. I was really proud of how this turned out in April. It’s still quite nice.

I still like the way this came out, but it’s not the best we can do.

This is one place where I’m obviously getting help from a tweaked AI upscaler model. Better processing might clean up the model a bit differently, but some of the features on the hull are slightly different shapes. This image benefits from a side-by-side comparison to the other at maximum resolution.

The gains are real. Every time I compare an upscale based on the old MKV files I’d created years ago against the benefit of going back to the DVD, the DVD wins. If you want the best version of Deep Space Nine, invest in the DVD set.

So How Good is It?

With careful processing and good upscaling, it’s possible to give Deep Space Nine a clarity that I think approaches that of what’s typically referred to as “HD” content, though it’s still limited to the NTSC color gamut as opposed to later standards like Rec. 709. At its worst — allowing for some deviations from perfection — it’ll still look like the best damn DVD you’ve ever seen. At its best — and I consider the shot of Sisko up there to be one of the best — I’d argue that he, at least, comes across in HD levels of detail.

The processing steps I’ve used repair some baked-in damage to the source, but not all of it. There’s a frame sequence in “The Way of the Warrior” that literally looks as though something was spilled on the film, and that’s not something I know how to fix. All of the encodes below are from Rio Grande.

Way of the Warrior: It’s No Illusion

“The Way of the Warrior” contains two of the best battle scenes of Season 4. In the first, Sisko takes the Defiant to rescue a Cardassian ship under heavy fire from the Klingons. You can see the VFX team experimenting with what amounts to an over-shoulder kill cam — not exactly the kind of thing Next Generation was known for. It’s obvious the VFX teams had a lot of fun with the Defiant.

The second major battle scene of the episode is below. There’s an excellent callback to an event from the show’s pilot, when Kira attempts to bluff a squadron of Cardassian warships into believing the station is far better armed than it actually is.

This time around, things go differently. Upscaled in Theia Fidelity first, at default settings, followed by Gaia-CG. Gaia-CG is what I’ve used for screenshots in the past:

I’ve included clips of normal human interaction in this article specifically to illustrate that these processing techniques don’t just work on starships and battle scenes. Deep Space Nine is about far more than just CGI combat — it just happens to have some really great examples of the latter, particularly by late-90s standards. Gaia-CG below:

Paradise Lost

The Defiant takes on the Lakota in this battle scene. Elements of Starfleet were in the process of slow-walking a coup attempt. This clip deliberately shows what happens when Rio Grande doesn’t get a scene entirely right. While this shot mostly works, it’s noticeably jerky at the end. I’ve confirmed that this only affects Rio Grande — Orinoco plays back perfectly smoothly in this situation. If the small amount of jerk doesn’t bother you, Rio Grande is your oyster. If it does, try Orinoco.

Far Beyond the Stars

One of the benefits of science fiction is that it allows creators to play fast and loose with the question of time travel. Here, Benjamin Sisko is flung back into the 1950s — at least, in his own mind — and finds himself face to face with the ugly racial politics of the era. Upscaled in Gaia-CG:

In the Pale Moonlight

The greatest episode of Deep Space Nine concludes with a chilling soliloquy. Upscaled in Gaia-CG:

The same scene, rendered in Theia Detail at 10 percent sharpening, 20 percent noise reduction, and 20 percent detail recovery.

Sacrifice of Angels

Y’all knew we’d be back to this episode, right? In First Fleet (below, rendered in Gaia-CG), the Federation closes to battle with the advance forces of the Cardassian / Dominion military. I quite like the Galaxy-class maneuvers that bookend the clip. The Galaxy-class often looked fat and ungainly from certain angles, while others showed it to great effect. It gets more of the latter and less of the former on DS9 than it did on TNG.

This initial engagement doesn’t provide the opening the Defiant needs in and of itself. Starfleet decides to try and force a breach through enemy lines, even though they’re outnumbered, 2:1. This is the scene I chose to render in a number of different ways. First, Gaia-CG:

Next, here’s Gaia-CG at CRF=20. The quality loss is modest here, but I don’t recommend going much higher. The problem with starting from DVD source is that there’s not a lot of detail to lose before quality starts suffering, and the interpolated/upscaled information is only just so good to start with. When you start pushing the quality downwards past a certain point, it drops like a rock. CRF=25 is way too high. I wouldn’t go lower than 20.

Gaia-HQ:

Theia Detail, default settings:

And finally, Theia Fidelity, at 20 percent sharpness, 30 percent detail recovery, and 20 percent noise reduction settings:

Although I’ve chosen to focus on the later seasons of the show, this encoding and upscaling method works on the early seasons, too. Just keep in mind that the actual source material for the early seasons isn’t as good as what we got in later years.

That concludes the demonstration portion of today’s program. Now, let’s talk about implementation.

How to Encode Rio Grande

To encode using Rio Grande, rip the DVD using DVD Decrypter and generate a D2V index file using DGIndex or an equivalent tool of your choice. Next, fire up whatever AviSynth front-end you use, if you use one at all. I use StaxRip, personally.

The AviSynth script is as follows:

TFM()
TDecimate()
QTGMC2 = QTGMC(Preset=”Very Slow”, SourceMatch=3, InputType=2, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset=”Very Slow”, MatchPreset2=”Very Slow”)
QTGMC3 = QTGMC(preset=”Very Slow”, inputType=3, prevGlobals=”Reuse”)
Repair(QTGMC2, QTGMC3, 9)

That’s it. You also have the option to inject some noise back into the video in the process of running these scripts — season to taste, basically. Noise added to one of the QTGMC runs will still come through after repair. While Rio Grande isn’t 100 percent frame-perfect, it works perfectly on all but one of the 29.97 fps scenes I’ve tested against. If it proves to have more issues than anticipated, we’ll modify it — and that’s where Orinoco comes in. Technically, it’s possible to render this specific scene at 59.94 fps and then include it in the 23.976 fps version by assembling a VFR file when you compile your image sequence back into video. I might explore this in a future update.

The QTGMC runs above are required to achieve the improved visuals, and while I’m completely open to alternatives, nothing I’ve ever tried comes anywhere close to what this script can do. You can tune the above script for strength and processing time, but you’ll notice the quality hit from stepping down to “Slow” from “Very Slow” and “Medium” is unsuited to the task. If you like your content noisy and you don’t mind more combing artifacts (or have a different solution for dealing with them), “Slow” may be a better option.

The one downside to running QTGMC this way is that it can occasionally produce a small ripple in one part of a scene. I find it trivial to ignore and the only solutions I’ve found for it so far cause far more damage than the ripple itself. While I’d like to get rid of it, it’s currently a tolerated “feature.”

Defiant, rounding the station.

The “InputType=2” and “InputType=3” sequences are essential. Running a single QTGMC instance will not produce the same effect. Running two instances of “InputType=2” will not achieve the same effect. There are nine potential ways to combine input types in a repair run like this and I’ve tested every single one of them. The “2,3” combination is what works the best, from “Emissary” to “What You Leave Behind.”

Similarly, I’ve tested all 25 functional “Repair” modes AviSynth supports. You want ,9. If you don’t like that output for whatever reason, try ,1. Most of the others didn’t have much of an impact one way or the other, though a few of them broke content in interesting ways. I’ve revisited them more than once when testing new filters, just to see how final output varied by mode.

If you have suggestions on how to improve output quality further, by all means, drop me a line.

How to Encode Orinoco

Orinoco is not simple, but it works far better than I feel like I had any right to expect.

For starters, we’re creating two files, not just one. Encode your first version of your target episode using the following script:

TDeint(mode=1, type=2, tryweave=true, mtnmode=3, full=false, ap=10, aptype=2, slow=2)

This script orders TDeint to produce doubled frame output, to perform kernel interpolation, to attempt to repair a frame by weaving if the result is fewer combing artifacts than deinterlacing would create, and to only deinterlace interlaced frames. This preserves the progressive frames baked into the NTSC source. Type=5 was the only option that came close to Type=2 in overall image quality, and the bi-directional blending Type=5 introduces causes more frame blending issues at scene boundaries. Type 5 occasionally fixes a problem in Type 2, but it causes them far more often than it repairs them.

Now we set this clip aside and turn our attention to the other. Run the following script against your second clip:

QTGMC2 = QTGMC(Preset=”Very Slow”, SourceMatch=3, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset=”Very Slow”, MatchPreset2=”Very Slow”)
QTGMC3 = QTGMC(preset=”Very Slow”, prevGlobals=”Reuse”)
Repair(QTGMC2, QTGMC3, 9)

While similar, this script is not identical to the script we used for Rio Grande. Rio Grande’s QTGMC implementation is run in progressive mode. For Orinoco, we want to double the frame rate, bringing our base 29.97fps up to 59.94fps. Now that QTGMC and TDeint are at the same frame rate, we run a repair against them, too. If you run two clips in “Repair” mode at two different frame rates, it’ll work… ish. You’ll know if you’ve done this if you’ve got a weird seam migrating across your video.

This is a bass-ackwards approach to video editing, at least as far as I’m aware, but there’s a method to my madness. The reason people don’t typically just inject a bunch of frames to bring VFR 23.976 / 29.97 content up to 59.94fps is that the process the interpolated and blended frames that boost the frame rate can also create visual errors and other problems. This creates a catch-22: Leave those frames out, and the single-rate output is choppy. Keep them in, and some of your scene boundary changes are going to look decidedly odd as a fireball mushrooms from someone’s head, or two completely different locations are blended together. It’s not a great situation.

But there’s a way to fix the problem.

clip1=FFVideoSource(“C:\DS9S6D2\Sacrifice-TDeint.mkv”)
clip2=FFVideoSource(“C:\DS9S6D2\Sacrifice-QTGMC-ToPairWithTDeint.mkv”)
Repair(clip1, clip2, 9)

The reason we create the interpolated QTGMC file is to serve as a sharp, clean source against the Sacrifice-TDeint file. The reason I used QTGMC for this is that — as I’ve said before — I’ve never found an equivalent method for creating clean footage. If you want less noise in your video, use “Very Slow” or “Slow” as an alternative. I do not recommend “Medium” for this purpose.

You can run QTGMC against this output again if you want, but the footage will start to look a bit overprocessed. The Theia models can compensate for this to some extent, thanks to a fairly effective (and tuneable) denoise algorithm. There are multiple examples of Theia models in the videos below.

Quality-wise, Orinoco and Rio Grande get you to the same place and I often cannot tell the difference between them when I compare them frame-by-frame (not counting interpolated frames, obviously). Rio Grande is to be preferred — it’s faster and simpler — but I cannot guarantee that it’ll work on all 176 episodes of the show. If you run into an episode with erratic or stuttering motion when its created via Rio Grande, try Orinoco as an alternative and let me know you’ve hit a snag.

Flaws and Shortcomings

Orinoco and Rio Grande are not perfect. I’m still hunting for a method of eliminating the ripple. Small as it is, it’s basically the one problem left (so far!) that I haven’t solved. The best method I’ve come up with so far involves using MCTemporalDenoise, and MCTD appears to be both entirely single-threaded and causes me other headaches in terms of source quality.

If you want to experiment with MCTemporalDenoise, I’d recommend using up to seven pixels but applying a relatively weak effect (strength 2-3). This will not completely eliminate it, but it helps in certain places. If you don’t like the impact this has on your visual quality, investigate the Theia upscaling models and try applying 10-20 percent noise reduction. While you don’t want to go nuts, you can lightly smooth away some defects. I don’t like using one filter just to squash problems another creates, and with any luck Orinoco won’t be needed in the first place.

While my goal has always been to create a one-size-fits-all template for this project, I’ll record any adjustments or changes that need to be made to Rio Grande or Orinoco to deal with problematic episodes, unless that episode is Move Along Home.

Next Time on… the Deep Space Nine Upscale Project

Now that I’ve finished this project, I get to actually start ripping episodes and encoding them. I’m going to write a set of tutorials that give an actual guide to how this works, build a benchmark for workstations, and solve any per-episode problems that need fixing. I’ve come up with a completely free method of upscaling that, while not as good as Topaz, presents a genuine improvement. I also want to look into PAL.

ViacomCBS doesn’t think Deep Space Nine is worth remastering. I say it’s wrong. There’s a limited amount of improvement we can squeeze from a DVD, which is why you don’t see me referring to the show as “4K” or “UHD”, even though it’s common to see those terms tossed around in projects like this. You can upscale Deep Space Nine to 4K resolution, but there’s no way (yet) to squeeze 4K quality out of a DVD source file. This is an imperfect solutions — but it’s light-years beyond any previous version of the show that’s ever existed. If it turns out that Rio Grande does throw more than very occasional 23.976 fps errors, we’ll find a solution for those episodes. There’s every indication that Nvidia’s upcoming GeForce RTX 3080 is going to make this kind of upscaling much faster, which will reduce the pain of a 59.94 conversion.

After today, no Star Trek fan has to wait for ViacomCBS to decide it makes financial sense to invest in its own back catalog. It’ll cost some money for the DVDs and the software, and a lot of compute time to get the work done (8-10 hours per episode on an RTX 2080 at 23.976, 2.5x longer at 59.94fps). Will most people do this? No. Most people will watch the show on Netflix, where the quality is much lower than what they’d get on DVD.

But as of today, nobody has to do that again. As of today, you can see the work of the VFX artists and actors who created in this show the way it was meant to be seen. Orinoco and Rio Grande aren’t perfect, but they illustrate just how much additional quality can be wrung from late-90s DVD source. GPUs are getting faster. AI upscalers are getting smarter. Even if the computer or GPU you own today isn’t capable of tackling a project like this, your next one very well might be.

May the Prophets guide you,

Hruska out.

I owe thank-yous to Stan Pennington, Gary Huff — man, Gary, I probably owe you a hearing aid — Cyril Niderprim, Robert Hallock, Andrew Swan, David Berry, Dan Greeney, Mark Renoden, the folks at Black Magic, my long-suffering fiancée who has listened to me talk about this topic for most of a year and contributed insights at numerous points, Ben Carter, Joe Robinson, multiple helpful individuals at Doom9, and a lot of other people. If I forgot names, I’m sorry. If you helped me, thank you. Most of all, thank you to the talented actors and artists who created Deep Space Nine in the first place. 

Now Read:

• DS9 Upscale Project Update: What I’ve Been Working On
• Deep Space Nine Upscale Project Season Finale: What We’ve Brought Ahead
• Deep Space Nine Upscale Project (DS9UP): Technical Goals and FAQ

sourse ExtremeTechExtremeTech https://ift.tt/3jXArx4