DF talks with Nvidia’s VP of applied deep learning research.
At CES 2025, Nvidia announced itsRTX 50-series graphics cardswithDLSS 4.
The interview follows below, with light edits for length and clarity as usual.
The full interview is available via the video embed below if you prefer.
So why switch over to this new transformer model?
To start, how does it improve super resolution specifically?
You have to innovate; you have to try something new.
Transformer models have this great property in that they’re very scalable.
We can train it on much more data, get a smarter model and then breakthrough results.
What are some key image characteristics that are improved with the new transform model in the super resolution mode?
We’re always trying to push on all of those dimensions, and they usually trade off.
It’s easier to get more detail if you accumulate more, but then that leads to ghosting.
Those are the standard problems with any sort of image reconstruction.
Is there better potential with this kind of model also?
Is there a better trajectory with a transformer model?
Bryan Catanzaro:Yeah, absolutely.
And on Blackwell, we think it’s the best way to play.
How is the transformer model improving these specific areas?
We’re analysing failure cases, adding them to our training sets and our evaluation methodology.
We consider those model failures; the model is just making a poor choice.
It needs to, for example, decide not to accumulate if that’s going to lead to ghosting.
Yeah, that was one of my key takeaways about DLSS 4.
It was really hard to teach the prior CNN models about that.
I hope we get to do a dedicated look at ray reconstruction.
Bryan Catanzaro:I think that’s true.
Bryan Catanzaro:Well, because we get better results that way.
Technology is always a function of the time in which it’s built.
One is it uses less memory, which is important as we’re always trying to save every megabyte.
So we needed to make image quality better.
And so that required a different algorithm.
Bryan Catanzaro:I think this is a question of optimisation, engineering and user experience.
What has changed with regards to frame pacing, between DLSS 3 frame generation and DLSS 4 frame generation?
Can you talk a bit about it?
With DLSS, we want to improve on all those areas.
We’re excited about Reflex 2 because it’s a new way of thinking about lowering latency.
If that happens, we warp the image to the new camera position.
The trick with a technique like Reflex 2 is filling in those holes to make a convincing-looking image?
But you’re almost getting the same effect.
Bryan Catanzaro:It’s pretty fun to me.
It feels totally different playing a game with Reflex 2, it just feels so much more connected.
DLSS has something for everybody.
The ability to choose two, three or four inserted frames with frame generation.
Coming onto multi frame generation, what is the lowest acceptable input frame-rate for MFG?
I think the challenges really have to do with how large the movement is between two consecutive rendered frames.
Where do you see the future of frame generation?
Where does frame generation fit into that future?
Bryan Catanzaro:Well, I’m excited about 1000Hz monitors.
One that you brought up earlier is subsurface scattering.
And so you ask yourself, what’s the role of a polygon?
They just they know what it’s supposed to look like.
It becomes very expensive and really challenging.
So that’s what I hope we see in the future.
Bryan Catanzaro:That’s where we’re headed, for sure.
