News NVidia Streamline to combine NVidia DLSS, Intel XeSS, and maybe AMD FSR in one interface

[Heartbreaker]

As soon as I saw Intel XeSS, and handful of supporting games, I was thinking we really need an abstraction layer so developers can write once, and support all the advanced Image scaling methods.

I was surprised that NVidia is the first to deliver this abstraction layer:

Nvidia Streamline Aims to Simplify Developer Support for All Upscaling Algorithms

Streamline is an open-source framework and Intel is already on board.

www.tomshardware.com

Thankfully, Streamline is both open-source and can accommodate super-resolution technologies from diverse hardware and game engine vendors. Intel is already on board, but we have no word from AMD on its plans.

...

"Intel believes strongly in the power of open interfaces," said Andre Bremer, VP of AXG and director of game engineering at Intel. "We are excited to support Streamline, an open, cross-IHV framework for new graphics effects. This will simplify game developers’ integration efforts and accelerate the adoption of new technology."

Nvidia says that Streamline is available today on GitHub supporting both DLSS and DLAA (Deep Learning Anti-Aliasing). NIS support is "coming soon." The framework is extensible beyond super-samplers, as evidenced by Nvidia including its Real-time Denoiser. Please note that Streamline supports DirectX 11 and DirectX 12 titles, but Vulkan compatibility is still in the beta testing stages.

 

[Aapje]

From a developer's standpoint, why would you ever bother with DLSS or XeSS if FSR 2 hits your entire target audience?

DLSS is a strong brand right now, so I do see it as a selling point. If FSR 2 and DLSS are extremely close in how well they do, I expect that different upscalers do better in different games and that people will have a preference for the artifacts of a specific upscaler. And you have the fan boys...

However, I doubt that this incentive is strong enough for developers to implement DLSS separately and quite plausibly even to support it in addition to FSR 2 with Streamline, where it doesn't cost that much developer time. After all, every additional option means more testing and more risk of bugs with specific combinations of settings and hardware that they didn't test. I can even see developers use Streamline, but then only enable FSR 2. That way they can switch if during development, another upscaler turns out to work better, but don't have the extra testing effort and risk of supporting 2 or 3 upscalers.

Ultimately, one of the main goals of AAA development is to get good reviews, which is why software with a ton of bugs gets released in the first place (because reviewers don't tend to care about bugs all that much). Reviewers also prefer fewer things to test, so having only one upscaler will actually earn them browny points with the reviewers.

My prediction is that DLSS is going to die and FSR 2 is going to take over, assuming that FSR 2 delivers on the promises. I suspect that XeSS will lag too far behind in quality and release date. Both Nvidia and AMD needed a version 2.0 to get good, but XeSS 1.0 that works on all hardware hasn't even released yet.

 

[Heartbreaker]

Thats not how it works. Game Devs DO NOT WANT three different upscalers. Just like display manufacturers didn't want two adaptive sync systems.

No, this is more like Ray Tracing where you have a common API, and each card maker does their own implementation. Would you claim developer want to write completely different Ray Tracing code, for each brand of card?

You are making it out as though devs just need to push a button and bam they support three different upscaler methods.

Essentially, that is what it does. That is the entire point of common API/Interface, is you write one set of application code, and it supports all options, by calling the appropriate implementation behind the scenes. Another new manufacturer could enter the picture, and use the common interface, and all four would be supported with no more developer work.

It's not push a button, but is similar work to supporting One Temporal Scaling, and you support all three, or four, or ten. That is why we use common interfaces for everything. This is first year CompSci stuff.

That is is the whole point. Write the application code once with a common interface, and support all implementations.

What do you think it does?

 

[Stuka87]

No, this is more like Ray Tracing where you have a common API, and each card maker does their own implementation. Would you claim developer want to write completely different Ray Tracing code, for each brand of card?

Ray Tracing is handled by DirectX. There is no vendor specific API. No proprietary system that uses a different implementation to accomplish the same thing.

Essentially, that is what it does. That is the entire point of common API/Interface, is you write one set of application code, and it supports all options, by calling the appropriate implementation behind the scenes. Another new manufacturer could enter the picture, and use the common interface, and all four would be supported with no more developer work.

It's not push a button, but is similar work to supporting One Temporal Scaling, and you support all three, or four, or ten. That is why we use common interfaces for everything. This is first year CompSci stuff.

That is is the whole point. Write the application code once with a common interface, and support all implementations.

What do you think it does?

You keep going on with this "no additional developer work". But that's not true. DLSS 2.x requires far less work from a developer to implement compared to DLSS 1.0, but they still have to do work.

I am just baffled as to why somebody is pushing so hard for a closed source, proprietary system that only works on very specific hardware to exist when an open standard is available. Its like saying AdaptiveSync should not have become the standard, and instead every single monitor should have nVidia's hardware inside their displays because then "everybody wins".

DLSS will ultimately go the same route as every other nVidia vendor lock in technology. nVidia comes out with it first, it only works on their latest generation of cards as a way to force people to upgrade or migrate from another brand if they want that feature. Then a short ways down the road, an open standard will replace it, and nVidia will begrudgingly drop their own system and support the open standard instead. Just in time for them to come out with something else.

 

[Dribble]

Really this just means each gpu manufacturers temporal upscaling solution will be available in all games, which is a win.

I think most of the hate is because the AMD fans strongly suspect that DLSS will be better, so it will remain an incentive to buy Nvidia cards. This is just backward thinking for consumers - what we want is all 3 manufacturers to be pushed by each other to do better, not to take away the competition so a substandard solution doesn't look so bad.

The only reason FSR exists is because of DLSS, competition is good.

I recently got told an old russian proverb that applies to fanboys like this: A man meets a genie is told he can have anything he likes but his neighbour will get twice what he is given. He asks to be made blind in one eye.

 

[darkswordsman17]

Really this just means each gpu manufacturers temporal upscaling solution will be available in all games, which is a win.

I think most of the hate is because the AMD fans strongly suspect that DLSS will be better, so it will remain an incentive to buy Nvidia cards. This is just backward thinking for consumers - what we want is all 3 manufacturers to be pushed by each other to do better, not to take away the competition so a substandard solution doesn't look so bad.

The only reason FSR exists is because of DLSS, competition is good.

I recently got told an old russian proverb that applies to fanboys like this: A man meets a genie is told he can have anything he likes but his neighbour will get twice what he is given. He asks to be made blind in one eye.

You do know there's nothing preventing that (well other than Nvidia's own self-imposed nonsense), already, right? You're also hilariously naive if you think that's actually what is happening here. And that's literally the only reason this is happening, is because both AMD and Intel are developing versions that can run on any hardware and not locked down like Nvidia does to try and push people to keep buying their newest GPU.

I like how you ignore the Nvidia fans trying to say FSR would never even work because Tensor cores are required for this stuff while having a moan that AMD fanboys supposedly are just mad that DLSS might be better. Not that I'm surprised at the double standard, same as it ever was.

Er, you know devs have been working on implementing upscaling on consoles for over a decade, right? The only reason DLSS exists is because Nvidia needed something to market their AI focused hardware towards consumers, coupled with a need to upscale because of the dumb decision to push half-baked ray-tracing that tanked performance.

Most likely this just ends up like Gameworks where Nvidia pays devs to let them shove in their implementation (just because Nvidia might offer an open source version doesn't prevent them from doing a closed source one), where it ends up black boxed and often causing problems, sometimes straight up explicitly screwing customers because of just flags of having other brands' GPU or even sometimes Nvidia's own just older ones. Meanwhile Nvidia gets to claim they're supporting others but its just effs things up for customers of the games they do it to. Nvidia basically has a 20 year history of doing this type of junk, which is why people are calling it out. Because we've seen this play out already. And every single time we get people like you going "AMD/ATi fanboys just mad, this is really the greatest thing ever!" wherein it plays out just like the people calling it out say. And then the people like you try and pretend it never happened (often straight up lying about previous instances), or if you admit the last time they did it was poor, you just forget and pull the same nonsense the next time Nvidia hypes something like this.

Further, this is just going backwards once again. The entire point of DX12/Vulkan was to undo all the tacked on middleware and translation to try and support it, which was sapping performance and actually limiting progress of graphics. I called this when they started pushing this weak half-baked ray-tracing, that we'll just go back to the way things were to try and get performance somewhere acceptable.

DLSS is a strong brand right now, so I do see it as a selling point. If FSR 2 and DLSS are extremely close in how well they do, I expect that different upscalers do better in different games and that people will have a preference for the artifacts of a specific upscaler. And you have the fan boys...

However, I doubt that this incentive is strong enough for developers to implement DLSS separately and quite plausibly even to support it in addition to FSR 2 with Streamline, where it doesn't cost that much developer time. After all, every additional option means more testing and more risk of bugs with specific combinations of settings and hardware that they didn't test. I can even see developers use Streamline, but then only enable FSR 2. That way they can switch if during development, another upscaler turns out to work better, but don't have the extra testing effort and risk of supporting 2 or 3 upscalers.

Ultimately, one of the main goals of AAA development is to get good reviews, which is why software with a ton of bugs gets released in the first place (because reviewers don't tend to care about bugs all that much). Reviewers also prefer fewer things to test, so having only one upscaler will actually earn them browny points with the reviewers.

My prediction is that DLSS is going to die and FSR 2 is going to take over, assuming that FSR 2 delivers on the promises. I suspect that XeSS will lag too far behind in quality and release date. Both Nvidia and AMD needed a version 2.0 to get good, but XeSS 1.0 that works on all hardware hasn't even released yet.

I don't think any of them will dominate. DLSS will lose out because there are other options that don't require lock-in to Nvidia's latest. The industry will work on doing their own implementation and it'll basically not even matter what hardware you have as it'll be baked into games in ways that don't need to support specific implementations. And the cycle will start all over again with whatever the next best thing becomes.

The problem is the entire industry is broken. We get all sorts of analysis of this technical stuff, with never ending "next best thing ever" nonsense, only for, as has always been the case and always will be, the art direction matters 1000% more than any of these graphics features.

 

[Heartbreaker]

I like how you ignore the Nvidia fans trying to say FSR would never even work because Tensor cores are required for this stuff

All FSR is today, is a basic decades old resize algorithm, mixed with some sharpening.

Nobody every thought you need Tensor cores for that kind of basic image manipulation.

FSR2 will step to a more modern Temporal scaling algorithm. Though again this has been around for a while and AMD won't be using Neural Networks (NN) to aid in the reconstruction, so even there, no expectation that NN would be necessary

So it is quite likely that FSR 2.0 won't quite match up with DLSS and XeSS because the latter two will use NN to aid in reconstruction.

 

[coercitiv]

All FSR is today, is a basic decades old resize algorithm, mixed with some sharpening.

What @darkswordsman17 was addressing had to do with some very interesting hand waving from @Dribble regarding FSR 2.0:

This is just another temporal upscaler using shaders of which there are several, to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware, and secondly write the maching learning software.

This hot take on AMD requiring specialized hardware for a proper DLSS competitor was wrong from the start, as Intel themselves have stated XeSS doesn't require specialized hardware. Moreover, the current take that AMD fans oppose Nvidia Streamline for fear of DLSS superiority is also fundamentally flawed: if Nvidia knew their tech is clearly better they would not give their competitive advantage away by opening access to AMD/Intel technology in every game that could be DLSS exclusive. Having FSR available on every super-sampling enabled game does not sell more Nvidia cards.

The problem with Streamline is not technical, this is not about whether the software can help the ecosystem by providing a healthy choice of super-sampling algos. The problem is one of TRUST in Nvidia, the self appointed warden of the common interface. This is a fiercely competitive company that never gives away a competitive advantage, they only do so when their edge is lost. Based on precedents they're also willing to sabotage the consumer if it makes their technology win benchmarks or alternatively if they get a marketing advantage from it.

If Streamline is such a win-win initiative (and it may still be), why not bring it under the care of more neutral parties such as Microsoft and/or big game engine developers. Why not ship it as a true standard?

 

[Heartbreaker]

This hot take on AMD requiring specialized hardware for a proper DLSS competitor was wrong from the start, as Intel themselves have stated XeSS doesn't require specialized hardware.

We have no idea at all what XeSS delivers, and initially it will only run on Intels proprietary MXM cores (Intel's Tensor core equivalents), so it will be using specialized HW.

When there is a DP4a solution for competitors, it will disadvantage competitor HW. This is not a one size fits all. It's proprietary solution with two tiers. The top tier that runs on Intel specialized HW it was initially designed for, and a DP4a secondary tier, running slower for everyone else.

So it's pretty hard to use that as proof of anything, until that DP4a solution is eventually delivered and we see how badly competitors are disadvantaged.

It's funny to see people freaking out about an Open Source API, which is nothing but a common interface, while lauding a proprietary closed source XeSS that relies on specialized HW, that will eventually have a DP4a solution that will almost certainly give competitors a performance disadvantage if they use it.

I can only image if NVidia had released a Generic implementation of DLSS, that didn't run on Tensor cores (and thus suffered a big performance hit on competitor HW), it wouldn't be so warmly received. Instead the same people would be screaming its "proof" of NVidia sabotaging competitors with lower performance.

Moreover, the current take that AMD fans oppose Nvidia Streamline for fear of DLSS superiority is also fundamentally flawed: if Nvidia knew their tech is clearly better they would not give their competitive advantage away by opening access to AMD/Intel technology in every game that could be DLSS exclusive. Having FSR available on every super-sampling enabled game does not sell more Nvidia cards.

That "logic" doesn't really hold up at all. It actually makes as much, if not more sense that if NVidia knew their tech is clearly better, they would invite the direct comparison of having competitor solutions available in every game, thus making it easier to clearly demonstrate those wins. Similarly if they thought their solution wasn't that good, they would want exclusivity, to duck those direct comparisons.

At best you could argue, we really don't know what this says about NVidia's confidence in the superiority of their solution.

If Streamline is such a win-win initiative (and it may still be), why not bring it under the care of more neutral parties such as Microsoft and/or big game engine developers. Why not ship it as a true standard?

That would be better, to end the whining about NVidia sabotage, if nothing else, but you need to find someone willing to take over.

This it's NOT an exciting project to maintain. It's just an API. If I was looking for a cool open source project to work on/maintain, this wouldn't be it. Best bet is if Microsoft/Khronos just grab it and make their own independent use of it. Microsoft probably wouldn't bother making it open source, but just pull it into Windows and use as they see fit.

This has an MIT license. Anyone can take it, and literally do anything they want with it, and it could be pulled into game engines in the meantime and just used/modded without the game companies officially taking it over.

 

[Heartbreaker]

The problem is one of TRUST in Nvidia, the self appointed warden of the common interface. This is a fiercely competitive company that never gives away a competitive advantage, they only do so when their edge is lost. Based on precedents they're also willing to sabotage the consumer if it makes their technology win benchmarks or alternatively if they get a marketing advantage from it.

This is seems to be the main opposition for many people, So I will address it separately.

A common interface is NOT complex code, and It's open source, so both AMD and Intel (or anyone else) will be able to examine the code and could and would immediately call out any sabotage, so this concern is really more the typical internet forum conspiracy nonsense, than a real concern.

 

[Aapje]

while lauding a proprietary closed source XeSS

Intel has promised to open source it eventually.

 

[coercitiv]

This is seems to be the main opposition for many people, So I will address it separately.

This is indeed the pressure point, however your explanation continues to focus on the technical aspect, which is (almost) irrelevant. There are ways to go about this that do not require Nvidia to sabotage the tech, but rather sabotage the purpose.

For example, there is a considerable number of AAA titles every year that come out with exclusive RTX & DLSS support. According to you, should AMD and Intel join Streamline:
A) these Nvidia exclusive titles will be built with Streamline to offer consumers the win-win scenario of multiple super-sampling tech support
B) these Nvidia exclusive titles will continue to be RTX & DLSS exclusive, as they bear the fruit of Nvidia marketing efforts

Remember FSR or any other competing tech does not need to be objectively better than DLSS, it needs to be subjectively close while providing consumers/devs with other important perks such as wider range of supporting hardware. It's the same Freesync vs. Gsync all over again, and Nvidia is clearly attempting to salvage the situation. The only reason Streamline exists is to be used as a wedge to make room for DLSS and prevent early asphyxiation of the tech.

You'll understand more when FSR 2.0 gets benchmarked on a 3090 Ti.

 

[Heartbreaker]

This is indeed the pressure point, however your explanation continues to focus on the technical aspect, which is (almost) irrelevant. There are ways to go about this that do not require Nvidia to sabotage the tech, but rather sabotage the purpose.

For example, there is a considerable number of AAA titles every year that come out with exclusive RTX & DLSS support. According to you, should AMD and Intel join Streamline:
A) these Nvidia exclusive titles will be built with Streamline to offer consumers the win-win scenario of multiple super-sampling tech support
B) these Nvidia exclusive titles will continue to be RTX & DLSS exclusive, as they bear the fruit of Nvidia marketing efforts

Remember FSR or any other competing tech does not need to be objectively better than DLSS, it needs to be subjectively close while providing consumers/devs with other important perks such as wider range of supporting hardware. It's the same Freesync vs. Gsync all over again, and Nvidia is clearly attempting to salvage the situation. The only reason Streamline exists is to be used as a wedge to make room for DLSS and prevent early asphyxiation of the tech.

You'll understand more when FSR 2.0 gets benchmarked on a 3090 Ti.

So reading through that, your perceived negative of Streamline, is that it might keep DLSS alive?

Exactly who is harmed by that?

 

[blckgrffn]

DLSS is a strong brand right now, so I do see it as a selling point.

My prediction is that DLSS is going to die and FSR 2 is going to take over, assuming that FSR 2 delivers on the promises. I suspect that XeSS will lag too far behind in quality and release date. Both Nvidia and AMD needed a version 2.0 to get good, but XeSS 1.0 that works on all hardware hasn't even released yet.

This is s damn near exactly what I was thinking.

DLSS is some sort of holy grail right now, but....

If FSR2 is close enough to another method that notably does have graphical drawbacks but is vendor agnostic, as a developer that's the *only* one worth doing.

Implementing an API that allows a method to be selected is still going to require testing and some extra complexity. Why bother?

I mean, I know why one might bother. Sponsorship money. Beyond that, simple is best.

Physx again.

 

[coercitiv]

So reading through that, your perceived negative of Streamline, is that it might keep DLSS alive?

My only perceived negative is people trying to present Streamline as a consumer oriented move. I have nothing against DLSS as a proprietary tech giving Nvidia a well earned competitive advantage, but that would stop the second they attempt to delay the adoption of open source solutions that can benefit a much wider consumer base. My interest as a consumer is to have access to competitive super-sampling tech no matter the brand of choice for my next purchase, or the one after that. Today I have no idea what brand my next video card will be.

So I will keep my reservations with respect to Streamline, as Nvidia no longer enjoys the benefit of the doubt when it comes to these matters. I'll wait and see how they go about using Streamline in conjunction with their usual marketing efforts, particularly sponsored games. BTW this applies to AMD and Intel as well, that is if they join Streamline and then start with "exclusive" FSR/XeSS shenanigans.

 

[DrMrLordX]

Exactly who is harmed by that?

Primarily developers, secondarily customers who have to pay more and/or wait longer for AAA titles.

 

[igor_kavinski]

BTW this applies to AMD and Intel as well, that is if they join Streamline and then start with "exclusive" FSR/XeSS shenanigans.

They have to do the "My milkshake is better than yours" routine. How else they get their milkshake to sell? Remember the Nvidia Dawn/AMD Ruby era? Going with one vendor will always make you miss out on the competitive benefits of the other industry player. It's a compromise.

 

[blckgrffn]

Primarily developers, secondarily customers who have to pay more and/or wait longer for AAA titles.

lol, can you imagine a strategic meeting where they draw the ven diagram of who benefits from DLSS, FSR and Intels solution?

Spending any time (money) architecting, developing, testing and maintaining support of a feature that has no unique users is a tough sell.

 

[DrMrLordX]

lol, can you imagine a strategic meeting where they draw the ven diagram of who benefits from DLSS, FSR and Intels solution?

Spending any time (money) architecting, developing, testing and maintaining support of a feature that has no unique users is a tough sell.

It would be a very short meeting if an NV rep is there to explain how many program incentives they'll get from supporting DLSS. Not sure how aggressive NV is with the money hat these days, though.

 

[Thala]

My only perceived negative is people trying to present Streamline as a consumer oriented move.

I do not understand this argument. With Streamline NVidias reasoning might not be inherently consumer friendly in the first place, but the result is. If some HW has some advantages with respect to some algorithms, which in this concrete case require some form of machine learning inference capabilities, why would you like to suppress such algorithms? If i bought HW with extended capabilities, i would like to have those capabilities utilized.

Today I have no idea what brand my next video card will be.

And yet, you do not have to roll a dice, when purchasing your next video card. You can always make an educated decision based on the value you personally put on the extended HW capabilities.

 

[coercitiv]

With Streamline NVidias reasoning might not be inherently consumer friendly in the first place, but the result is.

Maybe you're from the future, but today we have yet to see the results of this project. In fact if we had results, we would no longer talk about intent.

And yet, you do not have to roll a dice, when purchasing your next video card. You can always make an educated decision...

Wow, thank you so much, I never knew I could buy video cards based on the value of their HW capabilities!

 

[Heartbreaker]

Maybe you're from the future, but today we have yet to see the results of this project. In fact if we had results, we would no longer talk about intent.

The end result if this became widely utilized is easy to understand. It's a common interface, just like we have a common interface for Ray Tracing.

One common interface, is better for developers, and is better for consumers, than three separate interfaces to do the same thing.

It would ensure that every GPU company could do their own scaling implementation and have it supported. Just like every company can do their own Ray Tracing implementation and have it supported.

There seems to be a common theme emerging, that the people against the common interface, think FSR2 will kill DLSS, and common interface will prolong the lifespan of DLSS, and that appears to be the core of opposition. A desire for DLSS to fail.

I wonder why...

 

[coercitiv]

It would ensure that every GPU company could do their own scaling implementation and have it supported. Just like every company can do their own Ray Tracing implementation and have it supported.

You talk the talk but still haven't answered my question: after both AMD & Intel join Streamline, will there still be DLSS exclusive titles?

There seems to be a common theme emerging, that the people against the common interface, think FSR2 will kill DLSS, and common interface will prolong the lifespan of DLSS, and that appears to be the core of opposition. A desire for DLSS to fail.

this concern is really more the typical internet forum conspiracy nonsense, than a real concern.

 

[Heartbreaker]

You talk the talk but still haven't answered my question: after both AMD & Intel join Streamline, will there still be DLSS exclusive titles?

Changes like this take time to work through the system.

In the short term there will only be partial usage, and I expect there will still be titles that only exclusively support one of FSR, XeSS or DLSS.

In the long term if the API is universally embraced, I would expect, the exclusives will end

this concern is really more the typical internet forum conspiracy nonsense, than a real concern.

No, this is the kind of thing said clearly stated in this thread, by those that oppose common API. FSR is going to kill DLSS, and this the common API is NVidias attempt to keep DLSS alive.

 

[Aapje]

One common interface, is better for developers, and is better for consumers, than three separate interfaces to do the same thing.

There are also strong downsides to common interfaces, which is why we don't have them for everything.

A common interface tends to result in getting the lowest common denominator, rather than the best of each. Also, it stalls innovation, as it's hard to change things that are needed to make a step ahead.

It seems to be a bit early to be sure that temporal upscaling has matured enough for a common interface to make sense, when two out of the three systems haven't even been released yet.

 

[Heartbreaker]

It seems to be a bit early to be sure that temporal upscaling has matured enough for a common interface to make sense, when two out of the three systems haven't even been released yet.

No it isn't because API's evolve. We have had decades of GPU APIs that every maker adhered to and evolved over time, and what we have now is three version of the same thing with the same inputs.

We are only getting FSR2, and XeSS because DLSS existed. They are both essentially modeled on DLSS, and taking the same inputs as DLSS, and attempting to create the same result.

They are three flavors of the same thing with the same inputs, and same desired output. They should have a common API.

For all the worries about a basic Open Source common UI, it's far more benign and beneficial, than the alternative route NVidia could have gone to ensure DLSS survival.

If NVidia really wanted to promote DLSS over competing solutions, they would release a generic DLSS-Everywhere Shader version. An inferior non-Tensor version that runs on every card of sufficient power. I'm not sure how many recall, but DLSS 1.5 was a one off solution for the game "Control", that did NOT use Tensor cores. This solution could likely have easily been ported to everything.

Given the massive lead DLSS already has over competing solutions, having a lesser version for other cards would likely kill all competing scaling alternatives.

But IMO that would truly suck, because unlike a common API, DLSS-Everywhere would suppress alternative solutions, and I would much rather have a common API, and let GPU makers compete for the best implementation, that see any one solution prevail.