Question About Memory Management with Audio*

Hi everyone, I've started to go ahead and start working on some game projects with the intent of completion and delivery. However again I have my concerns around memory manage. My question this time is what is not stored in memory or effected by any possible memory limits or storage or at risk of capping/leaking?

For example can we load audio without it loading into memory? I most commonly use the OGG format.
Also is this the same case with PNG Sprites? I have had unexpected errors after using a plenty of PNGs after a while in the editor during tests without exiting GG.

Everything is loaded into memory, that's how computers work
Perhaps you just need to make sure things are being cleaned up correctly when you exit (DeleteImage etc).

unfortunately, all audio etc has to be loaded into memory. memory should always be deallocated once finished with.

if you convert your PNG's to DDS you will get some memory back PNG's are handed to the GPU via the CPU,

PNG stores images in a compressed, lossless format that is optimized for storage on disk. DDS stores textures in formats that are native to the GPU, and hence optimized for GPU consumption. For PNG's to be useable as texture data, they typically need processing to convert to a GPU format, and possibly also create mipmaps. This can take some time and memory if you have a lot of textures to load, particularly if you want to load them as a DXT/BC-compressed format so that they're optimized for GPU memory and bandwidth consumption. They will also need some manual processing if you want to load them as 1D, 3D, or Cube textures. DDS can store all of those formats natively, which means you can load them very quickly with little-to-no processing and less memory.

Forgot to edit my original post. I'm referring to PNG sprites. Not sure if DDS is supported with the sprite commands?



I'm running into problems with the sprites crashing GG over time, even if I'm just on one map. But I'm assuming this is happening because im within the editor and not standalone. I'm assuming the standalone would require clearing whenever you re-load the map from a save file. Maybe before loading an image i should add a deleteimage command, even though the image would not have been loaded that first time? with a nil check this might be possible i guess. I'm just really worried about hitting the same memory issues i have encountered, once i get to making standalone.

With audio should i be loading them in and deleting/deactivating them after they play? Not at home at the moment so dont have the global.lua on me to check commands.

Your sprites should work as DDS try it they may help eleviate the problem using GPU memory instead

I'll try that when I get home. Though i believe I might be limited to squared resolutions.

For audio im curious how game developers are able to host 100's of audio files without capping out the memory? Especially games like Bethesda produces? I work with their engine and it can support an infinite number of OGG files somehow?

it may be the sound processor they use, is fallout DirectX or openGL or both lol I think Direct X 12 as better sound management too

The other thing you will notice is that in DirectX 11 the Direct Sound API is still the same one from DirectX 8. The only major difference is that hardware sound mixing is generally not available on the latest Windows operating systems. The reason being is for security and operating system consistency all hardware calls now have to go through a security layer. The older sound cards used to have DMA (direct memory access) which was very fast but doesn't work with this new Windows security model. So all sound mixing is now done at the software level and hence no hardware acceleration is directly available to this API.
The nice thing about Direct Sound is that you can play any audio format you want. Direct Sound is so easy to use you just create a sound buffer with the play back format you would like and then copy your audio format into the buffer's format and then it is ready to play. You can see why so many applications use Direct Sound due to its pure simplicity.
Note that Direct Sound does use two different kinds of buffers which are primary and secondary buffers. The primary buffer is the main sound memory buffer on your default sound card, USB headset, and so forth. Secondary buffers are buffers you create in memory and load your sounds into. When you play a secondary buffer the Direct Sound API takes care of mixing that sound into the primary buffer which then plays the sound. If you play multiple secondary buffers at the same time it will mix them together and play them in the primary buffer. Also note that all buffers are circular so you can set them to repeat indefinitely. So I think GG is using the secondary buffer hence the memory issue, DirectX 11 doesn't take advantage of multiple cores like DX12, thus a single core is doing all the work while the others remain idle. The days of having a single CPU core and a single GPU core went out in the early aughts, and Microsoft is finally getting up to speed with this latest DirectX release.
Look at it this way: computers have moved from a single-lane to an eight-lane superhighway, allowing the CPU to throw rendering and compute commands to the GPU faster than ever before. For the gamer, that means better framerates and a better image quality. Also Windows 10 as a Native API for DX12.

I remember distinctly that Lee was able to utilise more cores in DX11 than in DX9, it may not be as good as 12 of course, but should still use more than a single core? I definitely remember Lee talking about handling different processes with different cores.

@Teabone. Deleting them as you suggest should work. When I do that sort of houskeeping I normally delete them on exiting the mini-game so when starting over, it just reloads them and you know they are no longer used. The nill option on loading would work as well, but just means it will take a little longer to load. Deleting on exit splits it up a little

Purely in itself, moving from DX9 to DX11 will have basically little effect on the look of the game, or its performance. Each version of DX is a superset of the previous one - it just adds new options and capabilities. In fact DX11 will still run on DX9 GPUs, as long as you don't use any of the new features. The API changed significantly from DX9 to 10 (and a little from 10 to 11), so DX9 titles need major rewrites of their graphics code to move to DX11, which takes time. At the end you'll have a DX11 title that takes no advantage of the new features, but after that you can start to make improvements.

For a flavour of what could be improved, some of the headline features of DX11 over DX9 are instancing, tessellation and compute shaders (plus a bunch of other smaller things I've probably forgotten).

'Instancing' means drawing multiple copies of objects in one go. In DX9 if you want to draw e.g. ten tanks, you set up the position/rotation of a tank and draw it, and repeat this ten times. In DX11 you can set up an array of transforms for each tank and draw ten at once, potentially saving a load of CPU time (each draw call is expensive). This may well be of use here.

'Tessellation' can be used to dynamically generating new vertices depending on where the camera is. It's useful for height-mapped ground terrains to generate the optimum number of vertices for the given camera angle, and avoid copying so much data to the GPU (again slow). I don't know if this is applicable at the moment, but it's pretty much a requirement for doing good looking dynamic water.

'Compute shaders' are for doing general purpose (not necessarily graphics) parallelisable work on the GPU. It's still mainly used to do graphics effects though and is great for doing super-quick particle systems, hundreds of dynamic lights, post-processing effects, stuff like that. Again I don't know how useful it'll be for this engine.

So, DX12. The main advantage of DX12 is that it can be much more efficient on the CPU, by stripping away most of the driver and giving direct access to the hardware. It's Windows 10 only though, so most (or all?) existing titles add DX12 in addition to DX11 support, as otherwise you're cutting your customer base. That doubles up your work though each time you add a new graphics feature, so personally I think DX11-only is the way to go for now.

At the end of the day though I doubt graphics rendering is actually the bottleneck, so there will be limitations on what a change of DX version can achieve for performance. I suspect that most of the CPU is used for AI, so even if you removed all CPU usage from the graphics, I doubt it would have that great an effect. The other coming improvements around AI efficiency will likely do more .
DirectX 12 now lets developers get closer to the “metal” of their users’ graphics hardware, providing access to features previously unavailable for developers. Exposing lower level hardware means that skilled developers can write more efficient code that executes faster. Specifically, developers can improve optimization with faster multi-threaded performance and more efficient CPU utilization. This could potentially allow for low-level optimizations similar to those we see in console gaming where games are tuned for extremely specific mid-range hardware, eeking out performance that would be impossible without such hyper-specific optimizations.
Previously, DirectX 11 only offered a fairly high level of hardware abstraction, which vastly simplifies the process of coding. In fact, that’s the whole idea behind what makes hardware APIs so powerful. But it also meant that opportunities for fine-tuning code against specific hardware features were less frequent. Microsoft’s Developer Blog provides some great insights into how, exactly, DirectX 12 accomplishes this. https://blogs.msdn.microsoft.com/directx/2014/03/20/directx-12/#where-does

you shouldn't technically get any issue with sound as sprites are handled differently to sounds in that a sound can be loaded but not playing while a sprite will always be "using" its loaded image even if not on screen so it can't really free up that memory.

it's important to remember that media such as sounds and images are compressed when in their normal file state and when being actually used in a program they tend to actually require much more memory.

if it does become an issue you can use the global sound commands to load and delete sounds as required (just remember they won't automatically copy to your standalone), and as i mentioned before, deleting unused sprites & images is always a good idea.
you may also want to look at cutting back on the number of images you are using if possible - you mentioned having many sprites, are they all different images? double check you aren't loading any duplicate images as a starting point.

What! I can't remember that being an issue. I rarely use standalone mode these days, I've encountered scripts not copying over recently when using include I didn't know the sounds also refused to copy over.

I think we need a seriuos bugbash and sort as many of these issues as possible

Edit
@ Graphix. Perhaps it was the newer Visual Studio features which gave more access to extra cores. Lee definitely said that he has already split some stuff onto extra cores even at this stage, think it was his blog, which was some time ago.

Everything should copy across in the standalone as long as it is explicitly referenced in the scripts. GG needs to be able to work out what to copy so if you build paths or filenames in a script at runtime GG can't figure that out and will not copy it over.

For example avoid doing things like this:


That is an example from my Asteroids demo from back when I didn't know any better.

Same thing applies to sounds, somewhere in your script you must make sure there is an explicit reference to the sound file.

Remember if a file doesn't copy across to standalone you can always add it to the ".fpp" file (see GIT update dated Sunday, 7 October 2018) to ensure it gets copied across with the appropriate encryption etc.

How to use the FPP properly:

[standalone add files]
characterkit\bodyandhead\
importer\temp\readme.txt

[standalone delete files]
databank\
effectbank\explosion\animatedspark.dds

Do not add a "\" in front of the folder like:
\effectbank\explosion\animatedspark.dds
That is not working!

Does the fpp system replace automatic copying or is it in addition to it?

i.e. if you include an .fpp file does GG expect it to list all files to be copied?

Its an addition to it … Im still trying to work it out but right now all my stuff copies fine anyway..

Myke seems to understand it better perhaps he will jump in

the fpp file should be next to the fpm file and must have the same name. All files, which are not copied over to the standalone, can be registered there and then the engine knows them and copies them over in the next build.
It is a addition to the export feature, because you can not prevent all usage of lua commands (like AmenMoses example with the for loop)

This line:
LoadImage ("scriptbank\\images\\Asteroids Cockpit\\Numbers\\Blue\\" .. i .. ".png")
will not work, but if you add this line to the fpp, it works:
scriptbank\images\Asteroids Cockpit\Numbers\Blue\

Put simply, it is a workaround for not copied over files

So it will copy over everything in a specified folder?

If you want to copy the folder with all files inside to the standalone create a .fpp with a texteditor and add following line:
[standalone add files]
scriptbank\images\Asteroids Cockpit\Numbers\Blue\

if you just want some image files, add:
[standalone add files]
scriptbank\images\Asteroids Cockpit\Numbers\Blue\1.png
scriptbank\images\Asteroids Cockpit\Numbers\Blue\2.png
scriptbank\images\Asteroids Cockpit\Numbers\Blue\3.png

You can add as many folders or files as you want

and the best is, everything is encrypted.

That works. Thank you.

Noticed something in the loading.lua file that gets added standalones. It appears only 63 sprites are supported?

SetLoadingResource(0,4) -- number of sprites up to 63

That's probably just for the main menus or something, I've used over 2,000 sprites before now with no issues. There used to be a hard limit of 1,000 but Lee raised that, IIRC the limit is 10,000.

Yeah I remember sending Lee a mail regarding the sprite limit years ago as it was causing issues with a 2D game I was doing at the time. He did increase it from the 1000 max at the time to 10000 as AmenMoses says. I can vouch there is no problem with thousands of sprites. I remember Lee saying he did not think anyone would ever use a thousand sprites but as ever had been surprised by the things GG users make :p

2000? You guys are machines

But this certainly is good news -- as i have a very complex inventory and UI system being developed which uses "a lot" of sprites.

My scrolling map thingy for the old Big Escape map used 2600 sprites. That was two years ago!

Yeah my old 2D shooter ran over the 1000 limit at one point and it was a fairly simple game.

I also started a scrolling map type game after that, which needed a lot more (never got far with it though). But it's pretty easy to use more than 1000 sprites in a game if you have a big map and/or lots of fancy explosions and such.

That is just cool DVader

Reliquia....