Product Chat / memory consumption of "GENERATING TERRAIN SUPER TEXTURE " out of bounds? Anyone an idea?

Author
Message
Ertlov
GameGuru BOTB Developer
17
Years of Service
User Offline
Joined: 18th Jan 2007
Location: Australia
Profile
Posted: 14th Oct 2018 02:21
Link
I just tried to testrun Father`s Island after some bugfixes and changes to videozones, and testgame doesnt run anymore now, same with standalone.

I turned logging on and found



which is interesting, as it shows GG spamming the RAM with additional 2GB in the Terrain Texture step.

The funny thing is, it stays that way on this map even if I revert my changes.

Anyone see that?

I have added that to the issue tracker, too, and the log is attached.
"I am a road map, I will lead and you will follow, I will teach and you will learn, when you leave my sprint planning you will be weapons, focused and full of JIRA tickets, Hot Rod rocket development gods of precision and strength, terrorizing across the repository and hunting for github submits."

Attachments

Login to view attachments
PM Email Website
Corno_1
GameGuru Tool Maker
13
Years of Service
User Offline
Joined: 3rd Nov 2010
Location:
Profile
Posted: 14th Oct 2018 10:17
Link
Did it happen in the testgame too or just in the standalone?
Ebe Editor Free - Build your own EBE structures with easy and without editing any text files
Thread and Download
PM
Ertlov
GameGuru BOTB Developer
17
Years of Service
User Offline
Joined: 18th Jan 2007
Location: Australia
Profile
Posted: 14th Oct 2018 21:07
Link
In both!
"I am a road map, I will lead and you will follow, I will teach and you will learn, when you leave my sprint planning you will be weapons, focused and full of JIRA tickets, Hot Rod rocket development gods of precision and strength, terrorizing across the repository and hunting for github submits."
PM Email Website
Teabone
Forum Support
17
Years of Service
User Offline
Joined: 8th Jun 2006
Location: Earth
Profile
Posted: 15th Oct 2018 03:01
Link
oh no


Twitter - Teabone3 | Youtube - Teabone3 | Twitch - Teabone3 | TGC Assets - Store Link | Patreon - Teabone3

i7 -2600 CPU @ 3.40GHz - Windows 7 - 8GB RAM - Nivida GeForce GTX 960
PM Website
DVader
20
Years of Service
User Offline
Joined: 28th Jan 2004
Location:
Profile
Posted: 15th Oct 2018 14:40
Link
Quote: "GENERATING TERRAIN SUPER TEXTURE"

Does GG even use this anymore? I thought it was removed way back with the terrain update. It certainly shouldn't be that huge either! Looks like a bug to me or some old legacy code for the old terrains possibly.
SPECS: Ryzen 1700 CPU. Nvidia 970GTX. 16 Gig Memory. Win 10.
PM Website
Teabone
Forum Support
17
Years of Service
User Offline
Joined: 8th Jun 2006
Location: Earth
Profile
Posted: 15th Oct 2018 22:42
Link
This is a serious sounding issue. Hopefully there is resolution.
Twitter - Teabone3 | Youtube - Teabone3 | Twitch - Teabone3 | TGC Assets - Store Link | Patreon - Teabone3

i7 -2600 CPU @ 3.40GHz - Windows 7 - 8GB RAM - Nivida GeForce GTX 960
PM Website
Ertlov
GameGuru BOTB Developer
17
Years of Service
User Offline
Joined: 18th Jan 2007
Location: Australia
Profile
Posted: 21st Oct 2018 00:49 Edited at: 21st Oct 2018 01:00
Link
Still the same behavior. The only way to get it to run is to make sure that it stays below 3.6GB total even with this insane spike.

Interesting:

- the RAM is freed in the reset LUA step again:
- Loading times are 3 - 4 times longer compared to a few versions ago (after the update reducing them)
- "entering options" takes 732MB?!

"I am a road map, I will lead and you will follow, I will teach and you will learn, when you leave my sprint planning you will be weapons, focused and full of JIRA tickets, Hot Rod rocket development gods of precision and strength, terrorizing across the repository and hunting for github submits."
PM Email Website
Avenging Eagle
18
Years of Service
User Offline
Joined: 2nd Oct 2005
Location: UK
Profile
Posted: 21st Oct 2018 08:25 Edited at: 21st Oct 2018 08:26
Link
Don't know if this is related but I found that on one particular iteration of Extraction Point, the filesize of the fpm gained 45MB for no real reason, and the load times in standalone shot up from ~90 seconds to 5 minutes. Ended up reverting to a previous save and adding my changes back in, saving as a different filename. Sadly, performance and load time was also helped significantly by stripping out some of my PBR assets, and going back to some X9 alternatives...kinda defeating the point of building in X11 but oh well...

AE
PM Email
cybernescence
GameGuru Master
11
Years of Service
User Offline
Joined: 28th Jan 2013
Playing: Cogwheel Chronicles
Profile
Posted: 23rd Oct 2018 13:03
Link
It seems the GG log file is reporting on page file usage. When I change it to working set I get a different perspective on ram usage per load stage. More representative of memory hogs perhaps?

I created a working set lua call and after 4 hours of running a game ram useage had risen 1.5GB. A slow trickle (memory leak?) can be seen every few frames. I think it might be connected to music playing but am still checking.

Doesnt really help your issue Ertlov but am still investigating .

Cheers.
PM Website
granada
Forum Support
21
Years of Service
User Offline
Joined: 27th Aug 2002
Location: United Kingdom
Profile
Posted: 23rd Oct 2018 13:10
Link
@cybernescence Ivan posted this on GitHub

Quote: "
IvanErtlov
commented 17 minutes ago
Ok it seems to be that video playback swallows RAM equal to each frame in all of the videos."


Dave
Windows 10 Pro 64 bit
GeForce GTX 1050 Ti
AMD FX (tm)-9590 Eight-core Processor
31.96 GB RAM
1920x1080,60 Hz
PM
Ertlov
GameGuru BOTB Developer
17
Years of Service
User Offline
Joined: 18th Jan 2007
Location: Australia
Profile
Posted: 23rd Oct 2018 13:23 Edited at: 23rd Oct 2018 13:25
Link
yes, currently it seems that video is the culprit. The log is a little bit misleading here.

Not confirmed yet, but I had help from Preben and my tests seem to verify it. Basically every single frame of all videos crawls into RAM at the start. From a functional point, that doesnt make sense to me as a video playback should be something streamed in on demand, but right now its the best lead at hands.
"I am a road map, I will lead and you will follow, I will teach and you will learn, when you leave my sprint planning you will be weapons, focused and full of JIRA tickets, Hot Rod rocket development gods of precision and strength, terrorizing across the repository and hunting for github submits."
PM Email Website
Preben
19
Years of Service
User Offline
Joined: 30th Jun 2004
Location:
Profile
Posted: 23rd Oct 2018 16:07
Link
No it only pre cache the first 16 frames "images" of the video , but it does this for all your videos on your level (19) so:

Your videos is: 1280x720
1280 * 720 * 4 (RGBX used) = 3686400b.
Then it pre cache 16 frames:
3686400b * 16 = 58982400b.
Then it does this for all your videos.
58982400b. * 19 = 1120665600b.
Then it create a render target for each video:
1280 * 720 * 4 (RGBX used) = 3686400 * 19 (videos) = 70041600b.

So memory used now:
1120665600b. + 70041600b. = 1190707200b.

1162800 kb.
1135,54 mb.
1,10 GB.

So it takes 1.1 GB of memory ONLY for your videos, before the game is even started (Thats why you could not run the game , out of memory ).

Anyway i fixed this problem and sent you a test version.



Subscribe and checkout great GameGuru/AGK video's here: Videos click here
Latest GameGuru Loader news: News click here
Get GameGuru Loader PBR version here: Steam click here
best regards Preben Eriksen,
PM Website
Teabone
Forum Support
17
Years of Service
User Offline
Joined: 8th Jun 2006
Location: Earth
Profile
Posted: 23rd Oct 2018 16:47
Link
wow preben you are a machine
Twitter - Teabone3 | Youtube - Teabone3 | Twitch - Teabone3 | TGC Assets - Store Link | Patreon - Teabone3

i7 -2600 CPU @ 3.40GHz - Windows 7 - 8GB RAM - Nivida GeForce GTX 960
PM Website
cybernescence
GameGuru Master
11
Years of Service
User Offline
Joined: 28th Jan 2013
Playing: Cogwheel Chronicles
Profile
Posted: 23rd Oct 2018 17:06
Link
That’s a great fix!

My issue must be something else as a slow incremental consumption of ram once the game is loaded.

Cheers.
PM Website
Ertlov
GameGuru BOTB Developer
17
Years of Service
User Offline
Joined: 18th Jan 2007
Location: Australia
Profile
Posted: 23rd Oct 2018 21:29 Edited at: 23rd Oct 2018 21:30
Link
thanks a lot! checking at night when back at home
"I am a road map, I will lead and you will follow, I will teach and you will learn, when you leave my sprint planning you will be weapons, focused and full of JIRA tickets, Hot Rod rocket development gods of precision and strength, terrorizing across the repository and hunting for github submits."
PM Email Website
Preben
19
Years of Service
User Offline
Joined: 30th Jun 2004
Location:
Profile
Posted: 25th Oct 2018 08:17
Link
cybernescence:
Quote: "My issue must be something else as a slow incremental consumption of ram once the game is loaded. "


M-Sliders.cpp
search:

replace:


Test using F11 (let it settle first), should make it way more easy to figure out
Subscribe and checkout great GameGuru/AGK video's here: Videos click here
Latest GameGuru Loader news: News click here
Get GameGuru Loader PBR version here: Steam click here
best regards Preben Eriksen,
PM Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 25th Oct 2018 09:36
Link
I made that code change and in test game with a small map and the player simply standing still we appear to be leaking memory at around 1Mb every minute or so!
Been there, done that, got all the T-Shirts!
PM
GraPhiX
Forum Support
19
Years of Service
User Offline
Joined: 15th Feb 2005
Playing:
Profile
Posted: 25th Oct 2018 10:17 Edited at: 25th Oct 2018 10:18
Link
@AmenMoses - what happens if you turn off the terrain, then try water, then sky then ai see if you can pinpoint the culprit I assume it will be terrain
Welcome to the real world!
Main PC - Windows 10 Pro x64 - Core i7-7700K @4.2GHz - 32GB DDR4 RAM - GeForce GTX 1060-6G 6GB - 1TB NVe SSD
Test PC - Windows 10 Pro x64 - G4400 @3.3GHz - 16GB DDR3 RAM - GeForce GTX 950 2GB - 500GB SSD
Laptop - Helios 300 Predator - i7 7700HQ - 32GB - Nvidia GTX1060 6GB - 525GB M2 - 500 SSD - 17.3" IPS LED Panel - Windows 10 Pro x64
Various Tutorials by me
PM Email Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 25th Oct 2018 10:31 Edited at: 25th Oct 2018 10:32
Link
Well, interestingly enough I then tested the Big Escape level and left it running for 30 minutes. No leak, memory usage was the same at the end as the beginning.

The 'small level' I first tested happens to be the fireworks display one! So when I get back from work I'm going to try it without any scripts running. Then again with just the fireworks scripts running but no fireworks triggered to see if I can figure out what caused the 'leak'.

There may be a subtle leak in the particles code itself in the engine that hasn't been noticed before because it hasn't really been used much up till now.
Been there, done that, got all the T-Shirts!
PM
cybernescence
GameGuru Master
11
Years of Service
User Offline
Joined: 28th Jan 2013
Playing: Cogwheel Chronicles
Profile
Posted: 25th Oct 2018 12:51
Link
Thanks Preben that’s pretty much what I have already done to monitor though also changed the smea to display working set memory (pmc) - which kind of tracks memory usage as in task viewer.

I now don’t think it is playing sounds causing the leak but it is a squirrely thing to pin down - still looking.

Hopefully with a few of us looking it will get narrowed down faster

Cheers.
PM Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 25th Oct 2018 19:23 Edited at: 25th Oct 2018 19:57
Link
So after a bit of experimenting the leak is Lua related, it does occur in the Big Escape level, which has no particle effects, as long as you just stand still and don't kill off the AI.

The reason that the leak is so rapid in the fireworks display is purely due to the number of scripts running, even when I altered the scripts to basically do nothing just the fact they are being called causes a slow 'leak'.

I suspect that for performance reasons Lee has turned off Lua garbage collection until a certain level is reached or something similar, I can experiment some more to establish if that is the case.

A secondary issue, which I already suspected was the case, repeated test runs show increasingly more memory used so it isn't being freed between test runs, or if it is fragmentation is causing more to be used each run.

Correction! The only thing that turns off the memory leak is disabling physics! Which makes no sense at all to me.

Lua garbage collection is on all the time, and appears to be working fine … hmmm.

Or maybe the leak has something to do with the player control? With physics off the player cannot move and possibly the player control logic is disabled.

Correction2! It's both, i.e. with Physics on there is some Lua calls going on under the covers, the playercontrol script for example, with AI also on it gets worse if there are a lot of scripts running.
Been there, done that, got all the T-Shirts!
PM
DVader
20
Years of Service
User Offline
Joined: 28th Jan 2004
Location:
Profile
Posted: 25th Oct 2018 19:56
Link
I remember someone saying the physics code is a bit of a mess awhile back, so the physics having a memory leak is quite possible. It hasn't been widely used that much up to recent months.
SPECS: Ryzen 1700 CPU. Nvidia 970GTX. 16 Gig Memory. Win 10.
PM Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 25th Oct 2018 20:01
Link
It's not the physics code itself, it's the Lua calls being made from/to the physics side I reckon.

I'm going to take a look at the Lua side some more, see how it has been interfaced to the C++.
Been there, done that, got all the T-Shirts!
PM
DVader
20
Years of Service
User Offline
Joined: 28th Jan 2004
Location:
Profile
Posted: 25th Oct 2018 20:32
Link
Ah you would know better than me, the only C++ code I've ever played with was Dark Basic C++ ages ago. Keep up the good work, you're a star!
SPECS: Ryzen 1700 CPU. Nvidia 970GTX. 16 Gig Memory. Win 10.
PM Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 25th Oct 2018 23:33
Link
Well I've been staring at the code all evening and can't see where the leak is.
Been there, done that, got all the T-Shirts!
PM
DVader
20
Years of Service
User Offline
Joined: 28th Jan 2004
Location:
Profile
Posted: 26th Oct 2018 00:24
Link
Nasty Never nice to get zero progress after spending several hours + on code, or anything really. I seem to remember Lee going on about the great debugging tools in the version of Visual Studio he was using. He specifically mentioned memory leaks in his blog at the time being easier to track. Maybe he can give some insight? You may already know about this of course, just a thought as reading your post triggered my memory of the blog. It may not be as simple.

This was actually well before the DX11 update, but he was tinkering I believe and tantalising about unlimited light sources using DX11 etc :p
SPECS: Ryzen 1700 CPU. Nvidia 970GTX. 16 Gig Memory. Win 10.
PM Website
Stab in the Dark software
GameGuru TGC Backer
21
Years of Service
User Offline
Joined: 12th Dec 2002
Location: Upstate New York USA
Profile
Posted: 26th Oct 2018 00:47
Link
Quote: "Well I've been staring at the code all evening and can't see where the leak is."


When I lasted looked at the physics code I saw several memory leaks in the main Bullet Physics code.
One example, the coder changed the Bullet physics character controller class which caused an error.
Instead of fixing the error they changed something else to try to solve the first error and caused a memory leak.
They should have just created a new class that inherits the character controller. Hint: the AGK source code is
now open so you could just copy that physics code over to GG.
The coffee is lovely dark and deep,and I have code to write before I sleep.
PM Email Website
Corno_1
GameGuru Tool Maker
13
Years of Service
User Offline
Joined: 3rd Nov 2010
Location:
Profile
Posted: 26th Oct 2018 12:40
Link
Quote: "M-Sliders.cpp

pastebitmapfont( cstr(cstr(Str(GetDisplayFPS()))+"fps mem: " + cstr(Str(SMEMAvailable(1)/1024)) +"mb." ).Get(),8,8,1,255);

Test using F11 (let it settle first), should make it way more easy to figure out "


Thanks. I now understand my FPS drops in some level
Ebe Editor Free - Build your own EBE structures with easy and without editing any text files
Thread and Download
PM
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 26th Oct 2018 20:11 Edited at: 26th Oct 2018 20:14
Link


This video shows GG memory usage in the top left and bottom centre the Lua memory usage (in Kb).

Now the interesting thing is that although the fireworks scripts are being called I have edited them to simply do nothing and return to the engine.

GG standard Lua scripts will also be being called of course but I have switched off Physics in the F11 menu so the player cannot actually move around.

You can see that the Lua memory usage increases and then falls, this is the garbage collection kicking in I think, but GG memory just keeps rising.

Maybe this may give someone some clues where to look, personally it just gives me a headache.

btw to see Lua memory simply do this in a script:
Been there, done that, got all the T-Shirts!
PM
cybernescence
GameGuru Master
11
Years of Service
User Offline
Joined: 28th Jan 2013
Playing: Cogwheel Chronicles
Profile
Posted: 27th Oct 2018 13:34
Link
I can see some differences between the agk and gg bullet code but nothing to suggest memory leak (but could be looking in wrong place I guess).

I’m not getting any success by turning things off to try and narrow it down for static inspection so may have to see if I can get the VS tools to work as Dvader suggests.

Cheers.
PM Website
AmenMoses
GameGuru Master
8
Years of Service
User Offline
Joined: 20th Feb 2016
Location: Portsmouth, England
Profile
Posted: 27th Oct 2018 14:06
Link
If you figure that out please let me know how!

I just get VS complaining that I can't use them with my 'toolset' and prattles on about VS11 SDK, I think it may be to do with the precompiled libraries, like Bullet.lib, not being built to the latest SDK or something.

I'm using VS2017 upgraded from VS2015 but when I look at the install folders the latest 'SDK' folder is 2014!


Been there, done that, got all the T-Shirts!
PM
Back to top

Login to post a reply

Server time is: 2024-04-20 15:01:15
Your offset time is: 2024-04-20 15:01:15