I think it is a combination of different parameters see for example
GDI Objects.
I am not sure what you mean by 'huge' frame? Lot of visual objects, lot of data (example array with numerous rows) lot of line of code, ...?
Anyway depending on where in our internal OpenROAD code we will have reach the limits for the process, we either trap it and are able to put a message in the trace or we exit.
For example I just created a frame 10x10 buttonfields in a matrix, and I call this frame in a loop. Depending on the machine where I am running it I can see the CreateWindow error in the trace, or OpenROAD just exit because it fails on some other API or when it needs to allocate more memory.
So now why are you reaching the limits with your application? May be we have some memory leaks in OR, may be the 4GL application is loading lot of data (example Tablefields with lot of rows).
Is this always happening at the same place in your code you may be able to identify the component generating the error and check the system resource at that time?
It seems you are running batch so I am confused on the fact you are running frames?