Out of memory whilst patching

[aeiltz]

We have recently had problems with Visual Patch giving Out of memory errors.
So we thought we better upgrade to the latest version of Visual Patch, but the problem still occurs.
The server that we build the patches on does not show any sign of running out of memory, so what is going on ?
Is it possibly related to the number of versions that we patch ?

[aeiltz]

See screenshot for error message and the stage when it happens.

[jassing]

There seems to be a lot of "including whole file" messages -- could you be getting low on physical disk space on the temp drive?

[aeiltz]

I have 15Gb of free disk space so that is not the problem. I have switched off the logging and verbose logging, this has reduced the number of "Out of memory" occurrences.

I decided to remove some of our older versions, we have about 50 version with each about 1800 files. Then I tried to build again and hey presto no more "Out of memory" errors.

So clearly Visual patch cannot cope with the great number of files/versions.

[jassing]

I would bet that it's not the disk size that it's complaining about; but rather RAM.

if most of your 1800 files change and you have many versions; I would suspect a out of memory condition at some point.

My understanding is that the underlaying patch engine is near limitless -- but it is limited by your machine... Physical ram & disk space...

Glad you got it sorted out.

[sgrainger]

We have been plagued by this with our largest project.

I have a Very Large project that is 2.76 GB in overall size spread out among 43,000 files per version.

We have not been able to make a new version between our latest update. It's running on a server that has over 100GB free in Disk Space and 2GB RAM on the machine itself.

We used Mem Gobbler to bring down the RAM according to some older suggestions we found. But nothing we do on this project seems to work. We moved it to different machines, but we get the "out of memory" error constantly now. Even when just trying to save the project.

Any suggestions?

[jassing]

I would add more ram.
memory "optimziers" rarely actually help.. infact, more often they slow the system down.

[sgrainger]

Mem Gobbler was suggested by one of the Developers in the past. One reason we tried. We do have an order with our facilities to upgrade the ram on this box.

[sgrainger]

I would add more ram.
memory "optimziers" rarely actually help.. infact, more often they slow the system down.

We upgraded the RAM to 3GB on this machine and reimaged the entire OS. The project has been slimmed down to 35000 files per version, but we are still running into this error constantly.

How much RAM would you recommend to handle this kind of project? If this is indeed the problem?

[SteveMartin60]

I just downloaded the trial to test out Visual Patch. The first time I tried to do a build I got this error a number of times. I'm running 4GB of RAM, although it's a 32 bit OS so only around 3.4 available.

Does anyone know if Visual Patch would work with Win 7 64 bit?

[jassing]

yes, it will run on win7 x64; but it's still a 32bit application, so it's memory usage will be similar.. Although if you run 64bit apps otherwise, you may get the full ram it can see...

What are you patching? I have one client that insists on patching every version he's ever built (not just released) we finally had to do a double patch -- one for exe's and another for dll's to get around memory usage issues.

You can try reducing the files you're patching to see if that helps.

[SteveMartin60]

I'm patching a CAD software package. 2.6GB and more than 22,000 files.

We're currently using some older patching software but I'm looking around to see if there is something better. I have to say my initial experience with Visual Patch doesn't fill me with enthusiasm. Especially as there are a number of people with the same issue and I don't see a lot of action from Indigo Rose staff in response on the forum.

I suspect the problem is that the entire list of files and differences is kept in memory during the build process rather than building a list of files to be patched and then patching them one at a time and writing the differences to disk as they go. Such an approach would definitely be faster for small patches but is a non-starter for really large applications.

[jassing]

Open a support ticket with indigorose -- while they do monitor & reply occasionally, this is mostly a user-to-user forum... I have found them to be very responsive.

The IR development team also remains quiet during build cycles, so they may be addressing this issue, I have no idea; but the way to find out is to contact them...

-josh

[SteveMartin60]

Thanks Josh. I'm just trying to build again on a high spec machine. Dual i7 Quad Core, 6GB RAM, Win 7 x64

If that still gives me the same issue I'll contact support and see how that goes.

[Lorne]

I suspect the problem is that the entire list of files and differences is kept in memory during the build process rather than building a list of files to be patched and then patching them one at a time and writing the differences to disk as they go. Such an approach would definitely be faster for small patches but is a non-starter for really large applications.

No, the files are differenced and written out individually, only one operation is performed within memory at a time.

We've had a handful of reports of this issue and we have investigated but haven't been able to reproduce it yet.

In some cases on systems with large amounts of RAM limiting the amount of free memory to under 2GB helped, although the code has been designed so that this shouldn't be necessary. It could be an undocumented variance in some versions of the Windows API calls that are used to investigate the memory available to a running process. (We've uncovered a couple of those already.)

The patching algorithm does its best to acquire the largest contiguous block of address space it can for the hash tables, and we even use high-performance unbuffered io routines to prevent the file system cache from stealing it back. The system is designed to operate with very small IO buffers when necessary however the differencing tables need a large contiguous block of address space. The differencing scheme I developed does not use chained hashing so the memory it needs does not increase during patching, it will instead do the best it can with whatever memory is available.

There are still some features I would like to add to the patching engine, e.g. 64-bit addressing, however I cannot say if/when I will be able to implement such changes given our current development schedule.