lzma error -- suggestions

[jassing]

Suggestion:
(Other than suggesting using the dll than shelling out)

When seeing if the "dictionary size" was fixed, I got this error when using anything over 128:

LZMA 4.43 Copyright (c) 1999-2006 Igor Pavlov  2006-06-04

Error: Can not allocate memory

Error 2091: Could not compress file using LZMA: C:\Clients\GrantSkellenger\FlashBack\HPS FlashBack Installation Files\DirectX\Redist\ManagedDX.CAB

I also tested the latest release of lzma.exe and the latest beta; same thing.

It would seem that a pre-launch memory availability could be done - or a parsing for "Can not allocate memory" so that the SUF8 can then offer "Try reducing the dictionary size" would be helpful.

The system has 2 gig of ram installed, with 400Meg "free" before the start of the build; during the build it dropped to around 135meg free...


update: I closed some other programs and got 800meg free; still anything over 128 yeilded cannot allocate memory.

[Lorne]

A 128 MB dictionary size is too big for your available memory. For LZMA compression the memory use is about 10 times more than the dictionary size. (For decompression it's about equal with the dictionary size.)

Depending on the kind of allocation scheme the LZMA compressor uses, it might even require contiguous address space, which means you might not have enough usable memory even if you have all of it free, because of address space fragmentation. (Memory can never become fragmented, but the address space that describes it can.)

[jassing]

Originally Posted by Lorne

A 128 MB dictionary size is too big for your available memory. For LZMA compression the memory use is about 10 times more than the dictionary size. (For decompression it's about equal with the dictionary size.)

Depending on the kind of allocation scheme the LZMA compressor uses, it might even require contiguous address space, which means you might not have enough usable memory even if you have all of it free, because of address space fragmentation. (Memory can never become fragmented, but the address space that describes it can.)

.... And since we know this up front, we can tell the user, instead of a cryptic 3rd party error message...

.... And since you capture the output of the 3rd party compression routine, you could give more advice to the user...

It was a suggestion to help the end user with 3rd party error messages. Since you're using a 3rd party comrpession program, not everyone will really understand that it's "not IR's problem" -- and you have enough information to help the user understand this error - in addition;you are able to help it from happening in the 1st place.

[Lorne]

...and since we're all perfect at IR and have infinite time...

I wasn't involved in the SF8 compression change or the docs for it. I was responding based on my knowledge of LZMA and compression in general.

In any case thanks for your suggestions, Josh. I've added them to the database. REF: 17794

I'll also suggest that a note about the relationship between dictionary size and memory use be added to the help file. REF: 17795

[jassing]

Originally Posted by Lorne

For LZMA compression the memory use is about 10 times more than the dictionary size. (For decompression it's about equal with the dictionary size.)

The more I think about this statement, the more I think that more serious warnings need to be given to the user.

Why have a 1 gig dictionary size at all? on a win32 machine; you cannot access 10 gigs of memory..