[Script] MD5 Verification

[ryanc]

Good Day

I have been using visual patch for some time now to build differential patches and must say, it's awesome! But sometimes when deploying it appears that files are not being updated. I now that is because of files not being upgraded in the past, due to an omission on my part :eek:

So I have written this function to scan the patched files post patching to confirm they have been updated. If the file is found to not be matching, it is added to a table which I interate through and download from our file server.

Now the first question I want to ask, am I eating a ice cream with a shovel? I s there some other way to do this?

Also, the other thing is that I update a scrolling text dialog. I added a pause to prevent the screen from flickering (this only seems to happen on lower end systems), is there some to avoid the redraw flicker?


--[[
************************************************** ******************************
Function: ManifestCheck
Purpose: Compare the installed files MD5 Hashes to the data contained in the
Manifest XML file. Also logs the progress in Scrolling Text Control if it is
present.
Arguments: None;
Returns: (boolean) true - all files match, false - there are mismatches

Say I have three files:
c:\Helpdesk\Helpdesk.exe,
c:\Helpdesk\LocaliseEN.dat
c:\Helpdesk\Reports\IssuesByDate.rpt

The Template for the manifest file is as follows

<?xml version="1.0" encoding="iso-8859-1"?>
<Manifest>
<helpdeskexe>
<File>Helpdesk.exe</File>
<MD5>058056d54f2ca3620ca1d34a28ff0a67</MD5>
<Size>560128</Size>
</helpdeskexe>
<localiseendat>
<File>LocaliseEN.dat</File>
<MD5>d77efbb37a61f53b654b0cf6eca1313a</MD5>
<Size>73728</Size>
</socketsdll>
<reportsissuesbydaterpt>
<File>Reports\IssuesByDate.rpt</File>
<MD5>42d33f5cbce69929aaffb96e1643aaaa</MD5>
<Size>66816</Size>
</reportsissuesbydaterpt>
</Manifest>

************************************************** ******************************
--]]
function ManifestCheck()

Log("Start Manifest Check");

-- MD5 Compare Result
local sMatchRes = "";

-- File to be Evaluated
local sFile = "";

-- Expected MD5 for Evaluated File
local sMD5 = "";

-- Actual MD5 of the file installed
local newMD5 = "";

-- The Path to Your Server from which the manifest file can be downloaded
local sHTTPServ = "www.myserver.com/updates/"

-- Boolean to return if the files are correct
local bSuccess = true


if File.DoesExist(SessionVar.Expand("%SourceFolder%\\Manifest.XML")) == false then
StatusDlg.Show();
HTTP.Download(sHTTPServ.."manifest.xml", SessionVar.Expand("%SourceFolder%\\Manifest.XML"), MODE_BINARY);
StatusDlg.Hide();
end;

XML.Load(SessionVar.Expand("%SourceFolder%\\Manifest.XML"));

local min = 1; -- The number to start at
local max = XML.Count("Manifest", "*"); -- The number to stop at
local count;

--Interate through our list and check for mismatches
for count = min, max do
sFile = XML.GetValue("Manifest/*:"..count.."/File");
sMD5 = XML.GetValue("Manifest/*:"..count.."/MD5");
sSize = XML.GetValue("Manifest/*:"..count.."/Size");
tDescription = String.SplitPath(sFile);

if File.DoesExist(SessionVar.Expand("%SourceFolder%\\"..sFile)) then
newMD5 = Crypto.MD5DigestFromFile(SessionVar.Expand("%SourceFolder%\\"..sFile));
if newMD5 ~= sMD5 then
sMatchRes = "MD5 Hashes for Files "..sFile.." are not correct redownload";
AddManifestDownload(sFile, sSize, tDescription.FileName);
bSuccess = false;
else
sMatchRes = "";
end;

else
sMatchRes = "File "..sFile.." is missing!";
AddManifestDownload(sFile, sSize, tDescription.FileName);
bSuccess = false;
end
if (CTRL_SCROLLTEXT_BODY) ~= nil then
if sMatchRes ~= "" then
DlgScrollingText.AppendLine(CTRL_SCROLLTEXT_BODY, sMatchRes, true, false);
Log(sMatchRes);
end;
end;

Application.Sleep(100);
end
DlgScrollingText.AppendLine(CTRL_SCROLLTEXT_BODY, "File Check Completed", true, false);
LogLine();

return bSuccess;

end;

--[[
************************************************** ******************************
Function: AddDownload
Purpose: Add A File to the Download List
Arguments:
FileName - Name of the File,
RemotePath - Path of the File relative to Base Path
LocalFile - Local Filename and path to save the file to
Execute - Execute the file after all downloads are completed
ForceOverWrite - Delete the current file before downloading - force install
Returns: True or False if the download was added to the list
************************************************** ******************************
--]]
function AddDownload(FileName, RemotePath, LocalFile, Execute, ForceOverWrite, FileSize, Description)
local tblNew = {};

if FileSize == nil then
FileSize = 0
end;

if Description == nil then
Description = FileName;
end;

tblNew.FileName = FileName;
tblNew.RemotePath = RemotePath;
tblNew.LocalFile = LocalFile;
tblNew.Execute = Execute;
tblNew.ForceOverWrite = ForceOverWrite;
tblNew.FileSize = FileSize;
tblNew.Description = Description;

local max = Table.Count(tblDownloads);

local i
local item
--Check for dupes
for i,item in tblDownloads do
if String.Lower(item.FileName) == String.Lower(FileName) then
return false
end;
end;

Table.Insert(tblDownloads, max + 1, tblNew);
return true;
end;

--[[
************************************************** ******************************
Function: AddManifestDownload
Purpose: Add A File from the Manifest list to the Download List
Arguments:
AFileName - Name of the File with the path relative to the aura installation
AFileSize - Size of the File
ADescription - Description of the File
************************************************** ******************************
--]]
function AddManifestDownload(AFileName, AFileSize, ADescription)
-- Convert the folder paths to be web friendly
local path_parts = String.SplitPath(AFileName);
local remotepath = String.Replace(path_parts.Folder, "\\", "/", false)

remotepath = "current/"..remotepath;

local remotefile = path_parts.Filename..path_parts.Extension;

AddDownload(remotefile, remotepath, SessionVar.Expand("%SourceFolder%")..path_parts.Folder.."\\"..remotefile, false, true, AFileSize, ADescription);
end;

[Lorne]

I have been using visual patch for some time now to build differential patches and must say, it's awesome!Thanks! :)

But sometimes when deploying it appears that files are not being updated. I now that is because of files not being upgraded in the past, due to an omission on my part :eek: What kind of omission do you mean, actually? Were you not including some files in your patch project? Or...?

So I have written this function to scan the patched files post patching to confirm they have been updated. If the file is found to not be matching, it is added to a table which I interate through and download from our file server.That's a pretty slick system, although it involves some duplicate work, since Visual Patch verifies MD5 checksums as part of the patching process. How are you generating the manifest XML file?
Now the first question I want to ask, am I eating a ice cream with a shovel? Is there some other way to do this? Possibly. It depends what problem you're trying to solve, though...if it's just from forgetting to include files in the project you might be able to move the validation to build time by using the project report tool or the build log.

Also, the other thing is that I update a scrolling text dialog. I added a pause to prevent the screen from flickering (this only seems to happen on lower end systems), is there some to avoid the redraw flicker?Sometimes combining lines can help (fewer draws)...but some systems just have slow redraws.

[ryanc]

Howdy

Thanks for the reply. I generate my manifest using an application I wrote for generating my updates. As my of my users are on dial up (South Africa is VERY behind in broadband deployment) I build differential patches for every possible release from the time I started using True Update.

The omission is more a conflict, as some support technicians in the past have had a very cowboy attitude in terms of releasing patches. So I have many installations with mismatching files. Now as I said I have the challenge of slow and expensive connections, thus my file per file manifest check.

At present my version range is 5.2.0 to 5.4.17 so when I release (presently 5.4.17) I have a differential patch for versions:
5.2.0 to 5.4.17
5.2.1 to 5.4.17
5.2.2 to 5.4.17
etc. etc. etc.

My program uses a templete file in which for the from and to versions I use the build time constants #OLDVERSION# and #NEWVERSION# as I have organised my release folders as such
\5.2.0
\5.2.1
\5.2.2
etc. etc. etc.

So doing an unattended build I generate the Manifest.xml, a changelog with the changes between #OLDVERSION# and #NEWVERSION# and the differential patch.

I will try drawing the lines in batches and only reporting mismatches.

[AdrianS]

We ran into this problem when users tried to manually edit resources in our DLLs. The editted files' MD5s didn't match what VP was expecting. When a file can't be patched because its MD5 doesn't match, VP is silent about the failure.

We resolved the problem by adding a g_OnFileProcessed callback function. If the ResultCode is 4, we display an error message indicating that the file could not be patched.

[ryanc]

Thanks for that bit of info Adrian, I will look to this as means of verifying file patch success. I do some sometimes use excessive force instead of the simple solution :wow overly vealous!

What I'll do then is that during patching and then prompt the user to download the mismatching files from a our file repos. I guess this makes my script kind of redundent, oh well it was a good learning experience.

[jlsf2]

We ran into this problem when users tried to manually edit resources in our DLLs. The editted files' MD5s didn't match what VP was expecting. When a file can't be patched because its MD5 doesn't match, VP is silent about the failure.

We resolved the problem by adding a g_OnFileProcessed callback function. If the ResultCode is 4, we display an error message indicating that the file could not be patched.

Adrian, where did you add this callback function? Thanks.

[AdrianS]

We put it in "Global Functions". For more info, you can search on g_OnFileProcessed in VP2's help. It's under Miscellaneous/Hidden Functions in the contents.