[Script] MD5 Verification

[ryanc]

Good Day

I have been using visual patch for some time now to build differential patches and must say, it's awesome! But sometimes when deploying it appears that files are not being updated. I now that is because of files not being upgraded in the past, due to an omission on my part

So I have written this function to scan the patched files post patching to confirm they have been updated. If the file is found to not be matching, it is added to a table which I interate through and download from our file server.

Now the first question I want to ask, am I eating a ice cream with a shovel? I s there some other way to do this?

Also, the other thing is that I update a scrolling text dialog. I added a pause to prevent the screen from flickering (this only seems to happen on lower end systems), is there some to avoid the redraw flicker?

--[[
********************************************************************************
Function: ManifestCheck
Purpose: Compare the installed files MD5 Hashes to the data contained in the
	Manifest XML file. Also logs the progress in Scrolling Text Control if it is
	present.
Arguments: None;
Returns:	(boolean) true - all files match, false - there are mismatches 

Say I have three files:
c:\Helpdesk\Helpdesk.exe, 
c:\Helpdesk\LocaliseEN.dat
c:\Helpdesk\Reports\IssuesByDate.rpt

The Template for the manifest file is as follows

<?xml version="1.0" encoding="iso-8859-1"?>
<Manifest>
  <helpdeskexe>
    <File>Helpdesk.exe</File>
    <MD5>058056d54f2ca3620ca1d34a28ff0a67</MD5>
    <Size>560128</Size>
  </helpdeskexe>
  <localiseendat>
    <File>LocaliseEN.dat</File>
    <MD5>d77efbb37a61f53b654b0cf6eca1313a</MD5>
    <Size>73728</Size>
  </socketsdll>  
  <reportsissuesbydaterpt>
    <File>Reports\IssuesByDate.rpt</File>
    <MD5>42d33f5cbce69929aaffb96e1643aaaa</MD5>
    <Size>66816</Size>
  </reportsissuesbydaterpt>
</Manifest>

********************************************************************************
--]]
function ManifestCheck()

	Log("Start Manifest Check");
	
	-- MD5 Compare Result
	local sMatchRes = "";
	
	-- File to be Evaluated
	local sFile = "";
	
	-- Expected MD5 for Evaluated File
	local sMD5 = "";
	
	-- Actual MD5 of the file installed
	local newMD5 = "";
	
	-- The Path to Your Server from which the manifest file can be downloaded
	local sHTTPServ = "www.myserver.com/updates/"
	
	-- Boolean to return if the files are correct
	local bSuccess = true
	
	
  if File.DoesExist(SessionVar.Expand("%SourceFolder%\\Manifest.XML")) == false then
  	StatusDlg.Show();
 		HTTP.Download(sHTTPServ.."manifest.xml", SessionVar.Expand("%SourceFolder%\\Manifest.XML"), MODE_BINARY);
   	StatusDlg.Hide();
	end;
	
	XML.Load(SessionVar.Expand("%SourceFolder%\\Manifest.XML"));

	local min = 1;  -- The number to start at
	local max = XML.Count("Manifest", "*");  -- The number to stop at
	local count;

	--Interate through our list and check for mismatches
	for count = min, max do
		sFile = XML.GetValue("Manifest/*:"..count.."/File");
		sMD5  = XML.GetValue("Manifest/*:"..count.."/MD5");
		sSize = XML.GetValue("Manifest/*:"..count.."/Size");
		tDescription = 	String.SplitPath(sFile);

		if File.DoesExist(SessionVar.Expand("%SourceFolder%\\"..sFile)) then
			newMD5 = Crypto.MD5DigestFromFile(SessionVar.Expand("%SourceFolder%\\"..sFile));
			if newMD5 ~= sMD5 then
				sMatchRes = "MD5 Hashes for Files "..sFile.." are not correct redownload";
				AddManifestDownload(sFile, sSize, tDescription.FileName);
				bSuccess = false;
			else
				sMatchRes = "";
			end;

		else
			sMatchRes = "File "..sFile.." is missing!";
   		AddManifestDownload(sFile, sSize, tDescription.FileName);
   		bSuccess = false;
		end
		if (CTRL_SCROLLTEXT_BODY) ~= nil then
			if sMatchRes ~= "" then
				DlgScrollingText.AppendLine(CTRL_SCROLLTEXT_BODY, sMatchRes, true, false);
				Log(sMatchRes);
			end;
		end;
		
		Application.Sleep(100);
	end
	DlgScrollingText.AppendLine(CTRL_SCROLLTEXT_BODY, "File Check Completed", true, false);
	LogLine();
	
	return bSuccess;

end;

--[[
********************************************************************************
Function: AddDownload
Purpose: Add A File to the Download List
Arguments:
	FileName - Name of the File,
	RemotePath - Path of the File relative to Base Path
	LocalFile - Local Filename and path to save the file to
	Execute - Execute the file after all downloads are completed
	ForceOverWrite - Delete the current file before downloading - force install
Returns: True or False if the download was added to the list
********************************************************************************
--]]
function AddDownload(FileName, RemotePath, LocalFile, Execute, ForceOverWrite, FileSize, Description)
	local tblNew = {};

	if FileSize == nil then
	 FileSize = 0
	end;
	
	if Description == nil then
		Description = FileName;
	end;

	tblNew.FileName = FileName;
	tblNew.RemotePath = RemotePath;
	tblNew.LocalFile = LocalFile;
	tblNew.Execute = Execute;
	tblNew.ForceOverWrite = ForceOverWrite;
	tblNew.FileSize = FileSize;
	tblNew.Description = Description;
	
	local max = Table.Count(tblDownloads);

	local i
	local item
	--Check for dupes
	for i,item in tblDownloads do
	  if String.Lower(item.FileName) == String.Lower(FileName) then
	    return false
	 end;
	end;

	Table.Insert(tblDownloads, max + 1, tblNew);
	return true;
end;

--[[
********************************************************************************
Function: AddManifestDownload
Purpose: Add A File from the Manifest list to the Download List
Arguments:
	AFileName - Name of the File with the path relative to the aura installation
	AFileSize - Size of the File
	ADescription - Description of the File
********************************************************************************
--]]
function AddManifestDownload(AFileName, AFileSize, ADescription)
	-- Convert the folder paths to be web friendly
	local path_parts = String.SplitPath(AFileName);
	local remotepath = String.Replace(path_parts.Folder, "\\", "/", false)
		
 	remotepath = "current/"..remotepath;
 	
	local remotefile = path_parts.Filename..path_parts.Extension;
	
  AddDownload(remotefile, remotepath, SessionVar.Expand("%SourceFolder%")..path_parts.Folder.."\\"..remotefile, false, true, AFileSize, ADescription);
end;

[Lorne]

Originally Posted by ryanc

I have been using visual patch for some time now to build differential patches and must say, it's awesome!

Thanks!

But sometimes when deploying it appears that files are not being updated. I now that is because of files not being upgraded in the past, due to an omission on my part

What kind of omission do you mean, actually? Were you not including some files in your patch project? Or...?

So I have written this function to scan the patched files post patching to confirm they have been updated. If the file is found to not be matching, it is added to a table which I interate through and download from our file server.

That's a pretty slick system, although it involves some duplicate work, since Visual Patch verifies MD5 checksums as part of the patching process. How are you generating the manifest XML file?

Now the first question I want to ask, am I eating a ice cream with a shovel? Is there some other way to do this?

Possibly. It depends what problem you're trying to solve, though...if it's just from forgetting to include files in the project you might be able to move the validation to build time by using the project report tool or the build log.

Also, the other thing is that I update a scrolling text dialog. I added a pause to prevent the screen from flickering (this only seems to happen on lower end systems), is there some to avoid the redraw flicker?

Sometimes combining lines can help (fewer draws)...but some systems just have slow redraws.

[ryanc]

Howdy

Thanks for the reply. I generate my manifest using an application I wrote for generating my updates. As my of my users are on dial up (South Africa is VERY behind in broadband deployment) I build differential patches for every possible release from the time I started using True Update.

The omission is more a conflict, as some support technicians in the past have had a very cowboy attitude in terms of releasing patches. So I have many installations with mismatching files. Now as I said I have the challenge of slow and expensive connections, thus my file per file manifest check.

At present my version range is 5.2.0 to 5.4.17 so when I release (presently 5.4.17) I have a differential patch for versions:
5.2.0 to 5.4.17
5.2.1 to 5.4.17
5.2.2 to 5.4.17
etc. etc. etc.

My program uses a templete file in which for the from and to versions I use the build time constants #OLDVERSION# and #NEWVERSION# as I have organised my release folders as such
\5.2.0
\5.2.1
\5.2.2
etc. etc. etc.

So doing an unattended build I generate the Manifest.xml, a changelog with the changes between #OLDVERSION# and #NEWVERSION# and the differential patch.

I will try drawing the lines in batches and only reporting mismatches.

[AdrianS]

We ran into this problem when users tried to manually edit resources in our DLLs. The editted files' MD5s didn't match what VP was expecting. When a file can't be patched because its MD5 doesn't match, VP is silent about the failure.

We resolved the problem by adding a g_OnFileProcessed callback function. If the ResultCode is 4, we display an error message indicating that the file could not be patched.

[ryanc]

Thanks for that bit of info Adrian, I will look to this as means of verifying file patch success. I do some sometimes use excessive force instead of the simple solution overly vealous!

What I'll do then is that during patching and then prompt the user to download the mismatching files from a our file repos. I guess this makes my script kind of redundent, oh well it was a good learning experience.

[jlsf2]

Originally Posted by AdrianS

We ran into this problem when users tried to manually edit resources in our DLLs. The editted files' MD5s didn't match what VP was expecting. When a file can't be patched because its MD5 doesn't match, VP is silent about the failure.

We resolved the problem by adding a g_OnFileProcessed callback function. If the ResultCode is 4, we display an error message indicating that the file could not be patched.

Adrian, where did you add this callback function? Thanks.

[AdrianS]

We put it in "Global Functions". For more info, you can search on g_OnFileProcessed in VP2's help. It's under Miscellaneous/Hidden Functions in the contents.