Problem getting HTML data from Web object

[johnraus]

Hi, I have a little problem getting HTML source from a secured website.
I need a Table with Data thats on this website
(HTTPS with logins/coockies and sessions and ...?)

This website is very tricky, but browsing with Web object is no problem ! event right click and view source works, but it shuold be an automated task.
I tried HTTPS.DownloadSecure / HTTP.SubmitSecure with correct ports and Logins to get file locally but no solution.

Now I created a new Generic TextOnly Printer, to store a file locally and use Web.Print to store the Text Data in a file. Then I process this data to a Table and back to HTML again ... :eek: ... But this is very sloppy and timeconsuming since the table is messed up by Text2File printing.

Does anybody have a more proper solution, to get the HTML source from the Web object ... ? :huh

[Worm]

do you want the source or the content on the page?

[johnraus]

do you want the source or the content on the page?
I would like to have the source.
This is because it's easier to get Data from the tables in the webpage

[johnraus]

do you want the source or the content on the page?
Some additional info:

I would like to login on a Website (the tricky one, see first post) with the Web Object

in AMS I'll make a button wich navigates to particular page on this server(this is already O.K.), get source code of displayed page, extract some data from HTML source code, run some statistics and create a local HTML file with a report of the calculations.

This is why I would like to be able to access HTML Source code from the page currently being displayed in the web object

This would make my AMS project really usefull.

PS
the second issue would be cool to automate a search on this website and see the results in the Web.Object,
but then I would need to POST Data to the WEB.OBJECT
the search URL is like:
https://aaa.bbb.com:1234/ABC/search_doc.jsp
Post DATA like:
Parameter: (Mime Type) Value application/x-www-form-urlencoded
Parameter: docid Value:123456789-012
tried HTTP.Submitsecure, but it seems it's not the same session as the WEB OBJECT so this fails, The only thing I seem to be able to automate is Browsing this website with the web object.

Any Help would really be arappriciated :rolleyes

[Worm]

Getting the source is simple enough once you're on the page


Web.LoadURL("Web1", "javascript:alert(document.all[0].outerHTML)")


How to get that info into AMS is the hard part. I tried passing it in the URL, but it is truncated. I thought maybe it could be set as a cookie, then retrieved within AMS some other way, but haven't had much of a chance to try that.

Another thought would be to use:

Web.LoadURL("Web1", "javascript:document.execCommand(\"SaveAs\",\"false\", \"C:\\\\htmlsource.txt\")")


In conjunction with the KeyStrokes dll to automate the save, but that doesn't seem real clean either. I'll see what I can come up with, but that's the best I have so far.


")

[Worm]

This isn't pretty, but it seems to do the trick. There has to be a better way.

Cookies are limited to 4K, so thats out...

[Worm]

Here's a much cleaner solution... it uses the luaCom plugin from (http://www.icynorth.com)

[johnraus]

Here's a much cleaner solution... it uses the luaCom plugin from (http://www.icynorth.com)

Thnx Worm for your interest, and helping me ...

I'll Try all of your solutions right now, so I'll be busy for a while

[Worm]

Okay, sorry to keep going on this, but this is too cool to not share.

I extended the luaCom project a little more. The issue before was that your user couldn't really interact with the "automated" Internet Explorer Window. It hit me that I might be able to embed the Internet Explorer Window in AMS to emulate the web object.

Guess What!? I can, and did! With a couple handy API calls to remove the title bar and to set it's parent, I was able to take the Internet Explorer Window and make it look like the Web Object within the AMS project. Now, you can control the Internet Explorer Window within your AMS project via luaCom, plus the users can interact with it too.

Anyhow, this should give you a seamless solution. This goes to prove to me, yet again, how powerful AMS can be.

[Intrigued]

[Can't resist]

"You don't know the power of the Code Side! Come Worm, join me... we will rule the galaxy together... as goofy guy 1 and ... goofy guy2!"

[/Can't resist]

Hard.Message("Dee", "Har-Har!")

:D

:yes (btw, this is pretty nifTAY!)

[johnraus]

Okay, sorry to keep going on this, but this is too cool to not share.

I extended the luaCom project a little more. The issue before was that your user couldn't really interact with the "automated" Internet Explorer Window. It hit me that I might be able to embed the Internet Explorer Window in AMS to emulate the web object.

Guess What!? I can, and did! With a couple handy API calls to remove the title bar and to set it's parent, I was able to take the Internet Explorer Window and make it look like the Web Object within the AMS project. Now, you can control the Internet Explorer Window within your AMS project via luaCom, plus the users can interact with it too.

Anyhow, this should give you a seamless solution. This goes to prove to me, yet again, how powerful AMS can be.

Yep, this is to COOL to be true THNX very much, you even provided solution before I could ask the question, because your previous "luaComGetSource" started another inctance of IE and I would end at the logonpage for that website, this one should solve this issue. :yes :cool

I'm not very expirienced so where can I find more info on controlling this Internet Explorer Window with the LUAcom plugin for (Back-Fwd button, get displayed url (like e_URL on loaded) or even Post data to this IE window. :huh

[johnraus]

:wow
WOW I just discovered info on controlling IE instance, in AMS project
This is more than I asked for .... :p

MSDN info in iternetexplorer object (http://msdn.microsoft.com/library/default.asp?url=/workshop/browser/webbrowser/reflist_vb.asp)

This is really COOooooollll :cool
THNX again Worm for your effort :yes

on minor drawback, is when th IE object has focus, the onmouse-over effect in AMS is gone till I click AMS window, but I can live with this. The control of this IE object is a boost for my creativity for me for now ... Lots of hours of playtime to go to expand my project ....

[Worm]

one minor drawback, is when th IE object has focus, the onmouse-over effect in AMS is gone till I click AMS window, but I can live with this.

No need to live with it when it can be overcome... :)

Here's the fix, I used my mouse.dll to determine which window the pointer is over, then use the User32.dll function SetFocus to put the focus on the correct window.

When/If you do more automation with IE, I'd love to see what you come up with.

I agree with you on this one... it opens up a whole new avenue with web integration. I'm very interested in how well this will perform. Please keep me updated.

[johnraus]

Hey Worm, you did it again :yes :D
I'll keep you posted...

[Worm]

Bug fix:

In On Preload the action:
oIE.Resizable = false;

needs to come after the DLL.CallFunction statements. Otherwise the IE window is resizable inside the AMS app. Although that's kinda cool, its not the what you're looking to do.

[Worm]

Bug Fix:
Needed to add code to set the focus only when needed in the timer event to keep the EnsureFocus function from taking the focus out of other objects like the input box.

Replace the code in the Global Functions with this.

function EnsureFocus()
if oIE then
nX=String.ToNumber(DLL.CallFunction("AutoPlay\\Docs\\mouse.dll", "GetMouseX"
, Application.GetWndHandle()
, DLL_RETURN_TYPE_LONG, DLL_CALL_STDCALL));

nY=String.ToNumber(DLL.CallFunction("AutoPlay\\Docs\\mouse.dll", "GetMouseY"
, Application.GetWndHandle()
, DLL_RETURN_TYPE_LONG, DLL_CALL_STDCALL));

tWinPos = Window.GetPos(oIE.hWnd)
tWinSize = Window.GetSize(oIE.hWnd)
if nY > 0 and nX >0 then
-- were in the AMS app window
if (nX < tWinPos.X or nX > tWinPos.X + tWinSize.Width) and (nY < tWinPos.Y or nY > tWinPos.Y + tWinSize.Height) then
--we're over the explorer window
if sLastObject ~= "IE" then
nResult = DLL.CallFunction(_SystemFolder.."\\User32.dll", "SetFocus", oIE.hWnd, DLL_RETURN_TYPE_LONG, DLL_CALL_STDCALL)
sLastObject = "IE"
end
else
--we're not over the explorer window
if sLastObject ~= "AMS" or sLastObject == nil then
nResult = DLL.CallFunction(_SystemFolder.."\\User32.dll", "SetFocus", Application.GetWndHandle(), DLL_RETURN_TYPE_LONG, DLL_CALL_STDCALL)
sLastObject = "AMS"
end
end
end
end
end

[Worm]

Here is the latest version with the above bug fixes.

[johnraus]

Yep, I'm back again, found another small problem.

I've created a Forward and Back button for oIE

with oIE.GoBack() and oIE.GoForward()

and they work .....
but if there are no I've visited and I click the FWD or BACK button I get a LUAcom error. is there any way to intercept these errors ?

[Worm]

Put this in your IE events table, it'll enable and disable the buttons for you as needed. You'll probably need to change the button names, but other than that, it's good to go. :p


function IE_Events:CommandStateChange(nCommand, blnEnable)
nCommand = String.ToNumber(nCommand)
if nCommand == 1 then
Button.SetEnabled("btnForward", blnEnable)
elseif nCommand == 2 then
Button.SetEnabled("btnBack", blnEnable)
end
end

[johnraus]

Thnx again Worm ;)

I saw yor other thread on IE events, and also read about these events on MSDN. the only difficulty for me is sometimes how to implement thes in AMS

[Worm]

I'm fortunate that I can fire up Visual Basic and take a look at the events that way too. It's shedding some interesting light on the use of luacom. At least for me it is.

I'm happy to help, anytime.

[johnraus]

Hi all

this embedded oIE really rocks, but I've come across another problem

in globals:

IE_Events={}

function IE_Events:BeforeNavigate2(pDisp, URL, Flags, TargetFrameName, PostData, Headers, Cancel)
Debug.Print("BeforeNavigate-URL : "..URL.."\r\n");
end

but this event never gets triggered ? I would like to extract the PostData when a user fills in a form on this website

I use XP sp2, and all other events seem to work, also tried BeforeNavigate, but never gets triggered

any suggestions ...

[Worm]

I'll take a look when I get home tonight and see if it's something I have wrong in the event functions.

[Intrigued]

Also, then could not we do such third party application embedding via luacom, say MS Word and MS Excel...

:D

[Intrigued]

Coo'! I just changed:

oIE.Resizable = false;

to...

oIE.Resizable = true;

[Worm]

I can confirm that the events don't seem to be firing... but I don't know why. At least not yet anyway :)

[johnraus]

Thnkx for looking into this issue

found related topic on microsoft support website Q311298, but that one is related to Visual Basic .NET 2002, and states it has something to do with the COM and ICOM interface, they offer a workaround for Visual Basic.Net, but my knowlidge of programming is to little to understand what they are saying.

http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q311298&ID=KB;EN-US

Maybe there is a qlue inside this article, how to fix this issue ...

[johnraus]

Hi all,

found somthing really cool getting HTML SOURCE or just TEXT
from a WEB object inside AMS, little dirty but very usefull.

The advantage is that you don't have to save it to disk first befor you can get inside a variable to process it

So if you don't wanna use LUAcom and IE, just plain AMS this one is for you.
(you need the clipboard plugin for this one


function GetSourceFromWeb(html) -- Retrieve Content from Embedded Web Object
-- if html is TRUE returns loaded HTML data
-- if html is FALSE returns loaded TEXT data

local datastring = "";
local remember = "";
local rcount = 0;

if Clipboard.IsTextAvailable()
then
remember = Clipboard.GetText(); -- Only works with text on clipboard
Clipboard.CopyText("");
end;
Application.Sleep(10);
if html
then Web.LoadURL("WEB", "javascript:void(window.clipboardData.setData(\"Text\", document.body.createTextRange().htmlText));");
else Web.LoadURL("WEB", "javascript:void(window.clipboardData.setData(\"Text\", document.body.createTextRange().text));");
end;
repeat
Application.Sleep(10);
rcount = rcount +1;
until Clipboard.IsTextAvailable() or rcount==50;
if rcount < 50
then
datastring = Clipboard.GetText();
Application.Sleep(10);
Debug.Print("Get Source from Web O.K. "..rcount.."\r\n");
else
Debug.Print("Get Source from Web FAILED\r\n");
end;
Clipboard.CopyText(remember);

return datastring;

end;

[TristanD]

worm, i love you XD :D :lol :p :p thank for leaing me here as i just found the whole **** list with Microsoft Browser Active X Commands and controls :D:D:D:D:D:D