ASP Scripts to Download Datafeed

I've been trying for the past couple days to get a script working that will download a tigerdirect datafeed (~7 mb). The code below works perfectly for smaller files that I've tried (6 kb) but I get an error when I try to download the larger files through the script. My eventual goal is to automate the downloading, and the loading of the datafeed into my database a couple times a week so I don't have to be bothered with it.

Response.Buffer = True
Server.ScriptTimeout = 3600
Set xml = Server.CreateObject("MSXML2.XMLHTTP")
xml.Open "GET", "ftp://usernameassword@ip.address/feed.txt", false
xml.Send()
If err.Number <> 0 then
xml.abort()
Set xml = Nothing
Else
fl1 = xml.responseText
Set xml = Nothing
fl1 = Replace(fl1,"ID_HERE","XXXX")
Set fs = Server.CreateObject("Scripting.FileSystemObject")
Set fl = fs.CreateTextFile("C:\Path\feed.txt",True,False)
fl.WriteLine fl1
fl.Close
Set fl = Nothing
Set fs = Nothing
End If

The error I get when this is executed is (BTW, it's actually loading the 7 MB file before the error):

Microsoft VBScript runtime error '800a0005'
Invalid procedure call or argument
/tdftp.asp, line 18

Line 18 is: fl.WriteLine fl1

If I change the last parameter of line 17 to True (to save the file as unicode instead of ASCII) it saves it with no error, but the output in the file is unreadable, ex: ÿþ atrldpo|cttpalraanec aam sezc kaa 4nihr Pmf

If anybody has a solution to fix this script so it will work with the larger files I would appreciate it, also any better solutions for downloading the datafeeds from a FTP site, I would appreciate hearing about them as well.

[This message was edited by Haiko on July 30, 2003 at 01:28 PM.]

if you are just using it to download, use wget or ncftpget on the command line...

syntax:
<pre class="ip-ubbcode-code-pre">wget -q http://user:pass@site.com/path/to/datafeed.txt</pre>
(-q means quiet)
for ncftpget, the syntax is the same.

--
"The greatest good you can do for another is not just to share your riches, but to reveal to him his own." – Benjamin Disraeli
--
Ford Fox-body Owners Club -- http://www.ford-fox.org

Thanks FFoc, If I go an alternate route I may use your suggestion. But for now the point of what I'm trying to do is to have the retrieval and processing be completely unassisted through the server-side scripting.

There may be some sort of file size limitation in play. PHP has a 2 meg buffer pre-configured in the php.ini file. Perhaps there is something similar in asp.

Just a thought.

Chris

----------------------------
Scriptsforyoursite.com - Featuring datafeed import scripts for Backcountry Store, Coldwater Creek, Mondera.com and many more.....

I agree cditty, I do think it could be a buffer size limitation, and I've searched for hours on the topic w/ no solid leads for ASP. I think I'll try to do the same things in ASP.NET, maybe it's more tolerant.

Hi Mikey,

We had the same problem with our sample script for affiliates. The problem are special characters in the Response Text.

We changed the Script to use ADODB.Stream instead of WriteLine and it worked.

Here is the Code:


Dim strFullURL
Dim objSrvHTTP
Dim strText
Dim strResultFilename
Dim objFso

'Declare constants
strFullURL = "ftp://usernameassword@ip.address/feed.txt"
strResultFilename = "C:\Path\feed.txt"

'Use Microsoft XMLHTTP to get the Data
Set objSrvHTTP = CreateObject("MSXML2.XMLHTTP")
objSrvHTTP.Open "GET", strFullURL, False
objSrvHTTP.Send ""
strText = objSrvHTTP.ResponseText
Set objSrvHTTP = Nothing

'Delete (old) target file on local hard drive
Set objFso = CreateObject("Scripting.FilesystemObject")
If objFso.FileExists(strResultFilename) Then
objFso.DeleteFile strResultFilename
End If

'Create the stream
Set objStream = CreateObject("ADODB.Stream")

'Initialize the stream
objStream.Open

'Reset the position and indicate the character encoding
objStream.Position = 0
objStream.Charset = "ascii"

'Write data to file
objStream.WriteText strText

'Save the stream to a file
objStream.SaveToFile strResultFilename

Set objFso = Nothing

Hope that will help.

CC

___________________________
Bevmo.com Affiliate Manager
5%-10% Commission, 60 day cookie, 2 AM's
http://www.bevmo.com
https://www.bevmo.com/affiliates/
http://www.bevmoaffiliates.com

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by mikey9:
...to have the retrieval and processing be completely unassisted through the server-side scripting.<HR></BLOCKQUOTE>
Yep, we have a thread going on the subject here: http://www.abestweb.com 6/ubb.x?a=tpc...89&m=9426003283

--
"The greatest good you can do for another is not just to share your riches, but to reveal to him his own." – Benjamin Disraeli
--
Ford Fox-body Owners Club -- http://www.ford-fox.org

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by AM - Bevmo.com:
We changed the Script to use ADODB.Stream instead of WriteLine and it worked.
<HR></BLOCKQUOTE>

CC,
You're awesome! I can't tell you how much I appreciate your insight into this problem, especially since you were dead on, and with your suggestion my script was working in less than a minute. Thanks again.
Mike

Mike,

You are very welcome.

You are also welcome to join our affiliate program... now that you got the TigerDirect Datafeed to work. Just leave me a PM or e-mail me.

We run our Affiliate Program through CJ, but provide Product Datafeeds and more via our In-house Affiliate Intranet.

We have a Forum here at ABW as well:
http://www.abestweb.com 6/ubb.x?a=frm...79&f=6046025762

CC

___________________________
Bevmo.com Affiliate Manager
5%-10% Commission, 60 day cookie, 2 AM's
http://www.bevmo.com
https://www.bevmo.com/affiliates/
http://www.bevmoaffiliates.com

BTW, the Bevmo "backroom" and feed system are VERY well done - I only wish that all the other merchants whose feeds I used could be even HALF as advanced.. (only ones that even come close are Mondera, followed by Irvs/EE)

--
"The greatest good you can do for another is not just to share your riches, but to reveal to him his own." – Benjamin Disraeli
--
Ford Fox-body Owners Club -- http://www.ford-fox.org

CC it's a very tempting invitation more than ever. Your generosity in helping me get the script working was wonderful, however the very program I set it up for stabbed their datafeed affiliates in the back by revoking all use of their datafeed this morning. If I wasn't LDS (Mormon) I'd promote your stuff in a heartbeat because it looks like you have a very organized, and well treated affiliate community.

I understand, no problem.
Btw. BevMo sells Glassware as well .

CC

___________________________
Bevmo.com Affiliate Manager
5%-10% Commission, 60 day cookie, 2 AM's
http://www.bevmo.com
https://www.bevmo.com/affiliates/
http://www.bevmoaffiliates.com

Do not forget object cleanup....

add:

Set objStream = Nothing

right before:

"Set objFso = Nothing"

IamJaloppy

Does the ftp server I'm grabbing the info from register the IP of the server or the IP address of the client? My guess is that it's the IP of the server which would be great for Linkshare's ridiculous requirement of having a static IP.

Yes, I use the IP of my server for my linkshare merchandiser account.
Use the server to grab the file then. much quicker & easier once you figure out how to do it.

I use either wget or NET::FTP in a perl script to grab & process mine.

Can you post how to use Wget or a link to good instructions?

Actually on second thought, the above script gives me more flexibility simply because I know Vbscript.

So those who have used the vbscript above, is the IP address of the server indeeded the one that the FTP site recognizes? Anyone successfully used it with Linkshare?

um, ok, So I do this and get a file not found error:

ftp://meritline:affiliate@66.206.24...atafeed1012.txt

Any ideas why this doesn't work. Copy and paste this into your browser and it will also fail. If I leave off the text file at the end it logs in fine and I can manually download the file. Any ideas?

Looks like you got the filename wrong, Steele, put this in the URL:

20031102-M-Datafeed.txt

--
scott@befree.com

They changed the filename over night. However the new filename does not work either:

ftp://meritline:affiliate@66.206.24...-M-Datafeed.txt