can C++ read a text file from the internet?

acejj26

Senior member
Dec 15, 1999
886
0
0
hey guys, i need some help again.

i'm writing a C++ program that needs to be able to read a text file from the internet. i will know the exact location of the file, so the location can be hardcoded.

i tried something like : fileloc="http://www.testsite.com/testfile.txt";

however, that didn't work.

is it possible to get this to work??
 

Descartes

Lifer
Oct 10, 1999
13,968
2
0
Absolutely, but not in standard C++. C++ has no notion of URLs and network resources, as accessing these resources is usually implementation defined (i.e. it's not defined in the ANSI standard). You didn't mention your platform, but if you're targeting win32, you can use the Internet* functions. On other platforms, you can use straight sockets and simply generate an HTTP GET request and read the response. I wrote a program a long time ago to use the win32 functions that I called "httprip" that simply took a URL, and saved the response to a local file. I used this to grab, and dynamically load, DLLs from remote resources into my applications (a primitive "automatic update", if you will). This was the console version, and the code is probably a little dirty, but hopefully it will get you on your way. It's a bit incomplete as I wrote another version in Java, then C#, and it's fairly long, so sorry about that... Let me know if you have any questions on it.

You can compile this by simply running "cl httprip.c wininet.lib" from the command-line.

#include <windows.h>
#include <wininet.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define HTTP_USER_AGENT "HTTPRIP"
#define BUFFER_SIZE 512

void usage(void);
void die(const char *message);

const char http[] = "HTTP";

int main(int argc, char **argv)
{
HANDLE file;
SECURITY_ATTRIBUTES sa;
HINTERNET internet;
HINTERNET url;
char *requesturl = NULL;
char *outputfile = NULL;
char *buf = NULL;
DWORD bytesread;
DWORD byteswritten;
int k;
int usestdout = 0;

if (argc == 2)
// user wants data directed to stdout
usestdout = 1;
else if (argc != 3)
usage();

requesturl = argv[1];

if (!usestdout)
outputfile = argv[2];

// validate protocol -- first 4 alphas should be http, case insensitive
for (k = 0; k < 4; k++)
if (requesturl[k] != http[k] && requesturl[k] != (char)(http[k] + 32))
die("Invalid url");

if (!usestdout)
{
file = CreateFile(outputfile,
GENERIC_WRITE,
FILE_SHARE_READ,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (file == INVALID_HANDLE_VALUE)
die("CreateFile()");
}

printf("Opening connection...\n");
internet = InternetOpen(HTTP_USER_AGENT,
INTERNET_OPEN_TYPE_PRECONFIG,
NULL,
NULL,
0);
if (internet == NULL)
die("InternetOpen()");

printf("Requesting url...\n");
url = InternetOpenUrl(internet,
requesturl,
NULL,
0,
INTERNET_FLAG_PRAGMA_NOCACHE,
0);
if (url == NULL)
die("InternetOpenUrl()");

buf = malloc(BUFFER_SIZE + 1);
if (buf == NULL)
die("Unable to allocate memory for receive buffer");

printf("Saving file");
while (InternetReadFile(url,
buf,
BUFFER_SIZE,
&bytesread))
{
if (bytesread == 0) break;

if (!usestdout)
{
WriteFile(file,
buf,
bytesread,
&byteswritten,
NULL);

putchar('.');
}
else
//fwrite( const void *buffer, size_t size, size_t count, FILE *stream );
fwrite(buf, bytesread, 1, stdout);
}
putchar('\n');

free(buf);
InternetCloseHandle(url);
InternetCloseHandle(internet);
CloseHandle(file);

printf("Done.\n");

return 0;
}

void usage(void)
{
fprintf(stderr, "http rip - Chad Osgood\n\nusage: httprip url filename\n");
exit(EXIT_FAILURE);
}

void die(const char *message)
{
fprintf(stderr, "%s", message);
exit(EXIT_FAILURE);
}

[edit]My use of variables named i for array indexes hosed the formatting. Fixed![/edit]
 

acejj26

Senior member
Dec 15, 1999
886
0
0
i get the following error when i try to compile your sample program:

fatal error C1010: unexpected end of file while looking for precompiled header directive

the program i have written is not a console app...it is a windows app, with forms and buttons. i need code inside this program to read in the text file that will be stored in a specified web address (i.e. "http://www.website.com/file.txt")

if i can embed said code, i'll be extremely happy
 

acejj26

Senior member
Dec 15, 1999
886
0
0
i get the same error regardless of whether i am using MFC

my gui app works without using MFC
 

singh

Golden Member
Jul 5, 2001
1,449
0
0
Originally posted by: acejj26
i get the same error regardless of whether i am using MFC

my gui app works without using MFC

I was just asking because I had some smaller code in case you were.
 

acejj26

Senior member
Dec 15, 1999
886
0
0
i don't really know much about MFC

could you post your smaller code and let me try it out?? i'm just not sure even where to begin on allowing this kind of functionality

any and all help is always appreciated, guys. you guys have helped me out countless times. this is such a great community.
 

Descartes

Lifer
Oct 10, 1999
13,968
2
0
Ok, here's the update. I segregated the actual process of retrieving the file, and saving it into it's own function, so you can just call that instead. More lucid error handling is probably needed, as right now it just indicates success or failure. Also, as I wrote this in C, there's really nothing C++ about it, it doesn't take advantage of any of C++'s advanced features. It will, however, compile without issue in C++ if you cast the return value of the malloc() call. I don't cast malloc calls in C, because the C standard defines that a void * can be assigned any type, and I only need to cast it to the appropriate type when dereferencing the value. All you need to do is call InternetSaveFile with the url, and the local filename as the parms, and simply access the local filename using conventional methods (open()/fopen()/ifstream, whatever). I've also posted a properly formatted version here for the time being. Hope this helps.

#include <windows.h>
#include <wininet.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define HTTP_USER_AGENT "HTTPRIP"
#define BUFFER_SIZE 512

void usage(void);

int InternetSaveFile(const char *url, const char *filename);

int main(int argc, char **argv)
{
FILE *input = NULL;
char buffer[BUFFER_SIZE] = { 0 };

if (argc != 3)
{
usage();
exit(EXIT_FAILURE);
}

if (InternetSaveFile(argv[1], argv[2]))
{
input = fopen(argv[2], "r");
if (input != NULL)
{
while (fgets(buffer, BUFFER_SIZE, input) != NULL)
printf("%s", buffer);

fclose(input);
}
else
printf("Unable to open input file!\n");
}
else
printf("Failure!\n");

return 0;
}

void usage(void)
{
fprintf(stderr, "http rip - Chad Osgood\n\nusage: httprip url filename\n");
}

int InternetSaveFile(const char *requesturl, const char *outputfile)
{
const char http[] = "HTTP";
HANDLE file;
SECURITY_ATTRIBUTES sa;
HINTERNET internet;
HINTERNET url;
char *buf = NULL;
DWORD bytesread;
DWORD byteswritten;
int k;

// validate protocol -- first 4 alphas should be http, case insensitive
for (k = 0; k < 4; k++)
if (requesturl[k] != http[k] && requesturl[k] != (char)(http[k] + 32))
return 0; // need error handling

file = CreateFile(outputfile,
GENERIC_WRITE,
FILE_SHARE_READ,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (file == INVALID_HANDLE_VALUE)
return 0; // need error handling

internet = InternetOpen(HTTP_USER_AGENT,
INTERNET_OPEN_TYPE_PRECONFIG,
NULL,
NULL,
0);
if (internet == NULL)
return 0; // need error handling

url = InternetOpenUrl(internet,
requesturl,
NULL,
0,
INTERNET_FLAG_PRAGMA_NOCACHE,
0);
if (url == NULL)
return 0; // need error handling

buf = malloc(BUFFER_SIZE + 1);
if (buf == NULL)
return 0; // need error handling

while (InternetReadFile(url,
buf,
BUFFER_SIZE,
&bytesread))
{
if (bytesread == 0) break;

WriteFile(file,
buf,
bytesread,
&byteswritten,
NULL);
}

free(buf);
InternetCloseHandle(url);
InternetCloseHandle(internet);
CloseHandle(file);

return 1;
}

[edit]Also, I currently have it printing the contents of the buffer to stdout after the call to fgets(), but you'll probably want to kill that...[/edit]
 

acejj26

Senior member
Dec 15, 1999
886
0
0
descartes,

i appreciate everything, but i have a question.....i tried compiling and i got this error:

error C2440: '=' : cannot convert from 'void *' to 'char *'
Conversion from 'void*' to pointer to non-'void' requires an explicit cast

the error occurs in the line:

buf = malloc(BUFFER_SIZE + 1);

any ideas?
 

Descartes

Lifer
Oct 10, 1999
13,968
2
0
ace: Yup, that's what I was talking about when I said it's not 100% C++ ready. Just convert this...

buf = malloc(BUFFER_SIZE + 1);

to this...

buf = (char *)malloc(BUFFER_SIZE + 1);

and I'd rename the extension to .cpp. That's a C-style cast, but it's still valid C++. As Stroustrup said, good C is also C++. I compiled using VC++6 by executing the following at the CLI...

cl httprip2.cpp wininet.lib

Btw, what compiler are you using?
 

singh

Golden Member
Jul 5, 2001
1,449
0
0
Originally posted by: Descartes
I compiled using VC++6 by executing the following at the CLI...

cl httprip2.cpp wininet.lib

Oh, command-line compilation, the horror! :Q

:p
 

acejj26

Senior member
Dec 15, 1999
886
0
0
i'm using VC++ 6, too

i get the following errors now:

httprip2.obj : error LNK2001: unresolved external symbol __imp__InternetCloseHandle@4
httprip2.obj : error LNK2001: unresolved external symbol __imp__InternetReadFile@16
httprip2.obj : error LNK2001: unresolved external symbol __imp__InternetOpenUrlA@24
httprip2.obj : error LNK2001: unresolved external symbol __imp__InternetOpenA@20

whaddaya think?? i've had similar issues before, and it was because i didn't have a "lib" file in the right place. am i correct in assuming that here? is this the "wininet.lib" you're referring to?? how do i create this?

EDIT: never mind...i found the file and included it in my project...it runs now :)
i'll play with it for a while and see if i can get it to read in my file from the internet
 

acejj26

Senior member
Dec 15, 1999
886
0
0
descartes,

i'm having some troubles figuring out what your program actually does now....i'm not too familiar with some of the calls you're making, what you're passing back and forth, and what your program is doing.

could you give me a broad overview of what it's trying to do, so that i'll be able to make it grab my file from a URL that i specify (hardcoded)??
 

Descartes

Lifer
Oct 10, 1999
13,968
2
0
you could use sockets. I've never used INET before but I've used sockets.

Indeed you could. I whipped up a quick example to show how sockets could be used. It's incomplete, as there's no dns resolution, and it only retrieves the root doc, but it could easily be changed to do otherwise. I've posted it here. Of course, it's so much easier in C#, I provided that here. I wrote them quickly just to show how to do it, so the usual caveats apply (need error handling, etc.).

 

acejj26

Senior member
Dec 15, 1999
886
0
0
one last question, and i think i'm good with this. is there any way to pass the url and the filename to the program without having to run it from the command line?? or, can i call this program from another program while passing the parameters?? here's what i'm doing:

a program that i have already written in C++ generates a file that is stored on the webserver.
this program that you guys have helped me with takes that file and stores it on the user's machine.

now, i what i want to do is this:

embed the code from the second program into the first program. when the user clicks on the print button (i'm writing a program that prints smart labels based on the information from the file stored on the webserver), i want the program to grab the file from the internet, store the file on the user's machine, then read this file. so, i'll have to somehow hardcode the url and the filename in the program and have the embedded source code read this in so it knows where to get the file and where to store it.

descartes, any help on this? i'm so close to finishing this....i just need a little more guidance
 

Descartes

Lifer
Oct 10, 1999
13,968
2
0
ace: All you need is my second example, httprip2.c. All you need to do is call InternetSaveFile() with your url, and local filename, and then read that file using any of the conventional file access methods (open()/fopen()/ifstream). Just a quick example, assuming you've copied my routine, InternetSaveFile() from httprip2.c into your new source file.

FILE *input = NULL;
if (InternetSaveFile("yoururl", "yourfilename"))
{
input = fopen("yourfilename", "r");
if (input == NULL) ; // handle error condition
// do whatever you need to do with the file
fclose(input);
}
else
fprintf(stderr, "Couldn't save file");

just plug that code into your main(), or wherever you need it and you should be kosher dill pickle. The httprip2.c is a complete example of this, so you only have to modify it to your needs. PM, or post, if you have any more questions...