PHP: Problem Retreveing XML file from remote server

[Syran]

I am having trouble retrieving an XML file from the OFAC Website when using PHP.

http://www.treasury.gov/ofac/downloads/sdn.xml
(Warning: BIG Link, 6Meg xml file)

I didn't have any problems with things working until about Feb of this year. The failures did not seem to match with any php update I've done as far as I can tell.

I can manually download it without a problem.
I can remotely copy it from a URL on my test server to my primary server via the PHP copy command.

The code i've used in the past is:

        ini_set("user_agent","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
	ini_set("max_execution_time", 0);
	ini_set("memory_limit", "10000M");
		
	$ol = get_ofac_link( $id );
	//function to retreive the SDN list from the Treasury Department. Will Error out if you cannot load the file.
	$xml = simplexml_load_file($ol ) or die("Error retreiving File: ".$ol );
			
	$pb = $xml->publshInformation;
	
	//Updates the information on the SDN List
	$uq = "UPDATE sdnlist
			SET date = '$pb->Publish_Date',
				records = '$pb->Record_Count'
			WHERE id = '$id'
			LIMIT 1";
	mysql_query( $uq) or die(mysql_error($link_list));
	
	if ($id == 1)
		{
		echo "Updating SDN Entries.<br>\n";
		flush();
	
		//This parses the XML file and passes the information into the database
		foreach ($xml->sdnEntry as $sdnEntry)
			{
			insert_into_sdn( $sdnEntry, $id );
			}
		}
	else
		{
		echo "Updating PLC Entries.<br>\n";
		flush();
		
		//This parses the XML file and passes the information into the database
		foreach ($xml->nspEntry as $sdnEntry)
			{
			insert_into_sdn( $sdnEntry, $id );
			}
		}
	
	$xml = NULL;

I have also tried using the copy command, and just get a 0 byte file. If I use links, I can access and pull it in normally. But I'm trying to keep this as something a user can do without me having to manually intervene.

My php.ini has both allow_url_fopen and allow_url_include = On.

Same server, I tried this code:

<?PHP
$url = "http://www.treasury.gov/ofac/downloads/sdn.xml";
copy($url, "/var/www/html/ofac/sdn.xml");
?>

It doesn't work. Creates the 0 byte file.

<?PHP
$url = "http://10.1.122.19/ofac/sdn.xml";
copy($url, "/var/www/html/ofac/sdn.xml");
?>

Works. Creates the 6.4Mb file.

Is there some wierd timeout I'm missing? It finishes the copy attempt in about 2 seconds either way.

 

[DaveSimmons]

Your user-agent is still IE 6.0, you might change that to IE 8.0 since some sites now block access from IE 6.

Other than that, the URL works for me in IE 8 and the trace from IEWatch (similar to Fiddler) doesn't show anything unusual going on.

It's HTTP so you could use a packet sniffer to see what is happening with the request / response, or just be lazy and use cURL or shell execture wget to grab the file if switching the user-agent doesn't help.

 

[Syran]

Your user-agent is still IE 6.0, you might change that to IE 8.0 since some sites now block access from IE 6.

Other than that, the URL works for me in IE 8 and the trace from IEWatch (similar to Fiddler) doesn't show anything unusual going on.

It's HTTP so you could use a packet sniffer to see what is happening with the request / response, or just be lazy and use cURL or shell execture wget to grab the file if switching the user-agent doesn't help.

Thanks!

I got lazy, tried curl, it wouldn't work either, but did get wget to work.

	$ol = get_ofac_link( $id );
	//function to retreive the SDN list from the Treasury Department. Will Error out if you cannot load the file.
	$link = $ol[httplink].$ol[hbasename];
	$locallink = "/var/www/html/ofac/".$ol[hbasename];
	shell_exec( "wget ".$link." -r -O ".$locallink );
	$xml = simplexml_load_file( $locallink ) or die("Error retreiving File: ".$locallink );

Not super-clean, but it works!

 

[uclabachelor]

Same server, I tried this code:

<?PHP
$url = "http://www.treasury.gov/ofac/downloads/sdn.xml";
copy($url, "/var/www/html/ofac/sdn.xml");
?>

It doesn't work. Creates the 0 byte file.

<?PHP
$url = "http://10.1.122.19/ofac/sdn.xml";
copy($url, "/var/www/html/ofac/sdn.xml");
?>

Works. Creates the 6.4Mb file.

Is there some wierd timeout I'm missing? It finishes the copy attempt in about 2 seconds either way.

DNS issue? The two URLs are also slightly different.