31 decembrie 2008

Incarcare in MySQL pentru fisiere CSV de dimensiuni mari

Am avut o "mica" problema, de vreo 100 Mb, un fisier CSV care trebuia importat intr0o baza de date MySQL din  PHPMyAdmin sau script PHP.
Fisierul CSV se gaseste la
http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity_20081201.zip
Daca foloseam :
<?php
$file=file_get_contents("blocks.csv"); $rows=explode("\n",$file);
foreach($rows as $r) mysql_query("INSERT INTO geolocate
VALUES($r)");?>

se genera Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes), si chiar daca nu aparea asta, sigur aparea limitarea timpului de executie a scriptului PHP (de obicei 30 secunde). Mai tarziu a venit si ideea :
Creez tabelele :
 
CREATE TABLE `blocks` (
  `ipn1` int(11) NOT NULL,
  `ipn2` int(11) NOT NULL,
  `locid` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
 
CREATE TABLE `location` (
  `locid` int(11) NOT NULL,
  `country` varchar(5) NOT NULL,
  `region` varchar(5) NOT NULL,
  `city` varchar(150) NOT NULL,
  `postcode` varchar(20) NOT NULL,
  `lat` int(11) NOT NULL,
  `lng` int(11) NOT NULL,
  `metro` varchar(11) NOT NULL,
  `area` varchar(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
 
fac upload la fisierele CSV prin FTP,  si apoi
 
<?php 
 mysql_query("LOAD DATA INFILE 'path/blocks.csv' REPLACE INTO TABLE blocks FIELDS ENCLOSED BY '\"' TERMINATED BY ','") or die('Invalid query: ' . mysql_error());
 mysql_query("LOAD DATA INFILE 'path/location.csv' REPLACE INTO TABLE location FIELDS OPTIONALLY ENCLOSED BY '\"' TERMINATED BY ','") or die('Invalid query: ' . mysql_error());
?>