elboriyorker
09-28-2006, 07:03 PM
Staggered import of large and very large MySQL Dumps (like phpMyAdmin 2.x Dumps) even through the web-servers with hard runtime limit and those in safe mode. The script executes only a small part of the huge dump and restarts itself. The next session starts where the last was stopped.
DOWNLOAD
BigDump ver. 0.24b (http://www.ozerov.de/pphlogger/dlcount.php?id=bigdump&url=http://www.ozerov.de/bigdump.zip) (beta) (6,8 KB ZIP-Archive)
SUPPORT
* Need help restoring or moving the large mySQL database?
* Need help converting your dump file to use BigDump?
* Need help developing new software that use BigDump?
* Need a feature that BigDump doesn't currently provide?
* Need help getting BigDump running?
* Need a bug fixed right away?
* Need someone you can ask if you run into a problem?
Support services for BigDump can provide the solution you need. For more information please contact me.
YOU WANT
To restore the very large backup of your mySQL database (or a part of it) into the new or the same mySQL database. You can't access the server shell and you can't import the dump using phpMyAdmin or any other scripts due to hard memory resp. runtime limit of the web-server.
YOU NEED
1. Bigdump-Script bigdump.php from the download above
2. Dump file(s) of your database created by phpMyAdmin I'll call it dump.sql from now on. You can also use GZip compressed dump files if you call them somehow like dump.gz. GZip support is only with PHP 4.3.0 and later
Note: Using a huge GZip compressed dump file can cause the script to exceed the PHP memory/runtime limit since the dump file has to be unpacked from the beginning everytime the session starts. If this happens use the uncompressed dump. It's your only chance.
3. Access account for your mySQL database
4. Access account for some web-server with PHP 4.1.0 or later installed. This web-server must be able to connect to the mySQL database (this ability is probably present if your web-server and the mySQL server are from the same ISP)
5. Some text editor like Notepad to edit the configuration file
6. Some FTP client to upload the files to the web-server
7. Common knowledge about files, PHP, mySQL databases, phpMyAdmin, FTP and HTTP
USAGE
1. Open bigdump.php in a text editor and adjust the database configuration
2. Drop the old tables on the target database if your dump doesn't contain "DROP TABLE" (use phpMyAdmin)
3. Create the working directory (e.g. dump) on your web-server
4. If you want to upload the dump files directly from the web-browser give the scripts writing permissions on the working directory (e.g. make chmod 777 on a Linux based system). You can upload the dump files from the browser up to the size limit set by the current PHP configuration of the web-server. Alternatively you can upload any files via FTP.
5. Upload bigdump.php and the dump files (*.sql or *.gz) via FTP to the working directory (take care of TEXT mode upload for bigdump.php and dump.sql but BINARY mode for dump.gz if uploading from MS Windows).
6. Run the bigdump.php from your browser via URL like http://www.yourdomain.com/dump/bigdump.php. Now you can select the file to be imported from the listing of your working directory.
7. BigDump will start every next import session automatically if you enable the JavaScript in your browser.
8. Relax and wait for the script to finish. Do not close the browser window!
9. IMPORTANT: Remove bigdump.php and your dump files from your server
Note 1: BigDump will fail processing large tables containing extended inserts. An extended insert contains all table entries within one SQL query. BigDump isn't able to split such SQL queries. In most cases BigDump will stop if some query includes to many lines. But if PHP complains that allowed memory size exhausted or MySQL server has gone away your dump probably also contains extended inserts. Please turn off extended inserts when exporting database from phpMyAdmin.
Note 2: Some web-servers disallow script execution in the directory with writing permissions for security reasons. If you changed the permissions on the working directory and you are getting a server error when running the script restore the permissions to their normal state for directories.
Note 3: If Timeout errors still occure you may need to adjust the $linespersession setting in bigdump.php.
Note 4: If mySQL server overrun occures you can use $delaypersession setting to let the script sleep some milliseconds or more before starting next session. This setting will only work if the JavaScript is activated.
Note 5: BigDump is currently not able to restore a single dump file with multiple databases inside (switched by the USE statement).
Note 6: If you experience problems with non-latin characters while using BigDump you have to adjust the $db_connection_char_set configuration variable in bigdump.php to match the encoding of your dump file.
VERSION HISTORY
Version 0.24b 2006-06-25
* Accurate directory detection (use SCRIPT_FILENAME if PATH_TRANSLATED not set)
Version 0.23b 2006-06-04
* print the current mySQL connection character set
* specify special mySQL connection character set in $db_connection_char_set
* suppress warning by set_time_limit in safe mode
* PHP in CGI mode bugfix (SCRIPT_FILENAME replaced by PATH_TRANSLATED)
Version 0.22b 2006-04-15
* minor bugfix for gzip mode
* per cent progress indication
* try to set time limit to unlimited
* read dump lines longer than 65536 characters
* HTML related improvements
Version 0.21b 2005-02-08
* extended inserts giveup bugfix (don't count linebreaks within text fields)
Version 0.20b 2005-01-05
* short_open_tag=off bugfix
* stop processing on extended insert
* disable upload of PHP files
* minor bugfixes
Version 0.19b 2004-05-31
* # comment in the text field bugfix
Version 0.18b 2004-03-18
* \\' in the text field bugfix
Version 0.17b 2003-12-14
* MySQL deprecated '-- ' comment bugfix
* Drop user-defined proprietary comment lines
* Improved anti-cache headers
Version 0.16b 2003-11-20
* Register globals bugfix
Version 0.15b 2003-11-02
* Dump file browser upload feature added (for dump files up to upload_max_filesize)
* Dump file mini-browser for the working directory added
Version 0.14b 2003-10-12
* GZip compressed files support added (available with PHP 4.3.0)
Version 0.13b 2003-10-09
* Query definition bugfix: don't stop query inside a text field ending by ';'
* Query definition bugfix: don't trim queries since spaces can be useful :-)
* $delaypersession setting added
Version 0.12 2003-07-11
* File seek by fseek() instead of line skipping should avoid time loss when skipping lines. Thanks to Anton Hummel for this idea
Version 0.11 2003-03-11
* Table definition bugfix: don't stop session inside an SQL query
* Queries statistics added
FAQ
Q: I get an error: "MySQL: Table 'some_tbl_name' already exists". Why?
A: Your dump file doesn't contain DROP queries. Use phpMyAdmin to drop all the tables on the target database which must be restored before you start the import.
Q: I get an error: "Fatal error: Allowed memory size of xxx bytes exhausted" or "MySQL server has gone away". Why?
A: Your dump file probably contains extended inserts. An extended insert contains all table entries within one SQL query. BigDump isn't able to split such SQL queries. Please turn off extended inserts when exporting database from phpMyAdmin.
Q: Why does BigDump fail putting strange SQL errors if I run it from the localhost on my Windows PC?
A: This is IMHO a bug in the PHP 4.3.x that is crashing BigDump on Win32 if using a dump file with DOS encoded line breaks (contact me for details). As workaround create your dump with only the UNIX line breaks or convert it into UNIX format using some text editor. Then you will be able to run BigDump also at your localhost.
Q: Can I use dump files created by other software than phpMyAdmin?
A: You can use any text dump file at your own risk. Although it's very important for BigDump to find a semicolon followed by a line break at the end of each query. It won't work else since it has no time to parse the queries to find their proper ends. Further BigDump behaviour can be unpredictable if your dump file is using double quotes to surround strings. BigDump won't work too if your dump file contains any proprietary comment lines (like in some dumps created by other tools). Although you can use the $comment setting in bigdump.php to drop non-standard comment lines by their first characters. Apart from that BigDump doesn't filter the dump file in any way.
Q: Why don't you provide any facility to input the database configuration from the browser form?
A: Input the database configuration from the form would be very insecure since the settings must be saved in the cookies or returned to your browser everytime the session ends.
You can read more info here: http://www.ozerov.de/bigdump.php
DOWNLOAD
BigDump ver. 0.24b (http://www.ozerov.de/pphlogger/dlcount.php?id=bigdump&url=http://www.ozerov.de/bigdump.zip) (beta) (6,8 KB ZIP-Archive)
SUPPORT
* Need help restoring or moving the large mySQL database?
* Need help converting your dump file to use BigDump?
* Need help developing new software that use BigDump?
* Need a feature that BigDump doesn't currently provide?
* Need help getting BigDump running?
* Need a bug fixed right away?
* Need someone you can ask if you run into a problem?
Support services for BigDump can provide the solution you need. For more information please contact me.
YOU WANT
To restore the very large backup of your mySQL database (or a part of it) into the new or the same mySQL database. You can't access the server shell and you can't import the dump using phpMyAdmin or any other scripts due to hard memory resp. runtime limit of the web-server.
YOU NEED
1. Bigdump-Script bigdump.php from the download above
2. Dump file(s) of your database created by phpMyAdmin I'll call it dump.sql from now on. You can also use GZip compressed dump files if you call them somehow like dump.gz. GZip support is only with PHP 4.3.0 and later
Note: Using a huge GZip compressed dump file can cause the script to exceed the PHP memory/runtime limit since the dump file has to be unpacked from the beginning everytime the session starts. If this happens use the uncompressed dump. It's your only chance.
3. Access account for your mySQL database
4. Access account for some web-server with PHP 4.1.0 or later installed. This web-server must be able to connect to the mySQL database (this ability is probably present if your web-server and the mySQL server are from the same ISP)
5. Some text editor like Notepad to edit the configuration file
6. Some FTP client to upload the files to the web-server
7. Common knowledge about files, PHP, mySQL databases, phpMyAdmin, FTP and HTTP
USAGE
1. Open bigdump.php in a text editor and adjust the database configuration
2. Drop the old tables on the target database if your dump doesn't contain "DROP TABLE" (use phpMyAdmin)
3. Create the working directory (e.g. dump) on your web-server
4. If you want to upload the dump files directly from the web-browser give the scripts writing permissions on the working directory (e.g. make chmod 777 on a Linux based system). You can upload the dump files from the browser up to the size limit set by the current PHP configuration of the web-server. Alternatively you can upload any files via FTP.
5. Upload bigdump.php and the dump files (*.sql or *.gz) via FTP to the working directory (take care of TEXT mode upload for bigdump.php and dump.sql but BINARY mode for dump.gz if uploading from MS Windows).
6. Run the bigdump.php from your browser via URL like http://www.yourdomain.com/dump/bigdump.php. Now you can select the file to be imported from the listing of your working directory.
7. BigDump will start every next import session automatically if you enable the JavaScript in your browser.
8. Relax and wait for the script to finish. Do not close the browser window!
9. IMPORTANT: Remove bigdump.php and your dump files from your server
Note 1: BigDump will fail processing large tables containing extended inserts. An extended insert contains all table entries within one SQL query. BigDump isn't able to split such SQL queries. In most cases BigDump will stop if some query includes to many lines. But if PHP complains that allowed memory size exhausted or MySQL server has gone away your dump probably also contains extended inserts. Please turn off extended inserts when exporting database from phpMyAdmin.
Note 2: Some web-servers disallow script execution in the directory with writing permissions for security reasons. If you changed the permissions on the working directory and you are getting a server error when running the script restore the permissions to their normal state for directories.
Note 3: If Timeout errors still occure you may need to adjust the $linespersession setting in bigdump.php.
Note 4: If mySQL server overrun occures you can use $delaypersession setting to let the script sleep some milliseconds or more before starting next session. This setting will only work if the JavaScript is activated.
Note 5: BigDump is currently not able to restore a single dump file with multiple databases inside (switched by the USE statement).
Note 6: If you experience problems with non-latin characters while using BigDump you have to adjust the $db_connection_char_set configuration variable in bigdump.php to match the encoding of your dump file.
VERSION HISTORY
Version 0.24b 2006-06-25
* Accurate directory detection (use SCRIPT_FILENAME if PATH_TRANSLATED not set)
Version 0.23b 2006-06-04
* print the current mySQL connection character set
* specify special mySQL connection character set in $db_connection_char_set
* suppress warning by set_time_limit in safe mode
* PHP in CGI mode bugfix (SCRIPT_FILENAME replaced by PATH_TRANSLATED)
Version 0.22b 2006-04-15
* minor bugfix for gzip mode
* per cent progress indication
* try to set time limit to unlimited
* read dump lines longer than 65536 characters
* HTML related improvements
Version 0.21b 2005-02-08
* extended inserts giveup bugfix (don't count linebreaks within text fields)
Version 0.20b 2005-01-05
* short_open_tag=off bugfix
* stop processing on extended insert
* disable upload of PHP files
* minor bugfixes
Version 0.19b 2004-05-31
* # comment in the text field bugfix
Version 0.18b 2004-03-18
* \\' in the text field bugfix
Version 0.17b 2003-12-14
* MySQL deprecated '-- ' comment bugfix
* Drop user-defined proprietary comment lines
* Improved anti-cache headers
Version 0.16b 2003-11-20
* Register globals bugfix
Version 0.15b 2003-11-02
* Dump file browser upload feature added (for dump files up to upload_max_filesize)
* Dump file mini-browser for the working directory added
Version 0.14b 2003-10-12
* GZip compressed files support added (available with PHP 4.3.0)
Version 0.13b 2003-10-09
* Query definition bugfix: don't stop query inside a text field ending by ';'
* Query definition bugfix: don't trim queries since spaces can be useful :-)
* $delaypersession setting added
Version 0.12 2003-07-11
* File seek by fseek() instead of line skipping should avoid time loss when skipping lines. Thanks to Anton Hummel for this idea
Version 0.11 2003-03-11
* Table definition bugfix: don't stop session inside an SQL query
* Queries statistics added
FAQ
Q: I get an error: "MySQL: Table 'some_tbl_name' already exists". Why?
A: Your dump file doesn't contain DROP queries. Use phpMyAdmin to drop all the tables on the target database which must be restored before you start the import.
Q: I get an error: "Fatal error: Allowed memory size of xxx bytes exhausted" or "MySQL server has gone away". Why?
A: Your dump file probably contains extended inserts. An extended insert contains all table entries within one SQL query. BigDump isn't able to split such SQL queries. Please turn off extended inserts when exporting database from phpMyAdmin.
Q: Why does BigDump fail putting strange SQL errors if I run it from the localhost on my Windows PC?
A: This is IMHO a bug in the PHP 4.3.x that is crashing BigDump on Win32 if using a dump file with DOS encoded line breaks (contact me for details). As workaround create your dump with only the UNIX line breaks or convert it into UNIX format using some text editor. Then you will be able to run BigDump also at your localhost.
Q: Can I use dump files created by other software than phpMyAdmin?
A: You can use any text dump file at your own risk. Although it's very important for BigDump to find a semicolon followed by a line break at the end of each query. It won't work else since it has no time to parse the queries to find their proper ends. Further BigDump behaviour can be unpredictable if your dump file is using double quotes to surround strings. BigDump won't work too if your dump file contains any proprietary comment lines (like in some dumps created by other tools). Although you can use the $comment setting in bigdump.php to drop non-standard comment lines by their first characters. Apart from that BigDump doesn't filter the dump file in any way.
Q: Why don't you provide any facility to input the database configuration from the browser form?
A: Input the database configuration from the form would be very insecure since the settings must be saved in the cookies or returned to your browser everytime the session ends.
You can read more info here: http://www.ozerov.de/bigdump.php