Import problem - FIXED! - Was a limit of the shared hosting package
5 posts by 3 authors in: Forums > CMS Builder: Plugins & Add-ons
Last Post: July 19, 2010 (RSS)
By gadefgaertgqe - July 12, 2010 - edited: July 21, 2010
Hi there,
I have been successfully using the Basic Auto add-on for around 8 or 9 months.
However over the last few days the data import we use has failed before it finishes. It gets close to the end but then stops.
I have carried out extensive testing and it seems to point to a limit in the total number of images (or something related to that) that can be imported into the database.
I'll explain my testing methods but first an overview of our setup which may help to give more clues....
Each night we have an import of used bike data in the form of a CSV file (the data) and a ZIP file (bike images) which has been uploaded to our server via FTP. We receive the data from a national magazine. We use their online system to input the bikes for print in their magazine, and to appear on their website.
The import script that we use is a modified version of the one we paid yourselves at Interactive Tools to write for us.
This modified version has worked fine for the last 8 or 9 months, until the last few days.
We recently upgraded our package with the bike magazine so that we can have more images in the adverts (from 6 to 9 images). Slowly we have been introducing these extra images with no problem until now. This change is what prompted me see if the extra images are causing the problem, as this is the only variable that has changed.
I tested my theory in a number of different ways.
1 - I split the data in half to 2 separate blocks. Each block imported fine. So I ruled out corrupted or malformed fields, and corrupted JPG files.
2 - I edited the entire data file so that each bike only had one image associated with it. This file also imported fine.
3 - I then edited the original data file which had all images listed in it, but removed the last 7 or so entries (near to where the import fails on the original data file) and this also imported fine.
4 - I then tried editing a copy of the original file this time removing one JPG reference at a time however this proved to be inconclusive as the number of JPGs being imported were different to the results in test number 3 above.
I have visually checked the data file for anything odd, including and hidden characters that may cause a problem.
At this point I am stuck and hoping that you can help me solve this problem ASAP.
Fingers crossed
Paul
I have been successfully using the Basic Auto add-on for around 8 or 9 months.
However over the last few days the data import we use has failed before it finishes. It gets close to the end but then stops.
I have carried out extensive testing and it seems to point to a limit in the total number of images (or something related to that) that can be imported into the database.
I'll explain my testing methods but first an overview of our setup which may help to give more clues....
Each night we have an import of used bike data in the form of a CSV file (the data) and a ZIP file (bike images) which has been uploaded to our server via FTP. We receive the data from a national magazine. We use their online system to input the bikes for print in their magazine, and to appear on their website.
The import script that we use is a modified version of the one we paid yourselves at Interactive Tools to write for us.
This modified version has worked fine for the last 8 or 9 months, until the last few days.
We recently upgraded our package with the bike magazine so that we can have more images in the adverts (from 6 to 9 images). Slowly we have been introducing these extra images with no problem until now. This change is what prompted me see if the extra images are causing the problem, as this is the only variable that has changed.
I tested my theory in a number of different ways.
1 - I split the data in half to 2 separate blocks. Each block imported fine. So I ruled out corrupted or malformed fields, and corrupted JPG files.
2 - I edited the entire data file so that each bike only had one image associated with it. This file also imported fine.
3 - I then edited the original data file which had all images listed in it, but removed the last 7 or so entries (near to where the import fails on the original data file) and this also imported fine.
4 - I then tried editing a copy of the original file this time removing one JPG reference at a time however this proved to be inconclusive as the number of JPGs being imported were different to the results in test number 3 above.
I have visually checked the data file for anything odd, including and hidden characters that may cause a problem.
At this point I am stuck and hoping that you can help me solve this problem ASAP.
Fingers crossed
Paul
Re: [8bit Gamer] Import problem (Is there maybe a total max limit of images that can be imported to the database?)
By Jason - July 12, 2010
Hi Paul,
That's a strange one. I can definitely look into this for you. If you could email me your CMS Builder Login Details and your FTP information to jason@interactivetools.com I can take a look.
Also, if you could tell me where I could find the original CSV and ZIP files that you were using for testing, I can see if I can replicate the problem on our internal sever.
Please only email this information, don't post it to the forum.
Hope this helps.
That's a strange one. I can definitely look into this for you. If you could email me your CMS Builder Login Details and your FTP information to jason@interactivetools.com I can take a look.
Also, if you could tell me where I could find the original CSV and ZIP files that you were using for testing, I can see if I can replicate the problem on our internal sever.
Please only email this information, don't post it to the forum.
Hope this helps.
---------------------------------------------------
Jason Sauchuk - Project Manager
interactivetools.com
Hire me! Save time by getting our experts to help with your project.
http://www.interactivetools.com/consulting/
Jason Sauchuk - Project Manager
interactivetools.com
Hire me! Save time by getting our experts to help with your project.
http://www.interactivetools.com/consulting/
Re: [Jason] Import problem (Is there maybe a total max limit of images that can be imported to the database?)
Hi Jason,
Wall of text has been emailed! [crazy]
Thanks
Paul
Wall of text has been emailed! [crazy]
Thanks
Paul
FIXED! - Update
By gadefgaertgqe - July 19, 2010 - edited: July 19, 2010
First of all thanks goes to Jason @ Interactive tools for jumping on this so quick and helping me diagnose the problem. Quality service as always!
As for the problem.....
We were hitting the max execution time of the PHP script. This issue was a limitation of our shared hosting package on 1&1, which has a execution timeout of 30 seconds by default. This value could not be changed on the shared hosting package we were on. The reason for that was so that our scripts would not affect other websites hosted on the same server, by taking up processing time and slowing it down.
Over the weekend we upgraded to our own dedicated server, with that we were able to increase the execution time and everything worked great!
For those of you who maybe having a similar problem this piece of code may help if you are not getting any errors back from the server (which is what was happening to me). Error reporting is disabled by default on 1and1, and after some research I found that creating a file called php.ini and placing in the same folder as my import script, with the following code:
[MySQL]
mysql.trace_mode = on
This then gave me the errors which pointed to the problem. The error was this:
Fatal error: Maximum execution time of 30 seconds exceeded in [the last bit here will refer to the PHP file that you are executing and the line number in that file]
Hope this helps someone :)
Paul
As for the problem.....
We were hitting the max execution time of the PHP script. This issue was a limitation of our shared hosting package on 1&1, which has a execution timeout of 30 seconds by default. This value could not be changed on the shared hosting package we were on. The reason for that was so that our scripts would not affect other websites hosted on the same server, by taking up processing time and slowing it down.
Over the weekend we upgraded to our own dedicated server, with that we were able to increase the execution time and everything worked great!
For those of you who maybe having a similar problem this piece of code may help if you are not getting any errors back from the server (which is what was happening to me). Error reporting is disabled by default on 1and1, and after some research I found that creating a file called php.ini and placing in the same folder as my import script, with the following code:
[MySQL]
mysql.trace_mode = on
This then gave me the errors which pointed to the problem. The error was this:
Fatal error: Maximum execution time of 30 seconds exceeded in [the last bit here will refer to the PHP file that you are executing and the line number in that file]
Hope this helps someone :)
Paul
Re: [8bit Gamer] FIXED! - Update
By Donna - July 19, 2010
Hi Paul,
That's great! Glad to hear you were able to track down what was going on there, thanks for sharing your solution!
That's great! Glad to hear you were able to track down what was going on there, thanks for sharing your solution!
Donna
--
support@interactivetools.com
--
support@interactivetools.com