Possible processing bug

Kosh · Post by **Kosh** » Sat Jun 27, 2009 7:01 pm

Just in the past few days I have started noticing occasional Processing Errors for my uploaded submissions. Most of my submissions are being processed successfully, and since I haven't been doing any pruning, etc., each submission will just be slightly larger / different from the previous one.

Here's a chunk of my submissions, showing a couple of errors:

Code: Select all

Jun 27, 09 14&#58;38&#58;56  	0  	0  	N
Jun 27, 09 02&#58;04&#58;37 	0 	0 	E
Jun 27, 09 01&#58;44&#58;42 	35 	507 	Y
Jun 26, 09 23&#58;56&#58;40 	14 	110 	Y
Jun 26, 09 23&#58;11&#58;00 	21 	268 	Y
Jun 26, 09 22&#58;49&#58;38 	24 	401 	Y
Jun 26, 09 21&#58;14&#58;04 	62 	811 	Y
Jun 26, 09 20&#58;12&#58;22 	0 	0 	E

Because a later submission is processing correctly, the site is eventually going to get everything, but I will (potentially) lose out on being "first" to see some of them.

I scrolled back through my upload output, and there were no differences between successful uploads that eventually processed correctly and ones that eventually processed as "Error". Here's the output from the last two (currently) processed submissions:

Code: Select all

&#91;Odin&#58;/Documents/download&#93; kosh% ./censusup.sh
censusup.sh&#58; Checking for updated CensusPlus.lua...
  lua file newer -- recompressing...
  Starting upload...
&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;
  % Total    % Received % Xferd  Average Speed          Time             Curr.
                                 Dload  Upload Total    Current  Left    Speed
100 1575k  100   569  100 1574k     20  57186  0&#58;00&#58;28  0&#58;00&#58;28  0&#58;00&#58;00 28871
Checking your file
Found&#58; CensusPlus Database; CensusPlus Info Data; Version&#58; 4.2.2; Locale&#58; US
Attempting to make entry
Username &#58; Kosh&#123;=
Kosh will be credited with this submission!

curl&#58; transferred 1574.68kb in 28.197 seconds &#40;55.8458 kb/s&#41;
&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;
  upload completed at 01&#58;44&#58;42

&#91;Odin&#58;/Documents/download&#93; kosh% ./censusup.sh
censusup.sh&#58; Checking for updated CensusPlus.lua...
  lua file newer -- recompressing...
  Starting upload...
&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;&#125;
  % Total    % Received % Xferd  Average Speed          Time             Curr.
                                 Dload  Upload Total    Current  Left    Speed
100 1576k  100   569  100 1575k     22  63094  0&#58;00&#58;25  0&#58;00&#58;25  0&#58;00&#58;00 32843
Checking your file
Found&#58; CensusPlus Database; CensusPlus Info Data; Version&#58; 4.2.2; Locale&#58; US
Attempting to make entry
Username &#58; Kosh&#123;=
Kosh will be credited with this submission!

curl&#58; transferred 1575.46kb in 25.569 seconds &#40;61.6161 kb/s&#41;
&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;&#123;
  upload completed at 02&#58;04&#58;38

Do you show any problems with my "Error" submissions on your side, or is it just an occasional hiccup?

[edit] Hmm, that's odd. For some reason the forum software is mangling any text that happens to be between a left angle bracket and a right angle bracket, even in a "code" block. Fixed by substituting braces.

Post by **Rollie** » Tue Jul 14, 2009 11:32 am

The likely culprit is that the file was just too big and timed out during processing.

Kosh · Post by **Kosh** » Wed Jul 15, 2009 12:02 am

OK, that seems reasonable, except... later submissions are even larger than the errored one, and they process correctly,

Is your "stop if too large" check on clock time or processing time (it must not be file size)? I wonder if other (periodic?) processes on the server are eating a lot of the CPU occasionally.

Anyway, it isn't happening often enough for me to discern any obvious pattern, so I will just keep watching. There happened to be a couple more early on Monday, but no other errors for quite a while before that.

Post by **Rollie** » Tue Jul 21, 2009 9:40 am

Yes, that is probably the likely culprit as well. It's not that the file is so large per say, but that the process was using too much time to process that file. Depending on other factors on the server, backups, updates, etc, then smaller files might trip the timeout as well.