Back to the main page.

Bug 1336 - StopOnError=false does not work with exceeded walltime errors

Reported 2012-02-21 17:04:00 +0100
Modified 2014-07-15 17:25:10 +0200
Product: FieldTrip
Component: qsub
Version: unspecified
Hardware: PC
Operating System: Linux
Importance: P3 major
Assigned to: Robert Oostenveld
Depends on:
See also:

Marcel Zwiers - 2012-02-21 17:04:30 +0100

qsubcellfun(@pause, {0,240,0}, 'timreq',1,'memreq',1024^2, 'StopOnError',false, 'Stack',1) submitting job marzwi_mentat304_p19124_b22_j001... qstat job id submitting job marzwi_mentat304_p19124_b22_j002... qstat job id submitting job marzwi_mentat304_p19124_b22_j003... qstat job id job marzwi_mentat304_p19124_b22_j003 returned, it required 0 seconds and 5.3 MB job marzwi_mentat304_p19124_b22_j001 returned, it required 0 seconds and 4.8 MB =>> PBS: job killed: walltime 188 exceeded limit 181 Warning: cleaning up all scheduled and running jobs, don't worry if you see warnings from "qdel" > In qsublist at 111 In qsubcellfun>cleanupfun at 429 In onCleanup>onCleanup.delete at 61 qdel: Request invalid for state of job MSG=invalid state for job - COMPLETE qdel Signal 42 ??? Error using ==> qsubget at 75 the batch queue system returned an error for job marzwi_mentat304_p19124_b22_j002, now aborting Error in ==> qsubcellfun at 321 [argout, options] = qsubget(jobid{collect}, 'output', 'cell', 'diary', diary, 'StopOnError', StopOnError);

Marcel Zwiers - 2012-02-28 21:07:27 +0100

I think that embedding the while (~all(collected)) .. end block (line 315-343 in qsubcellfun) in a try-catch statement, such as: try while (~all(collected)) .. end catch Exception if ~StopOnError rethrow Exception else % Notify the user and let qsubcellfun finish normally with the already collected output only end end would be a quick solution that would take away the main problem that all the already collected results are discarded after unexpected matlab-crashes / walltime errors. This could be an intermediate easy patch that would make qsubcellfun behave much more as expected when setting StopOnError=false (though I also see it is not the ultimate/ideal solution because it does stop -- just not with an error).

Marcel Zwiers - 2012-02-28 21:11:58 +0100

Ahum, the ~ in the if ~StopOnError statement should of course not be there :-)

Robert Oostenveld - 2012-02-28 23:11:52 +0100

StopOnError is functionality implemented in qsubget, not qsubcellfun. The error should be dealt with in qsubget in a similar fashion as the errors that are caught by the remote matlab. My idea is that the queue error (which is detected in the "master") is representded as error just like the error that is sent along by the remote matlab. The following section from line 154 onward in qsubget if StopOnError if ischar(err) error(err); else rethrow(err); end else warning('error during remote execution: %s', errmsg); end end % ~isempty(err) can then stay the same. The section to be changed is from line 70 onward, which states % the STDERR output log file should be empty, otherwise it indicates an error tmp = dir(logerr); if ~isempty(tmp) && tmp.bytes>0 % show the error that was printed on STDERR type(fullfile(curPwd,; error('the batch queue system returned an error for job %s, now aborting', jobid); end Instead of trowing the error at that location, it should result in err = ft_getopt(options, 'lasterr'); diarystring = ft_getopt(options, 'diary'); on line 108 and 109 to represent the oXXX and eXXX file contents in the err and diarystring.

Marcel Zwiers - 2014-04-16 16:33:17 +0200

I'm reviving this report as it would be a great feature to have (as discussed with Robert).

Robert Oostenveld - 2014-05-14 23:13:05 +0200

I have implemented consistent parsing of MATLAB and torque errors. This allows custom error handling in case torque jobs get killed. I tested it with this a = qsubcellfun(@myexit, {1, 0, 0}, 'memreq', 1e8, 'timreq', 300, 'StopOnError', 0) where myexit is a function that will exit MATLAB if the input is 1 and will continue if the input is 0. I also tested some other cases. I have not yet tested walltime/mem violations, but those should behave similarly as the test above. roboos@mentat001> svn commit Sending qsub/qsubcellfun.m Sending qsub/qsubget.m Sending qsub/qsublist.m Transmitting file data ... Committed revision 9531. @Marcel, please test and reopen if it does not work as expected.

Robert Oostenveld - 2014-07-15 17:25:10 +0200

Closed several bugs that were recently resolved. If it is closed in error, please reopen.