Back to the main page.

Bug 2029 - Speed problems after the latest read_neuralynx_ncs update

Reported 2013-03-05 13:06:00 +0100
Modified 2019-08-10 12:03:26 +0200
Product: FieldTrip
Component: fileio
Version: unspecified
Hardware: All
Operating System: Linux
Importance: P3 normal
Assigned to:
Depends on:
See also:

George Dimitriadis - 2013-03-05 13:06:19 +0100

After the read_neuralynx_ncs was updated in the morning of 04/03/2013 the speed at which I am able to access my neuralynx data dropped by about a factor of 10. Nothing gives an error, it just takes about 10 times more to do anything that requires reading from neuralynx files.

Roemer van der Meij - 2013-03-05 13:28:17 +0100

(related to bug 1998)

Martin Vinck - 2013-03-05 14:39:24 +0100

taken care of this by reading in much less blocks now, should be fixed. see bug 1998

Robert Oostenveld - 2013-03-05 14:59:47 +0100

George, can you please report on whether the speed is now acceptable again?

Martin Vinck - 2013-03-06 07:17:52 +0100

I actually don't get why it was a factor 10. We read out only the timestamps per block once moving the pointer by 512 every time. So relative to reading the data that should only give a small increase in computational load but not a factor 10. What do you think RObert?

George Dimitriadis - 2013-03-06 08:10:55 +0100

Hi guys, I just started testing my code with the latest update and it looks as if it is behaving ok (pretty much like before). Martin, regarding the slow down. I do preprocessing on two data sets each having 72 channels captured at around 8KHz and I cut them at 400 trials each dataset where each trial is 0.7secs long. That process took with the original code (and it seems with the newest one) around half an hour (roughly, I never actually timed it). With the delay that same process would take more than 5 hours. Even reading the buffer in when it was doing the trialfunction was extremely slow (minute or two instead of just a few seconds).

George Dimitriadis - 2013-03-06 08:12:11 +0100

In the last sentence of the previous message I meant header not buffer

Robert Oostenveld - 2013-03-06 09:14:33 +0100

> We read out only the timestamps per block once moving the pointer by 512 every > time. Reading a few bytes every 512 samples in the file (1044 bytes) means that with a modern filesystem on a large hard disk that every (file system) logical block has to be touched. E.g. see for windows. So the head of the hard disk has to pass along every block to get a few bytes out. Fseeking is fast if you skip a significant portion of the file (i.e. the head does not have to move to each block on the hard disk platter), but skipping 1044 bytes is not significant. George's dataset is 72*8000*4=2304000 bytes per second, or 7.9 GB per hour. Reading every 1044th byte from such a bulk of data takes time. So his observation does not surprise me.

Martin Vinck - 2013-03-06 09:16:26 +0100

I think the key thing is that read_neuralynx_ncs is called for every entry of the cfg.trl Because if it would be called only once (it's doing the same thing every time) it would be a lot faster and the load would never change by more than a factor 2 (but probably less).

Robert Oostenveld - 2019-08-10 12:03:26 +0200

This closes a whole series of bugs that have been resolved (either FIXED/WONTFIX/INVALID) for quite some time. If you disagree, please file a new issue describing the issue on