Back to the main page.
Bug 2185 - enhancement: keep time dimension in ft_sourcegrandaverage.
| Status | REOPENED |
| Reported | 2013-06-03 12:22:00 +0200 |
| Modified | 2015-02-11 15:53:26 +0100 |
| Product: | FieldTrip |
| Component: | core |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | Windows |
| Importance: | P3 enhancement |
| Assigned to: | Eelke Spaak |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: | http://bugzilla.fcdonders.nl/show_bug.cgi?id=386http://bugzilla.fcdonders.nl/show_bug.cgi?id=2517http://bugzilla.fcdonders.nl/show_bug.cgi?id=2528http://bugzilla.fcdonders.nl/show_bug.cgi?id=2197http://bugzilla.fcdonders.nl/show_bug.cgi?id=2544 |
Johanna - 2013-06-03 12:22:12 +0200
I have a source source1 = dim: [49 58 49] inside: [65596x1 double] outside: [73662x1 double] pos: [139258x3 double] time: [1 2] pow: [139258x2 double] per subject, that I would like to average over subjects but keep the time dimension (in this example, 2 time points). ft_sourcegrandaverage now sets dat = zeros(Nvoxel, Nsubject); but then I get a crash later on line 158 dat(:,i) = tmp(:); since tmp(:) includes time dimension. At a minimum, a more useful error message should be given if time is detected in the source structure, prior to reaching this line.
Robert Oostenveld - 2013-06-03 16:10:47 +0200
sourcegrandaverage is ancient. The whole function might have to be reconsidered. I do have some (related) good news: git seems to be in a decent state again. So we can probably move forward with the new source structure in a branch. As that is how I would like to address the sourcegrandaverage, i.e. make a new and clean implementation based on a more sane source structure.
Raghavan Gopalakrishnan - 2014-04-01 21:21:44 +0200
Has this been taken care of in the newer versions of fieldtrip?
Robert Oostenveld - 2014-04-01 23:15:55 +0200
(In reply to Raghavan Gopalakrishnan from comment #2) Hi Raghavan no, this has not yet been addressed due to lack of urgency and a dependency on restructuring the source data representation (which has been work-in-progress for a long time already). Do you have a clear example (i.e. with regards to input data) for which you would like to see this resolved? In which function do you want to feed the data after ft_sourcegrandaverage? It would help to have some concrete input to get this issue moving forward. E.g. it would help if you would contribute 2 (or more) source structures that you would want to append.
Robert Oostenveld - 2014-04-01 23:27:04 +0200
I made some changes to start addressing this. Most important is that I removed the old implementation from ft_sourcegrandaverage (there was an explicit old an new section in the code) and that in the "new" section I switched to using ft_selectdata_new rather than ft_selectdata_old. The difference between ft_selectdata_new and ft_selectdata_old is that new only selects, whereas old would also concatenate. The concatenation (in case of keepindividual=yes) or averaging (in case keepindividual=no) is now explicitly done in ft_sourcegrandaverage. Moving both sections of code from old to new indicates that I want this to move forward and that I am willing to take some risks in temporary breaking stuff. Note that test_bug2185 is the only test script that calls sourcegrandaverage, and that it is also not used in the tutorials. mac011> svn commit utilities/ft_selectdata_new.m Sending utilities/ft_selectdata_new.m Transmitting file data . Committed revision 9327. mac011> svn commit ft_sourcegrandaverage.m test Sending ft_sourcegrandaverage.m Adding test/test_bug2185.m Transmitting file data .. Committed revision 9328.
Raghavan Gopalakrishnan - 2014-04-01 23:29:12 +0200
(In reply to Robert Oostenveld from comment #3) Yes I would be glad to share some test data. It looks like this source_timelock_stim{1} ans = time: [1x1500 double] pos: [8196x3 double] inside: [8196x1 double] outside: [1x0 double] method: 'average' avg: [1x1 struct] cfg: [1x1 struct] The size is about 1 GB. How can I upload? After grandaverage, I would like to compute cluster based permutation statistics.
Robert Oostenveld - 2014-04-01 23:45:31 +0200
(In reply to Raghavan Gopalakrishnan from comment #5) Thanks for the additional info. Please look here for data sharing suggestions: http://fieldtrip.fcdonders.nl/faq/how_should_i_send_example_data_to_the_developers Just pick the one most convenient for you. I suggest you don't paste the link in bugzilla (as you don't want others to access the data) but email it to me directly using r.oostenveld at donders.ru.nl.
Robert Oostenveld - 2014-04-02 16:02:21 +0200
(In reply to Robert Oostenveld from comment #6) I received the data and copied it to /home/common/matlab/fieldtrip/data/test/bug2185.mat On 02 Apr 2014, Raghavan Gopalakrishnan wrote: The mat file has a cell matrix with 4 spatio temporal source reconstructed data using MNE. I would like to perform the following actions. 1. ft_sourcegrandaverage - grand avg the source (dipole) time series (within and between subjects) 2. It would also be great if the sources can be fed into the same function or different to compute the mean (or any other function) time series of an anatomical region of interest (Tzourio-Mazoyer et al 2002) 3. ft_sourcestatistics - Compute neighborhood matrix of sources and perform cluster based permutation stats.
Robert Oostenveld - 2014-04-03 18:25:51 +0200
I have been working on it, making various improvements along the way. roboos@mentat001> svn commit Sending ft_sourcegrandaverage.m Sending test/test_bug2185.m Sending utilities/ft_datatype_source.m Sending utilities/ft_selectdata_new.m Transmitting file data .... Committed revision 9341. Could you download the latest FT version (from ftp tonight, or from googlecode or github immediately) and have a look at the test_bug2185 script/function. As function it will not work for you, but you can copy-paste the relevant sections (at the bottom, where your data is loaded). I hope you can confirm that this is a step in the right direction in ft_sourcegrandaverage (with keep individual=yes/no). Oh, in case you wonder where the test script is, see https://code.google.com/p/fieldtrip/source/browse/trunk/test/test_bug2185.m The follow up of the analysis, i.e. doing something with the output of ft_sourcegrandaverage still needs more work. Your point 1 and 3 have some methodological concerns (i.e. not MATLAB related per see), but your point 2 should be almost done. The code for ROI selection on the basis of an anatomical atlas is present, so I will first continue there.
Robert Oostenveld - 2014-04-04 11:36:47 +0200
First thing to check is that ft_math can work with the newly formatted output of ft_sourceanalysis. It had problems with the mom field, which I just solved. mac011> svn commit ft_math.m test/test_ft_math.m Sending ft_math.m Sending test/test_ft_math.m Transmitting file data .. Committed revision 9351.
Robert Oostenveld - 2014-04-04 11:44:32 +0200
Hi Raghavan, I want to continue by ensuring that the ft_sourcegrandaverage output is compatible with ft_sourceplot. Am I correct that the source reconstruction is done on a cortical sheet? The number of vertices matches with fieldtrip/template/sourcemodel/cortex_8196.surf.gii but the positions don't. Can you tell me a bit more about the cortical sheet? Robert
Robert Oostenveld - 2014-04-04 11:46:48 +0200
(In reply to Robert Oostenveld from comment #10) Also the triangles of the FT template are inconsistent with your source model. Could you give me the triangle connectivity (i.e. the "tri" matrix in fieldtrip terminology)?
Robert Oostenveld - 2014-04-04 14:36:33 +0200
I improved data selection for source structures, now supports time and frequency selection and averaging, and averaging over source positions. Furthermore I extended the support for {pos}_ori_time structures. mac011> svn commit utilities/ft_selectdata_new.m test/test_bug2185.m Sending test/test_bug2185.m Sending utilities/ft_selectdata_new.m Transmitting file data .. Committed revision 9353.
Robert Oostenveld - 2014-04-04 14:39:56 +0200
Although the original data structure does not lend itself for easy plotting, the test script shows that it is possible to do some stuff with the source reconstruction and visualise it. Sofar the following functions are working on the new ft_sourcegrandaverage output representation: - ft_selectdata (new) - ft_math - ft_sourceinterpolate - ft_sourceplot The following ones still need to be checked: - ft_appendsource - ft_sourcedescriptives - ft_sourcemovie - ft_sourceparcellate - ft_sourcestatistics - ft_sourcewrite
Raghavan Gopalakrishnan - 2014-04-04 16:20:40 +0200
(In reply to Robert Oostenveld from comment #8) I did download the new version of FT 20140403 and tested ft_sourcegrandaverage. It works well, except for ft_math as you have pointed out. The new version of ft_math will be available on FT version 20140404? I will address your other comments one by one. Thanks very much.
Raghavan Gopalakrishnan - 2014-04-04 22:14:21 +0200
(In reply to Robert Oostenveld from comment #13) There seems to be one more problem. With ft_sourceinterpolate (and ft_sourcemovie and may be other functions you have listed). The funparameter looks for field 'avg.pow'. However, the output of the new ft_sourcegrandaverage does not have the 'avg.pow' field. Looks like 'avg' field is now stripped and 'pow' field is now separate.
Raghavan Gopalakrishnan - 2014-04-08 21:02:16 +0200
(In reply to Robert Oostenveld from comment #13) Any more improvements on this front?
Robert Oostenveld - 2014-04-08 22:39:23 +0200
(In reply to Raghavan Gopalakrishnan from comment #16) That the avg subfield is gone is part of the design. New source structures should be more similar to timelock and freq structures, and for example to the output of ft_sourcestatistics. That means that the avg and trial subfields have to be removed, the representation "flattened" and dimords better described. I have been working on ft_selectdata, but encountered some problems. To address them, I should implement ft_selectdata from scratch (i.e. make a 3rd attempt at the function, after the "old" and the "new" implementation that we now have). There is also the related report from Johanna that needs to be addressed with this change. I'll get to work on it tomorrow morning. best Robert
Robert Oostenveld - 2014-04-09 15:25:58 +0200
for reference, I copied the source model (with pnt and tri) to bug2185_sourcemodel.mat
Raghavan Gopalakrishnan - 2014-04-21 19:12:02 +0200
(In reply to Robert Oostenveld from comment #18) Dear Robert, I guess you have been working on this issue. Do you have any progress at this time? When do you anticipate the sourcegrandaverage and all related functions will go live? Thanks, Raghavan
Robert Oostenveld - 2014-04-22 09:46:44 +0200
(In reply to Raghavan Gopalakrishnan from comment #19) the implementation in ft_sourcegrandaverage is now complete, although there is a regression error according to the reported bug 2544. The updated code is also part of the fieldtrip release. You can confirm that it works with the following example code and check the output data structures. source = []; source.pos = rand(1320,3); source.time = 1:25; source.avg.pow = rand(10*11*12,25); source.inside = 1:660; source.outside = 661:1320; cfg = []; cfg.parameter = 'pow'; cfg.keepindividual = 'no'; grandavg1 = ft_sourcegrandaverage(cfg, source, source) cfg.keepindividual = 'yes'; grandavg2 = ft_sourcegrandaverage(cfg, source, source) Having completed this specific issue in ft_sourcegrandaverage does not mean that source activity that is expressed as function of time can be used throughout field trip. There is code further down in the analysis pipeline that may need to be further extended.
Robert Oostenveld - 2014-05-14 20:08:50 +0200
closed several of my bugs that have been resolved