update according to 22q project

Author: DouweHorsthuis

README.md

@@ -90,7 +90,7 @@ The data is down-sampled from 512Hz to 256 Hz.
 Externals are all deleted since not everyone has externals. So we cannot use them as a reference.  
 We apply a 1Hz and 50Hz filter.
 We add channel info to all the channel. For this we use the following 3 files: standard-10-5-cap385, BioSemi160, BioSemi64. The first 2 are from BESA and have the correct layout. The 3rd is needed for the MoBI data. You can find these in the Functions and files folder (inside the src folder).  
-Lastly this script uses eeglab's clean_artifacts function deletes the bad channels and bad parts of data. Channels will get deleted by the standard noise criteria, if they are flat over 5 seconds and the function checks if channels are overly correlated with each other.After that it devides the data in 0.5 second epochs. In these it looks if there data has peaks that go over 20 standard deviantion of the channels amplitlude. If so these get deleted. Because it's possible that between 2 of these moments there is a very short amount of data, we make sure to delete all the continues data if between 2 boundries there is less then 2 seconds of data. 
+Lastly this script uses eeglab's clean_artifacts function deletes the bad channels and bad parts of data. Channels will get deleted by the standard noise criteria, if they are flat over 5 seconds and the function checks if channels are overly correlated with each other.After that it devides the data in 0.5 second epochs. In these it looks if there data has peaks that go over 35 standard deviantion of the channels amplitlude. If so these get deleted. Because it's possible that between 2 of these moments there is a very short amount of data, we make sure to delete all the continues data if between 2 boundries there is less then 2 seconds of data. We got to the criteria of 35, because 20 is the default setting, but because resting state data is different from task based data, we had to find out a way that we still had enough data left, without it being noisy. 
 
 #### C_manual_check
 This script plots all the data in EEGlab as continues data and allows you to delete channels manually. 
@@ -104,11 +104,8 @@ This script loads a file with all the original channels, deletes the externals a
 We chose to interpolate before the ICA because like this we can still use the ICA weights for all the channels, and since we are setting a PCA (amount of ICs we want the ICA funtion to create) we account for the interpolated channels not being used for this. 
 
 This script will do an average reference.  
-This is followed by an [Independent Component Analysis](https://eeglab.org/tutorials/06_RejectArtifacts/RunICA.html). We use the pca option to prevent rank-deficiencies.
-After his we delete only eye components by using [IClabel](https://github.com/sccn/ICLabel). IClabel will only delete the component if it has more than 80% eye data and less then 10% brain data. We arrived at this criteria after comparing (for a different dataset) how many components we (Ana, Douwe and Filip) would delete manually and what threshold would get the closest to that.
-After that we use [pop_rejcont](https://github.com/wojzaremba/active-delays/blob/master/external_tools/eeglab11_0_4_3b/functions/popfunc/pop_rejcont.m). This function epochs the data temporally and deletes the epochs that are noisy. We set this to a threshold of 8, because this would delete between 0-20% of the data. We save a matlab structure with how much data of each participant get's deleted. 
-
-**note** for the Aging group, we use the [pop_rejcont](https://github.com/wojzaremba/active-delays/blob/master/external_tools/eeglab11_0_4_3b/functions/popfunc/pop_rejcont.m) function also right before the ICA. This is because the data was too noisy for more than 50% of the participants to find eye components. 
+This is followed by an [Independent Component Analysis](https://eeglab.org/tutorials/06_RejectArtifacts/RunICA.html). We use the pca option to prevent rank-deficiencies, as best as we can. We set it to be all channels before interpolation - 1. We don't want more because this gives "Ghost ICs" that are duplicates of existing ones. We also exclude an extra IC because of the avg ref, hence the -1.
+After his we delete eye, muscle and channel noise components by using [IClabel](https://github.com/sccn/ICLabel). IClabel will only delete the component if it has more than 70% eye data or 80% muscle or 70% channel noise and always less then 10% brain data. Normally we only delete eye components (>0.8), and we arrived at this criteria after comparing (for a different dataset) how many components we (Ana, Douwe and Filip) would delete manually and what threshold would get the closest to that. However, since this is different data, and it's un epoched so its harder to clean, we decided to also delete extra components. 
 
 
 #### G_preprocces4

src/B_preprocess1.m

@@ -9,20 +9,21 @@
 % ------------------------------------------------
 clear variables
 eeglab
-group = {'Control' '22q' 'schiz'};%
+group = { 'Control' '22q' 'schiz'};%
 lowpass_filter_hz=50; %50hz filter
 highpass_filter_hz=1; %1hz filter
-script_location= 'C:\Users\dohorsth\Documents\GitHub\resting-state-analysis-pipeline-microstates-frequency\src\';
+script_location= 'D:\restingstate\scripts\';
+
 for g=1:length(group)
     if strcmp(group{g},'22q')
-        subject_list = {'1101' '1164' '1808' '1852' '1855' '11014' '11094' '11151' '11170' '11275' '11349' '11516' '11558' '11583' '11647' '11729' '11735' '11768' '11783' '11820' '11912' '1106' '1108' '1132' '1134' '1154' '1160' '1173' '1174' '1179' '1190' '1838' '1839' '1874' '11013' '11056' '11098' '11106' '11198' '11244' '11293' '11325' '11354' '11369' '11375' '11515' '11560' '11580' '11667' '11721' '11723' '11750' '11852' '11896' '11898' '11913' '11927' '11958' '11965'}; %all the IDs for the indivual particpants
-        home_path  = 'D:\Data\';
+        subject_list = {'2201' '2202' '2204' '2207' '2212' '2216' '2222' '2229' '2231' '2243' '2256' '2257' '2260' '2261' '2267' '2270' '2274' '2281' '2284' '2286' '2292' '2295'};
+        home_path  = 'D:\restingstate\data\';
     elseif strcmp(group{g},'schiz')
-        subject_list = {'12022' '12023' '12031' '12081' '12094' '12188' '12255' '12335' '12339' '12362' '12364' '12372' '12376' '12390' '12398' '12407' '12408' '12451' '12454' '12457' '12458' '12459' '12468' '12478' '12498' '12510' '12517' '12532' '12564' '12631' '12633' '12634' '12636' '12665' '12670' '12696' '12719' '12724' '12751' '12763' '12769' '12776' '12790' '12806' '12814' '12823' '12830' '12847' '12851' '12855' '12856' '12857' '12859' '12871' '12872' '12892'};
-        home_path  = 'D:\Data\';
+        subject_list = {'7003' '7007' '7019' '7025' '7046' '7049' '7051' '7054' '7058' '7059' '7061' '7064' '7065' '7073' '7075' '7078' '7089' '7092' '7094' '7123' '7556' '7808'};
+        home_path  = 'D:\restingstate\data\';
     elseif strcmp(group{g},'Control')
-        subject_list = {'10281'};%'10158' '10165' '10384' '10407' '10451' '10467' '10501' '10534' '10615' '10620' '10639' '10844' '10956' '10033' '10130' '10131' '10257' '10281' '10293' '10360' '10369' '10394' '10438' '10446' '10463' '10476' '10526' '10545' '10561' '10562' '10581' '10585' '10616' '10748' '10780' '10784' '10822' '10858' '10906' '10915' '10929' '10935' '12005' '12007' '12010' '12215' '12328' '12360' '12413' '12512' '12648' '12651' '12707' '12727' '12739' '12750' '12815' '12898' '12899'};% ------------------------------------------------
-        home_path  = 'C:\Users\dohorsth\Desktop\practice\';%'C:\Users\dohorsth\Desktop\Testing restingstate\Remaining_controls\';
+        subject_list = {'10293' '10561' '10562' '10581' '10616' '10748' '10822' '10858' '10935' '12004' '12010' '12139' '12177' '12188' '12197' '12203' '12206' '12215' '12272' '12413' '12415' '12449' '12482' '12512' '12588' '12632' '12648' '12651' '12707' '12727' '12739' '12746' '12750' '12755' '12770' '12815' '12852' '12870'};
+        home_path  = 'D:\restingstate\data\';
     end
     deleted_channels=zeros(length(subject_list),2);
     deleted_data=zeros(length(subject_list),2);
@@ -77,16 +78,30 @@
         % components in ICA or if these are pre-deleted now (which they shouldn't)
         %EEG = pop_clean_rawdata(EEG,
         %'FlatlineCriterion',5,'ChannelCriterion',0.8,'LineNoiseCriterion',4,'Highpass','off','BurstCriterion','off','WindowCriterion','off','BurstRejection','off','Distance','Euclidian');%doesn't delete bad periods
-        EEG = pop_clean_rawdata(EEG, 'FlatlineCriterion',5,'ChannelCriterion',0.8,'LineNoiseCriterion',4,'Highpass','off','BurstCriterion',20,'WindowCriterion','off','BurstRejection','on','Distance','Euclidian'); % deletes bad chns and bad periods
+        %first at 'BurstCriterion',20, this caused too much data to be
+        %deleted, second time at 'BurstCriterion',50, this caused too few data to be
+        EEG = pop_clean_rawdata(EEG, 'FlatlineCriterion',5,'ChannelCriterion',0.8,'LineNoiseCriterion',4,'Highpass','off','BurstCriterion',35,'WindowCriterion','off','BurstRejection','on','Distance','Euclidian'); % deletes bad chns and bad periods
         EEG.deleteddata_wboundries=100-EEG.pnts/old_samples*100;
         new_n_chan = EEG.nbchan;
         deleted_sample=EEG.pnts;
-        for i = length(EEG.event)-1:-1:1
-            if strcmp(EEG.event(i).type, 'boundary') && strcmp(EEG.event(i+1).type, 'boundary') && EEG.event(i+1).latency/EEG.srate-EEG.event(i).latency/EEG.srate < 2 %following event is also a boundary and less then 2 seconds of "good" data between them
-               disp(i)
-               EEG = pop_select( EEG, 'notime',[EEG.event(i).latency/EEG.srate EEG.event(i+1).latency/EEG.srate] );
+        if ~isempty(EEG.event) %at least 1 participant with no events
+            %adding one boundary at the end to stop issues, will delete later
+            for i=1:length(EEG.event)
+                EEG.event(i).time=EEG.event(i).latency/EEG.srate
             end
+            EEG.event(length(EEG.event)+1)=EEG.event(length(EEG.event)); EEG.event(length(EEG.event)).type='temp';% EEG.event(length(EEG.event)).latency=EEG.event(length(EEG.event)).latency+100;EEG.event(length(EEG.event)).duration=EEG.event(length(EEG.event)).duration+100;
+            for i = length(EEG.event)-1:-1:1%12139 caused issue
+                if strcmp(EEG.event(i).type, 'boundary') && strcmp(EEG.event(i+1).type, 'boundary') && EEG.event(i+1).latency/EEG.srate-EEG.event(i).latency/EEG.srate < 2 %following event is also a boundary and less then 2 seconds of "good" data between them
+                    disp(i)
+                    EEG = pop_select( EEG, 'notime',[EEG.event(i).latency/EEG.srate EEG.event(i+1).latency/EEG.srate] );
+                    if strcmp(EEG.event(length(EEG.event)).type, 'boundary')
+                        EEG.event(length(EEG.event)+1)=EEG.event(length(EEG.event)); EEG.event(length(EEG.event)).type='temp';
+                    end
+                end
+            end
+            EEG.event(length(EEG.event)) = [];
         end
+        % deleting the event we added before
         new_samples=EEG.pnts;
         EEG.deleteddata=100-EEG.pnts/old_samples*100;
         deleted_channels(s,:) = [string(subject_list{s}), old_n_chan-new_n_chan] ;

src/C_manual_check.m

@@ -4,20 +4,10 @@
 % Deleting remaining bad channels by eye + seeing if there is something
 % of note going on with the raw data of each person
 clear variables
-%% ASD
-%subject_list = {'1101' '1164' '1808' '1852' '1855' '11014' '11094' '11151' '11170' '11275' '11349' '11516' '11558' '11583' '11647' '11729' '11735' '11768' '11783' '11820' '11912' '1106' '1108' '1132' '1134' '1154' '1160' '1173' '1174' '1179' '1190' '1838' '1839' '1874' '11013' '11056' '11098' '11106' '11198' '11244' '11293' '11325' '11354' '11369' '11375' '11515' '11560' '11580' '11667' '11721' '11723' '11750' '11852' '11896' '11898' '11913' '11927' '11958' '11965'}; %all the IDs for the indivual particpants
-%home_path  = 'C:\Users\dohorsth\Desktop\Testing restingstate\ASD\';
-%% aged matched controls
-%subject_list = {'10033' '10130' '10131' '10158' '10165' '10257' '10281' '10293' '10360' '10369' '10384' '10394' '10407'  '10438' '10446' '10451' '10463' '10467' '10476' '10501' '10526' '10534' '10545' '10561' '10562' '10581' '10585' '10616' '10615' '10620' '10639' '10748' '10780' '10784' '10822' '10858' '10906' '10915' '10929' '10935'  '10844' '10956'  '12005' '12007' '12010' '12215' '12328' '12360' '12413' '12512' '12648' '12651' '12707' '12727' '12739' '12750' '12815' '12898' '12899'};% ------------------------------------------------
-%home_path  = 'C:\Users\dohorsth\Desktop\Testing restingstate\Control\';
-%% extra controls
-subject_list = {'10297' '10331' '10385' '10399' '10497' '10553' '10590' '10640' '10867' '10906' '12002' '12004' '12006' '12122' '12139' '12177' '12188' '12197' '12203' '12206' '12230' '12272' '12415' '12449' '12474' '12482' '12516' '12534' '12549' '12588' '12632' '12735' '12746' '12755' '12770' '12852' '12870'};
-home_path  = 'C:\Users\dohorsth\Desktop\Testing restingstate\Remaining_controls\';
-%% MoBI
-% subject_list = {'12022' '12023' '12031' '12081' '12094' '12188' '12255' '12335' '12339' '12362' '12364' '12372' '12376' '12390' '12398' '12407' '12408' '12451' '12454' '12457' '12458' '12459' '12468' '12478' '12498' '12510' '12517' '12532' '12564' '12631' '12633' '12634' '12636' '12665' '12670' '12696' '12719' '12724' '12751' '12763' '12769' '12776' '12790' '12806' '12814' '12823' '12830' '12847' '12851' '12855' '12856' '12857' '12859' '12871' '12872' '12892'};
-% % controls and ASD subjects that need to be re-checked because of 160 channel info
-% %'10131' '10257' '10369' '10438' '10545' '10585' '12360' '12898' '1808' '1852' '1855' '11345' '1106' '1134' '1154' '1160' '1174' '1179' '1190' '1838' '11106' '11375' '11913'
-% home_path  = 'C:\Users\dohorsth\Desktop\Testing restingstate\Aging\';
+%subject_list = {'2201' '2202' '2204' '2207' '2212' '2216' '2222' '2229' '2231' '2243' '2256' '2257' '2260' '2261' '2267' '2270' '2274' '2281' '2284' '2286' '2292' '2295'};
+%subject_list = {'7003' '7007' '7019' '7025' '7046' '7049' '7051' '7054' '7058' '7059' '7061' '7064' '7065' '7073' '7075' '7078' '7089' '7092' '7094' '7123' '7556' '7808'};
+subject_list = {'10293' '10561' '10562' '10581' '10616' '10748' '10822' '10858' '10935' '12004' '12010' '12139' '12177' '12188' '12197' '12203' '12206' '12215' '12272' '12413' '12415' '12449' '12482' '12512' '12588' '12632' '12648' '12651' '12707' '12727' '12739' '12746' '12750' '12755' '12770' '12815' '12852' '12870'};
+home_path  = 'D:\restingstate\data\';
 for s=1:length(subject_list)
     clear bad_chan;
     fprintf('\n******\nProcessing subject %s\n******\n\n', subject_list{s});
@@ -33,3 +23,8 @@
     close all
 
 end
+%2204 a lot of alpha eyes closed
+%7064 muscle artifact, check ICA 
+%10561 noisy
+%12449 don't use only 16 chn left
+%12707 check n channels (seems few left)