Berlin Brain Connectivity Benchmark (BBCB)

Berlin Brain Connectivity Benchmark and Data Analysis Challenge

The deadline for BIOMAG 2016 Challenge submissions has been extended once again: 20th September

Also, please note that there has been a slight mistake in the evaluation script evaluate_performance.m. The mistake affects the rare case in which only one ROI is estimated, and leads to a wrong measurement of the localization performance when the estimated ROI matches one of the true ones. An corrected version of the script is provided here: evaluate_performance.m. The code package has also been updated to include the corrected version. The corrected script will be used for evaluating the challenge contributions.

The Berlin Brain Connectivity Benchmark (BBCB) is a Matlab based simulation framework for generating electro- and magnetoencephalographic (EEG and MEG) data within a realistic electromagnetic volume conductor (head) model for the purpose of benchmarking EEG/MEG based brain connectivity estimation methodologies. Its development is motivated by our feeling that, while brain connectivity analyses using interaction measures such as Granger Causality are frequently applied to EEG/MEG data, their validity to detect brain interactions rather than more trivial properties of the data is rarely validated in suitable simulations. Problems such as spurious connectivity affecting certain interaction measures are overlooked when essential features of real EEG/MEG data such as mixed brain signals or correlated noise are not reflected in a simulation (see e.g. [5-7]). Their impact might therefore not be entirely clear to all researchers employing such measures. The goal of BCCB framework is therefore to facilitate the assessment of entire source connectivity estimation pipelines, which may be composed of separate pre-processing, inverse source reconstruction and connectivity estimation steps, as a whole, as opposed to benchmarking each analysis step separately under conditions that may be too simplistic.

The BBCB includes head models for both EEG and MEG that allow one to create physically realistic sensor data in an average human head anatomy. It also includes the necessary Matlab scripts for generating time series on the brain source level, mapping them to EEG/MEG sensors, adding noise etc. .

The simulation framework is described in the following paper: Haufe and Ewald, 2016.

The paper also proposes a procedure for benchmarking EEG based brain connectivity estimation. We describe a model for generating EEG data with a well-defined connectivity structure (two alpha band sources that are either linearly and uni-directionally interacting or entirely independent). The task is then to answer the following three questions:

1. Localization: In which two regions (brain octants) are the two alpha sources located?
2. Interaction: Does the data contain interaction (between the two sources) at all?
3. Directionality: If so, which estimated source octant contains the sending and which one the receiving source?

We provide Matlab code for generating data according to the proposed interaction model, and to evaluate the correctness of answers to questions 1-3 for a given dataset. We also provide a script for running an example pipeline involving LCMV beamforming for source reconstruction and the phase-slope index for connectivity estimation. The package also includes many helper functions, most notably routines for plotting scalp topographies and cortical sources distributions.

The package including code and EEG/MEG head models can be downloaded here: BBCB_code.tar.gz (~1.1 GB).

A documentation of the code and data structures used is provided here: BBCB_supplement.pdf.

BIOMAG 2016 Data Analysis Challenge

For BIOMAG 2016 we are posing the benchmark described above as a public data analysis challenge. To this end, we have simulated 100 datasets according to the procedure outlined above (see the paper draft for details) and using the code provided above.

The data

The 100 simulated EEG datasets (108 channel) to be analyzed within the challenge can be downloaded here:

BBCB_data_EEG1.tar.gz (~1.5 GB).
BBCB_data_EEG2.tar.gz (~1.5 GB).

Corresponding MEG data (298 channel CTF system, same brain sources and brain noise, but different sensor noise of the same SNR) are here:

BBCB_data_MEG1.tar.gz (~1.7 GB).
BBCB_data_MEG2.tar.gz (~1.7 GB).
BBCB_data_MEG3.tar.gz (~1.7 GB).
BBCB_data_MEG4.tar.gz (~1.7 GB).
BBCB_data_MEG5.tar.gz (~1.7 GB).

Data and code archives should be extracted into the same folder. This can be done by entering the following commands into a Unix shell, where BBCB is an existing target folder.

tar -xzf BBCB_code.tar.gz -C BBCB
tar -xzf BBCB_EEG1.tar.gz -C BBCB
tar -xzf BBCB_EEG2.tar.gz -C BBCB
tar -xzf BBCB_MEG1.tar.gz -C BBCB
tar -xzf BBCB_MEG2.tar.gz -C BBCB
tar -xzf BBCB_MEG3.tar.gz -C BBCB
tar -xzf BBCB_MEG4.tar.gz -C BBCB
tar -xzf BBCB_MEG5.tar.gz -C BBCB

Remark: MEG head model: While the EEG head model is described in the paper [1] and in detail in [2], this is not the case for the MEG model. The MEG model was computed in Brainstorm [4] using the same underlying anatomy as for EEG (the ICBM152 v2009 template described in [3]). We used the three-shell boundary element (BEM) method to compute the MEG lead field for 298 MEG channels of a CTF system, for which channel positions are provided by Brainstorm. All geometrical information such as channel positions are moreover part of the software package.

The task

Participants of the challenge are required to download and analyze the data, and to provide their estimates regarding questions 1-3 for each of the 100 datasets.

Remarks:

Participants may use the example pipelines implemented in the scripts EEG_estimate_lcmv_imcoh_psi.m and MEG_estimate_lcmv_imcoh_psi.m that are provided as part of the software package as templates for building their own pipelines.
These scripts use the same EEG/MEG head models that were employed to generate the simulated data. Alternatively, it is also possible to use other toolboxes for forward and inverse modeling, as we generated the data in a standardized head anatomy (ICBM152) using standard sensor locations. The used ICBM152 anatomical template is shipped with major analysis packages such as Brainstorm and Fieldtrip, and can moreover be obtained from the authors' website. EEG electrodes are placed according to the 10-5 system (see [8]). MEG sensor locations as well as all other anatomical information including brain octant boundaries are provided in MNI space, which is the native space of the ICBM anatomical template.

Evaluation criteria

Results will be evaluated against the ground truth by the organizers in each of the categories localization, (presence of) interaction and directionality, where generally correct answers lead to positive scores and incorrect answers lead to negative scores. Participants can also refuse to answer a particular question for a particular dataset, which will lead to zero points. The details of the scoring system are outlined in the paper draft.

Submission

Participants are expected to provide their results as a Matlab cell array all_est, where each entry contains an est structure as described in the supplementary document. The structure should be saved in a Matlab .mat file and be emailed to stefan.haufe@tu-berlin.de using the subject [BIOMAG16]. Participants need to mention their names (at least one for group submissions) and the categories they want to compete in. There are separate categories for localization, interaction and directionality, as well as for results obtained on EEG, MEG and combined M/EEG data. Thus, there are nine different categories.

Dissemination and prizes

The final leaderboard for each category will be announced end of September here. Category leaders are also eligible for small prizes as well as for presentation of their results at the dedicated BIOMAG symposium, if they provide

executable self-contained Matlab code transforming the provided EEG and/or MEG data into a Matlab estimates array all_est as described above. (We apologize for the inconvenience this rule may cause to participants using other programming languages than Matlab, and are happy to discuss alternatives.)
a slide summarizing their approach.

Schedule

The schedule outline for the challenge is as follows

Submission deadline: 20th September

Evaluation of results: 25th September

Public dissemination of results: 1st October at BIOMAG

Mailing list

Questions regarding the challenge can be sent to the mailing list BIOMAG16@ml.tu-berlin.de. Participants are invited to subscribe here. A list of frequently asked questions will moreover be compiled here.

References

[1] Haufe S and Ewald A, 2016, A simulation framework for benchmarking EEG-based brain connectivity estimation methodologies. Brain Topography. In Press.
[2] Huang Y, Parra LC and Haufe S, 2016, The New York Head-A precise standardized volume conductor model for EEG source localization and tES targeting. Neuroimage, in press. NY Head Website
[3] Fonov V, Evans AC, Botteron K, Almli CR, McKinstry RC, Collins DL, 2011, Unbiased average age-appropriate atlases for pediatric studies. Neuroimage, 54 (1), 313-327.
[4] Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM, 2011. Brainstorm: A user-friendly application for MEG/EEG analysis. Computational Intelligence and Neuroscience.
[5] Nolte G, Bai O, Wheaton L, Mari Z, Vorbach S, Hallett M, 2004. Identifying true brain interaction from EEG data using the imaginary part of coherency. Clinical Neurophysiology 115 (10), 2292-2307.
[6] Haufe S, Nikulin VV, Muller KR, Nolte G, 2013. A critical assessment of connectivity measures for EEG data: a simulation study. Neuroimage, 64:120-133.
[7] Winkler I, Panknin D, Bartz D, Muller KR, Haufe S, 2016. Validity of time reversal for testing Granger causality. IEEE TSP, 64(11):2746-2760.
[8] Oostenveld R, Praamstra P, 2016. The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology, 112(4):713–719.

Researchers wishing to use data or code from this website in a publication are asked to cite [1] and [2]. Other papers should be cited as appropriate.

back