BCI Competition III

- Final Results -

[ remarks | winners | true labels | organizers ]

The announcement and the data sets of the BCI Competition III can be found here.

Results for download: all results [ pdf ] or presentation from the BCI Meeting 2005 [ pdf ]

A Kind Request

It would be very helpful for the potential organization of further BCI competitions to get some feedback, criticism and suggestions, about this competition. We are interested to hear from participants about their experiences as well as from non-participants reporting why they refrained from taking part. Feedback may concern general issues or specific data sets (e.g. pros and contras for each data set).
Write your feedback to Benjamin Blankertz <benjamin.blankertz@tu-berlin.de>

[ top ]

Remarks

The results should not be taken too seriously. Of course they do not provide an objective ranking of quality:

There is great variance of how much effort contributors put into preparing their submissions.
Given the time-pressure of an approaching deadline there are a lot of possible pitfalls in processing the test data which may lead to bad or even "random" results. [This is the voice of almost-experience talking, bb.]
When test sets (and the number of classes) are relatively small, luck plays a big role. Let's take a look at a fictitious example. Suppose in a binary problem there are 15 methods that are able to classify correctly 60% of the ideal set of all trials and that they have random output on the remaining 40%. The expected accuracy of all these methods is 80%, but on a fixed test set consisting of 100 trials, the expected difference between the best and the worst result is greater than 10% (assuming independence between methods and test trials).

Nevertheless we hope that

something can be learned from this competition by presenting methods that worked on real world problems.
people who had not much success here can do a failure analysis (we provided labels of the test sets for that purpose) and gain experience for the future.
last but not least it was fun.

Watch out for the article on the competition in IEEE Transactions on Biomedical Engineering where each winning team will present a detailed description of their algorithms.

[ top ]

Data set I [Tübingen]

The performance measure is classification accuracy.
-> Note: The expected accuracy, if classification is made by chance, is 50%. <-

#.	contributor	acc	research lab	co-contributors
1.	Qingguo Wei	91%	Tsinghua University, Beijing	Fei Meng, Yijun Wang, Shangkai Gao
2.	Paul Hammon	87%	University of California, San Diego
3.	Michal Sapinski	86%	Warsaw University, Poland
3.	Mao Dawei	86%	Zhejiang University, P.R.C.	Ke Daguan, Xie Mingqiang, Ding Jichang, Zheng Kening, Zhou Jie, Murat
3.	Alexander D'yakonov	86%	Moscow State University
3.	Liu Yang	86%	National University of Defense Technology Changsha, P.R.C.	Hu Dewen, Zhou Zongtan, Zang Guohua
7.	Florian Knoll	84%	TU Graz	Alois Schloegl, Martin Hieden, Carmen Vidaurre, Bernhard Graimann
7.	Zhou Zongtan	84%	National University of Defense Technology Changsha, P.R.C.	Liu Yang, Hu Dewen
9.	Jianzhao Qin	83%	South China University of Technology, China and Institute for Infocomm Research, Singapore	Yuanqing Li
10.	Matthias Krauledat	82%	Fraunhofer FIRST, Berlin
11.	Kiyoung Yang	81%	University of Southern California
11.	Martin Hieden	81%	TU Graz, Austria	Florian Knoll, Alois Schloegl, Carmen Vidaurre, Bernhard Graiman
13.	Archis Gore	79%	Fergusson College, Pune
13.	Elly Gysels	79%	CSEM, Neuchatel
15.	Xiaomei Pei	69%	Xi'an Jiaotong University, P.R.C.	Guangyu Bin
16.	Ehsan Arbabi	67%	Sharif University of Technology, Tehran	Mohammad Bagher Shamsollahi
17.	Florian Popescu	66%	Fraunhofer FIRST, Berlin
18.	Hyunjin Yoon	65%	University of Southern California
18.	Guido Nolte	65%	Fraunhofer FIRST
20.	Timothy Uy	60%	University of California, Irvine (?)
21.	Wit Jakuczun	59%	Warsaw University of Technology
22.	Ken Wong	58%	Stanford University
23.	Xi-Chen Sun	54%	Peking University	Jufu Feng, Lianwen Wu, Qiansheng Cheng
24.	Nanying Liang	50%	Nanyang Technological University, Singapore
25.	Bin An	48%	University of Science and Technology of China, Hefei	Yan Ning, Qiang Chen, Zhaohui Jiang, Huanqing Feng
26.	Miharu Nishino	44%	Univ. of Tokyo	Osamu Fukayama, Takashi Sato
27.	Yan Ning	22%	University of Science and Technology of China, Hefei	Bin An, Qiang Chen, Zhaohui Jiang, Huanqing Feng

1. Qingguo Wei, Tsinghua University, Beijing

with Fei Meng, Yijun Wang, Shangkai Gao
Features: Combination of Bandpower, CSSD or Waveform Mean, Fisher Discriminant Analysis
Classification: linear SVM
some details [ html | doc ]

2. Paul Hammon, University of California, San Diego

Features: ICA then combination of AR coeffients, spectral power (0-45Hz), wavelet coefficients
Classification: regularized logistic regression
some details [ txt ]

3. Michal Sapinski, Warsaw University, Poland

Features: offset and spectral power of hand selected channels
Classification: logistic regression
some details [ txt ]

3. Mao Dawei, Zhejiang University, P.R.C.

with Ke Daguan, Xie Mingqiang, Ding Jichang, Zheng Kening, Zhou Jie, Murat
Features: Standard deviation of Hilbert-Huang Transform for time-frequency windows (5Hz*0.2S), 7 channels
Classification: Mahalanobis distance between classes
some details [ html | doc ]

3. Alexander D'yakonov, Moscow State University

Features: 7 channels visualized and chosen by hand
some details [ txt ]

3. Liu Yang, National University of Defense Technology Changsha, P.R.C.

with Hu Dewen, Zhou Zongtan, Zang Guohua
Features: Bandpass (2.5Hz to 25Hz), CSP, CWT, correlation between trials.
Classification: LDA
some details [ txt ]

7. Florian Knoll, TU Graz

with Alois Schloegl, Martin Hieden, Carmen Vidaurre, Bernhard Graimann
Features: AAR coeffients and band power (10-12 Hz and 16-24 Hz) from 10 channels
Classification: LDA
some details [ txt ]

7. Zhou Zongtan, National University of Defense Technology Changsha, P.R.C.

with Liu Yang, Hu Dewen
Features: bandpass (5-30Hz), CSP, 12 channels, CWT with coefficients corr. to 4-24Hz, weighting of coefficients by t-test
Classification: LDA
some details [ txt ]

9. Jianzhao Qin, South China University of Technology, China and Institute for Infocomm Research, Singapore

with Yuanqing Li
some details [ pdf ]

10. Matthias Krauledat, Fraunhofer FIRST, Berlin

Features: bandpass (8-12Hz), Gaussian Mixture Model for channel selection and clustering.
Classification: via cluster mean
some details [ txt ]

11. Kiyoung Yang, University of Southern California

Features: correlation and covariance matrix coefficients
Classification: SVM
some details [ txt ]

11. Martin Hieden, TU Graz, Austria

with Florian Knoll, Alois Schloegl, Carmen Vidaurre, Bernhard Graiman
Features: AAR coefficents and bandpower (7-10,1 0-12, 16-24 Hz), 11 channels chosen
some details [ txt ]

13. Archis Gore, Fergusson College, Pune

Features: bandpower
Classification: ensemble of neural networks and fisher discriminant functions
some details [ txt ]

13. Elly Gysels, CSEM, Neuchatel

Features: phase, PSD, and Empirical Mode Decomposition
Classification: Robust Discriminant Analysis
some details [ txt ]

15. Xiaomei Pei, Xi'an Jiaotong University, P.R.C.

with Guangyu Bin
Features: band power of 4 channels, selected by Fisher ratio, multi-variate AR coefficients
Classification: FDA
some details [ html | doc ]

16. Ehsan Arbabi, Sharif University of Technology, Tehran

with Mohammad Bagher Shamsollahi
Features: bandpass (0.5-45Hz), statistical and parametric models, various transforming functions
Classification: Bayesian classifier and SVM
some details [ pdf ]

17. Florian Popescu, Fraunhofer FIRST, Berlin

Features: bandpass (8-40Hz), AR coefficients and 'coupled fits'
Classification: sparse model from 'finite prediction error'
some details [ txt ]

18. Hyunjin Yoon, University of Southern California

Features: channel reduction via feature subset selection, correlation coefficients of all trials
Classification: SVM
some details [ txt ]

18. Guido Nolte, Fraunhofer FIRST

Method: probabilistic classification of cross-spectrum in alpha frequency range
some details [ txt ]

20. Timothy Uy, University of California, Irvine (?)

21. Wit Jakuczun, Warsaw University of Technology

Features: discriminative biorthogonal bases
Classification: discriminative biorthogonal bases
some details [ pdf ]

22. Ken Wong, Stanford University

Features: downsampling, subset of 30 channels
Classification: linear model with Tikhonov regularization
some details [ txt ]

23. Xi-Chen Sun, Peking University

with Jufu Feng, Lianwen Wu, Qiansheng Cheng
Features: time-frequency coefficients chosen by Rayleigh Coefficients
Classification: Attribute Clustering Network via Nearest Neighbors
some details [ pdf ]

24. Nanying Liang, Nanyang Technological University, Singapore

Features: AR coefficients for overlapping windows
Classification: neural network (single layer FF) with ELM training
some details [ html | doc ]

25. Bin An, University of Science and Technology of China, Hefei

with Yan Ning, Qiang Chen, Zhaohui Jiang, Huanqing Feng
Features: 2 CSP components, AR coeffients
Classification: SVM
some details [ txt ]

26. Miharu Nishino, Univ. of Tokyo

with Osamu Fukayama, Takashi Sato
Features: cross-correlation between test and training trials
Classification: 1-nearest neighbor
some details [ txt ]

27. Yan Ning, University of Science and Technology of China, Hefei

with Bin An, Qiang Chen, Zhaohui Jiang, Huanqing Feng
Features: AR coefficients
Classification: Nearest neighbor, SVM
some details [ txt ]

[ top ]

Data set II [Albany]

The performance measure is classification accuracy. The first row shows the result where data from all 15 repetitions have been used. For the results of the second row only the first 5 repetitions have been used.
-> Note: The expected accuracy, if classification is made by chance, is 2.8%. <-

#.	contributor	acc (15)	acc (5)	research lab	co-contributors
1.	Alain Rakotomamonjy	96.5%	73.5%	PSI CNRS FRE-2645, INSA de Rouen, France	V. Guigue
2.	Li Yandong	90.5%	55.0%	Department of Automation Department of Biomedical Engineering, Tsinghua University, China	Gao Xiaorong, Ma Zhongwei, Lin Zhonglin, Lu Wenkai, Hong Bo
3.	Zhou Zongtan	90.0%	59.5%	Department of Automatic Control, National University of Defense Technology, China	Liu Yang, Hu Dewen, Zang Guohu
4.	Ulrich Hoffmann	89.5%	53.0%	Signal Processing Institute, Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland
5.	Lin Zhonglin	87.5%	57.5%	Department of Automation, Tsinghua University, China	Zhang Changshui, Gao Xiaorong, Zhou Jieyun
6.	Renzhi Lu	83.0%	54.5%	Rensselaer Polytechnic Institute, NY
7.	Gerardo Gentiletti	78.5%	46.0%	Neuroimaging Laboratory, Department of Electrical Engineering, Universidad Autonoma Metropolitana Iztapalapa, Mexico City	Oscar Suarez, Veronica Banuelos
8.	Gautam Kunapuli	75.0%	34.0%	Rensselaer Polytechnic Institute, NY
9.	Qi Hongzhi	33.5%	27.5%	Tianjin University, China	Zhang Qian, Wang Zhen, Wan Baikun
10.	Kiyoung Yang	7.5%	7.0%	University of Southern California	Cyrus Shahabi, Hyunjin Yoon

1. Alain Rakotomamonjy, PSI CNRS FRE-2645, INSA de Rouen, France

with V. Guigue
some details [ txt ]

2. Li Yandong, Department of Automation Department of Biomedical Engineering, Tsinghua University, China

with Gao Xiaorong, Ma Zhongwei, Lin Zhonglin, Lu Wenkai, Hong Bo
some details [ txt ]

3. Zhou Zongtan, Department of Automatic Control, National University of Defense Technology, China

with Liu Yang, Hu Dewen, Zang Guohu
some details [ txt ]

4. Ulrich Hoffmann, Signal Processing Institute, Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland

some details [ txt ]

5. Lin Zhonglin, Department of Automation, Tsinghua University, China

with Zhang Changshui, Gao Xiaorong, Zhou Jieyun
some details [ txt ]

6. Renzhi Lu, Rensselaer Polytechnic Institute, NY

some details [ txt ]

7. Gerardo Gentiletti, Neuroimaging Laboratory, Department of Electrical Engineering, Universidad Autonoma Metropolitana Iztapalapa, Mexico City

with Oscar Suarez, Veronica Banuelos
some details [ txt ]

8. Gautam Kunapuli, Rensselaer Polytechnic Institute, NY

some details [ txt ]

9. Qi Hongzhi, Tianjin University, China

with Zhang Qian, Wang Zhen, Wan Baikun
some details [ txt ]

10. Kiyoung Yang, University of Southern California

with Cyrus Shahabi, Hyunjin Yoon
some details [ txt ]

[ top ]

Data set IIIa [Graz]

The performance measure is kappa value. The first column shows the average across the three subjects, columns 2 to 4 show the results for the individual subjects.
-> Note: The expected kappa value, if classification is made by chance, is 0. <-

#.	contributor	kappa	K3	K6	L1	research lab	co-contributors
1.	Cuntai Guan	0.7926	0.8222	0.7556	0.8000	Neural Signal Processing Lab Institute for Infocomm Research, Singapore	Haihong Zhang, Yuanqin Li
2.	Gao Xiaorong	0.6872	0.9037	0.4333	0.7111	Tsinghua University, Beijing, China	Wu Wei, Wang Ruiping, Yang Fusheng
3.	Jeremy Hill	0.6272	0.9481	0.4111	0.5222	Max Planck Institute for Biological Cybernetics, Tuebingen and Tuebingen University	Michael Schroeder

1. Cuntai Guan, Neural Signal Processing Lab Institute for Infocomm Research, Singapore

with Haihong Zhang, Yuanqin Li
Method: Fisher ratios of channel-freqency-time bins, feature selection, designing mu- and beta passband, multiclass CSP, SVM
some details [ txt ]

2. Gao Xiaorong, Tsinghua University, Beijing, China

with Wu Wei, Wang Ruiping, Yang Fusheng
Method: surface laplacian, 8-30Hz filter, CSP (one-vs-rest), SVM+kNN+LDA, bagging
some details [ pdf ]

3. Jeremy Hill, Max Planck Institute for Biological Cybernetics, Tuebingen and Tuebingen University

with Michael Schroeder
Method: resampling 100Hz, detrending, Infomax ICA, Amplitude spectra (Welch), linear PCA, and SVM, scores are constant for each trial
some details [ txt ]

[ top ]

Data set IIIb [Graz]

The performance measure is maximal steepness (t0=3s) of mutual information [bits/s]. The first column shows the average across the three subjects, columns 2 to 4 show the results for the individual subjects.
-> Note: The expected steepness, if classification is made by chance, is 0. <-

#.	contributor	MI/t	O3	S4	X11	research lab	co-contributors
1.	S. Lemm	0.3190	0.1698	0.4382	0.3489	Fraunhofer (FIRST) IDA, Berlin Germany
2.	O. Burmeister	0.2506	0.1626	0.4174	0.1719	Forschungszentrum Karlsruhe, Germany	M. Reischl, R. Mikut
3.	Xiaomei Pei	0.1380	0.2030	0.0936	0.1173	Institute of Biomedical Engineering of Xian Jiaotong University, Xian, China	Guangyu Bin
4.	S. Parini	0.1184	0.1153	0.1218	0.1181	Politecnico di Milano, Italy	L. Piccini, L. Maggi, G. Panfili, G. Andreoni
5.	D. Coyle	0.1159	0.1039	0.1490	0.0948	University of Ulster, Northern Ireland	G. Prasad, M. McGinnity
6.	Dezhong Yao	0.1104	0.1184	0.1516	0.0612	University of Electronic Science and Technology of China (UESTC) Chengdu, China	Yu Yin, Xiang Liao
7.	K. Tavakolian	0.0474	0.0704	0.0229	0.0489	University of Northern British Columbia, Canada	S. Rezaei

1. S. Lemm, Fraunhofer (FIRST) IDA, Berlin Germany

Method: ERP and ERD (MU and BETA), propabilistic classification model, accumulative classifier
some details [ txt ]

2. O. Burmeister, Forschungszentrum Karlsruhe, Germany

with M. Reischl, R. Mikut
Method: Bandbower (BP), ratios and differences of BP; MANOVA for feature selection; SVM and linear combiner
some details [ pdf ]

3. Xiaomei Pei, Institute of Biomedical Engineering of Xian Jiaotong University, Xian, China

with Guangyu Bin
Method: FFT with Hanning window of 1s-segments; Fisher Discriminant Analysis
some details [ txt | doc ]

4. S. Parini, Politecnico di Milano, Italy

with L. Piccini, L. Maggi, G. Panfili, G. Andreoni
Method: optimization of frequency bands (using Cauer Elliptic Filter and Bandpower using LDA) and time-interval (using amplitude modulation envelope) using LDA; Classification with AR(4) parameters (Burg) and PSD from 1s-window with boosted regularized LDA;
some details [ pdf ]

5. D. Coyle, University of Ulster, Northern Ireland

with G. Prasad, M. McGinnity
Method: Preprocessing with Prediction Neural Networks (pNN); Short-Time-Fourier-Transform (STFT); Linear Discriminant Analysis;
some details [ pdf ]

6. Dezhong Yao, University of Electronic Science and Technology of China (UESTC) Chengdu, China

with Yu Yin, Xiang Liao
Method: Scales of Morlet-Wavelet optimized by statistical method; wavelet-based bandpower, Support-Vector-Machines, normalized weighted summation for accumulative classification result.
some details [ txt | doc ]

7. K. Tavakolian, University of Northern British Columbia, Canada

with S. Rezaei
Method: AAR(6) parameters from 1s-segments; Bayesian Network classifier
some details [ pdf ]

[ top ]

Data set IVa [Berlin]

The performance measure is accuracy. The first column shows the overall accuracy for all five subjects, columns 2 to 4 show the results for the individual subjects.
-> Note: The expected accuracy, if classification is made by chance, is 50\%. <-

#.	contributor	acc	aa	al	av	aw	ay	research lab	co-contributors
1.	Yijun Wang	94.17%	95.5%	100.0%	80.6%	100.0%	97.6%	Tsinghua University, Beijing	Han Yuan, Dan Zhang, Xiaorong Gao, Zhiguang Zhang, Shangkai Gao
2.	Yuanqing Li	85.12%	89.3%	98.2%	76.5%	92.4%	80.6%	Institute for Infocomm Research, Singapore	Xiaoyuan Zhu, Cuntai Guan
3.	Liu Yang	83.45%	82.1%	94.6%	70.4%	87.5%	88.1%	National University of Defense Technology, Changsha, Hunan	Zhou Zongtan, Zang Guohua, Hu Dewen
4.	Zhou Zongtan	72.62%	83.9%	100.0%	63.3%	50.9%	88.1%	National University of Defense Technology, Changsha, Hunan	Hu Dewen, Liu Yang
5.	Michael Bensch	69.17%	73.2%	96.4%	70.4%	79.9%	50.8%	University of Tuebingen and Max Planck Institute for Biological Cybernetics	Jeremy Hill
6.	Cedric Simon	68.57%	83.0%	91.1%	50.0%	87.9%	54.4%	Université Catholique de Louvain-la-Neuve (UCL-Belgium)	Lehembre Remy
7.	Elly Gysels	67.86%	69.6%	96.4%	64.3%	69.6%	61.9%	Swiss Center for Electronics and Microtechnology, Neuchatel, Switzerland
8.	Carmen Viduarre	64.05%	66.1%	92.9%	67.3%	68.3%	50.4%	Dp IEE, State University of Navarra (UPNA) / Institut for Human-Computer Interfaces/BCI Lab, TU Graz	Alois Schloegl, Martin Hieden, Florian Knoll, Rafael Cabeza
9.	Le Song	63.69%	66.1%	100.0%	63.3%	64.3%	54.4%	NICTA and School of Information Technology
10.	Ehsan Arbabi	62.74%	70.5%	94.6%	56.1%	63.8%	56.3%	Electrical Engineering Department, Sharif University of Technology, Tehran, Iran	Emad Fatemizadeh
11.	Cyrus Shahabi	59.52%	57.1%	76.8%	57.7%	64.3%	54.0%	University of Southern California	Kiyoung Yang, Hyunjin Yoon
12.	Kiyoung Yang	53.93%	52.7%	85.7%	61.2%	51.8%	43.7%	University of Southern California	Hyunjin Yoon, Cyrus Shahabi
13.	Wang Feng	51.90%	50.9%	53.6%	54.6%	56.2%	46.0%	Southeast University, Nanjing	Liu Jianmin, Gong Kainan, Wang Xiaonan, Tang Liyan
14.	Hyunjin Yoon	51.19%	50.0%	67.9%	52.6%	52.7%	45.6%	University of Southern California	Kiyoung Yang, Cyrus Shahabi

1. Yijun Wang, Tsinghua University, Beijing

with Han Yuan, Dan Zhang, Xiaorong Gao, Zhiguang Zhang, Shangkai Gao
Features: Combination of CSP, AR and Temporal Waves of the readiness potential. Adaptation by former classified test samples as extended training samples for the datasets with small amount of training data.
Classification: LDA
some details [ pdf ]

2. Yuanqing Li, Institute for Infocomm Research, Singapore

with Xiaoyuan Zhu, Cuntai Guan
Preprocessing: common average reference. Features: CSP on mu and beta band (frequency band and time bins were chosen by cross-validation).
Classification: extended EM algorithm
some details [ pdf ]

3. Liu Yang, National University of Defense Technology, Changsha, Hunan

with Zhou Zongtan, Zang Guohua, Hu Dewen
Preprocessing: CSP on bandpass filtered signals (2.5-28 Hz), Continous Wavelet Transformation. Features: Rows of the correlation matrix.
Classification: linear with penalisation of energy.

4. Zhou Zongtan, National University of Defense Technology, Changsha, Hunan

with Hu Dewen, Liu Yang
Preprocessing: CSP on bandpass filtered signals (3-28 Hz). Features: sum of continous wavelet transformation values (4-24 Hz) weighted by the two samples t-Test statistic on every time-frequency point.
Classification: LDA
some details [ txt ]

5. Michael Bensch, University of Tuebingen and Max Planck Institute for Biological Cybernetics

with Jeremy Hill
Preprocessing: Detrending. Channel selection by ROC plots on coherence values. Common electrodes for all subjects were chosen. 4 electrode groups per subject and 6 frequency bands.
Classification: nu-SVM on subject's own trials

6. Lehembre Remy, Université Catholique de Louvain-la-Neuve (UCL-Belgium)

with Cedric Simon
Features: CSP (Common Spatial Patterns) and CSSD (Common Spatial Subspace decomposition). Increasing Training set by Mahalannobis distance.
Classification: SVM
some details [ txt ]

7. Elly Gysels, Swiss Center for Electronics and Microtechnology, Neuchatel, Switzerland

Features: PLV (Phase locking Value) features with Fast Correlation Based Filter.
Classification: SVM or DA
some details [ txt ]

8. Carmen Viduarre, Dp IEE, State University of Navarra (UPNA) / Institut for Human-Computer Interfaces/BCI Lab, TU Graz

with Alois Schloegl, Martin Hieden, Florian Knoll, Rafael Cabeza
Features: concatenation of logarithmic band power estimation in frequency bands 10-12 and 16-24 Hz and AAR parameters with order 3 and update coefficient 0.0055. Monopolar/Bipolar channel selection based on leave-one-out cross-validation with LDA.
Classification: LDA based Kalman filtering
some details [ pdf ]

9. Le Song, NICTA and School of Information Technology

Method: common spatial pattern on band (10-30hz) pass-filtered EEG (without re-referencing).
some details [ txt ]

10. Ehsan Arbabi, Electrical Engineering Department, Sharif University of Technology, Tehran, Iran

with Emad Fatemizadeh
Features: several features of different types (statistical features, features related to parametric models, coefficient of different transforming functions ...)
Classification: Bayesian classifier, leave-one-out to select features and to decide whether training from all or only from one subject are used.
some details [ pdf ]

11. Cyrus Shahabi, University of Southern California

with Kiyoung Yang, Hyunjin Yoon
Preprocessing: channel selection by Corona on the row differences. Features: correlation coefficiant matrix (upper triangle), aggregated by mean or max function. Channel were selected by a linear SVM with cross-validation,
Classification: linear SVM
some details [ txt ]

12. Kiyoung Yang, University of Southern California

with Hyunjin Yoon, Cyrus Shahabi
Features: correlation coefficiant matrix or covariance matrix (upper triangle), aggregated by mean or max function. Feature extraction method (corr vs cov, and mean vs max) and channels were selected by a linear SVM with cross-validation.
Classification: linear SVM
some details [ txt ]

13. Wang Feng, Southeast University, Nanjing

with Liu Jianmin, Gong Kainan, Wang Xiaonan, Tang Liyan
Preprocessing: Channel elimination by energy and selection by correlation coefficients. Features: CSP and CSSP on several frequency band.
Classification: linear and nonlinear SVM.
some details [ txt ]

14. Hyunjin Yoon, University of Southern California

with Kiyoung Yang, Cyrus Shahabi
Preprocessing: channel selection by a feature subset selection method
Features: correlation coefficiant matrix (upper triangle)
Classification: linear SVM
some details [ pdf ]

[ top ]

Data set IVb [Berlin]

-> There was only one submission to this data set. For that reason the competition is not evaluated for this data set.<-

Data set IVc [Berlin]

The performance measure is the mean square error with respect to the target vector (-1 for class left, 0 for relax and 1 for foot).
-> Note: The result for a submission with constant output of 0 is 2/3. <-

#.	contributor	mse	research lab	co-contributors
1.	Dan Zhang	0.30	Department of Biomedical Engineering, Tsinghua University, Beijing	Yijun Wang
2.	Liu Yang	0.59	National University of Defense Technology, Changsha, Hunan	Hu Dewen, Zhou Zongtan, Zang Guohua
3.	Zhou Zongtan	0.60	National University of Defense Technology, Changsha, Hunan	Hu Dewen, Liu Yang
4.	Bo Hong	0.67	Johns Hopkins School of Medicine	Soumyadipta Acharya, Nitish V. Thakor
5.	Kiyoung Yang	0.74	Information Laboratory, University of Southern California	Hyunjin Yoon, Cyrus Shahabi
6.	Cyrus Shahabi	0.88	Information Laboratory, University of Southern California	Kiyoung Yang, Hyunjin Yoon
7.	Hyunjin Yoon	1.33	Information Laboratory, University of Southern California	Kiyoung Yang, Cyrus Shahabi

1. Dan Zhang, Department of Biomedical Engineering, Tsinghua University, Beijing

with Yijun Wang
Features. ERD by CSSD(common spatial subspace decomposition).
Classification: Fisher Discriminant Analysis.
some details [ pdf ]

2. Liu Yang, National University of Defense Technology, Changsha, Hunan

with Hu Dewen, Zhou Zongtan, Zang Guohua
Preprocessing: CSP on bandpass-filtered signals (2.5-25 Hz).
Features. Rows of the correlation matrix.
Classification: linear with penalisation of energy.
some details [ txt ]

3. Zhou Zongtan, National University of Defense Technology, Changsha, Hunan

with Hu Dewen, Liu Yang
Preprocessing: CSP on bandpass filtered signals (2.5-25 Hz).
Features. sum of continous wavelet transformation values (4-24 Hz) weighted by the two samples t-Test statistic on every time-frequency point.
Classification: LDA
some details [ txt ]

4. Bo Hong, Johns Hopkins School of Medicine

with Soumyadipta Acharya, Nitish V. Thakor
Features. ERD (alpha and beta) identification in Independent Components (IC)
Classification: Committee Neural Network
some details [ txt ]

5. Kiyoung Yang, Information Laboratory, University of Southern California

with Hyunjin Yoon, Cyrus Shahabi
Preprocessing: (row) differences used instead of raw data and channel selection by a feature subset selection
Features. correlation coefficient matrix (upper triangle) of (row) differences.
Classification: linear SVM

6. Cyrus Shahabi, Information Laboratory, University of Southern California

with Kiyoung Yang, Hyunjin Yoon
Preprocessing: (row) differences used instead of raw data and channel selection by a feature subset selection method
Features. correlation coefficient matrix (upper triangle) of (row) differences.
Classification: linear SVM

7. Hyunjin Yoon, Information Laboratory, University of Southern California

with Kiyoung Yang, Cyrus Shahabi
Preprocessing: channel selection by a feature subset selection method
Features. correlation coefficient matrix (upper triangle)
Classification: linear SVM

[ top ]

Data set V [Martigny]

Remark: The original description of this data set stated that class labels should be calculated from segments of one second of raw data resp. from one precomputed feature vector. Later the organizers realized that it might not be clear enough that this statement meant the requirement that no other (i.e. also past) samples must be used. (The reason for this requirement is that the system should have a fast response time, although this property it not neccessary for the competition's test set.) Then this point was clarified on the competition web site. Unfortunately some competitors did not read the clarification and did not understand the original formulation in this sense. Their methods included smoothing consequtive classifier output on longer time windows. Since these results are not comparable to the other, we had to take them out of the regular scoring, see the second table below.

The performance measure is classification accuracy. The first column shows the average across the three subjects, columns 2 to 4 show the results for the individual subjects. The column titled 'psd' indicates whether the submission used the precomputed psd features (y) or not (n).
-> Note: The expected accuracy, if classification is made by chance, is 33.33%. <-

#.	contributor	psd	acc	s1	s2	s3	research lab	co-contributors
1.	Ferran Galan	y	68.65	79.60	70.31	56.02	University of Barcelona	Francesc Oliva, Joan Guardia
2.	Xiang Liao	y	68.50	78.08	71.66	55.73	University of Electronic Science and Technology of China (UESTC)	Yu Yin, Dezhong Yao
3.	Walter	y	65.90	77.85	66.36	53.44	???
4.	Xiaomei Pei	y	65.67	76.03	69.36	51.61	Institute of Biomedical Engineering of Xi'an Jiaotong University	Guangyu Bin, Chongxun Zheng
5.	Irene Sturm	y	64.91	78.08	63.83	52.75	Fraunhofer FIRST (IDA), Berlin	Guido Dornhege
6.	Stephan Uray	y	64.60	81.05	73.04	39.68	TU Graz
7.	Julien Kronegg	y	64.04	76.06	64.83	51.18	University of Geneva	Douglas Rofes
8.	John Q. Gan	y	63.91	77.40	63.83	50.46	University of Essex, Colchester	Louis C.S. Tsui
9.	Shiliang Sun	n	62.83	74.31	62.32	51.99	Tsinghua University, Beijing	Changshui Zhang, Jie Pan
10.	J. Ignacio Serrano M. D. del Castillo	y	62.61	75.80	61.75	50.23	Instituto de Automatica Industrial. CSIC. Madrid
11.	Changshui Zhang	y	60.47	72.15	59.22	50.00	Tsinghua University, Beijing	Shiliang Sun, Feiping Nie
12.	Douglas Rofes	y	59.81	72.52	59.85	46.99	University of Geneva
13.	Alois Schloegl	n	52.71	69.00	57.05	32.29	TU Graz	Carmen Vidaurre
14.	Ehsan Arbabi	n	50.25	55.41	51.79	43.61	Sharif University of Technology	Mohammad Bagher Shamsollahi
15.	Remy Lehembre	y	50.23	72.60	46.31	31.65	Universite Catholique de Louvain-la-Neuve (UCL-Belgium)	Simon Cedric
16.	Georgios Lappas	y	45.72	71.78	33.81	31.39	Technological Educational Institution (TEI) of Western Macedonia, University of Hertfordshire	Andreas Albrecht
17.	Mohammad Bagher Shamsollahi	y	44.97	71.46	32.52	30.76	Sharif University of Technology	Ehsan Arbabi
18.	Ikaro Silva	y	30.68	38.98	27.45	25.55	???
19.	Ali Salehi	n	27.97	26.54	32.84	24.53	???
20.	Ikaro Silva2	y	14.24	5.82	10.54	26.38	???

1. Ferran Galan, University of Barcelona

with Francesc Oliva, Joan Guardia
some details [ pdf ]

2. Xiang Liao, University of Electronic Science and Technology of China (UESTC)

with Yu Yin, Dezhong Yao
some details [ htm | doc ]

3. Walter, ???

some details [ txt ]

4. Xiaomei Pei, Institute of Biomedical Engineering of Xi'an Jiaotong University

with Guangyu Bin, Chongxun Zheng
some details [ htm | doc ]

5. Irene Sturm, Fraunhofer FIRST (IDA), Berlin

with Guido Dornhege
some details [ txt ]

6. Stephan Uray, TU Graz

some details [ pdf ]

7. Julien Kronegg, University of Geneva

with Douglas Rofes
some details [ pdf ]

8. John Q. Gan, University of Essex, Colchester

with Louis C.S. Tsui; (re-submission without smoothing)
some details [ txt ]

9. Shiliang Sun, Tsinghua University, Beijing

with Changshui Zhang, Jie Pan
some details [ txt ]

10. J. Ignacio Serrano M. D. del Castillo, Instituto de Automatica Industrial. CSIC. Madrid

some details [ htm | doc ]

11. Changshui Zhang, Tsinghua University, Beijing

with Shiliang Sun, Feiping Nie
some details [ txt ]

12. Douglas Rofes, University of Geneva

some details [ txt ]

13. Alois Schloegl, TU Graz

with Carmen Vidaurre
some details [ txt ]

14. Ehsan Arbabi, Sharif University of Technology

with Mohammad Bagher Shamsollahi
some details [ pdf ]

15. Remy Lehembre, Universite Catholique de Louvain-la-Neuve (UCL-Belgium)

with Simon Cedric
some details [ txt ]

16. Georgios Lappas, Technological Educational Institution (TEI) of Western Macedonia, University of Hertfordshire

with Andreas Albrecht
some details [ txt ]

17. Mohammad Bagher Shamsollahi, Sharif University of Technology

with Ehsan Arbabi
some details [ pdf ]

18. Ikaro Silva, ???

some details [ txt ]

19. Ali Salehi, ???

some details [ txt | doc ]

20. Ikaro Silva2, ???

some details [ txt ]

[ top ]

Data Set V - Methods acting on Longer Time Segments

See the remark above. The first three submissions used some post-processing smoothing, the forth submission used longer time windows for classification.

#.	contributor	psd	acc	s1	s2	s3	research lab	co-contributors
1.	John Q. Gan	y	80.66	95.98	79.49	67.43	University of Essex, Colchester	Louis C.S. Tsui
2.	David R. Hardoon	y	78.98	90.18	79.03	67.66	University of Southampton	Charanpal Dhanjal, Zakria Hussain
3.	Louis C.S. Tsui	n	76.88	96.60	75.79	58.49	University of Essex, Colchester	John Q. Gan
4.	Nanying Liang	y	76.68	95.43	70.74	63.76	???

1. John Q. Gan, University of Essex, Colchester

with Louis C.S. Tsui
some details [ txt ]

2. David R. Hardoon, University of Southampton

with Charanpal Dhanjal, Zakria Hussain
some details [ pdf ]

3. Louis C.S. Tsui, University of Essex, Colchester

with John Q. Gan
some details [ txt ]

4. Nanying Liang, ???

some details [ htm | doc ]

[ top ]

Competition Winners

data set	research lab	contributor(s)
I	Tsinghua University, Beijing	Qingguo Wei, Fei Meng, Yijun Wang, Shangkai Gao
II	PSI CNRS FRE-2645, INSA de Rouen, France	Alain Rakotomamonjy, V. Guigue
IIIa	Neural Signal Processing Lab Institute for Infocomm Research, Singapore	Cuntai Guan, Haihong Zhang, Yuanqin Li
IIIb	Fraunhofer (FIRST) IDA, Berlin Germany	Steven Lemm
IVa	Tsinghua University, Beijing	Yijun Wang, Han Yuan, Dan Zhang, Xiaorong Gao, Zhiguang Zhang, Shangkai Gao
IVc	Department of Biomedical Engineering, Tsinghua University, Beijing	Dan Zhang, Yijun Wang
V	University of Barcelona	Ferran Galan, Francesc Oliva, Joan Guardia

Winners are asked to submit an article (short communication, 2 to 4 pages) on their algorithm to IEEE Transactions on Neural Systems and Rehabilitation Engineering until July 19th 2005. We will contact the winners to give them more details on this.

[ top ]

True Labels of competition test sets

Data set I: [ labels ]
Data set II: [ subject A | subject B ]
Data set IIIa: [ subject k3 | subject k6 | subject l1 ]
Data set IIIb: [ subject O3 | subject S4 | subject X11 ]
Data set IVa: [ subject aa | subject al | subject av | subject aw | subject ay ]
Data set IVb: [ labels ]
Note that the labels are delayed by 500ms with respect to the visual stimuli, see the description of data set IVb.
Data set IVc: [ labels ]
Data set V: for raw data [ subject1 | subject2 | subject3 ]
for psd features at sampling rate 2Hz [ subject1 | subject2 | subject3 ]
for psd features at sampling rate 16Hz [ subject1 | subject2 | subject3 ]

[ top ]

Organizers

Albany: Gerwin Schalk, Dean Krusienski, Jonathan R. Wolpaw

Berlin: Benjamin Blankertz, Guido Dornhege, Klaus-Robert Müller

Graz: Alois Schlögl, Bernhard Graimann, Gert Pfurtscheller

Martigny: Silvia Chiappa, José del R. Millán

Tübingen: Michael Schröder, Thilo Hinterberger, Thomas Navin Lal, Guido Widman, Niels Birbaumer

[ top ]

Contact:	Dr. Benjamin Blankertz <benjamin.blankertz@tu-berlin.de>
	Fraunhofer FIRST (IDA)
	Kekulestr. 7, D-12489 Berlin, Germany
	Tel: +49.(0)30.6392-1875