Dr Terry Windeatt
Lecturer
Email: t.windeatt@surrey.ac.uk
Phone: Work: 01483 68 9286
Room no: 27 AB 05
Further information
Publications
Highlights
- .
(2011) 'Minimising added classification error using walsh coefficients'. IEEE IEEE Transactions on Neural Networks, 22 (8), pp. 1334-1339.Full text is available at: http://epubs.surrey.ac.uk/37251/
Abstract
Two-class supervised learning in the context of a classifier ensemble may be formulated as learning an incompletely specified Boolean function, and the associated Walsh coefficients can be estimated without knowledge of the unspecified patterns. Using an extended version of the Tumer-Ghosh model, the relationship between Added Classification Error and second order Walsh coefficients is established. In this paper, the ensemble is composed of Multi-layer Perceptron (MLP) base classifiers, with the number of hidden nodes and epochs systematically varied. Experiments demonstrate that the mean second order coefficients peak at the same number of training epochs as ensemble test error reaches a minimum.
- .
(2011) 'Embedded feature ranking for ensemble MLP classifiers'. IEEE IEEE Transactions on Neural Networks, 22 (6), pp. 988-994.Full text is available at: http://epubs.surrey.ac.uk/6484/
Abstract
A feature ranking scheme for multilayer perceptron (MLP) ensembles is proposed, along with a stopping criterion based upon the out-of-bootstrap estimate. To solve multi-class problems feature ranking is combined with modified error-correcting output coding. Experimental results on benchmark data demonstrate the versatility of the MLP base classifier in removing irrelevant features.
- .
(2010) 'Facial Expression Detection using Filtered Local Binary Pattern Features with ECOC Classifiers and Platt Scaling.'. Microtome Publishing Journal of Machine Learning Research, Track 11, pp. 111-118.Full text is available at: http://epubs.surrey.ac.uk/7128/
Abstract
We outline a design for a FACS-based facial expression recognition system and describe in more detail the implementation of two of its main components. Firstly we look at how features that are useful from a pattern analysis point of view can be extracted from a raw input image. We show that good results can be obtained by using the method of local binary patterns (LPB) to generate a large number of candidate features and then selecting from them using fast correlation-based ltering (FCBF). Secondly we show how Platt scaling can be used to improve the performance of an error-correcting output code (ECOC) classi er.
- . (2008) 'Ensemble MLP classifier design'. Studies in Computational Intelligence, 137, pp. 133-147.
Journal articles
- .
(2012) 'Low training strength high capacity classifiers for accurate ensembles using walsh coefficients'. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7626 LNCS, pp. 701-709.Full text is available at: http://epubs.surrey.ac.uk/738164/
Abstract
If a binary decision is taken for each classifier in an ensemble, training patterns may be represented as binary vectors. For a two-class supervised learning problem this leads to a partially specified Boolean function that may be analysed in terms of spectral coefficients. In this paper it is shown that a vote which is weighted by the coefficients enables a fast ensemble classifier that achieves performance close to Bayes rate. Experimental evidence shows that effective classifier performance may be achieved with one epoch of training of an MLP using Levenberg-Marquardt with 64 hidden nodes. © 2012 Springer-Verlag Berlin Heidelberg.
- . (2012) 'Preface'. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7626 LNCS
- .
(2011) 'Bias-variance analysis of ECOC and bagging using neural nets'. Springer Studies in Computational Intelligence, 373/2011, pp. 59-73.Full text is available at: http://epubs.surrey.ac.uk/37252/
Abstract
One of the methods used to evaluate the performance of ensemble classifiers is bias and variance analysis. In this chapter, we analyse bootstrap aggregating (bagging) and Error Correcting Output Coding (ECOC) ensembles using a biasvariance framework; and make comparisons with single classifiers, while having Neural Networks (NNs) as base classifiers. As the performance of the ensembles depends on the individual base classifiers, it is important to understand the overall trends when the parameters of the base classifiers -nodes and epochs for NNs-, are changed.We show experimentally on 5 artificial and 4 UCI MLR datasets that there are some clear trends in the analysis that should be taken into consideration while designing NN classifier systems.
- .
(2011) 'Embedded feature ranking for ensemble MLP classifiers'. IEEE IEEE Transactions on Neural Networks, 22 (6), pp. 988-994.Full text is available at: http://epubs.surrey.ac.uk/6484/
Abstract
A feature ranking scheme for multilayer perceptron (MLP) ensembles is proposed, along with a stopping criterion based upon the out-of-bootstrap estimate. To solve multi-class problems feature ranking is combined with modified error-correcting output coding. Experimental results on benchmark data demonstrate the versatility of the MLP base classifier in removing irrelevant features.
- .
(2011) 'Facial Action Unit Recognition using Filtered Local Binary Pattern Features with Bootstrapped and Weighted ECOC Classifiers'. Springer Studies in Computational Intelligence, 373/2011, pp. 1-20.Full text is available at: http://epubs.surrey.ac.uk/37255/
Abstract
Within the context face expression classification using the facial action coding system (FACS), we address the problem of detecting facial action units (AUs). The method adopted is to train a single error-correcting output code (ECOC) multiclass classifier to estimate the probabilities that each one of several commonly occurring AU groups is present in the probe image. Platt scaling is used to calibrate the ECOC outputs to probabilities and appropriate sums of these probabilities are taken to obtain a separate probability for each AU individually. Feature extraction is performed by generating a large number of local binary pattern (LBP) features and then selecting from these using fast correlation-based filtering (FCBF). The bias and variance properties of the classifier are measured and we show that both these sources of error can be reduced by enhancing ECOC through the application of bootstrapping and class-separability weighting.
- .
(2011) 'Hybrid correlation and causal feature selection for ensemble classifiers'. Springer Studies in Computational Intelligence, 373, pp. 97-115.Full text is available at: http://epubs.surrey.ac.uk/37256/
Abstract
PC and TPDA algorithms are robust and well known prototype algorithms, incorporating constraint-based approaches for causal discovery. However, both algorithms cannot scale up to deal with high dimensional data, that is more than few hundred features. This chapter presents hybrid correlation and causal feature selection for ensemble classifiers to deal with this problem. Redundant features are removed by correlation-based feature selection and then irrelevant features are eliminated by causal feature selection. The number of eliminated features, accuracy, the area under the receiver operating characteristic curve (AUC) and false negative rate (FNR) of proposed algorithms are compared with correlation-based feature selection (FCBF and CFS) and causal based feature selection algorithms (PC, TPDA, GS, IAMB).
- .
(2011) 'Minimising added classification error using walsh coefficients'. IEEE IEEE Transactions on Neural Networks, 22 (8), pp. 1334-1339.Full text is available at: http://epubs.surrey.ac.uk/37251/
Abstract
Two-class supervised learning in the context of a classifier ensemble may be formulated as learning an incompletely specified Boolean function, and the associated Walsh coefficients can be estimated without knowledge of the unspecified patterns. Using an extended version of the Tumer-Ghosh model, the relationship between Added Classification Error and second order Walsh coefficients is established. In this paper, the ensemble is composed of Multi-layer Perceptron (MLP) base classifiers, with the number of hidden nodes and epochs systematically varied. Experiments demonstrate that the mean second order coefficients peak at the same number of training epochs as ensemble test error reaches a minimum.
- .
(2010) 'Facial Expression Detection using Filtered Local Binary Pattern Features with ECOC Classifiers and Platt Scaling.'. Microtome Publishing Journal of Machine Learning Research, Track 11, pp. 111-118.Full text is available at: http://epubs.surrey.ac.uk/7128/
Abstract
We outline a design for a FACS-based facial expression recognition system and describe in more detail the implementation of two of its main components. Firstly we look at how features that are useful from a pattern analysis point of view can be extracted from a raw input image. We show that good results can be obtained by using the method of local binary patterns (LPB) to generate a large number of candidate features and then selecting from them using fast correlation-based ltering (FCBF). Secondly we show how Platt scaling can be used to improve the performance of an error-correcting output code (ECOC) classi er.
- . (2010) 'Bootstrap feature selection for ensemble classifiers'. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 6171 LNAI, pp. 28-41.
- .
(2009) 'Weighted decoding ECOC for facial action unit classification'. Springer Studies in Computational Intelligence, 245, pp. 59-77.Full text is available at: http://epubs.surrey.ac.uk/7129/
Abstract
There are two approaches to automating the task of facial expression recognition, the first concentrating on what meaning is conveyed by facial expression and the second on categorising deformation and motion into visual classes. The latter approach has the advantage that the interpretation of facial expression is decoupled from individual actions as in FACS (Facial Action Coding System). In this chapter, upper face action units (aus) are classified using an ensemble of MLP base classifiers with feature ranking based on PCA components. When posed as a multi-class problem using Error-Correcting-Output-Coding (ECOC), experimental results on Cohn-Kanade database demonstrate that error rates comparable to two-class problems (one-versus-rest) may be obtained. The ECOC coding and decoding strategies are discussed in detail, and a novel weighted decoding approach is shown to outperform conventional ECOC decoding. Furthermore, base classifiers are tuned using the ensemble Out-of-Bootstrap estimate, for which purpose, ECOC decoding is modified. The error rates obtained for six upper face aus around the eyes are believed to be among the best for this database.
- . (2008) 'Ensemble MLP classifier design'. Studies in Computational Intelligence, 137, pp. 133-147.
- .
(2006) 'Accuracy/diversity and ensemble MLP classifier design.'. IEEE Trans Neural Netw, United States: 17 (5), pp. 1194-1211.Full text is available at: http://epubs.surrey.ac.uk/37277/
Abstract
The difficulties of tuning parameters of multilayer perceptrons (MLP) classifiers are well known. In this paper, a measure is described that is capable of predicting the number of classifier training epochs for achieving optimal performance in an ensemble of MLP classifiers. The measure is computed between pairs of patterns on the training data and is based on a spectral representation of a Boolean function. This representation characterizes the mapping from classifier decisions to target label and allows accuracy and diversity to be incorporated within a single measure. Results on many benchmark problems, including the Olivetti Research Laboratory (ORL) face database demonstrate that the measure is well correlated with base-classifier test error, and may be used to predict the optimal number of training epochs. While correlation with ensemble test error is not quite as strong, it is shown in this paper that the measure may be used to predict number of epochs for optimal ensemble performance. Although the technique is only applicable to two-class problems, it is extended here to multiclass through output coding. For the output-coding technique, a random code matrix is shown to give better performance than one-per-class code, even when the base classifier is well-tuned.
Conference papers
- .
(2011) 'A comparison of random forest with ECOC-based classifiers'. Springer Lecture Notes in Computer Science: Multiple Classifier Systems, Naples, Italy: 10th International Workshop, MCS 2011 6713, pp. 207-216.Full text is available at: http://epubs.surrey.ac.uk/37250/
Abstract
We compare experimentally the performance of three approaches to ensemble-based classification on general multi-class datasets. These are the methods of random forest, error-correcting output codes (ECOC) and ECOC enhanced by the use of bootstrapping and class-separability weighting (ECOC-BW). These experiments suggest that ECOC-BW yields better generalisation performance than either random forest or unmodified ECOC. A bias-variance analysis indicates that ECOC benefits from reduced bias, when compared to random forest, and that ECOC-BW benefits additionally from reduced variance. One disadvantage of ECOC-based algorithms, however, when compared with random forest, is that they impose a greater computational demand leading to longer training times.
- . (2010) 'FLIP-ECOC: A greedy optimization of the ECOC matrix'. Springer Lecture Notes in Electrical Engineering: Computer and Information Sciences, London, UK: 25th International Symposium on Computer and Information Sciences 62 (5), pp. 149-154.
- .
(2010) 'Bias-Variance Analysis of ECOC and Bagging
Using Neural Nets'. Proceedings of the the Third Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications, European Conference on Machine Learning, Barcelona, Spain: ECML - SUEMA 2010, pp. 109-118.Full text is available at: http://epubs.surrey.ac.uk/37303/
- .
(2010) 'Hybrid Correlation and Causal Feature Selection
for Ensemble Classifiers'. Proceedings of the the Third Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications, European Conference on Machine Learning, Barcelona, Spain: ECML - SUEMA 2010, pp. 23-32.Full text is available at: http://epubs.surrey.ac.uk/37315/
Abstract
PC and TPDA algorithms are robust and well known prototype algorithms, incorporating constraint-based approaches for causal discovery. However, both algorithms cannot scale up to deal with high dimensional data, that is more than few hundred features. This paper presents hybrid correlation and causal feature selection for ensemble classifiers to deal with this problem. The number of eliminated features, accuracy, the area under the receiver operating characteristic curve (AUC) and false negative rate (FNR) of proposed algorithms are compared with correlation-based feature selection (FCBF and CFS) and causal based feature selection algorithms (PC, TPDA, GS, IAMB).
- .
(2010) 'Facial Action Unit Recognition using Filtered Local Binary Pattern Features with Bootstrapped and Weighted ECOC Classifiers'. Barcelona, Spain: Supervised and Unsupervised Ensemble Methods and their Applications, ECMLFull text is available at: http://epubs.surrey.ac.uk/7132/
Abstract
Within the context face expression classication using the facial action coding system (FACS), we address the problem of detecting facial action units (AUs). The method adopted is to train a single error-correcting output code (ECOC) multiclass classier to estimate the probabilities that each one of several commonly occurring AU groups is present in the probe image. Platt scaling is used to calibrate the ECOC outputs to probabilities and appropriate sums of these probabilities are taken to obtain a separate probability for each AU individually. Feature extraction is performed by generating a large number of local binary pattern (LBP) features and then selecting from these using fast correlation-based ltering (FCBF). The bias and variance properties of the classifier are measured and we show that both these sources of error can be reduced by enhancing ECOC through the application of bootstrapping and class-separability weighting.
- .
(2010) 'A bias-variance analysis of bootstrapped class-separabilityweighting for error-correcting output code ensembles'. Proceedings - International Conference on Pattern Recognition, , pp. 61-64.doi: 10.1109/ICPR.2010.24Full text is available at: http://epubs.surrey.ac.uk/7130/
Abstract
We investigate the effects, in terms of a bias-variance decomposition of error, of applying class-separability weighting plus bootstrapping in the construction of error-correcting output code ensembles of binary classifiers. Evidence is presented to show that bias tends to be reduced at low training strength values whilst variance tends to be reduced across the full range. The relative importance of these effects, however, varies depending on the stability of the base classifier type. © 2010 IEEE.
- .
(2010) 'Class-Separability Weighting and Bootstrapping in Error Correcting Output Code Ensembles'. SPRINGER-VERLAG BERLIN MULTIPLE CLASSIFIER SYSTEMS, PROCEEDINGS, Cairo, EGYPT: 9th International Workshop on Multiple Classifier Systems 5997, pp. 185-194.Full text is available at: http://epubs.surrey.ac.uk/7131/
- . (2010) 'Correlation-Based and Causal Feature Selection Analysis for Ensemble Classifiers'. SPRINGER-VERLAG BERLIN ARTIFICIAL NEURAL NETWORKS IN PATTERN RECOGNITION, PROCEEDINGS, Cairo, EGYPT: 4th Workshop on Artificial Neural Networks in Pattern Recognition 5998, pp. 25-36.
- . (2010) 'Facial Action Unit Recognition using Filtered Local Binary Pattern Features with Bootstrapped and Weighted ECOC Classifiers'. Bacelona, Spain: Workshop on Supervised and Unsupervised Ensemble Methods and their Applications, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)
- . (2010) 'Preface: Lecture Notes in Computer Science: Structural, Syntactic and Statistical Pattern Recognition'. Springer Lecture Notes in Computer Science: Structural, Syntactic and Statistical Pattern Recognition, Izmir, Turkey: Joint IAPR International Workshop, SSPR&SPR 2010 6218, pp. v-vi.
- . (2009) 'Relevance and Redundancy Analysis for Ensemble Classifiers'. SPRINGER-VERLAG BERLIN MACHINE LEARNING AND DATA MINING IN PATTERN RECOGNITION, Leipzig, GERMANY: 6th International Conference on Machine Learning and Data Mining in Pattern Recognition 5632, pp. 206-220.
- . (2009) 'Relevant and Redundant Feature Analysis with Ensemble Classification'. IEEE COMPUTER SOC ICAPR 2009: SEVENTH INTERNATIONAL CONFERENCE ON ADVANCES IN PATTERN RECOGNITION, PROCEEDINGS, Indian Statist Inst, Kolkata, INDIA: 7th International Conference on Advances in Pattern Recognition, pp. 247-250.
- .
(2009) 'Upper Facial Action Unit Recognition'. SPRINGER-VERLAG BERLIN ADVANCES IN BIOMETRICS, Comp Vis Lab, Alghero, ITALY: 3rd IAPR/IEEE International Conference on Advances in Biometrics 5558, pp. 239-248.Full text is available at: http://epubs.surrey.ac.uk/135505/
Abstract
This paper concentrates on the comparisons of systems that are used for the recognition of expressions generated by six upper face action units (AUs) by using Facial Action Coding System (FACS). Haar wavelet, Haar-Like and Gabor wavelet coe cients are compared, using Adaboost for feature selection. The binary classi cation results by using Support Vector Machines (SVM) for the upper face AUs have been observed to be better than the current results in the literature, for example 96.5% for AU2 and 97.6% for AU5. In multi-class classi cation case, the Error Correcting Output Coding (ECOC) has been applied. Although for a large number of classes, the results are not as accurate as the binary case, ECOC has the advantage of solving all problems simultaneously; and for large numbers of training samples and small number of classes, error rates are improved.
- .
(2009) 'The Bias Variance Trade-Off in Bootstrapped Error Correcting Output Code Ensembles'. SPRINGER-VERLAG BERLIN MULTIPLE CLASSIFIER SYSTEMS, PROCEEDINGS, Univ Iceland, Reykjavik, ICELAND: 8th International Workshop on Multiple Classifier Systems 5519, pp. 1-10.Full text is available at: http://epubs.surrey.ac.uk/7133/
Abstract
By performing experiments on publicly available multi-class datasets we examine the effect of bootstrapping on the bias/variance behaviour of error-correcting output code ensembles. We present evidence to show that the general trend is for bootstrapping to reduce variance but to slightly increase bias error. This generally leads to an improvement in the lowest attainable ensemble error, however this is not always the case and bootstrapping appears to be most useful on datasets where the non-bootstrapped ensemble classifier is prone to overfitting.
- .
(2009) 'Improved Uniformity Enforcement in Stochastic Discrimination'. SPRINGER-VERLAG BERLIN MULTIPLE CLASSIFIER SYSTEMS, PROCEEDINGS, Univ Iceland, Reykjavik, ICELAND: 8th International Workshop on Multiple Classifier Systems 5519, pp. 335-343.Full text is available at: http://epubs.surrey.ac.uk/135503/
Abstract
There are a variety of methods for inducing predictive systems from observed data. Many of these methods fall into the field of study of machine learning. Some of the most effective algorithms in this domain succeed by combining a number of distinct predictive elements to form what can be described as a type of committee. Well known examples of such algorithms are AdaBoost, bagging and random forests. Stochastic discrimination is a committee-forming algorithm that attempts to combine a large number of relatively simple predictive elements in an effort to achieve a high degree of accuracy. A key element of the success of this technique is that its coverage of the observed feature space should be uniform in nature. We introduce a new uniformity enforcement method, which on benchmark datasets, leads to greater predictive efficiency than the currently published method.
- .
(2008) 'Weighted Decoding ECOC for Facial Action Unit Classification'. Applications of Supervised and Unsupervised Ensemble Methods, Patras Greece: ECAI - Workshop Supervised and Unsupervised Ensemble MethodsFull text is available at: http://epubs.surrey.ac.uk/7134/
- . (2008) 'Feature ranking ensembles for facial action unit classification'. SPRINGER-VERLAG BERLIN ARTIFICIAL NEURAL NETWORKS IN PATTERN RECOGNITION, PROCEEDINGS, Pierre & Marid Curie Univ, Paris, FRANCE: 3rd IAPR Workshop on Artificial Neural Networks in Pattern Recognition 5064, pp. 267-279.
- .
(2008) 'Ensemble Approaches to Facial Action Unit Classification'. SPRINGER-VERLAG BERLIN PROGRESS IN PATTERN RECOGNITION, IMAGE ANALYSIS AND APPLICATIONS, PROCEEDINGS, Havana, CUBA: 13th Iberoamerican Congress on Progress in Pattern Recognition, Image Analysis and Applications 5197, pp. 551-559.Full text is available at: http://epubs.surrey.ac.uk/135501/
Abstract
Facial action unit (au) classification is an approach to face expression recognition that decouples the recognition of expression from individual actions. In this paper, upper face aus are classified using an ensemble of MLP (Multi-layer perceptron) base classifiers with feature ranking based on PCA components. This approach is compared experimentally with other popular feature-ranking methods applied to Gabor features. Experimental results on Cohn-Kanade database demonstrate that the MLP ensemble is relatively insensitive to the feature-ranking method but optimized PCA features achieve lowest error rate. When posed as a multi-class problem using Error- Correcting-Output-Coding (ECOC), error rates are comparable to two-class problems (one-versus-rest) when the number of features and base classifier are optimized.
- . (2007) 'An ensemble dependence measure'. SPRINGER-VERLAG BERLIN Artificial Neural Networks - ICANN 2007, Pt 1, Proceedings, Oporto, PORTUGAL: 17th International Conference on Artificial Neural Networks (ICANN 2007) 4668, pp. 329-338.
- . (2007) 'Stopping criteria for ensemble-based feature selection'. SPRINGER-VERLAG BERLIN Multiple Classifier Systems, Proceedings, Prague, CZECH REPUBLIC: 7th International Workshop on Multiple Classifier Systems 4472, pp. 271-281.
- . (2006) 'Parameter mining using the out-of-bootstrap generalisation error estimate for Stochastic Discrimination and Random Forests'. IEEE COMPUTER SOC 18th International Conference on Pattern Recognition, Vol 2, Proceedings, Hong Kong, PEOPLES R CHINA: 18th International Conference on Pattern Recognition (ICPR 2006), pp. 498-501.
- . (2001) 'Face identification and verification via ECOC'. SPRINGER-VERLAG BERLIN AUDIO- AND VIDEO-BASED BIOMETRIC PERSON AUTHENTICATION, PROCEEDINGS, Halmstad, SWEDEN: 3rd International Conference on Audio- and Video- Based Biometric Person Authentication 2091, pp. 1-13.
