Zor C, Windeatt T, Kittler J (2013) ECOC pruning using accuracy, diversity and hamming distance information, 2013 21st Signal Processing and Communications Applications Conference, SIU 2013
Existing ensemble pruning algorithms in the literature have mainly been defined for unweighted or weighted voting ensembles, whose extensions to the Error Correcting Output Coding (ECOC) framework is not successful. This paper presents a novel pruning algorithm to be used in the pruning of ECOC, via using a new accuracy measure together with diversity and Hamming distance information. The results show that the novel method outperforms those existing in the state-of-the-Art. © 2013 IEEE.
One of the methods used to evaluate the performance of ensemble classifiers is bias and variance analysis. In this chapter, we analyse bootstrap aggregating (Bagging) and Error Correcting Output Coding (ECOC) ensembles using a biasvariance framework; and make comparisons with single classifiers, while having Neural Networks (NNs) as base classifiers. As the performance of the ensembles depends on the individual base classifiers, it is important to understand the overall trends when the parameters of the base classifiers - nodes and epochs for NNs -, are changed. We show experimentally on 5 artificial and 4 UCI MLR datasets that there are some clear trends in the analysis that should be taken into consideration while designing NN classifier systems. © 2011 Springer-Verlag Berlin Heidelberg.
Windeatt T, Zor C (2011) Minimising added classification error using Walsh coefficients., IEEE Trans Neural Netw 22 (8) pp. 1334-1339
Two-class supervised learning in the context of a classifier ensemble may be formulated as learning an incompletely specified Boolean function, and the associated Walsh coefficients can be estimated without the knowledge of the unspecified patterns. Using an extended version of the Tumer-Ghosh model, the relationship between added classification error and second-order Walsh coefficients is established. In this brief, the ensemble is composed of multilayer perceptron base classifiers, with the number of hidden nodes and epochs systematically varied. Experiments demonstrate that the mean second-order coefficients peak at the same number of training epochs as ensemble test error reaches a minimum.
Zor C, Windeatt T, Kittler J (2013) ECOC Matrix Pruning Using Accuracy Information., MCS 7872 pp. 386-397 Springer
Zor C, Yanikoçlu B (2011) Optimization of the ECOC matrix, 2011 IEEE 19th Signal Processing and Communications Applications Conference, SIU 2011 pp. 944-947
Error Correcting Output Coding (ECOC) is a classifier combination technique for multiclass classification problems. In this approach, several base classifiers are trained to learn different dichotomies of the classes, specified by the columns of a code matrix. These classifiers' output for an unknown pattern is compared to the codeword of each class which is the desired output of the dichotomizers, in an error correcting fashion. While ECOC is one of the best solutions to multiclass problems, the solution is suboptimal due to the fact that the code matrix and the dichotomizers are set or learned independently. In this paper, we show an iterative update algorithm for the code matrix that is designed to reduce this decoupling. It consists of updates to the initial code matrix so as to reduce the discrepancy between the code matrix and the output of the trained dichotomizers. We show that the proposed algorithm improves over the basic ECOC approach, for some well-known data sets. © 2011 IEEE.
Zor C, Yanikoglu B, Windeatt T, Alpaydin E (2010) FLIP-ECOC: A greedy optimization of the ECOC matrix, Lecture Notes in Electrical Engineering 62 LNEE pp. 149-154
Error Correcting Output Coding (ECOC) is a multiclass classification technique, in which multiple base classifiers (dichotomizers) are trained using subsets of the training data, determined by a preset code matrix. While it is one of the best solutions to multiclass problems, ECOC is suboptimal, as the code matrix and the base classifiers are not learned simultaneously. In this paper, we show an iterative update algorithm that reduces this decoupling. We compare the algorithm with the standard ECOC approach, using Neural Networks (NNs) as the base classifiers, and show that it improves the accuracy for some well-known data sets under different settings. © 2011 Springer Science+Business Media B.V.
If a binary decision is taken for each classifier in an ensemble, training patterns may be represented as binary vectors. For a two-class supervised learning problem this leads to a partially specified Boolean function that may be analysed in terms of spectral coefficients. In this paper it is shown that a vote which is weighted by the coefficients enables a fast ensemble classifier that achieves performance close to Bayes rate. Experimental evidence shows that effective classifier performance may be achieved with one epoch of training of an MLP using Levenberg-Marquardt with 64 hidden nodes. © 2012 Springer-Verlag Berlin Heidelberg.
This paper proposes a methodology for the automatic detec-
tion of anomalous shipping tracks traced by ferries. The ap-
proach comprises a set of models as a basis for outlier detec-
tion: A Gaussian process (GP) model regresses displacement
information collected over time, and a Markov chain based
detector makes use of the direction (heading) information. GP
regression is performed together with Median Absolute Devi-
ation to account for contaminated training data. The method-
ology utilizes the coordinates of a given ferry recorded on a
second by second basis via Automatic Identification System.
Its effectiveness is demonstrated on a dataset collected in the
Existing ensemble pruning algorithms in the literature have mainly been defined for unweighted or weighted voting ensembles, whose extensions to the Error Correcting Output Coding (ECOC) framework is not successful. This paper presents a novel pruning algorithm to be used in the pruning of ECOC, via using a new accuracy measure together with diversity and Hamming distance information. The results show that the novel method outperforms those existing in the state-of-the-art.
Two-class supervised learning in the context of a classifier ensemble may be formulated as learning an incompletely specified Boolean function, and the associated Walsh coefficients can be estimated without knowledge of the unspecified patterns. Using an extended version of the Tumer-Ghosh model, the relationship between Added Classification Error and second order Walsh coefficients is established. In this paper, the ensemble is composed of Multi-layer Perceptron (MLP) base classifiers, with the number of hidden nodes and epochs systematically varied. Experiments demonstrate that the mean second order coefficients peak at the same number of training epochs as ensemble test error reaches a minimum.
The state of classfier incongruence in decision making systems incorporating
multiple classifiers is often an indicator of anomaly caused by an unexpected
observation or an unusual situation. Its assessment is important as one of the
key mechanisms for domain anomaly detection. In this paper, we investigate
the sensitivity of Delta divergence, a novel measure of classifier incongruence, to
estimation errors. Statistical properties of Delta divergence are analysed both
theoretically and experimentally. The results of the analysis provide guidelines
on the selection of threshold for classifier incongruence detection based on this
In pattern recognition, disagreement between two
classifiers regarding the predicted class membership of an observation
can be indicative of an anomaly and its nuance. As
in general classifiers base their decisions on class aposteriori
probabilities, the most natural approach to detecting classifier
incongruence is to use divergence. However, existing divergences
are not particularly suitable to gauge classifier incongruence. In
this paper, we postulate the properties that a divergence measure
should satisfy and propose a novel divergence measure, referred
to as Delta divergence. In contrast to existing measures, it focuses
on the dominant (most probable) hypotheses and thus reduces the
effect of the probability mass distributed over the non dominant
hypotheses (clutter). The proposed measure satisfies other important
properties such as symmetry, and independence of classifier
confidence. The relationship of the proposed divergence to some
baseline measures, and its superiority, is shown experimentally.
A spectral analysis of a Boolean function is proposed for ap-
proximating the decision boundary of an ensemble of classifiers, and an in-
tuitive explanation of computing Walsh coefficients for the functional ap-
proximation is provided. It is shown that the difference between first and
third order coefficient approximation is a good indicator of optimal base
classifier complexity. When combining Neural Networks, experimental re-
sults on a variety of artificial and real two-class problems demonstrate un-
der what circumstances ensemble performance can be improved. For tuned
base classifiers, first order coefficients provide performance similar to ma-
jority vote. However, for weak/fast base classifiers, higher order coefficient
approximation may give better performance. It is also shown that higher
order coefficient approximation is superior to the Adaboost logarithmic
weighting rule when boosting weak Decision Tree base classifiers.
Error Correcting Output Coding (ECOC) is a multi-
class classification technique in which multiple binary classifiers
are trained according to a preset code matrix such that each one
learns a separate dichotomy of the classes. While ECOC is one of
the best solutions for multi-class problems, one issue which makes
it suboptimal is that the training of the base classifiers is done
independently of the generation of the code matrix.
In this paper, we propose to modify a given ECOC matrix
to improve its performance by reducing this decoupling. The
proposed algorithm uses beam search to iteratively modify the
original matrix, using validation accuracy as a guide. It does not
involve further training of the classifiers and can be applied to
any ECOC matrix.
We evaluate the accuracy of the proposed algorithm (BeamE-
COC) using 10-fold cross-validation experiments on 6 UCI
datasets, using random code matrices of different sizes, and base
classifiers of different strengths. Compared to the random ECOC
approach, BeamECOC increases the average cross-validation
of the experimental settings involving all
datasets, and gives better results than the state-of-the-art in
of the scenarios. By employing BeamECOC, it is also possible to
reduce the number of columns of a random matrix down to
and still obtain comparable or even better results at times.