Comparison of Second Order Algorithms for Function Approximation with Neural Networks

E. Boutalbia,b, L. Ait Gougamb and F. Mekideche-Chafab aLaboratory of Particle Physics and Statistical Physics, Ecole Normale, Supérieure BP 92 Vieux Kouba, Algeria bLaboratory of Theoretical Physics, Faculty of Sciences-Physics, USTHB, B.P. 32, El-Alia, Algeria

Full Text PDF

The Neural networks are massively parallel, distributed processing systems representing a new computational technology built on the analogy to the human information processing system. They are usually considered as naturally parallel computing models. The combination of wavelets with neural networks can hopefully remedy each other's weaknesses, resulting in wavelet based neural network capable of approximating any function with arbitrary precision. A wavelet based neural network is a nonlinear regression structure that represents nonlinear mappings as the superposition of dilated and translated versions of a function, which is found both in the space and frequency domains. The desired task is usually obtained by a learning procedure which consists in adjusting the "synaptic weights". For this purpose, many learning algorithms have been proposed to update these weights. The convergence for these learning algorithms is a crucial criterion for neural networks to be useful in different applications. In this paper, we use different training algorithms for feed forward wavelet networks used for function approximation. The training is based on the minimization of the least-square cost function. The minimization is performed by iterative first and second order gradient-based methods. We make use of the Levenberg-Marquardt algorithm to train the architecture of the chosen network and, then, the training procedure starts with a simple gradient method which is followed by a BFGS (Broyden, Fletcher, Glodfarb et Shanno) algorithm. The conjugate gradient method is then used. The performances of the different algorithms are then compared. It is found that the advantage of the last training algorithm, namely, conjugate gradient method, over many of the other optimization algorithms is its relative simplicity, efficiency and quick convergence.

DOI: 10.12693/APhysPolA.128.B-271 PACS numbers: 02.30.Mv, 07.05.Mh