Training of multi-class linear Classifier with BFGS method

Linear Classifier is a basis model in the Classification learning field. BGFS is a classical unconstraint optimization technique. In this work, we applied BGFS methods into two representative linear Classifiers including logistic regression Classifier and SVM. The classical multi-class logistic regression Classifier uses Newton method to optimize its loss function and suffers the expensive computations and the un-stable iteration process. For SVM, we converted it into an unconstraint optimization problem in the primal form instead of the time-consuming SMO algorithm. The results on 25 datasets show that for logistic regression model BFGS achieves better Classification accuracy than the Newton method. Moreover, BFGS has the lower time complexity, in contrast with Newton method. Finally, we also observe that SVM with BFGS method demonstrates the comparable performance with the SVM Classifier but with the lower running time.