-
Notifications
You must be signed in to change notification settings - Fork 4
/
README
122 lines (78 loc) · 4.14 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
LiblineaR is a wrapper around the LIBLINEAR C/C++ library for machine learning.
LIBLINEAR is a simple library for solving large-scale regularized linear
classification and regression. It currently supports
- L2-regularized logistic regression/L2-loss support vector classification (SVM)/
L1-loss support vector classification (SVM)
- L1-regularized L2-loss support vector classification/L1-regularized logistic regression
- L2-regularized L2-loss support vector regression/L1-loss support vector regression.
The main features of LiblineaR include multi-class classification (one-vs-the rest, and
Crammer & Singer method), cross validation for model selection, probability
estimates (logistic regression only) or weights for unbalanced data. The
estimation of the models is particularly fast as compared to other libraries.
The two first blocks of the package version indicates which version of LIBLINEAR is
currently supported by LiblineaR. For example: 1.32-14 means that the package supports
the version 1.32 of LIBLINEAR.
For more information on the C/C++ LIBLINEAR library itself, refer to:
R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin.
LIBLINEAR: A Library for Large Linear Classification,
Journal of Machine Learning Research 9(2008), 1871-1874.
Software available at http://www.csie.ntu.edu.tw/~cjlin/liblinear .
Copyright
---------
All of this software is copyrighted by the list of authors in the DESCRIPTION file of
the package and subject to the GNU GENERAL PUBLIC LICENSE, Version 2, see the file
COPYING for details. The LIBLINEAR C/C++ code is copyright Chih-Chung Chang and
Chih-Jen Lin.
Installation
------------
This package should be installed using the `R CMD INSTALL' mechanism, see
the R online help or R manuals for details on how to install packages.
Mathematical Details
--------------------
Formulations:
For L2-regularized logistic regression (-s 0), we solve
min_w w^Tw/2 + C \sum log(1 + exp(-y_i w^Tx_i))
For L2-regularized L2-loss SVC dual (-s 1), we solve
min_alpha 0.5(alpha^T (Q + I/2/C) alpha) - e^T alpha
s.t. 0 <= alpha_i,
For L2-regularized L2-loss SVC (-s 2), we solve
min_w w^Tw/2 + C \sum max(0, 1- y_i w^Tx_i)^2
For L2-regularized L1-loss SVC dual (-s 3), we solve
min_alpha 0.5(alpha^T Q alpha) - e^T alpha
s.t. 0 <= alpha_i <= C,
For L1-regularized L2-loss SVC (-s 5), we solve
min_w \sum |w_j| + C \sum max(0, 1- y_i w^Tx_i)^2
For L1-regularized logistic regression (-s 6), we solve
min_w \sum |w_j| + C \sum log(1 + exp(-y_i w^Tx_i))
For L2-regularized logistic regression (-s 7), we solve
min_alpha 0.5(alpha^T Q alpha) + \sum alpha_i*log(alpha_i) + \sum (C-alpha_i)*log(C-alpha_i) - a constant
s.t. 0 <= alpha_i <= C,
where
Q is a matrix with Q_ij = y_i y_j x_i^T x_j.
For L2-regularized L2-loss SVR (-s 11), we solve
min_w w^Tw/2 + C \sum max(0, |y_i-w^Tx_i|-epsilon)^2
For L2-regularized L2-loss SVR dual (-s 12), we solve
min_beta 0.5(beta^T (Q + lambda I/2/C) beta) - y^T beta + epsilon \sum |beta_i|
For L2-regularized L1-loss SVR dual (-s 13), we solve
min_beta 0.5(beta^T Q beta) - y^T beta + epsilon \sum |beta_i|
s.t. -C <= beta_i <= C,
where
Q is a matrix with Q_ij = x_i^T x_j.
If bias >= 0, w becomes [w; w_{n+1}] and x becomes [x; bias].
The primal-dual relationship implies that -s 1 and -s 2 give the same
model, -s 0 and -s 7 give the same, and -s 11 and -s 12 give the same.
We implement 1-vs-the rest multi-class strategy for classification.
In training i vs. non_i, their C parameters are (weight from -wi)*C
and C, respectively. If there are only two classes, we train only one
model. Thus weight1*C vs. weight2*C is used. See examples below.
We also implement multi-class SVM by Crammer and Singer (-s 4):
min_{w_m, \xi_i} 0.5 \sum_m ||w_m||^2 + C \sum_i \xi_i
s.t. w^T_{y_i} x_i - w^T_m x_i >= \e^m_i - \xi_i \forall m,i
where e^m_i = 0 if y_i = m,
e^m_i = 1 if y_i != m,
Here we solve the dual problem:
min_{\alpha} 0.5 \sum_m ||w_m(\alpha)||^2 + \sum_i \sum_m e^m_i alpha^m_i
s.t. \alpha^m_i <= C^m_i \forall m,i , \sum_m \alpha^m_i=0 \forall i
where w_m(\alpha) = \sum_i \alpha^m_i x_i,
and C^m_i = C if m = y_i,
C^m_i = 0 if m != y_i.