??c ?5?
? 8.0 ?y
? 8.1 ?5 ?
(Regularization Theory)
? 8.2 Reproducing Kernel Hilbert Space
(RKHS)
? 8.3 ?5?
(Regularization Networks)
8.0 ?y
?

T
MY?￥Dù5
? T

" ( X,Y )
),,(,),,(),,(
2211 nn
yxyxyxL
},,2,1{ Ky
i
L∈
,
d
i
x?∈
￥ ?YS
M
i
x
M
 ￥?
?]s?"'b
Y
M
 ￥?
?]s?"'b
X
8.0 ?y
? Df
"
?
>f

"S bW
Τ∈f
))(,( xfyL
8.0 ?y
?
üZμ
>f

? ?
ù?
>f

2
))(())(,( xfyxfyL?=
{

e|)(|
|)(|0
|)(|))(,(
xfy
xfy
xfyxfyL
≤?
=
=
ε
ε
ε
8.0 ?y
? Soft Margin
>f

? Hard Margin
>f

? s ?
>f

{
else
xfyxfy
xfyxfyL
0
0)()(
|)(|))(,(
>
=
=
+
{
else
xfy
xfyhxfyL
0
0)(1
))(())(,(
>?
=
=
))(())(,( xyfhxfyL?=
8.0 ?y
? ü?BKl
2
1
1
(
1
)(
))(,(
1
)(

?
∑
∑
=
=
=
=
n
i
iiemp
n
i
iiemp
xfy
n
fR
xfyL
n
fR
8.0 ?y
?

T
MY?￥,D,￥V?Yè
^ ill-
posed ù5
? B?
àμ·B3
? 3
H
^?×?￥ (overfitting)b
8.0 ?y
? 3 %￥÷E
? Ockham’s razor 
?? ;
? ?5ZE
?

? f
"￥ˉFò@
D SVM￥
±X ?
b
8.1 ?5 ?
eo
8.1 ?5 ?
eo
? L??′ù5L
! - Wi
L??
,
? X?T

"
>f

x
0
= fy
N
R
R
ijijNnij
n
i
N
j
jiji
nn
xAAA
fAyAf
yxyxyx
,,)(
)(
),(,),,(),,(
11
2
2
2211
L
==
=?
×
==
∑∑
y
T
n
T
iNiii
yyyxxxx ),,(,),,(
2121
LL== y
8.1 ?5 ?
eo
? ? A￥ s′s3
y
y
TT
TTT
T
nn
nN
NNnN
ii
nN
iNn
NNnn
T
NNNnnnNn
AAAyAf
AAAAAA
Af
UVAA
VUVUA
1
1
1
1
)(
)(
/1
0
,
+
+
+
×
×
×
+
×
+
×
×
××××××
==
=
=
Σ=
=Σ
>Σ
Σ=
V
I5 ?T
￥<l
I
￥?í
í?l
￥?í
í1 s′
￥?? ?
λλ
λ
8.1 ?5 ?
eo
? Df
" V[ AT
"S bW (Target
Space)?￥?b
? 3￥×??
? ￥l ??
^?? f￥ l ??$
N
RTf =∈
yy?+=?+ )(
0
ffA
ll$
y
8.1 ?5 ?
eo
? B?￥μ
Iù5 p f
YTA
HibertYTYTfAf
→
∈∈=
:
,,
0
bW
^yy
8.1 ?5 ?
eo
?
Iù5 Well-posedD ill-posed
? Hadamard1? well-posedù5￥?l
? i?
? Z??μ3 fb
? ·B? ￡
? Z??μ·B3 fb
? ×??
^ ???
 ￡
3￥×??b
y￥l ??? f￥ l ??b
Yy∈
Yy∈
1?
A
?
^ well-posed ü?1 ill-posedb
Hadamard y9μμ ill-posedù5? V3b
8.1 ?5 ?
eo
? ?5 ?
(Tikhonov 1943,1963)
? μBv ?ù5 Hadamardil/
^ ill-
posedù5? V[sB?×?￥í
3b
? Tikhonov￥
±X
? ??? bW
 p3ü3 K?B?0
"
 p3 V
¤?×?3b
YyHfyAf ∈∈=,,
f
T
TH?
8.1 ?5 ?
eo
? Tikhonovil/￥ well-posedù5
? iB?"
@
?  ?i ?i
? ￥3b
? ?￥3
^·B￥b
? ?￥3
^×?￥ K?"

^ ??￥b
? D Hadamardil/ well-posed?l￥ ó"
? =??N ?T
TH?
YHAy?∈ )(
H
H
H
1?
A
YHA?)(
YHATH == )(
8.1 ?5 ?
eo
?
I
1"￥ V
@μ×?3￥Hq$
? 1 p K ?"
 ??b
? ? ?
I
0￥×??
? ?T  ?" ??5 
9 ??b
? 8"s $
? °¤s+? V
b
? ?5ZEMs 1? 5?
M9F?
Hq?1L
! bW (Hypothesis Space)b
H
1?
A
)(HA
A
XH?
1?
A
)(HA
H
H
8.1 ?5 ?
eo
? L
! bW  è
? m¨[
T
? ?° ?="
? L?f
Z 7
})(|{
10
m
m
xaxaaxffH +++==L
)}cos()sin(
)cos()sin(2/)(|{
110
mxbmxa
xbxaaxffH
mm
++
+++==L
∑
=
==
m
i
ii
xKcxffH
1
)}()(|{
8.1 ?5 ?
eo
? L??′ù5
? ? V
I Hadamardil/
^ ill-posedù5b
? ?5ZE9F?
Hq
? Kl
A
Nn
RfRAf ∈∈=,,yy
μ≤f
2
y?Af
8.1 ?5 ?
eo
? ? Lagrangee0 ?

TT
T
AIAAA
fAfA
f
fL
fAffL
1
22
)(
02)(2
)(
))(()(
+
+=
=+?=
+?=
λ
λ
μλ
y
y
yy
TT
AIAAAf
1
)(
+
+== λ
λ
?53
8.1 ?5 ?
eo
? ?5ZED Ockham’s razor:
? Ockham’s razor:,entities should not be
multiplied beyond necessity,.
? ?5
))((min
2
fyAff
f
Φ+?= λ
λ
Φ
)(
2
f
yAf
μb
Trade off
?
Hq
8.1 ?5 ?
eo
? ? 5?
Mê4?]￥L
! bW
V[¤??]￥?53b
?  zW￥Hq/?53
^
? i￥
? ·B￥
? ??G ??
￥
}0,)(:{ >≤Φ∈= μμfTfHL
! bW
8.1 ?5 ?
eo
? á
ì?3
^,z￥a;á￥, ?
∫
∫
∞
∞?
∞
∞?
=Φ
=Φ
dxxff
dxxff
2''
2
2'
1
)()(
)()(
8.1 ?5 ?
eo
?
L′f
￥ FourierMD
IMD
? Parseva? ?
1
Lf ∈
∫
∫
∞
∞?
∞
∞?
=
=
ωω
π
ω
ω
ω
defxf
dxexff
xj
xj
)(
2
1
)(
)()(
∫∫
∞
∞?
∞
∞?
= ωω
π
dfdxxf
2
2
)(
2
1
)(
8.1 ?5 ?
eo
∫∫
∫∫
∫∫
∫∫
==
==Φ
==
==Φ
∞
∞?
∞
∞?
ω
ω
ω
π
ω
ω
ω
π
ωωω
π
ω
ω
ω
π
ω
ω
ω
π
ωωω
π
d
K
f
d
f
dfdxxff
d
K
f
d
f
dfdxxff
)(
)(
2
1
1
)(
2
1
)(
2
1
)()(
)(
)(
2
1
1
)(
2
1
)(
2
1
)()(
2
2
4
2
2
42''
2
1
2
2
2
2
22'
1
8.1 ?5 ?
eo
?f
??f
￥ Fourier
IM
D ?1f
b
? ?
)(ωK
∞→→≥ xxKxK,0)(,0)(
)()()()/()sin()(
1
1
)(
2
1
)(
)()(
/1)(12/)(
/1)(2/)(
2
||
2/2/
4
3
2
2222
+==
+
=?=
=?=
==
==

ωωωπ
ω
ω
ω
ωω
ωω
γ
σωσ
UUKxxxK
KexK
eKexK
KxxK
KxxK
x
x
8.1 ?5 ?
eo
? ?5ZE3 %f
/íù5
? "'
? ? pf

? ?
Hq
).,(,),,(),,(
2211 nn
yxyxyxL
0,
)(
)(
2
1
2
>≤
∫
μμω
ω
ω
π
d
K
f
∑
=
n
i
ii
xfyf
1
2
))((,Kl
8.1 ?5 ?
eo
? ? ? ( Duchon,1977; Meinguet,1978;
Wahba,1977; Madych and Nelson,1990;
Poggio and Girosi,1989; Girosi,1992) L
!
^f
￥ Fourier
IMDb5
Kl[/Wf￥f
∑
∫
∑
=
=
=
+?
n
i
ii
n
i
ii
xxKxf
d
K
f
yxf
1
2
1
2
)()(
.
)(
)(
2
1
))((
α
ω
ω
ω
π
λ
λ
μ
)(xf
λ
)(ωK
)(xK
8.1 ?5 ?
eo
? dL?ù5￥?5
? ?5id? K?3L?Z?
? ?5
±X V[?¨?3dL?Z?a3K
aù5??b
? ??tù5 ?
?ìs￥
4vv
9F?V?¨???5￥ZE V
[e??1?¨b
8.2 × Hilbert bW (RKHS)
8.2 × Hilbert bW (RKHS)
? Reproducing Kernel Hilbert Space (RKHS)
^B?dè×1￥
Dy Dá
ì
M1￥
5× ?losa |y
*ü?
a SVM+Y
^Kí?1 ￡
ê￥
Kernel machine??μM?￥?¨b
8.2 × Hilbert bW (RKHS)
? RKHS'c￥T¨?
￡
U
B ?
f
V
Uù51
I
1 SVMaRBF?
??C
+M]￥f
[#
ì-W￥
= ó"b
8.2 × Hilbert bW (RKHS)
?
x
f bW?
=
·,· —— ?ó f,g,h∈Ε
α∈?μ
00),(0),(.3
),(),(),,(),(),(.2
),(),(.1
**
*****
**
==≥
=+=+
=
fiffffandff
gfgfhghfhgf
fggf
αα
8.2 × Hilbert bW (RKHS)
? B? bW
^? ?￥ ?T
μ
l ??

?
l ?? bW?￥?b
? ?TB?
x
f bW
^? ?￥?1B?
Hilbert bWb ? ?
^ Hilbert bWb
n
RL,
2
8.2 × Hilbert bW (RKHS)
? RKHS?l
?f
bW
^ Hilbert bW

=1
?if
O
@
?S× ?1×
Hilbert bW (RKHS)b
H H
.))(),((),(
x
xgxfgf =
),,( yxK
,))(),,(()(
y
yfyxKxf =
.f?
H),( yxK
HxK ∈?),(,x?
8.2 × Hilbert bW (RKHS)
? RKHS￥ è0
? ??loMDo?￥ RKHS:
? ?MD
?
IMD
.)()()(),(
2
1
∫
==
dx
a
bx
xfafTbaF
wav
ψ
wavelets
.)(),(
1
)(
2
1
∫∫
=
dadb
a
bx
baF
a
Cxf ψ
ψ
BBè
8.2 × Hilbert bW (RKHS)
IMD} ??MD
.)()(),(
1
),(
2
1
2
1
∫∫ ∫
′
′
′′′′
′
=
dx
a
bx
a
bx
abdadbaF
a
CbaF ψψ
ψ
×
.)()(),;,(
2
1
∫
′
′
=
′′
dx
a
bx
a
bx
ababaK ψψ
RKHS￥
=
.),(),(
1
)),(),,((
2
1
,
dadbbaGbaF
a
CbaGbaF
ba
∫∫
=
ψ
.)),(),,;,((),(
,ba
baFbabaKbaF
′′
′′′′
=
×
8.2 × Hilbert bW (RKHS)
?/×f

? 1
I
1/×$
? ×f
?5?
a SVM?μ×1T¨b
?/ZE
? ¨L?í1￥f
?
?
?
/

n/B?
L),(),(
21
xx φφ
L,,
21
λλ
.)()(),(
∑
=
n
nnn
yxyxK φφλ
×
8.2 × Hilbert bW (RKHS)
=
∑
=
n
nn
xxf ),()( φα
∑
=
n
nn
xxg ),()( φβ
∑
=
n
n
nn
xgxf,))(),((
λ
βα

@
=Hq
∑
≥=
n
n
n
xfxf,0))(),((
2
λ
α
.0>
n
λ
×
.)()(
)(
))(),,((
∑∑
===
n
nn
n
n
nnn
xfx
x
yfyxK φα
λ
αφλ
8.2 × Hilbert bW (RKHS)
? è0
?
￥ RKHS:]2,0[ π
).),2cos(),2sin(),cos(),sin(,1()}({
0
Lxxxxx
nn
=
≥
φ
.coscossinsin),(
11
0 ∑∑
≥≥
′
++=
n
n
n
n
nynxnynxyxK λλλ
,0>
n
λ,0>
′
n
λ
@
￥ ?i
^×f
b),( yxK
8.2 × Hilbert bW (RKHS)
B?+
y f ?
,
nn
λλ
′
=L,2,1=n
.
)(cos
coscossinsin),(
)(
0
11
0
∑
∑
∑∑
∞=
∞=
≥
≥≥
=
=
′
++=
n
n
yxin
n
n
n
n
n
n
n
ec
yxn
nynxnynxyxK
λ
λλλ
<
≥
=
.0
2
.0
2
n
n
c
n
n
n
λ
λ
8.2 × Hilbert bW (RKHS)
.),(
)(
∑
∞=
∞=
=
n
n
yxin
n
ecyxK
o
^ ￥f
,),( yxK )( yx?
.),()(
)(
∑
∞=
∞=
==?
n
n
yxin
n
ecyxKyxK
Kè¨
7
^×
),( yxK
o3
0>
n
λ
L,2,1=n
o3
0>
n
c
￥ Fourier)
),( yxK
8.2 × Hilbert bW (RKHS)
? w<?
￥ RKHS,
K×1￥×
d
]2,0[ πRKHS RKHS
d
},:)({ Znx
n
∈φ
}:),({
d
x?∈ξξφ
,)()(),(
∑
=
n
nnn
yxyxK φφλ,),(),()(),( ξξφξφξλ dyxyxK
d
∫
=
∑
=
n
nn
xxf ),()( φα
.),()(
)( ξξφξ dxfxf
∫
=
,))(),((
∑
=
n
n
nn
xgxf
λ
βα
.
)(
)(
)(
))(),(( ξ
ξλ
ξξ
d
gf
xgxf
∫
=
8.2 × Hilbert bW (RKHS)
 RKHS
d
).cos(),sin(),( xxx= ξξξφ
? ? ?
1¤?
.)(),(
)(
∫

= ξξ
ξ
decyxK
yxi
),( yxK
^×o1
^?￥b
8.2 × Hilbert bW (RKHS)
Gauss￥
^?￥ ).exp(),(
2
yxyxK= γ
),
2
exp()(
2
z
zf?=,),,,(
21
d
d
zzzz?∈=L
∑
∑
∫
∫
=
=
==
=
=
d
i
i
d
i
iii
i
T
constconst
dzzi
z
dzzi
z
f
1
2
2
1
2
2
).
2
exp()
2
exp(
)exp()
2
exp(
)exp()
2
exp()(
ξ
ξ
ξ
ξξ
Gauss
^×
8.2 × Hilbert bW (RKHS)
? ×DdL? SVM:
? dL? SVM1s V[V
U1
=￥f

? ×
@1 py1
)).(),((),( yzxzyxK =
+? bW
),)())((()()(),(
∑∑
==
n
nnnn
n
nnn
yxyxyxK φλφλφφλ
.0>
n
λ
8.2 × Hilbert bW (RKHS)
? M
?
 ?T K
^×
*
1i
B??

P¤
))(,),(),(()(
),(2),(),()()(
2211
2
LLxxxx
xxKxxKxxKxx
nn
φλφλφλ=Φ
′
′′
+=
′
Φ?Φ
8.2 × Hilbert bW (RKHS)
?￥ê4e5 V[V
bW￥?

?
Hq
Z
I
nb 8?¨
? I
n ?D
L=ù5?5?
M￥2
b No Free Lunch!
?
￥M
?

?
￥VrZ
T
? L
! bW￥ê4
8.3 ?5?
8.3 ?5?
? ?5?
,Regularization Networks (RN)
? ?5?
￥?
? V"'?D￥V? V[ ATf
/íù5
? ?
^ ill-posedù5
).,(,),,(),,(
2211 nn
yxyxyxL
.,yxf →
DB?f
8.3 ?5?
? ?5ZE3 %f
/íù5
? "'
? ? pf

? ?5
).,(,),,(),,(
2211 nn
yxyxyxL
)())(()(
1
2
fyxffL
n
i
ii
Φ+?=
∑
=
λ
∑
=
n
i
ii
xfyf
1
2
))((,
?5"
8.3 ?5?
?1o
L
! bW H￥ê |b
? RKHS
.))(()(
2
1
2
H
n
i
ii
fyxffL λ+?=
∑
=
¨ RKHS?￥S

,)()(),(
∑
=
n
nnn
xxK yy φφλ
∑
==
n
n
n
H
fff,),(
2
2
λ
α
∑
=
n
nn
xxf ),()( φα
8.3 ?5?
? pf
o 3 p"
' Vb
f
L,,
21
αα
.2)())((2
1
j
j
n
i
ijii
j
xyxf
L
λ
α
λφ
α
+?=
∑
=
?5"
0=
j
L
α
.)())((
1
∑
=
=
n
i
ijii
j
j
xyxf φ
λ
λ
α
8.3 ?5?
∑
∑∑
∑∑
∑
=
=
=
=
=
=
=
n
i
iii
j
n
ij
ijjii
j
j
n
i
ijii
j
j
jj
xxKxfy
xxxfy
xxyxf
xxf
1
1
1
),())((
1
)()())((
1
)()}())(({
)()(
λ
φφλ
λ
φφ
λ
λ
φα
"
×
8.3 ?5?
yy
y
1
2121
1
1
1
)(,)(
),,,(,),,,(
),(
),()(
),()(
))((
1
),()(
=
=
=
+=+=
==
=
+=+=
=
==
∑
∑
∑
IKCCIK
cccCyyy
xxKK
ccxxKcxfy
xxKcxf
xfycxxKcxf
T
n
T
n
jiij
ji
n
i
jijjj
n
i
jiij
iii
n
i
ii
λλ
λλ
λ
5
7
LL
8.3 ?5?
? ?5?
(RN)
L
L
L
1
x
2
x
1
c
d
x
∑
),( xxK
i
2
c
n
c
xInput
8.3 ?5?
? 
}V
U
T
n
n
i
ii
i
n
i
i
xxKxxKxxKxK
xKIKx
xbyxxKIK
xKIKxKCxxKcxf
)),(,),,(),,(()(
)()()(
)()()()(
)()()(),()(
21
1
1
1
1
1
L=
+=
=?=+?=
+=?==
→
→
=
→
→
→
=
∑
∑
?1?Nλ
λ
λ
b
byy
y
8.3 ?5?
? RBF?
^?5?
￥B?y1 Gauss

^×
? SVMD?5?
￥
±Xdè¤ío
^
>f
￥?
T?]b SVMoG ??|?_

??5?
àμ]"￥?éb
8.3 ?5?
? SRMaSVM RN
∑∑
∑
∑
∑
+==
+?
+?
+
=
=
+
=
bxxKcxfxxKcxf
fxfy
n
RN
fxfy
n
SVMc
AfxfyL
n
SRM
iiii
K
n
i
ii
K
n
i
ii
m
K
m
n
i
ii
),()(),()(
))((
1
|)(|
1
)())(,(
1
2
1
2
2
1
2
1

λ
λ
λ
? IóD
? [1] V.A,Morozov,Methods for Solving Incorrectly Posed
Problems,Springer-Verlag,1974.
? [2] Vladimir N,Vapnik,Statistical Learning Theory,John
Wiley & Sons,Inc,1998,(?Bc￥?
cμe1o
b )
? [3] F,Girosi,An Equivalence Between Sparse
Approximation and Support Vector Machine,Neural
Computation,vol.10,1455-1480,1998,(Appendix A)
? [4] Alex J,Smola,A Tutorial on Support Vector
Regression.