1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
|
Virtual Routing and Forwarding (VRF)
====================================
The VRF device combined with ip rules provides the ability to create virtual
routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
Linux network stack. One use case is the multi-tenancy problem where each
tenant has their own unique routing tables and in the very least need
different default gateways.
Processes can be "VRF aware" by binding a socket to the VRF device. Packets
through the socket then use the routing table associated with the VRF
device. An important feature of the VRF device implementation is that it
impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
(ie., they do not need to be run in each VRF). The design also allows
the use of higher priority ip rules (Policy Based Routing, PBR) to take
precedence over the VRF device rules directing specific traffic as desired.
In addition, VRF devices allow VRFs to be nested within namespaces. For
example network namespaces provide separation of network interfaces at the
device layer, VLANs on the interfaces within a namespace provide L2 separation
and then VRF devices provide L3 separation.
Design
------
A VRF device is created with an associated route table. Network interfaces
are then enslaved to a VRF device:
+-----------------------------+
| vrf-blue | ===> route table 10
+-----------------------------+
| | |
+------+ +------+ +-------------+
| eth1 | | eth2 | ... | bond1 |
+------+ +------+ +-------------+
| |
+------+ +------+
| eth8 | | eth9 |
+------+ +------+
Packets received on an enslaved device and are switched to the VRF device
in the IPv4 and IPv6 processing stacks giving the impression that packets
flow through the VRF device. Similarly on egress routing rules are used to
send packets to the VRF device driver before getting sent out the actual
interface. This allows tcpdump on a VRF device to capture all packets into
and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
applied using the VRF device to specify rules that apply to the VRF domain
as a whole.
[1] Packets in the forwarded state do not flow through the device, so those
packets are not seen by tcpdump. Will revisit this limitation in a
future release.
[2] Iptables on ingress supports PREROUTING with skb->dev set to the real
ingress device and both INPUT and PREROUTING rules with skb->dev set to
the VRF device. For egress POSTROUTING and OUTPUT rules can be written
using either the VRF device or real egress device.
Setup
-----
1. VRF device is created with an association to a FIB table.
e.g, ip link add vrf-blue type vrf table 10
ip link set dev vrf-blue up
2. An l3mdev FIB rule directs lookups to the table associated with the device.
A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
l3mdev rule for IPv4 and IPv6 when the first device is created with a
default preference of 1000. Users may delete the rule if desired and add
with a different priority or install per-VRF rules.
Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
ip ru add oif vrf-blue table 10
ip ru add iif vrf-blue table 10
3. Set the default route for the table (and hence default route for the VRF).
ip route add table 10 unreachable default
4. Enslave L3 interfaces to a VRF device.
ip link set dev eth1 master vrf-blue
Local and connected routes for enslaved devices are automatically moved to
the table associated with VRF device. Any additional routes depending on
the enslaved device are dropped and will need to be reinserted to the VRF
FIB table following the enslavement.
The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
addresses as VRF enslavement changes.
sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
5. Additional VRF routes are added to associated table.
ip route add table 10 ...
Applications
------------
Applications that are to work within a VRF need to bind their socket to the
VRF device:
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
or to specify the output device using cmsg and IP_PKTINFO.
TCP services running in the default VRF context (ie., not bound to any VRF
device) can work across all VRF domains by enabling the tcp_l3mdev_accept
sysctl option:
sysctl -w net.ipv4.tcp_l3mdev_accept=1
netfilter rules on the VRF device can be used to limit access to services
running in the default VRF context as well.
The default VRF does not have limited scope with respect to port bindings.
That is, if a process does a wildcard bind to a port in the default VRF it
owns the port across all VRF domains within the network namespace.
################################################################################
Using iproute2 for VRFs
=======================
iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
section lists both commands where appropriate -- with the vrf keyword and the
older form without it.
1. Create a VRF
To instantiate a VRF device and associate it with a table:
$ ip link add dev NAME type vrf table ID
As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
device create.
2. List VRFs
To list VRFs that have been created:
$ ip [-d] link show type vrf
NOTE: The -d option is needed to show the table id
For example:
$ ip -d link show type vrf
11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 1 addrgenmode eui64
12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 10 addrgenmode eui64
13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 66 addrgenmode eui64
14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 81 addrgenmode eui64
Or in brief output:
$ ip -br link show type vrf
mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
3. Assign a Network Interface to a VRF
Network interfaces are assigned to a VRF by enslaving the netdevice to a
VRF device:
$ ip link set dev NAME master NAME
On enslavement connected and local routes are automatically moved to the
table associated with the VRF device.
For example:
$ ip link set dev eth0 master mgmt
4. Show Devices Assigned to a VRF
To show devices that have been assigned to a specific VRF add the master
option to the ip command:
$ ip link show vrf NAME
$ ip link show master NAME
For example:
$ ip link show vrf red
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
Or using the brief output:
$ ip -br link show vrf red
eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
5. Show Neighbor Entries for a VRF
To list neighbor entries associated with devices enslaved to a VRF device
add the master option to the ip command:
$ ip [-6] neigh show vrf NAME
$ ip [-6] neigh show master NAME
For example:
$ ip neigh show vrf red
10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
$ ip -6 neigh show vrf red
2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
6. Show Addresses for a VRF
To show addresses for interfaces associated with a VRF add the master
option to the ip command:
$ ip addr show vrf NAME
$ ip addr show master NAME
For example:
$ ip addr show vrf red
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 2002:1::2/120 scope global
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe00:202/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 2002:2::2/120 scope global
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe00:203/64 scope link
valid_lft forever preferred_lft forever
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
Or in brief format:
$ ip -br addr show vrf red
eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
eth5 DOWN
7. Show Routes for a VRF
To show routes for a VRF use the ip command to display the table associated
with the VRF device:
$ ip [-6] route show vrf NAME
$ ip [-6] route show table ID
For example:
$ ip route show vrf red
prohibit default
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
$ ip -6 route show vrf red
local 2002:1:: dev lo proto none metric 0 pref medium
local 2002:1::2 dev lo proto none metric 0 pref medium
2002:1::/120 dev eth1 proto kernel metric 256 pref medium
local 2002:2:: dev lo proto none metric 0 pref medium
local 2002:2::2 dev lo proto none metric 0 pref medium
2002:2::/120 dev eth2 proto kernel metric 256 pref medium
local fe80:: dev lo proto none metric 0 pref medium
local fe80:: dev lo proto none metric 0 pref medium
local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth2 proto kernel metric 256 pref medium
ff00::/8 dev red metric 256 pref medium
ff00::/8 dev eth1 metric 256 pref medium
ff00::/8 dev eth2 metric 256 pref medium
8. Route Lookup for a VRF
A test route lookup can be done for a VRF:
$ ip [-6] route get vrf NAME ADDRESS
$ ip [-6] route get oif NAME ADDRESS
For example:
$ ip route get 10.2.1.40 vrf red
10.2.1.40 dev eth1 table red src 10.2.1.2
cache
$ ip -6 route get 2002:1::32 vrf red
2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
9. Removing Network Interface from a VRF
Network interfaces are removed from a VRF by breaking the enslavement to
the VRF device:
$ ip link set dev NAME nomaster
Connected routes are moved back to the default table and local entries are
moved to the local table.
For example:
$ ip link set dev eth0 nomaster
--------------------------------------------------------------------------------
Commands used in this example:
cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
1 mgmt
10 red
66 blue
81 green
EOF
function vrf_create
{
VRF=$1
TBID=$2
# create VRF device
ip link add ${VRF} type vrf table ${TBID}
if [ "${VRF}" != "mgmt" ]; then
ip route add table ${TBID} unreachable default
fi
ip link set dev ${VRF} up
}
vrf_create mgmt 1
ip link set dev eth0 master mgmt
vrf_create red 10
ip link set dev eth1 master red
ip link set dev eth2 master red
ip link set dev eth5 master red
vrf_create blue 66
ip link set dev eth3 master blue
vrf_create green 81
ip link set dev eth4 master green
Interface addresses from /etc/network/interfaces:
auto eth0
iface eth0 inet static
address 10.0.0.2
netmask 255.255.255.0
gateway 10.0.0.254
iface eth0 inet6 static
address 2000:1::2
netmask 120
auto eth1
iface eth1 inet static
address 10.2.1.2
netmask 255.255.255.0
iface eth1 inet6 static
address 2002:1::2
netmask 120
auto eth2
iface eth2 inet static
address 10.2.2.2
netmask 255.255.255.0
iface eth2 inet6 static
address 2002:2::2
netmask 120
auto eth3
iface eth3 inet static
address 10.2.3.2
netmask 255.255.255.0
iface eth3 inet6 static
address 2002:3::2
netmask 120
auto eth4
iface eth4 inet static
address 10.2.4.2
netmask 255.255.255.0
iface eth4 inet6 static
address 2002:4::2
netmask 120
|
