Network Connection in the NDPF
The configuration files and the source to the generated ACLs is in subversion at https://ndpfsvn.nikhef.nl/repos/ndpf/nl.nikhef.ndpf.config/routers/. Use your farm username and password (if you're authorized to begin with, i.e. are a member of the NDPFAdministrators directory group to get access, or check out from SVN using ssh.
To view actual bandwidth usage, Cricket graphs are available at
http://www.dutchgrid.nl/ndpf/cricket/grapher.cgi (or on salado, the network management host, if you are within the Nikhef domain).
The high-level overview was drawn by Tristan at Nikhef network.
VLAN Identification and network naming
Colour coding: red is in use on deel, green is in use on nikgrid, and thus yellow is in use on both ...
VLAN Table
| VLAN ID |
logicalName |
Description
|
| 2 |
p4ctb |
Limited Access VL-e and BiGGrid P4 Certification Test Bed
|
| 3 |
public-comb |
NDPF production core and grid services combined
|
| 4 |
ipmi |
Local management network
|
| 5 |
interconnect |
Interconnect network to hef-router
|
| 6 |
gridsrv |
Service box network
|
| 7 |
vobox |
Class-1 VObox network
|
| 8 |
nordic |
Experimental Open Net
|
| 9 |
farmnet |
Worker nodes
|
| 10 |
opninterconnect |
Interconnect network to nikopn
|
| 11 |
(rofcluster) |
Reserved
|
| 12 |
deelinterconnect |
Interconnect network to deel
|
| 13 |
opnuplink |
LHCOPN up link to SARA
|
| 14 |
opnstorage |
Storage servers
|
Subnet allocations
Subnet Table
| Prefix |
Name |
Description
|
| 192.16.186.192/26 |
public-sec |
Public NDPF network with services (core part)
|
| 194.171.96.0/24 |
(misc) |
several special-purpose networks, please see NDPF Node Functions
|
| 194.171.97.0/24 |
public-grid |
Public NDPF network with services (grid and ui part)
|
| 194.171.98.0/23 |
farmnet |
Worker nodes
|
| 194.171.100.0/22 |
(new) |
(new)
|
| 2001:0610:0120::/48 |
IKONET-IPv6 |
Nikhef
|
Network Management System
A dedicated system is used to control the routers and the management switch. It has three ethe interfaces: eth0 connects it to the Nikhef guestnet (as salado.nikhef.nl) and should always be reachable, even if the deel and nikopn routers themselves are messed up. It's second interface (eth1) is connected to deel as a tagged interconnect, and links it to the public-sec network (as schoffel.nikhef.nl) as well as the ipmi network (as
salado.ipmi.nikhef.nl, i.e., 172.20.1.255). The third interface actually connects the management blades of deel and nikopn, and can be used to control sw-mngt-01 (the 10/100 switch in c14). It has the address 192.168.254.4/24, where on that network deel is 192.168.254.1, nikopn is 192.168.254.2, and sw-mngt-01 is 192.168.254.254.
Logging from this box is sent to boes.nikhef.nl, whose IP address is statically configured in /etc/hosts. Note that boes MUST be on a directly connected network to remain secure. It also has the MAC address of boes hardwires in /etc/ethers.
A copy of the cricket installation, configuration and data as of Oct 24, 2008, is available at (hefnet):/global/ices/grid/nikhef/network.
Setting and applying ACLs
The ACLs on deel and nikopn can only be inbound, and should eb applied to all physical and virtual interfaces in a consistent way in order to be effective. Since deel contains many different kinds of networks, it is logically easier to think of the protected to be applied to the outbound direction, i.e. what kind of traffic you allow to flow towards a particular subnet, instead of thinkging aboutn the traffic allowed inbound. However, the outbound ACLs you would need to implement the logical idea are not supported on the BigIron-RX series hardware. Similarly, access-policy-groups cannot be used.
To offset this limitation, a translation utility (tr-acl) has been built to convert logical access control rules in a set of inbound-only ACLs to be used on the foundry. It converts a high-level description of all connected subnets and the list of access controls to be applied in a set of inbound ALCs, one for each subnet. To work in this way, you:
- write or update the high-level description. The one used for deel and nikopn is in the private SVN "ndpf" repository (ndpf/nl.nikhef.ndpf.config/router/ruleset.in or ruleset-nikopn.in). Make sure to commit and log changes
- run the tr-acl command: tr-acl ruleset.in > /tftpboot/deel-acl
- load the new ACLs in the router: cop tftp run 192.168.254.4 deel-acl
- apply them to the interfaces: conf t and ip rebind-acl all
- check to make sure that the CAM has not been exhausted and all ACLs actually have been applied. There should be no errors if you give the command: sho log
Example ruleset
The ruleset language looks an awful lot like the foundry syntax, but defines a couple of new constructs like subnetwork definitions, fixed pre- and post-fixes, and simple (non-recursive!) macros. For example
! ruleset for new deel router
!
interface interconnect
connects 192.16.186.164/30
connects 0.0.0.0/0
excludes 192.16.186.192/26
excludes 192.16.186.128/30
excludes 194.171.96.0/21
excludes 145.100.9.44/30
excludes 172.16.0.0/12
excludes 239.0.0.0/8
excludes 0.0.0.0/32
! ban forged source IPs from entering
! do this once we configure IPs on our own VLANs
prepend deny ip 194.171.96.0/21 any
prepend deny ip 192.16.186.192/26 any
end
! macro definitions
stanza permit-webserver
permit tcp $src $dest eq http
permit tcp $src $dest eq ssl
end
!
! actual rules start here
ruleset
!
! always allow established connections but filter localdomain
permit tcp any any established
deny ip 127.0.0.0/8 any
deny ip any 127.0.0.0/8
!
! HTTP and SSL servers on gridsrv are accessible from anywhere
$permit-webserver(src=any,dest=194.171.96.64/28)
!
! and deny all other traffic
deny ip any any
!
end
Deel
Module 3 (rx-bi-10g-4-port)
Module 3 (rx-bi-10g-4-port)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1
|
| 2
|
| 3 |
10 |
K80501617 |
nikopn |
|
| 4
|
Module 6 (rx-bi-1g-24-port-copper)
ports 13-24 reserved for the public-sec to-be VLAN
Module 6 (rx-bi-1g-24-port-copper)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
5 |
HEF21/2-19 |
hef-router |
trunk 6/2
|
| 2 |
5 |
Cross 51 |
hef-router |
trunk 6/1
|
| 3 |
T3,4 |
26 |
c14: sw-public-sec-01 |
trunk 6/4
|
| 4 |
T3,4 |
27 |
c14: sw-public-sec-01 |
trunk 6/3
|
| 5
|
| 6
|
| 7
|
| 8
|
| 9
|
| 10
|
| 11
|
| 12
|
| 13
|
| 14
|
| 15
|
| 16
|
| 17
|
| 18
|
| 19
|
| 20
|
| 21
|
| 22
|
| 23
|
| 24
|
Module 7 (rx-bi-1g-24-port-fiber)
Module 7 (rx-bi-1g-24-port-fiber)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1
|
| 2
|
| 3 |
9 |
NIK1354 |
c26:sw-luilak2-01 |
|
| 4 |
9 |
NIK1319 |
c27:sw-luilak2-02 |
|
| 5 |
9 |
NIK1246 |
c23:sw-luilak1-01 |
|
| 6 |
9 |
NIK1247 |
c23:sw-luilak1-02 |
|
| 7
|
| 8
|
| 9
|
| 10
|
| 11
|
| 12
|
| 13 |
3 |
080402/009/010 |
c18:sw-public-comb |
|
| 14
|
| 15
|
| 16
|
| 17
|
| 18
|
| 19
|
| 20
|
| 21
|
| 22
|
| 23 |
|
|
|
1350nm
|
| 24 |
|
|
|
1350nm
|
Module 10 (rx-bi-1g-24-port-copper)
Module 10 (rx-bi-1g-24-port-copper)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
T3,4 |
1 |
c14:sw-public-grid-01 |
trunk 10/2
|
| 2 |
T3,4 |
2 |
c14:sw-public-grid-01 |
trunk 10/1
|
| 3
|
| 4
|
| 5
|
| 6 |
4 |
14 |
c14:sw-mngt-01
|
| 7 |
T3,4 |
NDPF-0814-2 |
c14:netmanager (salado)
|
| 8
|
| 9 |
3 |
NIK1440 |
c16:bedstee
|
| 10 |
|
| 11 |
4 |
(blue) |
c14:nikopn-rtr |
mngt net extension
|
| 12 |
4 |
F0 |
c22:terp (DRCS)
|
| 13
|
| 14
|
| 15
|
| 16
|
| 17 |
6 |
#200 |
c02:hek
|
| 18 |
6 |
#202 |
c02:gierput
|
| 19 |
6 |
#203 |
c02:kaasvat
|
| 20 |
6 |
HA-6 |
c23:keerder
|
| 21
|
| 22
|
| 23
|
| 24
|
Module 13 (rx-bi-1g-24-port-copper)
Module 13 (rx-bi-1g-24-port-copper)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
9 |
F1 |
c21:sw-bulldozer-01
|
| 2 |
9 |
|
??: farmnet-vulnassess
|
| 3
|
| 4
|
| 5
|
| 6
|
| 7 |
9 |
F2 |
c21:sw-bulldozer-02
|
| 8 |
9 |
valentine-5 |
c07:sw-valentine-01
|
| 9 |
9 |
valentine-7 |
c06:sw-valentine-02
|
| 10 |
9 |
valentine-8 |
c04:sw-valentine-03
|
| 11
|
| 12
|
| 13 |
2 |
generic-07 |
c26:span
|
| 14 |
8 |
7 (seven) |
c15:melktank
|
| 15
|
| 16 |
2 |
(unknown) |
c1:speeltuin hall/BI15k
|
| 17 |
2 |
48 |
kudde
|
| 18 |
2 |
46 |
toom
|
| 19 |
8 |
NIK1212 |
virtualschoollab
|
| 20 |
|
| 21 |
8 |
[43] |
arrone
|
| 22 |
8 |
[44] |
aulnes
|
| 23
|
| 24 |
2 |
HA-4 |
c23:bleek
|
Module 16 (rx-bi-1g-48-port-copper)
Do not use PPRC4 (ports 37-48) until a firmware fix has been provided by FN
Module 16 (rx-bi-1g-48-port-copper)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
3 |
F3 |
c22:dorsvlegel
|
| 2 |
3 |
F4 |
c22:tbn15
|
| 3 |
3 |
43 (dupl) |
c16:bedstee (alternate)
|
| 4 |
3 |
[41] |
c15:erf
|
| 5 |
3 |
[42] |
c15:hooiberg-1
|
| 6 |
3 |
[40] |
c15:hooiberg-2
|
| 7 |
3 |
HA-0 |
c23:hilde
|
| 8 |
3 |
HA-3 |
c23:kaf
|
| 9 |
3 |
HA-5 |
c23:kribbe
|
| 10 |
3 |
HA-7 |
c23:boszwijn
|
| 11 |
3 |
HA-8 |
c23:schuur
|
| 12 |
3 |
HA-9 |
c23:hoeve
|
| 13 |
6 |
#201 |
c02:beerput
|
| 14 |
6 |
0729-09 |
c17:kuiken
|
| 15 |
6 |
valentine-6 |
c05:rijf
|
| 16 |
6 |
generic-08 |
c26:gareel
|
| 17
|
| 18
|
| 19
|
| 20
|
| 21
|
| 22
|
| 23
|
| 24
|
| 25
|
| 26
|
| 27
|
| 28
|
| 29
|
| 30 |
8 |
(unknown) |
melkbus |
|
| 31 |
8 |
(undef) |
H140-patch-voip-burgers |
assigned-only
|
| 32 |
8 |
(undef) |
H140-patch-voip-ivocs |
assigned-only
|
| 33 |
8 |
HA-1 |
c23:ullr
|
| 34 |
8 |
0729-10 |
c18:kvasir
|
| 35 |
8 |
6109-1 |
windmolen
|
| 36 |
7 |
NIK1208 |
c18:kot
|
| 37 |
|
|
|
|
| 38 |
|
|
|
|
| 39 |
|
|
|
|
| 40 |
|
|
|
|
| 41 |
|
|
|
|
| 42 |
|
|
|
|
| 43 |
|
|
|
|
| 44 |
|
|
|
|
| 45 |
|
|
|
|
| 46 |
|
|
|
|
| 47 |
|
|
|
|
| 48 |
|
|
|
|
NikOPN
Module 1 (rx-bi-10g-4-port)
Module 1 (rx-bi-10g-4-port)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
13 |
NIK2109 |
SARA via H140
|
| 2 |
12 |
K80501617 |
c14:deel |
|
| 3
|
| 4
|
Module 2 (rx-bi-1g-48-port-copper)
Module 2 (rx-bi-1g-48-port-copper)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
14 |
6 ("six") |
c15:hooiwagen
|
| 2
|
| 3
|
| 4
|
| 5
|
| 6
|
| 7 |
14 |
|
c28:hooi-ei-01
|
| 8 |
14 |
|
c28:hooikoorts
|
| 9 |
14 |
|
c28:hooi-ei-03
|
| 10 |
14 |
|
c28:hooi-ei-06
|
| 11 |
14 |
|
c28:hooi-ei-04
|
| 12 |
14 |
|
c28:hooi-ei-01
|
| 13 |
14 |
|
c28:hooi-ei-03
|
| 14 |
14 |
[45] |
c16:hooizolder
|
| 15 |
14 |
|
|
| 16 |
14 |
garitxako |
c16:garitxako
|
| 17 |
14 |
0729-08 |
c18:hooivork
|
| 18 |
14 |
NIK1442 |
c16:hooibroei
|
| 19 |
14 |
10 ("ten") |
c15:hooikuil
|
| 20 |
14 |
HA-2 |
c23:hooibaal |
hosting tbn18
|
| 21 |
14 |
NIK1209 |
c18:hooikist
|
| 22 |
14 |
NIK1210 |
c18:hooischelf
|
| 23
|
| 24
|
| 25
|
| 26
|
| 27
|
| 28
|
| 29
|
| 30
|
| 31
|
| 32
|
| 33
|
| 34
|
| 35
|
| 36
|
| 37
|
| 38
|
| 39
|
| 40
|
| 41
|
| 42
|
| 43
|
| 44
|
| 45
|
| 46
|
| 47
|
| 48
|
Module 7 (rx-bi-10g-4-port)
Module 7 (rx-bi-10g-4-port)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
14 |
17040 |
c28:opnstorage-sw-easteregg-01 |
(check cable no!)
|
| 2
|
| 3
|
| 4
|
sw-public-grid-01
Public Grid (sw-public-grid-01, cab 14)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
3 |
generic-01 |
c27:dissel
|
| 2 |
3 |
generic-02 |
c27:bats
|
| 3 |
3 |
generic-03 |
c27:mesthoop
|
| 4 |
3 |
generic-04 |
c27:silo
|
| 5 |
3 |
generic-05 |
c27:moestuin
|
| 6 |
3 |
generic-10 |
c26:appelvanger
|
| 7 |
3 |
1 (c15) |
c15:trog
|
| 8 |
3 |
2 (c15) |
c15:stal
|
| 9 |
3 |
3 (c15/BL0/SL16) |
c15:zadel
|
| 10 |
3 |
4 (c15) |
c15:boswachter
|
| 11 |
3 |
5 (c15) |
c15:bosheks
|
| 12 |
3 |
8 (c15) |
c15:dorsvloer
|
| 13 |
3 |
9 (c15) |
c15:hooimijt
|
| 14
|
| 15
|
| 16
|
| 17
|
| 18
|
| 19
|
| 20
|
| 21
|
| 22 |
3 |
#4 |
H140 patchpanels |
H1.169 (H157)
|
| 23 |
T3,4 |
1 (c14) |
c14:deel |
trunk 24
|
| 24 |
T3,4 |
2 (c14) |
c14:deel |
trunk 23
|
sw-public-sec-01
Public Sec (sw-public-sec-01, cab 14)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1
|
| 2
|
| 3
|
| 4
|
| 5
|
| 6
|
| 7
|
| 8
|
| 9
|
| 10
|
| 11
|
| 12
|
| 13
|
| 14
|
| 15
|
| 16
|
| 17
|
| 18
|
| 19
|
| 20
|
| 21
|
| 22
|
| 23 |
T3,4 |
26 (c14) |
c14:deel |
trunk 24
|
| 24 |
T3,4 |
27 (c14) |
c14:deel |
trunk 23
|
sw-mngt-01
Net and IPMI Management switch (sw-mngt-01, cab 14)
| Port |
VLANs |
Cable |
Destination |
Comments
|
| 1 |
1 |
0814-3 |
c14:salado
|
| 2 |
1 |
black |
c14:deel-man
|
| 3
|
| 4
|
| 5 |
4 |
47 |
c18:toom-ipmi
|
| 6 |
4 |
49 |
c18:kudde-ipmi
|
| 7 |
4 |
red "crossed" |
c28:ipmi-switch
|
| 8 |
4 |
valentine-10 |
c6:linksys
|
| 9 |
4 |
101-boven |
c21:ipmi
|
| 10 |
4 |
102-boven |
c20:ipmi
|
| 11 |
4 |
103-boven |
c22:ipmi
|
| 12 |
4 |
104-boven |
c23:ipmi
|
| 13 |
1 |
20 |
c14:nikopn-man
|
| 14 |
1 |
? |
c14:equinoxELS TS
|
| 15
|
| 16
|
| 17 |
4 |
NIK1211 |
c18:drcs-ruif
|
| 18
|
| 19 |
4 |
NIK1441 |
c16:ipmi
|
| 20
|
| 21
|
| 22
|
| 23
|
| 24
|
| 25 |
4 |
?? |
c14:deel
|
| 26
|
