Difference between revisions of "NetworkDeelConnections"
m |
|||
| Line 74: | Line 74: | ||
== Setting and applying ACLs == | == Setting and applying ACLs == | ||
| + | |||
| + | The ACLs on deel and nikopn can only be ''inbound'', and should eb applied to all physical and virtual interfaces in a consistent way in order to be effective. Since deel contains many different kinds of networks, it is logically easier to think of the protected to be applied to the ''outbound'' direction, i.e. what kind of traffic you allow to flow towards a particular subnet, instead of thinkging aboutn the traffic allowed inbound. However, the outbound ACLs you would need to implement the logical idea are not supported on the BigIron-RX series hardware. Similarly, access-policy-groups cannot be used. | ||
| + | |||
| + | To offset this limitation, a translation utility ([https://www.nikhef.nl/grid/ndpf/files/packages/foundry-tracl/ <tt>tr-acl</tt>]) has been built to convert logical access control rules in a set of inbound-only ACLs to be used on the foundry. It converts a high-level description of all connected subnets and the list of access controls to be applied in a set of inbound ALCs, one for each subnet. To work in this way, you: | ||
| + | |||
| + | # write or update the high-level description. The one used for deel and nikopn is in the private SVN "ndpf" repository (<tt>ndpf/nl.nikhef.ndpf.config/router/ruleset.in</tt> or <tt>ruleset-nikopn.in</tt>). Make sure to commit and log changes | ||
| + | # run the tr-acl command: <tt>tr-acl ruleset.in > /tftpboot/deel-acl</tt> | ||
| + | # load the new ACLs in the router: <tt>cop tftp run 192.168.254.4 deel-acl</tt> | ||
| + | # apply them to the interfaces: <tt>conf t</tt> and <tt>ip rebind-acl all</tt> | ||
| + | # check to make sure that the CAM has not been exhausted and all ACLs actually have been applied. There should be '''no''' errors if you give the command: <tt>sho log</tt> | ||
| + | |||
| + | === Example ruleset === | ||
| + | |||
| + | The ruleset language looks an awful lot like the foundry syntax, but defines a couple of new constructs like subnetwork definitions, fixed pre- and post-fixes, and simple (non-recursive!) macros. For example | ||
| + | |||
| + | ! ruleset for new deel router | ||
| + | ! | ||
| + | interface interconnect | ||
| + | connects 192.16.186.164/30 | ||
| + | connects 0.0.0.0/0 | ||
| + | excludes 192.16.186.192/26 | ||
| + | excludes 192.16.186.128/30 | ||
| + | excludes 194.171.96.0/22 | ||
| + | excludes 145.100.9.44/30 | ||
| + | excludes 172.16.0.0/12 | ||
| + | excludes 239.0.0.0/8 | ||
| + | excludes 0.0.0.0/32 | ||
| + | ! ban forged source IPs from entering | ||
| + | ! do this once we configure IPs on our own VLANs | ||
| + | prepend deny ip 194.171.96.0/21 any | ||
| + | prepend deny ip 192.16.186.192/26 any | ||
| + | end | ||
| + | ! macro definitions | ||
| + | stanza permit-webserver | ||
| + | permit tcp $src $dest eq http | ||
| + | permit tcp $src $dest eq ssl | ||
| + | end | ||
| + | ! | ||
| + | ! actual rules start here | ||
| + | ruleset | ||
| + | ! always allow established connections and filter localdomain | ||
| + | permit tcp any any established | ||
| + | deny ip 127.0.0.0/8 any | ||
| + | deny ip any 127.0.0.0/8 | ||
| + | ! | ||
| + | ! The P4 CTB hosts 10/8 space we dont want to leak out | ||
| + | deny ip 10.0.0.0/8 any | ||
| + | ! HTTP and SSL servers on gridsrv are accessible from anywhere | ||
| + | $permit-webserver(src=any,dest=194.171.96.64/28) | ||
| + | ! | ||
| + | ! | ||
| + | end | ||
== Deel == | == Deel == | ||
Revision as of 07:06, 28 April 2009
Network Connection in the NDPF
The configuration files and the source to the generated ACLs is in subversion at https://ndpfsvn.nikhef.nl/repos/ndpf/nl.nikhef.ndpf.config/routers/. Use your farm username and password (if you're authorized to begin with, i.e. are a member of the NDPFAdministrators directory group to get access, or check out from SVN using ssh.
To view actual bandwidth usage, Cricket graphs are available at http://www.dutchgrid.nl/ndpf/cricket/grapher.cgi (or on salado, the network management host, if you are within the Nikhef domain).
The high-level overview was drawn by Tristan at Nikhef network.
VLAN Identification and network naming
Colour coding: red is in use on deel, green is in use on nikgrid, and thus yellow is in use on both ...
| VLAN ID | logicalName | Description |
| 2 | p4ctb | Limited Access VL-e and BiGGrid P4 Certification Test Bed |
| 3 | public-comb | NDPF production core and grid services combined |
| 4 | ipmi | Local management network |
| 5 | interconnect | Interconnect network to hef-router |
| 6 | gridsrv | Service box network |
| 7 | vobox | Class-1 VObox network |
| 8 | nordic | Experimental Open Net |
| 9 | farmnet | Worker nodes |
| 10 | opninterconnect | Interconnect network to nikopn |
| 11 | (rofcluster) | Reserved |
| 12 | deelinterconnect | Interconnect network to deel |
| 13 | opnuplink | LHCOPN up link to SARA |
| 14 | opnstorage | Storage servers |
Subnet allocations
| Prefix | Name | Description |
| 192.16.186.192/26 | public-sec | Public NDPF network with services (core part) |
| 194.171.96.0/24 | (misc) | several special-purpose networks, please see NDPF Node Functions |
| 194.171.97.0/24 | public-grid | Public NDPF network with services (grid and ui part) |
| 194.171.98.0/23 | farmnet | Worker nodes |
| 194.171.100.0/22 | (new) | (new) |
| 2001:0610:0120::/48 | IKONET-IPv6 | Nikhef |
Network Management System
A dedicated system is used to control the routers and the management switch. It has three ethe interfaces: eth0 connects it to the Nikhef guestnet (as salado.nikhef.nl) and should always be reachable, even if the deel and nikopn routers themselves are messed up. It's second interface (eth1) is connected to deel as a tagged interconnect, and links it to the public-sec network (as schoffel.nikhef.nl) as well as the ipmi network (as salado.ipmi.nikhef.nl, i.e., 172.20.1.255). The third interface actually connects the management blades of deel and nikopn, and can be used to control sw-mngt-01 (the 10/100 switch in c14). It has the address 192.168.254.4/24, where on that network deel is 192.168.254.1, nikopn is 192.168.254.2, and sw-mngt-01 is 192.168.254.254. Logging from this box is sent to boes.nikhef.nl, whose IP address is statically configured in /etc/hosts. Note that boes MUST be on a directly connected network to remain secure. It also has the MAC address of boes hardwires in /etc/ethers.
A copy of the cricket installation, configuration and data as of Oct 24, 2008, is available at (hefnet):/global/ices/grid/nikhef/network.
Setting and applying ACLs
The ACLs on deel and nikopn can only be inbound, and should eb applied to all physical and virtual interfaces in a consistent way in order to be effective. Since deel contains many different kinds of networks, it is logically easier to think of the protected to be applied to the outbound direction, i.e. what kind of traffic you allow to flow towards a particular subnet, instead of thinkging aboutn the traffic allowed inbound. However, the outbound ACLs you would need to implement the logical idea are not supported on the BigIron-RX series hardware. Similarly, access-policy-groups cannot be used.
To offset this limitation, a translation utility (tr-acl) has been built to convert logical access control rules in a set of inbound-only ACLs to be used on the foundry. It converts a high-level description of all connected subnets and the list of access controls to be applied in a set of inbound ALCs, one for each subnet. To work in this way, you:
- write or update the high-level description. The one used for deel and nikopn is in the private SVN "ndpf" repository (ndpf/nl.nikhef.ndpf.config/router/ruleset.in or ruleset-nikopn.in). Make sure to commit and log changes
- run the tr-acl command: tr-acl ruleset.in > /tftpboot/deel-acl
- load the new ACLs in the router: cop tftp run 192.168.254.4 deel-acl
- apply them to the interfaces: conf t and ip rebind-acl all
- check to make sure that the CAM has not been exhausted and all ACLs actually have been applied. There should be no errors if you give the command: sho log
Example ruleset
The ruleset language looks an awful lot like the foundry syntax, but defines a couple of new constructs like subnetwork definitions, fixed pre- and post-fixes, and simple (non-recursive!) macros. For example
! ruleset for new deel router ! interface interconnect connects 192.16.186.164/30 connects 0.0.0.0/0 excludes 192.16.186.192/26 excludes 192.16.186.128/30 excludes 194.171.96.0/22 excludes 145.100.9.44/30 excludes 172.16.0.0/12 excludes 239.0.0.0/8 excludes 0.0.0.0/32 ! ban forged source IPs from entering ! do this once we configure IPs on our own VLANs prepend deny ip 194.171.96.0/21 any prepend deny ip 192.16.186.192/26 any end ! macro definitions stanza permit-webserver permit tcp $src $dest eq http permit tcp $src $dest eq ssl end ! ! actual rules start here ruleset ! always allow established connections and filter localdomain permit tcp any any established deny ip 127.0.0.0/8 any deny ip any 127.0.0.0/8 ! ! The P4 CTB hosts 10/8 space we dont want to leak out deny ip 10.0.0.0/8 any ! HTTP and SSL servers on gridsrv are accessible from anywhere $permit-webserver(src=any,dest=194.171.96.64/28) ! ! end
Deel
Module 3 (rx-bi-10g-4-port)
| Port | VLANs | Cable | Destination | Comments |
| 1 | ||||
| 2 | ||||
| 3 | 10 | K80501617 | nikopn | |
| 4 |
Module 6 (rx-bi-1g-24-port-copper)
ports 13-24 reserved for the public-sec to-be VLAN
| Port | VLANs | Cable | Destination | Comments |
| 1 | 5 | HEF21/2-19 | hef-router | trunk 6/2 |
| 2 | 5 | Cross 51 | hef-router | trunk 6/1 |
| 3 | T3,4 | 26 | c14: sw-public-sec-01 | trunk 6/4 |
| 4 | T3,4 | 27 | c14: sw-public-sec-01 | trunk 6/3 |
| 5 | ||||
| 6 | ||||
| 7 | ||||
| 8 | ||||
| 9 | ||||
| 10 | ||||
| 11 | ||||
| 12 | ||||
| 13 | ||||
| 14 | ||||
| 15 | ||||
| 16 | ||||
| 17 | ||||
| 18 | ||||
| 19 | ||||
| 20 | ||||
| 21 | ||||
| 22 | ||||
| 23 | ||||
| 24 |
Module 7 (rx-bi-1g-24-port-fiber)
| Port | VLANs | Cable | Destination | Comments |
| 1 | ||||
| 2 | ||||
| 3 | 9 | NIK1354 | c26:sw-luilak2-01 | |
| 4 | 9 | NIK1319 | c27:sw-luilak2-02 | |
| 5 | 9 | NIK1246 | c23:sw-luilak1-01 | |
| 6 | 9 | NIK1247 | c23:sw-luilak1-02 | |
| 7 | ||||
| 8 | ||||
| 9 | ||||
| 10 | ||||
| 11 | ||||
| 12 | ||||
| 13 | 3 | 080402/009/010 | c18:sw-public-comb | |
| 14 | ||||
| 15 | ||||
| 16 | ||||
| 17 | ||||
| 18 | ||||
| 19 | ||||
| 20 | ||||
| 21 | ||||
| 22 | ||||
| 23 | 1350nm | |||
| 24 | 1350nm |
Module 10 (rx-bi-1g-24-port-copper)
| Port | VLANs | Cable | Destination | Comments |
| 1 | T3,4 | 1 | c14:sw-public-grid-01 | trunk 10/2 |
| 2 | T3,4 | 2 | c14:sw-public-grid-01 | trunk 10/1 |
| 3 | ||||
| 4 | ||||
| 5 | ||||
| 6 | 4 | 14 | c14:sw-mngt-01 | |
| 7 | T3,4 | NDPF-0814-2 | c14:netmanager (salado) | |
| 8 | ||||
| 9 | 3 | NIK1440 | c16:bedstee | |
| 10 | ||||
| 11 | 4 | (blue) | c14:nikopn-rtr | mngt net extension |
| 12 | 4 | F0 | c22:terp (DRCS) | |
| 13 | ||||
| 14 | ||||
| 15 | ||||
| 16 | ||||
| 17 | 6 | #200 | c02:hek | |
| 18 | 6 | #202 | c02:gierput | |
| 19 | 6 | #203 | c02:kaasvat | |
| 20 | 6 | HA-6 | c23:keerder | |
| 21 | ||||
| 22 | ||||
| 23 | ||||
| 24 |
Module 13 (rx-bi-1g-24-port-copper)
| Port | VLANs | Cable | Destination | Comments |
| 1 | 9 | F1 | c21:sw-bulldozer-01 | |
| 2 | 9 | ??: farmnet-vulnassess | ||
| 3 | ||||
| 4 | ||||
| 5 | ||||
| 6 | ||||
| 7 | 9 | F2 | c21:sw-bulldozer-02 | |
| 8 | 9 | valentine-5 | c07:sw-valentine-01 | |
| 9 | 9 | valentine-7 | c06:sw-valentine-02 | |
| 10 | 9 | valentine-8 | c04:sw-valentine-03 | |
| 11 | ||||
| 12 | ||||
| 13 | 2 | generic-07 | c26:span | |
| 14 | 8 | 7 (seven) | c15:melktank | |
| 15 | ||||
| 16 | 2 | (unknown) | c1:speeltuin hall/BI15k | |
| 17 | 2 | 48 | kudde | |
| 18 | 2 | 46 | toom | |
| 19 | 8 | NIK1212 | virtualschoollab | |
| 20 | ||||
| 21 | 8 | [43] | arrone | |
| 22 | 8 | [44] | aulnes | |
| 23 | ||||
| 24 | 2 | HA-4 | c23:bleek |
Module 16 (rx-bi-1g-48-port-copper)
Do not use PPRC4 (ports 37-48) until a firmware fix has been provided by FN
| Port | VLANs | Cable | Destination | Comments |
| 1 | 3 | F3 | c22:dorsvlegel | |
| 2 | 3 | F4 | c22:tbn15 | |
| 3 | 3 | 43 (dupl) | c16:bedstee (alternate) | |
| 4 | 3 | [41] | c15:erf | |
| 5 | 3 | [42] | c15:hooiberg-1 | |
| 6 | 3 | [40] | c15:hooiberg-2 | |
| 7 | 3 | HA-0 | c23:hilde | |
| 8 | 3 | HA-3 | c23:kaf | |
| 9 | 3 | HA-5 | c23:kribbe | |
| 10 | 3 | HA-7 | c23:boszwijn | |
| 11 | 3 | HA-8 | c23:schuur | |
| 12 | 3 | HA-9 | c23:hoeve | |
| 13 | 6 | #201 | c02:beerput | |
| 14 | 6 | 0729-09 | c17:kuiken | |
| 15 | 6 | valentine-6 | c05:rijf | |
| 16 | 6 | generic-08 | c26:gareel | |
| 17 | ||||
| 18 | ||||
| 19 | ||||
| 20 | ||||
| 21 | ||||
| 22 | ||||
| 23 | ||||
| 24 | ||||
| 25 | ||||
| 26 | ||||
| 27 | ||||
| 28 | ||||
| 29 | ||||
| 30 | 8 | (unknown) | melkbus | |
| 31 | 8 | (undef) | H140-patch-voip-burgers | assigned-only |
| 32 | 8 | (undef) | H140-patch-voip-ivocs | assigned-only |
| 33 | 8 | HA-1 | c23:ullr | |
| 34 | 8 | 0729-10 | c18:kvasir | |
| 35 | 8 | 6109-1 | windmolen | |
| 36 | 7 | NIK1208 | c18:kot | |
| 37 | ||||
| 38 | ||||
| 39 | ||||
| 40 | ||||
| 41 | ||||
| 42 | ||||
| 43 | ||||
| 44 | ||||
| 45 | ||||
| 46 | ||||
| 47 | ||||
| 48 |
NikOPN
Module 1 (rx-bi-10g-4-port)
| Port | VLANs | Cable | Destination | Comments |
| 1 | 13 | NIK2109 | SARA via H140 | |
| 2 | 12 | K80501617 | c14:deel | |
| 3 | ||||
| 4 |
Module 2 (rx-bi-1g-48-port-copper)
| Port | VLANs | Cable | Destination | Comments |
| 1 | 14 | 6 ("six") | c15:hooiwagen | |
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 | ||||
| 6 | ||||
| 7 | 14 | c28:hooi-ei-01 | ||
| 8 | 14 | c28:hooikoorts | ||
| 9 | 14 | c28:hooi-ei-03 | ||
| 10 | 14 | c28:hooi-ei-06 | ||
| 11 | 14 | c28:hooi-ei-04 | ||
| 12 | 14 | c28:hooi-ei-01 | ||
| 13 | 14 | c28:hooi-ei-03 | ||
| 14 | 14 | [45] | c16:hooizolder | |
| 15 | 14 | |||
| 16 | 14 | garitxako | c16:garitxako | |
| 17 | 14 | 0729-08 | c18:hooivork | |
| 18 | 14 | NIK1442 | c16:hooibroei | |
| 19 | 14 | 10 ("ten") | c15:hooikuil | |
| 20 | 14 | HA-2 | c23:hooibaal | hosting tbn18 |
| 21 | 14 | NIK1209 | c18:hooikist | |
| 22 | 14 | NIK1210 | c18:hooischelf | |
| 23 | ||||
| 24 | ||||
| 25 | ||||
| 26 | ||||
| 27 | ||||
| 28 | ||||
| 29 | ||||
| 30 | ||||
| 31 | ||||
| 32 | ||||
| 33 | ||||
| 34 | ||||
| 35 | ||||
| 36 | ||||
| 37 | ||||
| 38 | ||||
| 39 | ||||
| 40 | ||||
| 41 | ||||
| 42 | ||||
| 43 | ||||
| 44 | ||||
| 45 | ||||
| 46 | ||||
| 47 | ||||
| 48 |
Module 7 (rx-bi-10g-4-port)
| Port | VLANs | Cable | Destination | Comments |
| 1 | 14 | 17040 | c28:opnstorage-sw-easteregg-01 | (check cable no!) |
| 2 | ||||
| 3 | ||||
| 4 |
sw-public-grid-01
| Port | VLANs | Cable | Destination | Comments |
| 1 | 3 | generic-01 | c27:dissel | |
| 2 | 3 | generic-02 | c27:bats | |
| 3 | 3 | generic-03 | c27:mesthoop | |
| 4 | 3 | generic-04 | c27:silo | |
| 5 | 3 | generic-05 | c27:moestuin | |
| 6 | 3 | generic-10 | c26:appelvanger | |
| 7 | 3 | 1 (c15) | c15:trog | |
| 8 | 3 | 2 (c15) | c15:stal | |
| 9 | 3 | 3 (c15/BL0/SL16) | c15:zadel | |
| 10 | 3 | 4 (c15) | c15:boswachter | |
| 11 | 3 | 5 (c15) | c15:bosheks | |
| 12 | 3 | 8 (c15) | c15:dorsvloer | |
| 13 | 3 | 9 (c15) | c15:hooimijt | |
| 14 | ||||
| 15 | ||||
| 16 | ||||
| 17 | ||||
| 18 | ||||
| 19 | ||||
| 20 | ||||
| 21 | ||||
| 22 | 3 | #4 | H140 patchpanels | H1.169 (H157) |
| 23 | T3,4 | 1 (c14) | c14:deel | trunk 24 |
| 24 | T3,4 | 2 (c14) | c14:deel | trunk 23 |
sw-public-sec-01
| Port | VLANs | Cable | Destination | Comments |
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 | ||||
| 6 | ||||
| 7 | ||||
| 8 | ||||
| 9 | ||||
| 10 | ||||
| 11 | ||||
| 12 | ||||
| 13 | ||||
| 14 | ||||
| 15 | ||||
| 16 | ||||
| 17 | ||||
| 18 | ||||
| 19 | ||||
| 20 | ||||
| 21 | ||||
| 22 | ||||
| 23 | T3,4 | 26 (c14) | c14:deel | trunk 24 |
| 24 | T3,4 | 27 (c14) | c14:deel | trunk 23 |
sw-mngt-01
| Port | VLANs | Cable | Destination | Comments |
| 1 | 1 | 0814-3 | c14:salado | |
| 2 | 1 | black | c14:deel-man | |
| 3 | ||||
| 4 | ||||
| 5 | 4 | 47 | c18:toom-ipmi | |
| 6 | 4 | 49 | c18:kudde-ipmi | |
| 7 | 4 | red "crossed" | c28:ipmi-switch | |
| 8 | 4 | valentine-10 | c6:linksys | |
| 9 | 4 | 101-boven | c21:ipmi | |
| 10 | 4 | 102-boven | c20:ipmi | |
| 11 | 4 | 103-boven | c22:ipmi | |
| 12 | 4 | 104-boven | c23:ipmi | |
| 13 | 1 | 20 | c14:nikopn-man | |
| 14 | 1 | ? | c14:equinoxELS TS | |
| 15 | ||||
| 16 | ||||
| 17 | 4 | NIK1211 | c18:drcs-ruif | |
| 18 | ||||
| 19 | 4 | NIK1441 | c16:ipmi | |
| 20 | ||||
| 21 | ||||
| 22 | ||||
| 23 | ||||
| 24 | ||||
| 25 | 4 | ?? | c14:deel | |
| 26 |
