Difference between revisions of "HvABigDataVisualisation"
From PDP/Grid Wiki
Jump to navigationJump to search| (6 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
This data is used not only here, but by over 20000 researchers worldwide, who transfer the data back and forth between the data curation centres, the more than 300 compute centres, and thoudans of small analysis workstations and desktops. Data flows globally, and even just at Nikhef compute and disk clusters are interconnected by 240 gigabit-per-second links, and international connectivity exceeds 100 gigabit-per-second. | This data is used not only here, but by over 20000 researchers worldwide, who transfer the data back and forth between the data curation centres, the more than 300 compute centres, and thoudans of small analysis workstations and desktops. Data flows globally, and even just at Nikhef compute and disk clusters are interconnected by 240 gigabit-per-second links, and international connectivity exceeds 100 gigabit-per-second. | ||
| + | |||
| + | = More questions than answers? = | ||
| + | |||
| + | The Phase-II visualisation challenge leaves you with plenty of things to try out. Use your creativity to visualise, explain and analyse the data: big data lives by propaganda (and agitation)! | ||
| + | |||
| + | * How can global data flows be presented? | ||
| + | * Can one conceive visualisations for the general public? for users? or for both? | ||
| + | * Identifying troublesome inter-peer links (and local disk servers) via analytics techniques | ||
| + | |||
| + | |||
| + | |||
| + | * Which systems (or group of systems) uses the most bandwidth (in and out separately) | ||
| + | * Which systems (or group of systems) generates the most connections? | ||
| + | |||
| + | * What does the time distribution of transfers look like (bandwidth and also number)? | ||
| + | * What "funny behavior" is there (machine learning anomaly detection) | ||
| + | |||
| + | but there's surely more to do with the data you have! | ||
= About the data analytics cluster = | = About the data analytics cluster = | ||
| + | [[File:RTM-2D-busy-2007.jpg|200px|thumb|right|Jobs distribution across EGEE and WLCG]] | ||
| Line 15: | Line 34: | ||
The TI student team in 2015 set up the ELK cluster and has taken care of a continuous stream of fresh data being inserted in the database. At any time there should be about 1 month worth of historical detailed data in the database, and it is updated in near-real-time. | The TI student team in 2015 set up the ELK cluster and has taken care of a continuous stream of fresh data being inserted in the database. At any time there should be about 1 month worth of historical detailed data in the database, and it is updated in near-real-time. | ||
| − | An ElasticSearch (ES) interface and API is accessible (through an application proxy) on a dedicated system: "vm6.stud1.ipmi.nikhef.nl" (172.23.1.16) on port | + | An ElasticSearch (ES) interface and API is accessible (through an application proxy) on a dedicated system: "vm6.stud1.ipmi.nikhef.nl" (172.23.1.16) on port 9200/tcp. |
Details of the API and how to use the interface (alongside some example queries) are available in your course documentation pack, kindly prepared by Jouke, Olivier, and Rens. We expect clients to talk to the ElasticSearch API via the network. The client should run on your local system (own laptop, desktop, HvA VDI system, &c), and | Details of the API and how to use the interface (alongside some example queries) are available in your course documentation pack, kindly prepared by Jouke, Olivier, and Rens. We expect clients to talk to the ElasticSearch API via the network. The client should run on your local system (own laptop, desktop, HvA VDI system, &c), and | ||
connect to the system via the public API only. | connect to the system via the public API only. | ||
| Line 49: | Line 68: | ||
comp-lzo | comp-lzo | ||
verb 3 | verb 3 | ||
| + | cipher aes-256-cbc | ||
#auth-user-pass ../keys/secret-schaapscheerder.conf | #auth-user-pass ../keys/secret-schaapscheerder.conf | ||
| Line 76: | Line 96: | ||
-----END CERTIFICATE----- | -----END CERTIFICATE----- | ||
| − | and if you want auto-login, *only on your own laptop*, make a secure, protected file "auth-user-pass ../keys/secret-schaapscheerder.conf" and uncomment the auth-user-pass line above. The file "auth-user-pass ../keys/ | + | and if you want auto-login, *only on your own laptop*, make a secure, protected file "auth-user-pass ../keys/secret-schaapscheerder.conf" and uncomment the auth-user-pass line above. The file "auth-user-pass ../keys/secret-schaapscheerder.conf" should contain something like |
nvahva16x2342 | nvahva16x2342 | ||
