Difference between revisions of "HvABigDataVisualisation"
From PDP/Grid Wiki
Jump to navigationJump to search| (3 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
This data is used not only here, but by over 20000 researchers worldwide, who transfer the data back and forth between the data curation centres, the more than 300 compute centres, and thoudans of small analysis workstations and desktops. Data flows globally, and even just at Nikhef compute and disk clusters are interconnected by 240 gigabit-per-second links, and international connectivity exceeds 100 gigabit-per-second. | This data is used not only here, but by over 20000 researchers worldwide, who transfer the data back and forth between the data curation centres, the more than 300 compute centres, and thoudans of small analysis workstations and desktops. Data flows globally, and even just at Nikhef compute and disk clusters are interconnected by 240 gigabit-per-second links, and international connectivity exceeds 100 gigabit-per-second. | ||
| + | |||
| + | = More questions than answers? = | ||
| + | |||
| + | The Phase-II visualisation challenge leaves you with plenty of things to try out. Use your creativity to visualise, explain and analyse the data: big data lives by propaganda (and agitation)! | ||
| + | |||
| + | * How can global data flows be presented? | ||
| + | * Can one conceive visualisations for the general public? for users? or for both? | ||
| + | * Identifying troublesome inter-peer links (and local disk servers) via analytics techniques | ||
| + | * Identifying �hot� (interesting) data by analysing usage | ||
| + | * Pro-active security incident detection: are there users �behaving strangely�? | ||
| + | * What�s the way to explain WLCG data flows to the world? | ||
| + | * Which systems (or group of systems) uses the most bandwidth (in and out separately) | ||
| + | * Which systems (or group of systems) generates the most connections? | ||
| + | * What does the spectrum of transfers look like?� mostly big, mostly small, ... ? | ||
| + | * What does the time distribution of transfers look like (bandwidth and also number)? | ||
| + | * What "funny behavior" is there (machine learning anomaly detection) | ||
| + | |||
| + | but there's surely more to do with the data you have! | ||
= About the data analytics cluster = | = About the data analytics cluster = | ||
| Line 50: | Line 68: | ||
comp-lzo | comp-lzo | ||
verb 3 | verb 3 | ||
| + | cipher aes-256-cbc | ||
#auth-user-pass ../keys/secret-schaapscheerder.conf | #auth-user-pass ../keys/secret-schaapscheerder.conf | ||
| Line 77: | Line 96: | ||
-----END CERTIFICATE----- | -----END CERTIFICATE----- | ||
| − | and if you want auto-login, *only on your own laptop*, make a secure, protected file "auth-user-pass ../keys/secret-schaapscheerder.conf" and uncomment the auth-user-pass line above. The file "auth-user-pass ../keys/ | + | and if you want auto-login, *only on your own laptop*, make a secure, protected file "auth-user-pass ../keys/secret-schaapscheerder.conf" and uncomment the auth-user-pass line above. The file "auth-user-pass ../keys/secret-schaapscheerder.conf" should contain something like |
nvahva16x2342 | nvahva16x2342 | ||
