Arbitrary grouping and sorting in Events
Comes from the CTI project - we will need to support arbitrary GROUP BY and SORT. We know this can be unexpected performance hog, so let's start with implementing it in a trivial way - add new subsection (along with Event, Detector, Admin, ...) like "Postprocess", and add there possibility to choose grouping columns and sorting column, which will translate to straightforward GROUP BY and ORDER BY. We'll then explore the limits (and maybe enforce some). Also, big fat warning should be added there that query time can be unpredictable.
Activity
assigned to @Jan_Mach
By Pavel Kácha on 2024-12-26T14:06:36
changed milestone to %Backlog
I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.
By Radko Krkoš on 2019-01-30T15:16:41
-
@radko_krkos wrote:
I would advise against altering the Event search view and instead for introducing a new "Analytics" view capable of such arbitrary queries. From experience the users (such as UPJŠ Košice with timelines) are more accepting of a new feature that is not optimized yet, than breaking existing stuff. Also the proposed output grouping is in my opinion not really useful for event search.
My thought behind that was similar in that we can hide that "Postprocess" or "Analytics" button for admins, such as "Admins" button, until we're ready to publish it.
Pros/cons:
- additional button in Alerts means we would have to make result table display more flexible (because columns from grouping can be very different, and also we can have different sort). So I see what you mean by possibility of breaking existing stuff.
- Different view means duplication of pretty much all of Alerts query view plus adding some more.
Not sure which is better, maybe you're right for the beginning and we'll see if it can be integrated better somewhere.
Thinking about that, maybe it overlaps in funcionality partially with Timeline view - Timeline view is in fact very simple query grouped by various attributes (Counts, #abuses, #analysers, ...), sorted by resulting counts and shown on both timeline and piechart.
I've asked Martin for usecase CTI document to have more tangible image of what the customer actually wants, I'll post relevant excerpt here as soon as I have it.
By Pavel Kácha on 2019-01-31T09:59:54
-
Excerpt from CTI customer requirements (doc in czech):
Analýza
Kategorie:Proaktivita
Případ užití: CTI bude podporovat analýzu událostí uchovávaných v systému pomocí dotazování se na uchovávané události. Dotazy v sobě budou typicky zahrnovat výběr časového okna dotazu, filtr, agregaci, řazení (top-n) dle relevantních polí události.
Příklad uživatelského scénáře: GovCERT.CZ zaznamená zvýšený počet událostí typu hádání hesel. Analytik se dotazuje systému, aby odhalil, zda toto zvýšení koreluje s výskytem například nového malware.
By Pavel Kácha on 2019-02-01T09:52:31
-
From 2019-02-07 meeting: We should be able to do "analytics" from the same set of conditions as searching for Alerts. If we go for separate top level "tab", we will lose the possibility to easily run the same query for both alerts search and analytics - or we'll have to allow some user-conscious propagation of the query form between alerts search and analytics.
Not resolved on the meeting, will need some more discussion.
By Pavel Kácha on 2019-02-11T15:07:17
-
Forgotten update from live meeting:
- let's start with widening timeline form by using form from Alerts search template
- generate the same timeline info from obtained data in the same way
- add navigation links Alerts -> Timeline and Timeline -> Alerts to allow obtaining the same data in both full and aggregated way
By Pavel Kácha on 2019-05-14T09:34:01
changed milestone to %2.7
added Done 25% S-In Progress labels and removed Done 0% S-New labels
added S-Closed label and removed S-In Progress label
marked this issue as related to #4384 (closed)
By Pavel Kácha on 2024-12-26T14:16:58