Rename _CESNET key to _Mentat (or make it configurable)

Historically we save Mentat specific keys (abuse contacts, classes, enrichments, ...) within _CESNET namespace in Idea. That does not make sense for instances running in different organisations. We should thus rename the key (potentially to _Mentat), or even make the name configurable.

assigned to @rajmund_hruska

By Pavel Kácha on 2024-12-26T14:07:07

added Done 0% P4 S-New T-Feature labels

changed milestone to %2.8

Back Relation 'relates' from: 6799

Back Relation 'relates' from: 4231

Relation 'blocks' to: 6949

By Pavel Kácha on 2020-11-19T10:59:05

Relation Relates - Feature 4231 Implement and make use of cleanup mechanism for "_" namespaces from IDEA messages.

By Pavel Kácha on 2020-11-19T11:03:39

added S-In Progress label and removed S-New label

Will need (quite long) transition period, as we have Idea events, which are quite old (over year for CESNET networks).
Correct way would be to allow for "envelope" for Idea events, passing through the system, with metadata fields, however that would need quite a redesign. So we'll stay with current model, "injecting" our own fields into suitable place within Idea. Let's start with simply renaming to _Mentat, we might move some fields into more "standardized" positions later.

However it would be nice to also rename database fields, let's drop the "cesnet_" prefix in column names altogether to prevent another possible renames in the future.

As for transition/compatibility, the aim is to start adding fields into new (_Mentat prefixed) names in Enricher, Inspector and Storage, however consider both namespaces (_CESNET and _Mentat) when reading. That could be implemented in various ways (everywhere on use, translation in database layer, replacement by generated fields from metadata table, some combination thereof).

By Pavel Kácha on 2020-12-03T09:07:06

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

By Rajmund Hruška on 2020-12-14T09:16:05

assigned to @ph and unassigned @rajmund_hruska

By Rajmund Hruška on 2024-12-26T14:22:09

added S-Feedback label and removed S-In Progress label

added Done 75% label and removed Done 0% label

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

By Rajmund Hruška on 2020-12-15T12:36:15

added Done 50% label and removed Done 75% label

assigned to @rajmund_hruska and unassigned @ph

By Rajmund Hruška on 2024-12-26T14:22:10

added S-In Progress label and removed S-Feedback label

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/6800#note_200144:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

• If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:

[ST_SKEY_ABUSES,     '_CESNET.ResolvedAbuses',        KEY_UNKNOWN],
    [ST_SKEY_ABUSES,     '_Mentat.ResolvedAbuses',        KEY_UNKNOWN],

• Or make middle term multiple and modify statistical logic:

[ST_SKEY_ABUSES,     ('_CESNET.ResolvedAbuses', '_Mentat.ResolvedAbuses'),        KEY_UNKNOWN],

• Or - of course - hardwire somehow.

By Pavel Kácha on 2020-12-18T14:44:36

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/6800#note_200142:

I noticed that in pynspect, there are also _CESNET keys in the test files. Should those keys be renamed to _Mentat?

From live video: They are concerning only tests, only example Idea, and not probably really used anywhere. Would make sense to rename them or get rid of the altogether, but I'd do that after we're ready on our side.

By Pavel Kácha on 2020-12-18T14:46:26

From live video: Also, still to do is

• database schema change
• Alembic migration

By Pavel Kácha on 2020-12-18T14:47:24

Pavel Kácha wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/6800#note_200146:

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/6800#note_200144:

Also, in lib/mentat/stats/idea.py there is some work done with _CESNET (lines 99-103). In my previous commit, I just renamed the key but I am not sure whether it should be also compatible with the _CESNET or not.

From live video: I'm afraid it will have to be more clever - just adding new key would add two types of stats. Solutions (from the top of my head) to be considered:

• If LIST_AGGREGATIONS are considered linearly everywhere, solution might be just to add second line, like:

[...]

• Or make middle term multiple and modify statistical logic:

[...]

• Or - of course - hardwire somehow.

I decided to make tuples - 38233d03

By Rajmund Hruška on 2020-12-18T21:27:58

Pavel Kácha wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/6800#note_200148:

From live video: Also, still to do is

I think all is done now.

• database schema change

78149a32

• Alembic migration

cb23deef

By Rajmund Hruška on 2020-12-18T21:30:57

assigned to @ph and unassigned @rajmund_hruska

By Rajmund Hruška on 2024-12-26T14:22:12

added Done 100% S-Resolved labels and removed Done 50% S-In Progress labels

Relation Blocks - Config 6949 Merge inspector configs into one

By Pavel Kácha on 2021-02-05T09:03:36

added S-In Review label and removed S-Resolved label

Merged into devel.

By Jan Mach on 2021-02-19T08:26:55

added S-Closed label and removed S-In Review label

Seems there is still something fishy (500 on event search on mentat-alt), reopening.

By Pavel Kácha on 2021-02-19T10:03:05

assigned to @rajmund_hruska and unassigned @ph

By Pavel Kácha on 2024-12-26T14:22:14

added S-In Review label and removed S-Closed label

The problem was that the database wasn't migrated. When I ran migration manually by alembic upgrade head, the problem was resolved. Is database migration part of the build at alchemist? I can't find that quickly.

By Rajmund Hruška on 2021-02-22T13:15:04

During the meeting, we noticed that there are some indexes with the old name (they contain _cesnet). I updated the migration.

By Rajmund Hruška on 2021-05-26T08:23:34

added S-Closed label and removed S-In Review label

I noticed that there are still old index names in 'Upgrading PostgreSQL from 12.x to 13.x' section in the documentation so I updated the names.

By Rajmund Hruška on 2021-05-31T10:42:38

Reopening to not forget it on 2.8 release.

By Pavel Kácha on 2021-06-18T08:41:16

added S-In Progress label and removed S-Closed label

As it was discussed during the meeting on 2021-06-18, I added the note about upgrading to the latest version of Mentat prior to the sections "Upgrading PostgreSQL from 11.x to 12.x" and "Upgrading PostgreSQL from 12.x to 13.x".

By Rajmund Hruška on 2021-06-18T13:28:17

added S-Resolved label and removed S-In Progress label

Merged into devel.

By Rajmund Hruška on 2021-07-16T10:47:01

added S-In Review label and removed S-Resolved label

added S-Closed label and removed S-In Review label

closed

marked this issue as related to #4231 (closed)

By Rajmund Hruška on 2024-12-26T14:15:00

mentioned in issue #4231 (closed)

By Pavel Kácha on 2024-12-26T14:15:01