Implement more complex object data services

In task #5067 (closed) basic connection with NERD system based on links was implemented. It would be very useful to implement more object data services and use them in IDEA event detail view to automagically fettch additional data for objects like IP addressess. Following modules could be implemented.

• DNS resolver
• PassiveDNS resolver

assigned to @Jan_Mach

By Jan Mach on 2024-12-26T14:06:45

added C-Dev-GUI Done 100% P4 S-New T-Feature labels

changed milestone to %2.5

Work in progress can be revieved here:

https://mentat-alt.cesnet.cz/mentat/events/a86158e5-bfab-4633-bc5f-7099cbf71eb8/show
https://mentat-alt.cesnet.cz/mentat/dnsr/search?search=195.113.144.233&submit=Search
https://mentat-alt.cesnet.cz/mentat/pdnsr/search?search=195.113.144.233&submit=Search
https://mentat-alt.cesnet.cz/mentat/nerd/search?search=193.32.163.89&submit=Search

Relation 'relates' to: 5067

By Jan Mach on 2019-06-26T13:44:56

added S-In Progress label and removed S-New label

In case you want to see what is cooking;-)

By Jan Mach on 2019-06-26T13:45:42

assigned to @ph and unassigned @Jan_Mach

By Jan Mach on 2024-12-26T14:19:17

added S-Feedback label and removed S-In Progress label

Relation Relates - Feature 5067 Add link to NERD to IP address context menus

By Jan Mach on 2019-06-26T13:46:28

added Done 50% label and removed Done 100% label

Latest patch replaces information icon about reaching limit with clickable load button, that enables users fetch not yet fetched data on demand.

You may try it here:

https://mentat-alt.cesnet.cz/mentat/events/a86158e5-bfab-4633-bc5f-7099cbf71eb8/show

In case of weird behavior make sure to load fresh JavaScript files by opening developer console and reloading the page.

By Jan Mach on 2019-06-27T13:53:02

Very coool. Appreciate the work that went into it. Limit for number of items is per IP or per queried service? Also, some reasonable caching would make sense (shall I make new issue?)

By Pavel Kácha on 2019-06-28T08:45:11

assigned to @Jan_Mach and unassigned @ph

By Pavel Kácha on 2024-12-26T14:19:18

@ph wrote:

Very coool. Appreciate the work that went into it. Limit for number of items is per IP or per queried service?

Limit is set for (Source/Target).IP(4/6), current value is 10. Data services may register hooks for object types they are capable of handling (IP4/6,hostname,...). Additional data are then fetched from all registered services for that type. For example in case an IDEA message has 15 Source.IP4s, it will result in 10x5 API queries because there are 5 services registered to handle IP4s. If there are any Target.IP4, they will have separate limit counter. The idea behind this is to resolve some number for each object type, but not to overwhelm the service APIs.

Also, some reasonable caching would make sense (shall I make new issue?)

Yes, that is absolutely necessary before releasing it to production. There is no need for a new issue in my opinion.

By Jan Mach on 2019-06-28T10:34:59

@Jan_Mach wrote:

@ph wrote:

Very coool. Appreciate the work that went into it. Limit for number of items is per IP or per queried service?

Limit is set for (Source/Target).IP(4/6), current value is 10. Data services may register hooks for object types they are capable of handling (IP4/6,hostname,...). Additional data are then fetched from all registered services for that type. For example in case an IDEA message has 15 Source.IP4s, it will result in 10x5 API queries because there are 5 services registered to handle IP4s. If there are any Target.IP4, they will have separate limit counter. The idea behind this is to resolve some number for each object type, but not to overwhelm the service APIs.

Ok, makes sense.

Also, some reasonable caching would make sense (shall I make new issue?)

Yes, that is absolutely necessary before releasing it to production. There is no need for a new issue in my opinion.

Ack. Thx.

Seems to me Targets don't get info from DNS/PassiveDNS, that's intentional?

By Pavel Kácha on 2019-06-28T11:19:09

@ph wrote:

@Jan_Mach wrote:

@ph wrote:

Very coool. Appreciate the work that went into it. Limit for number of items is per IP or per queried service?

Limit is set for (Source/Target).IP(4/6), current value is 10. Data services may register hooks for object types they are capable of handling (IP4/6,hostname,...). Additional data are then fetched from all registered services for that type. For example in case an IDEA message has 15 Source.IP4s, it will result in 10x5 API queries because there are 5 services registered to handle IP4s. If there are any Target.IP4, they will have separate limit counter. The idea behind this is to resolve some number for each object type, but not to overwhelm the service APIs.

Ok, makes sense.

Also, some reasonable caching would make sense (shall I make new issue?)

Yes, that is absolutely necessary before releasing it to production. There is no need for a new issue in my opinion.

Ack. Thx.

Seems to me Targets don't get info from DNS/PassiveDNS, that's intentional?

They do, there is just an empty result. You can see for yourself by opening the developer console and checking console logs. All services are queried for all objects except for NERD, which does not support IPv6.

By Jan Mach on 2019-06-28T12:20:05

@Jan_Mach wrote:

@ph wrote:

Seems to me Targets don't get info from DNS/PassiveDNS, that's intentional?

They do, there is just an empty result. You can see for yourself by opening the developer console and checking console logs. All services are queried for all objects except for NERD, which does not support IPv6.

Mmmkay, my bad.

By Pavel Kácha on 2019-06-28T12:31:47

I consider current state as good enough for pilot operation. Further improvements will be done in separate issues.

By Jan Mach on 2019-08-29T11:35:19

added Done 100% S-Closed labels and removed Done 50% S-Feedback labels

closed

marked this issue as related to #5067 (closed)

By Pavel Kácha on 2024-12-26T14:19:09

mentioned in issue #5067 (closed)

By Jan Mach on 2024-12-26T14:19:10