Migration from MongoDB to PostgreSQL

After many months of rigorous testing we have finally decided to migrate from MongoDB to PostgreSQL. The MongoDB performance and long response times are decreasing the usability of the web interface and even of the reporting subsystem. We will now abandon the MongoDB in favor of PostgreSQL.

The migration will occur in following phases:

1. Define data models for utility objects (users, groups, etc.)
2. Implement new storage service module for utility object management.
3. Rewrite all current code working with utility objects.
4. Implement new storage service for IDEA event management (maybe it will be possible to use the same service as above, maybe there will have to be different one, highly optimized and with lower overhead).
5. Rewrite all current code working with IDEA messages through persistent database storage.

assigned to @Jan_Mach

By Jan Mach on 2024-12-26T14:05:59

added C-Dev-Core Done 0% P4 S-In Progress T-Task labels

changed milestone to %2.0

Notes for future production migration:

• both db engines on the same machine
• user/group management + relevant data – can be migrated independently of event db
  • then management GUI
  • we will need migration script, which will allow reporter/statistician use Perl DB until also rewritten
• statistician
  • simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
  • then statistician can be written in python, switched and conversion script ditched
• reporter
  • simple incremental conversion script from Mongo to PQ + GUI, which already looks into PQ
  • then reporter can be written in python, switched and conversion script ditched
• incoming events forked into both old (Mongo) storage and new (PQ) storage daemons (events go simultaneously into both DBs)
  • then GUI can be written independently

Back Relation 'relates' from: 3362

Back Relation 'relates' from: 2808

By Pavel Kácha on 2017-10-10T16:16:37

Relation Relates - Task 3362 Migrate Mentat system to new hardware

By Jan Mach on 2018-05-31T11:58:07

On tuesday 24.7. 2018 we have successfully migrated our production instance of Mentat system to version of Mentat system that is completely written in Python and uses PostgreSQL as database backend. Therefore this task can be considered complete.

By Jan Mach on 2018-07-27T09:19:05

added Done 100% S-Closed labels and removed Done 0% S-In Progress labels

closed

Relation Relates - Bug 2808 Python and batch_size constraint

By Jan Mach on 2018-07-27T09:54:01

marked this issue as related to #3362 (closed)

By Jan Mach on 2024-12-26T14:12:51