After much work and consideration I have concluded, that it will be much better to use Vagrant and therefore full featured virtual machine for local Mentat development due to the complexity of the whole environment. It would be somewhat possible to do this with Docker, as was originally intended, but the result would be either completely hacked and abused Docker concept, or really complex network of multiple Docker containers.
First commit introduces working, but unpolished and somewhat rough on the edges proof of concept for setting up and using development environment. There is still lots of work, but I am confident the result will be much more pleasant to work with and as a side effect many utility scripts will receive another look and rework.
By Jan Mach on 2021-05-19T16:38:42
Jiří Rážchanged title from Use Docker as alternative tool for local development to Use Vagrant as alternative tool for local development·
Imported
changed title from Use Docker as alternative tool for local development to Use Vagrant as alternative tool for local development
Documentation regarding development with Vagrant grows slowly here: doc/sphinx/_doclib/development.rst.
Vagrant up command takes roughly 15 mins, mostly due to the provisioning of the database (download time of the snapshot from mentat-alt is not included, because it is not necessary to do it every time). This can be improved by making the development snapshot of the database smaller.
please, do not incorporate (un)relocation patches just yet, make notes - separate issue, perhaps?
we would like parts of this work to be reusable on integration testing (Mek: noted)
if possible, test also with Docker backend (as GitLab uses Docker as default)
put in only a handful of necessary fixtures, make import from another live systems available on explicit action
(thinking after meeting:) or maybe no fixtures yet, just empty functioning system, and let us make this small "devel" fixture set gradually during integration testing implementation? I mean - it does not have to be within scope of this issue, just make the wiring ready and useful contents will grow based on later needs.
everybody: please test (and note functioning versions of Vagrand and VirtualBox)
(thinking after meeting:) or maybe no fixtures yet, just empty functioning system, and let us make this small "devel" fixture set gradually during integration testing implementation? I mean - it does not have to be within scope of this issue, just make the wiring ready and useful contents will grow based on later needs.
I concur this.
everybody: please test (and note functioning versions of Vagrand and VirtualBox)
Tested working:
VirtualBox 6.1.26 with Vagrant 2.2.15
Tested not-working:
VirtualBox 6.1.26 with Vagrant 1.8.7
VirtualBox 6.1.22 with Vagrant 1.8.7
Maybe we could create a table in wiki? I will look into that.
Summary of drawbacks of using current opproach with lightweight chroot and maximal application portability:
Configuration files for Babel translations need to be generated to provide the necessary portability. Opposing argument is, that this is not relevant, because babel is misused in this use case anyway and the translations of report contents should be handled differently.
There are many text (JSON) configuration files, which need a separate copy in case of local development within the Git repository and deployment on the server. It is untrivial matter to keep all instances of all configuration files in sync. Currently this is resolved by using .env files, but this solution brings is too magical to be clear.
I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.
Pavel, I have a heretic question regarding installation of Warden client on development Vagrant box. Current provisioning procedure already requires to fetch configuration file /etc/GeoIP.conf from mentat-alt, so that the developer does not have to register account on MaxMind site. Without this configuration file the provisioning succeeds, but system is not usable right away, because working geolocation service is a prerequisite. That`s why I opted to download it from mentat-alt. To clarify, this download is not part of the provisioning script, it is a separate step before vagrant up, a prerequisite.
And now for my question. Would you consider using same approach for warden client? Would it be possible to download a prepared set of certificate and key for warden client and use the same pair as a default for all developers on all Vagrant boxes? This would further greatly simplify the installation process. Of course I would suggest to implement it as optional feature, otherwise the Vagrant box would be too much tied to CESNET and Warden.
Pavel, another note: Docker backend is currently not available for debian/contrib-buster64 Vagrant box, so I am not able to perform the testing you requested.
While working on scripts for dumping database for development and importing it back later I have come across interesting issue, that have multiple solutions and I would like to brainstorm it with other developers and most importantly with Radko.
The issue lies in different ordering of columns on Vagrant box and our production and staging servers. This is caused by the fact, that when the Vagrant box is provisioned the database schema is created immediately to the latest state, while the production and staging servers are running and therefore database schema is maintained by migrations. Newly added columns are appended to the end of the table.
Currently I am using equivalent of the following command to do the time based dump of some unnecessarily big tables like reports and then import it back:
sudo-umentatpsqlmentat_main-c"COPY (SELECT * FROM ${table_name} WHERE createtime >= '${BACKUP_DATE_FROM}'::timestamptz) TO stdout">${BACKUP_DIR}/${table_name}.datsudo-upostgrespsqlmentat_main-c"COPY FROM stdin;"</vagrant/data/devsnapshot/chunks/reports_events.dat
I am using COPY command, because to my knowledge pg_dump does not support performing WHERE on the dumped tables.
I have considered following solutions, but I do not like any of them:
Change ordering of the columns on our servers. PostgreSQL does not support this natively, but there are workarounds, for example by creating new table and copying data between them, then dropping the former and renaming the latter. This however is not permanent fix, another migration can break it again.
List the columns in the select, so that they are in order expected for Vagrant box. This however would need to be maintained every time we add or remove column.
Use different dumping and importing mechanism that supports where to restrict the number of fetched rows by createtime.
I would like to kindly ask Radko for help with this, I am sure he has some neat trick right up his sleeve.
Side note: It is currently possible to run /etc/mentat/scripts/devsnapshot.sh script on mentat-alt server to prepare fresh snapshot and then experiment with this from within the Vagrant box with command make data-import-dbsnapshot, which currently fails due to the error discussed above.
!DEV! vagrant@mentat-devel /vagrant $ sudo-u postgres psql mentat_main -c"COPY reports_events FROM stdin;" < /vagrant/data/devsnapshot/chunks/reports_events.datERROR: invalid input syntax for type integer: "2021-10-26 04:40:03.114034"CONTEXT: COPY reports_events, line 1, column id: "2021-10-26 04:40:03.114034"
While working on scripts for dumping database for development and importing it back later I have come across interesting issue, that have multiple solutions and I would like to brainstorm it with other developers and most importantly with Radko.
The issue lies in different ordering of columns on Vagrant box and our production and staging servers. This is caused by the fact, that when the Vagrant box is provisioned the database schema is created immediately to the latest state, while the production and staging servers are running and therefore database schema is maintained by migrations. Newly added columns are appended to the end of the table.
Currently I am using equivalent of the following command to do the time based dump of some unnecessarily big tables like reports and then import it back:
I am using COPY command, because to my knowledge pg_dump does not support performing WHERE on the dumped tables.
I have considered following solutions, but I do not like any of them:
Change ordering of the columns on our servers. PostgreSQL does not support this natively, but there are workarounds, for example by creating new table and copying data between them, then dropping the former and renaming the latter. This however is not permanent fix, another migration can break it again.
List the columns in the select, so that they are in order expected for Vagrant box. This however would need to be maintained every time we add or remove column.
Use different dumping and importing mechanism that supports where to restrict the number of fetched rows by createtime.
I would like to kindly ask Radko for help with this, I am sure he has some neat trick right up his sleeve.
Yes, this is a tricky one. I see two possible solutions. First one is organizational: always add the new columns at the end when changing schema. Of course that has enforcement problems and certainly cannot help now.
As for your list, number two is the most versatile and universally accepted. The drawback of maintenance could be lifted as it is possible to get the existing ordering from the source table and exploit that COPY allows to define the column ordering even in its FROM variant. Therefore I would go with something like this:
sudo-umentatpsqlmentat_main-c"COPY (SELECT string_agg(data.column_name, ',') FROM (SELECT column_name FROM information_schema.columns WHERE table_name = '${table_name}' ORDER BY ordinal_position) AS data) TO stdout">${BACKUP_DIR}/${table_name}.colssudo-umentatpsqlmentat_main-c"COPY (SELECT * FROM ${table_name} WHERE createtime >= '${BACKUP_DATE_FROM}'::timestamptz) TO stdout">${BACKUP_DIR}/${table_name}.datsudo-upostgrespsqlmentat_main-c"COPY ${table_name} ($(cat ${BACKUP_DIR}/${table_name}.cols)) FROM stdin;"<${BACKUP_DIR}/${table_name}.dat
Is that what you were looking for?
As a side note, my first idea was to define a VIEW with the WHERE filtering and use pg_dump on that (so a variant of option 3), but the tool seems to handle views differently and only export their definition (what kind of makes sense), so that is a no go.
Yes, this is a tricky one. I see two possible solutions. First one is organizational: always add the new columns at the end when changing schema. Of course that has enforcement problems and certainly cannot help now.
As for your list, number two is the most versatile and universally accepted. The drawback of maintenance could be lifted as it is possible to get the existing ordering from the source table and exploit that COPY allows to define the column ordering even in its FROM variant. Therefore I would go with something like this:
[...]
Is that what you were looking for?
As a side note, my first idea was to define a VIEW with the WHERE filtering and use pg_dump on that (so a variant of option 3), but the tool seems to handle views differently and only export their definition (what kind of makes sense), so that is a no go.
Thank you, that is perfect! Again, you didn't disappoint.
I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.
I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.
I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.
I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.
This is actually related to #7121 and not this issue.
I have just merged this branch into devel. It was possible, because I have successfully removed all breaking changes. Now the new Vagrant configurations are just non breaking additions. This means, that now it will be possible to test the new Vagrant development workflow more easily.
I am not sure if something changed since this comment, but I have a trouble launching mentat-controller.py --command start. Apparently, now it requires user account mentat. Here is the error message - CRITICAL ERROR: "Requested unknown user account 'mentat'". When I create user mentat, I have some issues with permissions. Changing permissions of mentat-ng repository is not enough.
I have investigated this furthrer and came to the conclusion, that your issue was due to the wrong usage and perhaps misleading documentation. I had no troubles executing mentat-controller.py --command start when doing it under the mentat system user as it should be. I have improved the documentation to be more clear, but the short summary is this: there are two main system users: vagrant and mentat. the vagrant is deeply baked into the Vagrant system and acts as de facto root for the box. It would not be practical to change it to something else. The mentat user is there to mimic the production enviroment and is intended to run Mentat system services. Project files are mounted and owned by vagrant user and by default you are also connected as vagrant user, so that you may freely work with the files. When you need to execute Mentat, you need to switch to mentat system user, because everything is preconfigured for that.
Note: Ok, so now the problem is that the default configuration was changed to hard-code the usage of mentat:mentat system user and group to solve the issue with directories being occasionally created with incorrect permissions. I am wondering, if the best approach at this point shouldn`t be to stop supporting arbitrary installations for development. After all the whole purpose of having and using a tool like Vagrant is to be able to stop worrying with broken and/or hard installations.
Summary of drawbacks of using current opproach with lightweight chroot and maximal application portability:
Configuration files for Babel translations need to be generated to provide the necessary portability. Opposing argument is, that this is not relevant, because babel is misused in this use case anyway and the translations of report contents should be handled differently.
There are many text (JSON) configuration files, which need a separate copy in case of local development within the Git repository and deployment on the server. It is untrivial matter to keep all instances of all configuration files in sync. Currently this is resolved by using .env files, but this solution brings is too magical to be clear.
...
Also, there is a couple of files (mostly configuration) which I modified in order to be able to run some modules:
conf/core/reporting.json.conf
conf/core/statistics.json.conf
changed paths in both files to chroot
conf/mentat-hawat.py.conf
enabled 'hawat.blueprints.auth_dev'
conf/mentat-ideagen.py.conf
added all starting IP addresses of networks from negistry
scripts/sqldb-migrate.sh
changed path /etc/mentat/migrations-events to /home/rajmund/mentat-ng/migrations-events
When I was trying to replicate the issue described in #7548 I came upon another issue.
I deleted the virtual machine and the cloned repository. Then I cloned the repository again and switched to devel branch. When I executed make data-fetch-config the following error was printed on my terminal screen:
ssh: Could not resolve hostname mentat-alt: Temporary failure in name resolutionmake: ***[Makefile:486: data-fetch-config] Error 1
I checked the Makefile and there is indeed only mentat-alt and not the full address. Am I doing something wrong?
When I was trying to replicate the issue described in #7548 I came upon another issue.
I deleted the virtual machine and the cloned repository. Then I cloned the repository again and switched to devel branch. When I executed make data-fetch-config the following error was printed on my terminal screen:
[...]
I checked the Makefile and there is indeed only mentat-alt and not the full address. Am I doing something wrong?
No, you are not doing anything wrong. Locally I use the SSH`s config file and I automatically put the shortcut to SCP path. I have not discovered this error before, bacause it just worked for me correctly. I have put full URLs to Makefile, so it should be resolved now.