Wrong display of national characters in preloaded data in user registration form

The user registration form at: https://mentat-hub.cesnet.cz/mentat/auth_env/register displays the national characters in full name wrongly. The input can be corrected and then the right representation is visible in user management, so the trouble seems to lie in load from identity provider. The case when the name is not fixed by user was not tested. The bug is also present on mentat-alt.

added C-Dev-GUI Done 0% P5 S-New T-Bug labels

changed milestone to %Backlog

added P3 label and removed P5 label

assigned to @rajmund_hruska

By Pavel Kácha on 2024-12-26T14:24:45

added P4 label and removed P3 label

changed milestone to %2.10

Also observed in e-mail (see Full name):

Return-Path: <mentat@mentat-alt.cesnet.cz>
X-Original-To: krkos@office2.cesnet.cz
Delivered-To: krkos@office2.cesnet.cz
Received: from postino.cesnet.cz (postino.cesnet.cz [IPv6:2001:718:1:101::144:242])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by office2.cesnet.cz (Postfix) with ESMTPS id 915A3400076
	for <krkos@office2.cesnet.cz>; Thu, 12 May 2022 14:09:10 +0200 (CEST)
Received: by postino.cesnet.cz (Postfix, from userid 110)
	id 8CFEC580090; Thu, 12 May 2022 14:09:10 +0200 (CEST)
X-Spam-Status: No
X-Spam-cesnet-Status: No, score=-1.7 required=5.0 tests=BAYES_00,SPF_HELO_NONE,
	SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no
	version=3.4.6
X-Spam-Level: 
X-Spam-Virus: No
X-Spam-Report: 
	* -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
	*      [score: 0.0000]
	*  0.0 SPF_NONE SPF: sender does not publish an SPF Record
	*  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
	* -0.0 T_SCC_BODY_TEXT_LINE No description available.
	*  0.2 TXREP TXREP: Score normalizing based on sender's reputation
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on postino.cesnet.cz
X-postfwd: rbl:wl.cesnet.cz:<mentat-alt.cesnet.cz is listed on CESNET welcomelist since 1652341206.0.>
X-postfwd: rbl:wl.cesnet.cz:<mentat-alt.cesnet.cz is listed on CESNET welcomelist since 1652341206.0.>
X-postfwd: rbl:wl.cesnet.cz:<mentat-alt.cesnet.cz is listed on CESNET welcomelist since 1652341206.0.>
X-postfwd: rbl:wl.cesnet.cz:<mentat-alt.cesnet.cz is listed on CESNET welcomelist since 1652341206.0.>
Received: from mentat-alt.cesnet.cz (mentat-alt.cesnet.cz [195.113.161.44])
	by postino.cesnet.cz (Postfix) with ESMTP id 9D4D758008E;
	Thu, 12 May 2022 14:09:09 +0200 (CEST)
Received: by mentat-alt.cesnet.cz (Postfix)
	id 9393034C007B; Thu, 12 May 2022 14:09:09 +0200 (CEST)
Delivered-To: root@mentat-alt.cesnet.cz
Received: from mentat-alt.cesnet.cz (localhost [IPv6:::1])
	by mentat-alt.cesnet.cz (Postfix) with ESMTP id 8F9E234C007A
	for <root@mentat-alt.cesnet.cz>; Thu, 12 May 2022 14:09:09 +0200 (CEST)
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Subject: [Mentat] Feedback for report - M20220512EL-IA39J
From: mentat@mentat-alt.cesnet.cz
To: root@mentat-alt.cesnet.cz
Date: Thu, 12 May 2022 14:09:09 +0200
Message-ID: <165235734950.19524.7826368553439102015@mentat-alt.cesnet.cz>
Reply-To: Rajmund.Hruska@cesnet.cz

Dear administrator,

Feedback for report "M20220512EL-IA39J (regular_recon-scanning, ip:
2001:67c:1220:8b4:565:f268:b312:22a4)" has been send by user:

    Login:           hruska@cesnet.cz
    Full name:       Rajmund Hubert Hruška
    Email:           Rajmund.Hruska@cesnet.cz
    Organization:    CESNET, z. s. p. o.

Message by user:

    TEST by Rajmund

Link to report:

https://mentat.domain.org/mentat/reports/M20220512EL-IA39J/unauth#regular_recon-scanning

Have a nice day

-- Mentat System

By Radko Krkoš on 2022-05-12T12:55:15

So far I have concluded that the fullname given by the environment variable cn is encoded in UTF-8. I printed the bytes of the string given by the environment with this code ":".join("{:02x}".format(ord(c)) for c in s) and I used a online convertor to convert my name to bytes (https://onlineutf8tools.com/convert-utf8-to-bytes). I got the same sequence of bytes.

What I don't know yet is why it's displayed in this way. And why it's stored in the database as a wrong string.

added S-In Progress label and removed S-New label

I don't really know why, but this seems to be working:

item.fullname = flask.request.environ['cn'].encode("iso-8859-1").decode()

added Done 40% label and removed Done 0% label

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200632:

I don't really know why, but this seems to be working:

Yeah, incorrectly declared encoding somewhere. The mentioned code looks like a valid workaround to me. If you are not into more detective work, I am OK with using that.

By Radko Krkoš on 2022-05-25T12:04:26

Radko Krkoš wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200633:

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200632:

I don't really know why, but this seems to be working:

Yeah, incorrectly declared encoding somewhere. The mentioned code looks like a valid workaround to me. If you are not into more detective work, I am OK with using that.

Great! So it should be fixed in 4c89be6b.

Nothing much happens when the name isn't fixed by the user. For example on mentat-hub there is only one user for whom the full name wasn't fixed - https://mentat-hub.cesnet.cz/mentat/users/169/show.

added Done 100% S-Resolved labels and removed Done 40% S-In Progress labels

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200634:

Radko Krkoš wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200633:

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200632:

I don't really know why, but this seems to be working:

Yeah, incorrectly declared encoding somewhere. The mentioned code looks like a valid workaround to me. If you are not into more detective work, I am OK with using that.

Great! So it should be fixed in 4c89be6b.

Nothing much happens when the name isn't fixed by the user. For example on mentat-hub there is only one user for whom the full name wasn't fixed - https://mentat-hub.cesnet.cz/mentat/users/169/show.

And the same goes for organization name, based on your example. Please, evaluate the fix for all parameters taken from flask environment.

By Radko Krkoš on 2022-05-25T12:18:30

Radko Krkoš wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200636:

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200634:

Radko Krkoš wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200633:

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200632:

I don't really know why, but this seems to be working:

Yeah, incorrectly declared encoding somewhere. The mentioned code looks like a valid workaround to me. If you are not into more detective work, I am OK with using that.

Great! So it should be fixed in 4c89be6b.

Nothing much happens when the name isn't fixed by the user. For example on mentat-hub there is only one user for whom the full name wasn't fixed - https://mentat-hub.cesnet.cz/mentat/users/169/show.

And the same goes for organization name, based on your example. Please, evaluate the fix for all parameters taken from flask environment.

Yes, I noticed just a few seconds after writing the previous post :D I fixed that as well. The other used variables are login and email address so there shouldn't be any problem with those.

Radko, could you try implement migration for these columns?

By Pavel Kácha on 2022-05-31T09:15:21

assigned to @radko_krkos and unassigned @rajmund_hruska

By Pavel Kácha on 2024-12-26T14:24:48

added S-In Progress label and removed S-Resolved label

Pavel Kácha wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7553#note_200638:

Radko, could you try implement migration for these columns?

As discussed, a migration without plpython3u is available in branch krkos-bug-#7553-auth-demangle_wo_plpython3u

By Radko Krkoš on 2022-07-25T16:53:13

Merged into devel and deployed on mentat-alt. The migration was executed without any errors.

added S-In Review label and removed S-In Progress label

added S-Closed label and removed S-In Review label

closed