Dear all,

Here a follow-up:

After two full days of debugging and trying  out many variants and getting more gray hair, we think it is a problem how the hash $EPrints::Index::FREETEXT_CHAR_MAPPING in Index/Tokenizer.pm is addressed.

This behaves completely erratically, sometimes š is translated to s, sometimes not. It is as sometimes the hash would not exist.

This problem is observed when characters with UTF codepoint > 0x00ff are used (non-Ascii chars).

It might be that a “use 5.8.0” might remedy this (not tried out) by using the old Unicode implementation of perl.

However, we applied a solution now  that we also use cfg.d/optional_filename_sanitise.pl to transliterate file names and in several import plugins, which is much simpler and failsafe: Text::Unidecode

This library separates the upper and lower bytes of an UTF8 char and then adresses the transliteration tables, which are arrays, not hashes, by the respective integer value of the UTF8 bytes.

Since the transliteration tables are very extensive, maintaining $EPrints::Index::FREETEXT_CHAR_MAPPING is not necessary at all.

Also, it is possible to override the Text::Unidecode transliteration tables if one needs to. See https://metacpan.org/pod/Text::Unidecode

Also, I see that it’s part of the EPrints 3.3 package (but has been removed with EPrints 3.4).

Kind regards,

Martin