EPrints Technical Mailing List Archive

Message: #05425

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

[EP-tech] Antwort: Antwort: Searching fails when database field contains Å (utf8 %c3%85)

I have added

chr(0x00c5) => 'A',     # Å
chr(0xc385) => 'A',     # Å

to the $EPrints::Index::FREETEXT_CHAR_MAPPING list in Tokenizer.pm, restarted both the web server and indexer process, and reindexed the eprint that contained Ågren as author.

Now Advanced Search works with Å

Kind regards,

Martin

--
Dr. Martin Brändle
Zentrale Informatik
Universität Zürich
Stampfenbachstr. 73
CH-8006 Zürich


Inactive hide details for martin.braendle---18/02/2016 10:14:46---Hi, we can reproduce the behavior:martin.braendle---18/02/2016 10:14:46---Hi, we can reproduce the behavior:

Von: martin.braendle@id.uzh.ch
An: eprints-tech@ecs.soton.ac.uk
Datum: 18/02/2016 10:14
Betreff: [EP-tech] Antwort:  Searching fails when database field contains Å (utf8 %c3%85)
Gesendet von: eprints-tech-bounces@ecs.soton.ac.uk




Hi,

we can reproduce the behavior:

Advanced search (which goes to the SQL index): Ågren, ågren, "Ågren" and "ågren" all fail

Quick search (which goes to the Xapian index:) both creators_name:ågren and creators_name:Ågren find results   (creators_name is the field name we use for authors)

perl_lib/EPrints/Index/Tokenizer.pm contains a translation list that maps Unicode characters to ASCII - Å is missing there. Maybe this is the clue?

Best regards,

Martin

--
Dr. Martin Brändle
Zentrale Informatik
Universität Zürich
Stampfenbachstr. 73
CH-8006 Zürich


Inactive hide details for Christer Enkvist ---17/02/2016 17:20:34---Hello all! I have encountered a weird UTF-8 related problemChrister Enkvist ---17/02/2016 17:20:34---Hello all! I have encountered a weird UTF-8 related problem when querying names in the advanced sear

Von: Christer Enkvist <christer.enkvist@slu.se>
An: "eprints-tech@ecs.soton.ac.uk" <eprints-tech@ecs.soton.ac.uk>
Datum: 17/02/2016 17:20
Betreff: [EP-tech] Searching fails when database field contains Å (utf8 %c3%85)
Gesendet von: eprints-tech-bounces@ecs.soton.ac.uk



Hello all!

I have encountered a weird UTF-8 related problem when querying names in the advanced search.  If the name of an author contains Å, like Ångström, (UTF-8 %c3%85, A with a ring above) then querying will fail.  I have not seen the problem for any other character, e.g. no problem with ”å” (a with ring above), %c3%a5, or any other non A-Z letter such as ä,Ä,ö, or Ö.  The problem is when the database entry itself contains an Å, which is typically when the character is the first in the name like Ångström or in a hyphened name like Per-Åke.

Furthermore, if the queryterm contains an “Å” then it will fail.  A few examples:

Mårten – works
mårten – works
MåRTEN -- works
MÅRTEN -- fails
mÅrten -- fails

The query field is (normally) case insensitive so it shouldn’t matter if I write “ångström” or “Ångström”.  However, hit or miss in this case depends on if the database have an Å and/or the query term contains an Å as it seems like Eprints cannot handle “Å”.  Always, displays correct and is correctly written into the database.  Only problem is the advanced search.

Should add that querying the database using SQL works without any problems (incl all upper/lower combinations).  Any ideas what may be wrong with Eprints and where to start looking?  

Regards,
Christer


Christer Enkvist, Ph D
System Administrator/System Librarian
Division of Scholarly Communication
Swedish University of Agricultural Sciences
Uppsala, Sweden

Telephone: 018-671042
*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/
*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/
*** EPrints community wiki: http://wiki.eprints.org/
*** EPrints developers Forum: http://forum.eprints.org/