EPrints Technical Mailing List Archive

Message: #09173

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] Modification of language by script

CAUTION: This e-mail originated outside the University of Southampton.

Dear John,

Yes, there are some EPrints with multiple documents, and they might be of different language.
And yes, I do want to process everything.

However, not necessarily with the same tool. I have already have scripts for examining the
language settings, and also some tools for finding out what the language is. (At the moment
what I do is the following: get the title, run it through a spell checker with a given dictionary, and
find out the percentage of the known words in the title. This checks in one step whether
the first hypothesis [the language is Hungarian] is true or not. Later I might want to
process the full text layer of the PDF. Next I might repeat the same with the next
choice of language, say, English.)

Also, I might do the simple cases (only one document) first.

So fix_language might be jut the thing I need - I am looking at it.

Thank You!

András

----- Original Message -----
From: "John Salter" <J.Salter@leeds.ac.uk>
To: "holl andras" <holl.andras@konyvtar.mta.hu>
Cc: "eprints-tech" <eprints-tech@ecs.soton.ac.uk>
Sent: Tuesday, 17 January, 2023 10:35:06
Subject: RE: Modification of language by script

Hi András,
Do some EPrints have multiple documents, and can those documents be in different languages?

It sounds like you want to process everything, rather than searching for specific documents/EPrints to update.

>From the details in these pages:
- https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2Fw%2FAPI%3AEPrints%2FDataSet&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wYIzVFSsuq7yVD%2BtRJhfFKkZfIbPCMOcaW2Ogh5SIJY%3D&reserved=0
- https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.eprints.org%2Fw%2FAPI%3AEPrints%2FList&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K4NU9%2Bv%2FrpNVC7AlV3SRmNMPa24Olmwo%2FFqcdMadZEw%3D&reserved=0

I would start with something like this:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fjesusbagpuss%2Fa8cc8c5328aa6e33e068609bc6f3d6ca&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wYV60qVECDpcty5ZMu%2BUqJMXXKGqXGAKqO6zPe%2FB1Kk%3D&reserved=0

The bits you need to work out are in the 'process_eprint' function:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fjesusbagpuss%2Fa8cc8c5328aa6e33e068609bc6f3d6ca%23file-fix_language-L78&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fdGDRly1%2FYEGsHILCxAmtFlu2B2sRR%2FMGjK9n2FEdgA%3D&reserved=0 - how to calculate the language based on the EPrint details
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fjesusbagpuss%2Fa8cc8c5328aa6e33e068609bc6f3d6ca%23file-fix_language-L86&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YIi%2B02hlv4gc4r5lpy%2BjX%2FGrgRWnzdU70Z6iYavvmaY%3D&reserved=0 - how to calculate the language from the document

As-is, the script will not change anything. The 'commit' lines are commented out of safety.
It also references a field that might not exist (eprint.language).

You may want to check the existing setting for the language of the document before updating it.
If you have EPrints with multiple documents attached, you might want to do something like this:
        $eprint->set_under_construction(1);
        ... update (commit) multiple doc changes
        $eprint->set_under_construction(0);
        $eprint->commit;
This means that the EPrint will have one new revision, rather than a revision for each document updated.

Let me know if that helps at all!

Cheers,
John


- you may want t


-----Original Message-----
From: András Holl [mailto:holl.andras@konyvtar.mta.hu]
Sent: 16 January 2023 12:05
To: John Salter <J.Salter@leeds.ac.uk>
Cc: eprints-tech <eprints-tech@ecs.soton.ac.uk>
Subject: Re: Modification of language by script



Dear John,

I am using EPrints 3.3.15. So far, the scripts for 3.2 did work for me.

Since we have installed EPrints (around 2008), the language field for the
documents have been hidden. For each uploaded documents EPrints used the
language settings of the browser as a guess for the language of a document,
and we did not care.

Now we have embarked upon a text mining project, and suddenly it become
important what the language is. I will process the content of the repository
(some 200k items), and find out what the language is, based first on the language
of the title, and then maybe the language of the text layer of the PDFs.

But when I know (or have a reasonable guess), I might try to set the language
of the EPrint document.

With kind regards,

Andras Holl

--
Holl András
informatikai főigazgató-helyettes / deputy director (IT)
MTA Könyvtár és Információs Központ / MTA Library and Information Centre

----- Original Message -----
From: "John Salter" <J.Salter@leeds.ac.uk>
To: "eprints-tech" <eprints-tech@ecs.soton.ac.uk>, "holl andras" <holl.andras@konyvtar.mta.hu>
Sent: Monday, 16 January, 2023 12:37:30
Subject: RE: Modification of language by script

Hi András,
Which version of EPrints are you using?

The scripts you found were written against EPrints 3.2, so might not work if you are using EPrints 3.3 or 3.4.

How do you determine which documents need the language field to be updated?
Is there a field at the EPrint, or at the Document level that you need to search for, to work out which ones, or do you have a list of IDs, or something similar?

Cheers,
John


-----Original Message-----
From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of András Holl via Eprints-tech
Sent: 13 January 2023 12:51
To: eprints-tech@ecs.soton.ac.uk
Subject: [EP-tech] Modification of language by script

CAUTION: This e-mail originated outside the University of Southampton.

Dear All,

I would like to modify language settings of a document in a given EPrint by a script.

How should I do it, with the script search_and_modify.pl found at
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.eprints.org%2Fservices%2Ftraining%2Fresources%2Fscripts%2Feprints3_2%2Fbin%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XEgyU46H5xOTdapUZ1Njv4n5YuQXmmI9q4DK9MibX%2BM%3D&reserved=0 ?

With kind regards,

András Holl

--
Holl András
informatikai főigazgató-helyettes / deputy director (IT)
MTA Könyvtár és Információs Központ / MTA Library and Information Centre

*** Options: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ecs.soton.ac.uk%2Fmailman%2Flistinfo%2Feprints-tech&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gyLBQ9autQGUJJOhZm%2FhUrFVoRvge%2Fv3rtX0VUYYWtM%3D&reserved=0
*** Archive: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4yMwFuYFyQ%2BCnCiDsS9wmPWK6GkYlgzhbebQ%2BB6imM8%3D&reserved=0
*** EPrints community wiki: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=05%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7Cb2f1f5677cd447add93608daf873014f%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C638095470065899905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=F4fB%2FaXXQH3290Rh8cZ7VSq3j4vfmugT7y3NGApV7aI%3D&reserved=0
--
Holl András
informatikai főigazgató-helyettes / deputy director (IT)
MTA Könyvtár és Információs Központ / MTA Library and Information Centre