EPrints Technical Mailing List Archive

See the EPrints wiki for instructions on how to join this mailing list and related information.

Message: #07530

< Previous (by date) | Next (by date) > | < Previous (in thread) | Next (in thread) > | Messages - Most Recent First | Threads - Most Recent First

Re: [EP-tech] duplicate key value violates unique constraint "event_queue_pkey"

Hi All,
There may be an underlying issue here.
The 'eventqueueid' is generated from a hash of the parameters that will be used in the index event:
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Fblob%2Fv3.4.0%2Fperl_lib%2FEPrints%2FDataObj%2FEventQueue.pm%23L88-L110&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=zKzQAeFlURHW5zn6AVf37JTXuh0j3qL0HAv5PoPC8%2BE%3D&amp;reserved=0

This means that if *exactly* the same event (e.g. generate RDF for EPrint X) is created twice, the eventqueueid will be the same, and the new task should not get added to the index queue.

The EPrints::DataObj::EventQueue::create_unique code calls EPrints::DataObj create_from_data (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Fblob%2Fv3.4.0%2Fperl_lib%2FEPrints%2FDataObj.pm%23L221-L353&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=4dpBNMhGo4NcLzAYhSmYNR%2FhIJIbhsAcbzenO%2Fj0OsI%3D&amp;reserved=0) which then tries to add the data to the database: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Fblob%2Fv3.4.0%2Fperl_lib%2FEPrints%2FDataObj.pm%23L293-L295&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=azAcu2Ros6qKuPm4niJaKvOH9%2FbzTgSIcn0qHShOwh0%3D&amp;reserved=0 using EPrints::Database::add_record.

This code *is* working correctly:
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feprints%2Feprints3.4%2Fblob%2Fv3.4.0%2Fperl_lib%2FEPrints%2FDatabase.pm%23L1387-L1388&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=sODmv2OFxXQukiMYEtdJ13ueSaILAQfTJkrJDNjf2cA%3D&amp;reserved=0
An error is raised when a duplicate ID is used.
This will return '0' to the calling code, which in turn returns undef to the EventQueue call.

The question here is: which module should be responsible for checking whether the eventqueueid already exists before trying to insert it.
Most other modules use auto-increment keys, so this isn't an issue.

We could add a dataset search for the MD5 in EPrints::DataObj::EventQueue::create_unique before trying to re-create the event?

Cheers,
John
PS I'll add this as an issue to GitHub.
PPS I'll also check how this worked in 3.3 - I don't think this was an issue there.


-----Original Message-----
From: eprints-tech-bounces@ecs.soton.ac.uk [mailto:eprints-tech-bounces@ecs.soton.ac.uk] On Behalf Of Maher Abdellatif Ahmad Qahwash via Eprints-tech
Sent: 11 October 2018 10:08
To: th.lauke@arcor.de; eprints-tech@ecs.soton.ac.uk
Subject: Re: [EP-tech] duplicate key value violates unique constraint "event_queue_pkey"

Hi Thomas

Thank you for your support.

We have cleared the events by stopping and force starting the indexer and making sure there are no pending queues but the error is still appearing in the log file when creating a new item.  

These are the steps we did:

1.	Force started the indexer.
2.	checked the "Tasks" which showed that there are no pending tasks.
3.	checked the "event_queue" table which showed that it is empty.
4.	started to create a new item.
5.	the "ERROR:  duplicate key value violates unique constraint "event_queue_pkey" started appearing again in the error_log when we start creating a new item.
6.	The event_queue table shows data with a Key (eventqueueid)=(f1dbbc2cba81b956e9290a9d39c8ddc0) which causes the duplicate key error.

Any other ideas to solve this issue?

Thank you again for your support.
Maher

-----Original Message-----
From: th.lauke@arcor.de [mailto:th.lauke@arcor.de] 
Sent: Wednesday, October 10, 2018 7:40 PM
To: Maher Abdellatif Ahmad Qahwash; eprints-tech@ecs.soton.ac.uk
Subject: Re:[EP-tech] duplicate key value violates unique constraint "event_queue_pkey"

تحذير: هذه الرسالة مرسلة من خارج الجامعة. لا تفتح أي مرفق أو رابط ما لم تكن متأكداً من أنه آمن
Warning: This mail has been sent from outside KFUPM. Do not open links or attachments unless you are sure they are safe.
____________________________________________________________
Hi Maher,

> "could it be caused because we are using Postgresql instead of Mysql?"
presumably not, because your error message
> EPrints::DataObj::create_from_data('EPrints::DataObj::EventQueue', 'EPrints::Repository=HASH(0x55aa55a83468)', 'HASH(0x55aa592f72d0)', 'EPrints::DataSet=HASH(0x55aa57023030)')
indicates the arguments (which are no subject to Postgresql!?) for the eventID

> shows that the pluginid is "Event::RDF" and is "waiting".
Assuming you are still in test mode of an experimental installation (_NO_ production site!) I recommend a clean up of your indexer:
>From Admin > System Tools > Force start indexer you get this task done.
Afterwards all hanging/waiting events should be gone!?
Otherwise engage EPRINTS/bin/indexer from console.

Hth
Thomas

*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=fbinWl1eploDWc9WS0MqH50D36UxDj8PyfX8JydKY7k%3D&amp;reserved=0
*** EPrints community wiki: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=LyjEbOPAjZIYck2WiJfx%2BKWXw5epAkrb1dLv671XQ%2BA%3D&amp;reserved=0
*** EPrints developers Forum: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fforum.eprints.org%2F&amp;data=01%7C01%7Ceprints-tech%40ecs.soton.ac.uk%7C9556a66a5f3a4e0a2e6f08d62f5acb71%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=WFQL3VIaNlQCXFOHPIgr36ufeS8r6bJRtHO4Cuqo65A%3D&amp;reserved=0