Skip to content

[ENG-9044] Add manage command to resync preprint dois v1 #11617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Vlad0n20 wants to merge 14 commits into
base: feature/pbs-26-19
Choose a base branch
from
Open

[ENG-9044] Add manage command to resync preprint dois v1 #11617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
abfc3c9
Add manage command to resync preprint dois v1
Vlad0n20 Mar 2, 2026
af27fef
Update command
Vlad0n20 Mar 9, 2026
8e93ef4
Added a class-level docstring
Vlad0n20 Apr 8, 2026
6efa09a
Register unversioned DOI alongside versioned on Crossref submissions
Vlad0n20 Apr 13, 2026
78ba103
Add daily Celery beat task and unversioned DOI tracking for preprint …
Vlad0n20 Apr 13, 2026
03b77a5
fix test
Vlad0n20 Apr 14, 2026
ab09a25
fix comments
bodintsov Aug 19, 2026
88f467a
fix tests
bodintsov Aug 20, 2026
47798c1
add to low priority queue, fix short circuit
bodintsov Aug 28, 2026
9c4a270
monitor active tasks and spawn new tasks below threshold
bodintsov Sep 22, 2026
71a3038
update migration
bodintsov Sep 22, 2026
4895983
remove skipped, fix select logic, fix to latest preprint version
bodintsov Oct 9, 2026
3a6ad34
flake8
bodintsov Oct 9, 2026
140b15b
check prefix not full doi, add description to command
bodintsov Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions api/crossref/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

from api.crossref.permissions import RequestComesFromMailgun
from osf.models import Preprint, NotificationTypeEnum
from osf.models.base import Guid
from website import settings
from website.preprints.tasks import mint_doi_on_crossref_fail

Expand Down Expand Up @@ -48,6 +49,24 @@ def post(self, request):
if record.get('status').lower() == 'success' and doi:
msg = record.find('msg').text
created = bool(msg == 'Successfully added')

@adlius adlius Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So a DOI in the doi_unversioned category is only written when Crossref replies Successfully added. This could be a potential issue because if the DOI
is already registered, the email probably comes back with something like Successfully updated, so for those preprints
this branch never runs.

This would affect legacy preprints that exists before versioning was a thing. Their base guid is their original guid, so the DOI deposit here is the one registered years ago.

I checked against a local copy of prod db. One of preprint 22hq2 has 10.31219/osf.io/22hq2 already registered with crossref. So it's possible that the crossref response email wouldn't contain Successfully added but Successfully updated or some other language. Will need to confirm if that is indeed the case.

If that is the case, one possible fix is that for an unversioned DOI, store doi_unversioned on any successful record rather than only on Successfully added.

@adlius adlius Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another problem: if we run the resync task every five minutes, and if the preprint that was resynced in the last run still haven't received an email back confirming the update of the DOI, then in the next run it will be included in the resync task again. So we'd have to rethink how we should do this. One idea suggested by @cslzchen is that we follow the example in this PR and monitory the number of active tasks and spawn new tasks if the number of active task falls below a certain number (say 1000). In this case, we don't have to worry about running the task every five minutes.

# Unversioned DOIs (no _vN suffix, e.g. 10.31233/osf.io/tnaqp) are routing
# aliases that always resolve to the latest version via OSF's GUID routing.
# Store them as 'doi_unversioned' on the v1 preprint so we can track which
# preprint series have had their unversioned DOI registered.
_, version = Guid.split_guid(guid) if guid else (None, None)
if not version:
logger.info(f'Unversioned DOI confirmed by CrossRef: {doi}')
if guid:
v1_preprint = Preprint.objects.filter(
versioned_guids__guid___id=guid,
versioned_guids__version=1,
).first()
if v1_preprint:
v1_preprint.set_identifier_value(category='doi_unversioned', value=doi)
Preprint.objects.filter(id=v1_preprint.id).update(doi_resync_queued_at=None)

@bodintsov bodintsov Sep 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR comment: Added doi_resync_queued_at, so not include them to eligible for resync

dois_processed += 1
continue

legacy_doi = preprint.get_identifier(category='legacy_doi')
if created or legacy_doi:
# Sets preprint_doi_created and saves the preprint
Expand All @@ -56,6 +75,7 @@ def post(self, request):
elif 'possible preprint/vor pair' not in msg.lower():
# Directly updates the identifier
preprint.set_identifier_value(category='doi', value=doi)
Preprint.objects.filter(id=preprint.id).update(doi_resync_queued_at=None)

@bodintsov bodintsov Sep 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR comment: set doi_resync_queued_at to None when preprint is being processed


dois_processed += 1

Expand All @@ -67,9 +87,14 @@ def post(self, request):
if 'Relation target DOI does not exist' in record.find('msg').text:
logger.warning('Related publication DOI does not exist, sending metadata again without it...')
mint_doi_on_crossref_fail.apply_async(kwargs={'preprint_id': preprint._id})
# This error occurs when a single preprint is being updated several times in a row with the same metadata [#PLAT-944]
elif 'less or equal to previously submitted version' in record.find('msg').text and record_count == 2:
break
# This error occurs when a single preprint is being updated several times in a row
# with the same metadata [#PLAT-944]. Previously this broke out of the loop when
# record_count == 2 (single DOI submitted twice). Now batches legitimately contain
# 2 records (versioned + unversioned DOI), so we continue instead of break to allow
# the remaining record to be processed.
elif 'less or equal to previously submitted version' in record.find('msg').text:
dois_processed += 1
continue
else:
unexpected_errors = True
logger.info(f'Creation success email received from CrossRef for preprints: {guids}')
Expand Down
88 changes: 88 additions & 0 deletions api_tests/crossref/views/test_crossref_email_response.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,3 +220,91 @@ def test_confirmation_marks_legacy_doi_as_deleted(self, app, url, preprint):
app.post(url, context_data)

assert preprint.identifiers.get(category='legacy_doi').deleted

def test_unversioned_doi_confirmation_skips_identifier_update(self, app, url, preprint):
versioned_doi = settings.DOI_FORMAT.format(
prefix=preprint.provider.doi_prefix, guid=preprint._id
)
preprint.set_identifier_value(category='doi', value=versioned_doi)

base_guid = preprint.get_guid()._id # no _vN suffix
unversioned_doi = settings.DOI_FORMAT.format(
prefix=preprint.provider.doi_prefix, guid=base_guid
)
dual_confirmation_xml = """
<?xml version="1.0" encoding="UTF-8"?>
<doi_batch_diagnostic status="completed" sp="cs3.crossref.org">
<submission_id>1390675999</submission_id>
<batch_id>{batch_id}</batch_id>
<record_diagnostic status="Success">
<doi>{versioned_doi}</doi>
<msg>Successfully updated</msg>
</record_diagnostic>
<record_diagnostic status="Success">
<doi>{unversioned_doi}</doi>
<msg>Successfully added</msg>
</record_diagnostic>
<batch_data>
<record_count>2</record_count>
<success_count>2</success_count>
<warning_count>0</warning_count>
<failure_count>0</failure_count>
</batch_data>
</doi_batch_diagnostic>
""".format(
batch_id=preprint._id,
versioned_doi=versioned_doi,
unversioned_doi=unversioned_doi,
)

context_data = self.make_mailgun_payload(crossref_response=dual_confirmation_xml)
with capture_notifications(expect_none=True):
app.post(url, context_data)

preprint.reload()
assert preprint.get_identifier_value('doi') == versioned_doi
assert preprint.get_identifier_value('doi_unversioned') == unversioned_doi

def test_unversioned_doi_confirmation_update_does_not_store_doi_unversioned(self, app, url, preprint):
versioned_doi = settings.DOI_FORMAT.format(
prefix=preprint.provider.doi_prefix, guid=preprint._id
)
preprint.set_identifier_value(category='doi', value=versioned_doi)

base_guid = preprint.get_guid()._id
unversioned_doi = settings.DOI_FORMAT.format(
prefix=preprint.provider.doi_prefix, guid=base_guid
)
update_confirmation_xml = """
<?xml version="1.0" encoding="UTF-8"?>
<doi_batch_diagnostic status="completed" sp="cs3.crossref.org">
<submission_id>1390676000</submission_id>
<batch_id>{batch_id}</batch_id>
<record_diagnostic status="Success">
<doi>{versioned_doi}</doi>
<msg>Successfully updated</msg>
</record_diagnostic>
<record_diagnostic status="Success">
<doi>{unversioned_doi}</doi>
<msg>Successfully updated</msg>
</record_diagnostic>
<batch_data>
<record_count>2</record_count>
<success_count>2</success_count>
<warning_count>0</warning_count>
<failure_count>0</failure_count>
</batch_data>
</doi_batch_diagnostic>
""".format(
batch_id=preprint._id,
versioned_doi=versioned_doi,
unversioned_doi=unversioned_doi,
)

context_data = self.make_mailgun_payload(crossref_response=update_confirmation_xml)
with capture_notifications(expect_none=True):
app.post(url, context_data)

preprint.reload()
assert preprint.get_identifier_value('doi') == versioned_doi
assert preprint.get_identifier_value('doi_unversioned') == unversioned_doi
Loading
Loading