GitLab

Looks like the reflection sometimes fail in multiprocessing when multiple processes send INSERT statements.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/pdo/app/ffitools-versions/FFITools-0.11.5/qlp/bin/patools-rename", line 160, in <module>
  _ = list(tqdm(pool.imap_unordered(move_frame, tasks), total=len(tasks)))
 File "/pdo/app/python-versions/python-3.9.2/lib/python3.9/site-packages/tqdm/std.py", line 1178, in __iter__
  for obj in iterable:
 File "/pdo/app/python-versions/python-3.9.2/lib/python3.9/multiprocessing/pool.py", line 870, in next
  raise value
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "libraries_name_version_key"
DETAIL: Key (name, version)=(scipy, 1.10.1) already exists.

[SQL: INSERT INTO libraries (name, version, created_on) VALUES (%(name)s, %(version)s, now()) RETURNING libraries.id, libraries.created_on]
[parameters: {'name': 'scipy', 'version': '1.10.1'}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
<shporer@pdo6> (0.11.5) {/pdo/qlp-data/orbit-140/ffi/run} %

There are two easy ways to fix this.

1. Adjust usage to ensure this only happens once during invocation through caching (which will be copied over into each process's runtime memory).
2. Use ON CONFLICT IGNORE statements to have postgres silently accept the error.