Bad Multiprocess Pickle of DB Object
Operator ran into this error. Database sessions have registered process guards to prevent access to the same TCP socket. But in this case python is strictly disallowing the object to even begin pickling.
2024.03.08 10:05:51 QLP version 0.12.0b8
2024.03.08 10:06:42 TIC version set to 8.2
2024.03.08 10:07:14 TICA version set to 1.4.0
2024.03.08 10:07:14 Detrending method set to QSP
2024.03.08 10:07:14 KSPBSMAX set to 2.5
2024.03.08 10:07:14 Reading targets for orbit 153
2024.03.08 10:12:42 Found 351810 TIC IDs for cams:ccds 1:1,2,3,4.
2024.03.08 10:12:42 Reading targets for orbit 154
2024.03.08 10:14:40 Found 351797 TIC IDs for cams:ccds 1:1,2,3,4.
2024.03.08 10:14:40 Read 351830 total TICs from orbits (153, 154)
0%| | 0/351830 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/pdo/users/petitpas/.local/bin/qlp", line 8, in <module>
sys.exit(qlp_main())
File "/pdo/users/petitpas/FFITools/qlp/cli/qlp.py", line 120, in qlp_main
stage_main(QLPParser)
File "/pdo/users/petitpas/FFITools/qlp/cli/lctools.py", line 229, in lctools_main
main(args)
File "/pdo/users/petitpas/FFITools/qlp/lctools/bin/hlsp.py", line 391, in hlsp_main
list(tqdm(pool.imap_unordered(h5tofits_wrapper, tasks), total=len(tic_locations)))
File "/sw/python-versions/python-3.9.2/lib/python3.9/site-packages/tqdm/std.py", line 1178, in __iter__
for obj in iterable:
File "/sw/python-versions/python-3.9.2/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
File "/sw/python-versions/python-3.9.2/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/sw/python-versions/python-3.9.2/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/sw/python-versions/python-3.9.2/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'create_engine.<locals>.connect'
<petitpas@pdogpu6> (0.11.6) {/scratch/hlsp} %```
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information