parameterize what serialization is used in multiprocessing #72240

Activity

1. applio commented on Sep 9, 2016

   

   applio

   on Sep 9, 2016

   MemberAuthor

   More actions

   Currently multiprocessing uses the pickle module for its serialization of objects to be communicated between processes. Specifically v2 of the pickle protocols is now exclusively used to provide maximum compatibility, motivated by the desire for multiple versions of Python to be used simultaneously with multiprocessing.

   Per conversations in bpo-26507, bpo-23403, and others, multiprocessing should offer the option to specify what serialization is to be used for the transport of data between processes. Besides supporting requests to use a different version of the pickle protocol or using 3rd party tools like dill, a hook to specify the means for reducing objects to a transmittable form opens a door for other creative or higher performance strategies.

   Ultimately, this is not an enhancement to add functionality but rather to reorganize the existing internals of multiprocessing to permit better control over its use of serialization.
2. applio

   self-assigned this

   on Sep 9, 2016
3. applio

   added

   type-featureA feature request or enhancementA feature request or enhancement

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Sep 9, 2016
4. applio commented on Sep 9, 2016

   

   applio

   on Sep 9, 2016

   MemberAuthor

   More actions

   Attaching patch containing refactorizations but missing update to docs for the purposes of review.

   Introduces three new things:

   • a function to get the current serializer/reducer
   • a function to set the serializer/reducer
   • an abstract base class to assist others in rolling their own reducers
5. ericsnowcurrently commented on Sep 9, 2016

   

   ericsnowcurrently

   on Sep 9, 2016

   Member

   More actions

   LGTM
6. python-dev commented on Sep 9, 2016

   

   python-devmannequin

   on Sep 9, 2016

   Mannequin

   More actions

   New changeset 7381b1b50e00 by Davin Potts in branch 'default':
   Issue bpo-28053: Applying refactorings, docs and other cleanup to follow.
   https://hg.python.org/cpython/rev/7381b1b50e00
7. applio commented on May 19, 2017

   

   applio

   on May 19, 2017

   MemberAuthor

   More actions

   Docs need updating still.
8. applio

   added

   3.7 (EOL)end of lifeend of life

   on May 19, 2017
9. WillS commented on Jul 14, 2017

   

   WillSmannequin

   on Jul 14, 2017

   Mannequin

   More actions

   Documentation would be appreciated. I have a project that uses BaseManager, Client, and Listener to create some servers and clients. I would like to update the project to work with Python 3 and would prefer to update the clients and the servers separately (i.e. switch the client to Python 3 while the server is run with Python 2.7). However, BaseManager uses connection.Client which uses connection._ConnectionBase which uses reduction.ForkingPickler without a protocol argument. It seems the default protocol is 3 on Python 3.6 and 2 on Python 2.7 (contrary to the comment above about v2 being used). I just want to set the protocol version to 2 in Python 3.6. Can I do that with the changes added by this patch?

   I tried creating pickle2reducer.py like this:

   from multiprocessing.reduction import ForkingPickler, AbstractReducer
   
   class ForkingPickler2(ForkingPickler):
       def __init__(self, *args):
           if len(args) > 1:
               args[1] = 2
           else:
               args.append(2)
           super().__init__(*args)
   
       @classmethod
       def dumps(cls, obj, protocol=2):
           return ForkingPickler.dumps(obj, protocol)
   
   
   def dump(obj, file, protocol=2):
       ForkingPickler2(file, protocol).dump(obj)
   
   
   class Pickle2Reducer(AbstractReducer):
       ForkingPickler = ForkingPickler2
       register = ForkingPickler2.register
       dump = dump

   and then putting

   import pickle2reducer
   multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

   at the top of my module before

   import multiprocessing.connection

   but I still see "ValueError: unsupported pickle protocol: 3" on the server when I connect with a Python 3.6 client.

   👍React with 👍1Reacted by auxsvr
10. WillS commented on Jul 28, 2017

    

    WillSmannequin

    on Jul 28, 2017

    Mannequin

    More actions

    Just to follow up in case anyone comes across my last message later:

    I just had to change the last line from

    multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

    to

    multiprocessing.context._default_context.reducer = pickle2reducer.Pickle2Reducer()
11. pitrou commented on Mar 31, 2018

    

    pitrou

    on Mar 31, 2018

    Member

    More actions

    I'd like to know if the work here will be completed soon :-) This currently lacks documentation but also tests.

    In particular, looking at the code (and that is supported by Will S' comment above), I feel the API isn't working as intended (i.e. setting the "reducer" property won't actually change the underlying parameters since the "reduction" module is imported eagerly).
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022
13. serhiy-storchaka

    added this to Multiprocessing issueson May 20, 2022
14. vstinner

    added

    topic-multiprocessing

    on Jun 6, 2022