Replace pickle with json#1502
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1502 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 31 31
Lines 2535 2531 -4
Branches 432 432
=========================================
- Hits 2535 2531 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
23e2aeb to
333349c
Compare
|
Detailed blog post on the vulnerability and its excitability |
|
I think this has broken the ability to EDIT: Was able to fix the "object not JSON-serialisable" error by replacing For reference I am trying to use So nothing ever calls Was able to manually work around this by manually doing it with a stub server matching that of my main server instance in the other process: |
|
@judilsteve Thanks. These are problems that started after I dropped the use of Pickle, which handled all these serialization issues transparently. I'll make sure |
|
@judilsteve I do not see the crash you reported above. Can you share the code that you are using that causes this crash on line 37 of async_manager.py? Thanks. |
|
I've also run into an issue since the switch to json was made. I'm using I can manually change the package code to go from pickle to json and see that it works one way and not the other. Client handler socket.on('order_update', function (tag: string, order: Order)Prior to 5.14, both parameters are defined. After, tag is instead an array with two entries Server side # where tag is a string and order is a dict
await sio.emit("order_update", (tag, order), to=f"theplace") |
|
@shmcgough okay, yes, I need to figure out how to pass tuples over json. This change has really been a nightmare. There are lots of things that people were doing that are much harder to do without pickle! |
|
@shmcgough Would you be able to install the main branch of this repo and test your application with it? I think the support for multiple arguments via tuples is now restored. I'll cut a release in a few days, so let me know if you experience any remaining issues. Thanks! |
I should be able to test it tonight, I'll update you then. Thanks for your work on this. |
|
@miguelgrinberg I was able to test the main branch and can confirm it works! Thanks again. |
|
@shmcgough v5.15.1 is now available with this fix. |
It has been reported (credit: Ali Raza, BlueRock) that the use of the
picklepackage from the Python standard library to encode payloads that are passed between Socket.IO processes over a message queue (in a multi-server and/or external process configuration) can be exploited to create a remote code execution.For this exploit to be viable, the attacker needs to connect to the message queue directly, and push a malicious payload to it. Given that the message queue is an internal component of the server deployment that is not accessible from the public internet, the only reason for concern would be if the queue is configured to listen publicly by mistake.
To confirm that you have a secure deployment, make sure that:
I have added a section to the documentation on secure deployment of the message queue with the above recommendations.
In addition to the above, and considering that
pickledoes not provide any benefits over other serialization formats, I have decided to remove the use of pickle completely, so that any possibility of an attack, though unlikely, is eliminated. Version 5.14.0 uses JSON as the only communication format for message queue payloads (in previous versions both JSON and Pickle were supported, but Pickle was used by default).The only side effect of this change is that a mix deployment that includes servers from before and after this change may fail to communicate, because the older versions will attempt to push Pickle packets to the queue and the newer servers do not understand that format anymore. If all servers are upgraded to 5.14 or newer there shouldn't be any problems.