Fix multithreading by skinkie · Pull Request #157 · MMTIS/badger

skinkie · 2026-07-30T14:14:08Z

… must now always have a TXN argument, ty cleanups.

skinkie · 2026-07-31T01:36:36Z

root cause found, insert_database still called inherited insert function.

In order to faciliate a lighter importing pattern, we are moving out functions that are not using domain.netex.model. This allows the multiprocessing code to import more efficiently.

With Google Antigravity I was able to find the root cause of the performance degredation of multiprocessing. Rest are improvements. 1. import domain.netex.model takes long, with spawn it is executed every time that a process is created causing significant overhead. 2. Having a multithreaded consumer that is not dependent on anly NeTEx entities won't have startup costs when spawned. The alternative is to have it not spawned, but forked too. 3. There is noticable IPC overhead, by batching the queue entries we lower this overhead. 4. By not making the MdbxStorageMP part of the forked processes, they cannot conflic. 5. By solving the the problem where insert_database was not using the multithreaded insert (BUG). We significantly improve the total outcome. 6. A netex_to_db_mp has been added, which does the most naive insert (thus great for EPIP LINE OFFER, but swiss_to_db_mp still relevant).

ue71603

it works when I run

python -m tools.script_runner ./tools/tool_scripts/script_regression.txt reg.log all

Do you want me to run something else?

ue71603 · 2026-07-31T17:17:10Z

is still single-treaded use on my machine.

ue71603 · 2026-07-31T17:41:14Z

If I need some other parameters, environment, then tell me. Also, the multi-threaded version in my view is not described anywhere. and no pointers on how to use it with script_runner is provided.

skinkie · 2026-07-31T19:23:46Z

If I need some other parameters, environment, then tell me. Also, the multi-threaded version in my view is not described anywhere. and no pointers on how to use it with script_runner is provided.

To validate the code works on your system:
uv run -m unittest

ue71603

problem I got with running swissinterchange:


(badger) PS D:\development\github\badger> python -m tools.script_runner  ./tools/tool_scripts/script_regression.txt reg.log  swissinterchange
2026-07-31 23:34:48,215 - script_runner - INFO - swissinterchange - step: 1: clean_tmp d:/aux_testing_processing/swissinterchange
2026-07-31 23:34:48,219 - script_runner - INFO - Command 'clean_tmp' executed for folder: d:/aux_testing_processing/swissinterchange

2026-07-31 23:34:48,220 - script_runner - INFO - swissinterchange - step: 2: download_input_file d:/aux_testing_processing/swissinterchange
2026-07-31 23:34:48,220 - script_runner - INFO - File exists already. Will use: d:/aux_testing_processing/swissinterchange\swiss_interchange.zip
filepaths: d:/aux_testing_processing/swissinterchange\swiss_interchange.zip
2026-07-31 23:34:48,220 - script_runner - INFO - Command 'download_input_file' executed for url: https://github.com/user-attachments/files/18870701/swiss_interchange.zip

2026-07-31 23:34:48,221 - script_runner - INFO - swissinterchange - step: 3: conv.netex_to_db_mp d:/aux_testing_processing/swissinterchange\swiss_interchange.zip d:/aux_testing_processing/swissinterchange/03.mdbx
module 'multiprocessing.context' has no attribute 'ForkProcess'
2026-07-31 23:34:57,447 - script_runner - ERROR - module 'multiprocessing.context' has no attribute 'ForkProcess'
Traceback (most recent call last):
  File "D:\development\github\badger\tools\script_runner.py", line 413, in main
    result = load_and_run(script_name, script_args)
  File "D:\development\github\badger\tools\script_runner.py", line 75, in load_and_run
    mod = importlib.import_module(module_name)
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\importlib\__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "D:\development\github\badger\conv\netex_to_db_mp.py", line 6, in <module>
    from storage.mdbx.core.implementation_mp import MdbxStorageMP
  File "D:\development\github\badger\storage\mdbx\core\implementation_mp.py", line 11, in <module>
    class MdbxStorageMP(MdbxStorageQueue):
    ...<29 lines>...
            return super().__exit__(exception_type, exception_value, exception_traceback)
  File "D:\development\github\badger\storage\mdbx\core\implementation_mp.py", line 12, in MdbxStorageMP
    writer: mp.context.ForkProcess
            ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'multiprocessing.context' has no attribute 'ForkProcess'
2026-07-31 23:34:57,451 - script_runner - INFO - Execution time: 9.2 seconds for swissinterchange - step: 3: conv.netex_to_db_mp d:/aux_testing_processing/swissinterchange\swiss_interchange.zip d:/aux_testing_processing/swissinterchange/03.mdbx

2026-07-31 23:34:57,452 - script_runner - ERROR - Script conv.netex_to_db_mp returned an error. Terminating the block of scripts: swissinterchange
(badger) PS D:\development\github\badger>

with the unit tests I get:

and


Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2025.1.1\plugins\python-ce\helpers\pydev\pydevd.py", line 1570, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2025.1.1\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/Program Files/JetBrains/PyCharm Community Edition 2025.1.1/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py", line 38, in <module>
    sys.exit(main(argv=args, module=None, testRunner=unittestpy.TeamcityTestRunner,
             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  buffer=not JB_DISABLE_BUFFERING))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\unittest\main.py", line 103, in __init__
    self.parseArgs(argv)
    ~~~~~~~~~~~~~~^^^^^^
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\unittest\main.py", line 142, in parseArgs
    self.createTests()
    ~~~~~~~~~~~~~~~~^^
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\unittest\main.py", line 153, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
                                                   self.module)
                                                   ^^^^^^^^^^^^
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\unittest\loader.py", line 207, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
              ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Users\Gm\AppData\Local\Programs\Python\Python313\Lib\unittest\loader.py", line 137, in loadTestsFromName
    module = __import__(module_name)
ValueError: Empty module name
python-BaseException

skinkie · 2026-08-01T03:35:41Z

@ue71603 I'll test windows later.

It fails afterwards, because the script_runner fix is in a different branch, but this gets the multi-core going. MistralAI said to do it like that.

ue71603 · 2026-08-01T11:14:57Z

some small eror (but perhaps this is more mypy that is not happy). context is defined in the outer function. And it shows how windows get happy to load.

skinkie · 2026-08-01T13:57:39Z

+    if sys.platform == 'win32':
+        context = mp.get_context('spawn')
+    else:
+        context = mp.get_context('fork')
+    writer: context.Process


@ue71603 This is invalid, because you can't setup the variable here.

skinkie · 2026-08-01T14:17:32Z

@ue71603 please test again. Unittests work for me on Windows now

ue71603 · 2026-08-01T20:51:33Z

uv run -m unittest


(badger) PS D:\development\github\badger> uv run -m unittest
warning: `VIRTUAL_ENV=D:\development\github\reference\gtfs-netex-test\.venv\badger` does not match the project environment path `.venv` and will be ignored; use `--active` to target the active environment instead
Uninstalled 2 packages in 266ms
░░░░░░░░░░░░░░░░░░░░ [0/3] Installing wheels...                                                                                                                                                                                     warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 3 packages in 328ms
D:\development\github\badger\.venv\Lib\site-packages\xsdata\formats\dataclass\serializers\config.py:48: DeprecationWarning: Setting `pretty_print` is deprecated, use `indent` instead
  warnings.warn(
..........[load_object_by_reference] fallback prefix-scan for ref 1
.................
----------------------------------------------------------------------
Ran 27 tests in 20.182s

OK
(badger) PS D:\development\github\badger>

ue71603 · 2026-08-01T21:00:55Z

I changed swiss4

You still tryy to do a fork, this does not work for windows:


 {
       "block": "swiss4",
       "download_url":"https://github.com/user-attachments/files/18593829/swiss4.zip",
        "description":"Reduced Swiss data set for tests, still a bit large",
        "scripts": [
        {"script": "clean_tmp", "args": "%%dir%%"},
        {"script": "download_input_file", "args": "%%dir%%"},
        {"script": "conv.netex_to_db_mp", "args": "[%%inputfilepath%%] %%dir%%/03.mdbx"},
        {"script": "conv.epip_db_to_db", "args": "%%dir%%/03.mdbx %%dir%%/04.mdbx"},
        {"script": "conv.epip_db_to_xml", "args": "%%dir%%/04.mdbx %%dir%%/05-epip.xml.gz"},
        {"script": "conv.netex_to_db", "args": "[%%dir%%/05-epip.xml.gz] %%dir%%/06.mdbx"},
        {"script": "conv.gtfs_db_to_db", "args": "%%dir%%/06.mdbx %%dir%%/07.mdbx"},
        {"script": "conv.gtfs_db_to_gtfs", "args": "%%dir%%/07.mdbx %%dir%%/08-%%block%%-gtfs.zip"},
        {"script": "tools.gtfs_validator", "args": "%%dir%%/08-%%block%%-gtfs.zip %%dir%%"},
        {"script": "tools.gtfs_map_visualisation", "args": "%%dir%%/08-%%block%%-gtfs.zip map_file=%%dir%%/10-%%block%%-map.html max_routes=1"}
         ]
  },

pushed a fix

is working now:

(not fully related, but it needs to be done anyhow) and doesn't affect your code.

skinkie · 2026-08-02T10:10:21Z

@ue71603 due to how windows works, you don't get the benefit of a fork. Due to how python with spawn works, you have to import all code again. The future where Python would support actual multithreading (python without the global interpreter lock) looks promissing, but not enabled by default. Hence only python >=3.13 versions explicitly removed it at compilation time will benefit. We can and should put migration to threads on the backlog obviously.

skinkie added 2 commits July 30, 2026 16:11

Remove empty argument from db_write.insert_objects_on_queue, db_names…

707180c

… must now always have a TXN argument, ty cleanups.

Unchecked cleanups

17891b6

skinkie self-assigned this Jul 30, 2026

skinkie added enhancement New feature or request hygiene labels Jul 30, 2026

skinkie added 2 commits July 30, 2026 17:54

Fix unittest

d14f214

Multiprocessing is still slower than uni processing

6250448

skinkie added 2 commits July 31, 2026 15:23

RelationKeyCodec and const

8e908ce

In order to faciliate a lighter importing pattern, we are moving out functions that are not using domain.netex.model. This allows the multiprocessing code to import more efficiently.

skinkie marked this pull request as ready for review July 31, 2026 13:56

skinkie requested a review from ue71603 July 31, 2026 13:56

ue71603 approved these changes Jul 31, 2026

View reviewed changes

Give one script netex_to_db_mp

cbfcfc8

ue71603 reviewed Jul 31, 2026

View reviewed changes

Fix #161

38d72ca

make it work for windows

19e376b

It fails afterwards, because the script_runner fix is in a different branch, but this gets the multi-core going. MistralAI said to do it like that.

skinkie commented Aug 1, 2026

View reviewed changes

skinkie added 2 commits August 1, 2026 16:04

Make the stuff look better, and not fail.

660e994

Fix test in win32

4f1da09

skinkie requested a review from ue71603 August 1, 2026 14:23

fix problem and test with swiss4

210829e

ue71603 approved these changes Aug 1, 2026

View reviewed changes

Update log_analyser.py

823d910

(not fully related, but it needs to be done anyhow) and doesn't affect your code.

skinkie merged commit da6ee7b into binary_relation_serializer Aug 2, 2026
4 checks passed

skinkie deleted the fix_multithreading branch August 2, 2026 10:11

Uh oh!

Conversation

skinkie commented Jul 30, 2026

Uh oh!

skinkie commented Jul 31, 2026

Uh oh!

ue71603 left a comment

Choose a reason for hiding this comment

Uh oh!

ue71603 commented Jul 31, 2026

Uh oh!

ue71603 commented Jul 31, 2026

Uh oh!

skinkie commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ue71603 left a comment

Choose a reason for hiding this comment

Uh oh!

skinkie commented Aug 1, 2026

Uh oh!

ue71603 commented Aug 1, 2026

Uh oh!

skinkie Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skinkie commented Aug 1, 2026

Uh oh!

ue71603 commented Aug 1, 2026

Uh oh!

ue71603 commented Aug 1, 2026

Uh oh!

skinkie commented Aug 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

skinkie commented Jul 31, 2026 •

edited

Loading

skinkie Aug 1, 2026 •

edited

Loading