Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language.

0
votes
1answer
31 views

multiprocessing.Queue() into dictionary with large input size

I have a list called encrypted_messages containing 126018 strings. Every string is an encrypted message. I also have a function called decipher which, given a string and a key (an integer from 9 to 15,...
0
votes
1answer
38 views

How do I speed up an optimized CPU-bound process that runs within a parallelized program in Python?

A Python program of mine uses the multiprocessing module to parallelize iterations of a search problem. Besides doing other things, each iteration loops over a CPU-expensive process that is already ...
0
votes
0answers
22 views

Keep a class running in a python subprocess or thread or process

I am using Squish to automate a Qt-based GUI application. I look up qt objects in the application recursively. Since it's time-intensive, I would like to cache objects once found for later reuse. I ...
2
votes
2answers
83 views

Python Multiprocessing Pipe is very slow (>100ms)

I'm currently writing an image processing program in Python 3.x that needs to process frames in real-time (30 FPS) with low-latency (<60ms). I have 1 parent process that reads frames and sends them ...
0
votes
1answer
33 views

Is there a better way of doing multiprocessing than this? Preferably faster and not so hard on the memory

This is my code. I don't think this is the best way to do this task, Can you kindly help me? Thanks in advance. It responded once but when I ran it again it kept on running and didn't give out the ...
0
votes
0answers
10 views

multiprocessing.Queue\list (for use with writing to database in separate process) slower than session.add\commit in main thread

I believe This Answer might be what I'm running into, but I want to confirm and see if there's a better way to do what I'm wanting. In short, I have some pretty heavy calculations that I've got spread ...
0
votes
1answer
27 views

Python process TypeError: cannot pickle '_thread.lock' object

Ran into a wall here a little. Basically I've got a python program which can launch multiple bots from within, and I'm trying to launch each bot as a separate process. So i've created a class that ...
0
votes
0answers
25 views

Python multiprocessing, can't pickle thread.lock (pymongo.Cursor)

First, let me assure you I read all the relevant answers and they don't work for me. I am using multiprocessing Pool to parallelize my data creation. I am using Mongodb 5.0 and pymongo client. As you ...
0
votes
0answers
11 views

Pool apply function hangs and never executes

I am trying to fetch Rally data by using its python library pyral. Sequentially the same code works, but its slow. I thought of using python multiprocess package, however my pool.apply method gets ...
0
votes
0answers
36 views

How to classify very big csv data file based on classifier column by using parallelization?

I have below classification file. all_class_file.txt abc def ghi jkl mno pqr stu vwx yz1 adg trv full_data_file.txt HEADER = "dt","id","name","class","...
0
votes
1answer
27 views

Why does python multiprocessing Pool.map() RAM consumption increase over time?

Simplified scenario tl;dr: See code snippet below. Here I just explain what it's supposed to do and why. I have an object example_class_instance of a class ExampleClassWithTonOfData that holds a ton ...
0
votes
2answers
19 views

Python multiprocessing.Pool outputs nothing to file

from multiprocessing import Pool f = open("./testing.txt", "a") def predefined_fp(s: str): ''' This function doesn't actually write anything to file. ''' # ...
0
votes
0answers
27 views

Why child process is not printing the statement?

I am running the do_work function 5 times on different processors. I am aware that each process has different memory space and it runs the entire piece of this code except what is written inside if ...
0
votes
0answers
13 views

CPU: Problems with Python module torch.multiprocessing

Let me provide you with a MWE of a simple test of torch.multiprocessing: import torch import numpy as np import torch.multiprocessing as mp def square(i, x, queue): print('In process {}'.format(i)...
0
votes
2answers
31 views

Multiprocessing + Flask OSError: [Errno 48] Address already in use / OSError: [Errno 38] Socket operation on non-socket

I am trying to build a Flask app that runs a function that eventually utilizes the multiprocessing library. import time from flask import Flask, request import multiprocessing as mp app = Flask(...

15 30 50 per page
1
2 3 4 5
270