Multiprocessing python


I have a list of lists something like [[1,2,3], [4,5,6]------]]. I want to implement multiprocessing so that I can parallely process a group of lists. Currently I am using Threadpool(running a for loop on list of lists) but it does not seem to provide a reliable answer is there any way that I can do this.

I am completely new to multithreading in Python

The title of your post is multiprocessing, which is not multithreading, what about multithreading are you looking to understand?

What answer are you looking for?
What does ''reliable" mean exactly?
What code did you do to cause it?

Can you share minimal data (you can generate a fake example, no need to share your real data), and the code you are using so others can recreate your issue, then there can be some progress on this question.

Sorry I am using Threadpool of multiprocessing. I have a list of about 7000 objects and I am doing some preprocessing (transformation) on the object.
Initially I have used for loop (squential) It was taking me 1 hr to process completely.
When I am using Threadpool with (100) threads then I see a lot of objects are getting skipped in preprocessing (I am collecting any skipped / errored objects using Try except block in a list).

For loop --> unprocessed objects are 500 (out of 7400)
threadpool --> umprocessed objects varies from (2000 - 4000)

Given this information, I am unable to help further other than take random jabs.

Why would there be unprocessed objects in the for loop?
What about removing dirty data? (throw away the 500/7400 before comparing for loop to threadpool, should be able to identify exactly which 500 since it’s sequential execution).
Why use threadpool rather than pool? Threadpool uses threads while pool spawns new processes. I’m no expert but the conventional wisdom is CPU bound jobs should use multiprocessing not multithreading.
Does using concurrent.futures.processpoolexecutor help?
Why are there 2000-4000 (4-8x more) unprocessed objects in threadpool than for loop? Is it because your processing code is not threadsafe so some processing is interrupting/mutating other objects during/before their processing?

Thanks for suggestion will try and let you know