Instead of running ./manage.py import --all in sequence
We can use multiprocessing and try to use all CPU power.
suggestions:
-
we can use Celery and schedule tasks and run easily importers in parallel
(another benefit: we can run periodic Tasks and make importers synchronize )
-
we can make it simple and use some code from scancode.
this could be a solution for #833
Instead of running
./manage.py import --allin sequenceWe can use multiprocessing and try to use all CPU power.
suggestions:
we can use Celery and schedule tasks and run easily importers in parallel
(another benefit: we can run periodic Tasks and make importers synchronize )
we can make it simple and use some code from scancode.
this could be a solution for #833