Task Queue
The work of A-Parser is based on a task queue - you add tasks and they are executed in order, thus allowing you to load the scraper with work for a long time and not monitor when it will finish.
The main advantages of the task queue in A-Parser
- Support for executing multiple tasks simultaneously
- Pausing and full stop of task execution
- Saving intermediate results of tasks, which allows to continue work from the point of scraper shutdown (or computer/server shutdown)
- Moving individual tasks in the queue
- The ability to edit already created tasks, as well as copying tasks for reuse
- Completed tasks move to the completed tasks queue, where you can view the results of each task and download the result
Work algorithm
Working slots for tasks
The number of tasks working simultaneously is regulated by working slots, this parameter is specified in the general settings (Settings -> General settings -> Maximum active tasks). The default value is 1, i.e., one task working simultaneously, others will wait their turn.
Task statuses in the queue
Each task has its own status, here is a list of possible states:
| Status | Description |
|---|---|
waitSlot | Waiting for a free slot to execute the task, a new task always enters the queue with this status |
work | The task is being executed |
paused | The task is paused, while it holds the working slot, not allowing the next tasks to start |
stopped | The task is stopped, thereby freeing the working slot |
completed | The task is completed - all requests have been processed, the task moves to the completed tasks queue, the working slot is freed |
When setting the Maximum active tasks parameter to more than one, it should be taken into account that the number of threads will increase, as will the overall resource consumption by the scraper.
To achieve maximum working speed, always use only one task, with an optimal number of threads selected for a specific computer/server.