Task queue
The work of A-Parser is based on a task queue - you add tasks and they are executed in order, allowing you to load the scraper with work for a long time and not worry about when it will finish.
The main advantages of task queue in A-Parser
- Support for executing multiple tasks simultaneously
- Pausing and stopping task execution
- Saving intermediate results of task execution, which allows you to continue work from the moment the scraper was turned off (or the computer/server was turned off)
- Moving individual tasks in the queue
- Ability to edit already created tasks, as well as copy tasks for reuse
- Completed tasks go to the completed tasks queue, where you can view the results of each task and download the result
The algorithm of work
Working slots of tasks
The number of simultaneously working tasks is regulated by working slots, this parameter is specified in the general settings (Settings -> General settings -> Maximum active tasks). The default value is 1, i.e. one task is working simultaneously, others will wait for their turn.
Task statuses in A-Parser
Each task has its own status, here is a list of possible states:
| Status | Description |
|---|---|
waitSlot | Waiting for a free slot to execute the task, a new task always enters the queue with this status |
work | Task is being executed |
paused | Task is paused, while holding a working slot, preventing the next tasks from starting |
stopped | Task is stopped, freeing up a working slot |
completed | Task is completed - all requests are processed, the task goes to the completed tasks queue, the working slot is freed |
When setting the Maximum active tasks parameter to more than one, it should be taken into account that the number of threads will increase, as well as the overall resource consumption by the scraper.
To achieve maximum performance, always use only one task with the optimal number of threads for a particular computer/server.