Configuration via Interface

A-Parser allows you to set the configuration in a declarative style, as well as use the interface to create and edit presets without the need to modify the source code of scrapers

static defaultConf

static defaultConf = {
    version: '0.0.1',
    results: {
        flat: [
            ['title', 'Title'],
        ],
        arrays: {
        }
    },
    results_format: "Title: $title\n",
    exampleKey: 'value',
};

The scraper's default configuration, the config will be available in the class object through the this.conf property, the following fields are mandatory:

• results - describes in a declarative style the results that this scraper can return

• results_format - sets the default result format

All other fields are optional, there is the following list of parameters that affect the scraper's operation:

Parameter Name	Type	Description (default value)
timeout	number	Maximum request waiting time in seconds (60)
useproxy	boolean / 0 / 1	Determines whether to use a proxy (1)
max_size	number	Maximum file size of the result (1 * 1024 * 1024)
proxyretries	number	Number of attempts for each request, if a request fails after the specified number of attempts it is considered unsuccessful and is skipped (10)
requestdelay	number / string	Delay between requests in seconds (0). You can also set a random value in a range, for example 10,30 - delay from 10 to 30 seconds
proxybannedcleanup	number	Proxy ban time in seconds (600)
pagecount	number	Number of parsing pages (1)
parsecodes	{ [code: string]: any }	Response codes for requests that will be considered successful (any)
queryformat	string	Query format ($query)

note

You can also set arbitrary fields that can be available for editing through the interface

static editableConf

This setting defines a list of configuration fields that can be edited through the interface. There are the following types of fields in the interface:

• textfield - a field for arbitrary input of numeric and string values
• checkbox - a flag with states on/off
• combobox - a dropdown with a choice of one or several values
• selecting multiple values is set through the option { multiSelect: 1 }

editableConf is an array, each element of which describes the corresponding configuration field:

static editableConf: [
    ...[
        fieldName: string,
        fieldConfig: [
            fieldType: 'textfield' | 'combobox' | 'checkbox',
            fieldLabel: string,
            fieldOptions?: {},
            ...fieldValues: [fieldValue: any, valueTitle: string][]
        ]
    ][]
];

Example of declaring editable fields:

static get editableConf() {
    let editableConf: typeof BaseParser.editableConf = [
        ['device',
            ['combobox', 'Device',
                ['desktop', 'Modern desktop computer (Windows 10, Chrome 84)'],
                ['mobile', 'Mobile device (iPhone X, iOS 11)']
            ]
        ],
        ['pagecount', ['combobox', 'Pages count']],
        ['linksperpage',
            ['combobox', 'Links per page',
                [10, '10'],
                [20, '20'],
                [30, '30'],
                [50, '50']
            ]
        ],
    ];

    for (let page = 1; page <= 25; page++)
        editableConf[1][1].push([page, page]);

    return editableConf;
}

note

Note that in this example, a getter method is used for editableConf, which allows for additional processing, such as generating a list of pages. For simpler cases, you can set a static class property, similar to defaultConf

static parserOptions

parserOptions is an alternative way of setting options, the list of options is displayed as additional items in the scraper's context menu

Declaring options works similarly to editableConf:

static parserOptions: [
    ...[
        fieldName: string,
        menuTitle: string,
        fieldConfig: [
            fieldType: 'textfield' | 'combobox' | 'checkbox',
            fieldLabel: string,
            fieldOptions?: {},
            ...fieldValues: [fieldValue: any, valueTitle: string][]
        ]
    ][]
];

Example of declaring additional fields:

static parserOptions: typeof BaseParser.parserOptions = [
    ['parseAll', 'Parse all results',
        ['checkbox', 'Parse all results']
    ],
    ['parseLevel', 'Parse related to level',
        ['combobox', 'Parse Related to level', [1, 1], [2, 2], [3, 3]]
    ],
];