↑↓ selectEnter openEsc close

    1. Elbuz

    How the parser updates a price list: new products, reruns and interrupted runs

    16 min readUpdated

    The parser writes each product to the source price list once its page is parsed and picks pages itself: collected products aren't downloaded again, and an interrupted run resumes where it stopped.

    When you need this:

    • Fresh site prices and stock: with “Delete products before loading” on, each run reopens the pages.
    • Only new products: with the checkbox cleared, collected products' pages aren't opened.
    • A run broke off or was stopped: collected data stays, “Continue” finishes the rest.
    • The test shows a product the price list lacks: exactly four reasons, all in the source settings.

    Where to start: decide on “Delete products before loading” on the “Additional” tab.

    What a parser run can do

    How to update prices or pick up only new products

    The tells a new product from one it already loaded, and that decides which pages get opened during a run.

    There is only one criterion: the page address. Name, SKU and barcode play no part in it.

    Each queue task gets a key: a checksum of the lowercase address. Before adding a new task, the app checks whether the queue already has a row with the same key; if it does, no task is created. That's why the same product isn't downloaded twice, even if it sits in three sections of the site.

    The “Delete products before loading” checkbox relies on the same mechanism (“Additional” tab, “Data processing” group, on by default):

    • on: the source's products are deleted before the start, and the run collects everything again;
    • cleared: the addresses of all existing products in the price list go into the queue marked “Ignore”, and any task for such an address is discarded. Their pages simply aren't opened in this run. That's how the promise that only new products will be loaded works.

    A cleared checkbox has a side effect people don't think of right away: prices and stock of existing products won't be updated. Their pages weren't opened, so there's no new data. This mode is good for picking up new products, not for tracking prices.

    The second side effect: if the site changes a product's address, the parser treats it as a new product. The old one stays in the price list next to the new one.

    What stays in the price list if a run breaks off or is stopped

    Products are written to the price list one at a time: once a page is parsed, a row appears in the price list along with its specifications, photos, videos and files. There's no “save everything at the end” step, so a break leaves exactly the products whose pages were parsed in time.

    Price list operations, such as markup, stock rules and others, are not applied to the collected rows until the run is over. They start in three cases:

    • the queue has no unfinished tasks left: the run reached the end;
    • you clicked “Cancel” in the import window: operations are applied to what has been collected, and the log shows “Apply different operations to products”;
    • you clicked “Stop parsing data” in the “Parsing log” window: the price list's unfinished tasks are removed from the queue, your other unfinished tasks are marked as canceled, and operations are applied to what was collected.

    The “Parsing log” window opens with the “Parsing log” button on the “Additional” tab of the “Site data import settings” window, or from the source card: “Products” tab → additional operations menu → “Parsing log”. “Stop parsing data” is an icon button in its header; the label appears when you hover over it.

    The “Website parsing log” window with the “Stop parsing data” tooltip on the header icon and the job grid below
    1Stop parsing data icon
    2Parsing job grid
    1. Stop parsing data — the icon in the window header that stops collection; hover shows the tooltip, do not click it unless you really want to stop the run.
    2. Job grid — one row per page-loading job with its status, HTTP response code and response time (empty here because no run is active).

    If the run broke off on its own, because of a server failure or because the window was closed while your browser was opening the pages, operations don't start. The price list holds rows with site prices and no markup until the run is resumed and completed.

    Every branch except the bottom one ends with price list operations: markup, stock and others. The bottom branch, a break on its own, leaves the collected data without them until the run is completed
    no tasks left in the queue“Cancel” in the importwindow“Stop parsing data”in the “Parsing log” windowthe run's request to theserver returned an errorwhen the browser opensthe pagesserver failure or closedwindowwhen the browser opensthe pagesThe run stoppedHow exactlyPrice list operations on collecteddata:markup, stock and othersThe price list's unfinished tasksare removed from the queueOperations don't startthe price list has site prices withoutmarkupTasks wait in the queuefor “Continue”
    • Start or end
    • Action or result
    • Decision: the path depends on the answer

    Besides applying operations, “Cancel” in the import window removes the price list's unfinished tasks from the queue, both when the browser opens the pages and when “Use server to load data” is on.

    In server mode, “Cancel” works the same as “Stop parsing data”: the server stops the run and won't take the next batch, your other unfinished tasks are marked “Canceled”, the log shows “Operation canceled by the user”, and price list operations are applied to what was collected. You can't resume a canceled run with “Continue”; only “Start download” is left.

    How to finish an interrupted run: “Continue” or “Start download”

    What happens to individual pages:

    • a page that doesn't respond goes back to the queue for a retry: after 35 seconds when the browser opens the pages, after 150 seconds via the Elbuz server, which covers the server's longest page wait of up to 120 seconds in “Premium parsing (anti-bot bypass)” mode. After five failed attempts (twenty for hotline), the task is closed with the note “no response for more than 35 sec, attempt 5 of 5”. The note is the same in both modes, and the product from that page won't be in the price list;
    • a page that returned a 404 code is closed without retries;
    • tasks the run hasn't reached yet stay in the queue. Next time, the import window shows “There are unfinished jobs” with their count above the buttons, plus a “Continue” button.

    After a break, the two buttons work differently:

    • “Continue” takes up the same unfinished tasks. Nothing is deleted, and what was collected stays in the price list.
    • “Start download” starts the run over: it clears the task queue and the previous run's log, and the rest depends on “Delete products before loading”. If it's on, the price list's collected products are deleted and collected again. If it's cleared, addresses of products already collected go into the queue marked “Ignore”: they aren't collected twice, and only the products the interrupted run didn't reach are picked up.
    The left branch erases nothing. The right one clears the queue first, and what happens to products already collected depends on “Delete products before loading”
    “Continue”“Start download”onclearedThe run broke off,tasks are left in the queueWhich button was clickedThe same unfinished tasks aretaken upcollected data stays in the price listThe task queue and the previousrun's log are cleared“Delete productsbefore loading”Price list products are deletedand collected againAddresses of collected products areignoredonly the ones not reached arecollected
    • Start or end
    • Action or result
    • Decision: the path depends on the answer

    A product can be duplicated in one case worth knowing. A product is first written to the price list, and only at the next step is its task marked as done. If the break hit exactly between these two steps, the page is opened again after “Continue”, and a second row with the same address appears in the price list.

    It stays until the run finishes: at completion, rows with the same address are removed and the first one is kept.

    How many pages the parser takes at a time, and its limits

    You can't calculate in advance how long a run will take: that depends on the site. But the limits it works within are set as numbers in the app.

    Via the Elbuz server (“Use server to load data” is on):

    • pages are loaded in batches: by default 100 addresses in 10 threads (when the scheduler continues in the background, 300 addresses). The whole batch gets 30 seconds per address in it;
    • one page is awaited for up to 35 seconds, up to 55 seconds through your own proxies, and up to 120 seconds in “Premium parsing (anti-bot bypass)” mode;
    • with “Use server for rendering JavaScript” on (without your own proxies), batches are smaller: 4 addresses in 2 threads;
    • the run happens on the server, and the window only shows the log. Closing the window doesn't stop the run. In one launch, the server goes through up to 14 batches in a row, and a background scheduler task continues the remaining unfinished tasks;
    • you can follow progress in the log: each batch gets a “Loading data from links” row with the number of addresses in the batch and the number of unfinished tasks. Before each batch, the log names the mode: “Page loading mode:” followed by “server proxy”, “direct (no proxy)”, “user proxy” or “enhanced mode (protection bypass)”.

    Via the browser (the server isn't used): the import window itself runs the crawl. Every 3 seconds it takes one address from the queue and passes it to the Elbuz extension in the browser. Once the window is closed, no new pages are taken, and unfinished tasks wait for “Continue”.

    There's no limit on the number of products in one run. A separate number applies only to “Search by catalog products” mode: no more than 100,000 active catalog products are taken for searching the site.

    Why a product from the test didn't make it into the price list

    It happens: the rule test shows a full set of fields, but the product isn't in the price list. There are exactly four reasons, and all of them trigger after the page is parsed, at the moment of saving.

    1. In listing mode, there's no name or address. Such a row is skipped silently (listing fields).
    2. A product import rule was triggered. If the source has item selection conditions, every product found is run through them, and one that doesn't match isn't saved.
    3. The price is out of range. The “Import products with price — starting from / to” fields in the source card, “Prices” section. A product priced outside these limits isn't written.
    4. The stock status is on the stop list. In the “Availability statuses” directory, every status has a “Do not import products with this status (site parser)” toggle. Turn it on for “Out of stock”, and products with that text in “Stock status (from price list)” won't get into the price list. A separate rule, {EMPTY} (or {ПУСТО}; both spellings are accepted), filters out products with an empty stock status. By default, this toggle is off for all statuses.

    What to check after the first run

    The items point to what most often goes wrong; each one is covered in detail in What exactly the parser takes from a product page.

    The first run is finished

    Scenarios

    Track prices and stock on a supplier's site

    What to do. In the “Product card” group, fill in the “Price” and “Stock status (from price list)” selectors. On the “Additional” tab, leave “Delete products before loading” on.

    What you get. Each “Start download” deletes the source's products and opens the pages again, so prices and stock in the price list come from this run. When the queue has no tasks left, price list operations are applied to what was collected: markup, stock rules and others.

    What won't happen. Catalog product cards won't change: new prices get there through a base catalog update. If the run broke off on its own, operations won't start, and the price list will keep site prices without markup (what to do after a break).

    Pick up only new products from a site

    What to do. Clear “Delete products before loading” and run “Start download”.

    What you get. Addresses of products already in the price list go into the queue marked “Ignore”, and no tasks are created for them. Only pages with new addresses are opened.

    What won't happen. Prices and stock of products already collected won't be updated: their pages weren't opened in this run. A product whose address the site changed comes in as a new row, and the old one stays next to it.

    What to do when the price list looks wrong after a run

    What you see in the price list after a run and what to do about it
    What you seeWhyWhat to do
    The run finished, but existing products still have the old prices “Delete products before loading” is cleared: pages of existing products weren't opened in this run Turn the checkbox on if you need fresh prices (about new and old products)
    The price list has two identical products The site changed the product's address. To the parser it's a new product: it tells items apart only by address Delete the old item, or run with “Delete products before loading” on (about new and old products)
    The price list has site prices without markup, and stock status isn't recognized The run broke off on its own, and price list operations didn't start: they only run at the end of a run Click “Continue” and let the run finish (what stays after a break)

    FAQ

    Will prices update if I clear “Delete products before loading”?

    No. Addresses of existing products go into the queue marked “Ignore”, and their pages aren't opened in this run. Only products from new addresses come in (more details).

    Can I close the window while a run is in progress?

    Via the Elbuz server, yes: the run happens on the server, and the window only shows the log. When the browser opens the pages, a closed window takes no new pages, and unfinished tasks wait for “Continue”.

    How many products can I collect in one run?

    There's no limit on the number of products. A separate number applies only to “Search by catalog products” mode: no more than 100,000 active catalog products are taken for searching the site.

    Can I resume a run canceled with the “Cancel” button?

    No. “Cancel” removes the price list's unfinished tasks from the queue, so there's nothing left for “Continue” to do. Only “Start download” remains (more details).

    Why are there two rows with the same product address in the price list?

    The break happened between writing the product and marking its task as done, and the page was opened again after “Continue”. The second row is removed when the run finishes, and the first one is kept (more details).

    Where to find it and who has access

    Menu
    Directories → Processing price lists → List of price lists
    Run checkboxes
    source row → “Setup” → “Additional” tab
    Start
    “Import data from the site” window: “Start download”, “Continue”, “Cancel”
    Parsing log
    “Additional” tab → “Parsing log”, or source card → “Products” → menu
    Access right
    the “Price lists” window (in the menu: “List of price lists”)

    How to set up a parser source and run it manually or on a schedule is covered in Site parsing. Which fields a run collects is covered in What exactly the parser takes from a product page.

    Related topics

    Was this article helpful?

    Missing a feature? Tell us.

    We will add it for free if other users need it too.

    If only you need it, we will agree on it separately.

    Article contents

    Suggest an improvement

    Try Elbuz free