↑↓ selectEnter openEsc close

    1. Elbuz

    How to run site parsing manually and on a schedule

    17 min readUpdated

    Elbuz crawls a site from a page queue, then applies markup, stock, and other price list operations to the collected products. A run starts with a button or on a server schedule, with no app window open.

    When you need this:

    • Collect a site now — “Import” opens the run window and starts the crawl.
    • Update prices nightly — in server mode, the schedule runs it unattended.
    • The last run broke off — “Continue” finishes the queue, clearing nothing.
    • The site blocks the server — “Directly, without a proxy”, own proxies, or a bypass service.

    Where to start: the “Import” button in the right panel of “List of price lists”.

    What running a parser can do

    • Manual run, pause, and resume — for one or for all site parsers one after another.
    • Two modes — pages are opened either by your browser through the extension or by the Elbuz server in the background.
    • New products only — pages of products you already have aren't opened during the run.
    • Schedule — daily or weekly runs with nobody involved, in server mode.
    • Page loading mode — through a proxy, directly, or through a protection-bypass service.

    How to run parsing manually

    For a parser source, the “Import” button in the right panel header doesn't open a file import. It opens the “Import data from the site” window, and the crawl starts right away, with no extra click. The same action is on the “Products” tab of the source card, where the button is also labeled “Import”. Most of the window is the operation feed, which fills up as the work goes on. In the header:

    • “Start download” — creates the jobs and starts the crawl;
    • “Continue” — appears when unfinished jobs are left from last time; next to it are the counter “There are unfinished jobs N” and the hint “Click ‘Continue’ to resume.”;
    • “Pause” — visible only while running; the feed shows “Job paused”, and the queue stays where it is;
    • “Cancel” — also visible only while running; the feed shows “Job canceled”, and then the run ends;
    • the “working” label — while it pulses, the run is alive; when feed lines are sparse, there's no other way to tell work from a hang;
    • the parsing log icon.

    What you see in the header depends on whether any unfinished jobs are left from last time. If not, the crawl is already running: the header shows only “Pause”, “Cancel”, and “working”. If there are, the crawl hasn't started, and the header shows “Start download” and “Continue”, the counter “There are unfinished jobs N”, and the hint below them.

    What happens next depends on who opens the pages: your browser or the server.

    Browser mode works without the server module. Every three seconds, the window asks the server for a batch of jobs and passes them to the Elbuz extension, which opens the pages and returns their markup. This has two consequences worth knowing up front: without the extension nothing is collected, and closing the window stops the run. A run counts as finished after five empty responses in a row.

    Browser mode: the extension opens each page in your browser, and the run window passes it to the server. Closing the window breaks this chain at the very first arrow
    SiteElbuz extensionElbuz serverRun windowparsing by rules, new jobs in the queueloop[every 3 seconds while the window is open]five empty responses in a row — run finishedgive me a batch of jobspage URLsopen these URLsopens the page in a browser tabpage markupmarkupmarkup for parsing
    • Request or action
    • Reply
    • loop — repeats
    • Note

    Server mode is turned on with the “Use server to load data” checkbox and works only if the matching module is activated in your account. The Elbuz server then downloads the pages, 100 URLs at a time in 10 threads (20 threads in “Directly, without a proxy” mode), and the window only shows the feed. You can close it.

    The “Install extension” link appears in the settings window when the parser's “Use server to load data” checkbox is cleared, which is exactly when you need the extension. How to install it: see Elbuz extension for Chrome.

    When the crawl ends, the app doesn't just close the window. It applies all price list operations to the collected data (markup, stock status, and the rest), and the feed shows “Apply different operations to products” meanwhile. Then it reindexes the source's products and refreshes the grid, so you aren't looking at old data. The last feed lines are “Job completed” and “Spent: mm:ss”.

    To run everything at once, use “Run all site parsers” in the “Import sources” group of the price list menu. The app picks up all active parser sources and processes them one by one: when one finishes, the window closes and opens again for the next one a second and a half later.

    What happens after “Start download”

    “Start download” doesn't go to the site right away. First the app prepares the queue, and this is where it's decided whether there will be anything to crawl at all.

    1. The queue and the live feed of the previous run are cleared. So read the “Parsing log” before the next run, not after it. “Continue” doesn't clear anything.
    2. If “Delete products before loading” is checked, the source's products are deleted. If it's cleared, the URLs of existing products go into the queue marked “Ignore”, and their pages aren't opened at all in this run. That's how the promise “only new products will be downloaded from the site” works.
    3. If “Link to sitemap (XML file)” is filled in, the sitemap is downloaded again on every run, not just once during setup.
    4. If the “Links” tab has URLs, each one becomes a job. A URL with the “This is a link to a product” checkbox goes straight to the “Product card” operation; the rest go to the “List of product links” operation.
    5. If “Links” is empty, the app creates exactly one job: “List of product category links” for the URL in the “Category link” field, or for the site's home page if that field is empty.
    The top fork is which of the two buttons was clicked: “Continue” clears nothing. The lower right branch is the case where no queue is created at all and the run ends before it starts
    “Continue”“Start download”yesnonoyesyesnoA button is clicked in the run window“Continue” or “Start download”?The crawl uses the existing queue,nothing is clearedThe queue and live feed of theprevious run are cleared“Delete products before loading”?The source's products are deletedURLs of existing products go intothe queue as “Ignore”Is “Links” empty?Each URL becomes a job: with “Thisis a link to a product” checked —“Product card”, the rest — “List ofproduct links”Category “Operation selector” filledin or “Auto-detect selectors” on?One job for “Category link” or thehome pageNo queue: after five emptyresponses — “Job completed”
    • Action or result
    • Decision: the path depends on the answer

    The last step has a condition behind most “the run finished, but nothing was collected” cases. The home page job is created only if the “Operation selector” row of the “List of product category links” group has something in it (a selector or a regular expression), or “Auto-detect selectors” is on. An empty selector without auto-detect plus an empty “Links” list means no queue at all: the feed shows “Job completed” at once, and the parsing log has no lines.

    Two checks run on every job, and it's worth knowing about them in advance. Each URL enters the queue only once: a repeated link to the same page is dropped, so the same product isn't downloaded twice even if it sits in three sections. And a URL on another domain never enters the queue: the feed shows a red line Main domain not exist in url! with the site address and the rejected link. If the site serves products on a second domain, enter it in the “Alias link” field on the “Additional” tab.

    How to run parsing on a schedule

    The schedule opens with the “Schedule setting” button in the top row of the “Additional” tab, next to “Parsing log” and “List of proxy servers”.

    Row of three buttons above the parsing options card
    1Parsing log
    2Schedule setting
    3List of proxy servers
    1. Parsing log — line-by-line history for every page.
    2. Schedule setting — regular collection; works only in server mode.
    3. List of proxy servers — your own addresses for bypassing blocks.

    The main condition: the schedule works only in server mode. If the server module is off, the scheduled job stops on its very first line and collects nothing, and you won't see any warning or error.

    The “Schedule setting” window
    1Active
    2Launch time
    1. Active — switches the schedule on.
    2. Launch time — the hour and minute of the run.

    The second condition concerns the source itself. If the parser runs in the browser, the schedule skips it and logs: “Source ID 123 was skipped by the schedule: the parser runs in the browser and is not started on schedule”. If the parser is set to server mode but the module isn't enabled in the account, the message is different: “Source ID 123 was skipped by the schedule: server-side parsing is not enabled for the account”. Look for both lines in the Operation log: they show that the schedule fired, but the source wasn't processed.

    Two scheduled runs never start at the same time. If another run is already in progress at start time, the log gets “Scheduled download canceled because another parsing template is being loaded (data processing conflict).”, and this run simply doesn't happen. If it happens every time, spread the sources across different hours.

    One more detail, shared by both modes: a job that gets no response for too long goes back into the queue with a log entry and a retry counter. The app waits 35 seconds in browser mode and 150 in server mode. The default is five retries; some marketplaces get more. In browser mode, this usually means a minimized window; in server mode, a slow site.

    What to set up on the “Additional” tab

    This window has no “Save” button. The form is sent to the server on its own each time a field changes, with no confirmation. This leads to two opposite mistakes: you look for the button and think the settings weren't saved, or you're “just looking”, click a checkbox, and have already changed how parsing works. There's only one way to check what's saved: close the window and open it again, and the values are re-read from the database.

    The main card of the tab is “Parsing options”, split into five groups.

    “What data to collect”. “Load attributes” (on; the hint warns that they're charged extra when the base catalog is updated), “Load photos” (on), “Load reviews” (off), “Save meta tags” (off), “Save a text copy of site pages” (off).

    The first three checkboxes of this group (“Load attributes”, “Load photos”, “Load reviews”), together with “Load prices” from the last group, are read by ready-made modules. The universal parser ignores them: what it collects is defined by the rules grid, and a field with an empty selector isn't collected whatever the checkbox says. The last two checkboxes of the group, “Save meta tags” and “Save a text copy of site pages”, work in both cases.

    “Page loading mode”. The “How the server fetches this site” dropdown has three values: “Through a proxy (regular server mode)”, the default; “Directly, without a proxy”, when the supplier has allowed parsing and requests are free; “Through a protection-bypass service”, for sites that otherwise won't serve pages, paid separately on any plan. The third adds a “Request country” field set to “Auto (default)” and forces server mode on. The group also has “Use server to load data”, “Use server for rendering JavaScript”, “Load JavaScript”, and “Load products and prices without opening the product page”.

    You don't need to pick a protection-bypass service. The app tries the cheaper one first and moves to the next one if it's refused. If a more expensive one worked, the source remembers it for 7 days. That's why the first run on a heavily protected site is slower than the following ones. A page where the site showed a protection check doesn't make it into the price list, but if the service charged for it, it's billed under the same rule (how much it costs).

    The “Parsing options” card on the Additional tab of the site data import settings window
    1What data to collect
    2How the server fetches this site
    3Use server to load data
    4Automation and debugging
    5Price collection
    1. What data to collect — attributes, photos, reviews, meta tags and a text copy of the pages.
    2. How the server fetches this site — through a proxy, directly, or through a protection-bypass service.
    3. Use server to load data — without it your browser opens the pages, so the window must stay open.
    4. Automation and debugging — Debug mode keeps the page code in the log.
    5. Price collection — whether prices are loaded.

    Two checkboxes here need a closer look. “Use server for rendering JavaScript” is for sites that show a blank page without scripts, and it slows the crawl sharply: the server takes 4 URLs at a time in 2 threads instead of the usual 100 in 10. “Load products and prices without opening the product page” no longer affects the universal parser once the source exists: listing mode is set by the “Products in this operation” column in the “Operation selector” row of the “List of product links” group, so switch that instead.

    “Data processing”. “Fix invalid HTML” (on); “Clean up links”, which for the universal parser strips the part after # from listing URLs, while parameters like ?param=value are cut only by ready-made modules; and “Delete products before loading” (checked by default). Make sure you understand the last one: if you clear it, only new products are downloaded from the site, and anything that has disappeared from the site stays in your price list.

    “Automation and debugging”. “Auto-detect selectors”: the app picks a selector wherever it's empty, on every run. “Search by catalog products”: instead of crawling sections, the app searches the site for your products. In the “Product search link” field, enter the search URL with the {NAME}, {SKU}, or {EAN} macro, and the app fills in the name, SKU, or barcode of each base catalog product. “Debug mode” saves the markup of each page in the parsing log; without it, there'll be nothing to look at there.

    “Price collection”. The “Load prices” checkbox.

    Below is the “Site addresses” card: “Site home page URL”, “Alias link” (a second domain of the same site, so links to it aren't rejected), “Category link” (the page the crawl starts from if it isn't the home page), “Product search link”, “Category ID”, and “Ignore pages containing text”. The last field filters out service pages: a page whose text contains your string isn't parsed. It works only in server mode; pages brought in by the browser extension skip this check.

    The “Additional” accordion at the bottom of the tab holds rare settings some sites need: “Find” / “Replace” (whole-page replacement before parsing), “CSS selectors to remove unnecessary blocks”, “List of tags to remove”, removal of own and foreign domain links, cookies to delete and to set, “HTTP headers”, “Site Encoding”, “Max. number of pages from category”, and “Max. number of products from the category”. These two limits apply to ready-made modules; the universal parser uses the “Max. number of results” column in the rule row. Also here: “Type of items to load” with “Product” (default), “Article”, “News”, “Recipe”.

    Scenarios

    Add only new products without re-downloading existing ones

    On the “Additional” tab, clear “Delete products before loading”. When the run starts, the URLs of products already in the price list go into the queue marked “Ignore”, and their pages aren't opened.

    What won't happen: in a regular crawl that opens product pages, prices and stock of existing products won't be updated in this run, and products that have disappeared from the site will stay in your price list.

    Collect a site every night with nobody involved

    On the “Additional” tab, check “Use server to load data” (the server parsing module must be enabled in your account), open “Schedule setting”, set the day and time, and turn on “Active”. The Elbuz server downloads the pages, so you don't need to open the app window.

    What won't happen: without the server parsing module, the scheduled job collects nothing. If another scheduled parsing run is already in progress at start time, this run is canceled with the entry “Scheduled download canceled because another parsing template is being loaded (data processing conflict).”

    What to do when a run won't start or stops

    What you seeWhyWhat to do
    “Job completed” right after the start, with no lines in the parsing log No queue was created: “Links” is empty, and the “List of product category links” group has neither a selector nor “Auto-detect selectors”. In browser mode, a missing looks the same Fill in the “Operation selector” of the first step or add URLs to “Links”; check the extension (how the crawl starts)
    A red line in the feed: Main domain not exist in url! and two URLs The selector captured a link to another domain, and such links don't enter the queue Narrow down the selector. If the site really serves products on a second domain, enter it in “Alias link” (about the crawl)
    “Source ID 123 was skipped by the schedule: the parser runs in the browser and is not started on schedule” The parser's “Use server to load data” checkbox is cleared, and the schedule can only start server runs Turn on server mode or run this source manually (about the schedule)
    “Source ID 123 was skipped by the schedule: server-side parsing is not enabled for the account” The parser is set to server mode, but the module itself isn't activated in the account Enable the “Use server to load data” module (where to find it and who has access)
    “Scheduled download canceled because another parsing template is being loaded (data processing conflict).” Another scheduled run is already in progress; two never run at once Wait for it to finish. If it happens every time, spread the source schedules across different hours (about the schedule)
    “Operation canceled by the user” “Cancel” was clicked, here or in another window Nothing. Unfinished jobs are kept: next time the window will offer “Continue”
    “There are unfinished jobs N” and “Click ‘Continue’ to resume.” The previous run didn't reach the end “Continue” resumes the crawl from the same place. “Start download” starts over and clears the previous run's log (about running)
    Settings on the “Additional” tab “weren't saved” The window has no “Save” button: the form is sent to the server each time a field changes, but shows no confirmation Close and reopen the window, and the values are re-read from the database (about the “Additional” tab)

    FAQ

    The schedule is set up, but the run doesn't start. What should I check?

    Server mode: without the “Use server to load data” module, the schedule does nothing. Then check whether another parsing run was in progress at that time, and whether the Operation log has a “Source ID … was skipped by the schedule” line with a reason (about the schedule).

    The run finished, but the products in the price list are old. Why?

    Most often “Delete products before loading” is cleared: then the pages of existing products aren't opened, and their prices aren't updated. The second reason is that the run never reached the product pages: check the “Product card” jobs in the parsing log.

    Why is parsing slow with JavaScript rendering on?

    In this mode the server takes 4 URLs at a time in 2 threads instead of the usual 100 URLs in 10 threads. Turn it on only for sites that show a blank page without scripts.

    Can I run all site parsers at once?

    Yes: use “Run all site parsers” in the “Import sources” group of the price list menu. Active parser sources are processed in turn, one after another.

    How much does “Through a protection-bypass service” mode cost?

    From €0.0045 per page on any plan; on heavily protected sites it's three or five times more: €0.0135 and €0.0225. “Directly, without a proxy” requests aren't charged. If the account has no fixed price, the run stops when the balance runs out, with an insufficient funds message.

    Where to find it and who has access

    Menu
    Directories → Processing price lists → List of price lists
    Run
    the “Site parser” row → “Import”; in the source card, “Products” tab → “Import”
    Schedule, log, proxies
    parser settings window, “Additional” tab, top row of buttons
    Access right
    the “Price lists” window
    Module
    “Use server to load data” — for background runs and the schedule
    Prepare
    for runs without the server — the Elbuz extension for Chrome

    Related topics

    Was this article helpful?

    Missing a feature? Tell us.

    We will add it for free if other users need it too.

    If only you need it, we will agree on it separately.

    Article contents

    Suggest an improvement

    Try Elbuz free