Bulk Adding Products: How to Upload Thousands of SKUs in Minutes

When is bulk import needed?

Bulk import becomes a necessity in several key scenarios:

1. Launching a new store

Initial catalog loading is the most common use case for bulk import.

• Volume: from 500 to 100,000+ products depending on the niche
• Source: Supplier price lists, export from legacy systems, databases
• Peculiarity: All data needs to be loaded at once, including images, descriptions, and specifications
• Risks: There's no way to test on a small sample - import everything at once

2. Adding a new supplier

Expanding the product range through new partners.

• Volume: 1,000-10,000 new SKUs
• Problem: duplication of goods, if the supplier offers the same thing that is already in the catalog
• Solution: import with checking for existing goods by EAN/article

3. Platform migration

Transferring an online store from one CMS to another.

• Examples: OpenCart → WooCommerce, Magento 1 → Magento 2, Shopify → PrestaShop
• Peculiarity: It is necessary to save URL, SEO data, reviews, history
• Complexity: different data structures between platforms

4. Regular updates

Periodic synchronization with suppliers or marketplaces.

• Frequency: daily, weekly or in real time
• Volume: updating 10,000-50,000 positions at a time
• Requirement: minimal site downtime, incremental updates

5. Seasonal promotions

Quickly add special product selections for the holidays.

• Examples: New Year's gifts, back-to-school, Black Friday
• Volume: 500-5,000 SKU
• Urgency: needs to be downloaded within a few hours

Preparing data before importing

Proper data preparation is the key to a successful import. 80% of problems arise from incorrect source files.

Data Preparation Checklist

Typical errors in source data

An example of a correct CSV structure

 sku;name;price;stock;category;brand;ean;image_url;status SM-S921-256-BLK;Samsung Galaxy S24 256GB Black;849.00;15;Smartphones;Samsung;8806095184234;https://cdn.shop.com/s24.jpg;active IP-15PRO-512-BLU;Apple iPhone 15 Pro 512GB Blue Titanium;1299.00;8;Smartphones;Apple;194253775349;https://cdn.shop.com/ip15.jpg;active HP-PAV-15-I5;HP Pavilion 15 Laptop i5 16GB 512GB;649.00;23;Laptops;HP;195161197042;https://cdn.shop.com/hp15.jpg;active

Bulk import tools for different CMSs

Each e-commerce platform has its own tools for bulk importing. Let's look at solutions for popular CMSs.

WooCommerce (WordPress)

Built-in importer

Path: WooCommerce → Products → Import

• Format: CSV (template available for download)
• Limits: up to 5,000 products at a time (depending on hosting)
• Peculiarities: support for variations, updating existing products
• Pros: free, easy to use
• Cons: slow for large directories, limited error handling

Product Import Export for WooCommerce (plugin)

• Formats: CSV, XML, Excel
• Possibilities: Import scheduling, import via URL/FTP, logging
• Price: $69-$199/year
• Performance: up to 50,000 items

WP All Import

• Formats: CSV, XML, Excel, JSON
• Peculiarity: visual drag-and-drop field mapping
• Price: $99-$299/year (requires WooCommerce Add-On)
• Pros: the most flexible tool, support for complex structures

Shopify

Built-in import

Path: Products → Import

• Format: CSV (Shopify Strict Template)
• Limits: Up to 20,000 items per file, 50,000 variations
• Peculiarities: automatic creation of collections and tags
• Pros: reliable, good documentation
• Cons: Rigid CSV structure, cannot import meta fields

Matrixify (app)

• Formats: Excel, CSV, Google Sheets
• Possibilities: import of metadata, redirects, blogs
• Price: $20-$200/month
• Performance: up to 500,000 items

Magento 2

System → Import

• Format: CSV
• Modes: Add/Update, Replace, Delete
• Validation: Pre-check file with error report
• Peculiarity: import attributes, categories, images

FireBear Improved Import & Export

• Formats: CSV, XML, JSON, ODS, Excel
• Sources: URL, FTP, Dropbox, Google Sheets
• Automation: cron tasks, schedule
• Price: $399-$1,499

PrestaShop

Catalog → Import

• Format: CSV
• Types: Products, Categories, Customers, Addresses
• Peculiarity: Multiple values separator (default;)
• Limits: Recommended for up to 10,000 items at a time

Store Commander

• Interface: An Excel-like editor for bulk edits
• Import: CSV, Excel with advanced mapping
• Price: €99-€399

Validating data before importing

Validating data before running an import helps avoid catalog integrity issues and save time on error correction.

Validation levels

1. Structural validation

Checking the correctness of the file format:

• Encoding: the file is read without artifacts
• Separators: consistent number of columns in all rows
• Headlines: presence of all required fields
• Syntax: Correct quotation marks, escaping special characters

2. Data typing

Matching values to expected types:

• Prices: numerical values not less than 0
• Quantity: integers not less than 0
• URL: correct link format (http/https)
• Email: valid email addresses
• Dates: correct date format

3. Business rules

Logical correctness of data:

• SKU uniqueness: no duplicate articles
• Existence of dependencies: categories, brands, attributes exist in the system
• Pricing policy: price > cost price, price range corresponds to category
• Product restrictions: weight< максимального для доставки, габариты заполнены для крупногабаритных

4. Image validation

Checking the availability and correctness of images:

• URL availability: images are loading (HTTP 200)
• Format: JPG, PNG, WebP, GIF
• Size: at least 500x500px for the main image
• File size: optimally up to 500KB

Validation tools

Online CSV validators

• CSV Lint: Checking the structure of CSV files
• CSVkit: console utility for working with CSV
• Google Sheets: open the file and check it visually

Validation scripts

Example of a Python script for price validation:

 import pandas as pd import validators def validate_product_file(file_path): errors = [] # Load file df = pd.read_csv(file_path, sep=';', encoding='utf-8') # Check for required fields required_fields = ['sku', 'name', 'price', 'category'] missing_fields = set(required_fields) - set(df.columns) if missing_fields: errors.append(f"Missing fields: {missing_fields}") # Check for SKU uniqueness duplicates = df[df.duplicated('sku', keep=False)] if not duplicates.empty: errors.append(f"Duplicate SKUs: {duplicates['sku'].tolist()}") # Check prices invalid_prices = df[df['price']<= 0] if not invalid_prices.empty: errors.append(f"Некорректные цены в строках: {invalid_prices.index.tolist()}") # Проверка URL изображений if 'image_url' in df.columns: for idx, url in df['image_url'].items(): if pd.notna(url) and not validators.url(url): errors.append(f"Некорректный URL в строке {idx}: {url}") return errors # Использование errors = validate_product_file('products.csv') if errors: print("Найдены ошибки:") for error in errors: print(f"- {error}") else: print("Файл валиден, готов к импорту")

Handling errors during import

Even with careful data preparation, importing may encounter errors. It's important to handle them correctly.

Types of import errors

1. Parsing errors

Reasons:

• Incorrect encoding (Cyrillic is displayed as hieroglyphs)
• Inconsistent delimiters in CSV
• Unclosed quotes in fields

Solution: Resave the file in UTF-8, check the delimiters

2. Validation errors

Examples:

• "SKU already exists" - duplicate item
• "Category not found" - non-existent category
• "Incorrect price format" - the price contains letters

Solution: Correct specific lines, re-import

3. Resource errors

Problems:

• Timeout: The import was interrupted due to a time limit.
• Memory limit: not enough memory to process
• Disk space: server space is running out (especially when loading images)

Solution: Increase PHP limits (max_execution_time, memory_limit) or split the file into parts

4. Image loading errors

Reasons:

• URL unavailable (404, 500)
• Hotlink protection on the CDN side
• Incorrect image format
• The image is too large

Solution: Import products without images, then upload images separately

Error handling strategies

Strategy 1: Fail-fast

• Behavior: import aborts on first error
• Pros: critical problems are quickly identified
• Cons: No goods are imported until all errors are corrected
• Application: initial import, migration

Strategy 2: Skip errors

• Behavior: import continues, problematic lines are skipped
• Pros: the maximum amount of goods is imported
• Cons: you may miss important items
• Application: Regular updates, import from multiple suppliers

Strategy 3: Partial import

• Behavior: The product is imported with all fields filled in, but the problematic fields remain empty.
• Example: The product is added without an image if the URL is unavailable.
• Pros: all products are included in the catalog
• Cons: incomplete cards require revision
• Application: urgent launch, gradual filling

Logging and reporting

Detailed import logs are necessary for problem analysis:

 ============================================ IMPORT REPORT =========================================== Date: 2025-10-21 14:23:15 File: suppliers_catalog.csv Lines in file: 5,243 RESULTS: ✓ Successfully imported: 5,107 products (97.4%) ✗ Skipped due to errors: 136 products (2.6%) STATISTICS: - New products created: 4,823 - Existing ones updated: 284 - Images uploaded: 18,342 ERRORS: 1. Duplicate SKUs (23 product): - Line 145: SKU "SM-S24-256" already exists - Line 891: SKU "IP-15-PRO" already exists... 2. Non-existent categories (58 products): - Line 234: Category "Smartwatches" not found - Line 456: Category "TWS Headphones" not found... 3. Image loading errors (55 products): - Line 789: https://cdn.supplier.com/img123.jpg - 404 Not Found - Line 1023: https://cdn.supplier.com/img456.jpg - Timeout... RECOMMENDATIONS: 1. Create categories: "Smartwatches", "TWS Headphones" 2. Fix duplicate SKUs in the source file 3. Check image availability on the supplier side Full log saved: /var/log/import_20251021_142315.log ========================================

Automatic error correction

Some errors can be corrected automatically:

• SKU autogeneration: if missing, create from name + ID
• Creating categories: automatically create missing categories
• Replacing images: If the URL is unavailable, use a placeholder
• Price normalization: Automatically remove currency symbols, convert commas to periods
• CNC transliteration: generate URL-friendly slug from name

Rollback for critical issues

The rollback mechanism allows you to return the directory to its pre-import state in case of critical errors.

When is a rollback needed?

• Bulk rewrite of products: The import accidentally updated all products with incorrect data.
• Price error: All prices are loaded with an error (wrong currency, wrong exchange rate)
• Deleting data: Import in "replace" mode accidentally deleted existing products
• Structure breakdown: the connections between categories, attributes, and variations are broken

Rollback implementation methods

1. Backup the database before importing

Process:

1. Create a full database dump before importing
2. Run import
3. If you have problems, restore the database from a dump.

Pros: 100% condition recovery

Cons: takes a long time for large databases (100GB+), blocks all operations

 # Create a backup before import (MySQL) mysqldump -u username -p database_name > backup_before_import_20251021.sql # Recover from problems mysql -u username -p database_name< backup_before_import_20251021.sql

2. Transactional import

Process:

1. Begin a database transaction (BEGIN TRANSACTION)
2. Perform import
3. If successful, COMMIT; if an error occurs, ROLLBACK.

Pros: instant rollback, no intermediate states

Cons: requires transaction support (InnoDB in MySQL), high load on the database

3. Product versioning

Process:

1. Save the current version of the products in a separate table
2. Perform import
3. If there are problems, restore from the versioned table

Pros: the ability to roll back only products without affecting orders

Cons: requires additional space in the database

4. Staged import

Process:

1. Import products into the staging table
2. Conduct validation and checks
3. If successful, move the tables to production.

Pros: safe, can be tested

Cons: more difficult to implement

Rollback best practices

Partial rollback (for goods)

If you need to roll back only some of the goods:

 # Rollback specific products according to the SKU list UPDATE products SET price = backup_products.price, name = backup_products.name, stock = backup_products.stock FROM backup_products WHERE products.sku = backup_products.sku AND products.sku IN ('SKU1', 'SKU2', 'SKU3');

High volume productivity

Importing tens of thousands of products requires optimization for acceptable speed and reliability.

Performance bottlenecks

1. File reading speed

Problem: Reading and parsing a large file can take a significant amount of time.

Solutions:

• CSV instead of Excel: CSV parses 10-20 times faster than XLSX
• Streaming parsing: read the file line by line, rather than loading the entire file into memory
• Compression: CSV.gz takes up less space and is transferred faster over the network.

2. Recording in the database

Problem: Inserting products one by one is slow (1000 products = 1000 requests = minutes).

Solutions:

• Batch insert: insert 100-500 products per request
• Prepared statements: Reduces SQL parsing overhead
• Disabling indexes: temporarily disable indexes, create after import
• Bulk insert API: use specialized database methods

3. Uploading images

Problem: Downloading and processing images is the slowest part.

Solutions:

• Asynchronous loading: First import the products, then upload the images in the background
• Parallel loading: download 10-20 images at a time
• CDN: If the images are already on a CDN, use URLs instead of local copies
• Lazy loading: Import URL only, load image on first view

4. Indexing and cache

Problem: Recalculating indexes and invalidating cache slows down import.

Solutions:

• Deferred indexing: index after import, not after each product
• Clearing cache: clear cache once after import
• Disabling hooks: temporarily disable triggers and event listeners

Optimizing batch processing

Batch size

The optimal batch size depends on the data:

• Simple goods (data only): 500-1000 per batch
• Products with variations: 200-500 per batch
• Products with images: 50-100 per batch
• Complex products (attributes, relationships): 50-200 per batch

Rule of thumb: batch size = 1-2 seconds of processing

Batch Import Example (PHP for WooCommerce)

<!--?php // Batch-импорт товаров в WooCommerce function batch_import_products($file_path, $batch_size = 500) { $handle = fopen($file_path, 'r'); $header = fgetcsv($handle, 0, ';'); $batch = []; $imported = 0; $total_time = microtime(true); while (($row = fgetcsv($handle, 0, ';'))!== false) { $product_data = array_combine($header, $row); $batch[] = $product_data; // Импортируем батч if (count($batch) --> = $batch_size){ $batch_time = microtime(true); import_batch($batch); $batch_time = microtime(true) - $batch_time; $imported += count($batch); echo "Импортировано: $imported товаров (батч за ". round($batch_time, 2). " сек)\n"; $batch = []; gc_collect_cycles(); // Очистка памяти } } // Import the remainder if (!empty($batch)) { import_batch($batch); $imported += count($batch); } fclose($handle); $total_time = microtime(true) - $total_time; echo "Total imported: $imported products in ". round($total_time, 2). " sec\n"; echo "Speed: ". round($imported / $total_time, 2). " products/sec\n"; } function import_batch($batch) { global $wpdb; // Prepare bulk insert $values = []; $placeholders = []; foreach ($batch as $product) { $placeholders[] = "(%s, %s, %f, %d, 'publish')"; $values[] = $product['sku']; $values[] = $product['name']; $values[] = $product['price']; $values[] = $product['stock']; } $query = "INSERT INTO {$wpdb->posts} (post_name, post_title, meta_price, meta_stock, post_status) VALUES ". implode(', ', $placeholders); $wpdb->query($wpdb->prepare($query, $values)); }?>

Parallel processing

Multithreaded import

Split the file into parts and import in parallel:

1. Split the file: 10,000 products → 10 files of 1,000
2. Run multiple processes: each one processes its own file
3. Combine the results: collect statistics and logs

Acceleration: 3-5x on a multi-core server

Background task queue

Use a queuing system (Redis, RabbitMQ):

1. The main process reads the file and creates tasks
2. Workers process tasks from the queue.
3. The number of workers can be scaled

Performance comparison

 max_execution_time = 3600; 1 hour memory_limit = 512M; or more for huge files upload_max_filesize = 100M post_max_size = 100M max_input_time = 600

Conclusion

Bulk product import is a critical operation for online stores of any size. Proper data preparation, optimal tools, and error handling strategies transform this potentially risky process into a routine one.

Key findings

• Подготовка данных критична: 80% успеха импорта — корректно подготовленный файл. Используйте чек-лист валидации перед загрузкой.
• Выбирайте правильный инструмент: встроенные импортеры CMS подходят для базовых задач, специализированные плагины — для сложных сценариев, универсальные платформы типа Elbuz — для автоматизации.
• Валидация обязательна: проверяйте данные до импорта, а не после. Исправить 100 ошибок в файле проще, чем 100 неправильных карточек в каталоге.
• Обрабатывайте ошибки gracefully: используйте стратегию skip errors для регулярных обновлений, fail-fast для критичных импортов.
• Всегда имейте plan B: backup перед импортом, механизм rollback при проблемах, staging-среда для тестирования.
• Оптимизируйте для масштаба: batch-обработка и параллелизм ускоряют импорт в 10-50 раз. Для 100,000+ товаров — это не опция, а необходимость.

Practical recommendations

1. Start small: протестируйте импорт на 100 товарах перед полной загрузкой
2. Документируйте процесс: создайте внутреннюю инструкцию по импорту для вашей команды
3. Автоматизируйте рутину: если импортируете регулярно, настройте автоматическую загрузку
4. Мониторьте результаты: анализируйте логи импорта, находите паттерны ошибок
5. Train your team: менеджеры должны понимать базовые принципы импорта

Next steps

После освоения массового импорта рассмотрите автоматизацию смежных процессов:

• Автоматическая загрузка прайсов от поставщиков по расписанию
• Объединение данных от нескольких поставщиков в единый каталог
• Синхронизация остатков в реальном времени
• Автоматическое обновление цен с учетом курсов валют и наценок