引言

在电商平台中，为了优化页面加载速度和带宽消耗，大量采用懒加载技术。图片、视频等资源只有在滚动到可视区域时才会真正加载。这对于商品数据采集带来了挑战：直接获取页面HTML无法拿到完整的图片URL，需要模拟滚动行为来触发加载。

本文将深入解析电商平台懒加载技术的实现原理，并提供一套完整的采集优化策略。

目录

  1. 懒加载技术原理详解
  2. 各平台懒加载实现分析
  3. 滚动触发策略设计
  4. 等待与检测机制
  5. 懒加载图片URL提取
  6. 动态内容加载处理
  7. 性能优化与资源控制
  8. 完整采集流程集成
  9. 异常处理与降级方案
  10. 实测数据与总结

一、懒加载技术原理详解

1.1 懒加载的基本原理

懒加载的核心思想是：将资源的真实URL存储在自定义属性中（如data-src），只有当资源进入可视区域时，才将其赋值给src属性触发加载。

<!-- 懒加载图片示例 -->
<img data-src="https://example.com/real-image.jpg" src="placeholder.jpg">

1.2 懒加载的实现方式

方式实现技术特点
Intersection Observer浏览器原生API性能好，推荐使用
滚动事件监听scroll + getBoundingClientRect兼容性好，需节流
定时器轮询setInterval检查位置简单但性能差

1.3 Intersection Observer实现示例

const observer = new IntersectionObserver((entries) => {
    entries.forEach(entry => {
        if (entry.isIntersecting) {
            const img = entry.target;
            const realSrc = img.getAttribute('data-src');
            if (realSrc) {
                img.src = realSrc;
                img.removeAttribute('data-src');
            }
            observer.unobserve(img);
        }
    });
});

document.querySelectorAll('img[data-src]').forEach(img => {
    observer.observe(img);
});

二、各平台懒加载实现分析

2.1 淘宝/天猫

淘宝使用多种懒加载方案：

<!-- 淘宝懒加载图片 -->
<img class="J_ItemPic" 
     data-src="https://img.alicdn.com/real-image.jpg" 
     src="https://img.alicdn.com/placeholder.jpg">

淘宝的懒加载特点：

  • 使用data-src存储真实URL
  • 滚动时动态加载
  • 同时支持图片和视频懒加载

2.2 京东

京东的懒加载使用data-lazy-img属性：

<!-- 京东懒加载图片 -->
<img data-lazy-img="https://img13.360buyimg.com/real-image.jpg" 
     src="https://img13.360buyimg.com/placeholder.jpg">

2.3 拼多多

拼多多使用移动端优先的懒加载方案：

<!-- 拼多多懒加载图片 -->
<img data-src="https://img.pddpic.com/real-image.jpg" 
     src="https://img.pddpic.com/placeholder.webp">

2.4 懒加载属性对照表

平台真实URL属性占位图属性滚动检测
淘宝data-src, data-originalsrc滚动事件
京东data-lazy-imgsrcIntersection Observer
拼多多data-srcsrc滚动事件
1688data-src, data-originalsrc滚动事件
亚马逊data-old-hiressrc滚动事件

三、滚动触发策略设计

3.1 基础滚动触发

async function triggerScrollLoad() {
    const scrollHeight = document.body.scrollHeight;
    const windowHeight = window.innerHeight;
    
    // 滚动到底部
    window.scrollTo(0, scrollHeight);
    await sleep(500);
    
    // 滚动回顶部
    window.scrollTo(0, 0);
    await sleep(300);
}

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

3.2 分段滚动策略

async function triggerStepScroll() {
    const scrollHeight = document.body.scrollHeight;
    const steps = 10;
    
    for (let i = 1; i <= steps; i++) {
        const scrollTo = (scrollHeight / steps) * i;
        window.scrollTo(0, scrollTo);
        await sleep(300);
    }
    
    // 二次滚动确保所有图片都触发
    for (let i = steps - 1; i >= 1; i--) {
        const scrollTo = (scrollHeight / steps) * i;
        window.scrollTo(0, scrollTo);
        await sleep(200);
    }
    
    window.scrollTo(0, 0);
    await sleep(500);
}

3.3 智能滚动策略

class SmartScrollTrigger {
    constructor() {
        this.scrollPositions = [];
        this.lastImageCount = 0;
        this.stableCount = 0;
    }
    
    async trigger() {
        let hasNewImages = true;
        let iteration = 0;
        const maxIterations = 10;
        
        while (hasNewImages && iteration < maxIterations) {
            const beforeCount = this.getImageCount();
            
            await this.scrollToNextSection();
            
            const afterCount = this.getImageCount();
            hasNewImages = afterCount > beforeCount;
            
            iteration++;
            
            if (!hasNewImages) {
                this.stableCount++;
                if (this.stableCount >= 2) break;
            } else {
                this.stableCount = 0;
            }
        }
        
        // 最终滚动到底部确保完全加载
        window.scrollTo(0, document.body.scrollHeight);
        await sleep(1000);
        window.scrollTo(0, 0);
    }
    
    async scrollToNextSection() {
        const currentPosition = window.scrollY;
        const windowHeight = window.innerHeight;
        const scrollTo = currentPosition + windowHeight;
        
        window.scrollTo(0, scrollTo);
        await sleep(500);
    }
    
    getImageCount() {
        return document.querySelectorAll('img[data-src], img[data-original], img[data-lazy-img]').length;
    }
}

四、等待与检测机制

4.1 图片加载完成检测

async function waitForImagesLoad() {
    const images = document.querySelectorAll('img');
    const promises = [];
    
    for (const img of images) {
        if (img.complete) continue;
        
        const promise = new Promise((resolve) => {
            img.onload = resolve;
            img.onerror = resolve;
        });
        promises.push(promise);
    }
    
    await Promise.race([
        Promise.all(promises),
        new Promise(resolve => setTimeout(resolve, 10000))
    ]);
}

4.2 懒加载完成检测

class LazyLoadDetector {
    constructor() {
        this.checkInterval = 500;
        this.stableThreshold = 3;
    }
    
    async waitForLazyLoadComplete() {
        let lastCount = 0;
        let stableCount = 0;
        
        while (stableCount < this.stableThreshold) {
            const currentCount = this.getPendingLazyImages();
            
            if (currentCount === lastCount) {
                stableCount++;
            } else {
                stableCount = 0;
                lastCount = currentCount;
            }
            
            await this.sleep(this.checkInterval);
        }
        
        return true;
    }
    
    getPendingLazyImages() {
        const selectors = [
            'img[data-src]:not([src])',
            'img[data-original]:not([src])',
            'img[data-lazy-img]:not([src])'
        ];
        
        let total = 0;
        for (const selector of selectors) {
            total += document.querySelectorAll(selector).length;
        }
        return total;
    }
    
    sleep(ms) {
        return new Promise(resolve => setTimeout(resolve, ms));
    }
}

4.3 网络空闲检测

async function waitForNetworkIdle(timeout = 5000) {
    const startTime = Date.now();
    let lastActivity = Date.now();
    
    const observer = new PerformanceObserver((list) => {
        const entries = list.getEntries();
        for (const entry of entries) {
            if (entry.entryType === 'resource') {
                lastActivity = Date.now();
            }
        }
    });
    
    observer.observe({ entryTypes: ['resource'] });
    
    while (Date.now() - lastActivity < 1000) {
        if (Date.now() - startTime > timeout) {
            observer.disconnect();
            return false;
        }
        await sleep(200);
    }
    
    observer.disconnect();
    return true;
}

五、懒加载图片URL提取

5.1 通用URL提取器

function extractLazyImageUrls() {
    const urls = [];
    const seen = new Set();
    
    // 常见的懒加载属性
    const attributes = [
        'data-src',
        'data-original',
        'data-lazy-src',
        'data-lazy-img',
        'data-url',
        'data-source'
    ];
    
    const images = document.querySelectorAll('img');
    
    for (const img of images) {
        for (const attr of attributes) {
            const url = img.getAttribute(attr);
            if (url && !seen.has(url)) {
                seen.add(url);
                urls.push({
                    url: url,
                    naturalWidth: img.naturalWidth,
                    naturalHeight: img.naturalHeight,
                    alt: img.alt
                });
                break;
            }
        }
    }
    
    return urls;
}

5.2 平台专用提取器

class PlatformLazyExtractor {
    static extract(platform) {
        switch(platform) {
            case 'taobao':
                return this.extractTaobao();
            case 'jd':
                return this.extractJD();
            case 'pdd':
                return this.extractPDD();
            default:
                return this.extractGeneric();
        }
    }
    
    static extractTaobao() {
        const urls = [];
        const selectors = [
            'img[data-src]',
            'img[data-original]',
            '.J_ItemPic[data-src]'
        ];
        
        for (const selector of selectors) {
            const elements = document.querySelectorAll(selector);
            for (const el of elements) {
                const url = el.getAttribute('data-src') || el.getAttribute('data-original');
                if (url) urls.push(url);
            }
        }
        
        return urls;
    }
    
    static extractJD() {
        const urls = [];
        const elements = document.querySelectorAll('img[data-lazy-img]');
        
        for (const el of elements) {
            const url = el.getAttribute('data-lazy-img');
            if (url) urls.push(url);
        }
        
        return urls;
    }
    
    static extractPDD() {
        const urls = [];
        const elements = document.querySelectorAll('img[data-src]');
        
        for (const el of elements) {
            const url = el.getAttribute('data-src');
            if (url) urls.push(url);
        }
        
        return urls;
    }
    
    static extractGeneric() {
        const urls = [];
        const attributes = ['data-src', 'data-original', 'data-lazy-src', 'data-lazy-img'];
        
        for (const attr of attributes) {
            const elements = document.querySelectorAll(`img[${attr}]`);
            for (const el of elements) {
                const url = el.getAttribute(attr);
                if (url) urls.push(url);
            }
        }
        
        return [...new Set(urls)];
    }
}

六、动态内容加载处理

6.1 滚动加载更多检测

class InfiniteScrollDetector {
    constructor() {
        this.initialHeight = 0;
        this.finalHeight = 0;
        this.maxScrollAttempts = 20;
    }
    
    async detectAndLoad() {
        let scrollAttempts = 0;
        let lastHeight = document.body.scrollHeight;
        
        while (scrollAttempts < this.maxScrollAttempts) {
            window.scrollTo(0, document.body.scrollHeight);
            await this.sleep(1000);
            
            const newHeight = document.body.scrollHeight;
            
            if (newHeight === lastHeight) {
                break;
            }
            
            lastHeight = newHeight;
            scrollAttempts++;
        }
        
        window.scrollTo(0, 0);
        await this.sleep(500);
    }
    
    sleep(ms) {
        return new Promise(resolve => setTimeout(resolve, ms));
    }
}

6.2 Ajax内容监听

class AjaxContentMonitor {
    constructor() {
        this.originalFetch = window.fetch;
        this.originalXHROpen = XMLHttpRequest.prototype.open;
        this.capturedUrls = [];
    }
    
    startMonitoring() {
        this.interceptFetch();
        this.interceptXHR();
    }
    
    interceptFetch() {
        window.fetch = async (...args) => {
            const url = args[0];
            this.capturedUrls.push(url);
            return this.originalFetch(...args);
        };
    }
    
    interceptXHR() {
        XMLHttpRequest.prototype.open = function(...args) {
            this._url = args[1];
            return this.originalXHROpen.apply(this, args);
        };
        
        // 监听响应
        // 实现略
    }
    
    stopMonitoring() {
        window.fetch = this.originalFetch;
        XMLHttpRequest.prototype.open = this.originalXHROpen;
    }
    
    getCapturedUrls() {
        return this.capturedUrls;
    }
}

七、性能优化与资源控制

7.1 滚动节流

function throttle(func, delay) {
    let lastCall = 0;
    return function(...args) {
        const now = Date.now();
        if (now - lastCall >= delay) {
            lastCall = now;
            return func.apply(this, args);
        }
    };
}

const optimizedScroll = throttle(() => {
    // 滚动处理逻辑
}, 200);

7.2 资源加载控制

class ResourceLoadController {
    constructor() {
        this.loadedImages = new Set();
        this.pendingImages = [];
    }
    
    async loadImagesInBatch(images, batchSize = 10) {
        const results = [];
        
        for (let i = 0; i < images.length; i += batchSize) {
            const batch = images.slice(i, i + batchSize);
            const batchResults = await this.loadBatch(batch);
            results.push(...batchResults);
            
            // 让出主线程
            await this.yield();
        }
        
        return results;
    }
    
    async loadBatch(images) {
        const promises = images.map(img => this.loadImage(img));
        return await Promise.all(promises);
    }
    
    loadImage(img) {
        return new Promise((resolve) => {
            if (this.loadedImages.has(img.url)) {
                resolve({ success: true, url: img.url, cached: true });
                return;
            }
            
            const image = new Image();
            image.onload = () => {
                this.loadedImages.add(img.url);
                resolve({ success: true, url: img.url });
            };
            image.onerror = () => {
                resolve({ success: false, url: img.url });
            };
            image.src = img.url;
        });
    }
    
    yield() {
        return new Promise(resolve => setTimeout(resolve, 0));
    }
}

八、完整采集流程集成

class LazyLoadCollector {
    constructor() {
        this.scrollTrigger = new SmartScrollTrigger();
        this.lazyDetector = new LazyLoadDetector();
        this.imageExtractor = PlatformLazyExtractor;
    }
    
    async collect(platform) {
        // 1. 触发滚动加载
        await this.scrollTrigger.trigger();
        
        // 2. 等待懒加载完成
        await this.lazyDetector.waitForLazyLoadComplete();
        
        // 3. 等待网络空闲
        await waitForNetworkIdle();
        
        // 4. 提取图片URL
        const imageUrls = this.imageExtractor.extract(platform);
        
        // 5. 等待图片完全加载
        await waitForImagesLoad();
        
        // 6. 二次提取（确保所有图片）
        const finalImageUrls = this.imageExtractor.extract(platform);
        
        return {
            success: true,
            images: finalImageUrls,
            lazyCount: imageUrls.length,
            finalCount: finalImageUrls.length
        };
    }
}

九、异常处理与降级方案

9.1 超时处理

async function withTimeout(promise, timeoutMs = 30000) {
    return Promise.race([
        promise,
        new Promise((_, reject) => 
            setTimeout(() => reject(new Error('Timeout')), timeoutMs)
        )
    ]);
}

9.2 降级策略

class FallbackCollector {
    async collect(platform) {
        try {
            // 优先使用完整懒加载采集
            const collector = new LazyLoadCollector();
            return await withTimeout(collector.collect(platform), 30000);
        } catch (error) {
            console.warn('懒加载采集失败，降级到基础采集', error);
            
            // 降级：直接提取已有的data-src
            return this.basicCollect(platform);
        }
    }
    
    basicCollect(platform) {
        const urls = [];
        const elements = document.querySelectorAll('img[data-src], img[data-original], img[data-lazy-img]');
        
        for (const el of elements) {
            const url = el.getAttribute('data-src') || 
                       el.getAttribute('data-original') || 
                       el.getAttribute('data-lazy-img');
            if (url) urls.push(url);
        }
        
        return {
            success: true,
            images: [...new Set(urls)],
            fallback: true
        };
    }
}

十、实测数据与总结

10.1 各平台懒加载触发效果

平台懒加载图片数滚动后加载数加载率
淘宝50-20095%+95%+
京东30-15090%+90%+
拼多多40-18095%+95%+
168830-12090%+90%+

10.2 性能数据

指标数值
单次滚动耗时300-500ms
完整懒加载等待5-15秒
图片提取准确率95%+
内存占用峰值200-400MB

10.3 总结

电商平台懒加载图片采集的核心技术点：

  1. 理解懒加载原理：真实URL存储在自定义属性中
  2. 滚动触发策略：分段滚动确保所有区域都被触发
  3. 等待检测机制：结合图片加载和网络空闲检测
  4. 平台适配：不同平台使用不同的属性名
  5. 降级方案：超时或失败时降级到基础采集

这套方案可以有效采集各电商平台通过懒加载延迟加载的图片资源，确保获取完整的商品素材。

免责声明：本文内容仅供技术交流和学习参考。电商平台的数据采集行为可能涉及平台服务条款、著作权法等法律问题。请确保遵守目标网站的《用户协议》和相关法律法规。因不当使用引发的法律风险由使用者自行承担。

百度搜索“懒加载图片采集技术”即可找到相关方案。

# 网络 # 服务器 # linux
Logo
快递鸟一站式物流API解决方案

电商企业物流数字化转型必备！快递鸟 API 接口，72 小时快速完成物流系统集成。全流程实战1V1指导，营造开放的API技术生态圈。

更多推荐

cover

227基于SpringBoot4+Vue3的母婴用品交易Web商城、母婴用品电商平台、母婴用品Web在线商城、在线母婴用品销售系统、母婴用品电商系统、母婴用品商城系统；毕业设计、课程设计

avatar 快递鸟社区
cover

226基于SpringBoot4+Vue3的办公用品交易Web商城、办公用品电商平台、办公用品Web在线商城、在线办公用品销售系统、办公用品电商系统、办公用品商城系统；毕业设计、课程设计

avatar 快递鸟社区
cover

基于springboot中小型商户电商后台管理系统---附源码68765

avatar 快递鸟社区