haikson/sitemap-generator

Sitemap crawler and generator class

dev-master 2018-11-22 19:04 UTC

Requires

• beautifulsoup4: >=4.4.0
• mechanize: >=0.2.5

Apache License

• Kamo Petrosyan

• Sitemap
• crawler

README

Sitemap generator

installing

pip install sitemap-generator

Gevent

Sitemap-generator uses gevent to implement multiprocessing. Install gevent:

pip install gevent

example

import pysitemap


if __name__ == '__main__':
    url = 'http://www.example.com/'  # url from to crawl
    logfile = 'errlog.log'  # path to logfile
    oformat = 'xml'  # output format
    crawl = pysitemap.Crawler(url=url, logfile=logfile, oformat=oformat)
    crawl.crawl()

multiprocessing example

import pysitemap


if __name__ == '__main__':
    url = 'http://www.example.com/'  # url from to crawl
    logfile = 'errlog.log'  # path to logfile
    oformat = 'xml'  # output format
    crawl = pysitemap.Crawler(url=url, logfile=logfile, oformat=oformat)
    crawl.crawl(pool_size=10)  # 10 parsing processes