项目文件夹

文件

T

Xirui 8bc9c8bea1 feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57 )

2. add --craw-delay flag to specify/override robots directive.

2026-06-22 16:16:54 +07:00

cloner_test.go

2026-06-22 16:16:54 +07:00

cloner.go

2026-06-22 16:16:54 +07:00

config.go

2026-06-22 16:16:54 +07:00

dedup_test.go

2026-06-15 19:07:49 +07:00

frontier_test.go

2026-06-14 18:22:25 +07:00

frontier.go

2026-06-14 18:22:25 +07:00

sitemap.go

2026-06-14 18:22:25 +07:00

stats.go

2026-06-15 20:27:47 +07:00

wantasset_test.go

2026-06-15 20:27:47 +07:00