make后执行： ./crawler 服务器ip地址 端口号 url.txt
-
Notifications
You must be signed in to change notification settings - Fork 8
ChenyuGao/Crawler-Parallel
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
C语言并行爬虫（epoll），爬取服务器的16W个有效网页，通过爬取页面源代码进行确定性自动机匹配和布隆过滤器去重，对链接编号并写入url.txt文件，并通过中间文件和三叉树去除掉状态码非200的链接关系，将正确的链接关系继续写入url.txt
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published