网络爬虫程序源码,开发辅助,Asp.net源码

bin
- download
  - 3891570720e771c847e5ac23e28aa6cc.html[105KB]
  - 417d9e708c95da24b75705338598087f.html[10B]
  - 84675a6817fc8715e33bc1c631154b5d.html[417KB]
  - c6b7e4c627243167faa5495e3aa583ec.html[4KB]
  - da78a112d1275115651d236d9c42ee97.html[302B]
  - e0b5fdbe393b18e9d9f30feb89c3e695.html[10B]
  - f82b75ac411ea1f58e2ed8f6fc0b82c3.html[247B]
  - 0003be8238c8302e17c799d9f5d65876.gif[35KB]
  - 0718ad68487fa12de0cc75b20f7be03c.html; charset=utf-8[72KB]
  - 082e9d970f371da4f6e74dbe2c97f6e2.html; charset=utf-8[47KB]
  - 132949602460dfebc35da092329cba0c.gif[317B]
  - 1695505243ceaa9c68e5a00061d1763f.javascript[4KB]
  - 1df7133090a0d07c5cec8fccbf6fd8dd.html; charset=utf-8[14KB]
  - 203557adfb69f0b4da4e237df2c0899a.html; charset=gb2312[164B]
  - 23e5f50b0b42662c6694e574e74835cd.html; charset=utf-8[14KB]
  - 24eebf7019dc355f064372d6a889c60a.html; charset=gb2312[62KB]
  - 27439efce81b9ca84182d54aa411418e.html; charset=gb2312[53KB]
  - 2a2f02ca86459cde185fc8e8e9045bed.html; charset=utf-8[35KB]
  - 349427e49e96cbca35651e55ef94353d.gif[287B]
  - 3ff2932f670fc24203b1290df195dabf.gif[322B]
  - 44b19dec343bee7540d2e563399518f6.html; charset=gb2312[45KB]
  - 46e1c646c9965ce2581be0e2baa182cf.html; charset=utf-8[21KB]
  - 48bfe5c4818bc6d7d0a86b7c5d5a963a.javascript[4KB]
  - 4cef95f512517e118d0427cdf40d8d91.javascript[11KB]
  - 54cd270476c08dc49137cc587d5420e7.html; charset=utf-8[47KB]
  - 5ae7c8b442091b3c740b5f89f2202977.gif[305B]
  - 5f194c03340af2c82af0806b4cd95f44.html; charset=gb2312[45KB]
  - 6a78a05748d064e4491b674a391174c7.javascript[38KB]
  - 6ba086f85f3602a364dae60f740138c5.html; charset=gb2312[72KB]
  - 73e9259e079ac68519bd2cf67af06c13.html; charset=utf-8[91KB]
  - 753a67d9417f20f83e1dce17d6146f85.gif[1KB]
  - 767223508f1bd57304d84720065f9ee8.x-javascript[3KB]
  - 7780c2d0134fad8b7a05a95d0f7b3378.html; charset=gb2312[101KB]
  - 7a6721fd05029de13a9df0e2a0948f25.html; charset=UTF-8[205B]
  - 7eedab1d5fa988b034a32f14e08a97c0.gif[2KB]
  - 857c3c382495ba1593a316498236e4f8.html; charset=gb2312[58KB]
  - 8769fd41800599144d3fffb49173cf71.x-icon[23KB]
  - 89253cefeda362f9b403341ccec22420.gif[314B]
  - 8d52d7ccdc272a6bcaf36ae22d856dfc.html; charset=utf-8[46KB]
  - 9339d79eed585c1e0b126588c50477a8.javascript[32KB]
  - 93c0e58661019bd4a98aa3790a400cdf.x-javascript[9KB]
  - 94f1e7adbd48cf364b19771319db6b3f.gif[807B]
  - 956119ce46fe84d5c1e240ef7d417bdb.html; charset=gb2312[55KB]
  - 9d71e4ab781e1b9bf3eccf2a47568d6e.html; charset=utf-8[45KB]
  - a2418875c3955a694b18cf795764164a.html; charset=gb2312[45KB]
  - a490c2a29b5986e5cd4e114a0b50d394.html; charset=gb2312[46KB]
  - a6275663cfbb6142241df064c6f249f9.html; charset=gb2312[74KB]
  - a776c9fb2eafab1f75def2a07a40c6ff.html; charset=gb2312[63KB]
  - b49950b51a7090372fa275d86a0bbae6.html; charset=gb2312[240KB]
  - baaba63486a5eaa09b34f56b5ffbfe99.html; charset=gb2312[47KB]
  - bbcff706ddc752ee730069aa036a390b.html; charset=gb2312[46KB]
  - cb8c4ddd3d55475825bf08ed71e11da7.gif[310B]
  - d37c07e10a22a9698fe474154fecaef1.html; charset=utf-8[20KB]
  - d646baff77dd8a709ef5b83ab084dfa1.html; charset=gb2312[58KB]
  - d655e24ee8c02fb2ef11764588cec317.x-javascript[2KB]
  - d700dde53028a9f4cb83aaed8df7ab23.x-javascript[2KB]
  - dce581b36d215edeae8f9fdc9c07529e.html; charset=gb2312[116KB]
  - dfcb93920e639c9f7963e66ad84c9a46.gif[1KB]
  - e030fe253f6880680bdd7dec04fbf67d.html; charset=gb2312[45KB]
  - e1b0f26b9a2eb96cbcfbe8c6d88d0344.html; charset=utf-8[64KB]
  - e2ab7c468bc700b7b381b1311e84dbf7.html; charset=gb2312[59KB]
  - e49f46d2d3f40d8f28669220f8f1588f.gif[114B]
  - e9919aa1644ed6c90b06e6cf15e9ce8e.html; charset=gb2312[72KB]
  - e9f8ab2ef6b78283a0331b975474b969.html; charset=gb2312[45KB]
  - ed87a5aafc2dfef72e87fcd107f89757.html; charset=utf-8[45KB]
  - f4d4aab293f2a305cb5e950f8c81b106.x-javascript[862B]
  - fd5056ed555d77b134e33c2ca7e50c0e.html; charset=gb2312[394KB]
- dump.txt[79KB]
- config.ini[162B]
- NWebCrawler.exe[61KB]
- NWebCrawler.pdb[49KB]
- NWebCrawler.vshost.exe[13KB]
- NWebCrawler.vshost.exe.manifest[490B]
- NWebCrawlerLib.exe[22KB]
- NWebCrawlerLib.pdb[59KB]
- NWebCrawlerLib_1_5_2010.log[660B]
NWebCrawler
- obj
  - Debug
    - TempPE
    - NWebCrawler.csproj.FileListAbsolute.txt[989B]
    - NWebCrawler.csproj.GenerateResource.Cache[915B]
    - NWebCrawler.exe[61KB]
    - NWebCrawler.MainForm.resources[23KB]
    - NWebCrawler.pdb[49KB]
    - NWebCrawler.Properties.Resources.resources[180B]
    - NWebCrawler.SettingsForm.resources[180B]
    - ResolveAssemblyReference.cache[6KB]
- Properties
  - AssemblyInfo.cs[1KB]
  - Resources.Designer.cs[2KB]
  - Resources.resx[5KB]
  - Settings.Designer.cs[1KB]
  - Settings.settings[249B]
- MainForm.cs[6KB]
- MainForm.Designer.cs[22KB]
- MainForm.resx[43KB]
- NWebCrawler.csproj[4KB]
- Program.cs[506B]
- SettingsForm.cs[2KB]
- SettingsForm.Designer.cs[57KB]
- SettingsForm.resx[5KB]
- config.ini[109B]
NWebCrawlerLib
- bin
- Common
  - Logger.cs[2KB]
  - PriorityQueue.cs[4KB]
- obj
  - Debug
    - TempPE
    - NWebCrawlerLib.csproj.FileListAbsolute.txt[487B]
    - NWebCrawlerLib.exe[22KB]
    - NWebCrawlerLib.pdb[59KB]
- Properties
  - AssemblyInfo.cs[1KB]
- CrawleHistroyEntry.cs[323B]
- CrawlerThread.cs[10KB]
- Downloader.cs[5KB]
- NWebCrawlerLib.csproj[2KB]
- Parser.cs[2KB]
- Program.cs[1KB]
- Settings.cs[3KB]
- UrlFrontierQueueManager.cs[3KB]
- Utility.cs[3KB]
51aspx源码必读.txt[2KB]
from.gif[4KB]
NWebCrawler.sln[1KB]
NWebCrawler.suo[21KB]
最新Asp.Net源码下载.url[125B]

源码介绍

源码参数

源码类别：【开发辅助】
源码类型：C/S
适合人群：菜鸟进阶
授权类型：免费版
开发语言：C#
数据库：无数据库
源码大小：780K
相关网址：【免费浏览】
源码指数： 
完整度： 

网络爬虫程序源码

这是一款用 C# 编写的网络爬虫
主要特性有：
可配置：线程数、线程等待时间，连接超时时间，可爬取文件类型和优先级、下载目录等。
状态栏显示统计信息：排入队列URL数，已下载文件数，已下载总字节数，CPU使用率和可用内存等。
有偏好的爬虫：可针对爬取的资源类型设置不同的优先级。
健壮性：十几项URL正规化策略以排除冗余下载、爬虫陷阱避免策略的使用等、多种策略以解析相对路径等。
较好的性能：基于正则表达式的页面解析、适度加锁、维持HTTP连接等。

今后有空可能加入的特性：
新特性介绍
爬取文件用Berkeley DB存储提高性能：常用操作系统不善于处理大量小文件
基于URL Ranking的优先级队列主题爬虫：机器学习算法对链接与主题相关度进行评估，并按照得出的优先级顺序进行爬取
爬虫礼仪遵循爬虫禁止协议、以及避免对服务器资源的过度使用等
性能优化用UDP取代封装好的HttpWebRequest/Response
DNS缓存
异步的DNS地址解析
硬盘缓存或内存数据库以避免频繁的磁盘寻道
分布式爬虫以扩展单机能力（CPU、内存和硬盘访问）

责任编辑：wxc