-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
executable file
·310 lines (277 loc) · 10.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Walker</title>
<style>
.outline {
margin: 0px;
max-width: 900px;
overflow: auto;
text-overflow: ellipsis;
border: 1px solid #e1e4e8;
padding: 10px 40px;
padding-bottom: 20px;
border-radius: 2px;
margin-left: auto;
margin-right: auto;
}
body {
font: 400 16px/1.5 'Helvetica Neue', Helvetica, Arial, sans-serif;
color: #111;
background-color: #fdfdfd;
-webkit-text-size-adjust: 100%;
-webkit-font-feature-settings: 'kern' 1;
-moz-font-feature-settings: 'kern' 1;
-o-font-feature-settings: 'kern' 1;
font-feature-settings: 'kern' 1;
font-kerning: normal;
padding: 30px;
}
@media only screen and (max-width: 600px) {
body {
padding: 5px;
}
main {
padding: 0px 20px 20px 20px !important;
}
}
main {
margin: 0px;
max-width: 900px;
border: 1px solid #e1e4e8;
padding: 10px 40px;
padding-bottom: 20px;
border-radius: 2px;
margin-left: auto;
margin-right: auto;
}
hr {
color: #bbb;
background-color: #bbb;
height: 1px;
flex: 0 1 auto;
margin: 1em 0;
padding: 0;
border: none;
}
/**
* Links
*/
a {
color: #0366d6;
text-decoration: none;
}
a:visited {
color: #0366d6;
}
a:hover {
color: #0366d6;
text-decoration: underline;
}
pre {
background-color: #f6f8fa;
border-radius: 3px;
font-size: 85%;
line-height: 1.45;
overflow: auto;
padding: 16px;
}
/**
* Code blocks
*/
code {
background-color: rgba(27, 31, 35, 0.05);
border-radius: 3px;
font-size: 85%;
margin: 0;
word-wrap: break-word;
padding: 0.2em 0.4em;
font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier,
monospace;
}
pre > code {
background-color: transparent;
border: 0;
display: inline;
line-height: inherit;
margin: 0;
overflow: visible;
padding: 0;
word-wrap: normal;
font-size: 100%;
}
/**
* Blockquotes
*/
blockquote {
margin-left: 30px;
margin-top: 0px;
margin-bottom: 16px;
border-left-width: 3px;
padding: 0 1em;
color: #828282;
border-left: 4px solid #e8e8e8;
padding-left: 15px;
font-size: 18px;
letter-spacing: -1px;
font-style: italic;
}
blockquote * {
font-style: normal !important;
letter-spacing: 0;
color: #6a737d !important;
}
/**
* Tables
*/
table {
border-spacing: 2px;
display: block;
font-size: 14px;
overflow: auto;
width: 100%;
margin-bottom: 16px;
border-spacing: 0;
border-collapse: collapse;
}
td {
padding: 6px 13px;
border: 1px solid #dfe2e5;
}
th {
font-weight: 600;
padding: 6px 13px;
border: 1px solid #dfe2e5;
}
tr {
background-color: #fff;
border-top: 1px solid #c6cbd1;
}
table tr:nth-child(2n) {
background-color: #f6f8fa;
}
/**
* Others
*/
img {
max-width: 100%;
}
p {
line-height: 24px;
font-weight: 400;
font-size: 16px;
color: #24292e;
}
ul {
margin-top: 0;
}
li {
color: #24292e;
font-size: 16px;
font-weight: 400;
line-height: 1.5;
}
li + li {
margin-top: 0.25em;
}
* {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial,
sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';
color: #24292e;
}
a:visited {
color: #0366d6;
}
h1 {
border-bottom: 1px solid #eaecef;
color: #111;
/* Darker */
}
h2 {
border-bottom: 1px solid #eaecef;
color: #111;
/* Darker */
}
h3 {
color: #111;
}
</style>
</head>
<body>
<div class="outline">
<h1>Walker</h1>
<p>Walker is a tool that performs a recursive analysis of a website to search for faulty links. It operates under the premise that the homepage of the site in question leads to other pages within the website, which subsequently lead to yet more pages.</p>
<p>It is worth noting that this process may take an indeterminate amount of time, as a website may contain an infinite number of nested links. A potential future update to the recursive function could be the addition of a <code>depth</code> parameter, which would restrict recursion to only a specified number of levels. I don't plan on implementing this any time soon due to CBSE board exams and college applications, though!</p>
<p>Also, <code>walker</code> is fast as fuck. After it acquires all the URLs in a website, it sends them parallel requests to quickly verify if they're broken or not. There is a <a href="#rate-limits">rate-limiting issue</a> that I am quite not sure how to fix, but I think I'll eventually figure it out. Foe what it's worth, the only cost-heavy part of <code>walker</code> is acquiring the URLs. That process is neither concurrent, nor parallel, and hence takes a while.</p>
<h2>Quirks</h2>
<h3>Client side rendering</h3>
<p>Since the implementation of this tool works through fetching the HTML of the website in question, it would be impossible for it to retrieve HTML for pages that render on the client. Basically, only the HTML available when you view the page source is analyzed. This often includes websites that are either statically generated or rendered on the server.</p>
<h3>Rate limits</h3>
<p>Since I don't wait between each request, some websites might enforce their rate-limiting policies on <code>walker</code>, and hence cause it to error out for URLs which are working perfectly fine. In these cases, if the API is returning semantic error codes, <code>walker</code> will display something like <code>429 Too Many Requests</code>.</p>
<p><code>walker</code> has a timeout of 5 seconds between each request. If the URL does not return a response within 5 seconds, it will error out and show that the operation was timed out.</p>
<h3>Head requests</h3>
<p>To save on bytes, <code>walker</code> performs <code>HEAD</code> requests instead of <code>GET</code> requests. However, some websites might deny responding to this method, which could lead to false negatives. In these cases, <code>walker</code> will show an error like <code>405 Method Not Allowed</code>.</p>
<h2>Install</h2>
<p>To install <code>walker</code>, you will need to have <code>cargo</code> configured in your environment. You can see how to do so <a href="https://doc.rust-lang.org/cargo/getting-started/installation.html">here</a>. When you're done doing so, run:</p>
<pre><code class="language-zsh">cargo install link-walker
</code></pre>
<p>...and let it install. Once <code>link-walker</code> is done installing, you can simply run:</p>
<pre><code class="language-zsh">link-walker -h
</code></pre>
<p>...to get information on how to use the tool.</p>
<h2>Usage</h2>
<p>Using <code>walker</code> is easy. It has a dead-simple CLI interface that can be used to visually see the results of the analysis. Options for it are:</p>
<pre><code class="language-bash">Usage: link-walker [OPTIONS] --url <URL>
Options:
-u, --url <URL> URL of the website to analyze links from
-r, --relative Whether to perform a deep search or not
-d, --debug Shows what URL walker is currently on
-c, --construct Constructs the stream of responses into a string and copies it to the clipboard
-s, --singular Checks if the domain of the URL is resolvable
-h, --help Print help
-V, --version Print version
</code></pre>
<p>So, for example, doing:</p>
<pre><code class="language-bash">link-walker --url "https://ynb.sh"
</code></pre>
<p>...would result in:</p>
<pre><code class="language-bash">Received 3 links. Iterating now...
https://ynb.sh/posts: 200 OK
https://github.com/ynbh: 200 OK
https://ynb.sh: 200 OK
Stats
Time to get all links: 0 seconds
Time to verify links: 0 seconds
</code></pre>
<p>But when used with the <code>-r</code> argument, it would result in something like:</p>
<pre><code class="language-bash">Received 17 links. Iterating now...
https://en.wikipedia.org/wiki/Test_of_English_as_a_Foreign_Language: 200 OK
https://ynb.sh/assets/toefl-listening.png: 200 OK
https://ynb.sh/assets/toefl-reading.png: 200 OK
https://ynb.sh/posts/preparing-for-and-writing-the-TOEFL: 200 OK
An error occurred: error sending request for url (http://www.icsscsummerofcode.com/): error trying to connect: dns error: failed to lookup address information: nodename nor servname provided, or not known
https://ynb.sh/posts: 200 OK
https://ynb.sh/assets/toefl-writing-template.png: 200 OK
https://www.toeflresources.com/speaking-section/toefl-speaking-templates: 200 OK
https://ynb.sh/assets/toefl-speaking.png: 200 OK
https://ynb.sh: 200 OK
https://ynb.sh/assets/toefl-writing.png: 200 OK
http://zeroclipboard.org: 500 Internal Server Error
https://github.com/ynbh: 200 OK
https://ynb.sh/posts/free-speech-and-some-concerns: 200 OK
https://ynb.sh/posts/black-panther-wakanda-forever-review: 200 OK
https://ynb.sh/assets/toefl-speaking-template.png: 200 OK
Stats
Time to get all links: 1 seconds
Time to verify links: 0 seconds
</code></pre>
<p>To debug what URL <code>walker</code> is currently fetching, simply pass it a <code>-d</code> debug flag.</p>
<p>In situations where <code>walker</code> is unable to resolve some arbitrary URL, it will accurately display the exact error that occurred. Often, this error occurs because <code>reqwest</code> is unable to resolve its DNS. You can check if that is the case by running <code>link-walker --url <URL> -s</code>. If it does not return an error, there is probably something else going on with the URL that needs to be looked at.</p>
<h2>Extra</h2>
<p>Note that this is an experimental tool, and I only made this to learn more about Rust. I wrote a <a href="https://ynb.sh/posts/walker">blog post detailing how I made it!</a></p>
</div>
</body>
</html>