diff --git a/README.md b/README.md index d5d02f0..77c5564 100644 --- a/README.md +++ b/README.md @@ -270,3 +270,6 @@ In the above example the `cpu` is the hex represantation of `some-kinda-cpu` and - No. The resolver must be an IPv4 address. We pass this value to `inet_addr()` function which accepts an IPv4. 4. How fast it can scan domain names? - It highly depends on your network and the (remote) resolver you use. +5. Why scanning one domain name takes much time? + - bulkDNS designed to be used for large-scale measurement. At the time of initialization, it launches dozens of threads in the memory. Therefore, + it's not suitable for scanning one domain name. You can use dig for that! diff --git a/modules/README.md b/modules/README.md new file mode 100644 index 0000000..ae04c50 --- /dev/null +++ b/modules/README.md @@ -0,0 +1,195 @@ +## Modules for bulkDNS + + +bulkDNS modules are written in Lua programming language. This tutorial shows how to write a module for a customized scan scenarios using Lua and bulkDNS. Before writing modules, you need to make sure to compile bulkDNS with Lua (using `make with-lua', by following the instruction in the main README file.) + +### How to write a module for customized scan scenario + +BulkDNS accepts a Lua script file using the switch `--lua-script`. After launching the scanner, each thread (pthread) creates a Lua state in the memory, run the file one time and then for each entry, it calls the `main` function in the Lua file. Therefore, your Lua script must have a `main` function which accepts only one parameter: *one line of the input file passed to bulkDNS*. + +The `main` function must return exactly one value: _whatever you want to log in the output file_. + +Therefore, here is the structure of the Lua script you pass to bulkDNS. + +```lua + -- whatever import module you want + -- + -- + -- whatever code or function you want to have + + -- if you define a global variable here, it will be + -- available as long as bulkDNS is running + -- for examle: + -- global_cache = {} + + + function main(input_line) + -- input_line: one of the entries of the input file + -- you passed to bulkDNS + + -- body of the function + -- body of the function + -- body of the function + + return "whatever you return will be stored in output" + end +``` + +If you return `nil` from the `main` function, then nothing will be logged in the output. This is very useful and we'll see an example later. + +It's also very important to note that whatever global variable you define in your Lua file will be available until the end of the scan. This is on purpose! In this way, you can keep the states for different entries and have a dynamic scan. For examle, one use-case of this feature is to implement a global LRU DNS cache in your Lua file! + +#### First example: find NXdomains + +Here is the scan scenario: I have a list of domain names and I want to output only those with DNS response code of NXDomain. +We know that NXDomain is `rcode=3` in a DNS packet. +So here is the code in Lua: + +```lua +-- save this code in a file nxscanner.lua + +local sdns = require("libsdns") +assert(sdns) + +function main(line) + -- create a query packet + local query = sdns.create_query(line, "A", "IN") + -- make sure the query packet created successfully + if query == nil then return nil end + + -- parameters for sending to cloudflare servers + tbl_send = {dstport=53, timeout=3, dstip="1.1.1.1"} + + -- make a payload from our query packet + to_send = sdns.to_network(query) + + -- make sure we created the payload successfully + if to_send == nil then return nil end + + -- add it to our parameters + tbl_send.to_send = to_send + + -- send it using sdns library + from_udp = sdns.send_udp(tbl_send) + + -- make sure we have the answer payload + if from_udp == nil then return nil end + + -- convert the payload to DNS packet + answer = sdns.from_network(from_udp) + + -- make sure the conversion was successful + if answer == nil then return nil end + + -- get the header of the DNS packet + header = sdns.get_header(answer) + + -- make sure you got the header + if header == nil then return nil end + + -- check if rcode==3 or not + if header.rcode == 3 then + -- send it to the output + return line + else + return nil + end +end +``` + +That's it! You just made a nice NX scanner! + +Before running bulkDNS, make sure you don't have any syntax error. You can do it by running `lua nxscanner.lua` in your bash. You should get nothing as output. + +Now create a list of domain names and store it in input_file.txt: +```text +microsoft.com +google.com +yahoo.com +filovirid.com +urlabuse.com +nonexistentdomainslakfjas.com +secondnotexistdomain234234.net +``` + +The last two domains must be in the output as they don't exist. + +Run bulkDNS like this: +```bash +./bulkdns --lua-script=nxscanner.lua --concurrency=10 input_file.txt +``` + +it prints out the output in your terminal (you can specify a file to save the output using `-o` or `--output` switch). + +You can download both `nxscanner.lua` and `input_file.txt` from this directory. + +It's important to note that I used `sdns` lua library for doing DNS operation in Lua. However, you can use whatever Lua library that you prefer. For socket operation, you can also use other Lua libraries like [this one](https://lunarmodules.github.io/luasocket/). However, if you want to use `sdns` Lua library, make sure you follow [this tutorial](https://github.com/maroofi/sdns/blob/main/lua/README.md). + + +#### Second example: SPF scanner + +In case you forgot, SPF stands for _**S**ender **P**olicy **F**ramework_. + +We want to extract **only** the SPF records (not the whole TXT records) of a list of domain names. Here is the code to do the job: + +```lua +-- save it in a file: spfscanner.lua +local json = require "json" +local sdns = require "libsdns" + +local find = string.find +local json_encode = json.encode +local insert = table.insert + +function main(line) + local query = sdns.create_query(line, "TXT", "IN") + if query == nil then return nil end + tbl_send = {dstport=53, timeout=3, dstip="1.1.1.1"} + to_send = sdns.to_network(query) + if to_send == nil then return nil end + tbl_send.to_send = to_send + from_udp = sdns.send_udp(tbl_send) + if from_udp == nil then return nil end + answer = sdns.from_network(from_udp) + if answer == nil then return nil end + local header = sdns.get_header(answer) + if header == nil then return nil end + if header.tc == 1 then + -- we need to do TCP + from_tcp = sdns.send_tcp(tbl_send) + if from_tcp == nil then return nil end + answer = sdns.from_network(from_tcp) + if answer == nil then return nil end + header = sdns.get_header(answer) + end + local spf = {} + question = sdns.get_question(answer) or {} + local num = header.ancount or 0 + if num == 0 then return nil end + for i=1, num do + a = sdns.get_answer(answer, i) + a = ((a or {}).rdata or {}).txtdata or nil + if a == nil then goto continue end + if find(a, "^[vV]=[sS][pP][fF]1%s+") ~= nil then + table.insert(spf, a) + end + ::continue:: + end + return json_encode({name=line, data=spf}); +end +``` + +And run it like: +```bash +./bulkdns --lua-script=spfscanner.lua --concurrency=10 input_file.txt +``` + +I am using the json library from [here](https://github.com/rxi/json.lua). I sotred it in the module directory. + + +You can find more modules in this directory and all are documented. + + +### Running bulkDNS in Server mode + +**TODO**: write this part. \ No newline at end of file diff --git a/modules/input_file.txt b/modules/input_file.txt new file mode 100644 index 0000000..68cbf7e --- /dev/null +++ b/modules/input_file.txt @@ -0,0 +1,7 @@ +microsoft.com +google.com +yahoo.com +filovirid.com +urlabuse.com +nonexistentdomainslakfjas.com +secondnotexistdomain234234.net diff --git a/modules/json.lua b/modules/json.lua new file mode 100644 index 0000000..711ef78 --- /dev/null +++ b/modules/json.lua @@ -0,0 +1,388 @@ +-- +-- json.lua +-- +-- Copyright (c) 2020 rxi +-- +-- Permission is hereby granted, free of charge, to any person obtaining a copy of +-- this software and associated documentation files (the "Software"), to deal in +-- the Software without restriction, including without limitation the rights to +-- use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +-- of the Software, and to permit persons to whom the Software is furnished to do +-- so, subject to the following conditions: +-- +-- The above copyright notice and this permission notice shall be included in all +-- copies or substantial portions of the Software. +-- +-- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +-- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +-- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +-- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +-- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +-- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +-- SOFTWARE. +-- + +local json = { _version = "0.1.2" } + +------------------------------------------------------------------------------- +-- Encode +------------------------------------------------------------------------------- + +local encode + +local escape_char_map = { + [ "\\" ] = "\\", + [ "\"" ] = "\"", + [ "\b" ] = "b", + [ "\f" ] = "f", + [ "\n" ] = "n", + [ "\r" ] = "r", + [ "\t" ] = "t", +} + +local escape_char_map_inv = { [ "/" ] = "/" } +for k, v in pairs(escape_char_map) do + escape_char_map_inv[v] = k +end + + +local function escape_char(c) + return "\\" .. (escape_char_map[c] or string.format("u%04x", c:byte())) +end + + +local function encode_nil(val) + return "null" +end + + +local function encode_table(val, stack) + local res = {} + stack = stack or {} + + -- Circular reference? + if stack[val] then error("circular reference") end + + stack[val] = true + + if rawget(val, 1) ~= nil or next(val) == nil then + -- Treat as array -- check keys are valid and it is not sparse + local n = 0 + for k in pairs(val) do + if type(k) ~= "number" then + error("invalid table: mixed or invalid key types") + end + n = n + 1 + end + if n ~= #val then + error("invalid table: sparse array") + end + -- Encode + for i, v in ipairs(val) do + table.insert(res, encode(v, stack)) + end + stack[val] = nil + return "[" .. table.concat(res, ",") .. "]" + + else + -- Treat as an object + for k, v in pairs(val) do + if type(k) ~= "string" then + error("invalid table: mixed or invalid key types") + end + table.insert(res, encode(k, stack) .. ":" .. encode(v, stack)) + end + stack[val] = nil + return "{" .. table.concat(res, ",") .. "}" + end +end + + +local function encode_string(val) + return '"' .. val:gsub('[%z\1-\31\\"]', escape_char) .. '"' +end + + +local function encode_number(val) + -- Check for NaN, -inf and inf + if val ~= val or val <= -math.huge or val >= math.huge then + error("unexpected number value '" .. tostring(val) .. "'") + end + return string.format("%.14g", val) +end + + +local type_func_map = { + [ "nil" ] = encode_nil, + [ "table" ] = encode_table, + [ "string" ] = encode_string, + [ "number" ] = encode_number, + [ "boolean" ] = tostring, +} + + +encode = function(val, stack) + local t = type(val) + local f = type_func_map[t] + if f then + return f(val, stack) + end + error("unexpected type '" .. t .. "'") +end + + +function json.encode(val) + return ( encode(val) ) +end + + +------------------------------------------------------------------------------- +-- Decode +------------------------------------------------------------------------------- + +local parse + +local function create_set(...) + local res = {} + for i = 1, select("#", ...) do + res[ select(i, ...) ] = true + end + return res +end + +local space_chars = create_set(" ", "\t", "\r", "\n") +local delim_chars = create_set(" ", "\t", "\r", "\n", "]", "}", ",") +local escape_chars = create_set("\\", "/", '"', "b", "f", "n", "r", "t", "u") +local literals = create_set("true", "false", "null") + +local literal_map = { + [ "true" ] = true, + [ "false" ] = false, + [ "null" ] = nil, +} + + +local function next_char(str, idx, set, negate) + for i = idx, #str do + if set[str:sub(i, i)] ~= negate then + return i + end + end + return #str + 1 +end + + +local function decode_error(str, idx, msg) + local line_count = 1 + local col_count = 1 + for i = 1, idx - 1 do + col_count = col_count + 1 + if str:sub(i, i) == "\n" then + line_count = line_count + 1 + col_count = 1 + end + end + error( string.format("%s at line %d col %d", msg, line_count, col_count) ) +end + + +local function codepoint_to_utf8(n) + -- http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=iws-appendixa + local f = math.floor + if n <= 0x7f then + return string.char(n) + elseif n <= 0x7ff then + return string.char(f(n / 64) + 192, n % 64 + 128) + elseif n <= 0xffff then + return string.char(f(n / 4096) + 224, f(n % 4096 / 64) + 128, n % 64 + 128) + elseif n <= 0x10ffff then + return string.char(f(n / 262144) + 240, f(n % 262144 / 4096) + 128, + f(n % 4096 / 64) + 128, n % 64 + 128) + end + error( string.format("invalid unicode codepoint '%x'", n) ) +end + + +local function parse_unicode_escape(s) + local n1 = tonumber( s:sub(1, 4), 16 ) + local n2 = tonumber( s:sub(7, 10), 16 ) + -- Surrogate pair? + if n2 then + return codepoint_to_utf8((n1 - 0xd800) * 0x400 + (n2 - 0xdc00) + 0x10000) + else + return codepoint_to_utf8(n1) + end +end + + +local function parse_string(str, i) + local res = "" + local j = i + 1 + local k = j + + while j <= #str do + local x = str:byte(j) + + if x < 32 then + decode_error(str, j, "control character in string") + + elseif x == 92 then -- `\`: Escape + res = res .. str:sub(k, j - 1) + j = j + 1 + local c = str:sub(j, j) + if c == "u" then + local hex = str:match("^[dD][89aAbB]%x%x\\u%x%x%x%x", j + 1) + or str:match("^%x%x%x%x", j + 1) + or decode_error(str, j - 1, "invalid unicode escape in string") + res = res .. parse_unicode_escape(hex) + j = j + #hex + else + if not escape_chars[c] then + decode_error(str, j - 1, "invalid escape char '" .. c .. "' in string") + end + res = res .. escape_char_map_inv[c] + end + k = j + 1 + + elseif x == 34 then -- `"`: End of string + res = res .. str:sub(k, j - 1) + return res, j + 1 + end + + j = j + 1 + end + + decode_error(str, i, "expected closing quote for string") +end + + +local function parse_number(str, i) + local x = next_char(str, i, delim_chars) + local s = str:sub(i, x - 1) + local n = tonumber(s) + if not n then + decode_error(str, i, "invalid number '" .. s .. "'") + end + return n, x +end + + +local function parse_literal(str, i) + local x = next_char(str, i, delim_chars) + local word = str:sub(i, x - 1) + if not literals[word] then + decode_error(str, i, "invalid literal '" .. word .. "'") + end + return literal_map[word], x +end + + +local function parse_array(str, i) + local res = {} + local n = 1 + i = i + 1 + while 1 do + local x + i = next_char(str, i, space_chars, true) + -- Empty / end of array? + if str:sub(i, i) == "]" then + i = i + 1 + break + end + -- Read token + x, i = parse(str, i) + res[n] = x + n = n + 1 + -- Next token + i = next_char(str, i, space_chars, true) + local chr = str:sub(i, i) + i = i + 1 + if chr == "]" then break end + if chr ~= "," then decode_error(str, i, "expected ']' or ','") end + end + return res, i +end + + +local function parse_object(str, i) + local res = {} + i = i + 1 + while 1 do + local key, val + i = next_char(str, i, space_chars, true) + -- Empty / end of object? + if str:sub(i, i) == "}" then + i = i + 1 + break + end + -- Read key + if str:sub(i, i) ~= '"' then + decode_error(str, i, "expected string for key") + end + key, i = parse(str, i) + -- Read ':' delimiter + i = next_char(str, i, space_chars, true) + if str:sub(i, i) ~= ":" then + decode_error(str, i, "expected ':' after key") + end + i = next_char(str, i + 1, space_chars, true) + -- Read value + val, i = parse(str, i) + -- Set + res[key] = val + -- Next token + i = next_char(str, i, space_chars, true) + local chr = str:sub(i, i) + i = i + 1 + if chr == "}" then break end + if chr ~= "," then decode_error(str, i, "expected '}' or ','") end + end + return res, i +end + + +local char_func_map = { + [ '"' ] = parse_string, + [ "0" ] = parse_number, + [ "1" ] = parse_number, + [ "2" ] = parse_number, + [ "3" ] = parse_number, + [ "4" ] = parse_number, + [ "5" ] = parse_number, + [ "6" ] = parse_number, + [ "7" ] = parse_number, + [ "8" ] = parse_number, + [ "9" ] = parse_number, + [ "-" ] = parse_number, + [ "t" ] = parse_literal, + [ "f" ] = parse_literal, + [ "n" ] = parse_literal, + [ "[" ] = parse_array, + [ "{" ] = parse_object, +} + + +parse = function(str, idx) + local chr = str:sub(idx, idx) + local f = char_func_map[chr] + if f then + return f(str, idx) + end + decode_error(str, idx, "unexpected character '" .. chr .. "'") +end + + +function json.decode(str) + if type(str) ~= "string" then + error("expected argument of type string, got " .. type(str)) + end + local res, idx = parse(str, next_char(str, 1, space_chars, true)) + idx = next_char(str, idx, space_chars, true) + if idx <= #str then + decode_error(str, idx, "trailing garbage") + end + return res +end + + +return json diff --git a/modules/nxscanner.lua b/modules/nxscanner.lua new file mode 100644 index 0000000..60b88bc --- /dev/null +++ b/modules/nxscanner.lua @@ -0,0 +1,47 @@ +local sdns = require("libsdns") +assert(sdns) + +function main(line) + -- create a query packet + local query = sdns.create_query(line, "A", "IN") + -- make sure the query packet created successfully + if query == nil then return nil end + + -- parameters for sending to cloudflare servers + tbl_send = {dstport=53, timeout=3, dstip="1.1.1.1"} + + -- make a payload from our query packet + to_send = sdns.to_network(query) + + -- make sure we created the payload successfully + if to_send == nil then return nil end + + -- add it to our parameters + tbl_send.to_send = to_send + + -- send it using sdns library + from_udp = sdns.send_udp(tbl_send) + + -- make sure we have the answer payload + if from_udp == nil then return nil end + + -- convert the payload to DNS packet + answer = sdns.from_network(from_udp) + + -- make sure the conversion was successful + if answer == nil then return nil end + + -- get the header of the DNS packet + header = sdns.get_header(answer) + + -- make sure you got the header + if header == nil then return nil end + + -- check if rcode==3 or not + if header.rcode == 3 then + -- send it to the output + return line + else + return nil + end +end diff --git a/modules/spfscanner.lua b/modules/spfscanner.lua new file mode 100644 index 0000000..53eb40b --- /dev/null +++ b/modules/spfscanner.lua @@ -0,0 +1,43 @@ +local json = require "json" +local sdns = require "libsdns" + +local find = string.find +local json_encode = json.encode +local insert = table.insert + +function main(line) + local query = sdns.create_query(line, "TXT", "IN") + if query == nil then return nil end + tbl_send = {dstport=53, timeout=3, dstip="1.1.1.1"} + to_send = sdns.to_network(query) + if to_send == nil then return nil end + tbl_send.to_send = to_send + from_udp = sdns.send_udp(tbl_send) + if from_udp == nil then return nil end + answer = sdns.from_network(from_udp) + if answer == nil then return nil end + local header = sdns.get_header(answer) + if header == nil then return nil end + if header.tc == 1 then + -- we need to do TCP + from_tcp = sdns.send_tcp(tbl_send) + if from_tcp == nil then return nil end + answer = sdns.from_network(from_tcp) + if answer == nil then return nil end + header = sdns.get_header(answer) + end + local spf = {} + question = sdns.get_question(answer) or {} + local num = header.ancount or 0 + if num == 0 then return nil end + for i=1, num do + a = sdns.get_answer(answer, i) + a = ((a or {}).rdata or {}).txtdata or nil + if a == nil then goto continue end + if find(a, "^[vV]=[sS][pP][fF]1%s+") ~= nil then + table.insert(spf, a) + end + ::continue:: + end + return json_encode({name=line, data=spf}); +end