-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathREADME
144 lines (117 loc) · 6.96 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
### readme version v10.4
*) Bash scripts to update blacklisted domains in BIND9-Response Policy Zone
*) this v10.4 version is the next release after the previous release. it's a maintenance
release which improves version compatibility with sources-list in grab_urls
### author:
ngadimin@warnet-ersa.net
~ https://github.com/ngadmini/
~ https://gist.github.com/ngadmini
### license:
*) Creative Commons Attribution-ShareAlike 4.0 International License
*) For more information, please refer to <https://creativecommons.org/licenses/by-sa/4.0/>
### what's new in v10.4:
*) splitting trust+ category into 3 sub-categories, i.e: trust+aa.domain, trust+ab.domain and trust+ac.domain
### Get url to download script-pack (latest version) :
~ curl -s https://api.github.com/repos/ngadmini/Grabbing-Blacklist-for-Bind9-RPZ/releases/latest \
| grep download_url | awk -F\" '{print $4}'
### file parts
1. grab_http.sh --grabbing and proccessing raw domains.
*) run this script as long as the below script and rpz.* are available in the same directory.
*) script completed by offering 4 options. IF no option are selected, it just generates a txt.*
which are not free from duplicate entries and invalid TLDs
2. grab_duplic.sh --pruning ipv4, duplicate & sub-domain (if parent domain exist) entries across CATEGORIE
*) you can run this scipt separately, if only txt.* are available in the same directory.
~ ls | grep '^txt' | tr "\n" " "
txt.adult txt.ipv4 txt.malware txt.publicite txt.redirector txt.trust+
3. grab_build.sh --rewriting to BIND9-rpz format (generate to db.* files)
*) you can run this scipt separately, if only txt.* are available in the same directory.
4. grab_cereal.sh --incrementing serial zones.
*) you can run this scipt separately, if only rpz.* are available in the same directory.
if not, it try to get from bind9-server. it generate new serial at zone-files
5. grab_config --configuration file.
*) please SEE prerequisites section at this README
6. grab_library --a library of functions and fundamental values
*) NO need modify, except you want change RPZ-policy triggers and actions.
*) please SEE prerequisites section at this README
7. grab_urls --contain urls list of source files.
*) case sensitive, sort as is, line count and no blank lines
8. grab_regex --contain reguler expressions.
*) case sensitive, sort as is, line count and no blank lines
*) intended for filtering, fixing and removing invalid entries, i.e :
- less than 4 characters and TLDs larger than 24 characters (currently in IANA)
~ curl -s "https://data.iana.org/TLD/tlds-alpha-by-domain.txt" \
| sed '/#/d;s/[A-Z]/\L&/g' | awk '{ print length(), $0 | "sort -n"}' | tail -1
24 xn--vermgensberatung-pwb
- construct with international characters (non ASCII). check it's with :
~ LC_ALL=C grep -P -n '[^\x00-\x7F]'
- explicit TLDs (adult & gambling)
~ \.(adult|bet|cam(era)?|cas(h|ino)|gay|goog|lgbt|lotto|poker|porn|sexy?|tube|webcam|xxx)$
- turns back passes filtered to the proper category
- ipv4-address filtering
9. grab_rsync.sh --syncronizing updated db.* and rpz.* files to BIND9-server.
*) in addition to synchronizing, this script also generates a tar-file to backup
old db.* and rpz.* files in BIND9-server and save it in $HOME directory
10. rpz.adultaa rpz.adultab rpz.adultac rpz.adultad rpz.adultae rpz.adultaf rpz.adultag
rpz.ipv4 rpz.malware rpz.publicite rpz.redirector rpz.trust+aa rpz.trust+ab rpz.trust+ac
11. .shellcheckrc --intended to shellcheck portability
### prerequisites:
*) on your linux desktop workstation
# check required package (dependencies)
~$ sudo apt policy curl dos2unix idn faketime rsync libnet-netmask-perl
~$ sudo apt install -y curl dos2unix idn faketime rsync libnet-netmask-perl
~$ echo 'insecure' | tee -a ~/.curlrc
*) modify file grab_config according to your BIND9-server environment
# modify file grab_config to fit your BIND9-server environment
# *) type in format VARIABLE=value
# *) blank lines and leading '#' will be ignored
# *) all variables must be determined. if NOT it will fallback to their default
# values. THEN and OR
# *) change all variables in grab_library refer to this file. go to line:
# ~ nano +$(grep -n 'f_dft()' grab_library | cut -d: -f1) grab_library
# specify in fqdn or ip-address
HOST=rpz.warnet-ersa.net
# fixed number of lines grab_regex. DON'T CHANGE
REGEX=4
# reload Remote Name Daemon Control (rndc)
# set 'yes' to run <rndc reload> or 'no' to <reboot>
RNDC_RELOAD=yes
# change Resource Record (RR) and Right Hand Value (RH) if you have other
# policy of RR and RH Value
# for example:
# RPZ_DOM=CNAME block.example.net.
# or:
# RPZ_DOM=IN CNAME block.example.net.
# RPZ_IP4=rpz-ip CNAME block.example.net.
# for more detail about rpz policy action, please see:
# <https://www.zytrax.com/books/dns/ch7/rpz.html#policy-actions>
# defining for 'RR=CNAME' and 'RH=.'
RPZ_DOM=CNAME .
# defining for 'RR=rpz-ip CNAME' and 'RH=.'
RPZ_IP4=rpz-ip CNAME .
# fixed number of lines grab_urls. DON'T CHANGE
URLS=19
# location of zone-files and dataBase (rpz.* and db.*) in bind9-server
ZONE_DIR=/etc/bind/zones-rpz
*) on your BIND9-server
check required package (dependencies)
~$ apt policy rsync
~$ apt install -y rsync
*) passwordless SSH
your linux desktop and BIND9-server, both must configure to use passwordless for
ssh, scp dan rsync connections. please reffer to :
~ https://github.com/ngadmini/Grabbing-Blacklist-for-Bind9-RPZ/wiki/Passwordless-SSH-and-sudo
*) reconfigure your bind9-server to run BIND9-RPZ features. please reffer to :
~ https://github.com/ngadmini/Grabbing-Blacklist-for-Bind9-RPZ/wiki/Fitting-Environment
### usage:
*) execute grab_http.sh with non root privileges either directly as a root user or by use
of sudo command, from your linux desktop workstation then follow the next step
### output:
*) db.adultaa db.adultab db.adultac db.adultad db.adultae db.adultaf db.adultag
db.ipv4 db.malware db.publicite db.redirector db.trust+aa db.trust+ab db.trust+ac
*) rpz.adultaa rpz.adultab rpz.adultac rpz.adultad rpz.adultae rpz.adultaf rpz.adultag
rpz.ipv4 rpz.malware rpz.publicite rpz.redirector rpz.trust+aa rpz.trust+ab rpz.trust+ac
*) txt.adult txt.ipv4 txt.malware txt.publicite txt.redirector txt.trust+
~ https://github.com/ngadmini/partial-output
### questions, feedback and issues:
~ https://github.com/ngadmini/Grabbing-Blacklist-for-Bind9-RPZ/discussions
~ https://github.com/ngadmini/Grabbing-Blacklist-for-Bind9-RPZ/issues