code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: CSV generation scripts, API filter params, enriched CSV stubs

Browse Source 
- scripts/generate_bot_ip.py: download Tor exit nodes + curate scanner IPs (1353 entries)
- scripts/generate_bot_ja4.py: 31 bot JA4 fingerprints across 16 families
- scripts/generate_asn_data.py: 38 ASNs + 96 IP-to-ASN prefixes
- scripts/update-csv-data.sh: master orchestrator with --install-stubs
- api.py: add asn_org/country_code/ja4/bot_name filters on detections+scores
- pages.py: add /network route
- csv-stubs: enriched with generated data (Tor nodes, scanner IPs, etc.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-08 15:05:43 +02:00
parent c6ca352db9
commit b6184e6529
14 changed files with 3779 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
tests/integration/platform/csv-stubs/asn_reputation.csv
View File

@ -2,13 +2,38 @@ src_asn,label
16276,human
15557,human
3215,human
12322,human
5432,human
3320,human
6805,human
1136,human
1103,human
2856,human
8913,human
5607,human
3352,human
3269,human
7922,human
7018,human
701,human
20115,human
2516,human
4713,human
15169,human
8075,human
32934,human
13414,human
210644,datacenter
209083,datacenter
14061,datacenter
16509,datacenter
396982,datacenter
8560,datacenter
24940,datacenter
20473,datacenter
63949,datacenter
13335,datacenter
197695,hosting
51167,hosting
46606,hosting
26496,hosting

1 src_asn label
2 16276 human
3 15557 human
4 3215 human
5 12322 human
6 5432 human
7 3320 human
8 6805 human
9 1136 human
10 1103 human
11 2856 human
12 8913 human
13 5607 human
14 3352 human
15 3269 human
16 7922 human
17 7018 human
18 701 human
19 20115 human
20 2516 human
21 4713 human
22 15169 human
23 8075 human
24 32934 human
25 13414 human
26 210644 datacenter
27 209083 datacenter
28 14061 datacenter
29 16509 datacenter
30 396982 datacenter
31 8560 datacenter
32 24940 datacenter
33 20473 datacenter
34 63949 datacenter
35 13335 datacenter
36 197695 hosting
37 51167 hosting
38 46606 hosting
39 26496 hosting

1365
tests/integration/platform/csv-stubs/bot_ip.csv
View File

File diff suppressed because it is too large Load Diff

26
tests/integration/platform/csv-stubs/bot_ja4.csv
View File

@ -1,5 +1,31 @@
t13d030500_ffd59bab1b39_6e7f7df63e98,curl_scanner
t13d030600_ffd59bab1b39_6e7f7df63e98,curl_scanner
t13d020400_ffd59bab1b39_6e7f7df63e98,curl_scanner
t12d030500_ffd59bab1b39_6e7f7df63e98,curl_scanner
t13d020300_6b9b1b2c3d4e_ffd59bab1b39,python_requests_scanner
t13d020200_6b9b1b2c3d4e_ffd59bab1b39,python_requests_scanner
t13d010300_6b9b1b2c3d4e_aabbccddeeff,python_requests_scanner
t12d020300_6b9b1b2c3d4e_ffd59bab1b39,python_requests_scanner
t13d1517h2_8daaf6152771_b0da82dd1658,go_http_scanner
t13d1517h2_8daaf6152771_02713d6af862,go_http_scanner
t12d1517h2_8daaf6152771_b0da82dd1658,go_http_scanner
t10d170000_0a1b2c3d4e5f_1b2c3d4e5f60,Masscan
t10d010000_0a1b2c3d4e5f_000000000000,Masscan
t12d050700_5a6b7c8d9e0f_1a2b3c4d5e6f,zgrab_scanner
t12d050600_5a6b7c8d9e0f_1a2b3c4d5e6f,zgrab_scanner
t12d030400_5a6b7c8d9e0f_0000deadbeef,zmap_scanner
t13d010100_aabbccddeeff_0011223344aa,Headless_Chrome_Automation
t13d010100_aabbccddeeff_ffeeddccbbaa,Headless_Chrome_Automation
t13d1517h2_aabbccddeeff_0011223344aa,Headless_Chrome_Automation
t13d030500_deadbeef1234_cafebabe5678,node_scanner
t13d020300_deadbeef1234_cafebabe5678,node_scanner
t13d1517h2_1234567890ab_abcdef012345,java_scanner
t12d1517h2_1234567890ab_abcdef012345,java_scanner
t13d020300_fedcba987654_0123456789ab,ruby_scanner
t12d010100_aabbccddeeff_deadbeefdead,nikto_scanner
t12d010100_ffeeddccbbaa_baddcafef00d,sqlmap_scanner
t13d030600_deadbeefcafe_babe12345678,nuclei_scanner
t13d020200_abcdef012345_fedcba987654,scrapy_crawler
t13d020300_abcdef012345_1234abcd5678,scrapy_crawler
t10d010000_0000000000_000000000000,malware_c2_minimal
t12d010100_1111111111_222222222222,cobalt_strike_beacon

1 t13d030500_ffd59bab1b39_6e7f7df63e98 curl_scanner
2 t13d030600_ffd59bab1b39_6e7f7df63e98 curl_scanner
3 t13d020400_ffd59bab1b39_6e7f7df63e98 curl_scanner
4 t12d030500_ffd59bab1b39_6e7f7df63e98 curl_scanner
5 t13d020300_6b9b1b2c3d4e_ffd59bab1b39 python_requests_scanner
6 t13d020200_6b9b1b2c3d4e_ffd59bab1b39 python_requests_scanner
7 t13d010300_6b9b1b2c3d4e_aabbccddeeff python_requests_scanner
8 t12d020300_6b9b1b2c3d4e_ffd59bab1b39 python_requests_scanner
9 t13d1517h2_8daaf6152771_b0da82dd1658 go_http_scanner
10 t13d1517h2_8daaf6152771_02713d6af862 go_http_scanner
11 t12d1517h2_8daaf6152771_b0da82dd1658 go_http_scanner
12 t10d170000_0a1b2c3d4e5f_1b2c3d4e5f60 Masscan
13 t10d010000_0a1b2c3d4e5f_000000000000 Masscan
14 t12d050700_5a6b7c8d9e0f_1a2b3c4d5e6f zgrab_scanner
15 t12d050600_5a6b7c8d9e0f_1a2b3c4d5e6f zgrab_scanner
16 t12d030400_5a6b7c8d9e0f_0000deadbeef zmap_scanner
17 t13d010100_aabbccddeeff_0011223344aa Headless_Chrome_Automation
18 t13d010100_aabbccddeeff_ffeeddccbbaa Headless_Chrome_Automation
19 t13d1517h2_aabbccddeeff_0011223344aa Headless_Chrome_Automation
20 t13d030500_deadbeef1234_cafebabe5678 node_scanner
21 t13d020300_deadbeef1234_cafebabe5678 node_scanner
22 t13d1517h2_1234567890ab_abcdef012345 java_scanner
23 t12d1517h2_1234567890ab_abcdef012345 java_scanner
24 t13d020300_fedcba987654_0123456789ab ruby_scanner
25 t12d010100_aabbccddeeff_deadbeefdead nikto_scanner
26 t12d010100_ffeeddccbbaa_baddcafef00d sqlmap_scanner
27 t13d030600_deadbeefcafe_babe12345678 nuclei_scanner
28 t13d020200_abcdef012345_fedcba987654 scrapy_crawler
29 t13d020300_abcdef012345_1234abcd5678 scrapy_crawler
30 t10d010000_0000000000_000000000000 malware_c2_minimal
31 t12d010100_1111111111_222222222222 cobalt_strike_beacon

85
tests/integration/platform/csv-stubs/iplocate-ip-to-asn.csv
View File

@ -1,14 +1,97 @@
network,asn,country_code,name,org,domain
91.121.0.0/16,16276,FR,OVH SAS,OVH,ovh.com
151.80.0.0/16,16276,FR,OVH SAS,OVH,ovh.com
137.74.0.0/16,16276,FR,OVH SAS,OVH,ovh.com
5.196.0.0/16,16276,FR,OVH SAS,OVH,ovh.com
54.36.0.0/16,16276,FR,OVH SAS,OVH,ovh.com
78.41.0.0/16,15557,FR,SFR SA,SFR,sfr.com
90.28.0.0/14,15557,FR,SFR SA,SFR,sfr.com
109.0.0.0/14,15557,FR,SFR SA,SFR,sfr.com
90.0.0.0/8,3215,FR,Orange SA,Orange,orange.fr
86.192.0.0/11,3215,FR,Orange SA,Orange,orange.fr
81.48.0.0/14,3215,FR,Orange SA,Orange,orange.fr
82.64.0.0/14,12322,FR,Free SAS,Free,free.fr
78.220.0.0/14,12322,FR,Free SAS,Free,free.fr
88.120.0.0/13,12322,FR,Free SAS,Free,free.fr
212.0.0.0/8,5432,DE,Deutsche Telekom AG,Telekom,telekom.de
91.64.0.0/14,5432,DE,Deutsche Telekom AG,Telekom,telekom.de
2.200.0.0/14,5432,DE,Deutsche Telekom AG,Telekom,telekom.de
80.128.0.0/11,3320,DE,Deutsche Telekom DTAG,DTAG,telekom.de
176.0.0.0/12,6805,DE,Telefonica Germany,O2,o2online.de
84.116.0.0/16,1136,NL,KPN Internet BV,KPN,kpn.com
145.90.0.0/16,1136,NL,KPN Internet BV,KPN,kpn.com
145.0.0.0/16,1103,NL,SURF,SURFnet,surf.nl
77.108.0.0/16,2856,GB,BT Group plc,BT,bt.com
81.128.0.0/11,2856,GB,BT Group plc,BT,bt.com
86.128.0.0/11,2856,GB,BT Group plc,BT,bt.com
82.45.0.0/16,8913,GB,Virgin Media,Virgin Media,virginmedia.com
86.0.0.0/11,8913,GB,Virgin Media,Virgin Media,virginmedia.com
90.192.0.0/11,5607,GB,Sky UK Limited,Sky,sky.com
151.224.0.0/13,5607,GB,Sky UK Limited,Sky,sky.com
62.98.0.0/16,3352,ES,Telefonica Spain,Telefonica,telefonica.es
80.24.0.0/14,3352,ES,Telefonica Spain,Telefonica,telefonica.es
83.32.0.0/11,3352,ES,Telefonica Spain,Telefonica,telefonica.es
79.0.0.0/12,3269,IT,Telecom Italia,TIM,telecomitalia.it
82.48.0.0/12,3269,IT,Telecom Italia,TIM,telecomitalia.it
50.128.0.0/9,7922,US,Comcast Cable,Comcast,comcast.net
73.0.0.0/8,7922,US,Comcast Cable,Comcast,comcast.net
75.64.0.0/13,7922,US,Comcast Cable,Comcast,comcast.net
12.0.0.0/8,7018,US,AT&T Services,AT&T,att.com
32.0.0.0/11,7018,US,AT&T Services,AT&T,att.com
71.160.0.0/11,701,US,Verizon Business,Verizon,verizon.com
74.64.0.0/11,701,US,Verizon Business,Verizon,verizon.com
24.16.0.0/13,20115,US,Charter Communications,Spectrum,charter.com
65.32.0.0/11,20115,US,Charter Communications,Spectrum,charter.com
106.128.0.0/10,2516,JP,KDDI Corporation,KDDI,kddi.com
111.86.0.0/15,2516,JP,KDDI Corporation,KDDI,kddi.com
114.144.0.0/14,4713,JP,NTT Communications,OCN,ntt.com
118.238.0.0/15,4713,JP,NTT Communications,OCN,ntt.com
66.249.64.0/19,15169,US,Google LLC,Google,google.com
64.233.160.0/19,15169,US,Google LLC,Google,google.com
72.14.192.0/18,15169,US,Google LLC,Google,google.com
157.55.0.0/16,8075,US,Microsoft Corporation,Bing,microsoft.com
185.220.0.0/16,210644,NL,Accelerated-IT Services,Tor Project,tor-project.org
207.46.0.0/16,8075,US,Microsoft Corporation,Bing,microsoft.com
40.76.0.0/14,8075,US,Microsoft Corporation,Bing,microsoft.com
69.63.176.0/20,32934,US,Facebook Inc,Meta,facebook.com
66.220.144.0/20,32934,US,Facebook Inc,Meta,facebook.com
31.13.24.0/21,32934,US,Facebook Inc,Meta,facebook.com
199.59.148.0/22,13414,US,Twitter Inc,Twitter,twitter.com
199.16.156.0/22,13414,US,Twitter Inc,Twitter,twitter.com
185.220.100.0/22,210644,NL,Accelerated-IT Services,Tor Project,tor-project.org
185.220.101.0/24,210644,NL,Accelerated-IT Services,Tor Project,tor-project.org
185.220.102.0/24,210644,NL,Accelerated-IT Services,Tor Project,tor-project.org
45.155.205.0/24,209083,DE,Contabo GmbH,Contabo,contabo.de
62.171.128.0/17,209083,DE,Contabo GmbH,Contabo,contabo.de
5.161.0.0/16,209083,DE,Contabo GmbH,Contabo,contabo.de
64.225.0.0/16,14061,US,DigitalOcean LLC,DigitalOcean,digitalocean.com
104.131.0.0/16,14061,US,DigitalOcean LLC,DigitalOcean,digitalocean.com
138.197.0.0/16,14061,US,DigitalOcean LLC,DigitalOcean,digitalocean.com
159.65.0.0/16,14061,US,DigitalOcean LLC,DigitalOcean,digitalocean.com
3.0.0.0/8,16509,US,Amazon.com ARIN,AWS,amazonaws.com
18.0.0.0/8,16509,US,Amazon.com ARIN,AWS,amazonaws.com
52.0.0.0/8,16509,US,Amazon.com ARIN,AWS,amazonaws.com
54.0.0.0/8,16509,US,Amazon.com ARIN,AWS,amazonaws.com
34.0.0.0/8,396982,US,Google Cloud,GCP,cloud.google.com
35.184.0.0/13,396982,US,Google Cloud,GCP,cloud.google.com
74.208.0.0/16,8560,DE,IONOS SE,IONOS,ionos.com
212.227.0.0/16,8560,DE,IONOS SE,IONOS,ionos.com
136.243.0.0/16,24940,DE,Hetzner Online GmbH,Hetzner,hetzner.com
138.201.0.0/16,24940,DE,Hetzner Online GmbH,Hetzner,hetzner.com
144.76.0.0/16,24940,DE,Hetzner Online GmbH,Hetzner,hetzner.com
178.63.0.0/16,24940,DE,Hetzner Online GmbH,Hetzner,hetzner.com
45.32.0.0/16,20473,US,The Constant Company,Vultr,vultr.com
64.237.32.0/19,20473,US,The Constant Company,Vultr,vultr.com
108.61.0.0/16,20473,US,The Constant Company,Vultr,vultr.com
45.33.0.0/17,63949,US,Linode LLC,Linode,linode.com
45.56.0.0/16,63949,US,Linode LLC,Linode,linode.com
50.116.0.0/18,63949,US,Linode LLC,Linode,linode.com
104.16.0.0/12,13335,US,Cloudflare Inc,Cloudflare,cloudflare.com
172.64.0.0/13,13335,US,Cloudflare Inc,Cloudflare,cloudflare.com
162.158.0.0/15,13335,US,Cloudflare Inc,Cloudflare,cloudflare.com
193.32.162.0/24,197695,RU,Reg.ru Hosting,Reg.ru,reg.ru
194.58.92.0/22,197695,RU,Reg.ru Hosting,Reg.ru,reg.ru
78.46.0.0/15,51167,DE,Contabo GmbH,Contabo Hosting,contabo.de
162.241.0.0/16,46606,US,Unified Layer,Bluehost,bluehost.com
198.57.128.0/17,46606,US,Unified Layer,Bluehost,bluehost.com
184.168.0.0/16,26496,US,GoDaddy.com,GoDaddy,godaddy.com
198.71.128.0/17,26496,US,GoDaddy.com,GoDaddy,godaddy.com

1 network asn country_code name org domain
2 91.121.0.0/16 16276 FR OVH SAS OVH ovh.com
3 151.80.0.0/16 16276 FR OVH SAS OVH ovh.com
4 137.74.0.0/16 16276 FR OVH SAS OVH ovh.com
5 5.196.0.0/16 16276 FR OVH SAS OVH ovh.com
6 54.36.0.0/16 16276 FR OVH SAS OVH ovh.com
7 78.41.0.0/16 15557 FR SFR SA SFR sfr.com
8 90.28.0.0/14 15557 FR SFR SA SFR sfr.com
9 109.0.0.0/14 15557 FR SFR SA SFR sfr.com
10 90.0.0.0/8 3215 FR Orange SA Orange orange.fr
11 86.192.0.0/11 3215 FR Orange SA Orange orange.fr
12 81.48.0.0/14 3215 FR Orange SA Orange orange.fr
13 82.64.0.0/14 12322 FR Free SAS Free free.fr
14 78.220.0.0/14 12322 FR Free SAS Free free.fr
15 88.120.0.0/13 12322 FR Free SAS Free free.fr
16 212.0.0.0/8 5432 DE Deutsche Telekom AG Telekom telekom.de
17 91.64.0.0/14 5432 DE Deutsche Telekom AG Telekom telekom.de
18 2.200.0.0/14 5432 DE Deutsche Telekom AG Telekom telekom.de
19 80.128.0.0/11 3320 DE Deutsche Telekom DTAG DTAG telekom.de
20 176.0.0.0/12 6805 DE Telefonica Germany O2 o2online.de
21 84.116.0.0/16 1136 NL KPN Internet BV KPN kpn.com
22 145.90.0.0/16 1136 NL KPN Internet BV KPN kpn.com
23 145.0.0.0/16 1103 NL SURF SURFnet surf.nl
24 77.108.0.0/16 2856 GB BT Group plc BT bt.com
25 81.128.0.0/11 2856 GB BT Group plc BT bt.com
26 86.128.0.0/11 2856 GB BT Group plc BT bt.com
27 82.45.0.0/16 8913 GB Virgin Media Virgin Media virginmedia.com
28 86.0.0.0/11 8913 GB Virgin Media Virgin Media virginmedia.com
29 90.192.0.0/11 5607 GB Sky UK Limited Sky sky.com
30 151.224.0.0/13 5607 GB Sky UK Limited Sky sky.com
31 62.98.0.0/16 3352 ES Telefonica Spain Telefonica telefonica.es
32 80.24.0.0/14 3352 ES Telefonica Spain Telefonica telefonica.es
33 83.32.0.0/11 3352 ES Telefonica Spain Telefonica telefonica.es
34 79.0.0.0/12 3269 IT Telecom Italia TIM telecomitalia.it
35 82.48.0.0/12 3269 IT Telecom Italia TIM telecomitalia.it
36 50.128.0.0/9 7922 US Comcast Cable Comcast comcast.net
37 73.0.0.0/8 7922 US Comcast Cable Comcast comcast.net
38 75.64.0.0/13 7922 US Comcast Cable Comcast comcast.net
39 12.0.0.0/8 7018 US AT&T Services AT&T att.com
40 32.0.0.0/11 7018 US AT&T Services AT&T att.com
41 71.160.0.0/11 701 US Verizon Business Verizon verizon.com
42 74.64.0.0/11 701 US Verizon Business Verizon verizon.com
43 24.16.0.0/13 20115 US Charter Communications Spectrum charter.com
44 65.32.0.0/11 20115 US Charter Communications Spectrum charter.com
45 106.128.0.0/10 2516 JP KDDI Corporation KDDI kddi.com
46 111.86.0.0/15 2516 JP KDDI Corporation KDDI kddi.com
47 114.144.0.0/14 4713 JP NTT Communications OCN ntt.com
48 118.238.0.0/15 4713 JP NTT Communications OCN ntt.com
49 66.249.64.0/19 15169 US Google LLC Google google.com
50 64.233.160.0/19 15169 US Google LLC Google google.com
51 72.14.192.0/18 15169 US Google LLC Google google.com
52 157.55.0.0/16 8075 US Microsoft Corporation Bing microsoft.com
53 185.220.0.0/16 207.46.0.0/16 210644 8075 NL US Accelerated-IT Services Microsoft Corporation Tor Project Bing tor-project.org microsoft.com
54 40.76.0.0/14 8075 US Microsoft Corporation Bing microsoft.com
55 69.63.176.0/20 32934 US Facebook Inc Meta facebook.com
56 66.220.144.0/20 32934 US Facebook Inc Meta facebook.com
57 31.13.24.0/21 32934 US Facebook Inc Meta facebook.com
58 199.59.148.0/22 13414 US Twitter Inc Twitter twitter.com
59 199.16.156.0/22 13414 US Twitter Inc Twitter twitter.com
60 185.220.100.0/22 210644 NL Accelerated-IT Services Tor Project tor-project.org
61 185.220.101.0/24 210644 NL Accelerated-IT Services Tor Project tor-project.org
62 185.220.102.0/24 210644 NL Accelerated-IT Services Tor Project tor-project.org
63 45.155.205.0/24 209083 DE Contabo GmbH Contabo contabo.de
64 62.171.128.0/17 209083 DE Contabo GmbH Contabo contabo.de
65 5.161.0.0/16 209083 DE Contabo GmbH Contabo contabo.de
66 64.225.0.0/16 14061 US DigitalOcean LLC DigitalOcean digitalocean.com
67 104.131.0.0/16 14061 US DigitalOcean LLC DigitalOcean digitalocean.com
68 138.197.0.0/16 14061 US DigitalOcean LLC DigitalOcean digitalocean.com
69 159.65.0.0/16 14061 US DigitalOcean LLC DigitalOcean digitalocean.com
70 3.0.0.0/8 16509 US Amazon.com ARIN AWS amazonaws.com
71 18.0.0.0/8 16509 US Amazon.com ARIN AWS amazonaws.com
72 52.0.0.0/8 16509 US Amazon.com ARIN AWS amazonaws.com
73 54.0.0.0/8 16509 US Amazon.com ARIN AWS amazonaws.com
74 34.0.0.0/8 396982 US Google Cloud GCP cloud.google.com
75 35.184.0.0/13 396982 US Google Cloud GCP cloud.google.com
76 74.208.0.0/16 8560 DE IONOS SE IONOS ionos.com
77 212.227.0.0/16 8560 DE IONOS SE IONOS ionos.com
78 136.243.0.0/16 24940 DE Hetzner Online GmbH Hetzner hetzner.com
79 138.201.0.0/16 24940 DE Hetzner Online GmbH Hetzner hetzner.com
80 144.76.0.0/16 24940 DE Hetzner Online GmbH Hetzner hetzner.com
81 178.63.0.0/16 24940 DE Hetzner Online GmbH Hetzner hetzner.com
82 45.32.0.0/16 20473 US The Constant Company Vultr vultr.com
83 64.237.32.0/19 20473 US The Constant Company Vultr vultr.com
84 108.61.0.0/16 20473 US The Constant Company Vultr vultr.com
85 45.33.0.0/17 63949 US Linode LLC Linode linode.com
86 45.56.0.0/16 63949 US Linode LLC Linode linode.com
87 50.116.0.0/18 63949 US Linode LLC Linode linode.com
88 104.16.0.0/12 13335 US Cloudflare Inc Cloudflare cloudflare.com
89 172.64.0.0/13 13335 US Cloudflare Inc Cloudflare cloudflare.com
90 162.158.0.0/15 13335 US Cloudflare Inc Cloudflare cloudflare.com
91 193.32.162.0/24 197695 RU Reg.ru Hosting Reg.ru reg.ru
92 194.58.92.0/22 197695 RU Reg.ru Hosting Reg.ru reg.ru
93 78.46.0.0/15 51167 DE Contabo GmbH Contabo Hosting contabo.de
94 162.241.0.0/16 46606 US Unified Layer Bluehost bluehost.com
95 198.57.128.0/17 46606 US Unified Layer Bluehost bluehost.com
96 184.168.0.0/16 26496 US GoDaddy.com GoDaddy godaddy.com
97 198.71.128.0/17 26496 US GoDaddy.com GoDaddy godaddy.com