code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(anubis): simplify to IP/CIDR + ASN only, remove UA and Country rules

Browse Source 
- Remove UA regex extraction (extract_ua_regex, _extract_ua_from_all/any)
- Remove Country rule collection from parse_bot_policies_inline
- Simplify fetch_rules.py: collect_all_rules returns (ip_rules, asn_rules)
- Remove insert_ua_rules and insert_country_rules functions
- reload_dicts now only reloads dict_anubis_ip + dict_anubis_asn
- Simplify CASE blocks in 04_mv_http_logs.sql, 07_ai_features_view.sql,
  view_ai_features_anubis.sql, mv_http_logs.sql: IP > ASN (was 5-level
  UA+IP > UA > IP > ASN > Country cascade)
- Remove dict_anubis_country + dict_anubis_ua from 03_anubis_tables.sql
  (UA table kept as stub for REGEXP_TREE catch-all compatibility)
- Remove anubis_country_rules table from schema
- Remove Anubis UA and Country tabs from dashboard reflists page
- Remove anubis_ua_rules/country_rules from API reflist queries
- deploy_schema.sql simplified from 339 to 122 lines
- 764 lines removed across 9 files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-09 15:25:33 +02:00
parent 98abbc80c7
commit 8180f4af04
9 changed files with 136 additions and 762 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
services/dashboard/backend/routes/api.py
View File

@ -1499,10 +1499,8 @@ _REFLIST_SORT = {
"browser_ja4": {"ja4", "browser_family", "tls_library"},
"asn_reputation": {"src_asn", "label"},
"iplocate_asn": {"asn", "country_code", "name", "network"},
"anubis_ua_rules": {"id", "regexp", "bot_name", "action", "category"},
"anubis_ip_rules": {"prefix", "bot_name", "action", "category"},
"anubis_asn_rules": {"asn", "bot_name", "action", "category"},
"anubis_country_rules": {"country_code", "bot_name", "action", "category"},
}
_REFLIST_SEARCH_COLS: dict[str, list[str]] = {
@ -1511,10 +1509,8 @@ _REFLIST_SEARCH_COLS: dict[str, list[str]] = {
"browser_ja4": ["ja4", "browser_family", "tls_library", "context"],
"asn_reputation": ["toString(src_asn)", "label"],
"iplocate_asn": ["network", "toString(asn)", "country_code", "name"],
"anubis_ua_rules": ["regexp", "bot_name", "action", "category"],
"anubis_ip_rules": ["prefix", "bot_name", "action", "category"],
"anubis_asn_rules": ["toString(asn)", "bot_name", "action", "category"],
"anubis_country_rules": ["country_code", "bot_name", "action", "category"],
}
@ -1532,22 +1528,12 @@ _REFLIST_QUERIES: dict[str, str] = {
f"SELECT network, asn, country_code, name "
f"FROM dictionary('{_DB}.dict_iplocate_asn')"
),
"anubis_ua_rules": (
f"SELECT id, parent_id, regexp, "
f" arrayElement(values, indexOf(keys, 'bot_name')) AS bot_name, "
f" arrayElement(values, indexOf(keys, 'action')) AS action, "
f" arrayElement(values, indexOf(keys, 'category')) AS category "
f"FROM {_DB}.anubis_ua_rules"
),
"anubis_ip_rules": (
f"SELECT prefix, bot_name, action, category FROM {_DB}.anubis_ip_rules"
),
"anubis_asn_rules": (
f"SELECT asn, bot_name, action, category FROM {_DB}.anubis_asn_rules"
),
"anubis_country_rules": (
f"SELECT country_code, bot_name, action, category FROM {_DB}.anubis_country_rules"
),
}