Suricata + Lua实现本地情报对接

摘要

背景

由于近期网站遭受恶意攻击, 通过对于登录接口的审计与分析, 现已确定了一批可疑账号。既然之前写过一个登录接口的审计脚本, 那么完全可以通过扩展这个脚本来实现对于可疑账号的比对。主要思路: 通过将可疑账存进Redis中, 再利用Lua脚本调用Redis接口进行账号的比对。

先说一下Suricata默认是存在黑名单机制的, 如下:

# IP Reputation #reputation-categories-file: /etc/suricata/iprep/categories.txt #default-reputation-path: /etc/suricata/iprep #reputation-files: # - reputation.list

在Suricata 5.0版本中更是增加了新的功能Datasets。大概看了一下, 可以通过在规则中使用dataset和datarep关键字将大量数据与sticky buffer进行匹配。确实是个很赞的功能!

alert http any any -> any any (http.user_agent; dataset:set, ua-seen, type string, save ua-seen.lst; sid:1;) alert dns any any -> any any (dns.query; to_sha256; dataset:set, dns-sha256-seen, type sha256, save dns-sha256-seen.lst; sid:2;) alert http any any -> any any (http.uri; to_md5; dataset:isset, http-uri-md5-seen, type md5, load http-uri-md5-seen.lst; sid:3;)

但是… 这并不适用我现在的场景。因为在我的场景中, 用户的登录请求存在于POST Body中, 默认的Suricata方法并不能准确定位到我们需要的账号。这个时候我们就只能依赖于Lua脚本来扩展。当然这些需求Zeek也可以满足, 只是…Zeek的脚本真是难写…主要是我技术太low~

准备阶段

运行环境

OS:Ubuntu 18.04

Suricata: Suricata 5.0.0 RELEASE (我是AWS的流量镜像, 必须使用4.1.5或者5.0版本, 因为要解析VXLAN)

LuaRocks

1.由于Ubuntu默认没有安装LuaRocks(LuaRocks is the package manager for Lua modules), 这里需要我们手动安装。

# 通过apt直接安装, 简单省事儿。 $ apt-get install luarocks 

2. 通过luarocks安装我们所需要的lua模块, 这里我们需要用到redis-lualuasocket这两个模块。

# Install Modules
$ luarocks install luasocket
$ luarocks install redis-lua

$ ll /usr/local/share/lua/5.1/
total 72 drwxr-xr-x 3 root root  4096 Oct 25 03:35 ./
drwxr-xr-x 3 root root  4096 Sep 17 14:14 ../
-rw-r--r-- 1 root root  8331 Oct 25 03:34 ltn12.lua -rw-r--r-- 1 root root  2487 Oct 25 03:34 mime.lua -rw-r--r-- 1 root root 35599 Oct 25 03:35 redis.lua drwxr-xr-x 2 root root  4096 Oct 25 03:34 socket/
-rw-r--r-- 1 root root  4451 Oct 25 03:34 socket.lua 

3. 安装成功后, 可以简单的测试一下。

3.1 利用Docker启动Redis容器

$ docker run -ti -d -p 6379:6379 redis 

3.2 测试脚本 hello_redis.lua

local redis = require "redis" local client = redis.connect("127.0.0.1", 6379) local response = client:ping() if response == false then return 0
end

client:set("hello", "world") local var = client:get("hello") print(var) 

3.3 可能会存在环境变量不对导致的报错

$ luajit hello_redis.lua
	luajit: /usr/local/share/lua/5.1/redis.lua:793: module 'socket' not found: no field package.preload['socket'] no file './socket.lua' no file '/usr/local/share/luajit-2.0.5/socket.lua' no file '/usr/local/share/lua/5.1/socket.lua' no file '/usr/local/share/lua/5.1/socket/init.lua' no file './socket.so' no file '/usr/local/lib/lua/5.1/socket.so' no file '/usr/local/lib/lua/5.1/loadall.so' stack traceback:
	[C]: in function 'require' /usr/local/share/lua/5.1/redis.lua:793: in function 'create_connection' /usr/local/share/lua/5.1/redis.lua:836: in function 'connect' a.lua:3: in main chunk
	[C]: at 0x56508049e440 

3.4 执行luarocks path –bin 并将结果输入

$ luarocks path --bin Warning: The directory '/home/canon/.cache/luarocks' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing /usr/local/bin/luarocks with sudo, you may want sudo's -H flag.
export LUA_PATH='/home/canon/.luarocks/share/lua/5.1/?.lua;/home/canon/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/share/luajit-2.0.5/?.lua'
export LUA_CPATH='/home/canon/.luarocks/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/loadall.so'
export PATH='/home/canon/.luarocks/bin:/usr/local/bin:/home/canon/anaconda3/bin:/home/canon/anaconda3/condabin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin' 

3.5 执行脚本, 将会看到如下输出

$ luajit hello_redis.lua
world 

CJson

这里建议大家使用CJson模块, 我之前为了测试随便从github上找了个json模块来使用。这几天发现在网站的高峰时期 Suricataapp_layer.flow这个字段非常的大, 从而导致了kernel_drops。由于我们的网站是面对海外用户想定位问题又存在时差, 经过几天的熬夜最终定位到是由于json模块太过于消耗性能而导致。可以看下这个截图:

a.Suricata监控图 -启用CJson模块之前

b.Suricata监控图 -启用CJson模块之后

1.下载 CJson

# wget 下载 $ wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz # Git Clone $ git clone git@github.com:mpx/lua-cjson.git 

2. 根据Lua环境修改Makefile(个人配置)

##### Build defaults ##### LUA_VERSION = 5.1 TARGET =            cjson.so PREFIX =            /usr/local #CFLAGS =            -g -Wall -pedantic -fno-inline CFLAGS =            -O3 -Wall -pedantic -DNDEBUG CJSON_CFLAGS =      -fpic CJSON_LDFLAGS =     -shared LUA_INCLUDE_DIR =   $(PREFIX)/include/luajit-2.0 LUA_CMODULE_DIR =   $(PREFIX)/lib/lua/$(LUA_VERSION) LUA_MODULE_DIR =    $(PREFIX)/share/lua/$(LUA_VERSION) LUA_BIN_DIR =       $(PREFIX)/bin 

3. 安装 CJson

$ make && make install

登录接口代码示例

json = require "cjson.safe" md5 = require "md5" redis = require "redis" -- 登录接口
login_url = "/login" -- 根据实际接口而定
-- 登录错误提示
success_code = 0 -- event_name
event_name = "login_audit" -- event_type
event_type = "lua" -- logs
name = "login_audit.json" -- 协议
proto = "TCP" -- redis_config
host = "127.0.0.1" port = 6379 -- common_mapping 通用请求头
http_common_mapping = '{"accept":"accept","accept-charset":"accept_charset","accept-encoding":"accept_encoding","accept-language":"accept_language","user-agent":"user_agent"}' common_mapping_table = json.decode(http_common_mapping)

-- request_mapping 自定义请求头
http_request_mapping = '{"content-length":"request_content_length","content-type":"request_content_type"}' request_mapping_table = json.decode(http_request_mapping)

-- response_mapping 自定义响应头
http_response_mapping = '{"content-length":"response_content_length","content-type":"response_content_type"}')


-- custom defind functioin
function md5Encode(args)
    m = md5.new() m:update(args) return md5.tohex(m:finish()) end function formatBody(args)
    t = {}
    ios = string.match(args, 'canon') if ios ~= nil then mail = 'email"%s+(.-)%s' t['email'] = string.match(args, mail) else data = string.split(args, '&') for n, v in ipairs(data) do d = string.split(v, '=')
            t[d[1]] = d[2] end end return t end function string.split(s, p)
    rt = {}
    string.gsub(s, '[^'..p..']+', function(w) table.insert(rt, w) end ) return rt end -- default function
function init (args)
    local needs = {}
    needs["protocol"] = "http" return needs end function setup (args)
    filename = SCLogPath() .. "/" .. name
    file = assert(io.open(filename, "a"))
    SCLogInfo("app_login_audit filename: " .. filename)
    http = 0 -- Connect Redis Server 连接Redis服务器
    SCLogInfo("Connect Redis Server...")
    client = redis.connect(host, port)
    response = client:ping() if response then SCLogInfo("Redis Server connection succeeded.") end end function log(args)
    -- init tables
    http_table = {}

    -- ti tables
    ti = {
        tags = {}
    }

    -- init score 初始分数(为后期规则判断而准备, 符合规则进行加分。)
    score = 50 -- http_hostname & http_url
    http_hostname = HttpGetRequestHost()
    http_url = HttpGetRequestUriNormalized()
    
    -- http_method
    rl = HttpGetRequestLine() if rl then http_method = string.match(rl, "%w+") if http_method then http_table["method"] = http_method end end -- 为了保证 Suricata 的性能不受影响, 指定登录接口以及请求才能进入此逻辑。 if http_url == login_url and http_method == "POST" then http_table["hostname"] = http_hostname
        http_table["url"] = http_url
        http_table["url_path"] = http_url
        
        -- http_status & http_protocol
        rsl = HttpGetResponseLine() if rsl then status_code = string.match(rsl, "%s(%d+)%s")
            http_table["status"] = tonumber(status_code)

            http_protocol = string.match(rsl, "(.-)%s")
            http_table["protocol"] = http_protocol end -- login_results
        a, o, e = HttpGetResponseBody() if a then for n, v in ipairs(a) do body = json.decode(v)
                results_code = tonumber(body["code"]) if results_code == success_code then http_table["results"] = "success" else http_table["results"] = "failed" end end http_table["results_code"] = results_code end --[[ 1. 获取用户登录email并查询Redis中是否存在该账号 2. 根据结果进行相应的打分以及tags标注
        --]]
        a, o, e = HttpGetRequestBody() if a then for n, v in ipairs(a) do res = formatStr(v) if res["email"] then -- 查询Redis对比黑名单
                    black_ioc = client:get(res["email"]) if black_ioc then ti["provider"] = "Canon" ti["producer"] = "NTA" table.insert(ti["tags"], "account in blacklist")
                        score = score + 10 end end end end -- RequestHeaders 根据自定义的请求头进行获取, 对于业务安全来说有些请求头还是有必要获取的。
        rh = HttpGetRequestHeaders() if rh then for k, v in pairs(rh) do key = string.lower(k)
                common_var = common_mapping_table[key]
                if common_var then                     http_table[common_var] = v
                end     
                request_var = request_mapping_table[key]
                if request_var then                     http_table[request_var] = v
                end end end -- ResponseHeaders 自定义获取响应头
        rsh = HttpGetResponseHeaders() if rsh then for k, v in pairs(rsh) do key = string.lower(k)
                common_var = common_mapping_table[key]
                if common_var then                     http_table[common_var] = v
                end         
                response_var = response_mapping_table[key]
                if response_var then                     http_table[response_var] = v
                end end end -- timestring
        sec, usec = SCPacketTimestamp()
        timestring = os.date("!%Y-%m-%dT%T", sec) .. '.' .. usec .. '+0000' -- flow_info
        ip_version, src_ip, dst_ip, protocol, src_port, dst_port = SCFlowTuple()

        -- flow_id
        id = SCFlowId()
        flow_id = string.format("%.0f", id)
        flow_id = tonumber(flow_id)

        -- alerts 查询这笔flow是否存在特征匹配后的告警
        has_alerts = SCFlowHasAlerts()

        -- true_ip
        true_client_ip = HttpGetRequestHeader("True-Client-IP") if true_client_ip ~= nil then src_ip = true_client_ip end -- session_id
        tetrad = src_ip .. src_port .. dst_ip .. dst_port
        session_id = md5Encode(tetrad)

        -- table
        raw_data = {
            timestamp = timestring,
            flow_id = flow_id,
            session_id = session_id,
            src_ip = src_ip,
            src_port = src_port,
            proto = proto,
            dest_ip = dst_ip,
            dest_port = dst_port,
            event_name = event_name,
            event_type = event_type,
            app_type = app_type,
            http = http_table,
            alerted = has_alerts,
            ti = ti,
            score = score
        }

        -- json encode
        data = json.encode(raw_data) file:write(data .. "\n") file:flush()

        http = http + 1 end end function deinit (args)
    SCLogInfo ("app_login_audit transactions logged: " .. http); file:close(file) end

1. 简单说下以上脚本的功能:

a.登录接口的用户名审计(废话…);

b.通过请求Redis比对当前用户是否在黑名单中, 并进行相应的打分、标签处理;

c.根据自定义的需求获取的http headers, 个人觉得这个对于业务安全上还是有点用的;

d. 新增字段”session_id”, 主要考虑是针对CDN或者Nginx这种方向代理的场景下, 可以直接对 xff 或者 true_client_ip 进行四元组的hash, 得到session_id, 这样溯源的时候会比较方便。因为在这种场景下传统的四层flow_id就不是那么有用了。

e.后续可以追加一些简单的检测方法, 例如: (这些适用于我们, 其他的请头脑风暴)

检查请求头中的字段是否完整;

检查请求头中的某个字段长度是否符合合规;

头脑风暴…

2. 配置Suricata启用Lua脚本

- lua:  enabled: yes
    scripts-dir: /etc/suricata/lua-output/  scripts: - login_audit.lua

3. 启动Suricata

$ suricata -vvv --pfring -k none -c /etc/suricata/suricata.yaml 

注: 这里-vvv 参数建议加上. 如果你的Lua脚本有一些问题, 如果加上了这个参数, 就可以通过这个日志看出。

$ tailf /data/logs/suricata/suricata.log 4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355. 4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355. 4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355. 4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355. 4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355. 4/11/2019 -- 02:28:03 - <Info> - failed to run script: /usr/local/share/luajit-2.0.5/md5.lua:347: attempt to get length of local 's' (a nil value) 

输出日志样例

{ "src_port": 62722, "score": 60, "session_id": "c863aeb2ef8d1b37f3257f8c210bf440", "ti": { "tags": [ "account in blacklist" ], "provider": "Canon", "producer": "NTA" }, "alert": { "alerted": true, "rules": { "请求头校验": "dev-id" }
    }, "proto": "TCP", "flow_id": "1064295903559076", "timestamp": "2019-10-25T08:33:55.585519+0000", "event_type": "lua", "src_ip": "1.1.1.1", "dest_port": 80, "http": { "response_content_length": "96", "response_content_type": "application/json; charset=UTF-8", "accept_encoding": "gzip", "accept": "application/json", "results_code": 400504, "server": "nginx", "date": "Fri, 25 Oct 2019 08:33:55 GMT", "app_version": "6.6.0", "request_content_type": "application/x-www-form-urlencoded", "user_agent": "okhttp/3.12.0", "url": "/login", "email": "canon@gmail.com", "results": "failed", "pragma": "no-cache",- "cache_control": "no-cache, max-age=0, no-store", "connection": "keep-alive", "status": 200, "protocol": "HTTP/1.1", "hostname": "x.x.x.x", "url_path": "/login", "method": "POST", "device": "RMX1920 Android8.0.0", "device_type": "Android", "request_content_length": "39" }, "event_name": "login_audit", "dest_ip": "2.2.2.2" } 

目前评论:0 条

发表评论