Home » Blogs » Ubiquiti Dream Machine Pro Logs to Microsoft Sentinel

Ubiquiti Dream Machine Pro Logs to Microsoft Sentinel

Many IT geeks may have at home an Ubiquiti Dream Machine Pro to manage and protect their network. It’s a perfect device to build test networks and having some cool options like having an Azure Site to Site VPN tunnel. But can I also leverage the logs in Microsoft Sentinel?

There is an available Solution in the Content Hub with a few Analytic rules, a Data Connector and even a Parser.

After installing the Solution I checked the Connector and it is by default leveraging the Linux or Windows Log Analytics Agent. I don’t like this approach as I would like to leverage the Logstash engine where I have options to filter and transform data. Deploying Logstash is simple by following the documentation of Microsoft. But can we still leverage the Microsoft provided solution? in the next steps I will highlight the important steps to successfully implement this configuration.

Non-Linux Admins like me

If you are working the first time or not often with Linux, you can setup access to the VM including copy/paste via RDP by installing xrdp. This can be achieved by the following script:

sudo apt update
sudo apt install kde-plasma-desktop

sudo apt install xrdp
sudo systemctl start xrdp
sudo systemctl enable xrdp
sudo ufw allow 3389/tcp

Install Logstash

Microsoft Sentinel’s new Logstash output plugin supports pipeline transformations and advanced configuration via Data Collection Rules (DCRs). The plugin can forward any type of logs into custom or standard tables in Log Analytics or Microsoft Sentinel. To simplify deployment, I created a small script to install and directly create a sample file which can be used to configure the data collection rules.

Change It is required to change administrator on line 11 to your username.

sudo apt update
sudo apt install apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt install logstash
sudo /usr/share/logstash/bin/logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-plugin


sudo mkdir /tmp/logstash
sudo touch /etc/logstash/conf.d/unify.conf
sudo chown administrator /etc/logstash/conf.d/unify.conf
sudo chmod u+w /etc/logstash/conf.d/unify.conf
echo "input {
     syslog {
         port => 514
    }
}

output {
    microsoft-sentinel-log-analytics-logstash-output-plugin {
      create_sample_file=> true
      sample_file_path => \"/tmp/logstash\"
    }
}
}" > /etc/logstash/conf.d/unify.conf

sudo /usr/share/logstash/bin/logstash -r -f /etc/logstash/conf.d/unify.conf

Logstash is now waiting on syslog events. To get the first one in the sample file you have to configure your Ubiquiti Dream Machine accordingly. First, we need to enable the syslog logging, set the correct Host and Port in the System Advanced settings page.

Then we need to ensure, that firewall actions are logged. First, we can enable the Default Action logging which will enable logging of all default rules which can’t be edited:

Second, and only if you have firewall rules defined, then you can edit them and enable Logging on a per rule basis:

After a few seconds the sample logfile should be generated:

Setting up Data Collection Rule Resources

The next steps are well documented on Microsoft Learn. Basically, you have to create an Application Registration in Microsoft Entra, Create data collection endpoint and then creating a new table in Log Analytics Workspace. Here it is important to provide the table name “Ubiquiti”. If you choose another name, you should just remember to change the table name in the parser/function provided by the Content Hub solution.

Then you need to upload the sample file on the next page which will create an error as there is no TimeGenerated field available.

Specifying the following simple transformation will correct the issue:

source 
| extend TimeGenerated=ls_timestamp

After finishing the wizard, we have created a data collection endpoint, a data collection rule and an application registration. Now, we need to grant permission to the application to be able to submit logs.

As a last task before we get data in Microsoft Sentinel, we have to adjust the Logstash Configuration file to submit data to Microsoft Sentinel instead of the sample file. For this we can leverage again a script, which then also starts Logstash as a service.

echo "input {
     syslog {
         port => 514
    }
}

output {
    microsoft-sentinel-log-analytics-logstash-output-plugin {
      client_app_Id => \"<enter your client_app_id value here>\"
      client_app_secret => \"<enter your client_app_secret value here>\"
      tenant_id => \"<enter your tenant id here> \"
      data_collection_endpoint => \"<enter your DCE logsIngestion URI here>\"
      dcr_immutable_id => \"<enter your DCR immutableId here>\"
      dcr_stream_name => \"<enter your stream name here>\"
      create_sample_file=> false
      sample_file_path => \"/tmp/logstash\"
      proxy => \"http://proxy.example.com\"
    }
}
}" > /etc/logstash/conf.d/unify.conf


sudo systemctl daemon-reload
sudo systemctl enable logstash.service
sudo systemctl start logstash.service
sudo systemctl status logstash.service # if you can see "active", it's installed successfully.

Important For security reasons, I recommend to not write the client_app_Id, client_app_secret, tenant_id, data_collection_endpoint, and dcr_immutable_id attributes in the Logstash configuration file. A better option is to store this sensitive information in a Logstash KeyStore.

First result

After a short while you should see first entries populated within Microsoft Sentinel:

Please keep in mind, that if you are working in a test environment with limited budget, such raw logs will generate without much workload already at least 1GB data ingest per day. Filtering and data transformation in Logstash or in the Data Collection Rule can reduce this amount. For example, modifying or removing a single property can have a huge impact. As an example, the logsource is by default DreamMachinePro which can be 15 bytes. Changing it to DM reduces it to 2 bytes. The best thing is to set a daily cap and to monitor on the first days the costs and data ingest closely.

Parsing and Tuning

All the provided Analytic rules are leveraging the UbiquitiAuditEvent function which parses the logs.

To check if everything is working, I execute the specific functions and can detect, that just a subset of row’s is returned. After having a closer look, I could detect, that this solution probably works for Ubiquiti USG Pro, but not for the Dream Machine Pro. The schema is sadly really different especially for the firewall messages:

Changed format
Action like Reject, Deny and Accept formatted differently.
Column Message renamed to lowercase.

To change this, you can open the Workspace function:

Then you can replace the content with:

let EventData = Ubiquiti_CL
| extend EventVendor = 'Ubiquiti'
| extend EventTime = extract(@'\<\d+\>(\w+\s+\w+\s+\d+:\d+:\d+)\s\w+,', 1, message)
| extend DvcType = iif(extract(@'\d+\:\d+\:\d+\s(\w+),[A-Fa-f0-9]{12}', 1, message)!="", extract(@'\d+\:\d+\:\d+\s(\w+),[A-Fa-f0-9]{12}', 1, message), extract(@'\d+\:\d+\:\d+\s[A-Fa-f0-9]{12},([A-Za-z-]+)-', 1, message))
| extend DvcMacAddr = replace(@'(:)$', @'', replace(@'(\w{2})', @'\1:', extract(@'([A-Fa-f0-9]{12}),' , 1, message)))
| extend FirmwareVersion = iif(extract(@'[A-Fa-f0-9]{12},v(.*?)\:', 1, message)!="", extract(@'[A-Fa-f0-9]{12},v(.*?)\:', 1, message), extract(@'[A-Fa-f0-9]{12},[A-Za-z-]+([\d\.\+]+)[\:\s]', 1, message))
| extend DvcIpAddr = host;
let ubiquiti_dropbear_events =() {
EventData
| where message contains 'dropbear'
| extend EventCategory = 'dropbear'
| extend Eventmessage = extract(@' dropbear\[\d+\]\:\s(.*)', 1, message)
| extend SrcIpAddr = extract(@'from (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\:\d{1,5}', 1, message)
| extend SrcPortNumber = extract(@'from \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:(\d{1,5})', 1, message)
};
let ubiquiti_hostapd_events =() {
EventData
| where message contains 'hostapd'
| extend EventCategory = 'hostapd'
| extend WlanId = extract(@'hostapd:\s(\w+)', 1, message)
| extend SrcType = extract(@':\s(\w+)\s[A-Fa-f0-9:]{17}', 1, message)
| extend SrcMacAddr = extract(@':\s(\w+)\s([A-Fa-f0-9:]{17})', 2, message)
| extend DstMacAddr = extract(@'addr=([a-fA-F0-9:]{17})', 1, message)
| extend Service = extract(@'[A-Fa-f0-9:]{17}\s(.+):', 2, message)
| extend Eventmessage = extract(@'[A-Fa-f0-9:]{17}\s(.*):\s(.*)', 2, message)
};
let ubiquiti_firewall_events =() {
EventData
| where message matches regex @'\[.+\]\sDESCR="+[A-z\s]+"\sIN='
| extend EventCategory = 'firewall'
| extend FlowId = extract(@'ID=(.*?)\s', 1, message)
| extend DvcInboundInterface = extract(@'IN=(.*?)\s', 1, message)
| extend DvcOutboundInterface = extract(@'OUT=(.*?)\s', 1, message)
| extend dvc_action = extract(@'\[.+(-(RET|D|R)-).+\]', 2, message)
| extend DvcAction = case(dvc_action == "RET", "Accepted",
                          dvc_action == "D", "Denied",
                          dvc_action == "R", "Rejected",
                          "Other")
| extend NetworkRuleName = extract(@'kernel:\s+\[(\S+)-\w\]', 1, message)
| extend DstMacAddr = extract(@'MAC=([a-fA-F0-9:]{17}):', 1, message)
| extend SrcMacAddr = extract(@'MAC=[a-fA-F0-9:]{17}:([a-fA-F0-9:]{17})\s', 1, message)
| extend SrcIpAddr = extract(@'SRC=(.*?)\s', 1, message)
| extend SrcPortNumber = extract(@'SPT=(.*?)\s', 1, message)
| extend DstIpAddr = extract(@'DST=(.*?)\s', 1, message)
| extend DstPortNumber = extract(@'DPT=(.*?)\s', 1, message)
| extend NetworkBytes = extract(@'LEN=(.*?)\s', 1, message)
| extend Tos = extract(@'TOS=(.*?)\s', 1, message)
| extend Prec = extract(@'PREC=(.*?)\s', 1, message)
| extend Ttl = extract(@'TTL=(.*?)\s', 1, message)
| extend NetworkProtocol = extract(@'PROTO=(.*?)\s', 1, message)
| extend Window = extract(@'WINDOW=(.*?)\s', 1, message)
| extend Res = extract(@'RES=(.*?)\s', 1, message)
| extend Mark = extract(@'MARK=(.*?)\s', 1, message)
};
let ubiquiti_dns_timeout_events =() {
EventData
| where message contains "DNS request timed out"
| extend EventCategory = 'dnstimeout'
| extend Eventmessage = 'DNS request timed out'
| extend SrcType = extract(@'\[(\w+):\s[a-fA-F0-9:]{17}\]', 1, message)
| extend DvcMacAddr = extract(@'\[\w+:\s([a-fA-F0-9:]{17})\]', 1, message)
| extend DnsQuery = extract(@'QUERY:(.*?)\]', 1, message)
| extend DnsServer = extract(@'DNS_SERVER\s?:(.*?)\]', 1, message)
};
let ubiquiti_stahtd_events =() {
EventData
| where message contains 'stahtd'
| extend EventCategory = extract(@'\"message_type\":\"(.*?)\"', 1, message)
| extend SrcDvcMacAddr = extract(@'\"mac\":\"(.*?)\"', 1, message)
| extend WlanId = extract(@'\"vap\":\"(.*?)\"', 1, message)
| extend AssocStatus = extract(@'\"assoc_status\":\"(.*?)\"', 1, message)
| extend EventResult = extract(@'\"event_type\":\"(.*?)\"', 1, message)
| extend Eventmessage = extract(@'\}\s-\s(.*)', 1, message)
};
let ubiquiti_EVT_AP_STA_ASSOC_TRACKER_DBG =() {
EventData
//| where message contains 'libubnt'
| where message contains 'EVT_AP_STA_ASSOC_TRACKER_DBG'
| extend EventCategory = 'libubnt'
| extend WlanId = extract(@'vap:\s(.*?)', 1, message)
| extend SrcMacAddr = extract(@'sta_mac:\s(.*?)', 1, message)
| extend EventResult = extract(@'event_type:\s(.*)', 1, message)
| extend Eventmessage = 'Client failed to associate with an AP'
};
let ubiquiti_EVENT_STA_ =() {
EventData
//| where message contains 'libubnt'
| where message contains 'EVENT_STA_'
| extend EventCategory = 'libubnt'
| extend WlanId = extract(@'EVENT_STA_(JOIN|LEAVE|IP)\s(\w+):', 2, message)
| extend DvcAction = extract(@'EVENT_STA_(JOIN|LEAVE|IP)', 1, message)
| extend Eventmessage = case(DvcAction == 'JOIN', 'Client joined AP',
                             DvcAction == 'LEAVE', 'Client disconnected from AP',
                             'Client IP info')
| extend SrcMacAddr = extract(@':\s([A-Fa-f0-9:]{17})', 1, message)
| extend SrcIpAddr = extract(@'\/\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', 1, message)
};
let ubiquiti_syswrapper_events =() {
EventData
| where message contains 'syswrapper'
| extend EventCategory = 'syswrapper'
| extend Eventmessage = extract(@'syswrapper:\s(.*)', 1, message)
};
let ubiquiti_logread_events =() {
EventData
| where message contains 'logread'
| extend EventCategory = 'logread'
| extend DstIpAddr = extract(@'to\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', 1, message)
| extend DstPortNumber = extract(@'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:(\d{1,5})', 1, message)
| extend Eventmessage = extract(@'logread\[\d+\]:\s(.*)', 1, message)
};
let ubiquiti_stamgr_events =() {
EventData
| where message contains'stamgr'
| extend EventCategory = 'stamgr'
| extend DstMacAddr = extract(@'\s([A-Fa-f0-9:]{17})', 1, message)
| extend WlanId = extract(@'\s[A-Fa-f0-9:]{17}\s(\S+)', 1, message)
| extend Eventmessage = extract(@'stamgr:(.*?)\(', 1, message)
| extend EventResultDetails = extract(@'reason:(.*?)\)', 1, message)
};
let ubiquiti_kernel_events =() {
EventData
| where message contains 'kernel'
| where message contains 'FWLOG' or message contains 'set_ratelimit'
| extend EventCategory = 'kernel'
| extend Eventmessage = case(message matches regex "kernel.*FWLOG", extract(@'FWLOG:\s\[\d+\]\s(.*)', 1, message),
                             message matches regex "kernel.*_set_ratelimit", extract(@'_set_ratelimit:\s(.*)', 1, message),
                             "Check raw_message for details")
};
let ubiquiti_dns_events =() {
EventData
| where message matches regex @'dnsmasq\[\d+\]:'
| extend DstMacAddr = extract(@'MAC=([a-fA-F0-9:]{17}):', 1, message)
| extend SrcMacAddr = extract(@'MAC=[a-fA-F0-9:]{17}:([a-fA-F0-9:]{17})\s', 1, message) 
| extend DnsQuery = extract(@'dnsmasq\[\d+\]:\s(.*?)\[\w+\]|\s(\S+)\sfrom\s\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', 2, message)
| extend SrcIpAddr = extract(@'dnsmasq\[\d+\]:\s(.*?)\[\w+\]|\s(.*?)from\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', 3, message)
};
union isfuzzy=true ubiquiti_dropbear_events, ubiquiti_hostapd_events, ubiquiti_firewall_events, ubiquiti_dns_timeout_events, ubiquiti_stahtd_events, ubiquiti_EVT_AP_STA_ASSOC_TRACKER_DBG, ubiquiti_EVENT_STA_, ubiquiti_syswrapper_events, ubiquiti_logread_events, ubiquiti_stamgr_events, ubiquiti_kernel_events, ubiquiti_dns_events
| project TimeGenerated
        , EventVendor
        , EventTime
        , EventCategory
        , DvcIpAddr
        , DvcType
        , DvcMacAddr
        , FirmwareVersion
        , Eventmessage
        , WlanId
        , SrcType
        , Service
        , FlowId
        , DvcInboundInterface
        , DvcOutboundInterface
        , DvcAction
        , NetworkRuleName
        , SrcMacAddr
        , SrcIpAddr
        , SrcPortNumber
        , DstMacAddr
        , DstIpAddr
        , DstPortNumber
        , NetworkBytes
        , Tos
        , Prec
        , Ttl
        , NetworkProtocol
        , Window
        , Res
        , Mark
        , DnsQuery
        , DnsServer
        , SrcDvcMacAddr
        , AssocStatus
        , EventResult
        , EventResultDetails
        , message
UbiquitiAuditEvent

Result

If you did everything correctly you should be able to see entries which are correctly parsed. In a next blog I will then show how to create an ASIM parser for these logs as well.

About
Latest Posts

Follow me

Thomas Kurth

Principal Security Consultant | MVP at baseVISION AG

I’m a consultant, trainer and architect for modern workplace and enterprise mobility projects with Microsoft Technologies in the past ten years. I love to push and design the modern workplace based on Microsoft 365 for my customers which is the only answer for the current security threats, agile world and the fast-changing business requirements of my customers. Important for me is to simplify and automate the operational processes, because there are the highest costs.

Follow me

Ubiquiti Dream Machine Pro Logs to Microsoft Sentinel

Non-Linux Admins like me

Install Logstash

Setting up Data Collection Rule Resources

First result

Parsing and Tuning

Result

0 Comments

Leave a Reply Cancel reply

Recent Posts

Hosting sponsored by:

Subscribe

Ubiquiti Dream Machine Pro Logs to Microsoft Sentinel

Non-Linux Admins like me

Install Logstash

Setting up Data Collection Rule Resources

First result

Parsing and Tuning

Result

0 Comments

Leave a Reply Cancel reply

Recent Posts

Hosting sponsored by:

Subscribe

Tags