Incorporating Virustotal Data to Elasticsearch

Now that we’re collecting logs from various sources including Sysmon, we have access to file hash information. A while back I came across this SANS article on incorporating Virustotal to Elasticsarch here. The article is from 2015 when Elasticsearch was still in version 1.5, the creator was Jason Kendall and the github project is posted here. Unfortunately I have tried to get this working on Logstash version 5.x without success; Therefore, I stayed with version 2.3.4. My test environment was running Elasticsaerch version 2.3.4 as well as Logstash 2.3.4 in which the Virustotal plug-in did work!

Since my environment is on Elasticsearch version 5.x, I ended up creating a small VM with Logstash 2.3.4 and setup the virustotal plugin here, the data is still going to my current server so I’m happy with that.

Prerequisites:

  • Logstash 2.3.4 or earlier
  • Git  (yum install git)
  • Ruby Gem
  • Nano (my preferred text editor)
  • Sign up for a Virustotal public API https://www.virustotal.com

Step 1: Install Ruby On Rails

Say Yes to the promt

Install Ruby Gems & Update

Step 2: Install Virustotal Plug-in

Since this script was written for versions prior to 2.0, we’re going to edit the .gemspec file.

Edit the following, and change it to <3.0.0 or whatever version you choose.
s.add_runtime_dependency logstash-core, >= 1.4.0, < 2.0.0

Save it (CTRL-O) and exit.
let’s continue with the installation.

Note: you might get a warning message of “fatal: Not a git repository (or any of the parent directories): .git” which you may ignore.

You may now restart logstash.

Lastly, confirm that your Virustotal plugin is part of the logstash plugins.
under /opt/logstash run the following:

You should see logstash-filter-virustotal listed

Done!

Now let’s configure Logstash to lookup file hashes that are being generated from Sysmon.

In my use case I’m only looking focusing on EventID 15 in Sysmon
Event ID 15: FileCreateStreamHash

My Logstash configuration does the following:

  • Only looks at Event ID 15 and Hash field.
  • Focuses on the highlighted information from Virustotal. (Note: when the file queries Virustotal, the Anti-virus list will also be populated; therefore, my logstash configuration will automatically remove that information).

 

Here’s my logstash configuration for Virustotal. (Note: I”m using NXlog to send the logs from my Windows system to Logstash.

Update: 07-06-2017 – Logstash configuration to receive from Winlogbeat

 

Finally, here’s a sample log of Putty.exe without the unnecessary anti-virus vendor information.

Here’s a sample dashboard of Elasticsearch, which makes it easy to sort by high number of positives.

One last thing. Virustotal limits you to  the following if you have a public key:

Privileges public key
Request rate 4 requests/minute
Daily quota 5760 requests/day
Monthly quota 178560 requests/month
Status Key enabled

Therefore please ensure that you are not abusing the daily limit; otherwise, you might need to upgrade to a Private API.

Here’s a sample of my current API Consumption:

Thanks for reading.

 

Leave a Reply