Threat Intelligence Feeds

Carbon Black EDR (Endpoint Detection and Response) is the new name for the product formerly called CB Response.

EDR ships with support for threat intelligence feeds. The Indicators of Compromise (IOCs) contained in the feeds are compared to the sensor data as it arrives on the server. Any activity matching an IOC is tagged; users can search for the tags and, optionally, register for e-mail alerts.

Feeds allow EDR servers to use freely available threat intelligence, proprietary customer threat data, and provides a mechanism to feed threat indicators from on-premise analytic sources to EDR for verification, detection, visibility and analysis.

The EDR 4.0+ server supports three types of indicators:

Binary MD5s
IPv4 addresses
DNS names

The EDR 5.0+ server adds support for two new types of indicators:

Process Queries (Process Searches)
Binary Queries (Binary Searches)

The EDR 6.1+ server adds support for one new type of indicator:

IPv6 addresses

Please note that query IOC types have significant differences as compared to MD5s, IPv4/IPv6 addresses, and DNS names. Please see notes below regarding their usage.

The feed format, described in the “Feed Structure” section below, is designed for simplicity. This should make it easy to add support for feed data from any input source.

Example feed creation scripts are included. See the ‘Examples’ section in this document for a listing of the examples.

Using the EDR Feeds API

The EDR Threat Intelligence Feed API (Feeds API) can be found on GitHub

The Feeds API is a collection of documentation, example scripts, and a helper library to help create and validate Carbon Black feeds. It is not required in order to build a EDR feed - a feed can be created in any language that allows for building JSON, or even built by hand. The feed file itself must match the feed structure, or schema, defined in the “Feed Structure” section below.

Getting started

Install git as needed

    [root@localhost carbonblack]# yum install git
    ...

Clone the github cbfeed repository:

    [root@localhost carbonblack]# git clone https://github.com/carbonblack/cbfeeds.git
    Initialized empty Git repository in /root/repos/carbonblack/cbfeeds/.git/
    remote: Reusing existing pack: 80, done.
    remote: Counting objects: 25, done.
    remote: Compressing objects: 100% (25/25), done.
    Receiving objects: 100% (105/105), 38.03 KiB | 17 KiB/s, done.
    Resolving deltas: 100% (50/50), done.
    remote: Total 105 (delta 10), reused 0 (delta 0)

Navigate to the newly-created cbfeeds directory

    [root@localhost carbonblack]# ls
    cbfeeds
    [root@localhost carbonblack]# cd cbfeeds/
    [root@localhost cbfeeds]# ls
    cbfeeds/  LICENSE.md               README.md         setup.py  validate_feed.py
    example/  percent_encode_query.py  requirements.txt  test.py

Navigate to the example directory and use the example generate_tor_feed.py (inside the example/tor/ directory) script to generate a feed from live tor egress IPs

    [root@localhost cbfeeds]# cd example/
    [root@localhost example]# python tor/generate_tor_feed.py example_tor_feed.feed
    [root@localhost example]# ls -l example_tor_feed.feed
    -rw-r--r--. 1 root root 2179084 Mar 25 08:09 example_tor_feed.feed

Use the example validate_feed.py (inside the parent cbfeeds/ directory) script to validate the tor feed (or a feed of your choosing)

    [root@localhost cbfeeds]# python validate_feed.py --feedfile example/example_tor_feed.feed
    -> Validated that file exists and is readable
    -> Validated that feed file is valid JSON
    -> Validated that the feed file includes all necessary CB elements
    -> Validated that all element values are within CB feed

Feed Structure

Feed: a EDR feed
- FeedInfo: Feed metadata: name, description, etc
- Reports: a list of report
  - Report metadata: title, id, URL
  - IOCs for this report

A feed is a JSON structure with two entries:

feedinfo
reports

The feedinfo structure is a list of basic feed metadata. reports is a list of report structures. Each report has report metadata and a list of IOCs.

feedinfo

feedinfo is a JSON structure with the following entries:

name	status	description
`name`	REQUIRED	Internal name; must not include spaces or special characters. See Notes.
`display_name`	REQUIRED	Display name for the user interface.
`provider_url`	REQUIRED	Human-consumpable link to view more information about this feed.
`summary`	REQUIRED	A short description of this feed.
`tech_data`	REQUIRED	More detailed technical description, to include data sharing requirements (if any)
`icon`	OPTIONAL	A base64 encoded version of the image to use in the user interface
`icon_small`	OPTIONAL	A base64 encoded version of a smaller icon
`category`	OPTIONAL	Category of the feed i.e. Open Source, Partner, Connector, First Party etc

Notes:

The ‘name’ field must not include spaces or special characters. Typically, it should be unique per-feed on a single server.

Explanation of category parameters:

Category Name	Description
`Partner`	Proprietary threat intelligence provided to the Threat Intelligence Cloud via a partner agreement.
`Open Source`	Open Source intelligence that is generally available to the public
`Carbon Black First Party`	Intelligence generated inside the Threat Intelligence Cloud by the Carbon Black Research team
`Connectors`	Intelligence connectors from third party technologies EDR integrates with
`Carbon Black`	Intelligence based on output from host-based integrations
`Meta-feed`	Includes a theme-based aggregate of selected intelligence indicators from other feeds

An example feedinfo structure, from the generate_tor_feed.py script:

{
  "feedinfo": {
    "name": "tor",
    "display_name": "Tor Exit Nodes",
    "provider_url": "https://torproject.org/",
    "summary": "This feed is a list of Tor Node IP addresses, updated every 30 minutes.",
    "tech_data": "There are no requirements to share any data to receive this feed.",
    "icon": "tor.png",
    "icon_small": "tor.small.png",
    "category": "Open Source"
   }
}

report

A report is a JSON structure with the following entries:

name	status	description
`timestamp`	REQUIRED	Time this report was last updated, in seconds since epoch (GMT). This should always be updated whenever the content of the eport changes.
`id`	REQUIRED	A report id, must be unique per feed `name` for the lifetime of the feed. Must be alphanumeric (including no spaces).
`link`	REQUIRED	Human-consumbable link to information about this report.
`title`	REQUIRED	A one-line title describing this report.
`score`	REQUIRED	The severity of this report from -100 to 100, with 100 most critical.
`iocs`	REQUIRED	The IOCs for this report. A match on any IOC will cause the activity to be tagged with this report id. The IOC format is described below.
`description`	OPTIONAL	A short description of the contents of the report.
`tags`	OPTIONAL	An array of strings can be provided as tags for the context as to the contents of the report.

iocs

EDR 4.0 supports three types of IOCs:

IPv4 addresses
domain names
md5s

EDR 5.0+ supports all 4.0 IOCs and adds one additional type:

query - this contains query related to modules or events

EDR 6.1+ supports all 5.0 IOCs and adds one additional type:

IPv6 addresses

iocs is a structure with one or more of these entries:

name	status	description
`ipv4`	OPTIONAL	A list of IPv4 addresses in dotted decimal form
`ipv6`	OPTIONAL	A list of IPv6 addresses
`dns`	OPTIONAL	A list of domain names
`md5`	OPTIONAL	A list of md5
`query`	OPTIONAL	A query of type “events” or “modules”

An example reports list with two report structures, each with one IPv4 IOC, from the example_tor.py script:

  "reports": [
    {
      "timestamp": 1380773388,
      "iocs": {
        "ipv4": [
          "100.2.142.8"
        ]
      },
      "link": "https://www.dan.me.uk/tornodes",
      "id": "TOR-Node-100.2.142.8",
      "title": "As of Wed Oct  2 20:09:48 2013 GMT, 100.2.142.8 has been a TOR exit for 26 days, 0:44:42. Contact: Adam Langley <agl@imperialviolet.org>",
      "score": 50
    },
    {
      "timestamp": 1380773388,
      "iocs": {
        "ipv4": [
          "100.4.7.69"
        ]
      },
      "link": "https://www.dan.me.uk/tornodes",
      "id": "TOR-Node-100.4.7.69",
      "title": "As of Wed Oct  2 20:09:48 2013 GMT, 100.4.7.69 has been a TOR exit for 61 days, 2:07:23. Contact: GPG KeyID: 0x1F40CBDC Jeremy <jeremy@acjlaw.net>",
      "score": 50
    }
  ]

Another example with “query” IOC:

"reports":
[
    {
      "title": "Notepad processes",
      "timestamp": 1388538906,
      "iocs": {
        "query": [
          {
            "index_type": "events",
            "search_query": "cb.urlver=1&q=process_name%3Anotepad.exe"
          }
        ]
      },
      "score": 50,
      "link": "http://www.myfeedserver/feed/report/notepad_proc",

      "id": "notepad_proc"
    },
    {
      "title": "Newly loaded modules",
      "timestamp": 1388570000,
      "iocs":
      {
        "query": [
          {
            "index_type": "modules",
            "search_query": "cb.urlver=1&q=is_executable_image%3Afalse"
          }
        ]
      },
      "score": 50,

      "link": "http://www.dxmtest1.org/02",
      "id": "new_mod_loads"
    }
]

Notes on “query” IOC reports

The following conditions apply for “query” IOC reports

the “iocs” element can only contain one “query” element
only “events” and “modules” are valid values for “index_type” element
a report with a query CANNOT also have other IOCs

The “search_query” syntax is particularly noteworthy. The following conditions apply for the “search_query” field:

the “search_query” syntax is described in CB Enterprise Server Query Overview documentation
the query itself should be prepended with a q=
the query should be percent-encoded. This can be accomplished in several ways, including:
- by copying a query from the EDR UI
- by using a quoting library such as included with python in urllib.
- by using the included percent_encode_query.py script

As with all feeds, it is highly recommended to provide initial validation of the feed with the included validate_feed.py script. For any feeds that include query IOCs, it is recommended to run feed_query_validate.py in the cbapi github repo.

Performance implications

Queries IOCs impose a much higher performance cost on the EDR than md5, dns, and ip IOCs. Furthermore, the relative costs of queries can very signficantly. As a general rule, ‘events’ queries are more expensive than ‘modules’ queries. The use of wildcards, long paths, joined searches, or multiple terms are also expensive.

It is recommended that feed developers take care in constructing query IOCs and test against representative server prior to deploying in production.

Feed Synchronization

The Carbon Black server periodically synchronizes enabled feeds. There are two types of feed synchronization:

Incremental
Full

Incremental synchronization updates any new reports and reports with updated timestamps. Deleted reports and those reports which have been changed, but without a change to the report timestamp, are not synchronized.

Full synchronization accounts for all feed changes, including when a report is deleted. Full synchronization also requires changes to report timestamps. To update a Feed without changes to timestamps, you can add the ignoreTimestampOnFeedUpdate flag to /etc/cb/cb.conf:

# --== Solr Feed and Report Settings ==--

# Setting IgnoreTimestampOnFeedUpdate=True ensures a feed will be updated even if
# its timestamp is not up to date after changes are made. You may need to enable this
# setting if you have manually added a feed that does not update its timestamp after
# modifying reports or the feed itself. This may often be the case with feeds that
# are generated and added to the system by third-party connectors.
#
# Warning: the solr process may consume more resources by enabling this setting.
#IgnoreTimestampOnFeedUpdate=False

If IgnoreTimestampOnFeedUpdate=True Feeds will update even when timestamps are not updated. Warning: the solr process may consume more resources by enabling this setting.

For changes to /etc/cb/cb.conf to take effect, Carbon Black EDR needs to be restarted by executing service cb-enterprise restart.

Full synchronization occurs less frequently than incremental synchronization. It can be triggered manually via the web console or via the Carbon Black Client API. Alternatively, the following practices will result in all report changes being synchronized via incremental synchronization:

Update all report timestamps whenever there is a change to the report. The accuracy of the timestamp is less important than the fact that the timestamp increases.
For reports to be deleted, remove all IOCs from the report and update the timestamp rather than removing the report.

Examples

Several example scripts are available that illustrate using the Carbon Black cbfeeds API to generate Carbon Black feeds from a variety of data sources. The cb-airgap-feed tool allows for importing Carbon Black-provided threat intelligence feeds into an air-gapped EDR server.

Data Source	Description	Location
abuse.ch	The Swiss security blog abuse.ch tracks C&C servers for Zeus, SpyEye and Palevo malware.	cb-airgap-feed
iSIGHT Partners	iSIGHT Partners customers can use their API key to generate a Carbon Black feed from iSIGHT Partners cyber threat intelligence.	iSight Connector GitHub Repo
Malware Domain List	Malware Domain List is a non-commercial community project to track domains used by malware.	cb-airgap-feed
Tor	Provide a Carbon Black feed from a live list of Tor exit nodes provided by torproject.org	cb-airgap-feed

Last modified on January 1, 0001

Carbon Black EDR

Page Contents