Exercise 4: Threat Feed Ingestor

Objective

Fetch IP addresses from external threat intelligence feeds and return them as structured JSON data that can be consumed by security tools and firewalls.

This exercise teaches you how to build threat intelligence automation that aggregates security data from multiple sources and provides it in a consumable format for security operations.

What You'll Learn

• Threat Intelligence: Consume and process external security feeds
• Data Transformation: Parse and normalize threat data formats
• API Design: Create RESTful endpoints for security data
• Caching Strategies: Implement efficient data retrieval patterns

Architecture Overview

External Threat Feeds → Worker (HTTP) → Data Processing → JSON API Response
                             ↓
                        Format & Filter
                             ↓
                       Structured Output

• Trigger: HTTP requests to /threats endpoint
• Processing: Fetch, parse, and normalize threat feed data
• Output: Clean JSON format with IP addresses and metadata
• Pattern: Simple API endpoint with data transformation

Prerequisites

Before starting this exercise, ensure you have:

• Understanding of threat intelligence concepts
• Knowledge of HTTP API design
• Familiarity with data parsing and transformation
• Basic security operations awareness

Instructions

1. Project Setup

Create a new Worker project for threat intelligence:

npm create cloudflare@latest threat-feed-ingestor
# Select "Hello World" Worker
# Choose TypeScript
# Select "Yes" to deploy
cd threat-feed-ingestor

2. Understanding Threat Feeds

Common threat intelligence feed formats you'll work with:

• Plain text lists: Simple newline-separated IP addresses
• CSV format: Structured data with additional context
• JSON feeds: Rich metadata with indicators
• Comment-based: Feeds with metadata in comments

3. Implementation Challenge

Your task is to implement a Worker that:

• Provides a /threats endpoint that accepts HTTP GET requests
• Fetches data from multiple threat intelligence sources
• Parses different feed formats (text, CSV, JSON)
• Filters and validates IP addresses
• Returns structured JSON with threat indicators
• Implements basic caching to reduce external API calls
• Handles feed failures gracefully with fallback data

Key Requirements:

• Support multiple threat feed sources
• Parse and validate IP addresses correctly
• Include metadata like threat type, confidence, and timestamps
• Implement rate limiting protection for external feeds
• Return consistent JSON format regardless of source

🎯 Hint: Popular Threat Feeds

Use these free/public threat intelligence sources:

const threatFeeds = [
  'https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt',
  'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt',
  'https://www.binarydefense.com/banlist.txt',
  'https://cinsscore.com/list/ci-badguys.txt'
];

📊 Hint: Output Format

Structure your JSON response like this:

{
  "success": true,
  "timestamp": "2024-07-25T10:30:00Z",
  "total_threats": 1250,
  "sources": ["ipsum", "emerging-threats", "binarydefense"],
  "threats": [
    {
      "ip": "192.0.2.1",
      "type": "malware",
      "confidence": "high",
      "first_seen": "2024-07-20T14:30:00Z",
      "source": "ipsum",
      "description": "Known botnet C&C server"
    }
  ]
}

🔧 Hint: IP Validation

Implement robust IP validation:

function isValidIP(ip) {
  const ipv4Regex = /^(\d{1,3}\.){3}\d{1,3}$/;
  if (!ipv4Regex.test(ip)) return false;
  
  return ip.split('.').every(octet => {
    const num = parseInt(octet, 10);
    return num >= 0 && num <= 255;
  });
}

Testing Your Solution

1. Basic Functionality: Test the /threats endpoint returns valid JSON
2. Data Quality: Verify IP addresses are properly validated
3. Source Integration: Confirm multiple feeds are processed
4. Error Handling: Test behavior when feeds are unavailable
5. Performance: Measure response times and implement caching

Success Criteria

Your implementation should:

• ✅ Successfully fetch from multiple threat intelligence sources
• ✅ Parse different feed formats correctly
• ✅ Validate and filter IP addresses accurately
• ✅ Return consistent JSON format with proper metadata
• ✅ Handle feed failures without breaking the entire response
• ✅ Implement basic caching for performance
• ✅ Include proper HTTP status codes and error responses

Solution

🔒 Complete Threat Intelligence API

This content is password-protected. Enter the password to view the content.

Next Steps

After completing this exercise, you've built a professional threat intelligence API that:

• Aggregates data from multiple reliable threat feeds
• Provides clean, structured threat intelligence data
• Handles errors gracefully and maintains high availability
• Offers flexible querying and multiple output formats

This API can be integrated into firewalls, SIEM systems, security automation workflows, and other security tools to enhance your organization's threat detection capabilities!