Dataset
A large-scale, multi-source cybersecurity knowledge dataset containing 1.19M records across 16 categories, collected exclusively for academic, non-commercial research purposes. Last updated: 2026-05-27.
Disclaimer: This dataset is provided for academic research only. All content is aggregated from publicly available sources. The views, opinions, and information expressed in the dataset content do not represent the views or positions of the research team. The research team does not endorse, support, or take responsibility for any of the content. Users access and use this dataset at their own risk.
Security Notice: This dataset contains information about cybersecurity vulnerabilities, exploitation techniques, and offensive security methods. This information is already publicly available and is collected here solely for defensive security research and education. Misuse of this information to attack systems without authorization is illegal. Users must comply with all applicable laws and regulations.
CyberSecurity-1M aggregates publicly available cybersecurity content from diverse sources into a unified, categorized, and quality-annotated dataset. It is designed to support cybersecurity research, LLM fine-tuning for security domains, threat intelligence analysis, and security education.
The dataset covers the full spectrum of cybersecurity knowledge:
CyberSecurity-1M/
├── merged/ # Categorized & merged data (16 categories)
│ ├── vulnerability.jsonl # 878,815 records
│ ├── cn_sec.jsonl # 58,408 records
│ ├── framework.jsonl # 57,925 records
│ ├── reference.jsonl # 44,453 records
│ ├── ctf.jsonl # 43,174 records
│ ├── tool.jsonl # 18,972 records
│ ├── vuln_research.jsonl # 19,189 records
│ ├── incident_response.jsonl # 18,049 records
│ ├── bug_bounty.jsonl # 16,927 records
│ ├── threat_intel.jsonl # 13,663 records
│ ├── offsec.jsonl # 10,087 records
│ ├── conference.jsonl # 4,514 records
│ ├── books.jsonl # 3,124 records
│ ├── news.jsonl # 2,100 records
│ ├── ai_security.jsonl # 1,761 records
│ └── ics_ot.jsonl # 944 records
└── source_registry.json # Inventory of all sources with quality status
Each JSONL record follows the CyberRecord schema:
{
"id": "a1b2c3d4e5f6",
"title": "CVE-2024-1234: Remote Code Execution in Framework X",
"source": "vuln_db",
"_original_source": "vuln_db",
"url": "https://example.com/advisory/CVE-2024-1234",
"category": "vulnerability",
"cve": "CVE-2024-1234",
"author": null,
"date": "2024-03-15",
"tags": ["rce", "critical"],
"description": "A remote code execution vulnerability exists in...",
"markdown": "# CVE-2024-1234\n\n## Description\nA remote code execution vulnerability...",
"exploit_code": null,
"bounty": null,
"extra": {},
"scraped_at": "2026-05-08T12:00:00+00:00",
"schema_version": 1
}
| Field | Type | Description |
|---|---|---|
id | string | Unique record identifier |
title | string | Record title |
source | string | Source identifier |
_original_source | string | Same as source; preserved for provenance after merging |
url | string | Original URL of the content |
category | string | Category name (one of 16 categories) |
cve | string | CVE identifier, if applicable |
author | string | Author name(s) |
date | string | Publication date |
tags | list[string] | Content tags/topics |
description | string | Brief description/summary |
markdown | string | Full text content in markdown format |
exploit_code | string | Source code / payload content (for tool sources) |
bounty | string | Bug bounty amount (for bug_bounty category) |
extra | object | Source-specific metadata |
scraped_at | string | ISO 8601 timestamp of when the record was collected |
schema_version | int | Schema version (currently 1) |
This dataset uses a single train split. Records are organized into 16 category-based configurations (see configs in YAML metadata). Each configuration can be loaded independently:
from datasets import load_dataset
# Load a specific category
ds = load_dataset("WhitzardAgent/CyberSecurity-1M", "vulnerability", split="train")
# Load all categories
ds = load_dataset("WhitzardAgent/CyberSecurity-1M", split="train")
| Category | Records | Size | Description |
|---|---|---|---|
| vulnerability | 878,815 | 3.3GB | CVE records, security advisories, exploit databases |
| cn_sec | 58,408 | 627MB | Chinese-language security content |
| framework | 57,925 | 710MB | Security frameworks, detection rules, standards |
| reference | 44,453 | 941MB | Documentation, cheat sheets, curated lists |
| ctf | 43,174 | 1.6GB | CTF writeups and exercises |
| tool | 18,972 | 166MB | Security tools, templates, cloud audit samples |
| vuln_research | 19,189 | 734MB | Vulnerability research, CWE, detection rules (YARA, Elastic, Sigma) |
| incident_response | 18,049 | 175MB | Incident response, DFIR, forensics, security log data |
| bug_bounty | 16,927 | 364MB | Bug bounty writeups and HackerOne disclosures |
| threat_intel | 13,663 | 991MB | Threat research, APT reports, IOCs (ThreatFox, MalwareBazaar, URLhaus, MISP) |
| offsec | 10,087 | 565MB | Offensive security and penetration testing |
| conference | 4,514 | 336MB | Security conference papers and presentations |
| books | 3,124 | 301MB | OCR-extracted cybersecurity books |
| news | 2,087 | 168MB | Security news and podcasts |
| ai_security | 1,761 | 14MB | LLM/AI security, Web3 security |
| ics_ot | 944 | 38MB | ICS/OT/SCADA security |
Each category contains different types of knowledge. Understanding these types helps select the right data for your use case:
| Category | Knowledge Type | Content Characteristics | Example Sources |
|---|---|---|---|
| vulnerability | Structured factual data | CVE records with standardized fields (CVSS scores, affected products, references). Highly structured, ID-keyed, minimal prose. Good for lookup, classification, and vulnerability assessment training. | NVD, OSV, ExploitDB, GitHub Advisories, CISA KEV, vendor security bulletins |
| cn_sec | Mixed (articles + tutorials) | Chinese-language security articles spanning vulnerability analysis, tool tutorials, and threat commentary. Mix of knowledge-type reference and step-by-step guides. Language is zh-CN. | Anquanke, SecWiki, Xianzhi, Seebug, Govuln, Kanxue, Tttang |
| reference | Knowledge-type reference | Curated link collections (awesome-lists), cheat sheets, quick-reference cards, and documentation indexes. High link density, concise descriptions, broad coverage. Best for building knowledge graphs and resource discovery. | awesome-security, awesome-web-security, Gtfobins, Lolbas, Seclists, GitHub security markdown |
| framework | Rule-based structured data | Detection rules (Sigma, YARA, Suricata/Snort ET Open), MITRE ATT&CK mappings, OWASP standards (ASVS, WSTG, CheatSheetSeries, Top10), and security standards/benchmarks (CIS, NIST). Highly structured, machine-parseable format. Ideal for rule generation and detection engineering. | SigmaHQ, MITRE ATT&CK, CAPEC, YARA rules, ET Open rules, capa rules, CIS benchmarks, OWASP projects |
| ctf | Tutorial/procedural | Step-by-step CTF writeups with detailed exploitation chains, debugging notes, and solution walkthroughs. Rich procedural reasoning — shows how to think about a problem, not just the answer. | CTFtime, CTFSearch, CTF Wiki, LiveOverflow, PentesterLand, Oxdf |
| tool | Code + documentation | Security tool configurations (Nuclei templates, Metasploit modules), scanner rules, and tool documentation. Mixed code-and-prose format. Useful for tool usage training and rule authoring. | Nuclei templates, Metasploit, Loldrivers, CloudGoat, Kubesploit |
| incident_response | Procedural + knowledge + log data | DFIR playbooks, forensic analysis guides, memory forensics documentation, attack sample datasets (EVTX, PCAP), and security log data (Splunk BOTS, Mordor, EVTX-ATTACK-SAMPLES). Mix of step-by-step procedures, reference knowledge, and raw log samples for training. |
This dataset is compiled and distributed strictly for academic, non-commercial research purposes. Any commercial use, redistribution for profit, or application in commercial products is strictly prohibited without explicit written authorization. The research team receives no financial benefit from this dataset.
The content in this dataset is aggregated from publicly available sources and represents the views of the original authors, not the research team. The research team:
This dataset contains technical information about vulnerabilities, exploitation methods, and offensive security techniques. While this information is already publicly available, users should be aware that:
The dataset compilation is released under Apache 2.0 for academic, non-commercial use. Individual content items retain their original source licensing. Some content may have specific terms of service or attribution requirements — users must verify licensing for specific content before any redistribution. Commercial use is prohibited.
@dataset{cybersecurity-1m,
title={CyberSecurity-1M: A Large-Scale Multi-Source Cybersecurity Knowledge Dataset},
author={WhitzardAgent Team (SIIxFudan)},
year={2026},
publisher={Hugging Face},
url={https://huggingface.co/datasets/WhitzardAgent/CyberSecurity-1M}
}
| Volatility, FLARE-FLOSS, Mordor datasets, Malware Traffic Analysis, DFIR KB, Splunk BOTS, Security Log Data |
| offsec | Tutorial/procedural | Red team infrastructure guides, penetration testing methodology docs, attack chain walkthroughs, and offensive tool usage. Strong procedural content showing step-by-step exploitation. | HackTricks, IRedTeam, PentestBook, PSTips, Offensive Security KB, Red Team Cheatsheets |
| threat_intel | Analytical/report + IOC data | APT campaign analysis, malware family reports, threat landscape assessments, and structured IOC collections (ThreatFox, MalwareBazaar, URLhaus, MISP OSINT feeds). Mix of long-form analytical prose with TTP mapping and structured indicator data. Best for threat reasoning, report summarization, and IOC extraction. | Unit42, Kaspersky, CrowdStrike, ESET, Check Point, Mandiant, ThreatFox, MalwareBazaar, URLhaus, MISP feeds |
| bug_bounty | Tutorial/procedural | Bug bounty writeups with reproduction steps, bypass techniques, and bounty amounts. Step-by-step vulnerability discovery narratives. Excellent for vulnerability discovery training. | PentesterLand, HackerOne Hacktivity, Bug Bounty KB |
| vuln_research | Analytical/deep-dive + rule data | In-depth vulnerability analysis, exploit development writeups, reverse engineering case studies, CWE weakness taxonomy, and detection rules (YARA, Elastic, Sigma, capa). Mix of analytical prose and structured rule/code content. Shows why a vulnerability exists and how it was found and detected. | Trail of Bits, PortSwigger Research, Google Project Zero, GH Security Lab, full-disclosure, CWE KB, YARA Rules, Elastic Detection Rules, Sigma |
| books | Long-form knowledge | Full-length cybersecurity books (OCR-extracted from PDFs). Covers topics from network security to cryptography to forensics. Long coherent text ideal for domain pre-training. | Local book collection |
| conference | Academic/presentation | Security conference papers (USENIX Security, NDSS) and presentation slides (DEF CON, BlackHat). Mix of rigorous academic prose and slide-format bullet points. | USENIX papers, NDSS papers, DEF CON media, security conference repos |
| news | Factual/news | Security industry news articles, breach reports, and podcast transcripts. Timely, event-driven content. Good for security awareness and event classification. | Krebs on Security, Security Now, Dark Reading, Darknet Diaries, BleepingComputer |
| ai_security | Research/tutorial | LLM security research (jailbreaks, adversarial prompts), AI vulnerability scanning tools, and Web3/smart-contract security. Emerging domain with mixed analytical and procedural content. | LLM-attacks, NVIDIA garak, GPTFuzz, Guardrails, Web3 Security KB |
| ics_ot | Specialized reference | Industrial control system security: SCADA protocols, OT pentesting guides, ICS-specific detection rules. Niche domain with specialized terminology and procedures. | ICS Security Tools, Redpoint, ThreatHunting Keywords, GRASSMARLIN |