> Detection Engineering: Implementing Detection-as-Code in an Existing Environment: Part 3 – Validation and Future Possibilities

In Part 2, we described how we restructured the data to move all detection data into the new repository while preserving the existing Git history. In this third part, we examine data quality and the use of the data beyond the internal needs of the Detection Engineering team. We cover how systematic validation improved data quality, the first practical uses of the consolidated data, larger planned initiatives, and future opportunities to further improve and automate the Detection Engineering process. Consistency Validation At this point, all detection data had been successfully consolidated in the new repository. The format of the different files was largely kept unchanged to make the migration as easy as possible for detection developers and to avoid introducing additional friction. This also means that some information is stored in multiple files. For example, platform and severity levels are stored in both the metadata file and the implementation, while title and description are stored in the metadata file, the implementation, and the Atomic Red Team test definition. This is not inherently problematic; the data had been structured this way for a long time. However, it creates issues when the different files contain inconsistent values. If the metadata file states a different severity from that in the actual implementation, analyses and reports based on the metadata files are incorrect. Therefore, such inconsistencies needed to be identified and corrected, and we needed to ensure that no new inconsistencies would be introduced into the repository. To accomplish this, a validator was implemented. This validator contains numerous test cases that check files against defined requirements. These checks cover different questions: Whether files are parseable, preventing malformed files from breaking other scripts and automations. Whether files fulfil the defined schema, identifying missing, misspelled, or surplus keys. Whether values stored in multiple files are consistent. The checks are divided into different levels: required: These checks must always pass. required_on_change: These checks must pass when a detection is modified; for existing, unchanged detections, they are considered optional. optional: These checks create remarks for the developer but do not block a change. The differentiation between these levels makes the initial improvement of consistency manageable. While some issues need to be fixed before the data can be relied upon, such as severity values, MITRE ATT&CK IDs, and platforms, other data should be consistent long-term but would require too much effort to correct in a single step. Their inconsistency does not cause immediate problems, for example where Atomic Red Team test cases do not match a naming schema. These non-urgent checks become mandatory once a developer modifies the affected detection; until then, they remain optional. Furthermore, there are scenarios in which fields that are usually identical differ for legitimate reasons. For example, some descriptions contain variables in the implementation that are populated in the Splunk UI, while the description in the metadata document remains generic. The validator is built as a library that is used in two ways. Firstly, it is available as a CLI tool. This tool has parameters that define which levels of checks to report. With this tool, we identified inconsistencies and corrected them manually. This sometimes required research in old tickets to determine which of two values for a field was correct, as an error could have been introduced either when defining the metadata file or when implementing the detection. Secondly, validation is integrated into the development process through CI pipelines. When a pull request is opened, the CI pipeline runs the validation checks on the detection folders included in the pull request changeset. The result and detailed output are displayed in a GitHub check in the pull request to provide direct feedback to the detection developer. If any required or required_on_change checks fail, the pull request is automatically blocked from merging. When optional checks fail, the output is displayed in the check, but the pull request is not blocked. This ensures that the data in the repository remains consistent over the long term and that non-urgent inconsistencies are corrected over time. Automations, Reports and Other Integrations At this point, we had achieved several important results: We designed a new data structure tailored to the workflows of the Detection Engineering team. We reorganized the existing data into this new structure. We migrated additional information scattered across manually updated Excel files and reports into the repository. We removed existing inconsistencies in the data and technically ensured that no new inconsistencies are introduced during the development process. This allowed us to establish the Git repository as the single source of truth for all detection-related data. The validator also ensures consistency across all files in the repository – for example, between the metadata documents and the corresponding technical implementations. As a result, the metadata documents can be used reliably for analysis and reporting, as they consistently reflect the deployed state in the production platforms. In Part 2 of this series, we already mentioned the CLI tool used to generate the existing Excel reports from the repository. With the achieved data quality in the repository, these reports are now far more reliable than the previously manually updated versions, which partially contained outdated data and errors introduced through human error. Furthermore, these reports can be generated automatically in pipelines and provided to other departments, such as Compliance, on a regular basis without manual effort from the Detection Engineering team. Another analysis tool uses the repository data and the DeTTECT framework to create layer files for the MITRE ATT&CK Navigator. These files provide a visual overview of detection coverage and can support the Threat Intelligence team in comparing current coverage with the tactics, techniques, and procedures (TTPs) profiles of potential attacker groups. Furthermore, we initiated a larger project: a full rework of reporting on the detection library by synchronising detection metadata into their ServiceNow ITSM (IT service management) solution. The goal is to fully remove manual reporting work, even though it is already partially automated. Non-developer stakeholders, such as management, internal audit, and external auditors, should not be expected to work with plain YAML files or navigate GitHub repositories. However, these stakeholders have a valid interest in the detection library, as they may need to ensure coverage of specific techniques for compliance reasons, oversee the development backlog, or evaluate Key Performance Indicators (KPIs). As ServiceNow is already established as a central location for different types of data, it is well suited to provide information about the detection library and make the latest data available to these non-developer stakeholders. This removes the requirement to manually update Excel files for audit purposes. ServiceNow can provide a complete view that answers questions such as: “Which detections exist for MITRE ATT&CK technique T1003.001, which are currently in development, which are planned, and which existed in the past?” In addition, ServiceNow maintains a trusted change log of values to further fulfil compliance requirements. ServiceNow will contain only information from the metadata documents, not implementation contents. No detection data can be edited from ServiceNow; GitHub remains the single source of truth. The data is collected from the repository and pushed to ServiceNow by pipelines. In addition to data for active detections from the metadata files, information displayed in ServiceNow is also taken from issues on the GitHub Project board and metadata files in the detections_retired folder. With the availability of the detection library data in ServiceNow, there is the possibility to move more related processes into ServiceNow workflows that are currently are integrated into GitHub directly. These processes could include coverage analysis (comparing the current library with threat actor profiles) as well as requesting the development of new detections. Such a ServiceNow workflow could open a new issue in GitHub on the project board. This would further reduce the number of people, who are not detection developers, that require access to the GitHub repository. Future Possibilities After these modifications to the data storage and development processes have been implemented and are running smoothly, there are further improvement opportunities to evaluate. Auto-Deployment of Detections to Test Environments With detections now organised as modular, encapsulated entities rather than scattered across files and repositories, it becomes possible to automatically create test bundles. It would be possible to fully automate the deployment of detections to a test environment. When the ticket for an ongoing development is moved to the testing status in the GitHub Project board, the modified detection in the linked pull request could be deployed to a test environment. As soon as the issue progresses further, for example to the review or completed status, the detection could be removed from the test environment again. This would ensure that every detection is tested in the exact form in which it will later be deployed because the production deployment logic would also be used in test environments. It would also remove manual steps that could introduce human error. Autonomous Test Execution In the current process, every detection must be tested before release. The detection is deployed to a test environment, its Atomic Red Team test definition is executed on a client machine in that environment, and the resulting alert is recorded in the review documentation. Currently, all these steps are performed manually. Building on the deployment automation described in the previous section, test execution and result gathering could also be automated. Following the restructuring, Atomic Red Team test definitions are organised per detection and can therefore be copied to the appropriate test machine and executed from a CI pipeline. A regularly executed job could then collect alerts generated in the test environments and match them to the corresponding test executions. Both test execution and alert collection could be reported through separate GitHub checks in the relevant pull requests, alongside the existing validator output. These CI pipeline steps could be linked to the status of an issue on the GitHub Project board, ensuring that tests run only when a detection is ready for testing and thereby keeping noise in the test environment low. This would remove repetitive manual tasks for detection engineers and reduce the risk of errors, such as executing an outdated test version. It would also remove several operational hurdles, including the need for every detection developer to have interactive access to the test machines and a limited number of concurrent RDP sessions. With this automation in place, a developer would simply move an issue to the testing status after implementing a detection and its test. The CI pipelines would then perform validation and testing, and provide the reviewer with all relevant success or failure details. This setup would also enable regular testing of the complete detection library. Without automation, existing detections can only be tested on a sample basis. With automation, the full library could be tested more frequently, allowing broken detections, for example because of missing log sources, to be identified and scheduled for rework. Generation of Splunk Configurations In the new setup, redundant information is still stored in different files. One possible next step would be to remove this redundancy by storing only the parts of a Splunk search that cannot be derived elsewhere, mainly SPL and drill-down searches. Other information could be taken from the metadata documents, such as title, severity, description, and tags, or from general configurations, such as the schedule. Transforms for simple false-positive lookups could also be generated from the filename and the CSV header in the lookup file. This would further reduce manual work, free resources, and lower the risk of errors. However, it requires auto-deployment to test environments, as there would no longer be a complete Splunk configuration in the repository that could be copied easily to a test environment. Wrapping Up Together with the customer, we streamlined the Detection Engineering process, ensured data consistency, and implemented initial reporting and automations based on the new data foundation. We also supported the definition and initiation of larger efforts that build on this foundation. All tooling implementations were built to be as modular as possible, allowing reuse and making it easier to integrate additional automations into the setup in the future.