After five months of development, IncidentRelay 1.1 has been released. It develops an open-source system for on-call management, alert routing, and incident response, running on a self-hosted server. IncidentRelay 1.1 is marked as the first stable release (the 1.0 branch was in beta). The project is targeted at SREs, DevOps, and infrastructure teams seeking a locally deployed alternative to SaaS services for on-call management, escalation policies, and incident response. The project code is written in Python and distributed under the MIT license.
IncidentRelay receives events from monitoring systems, matches them with the service, team, and rotation, and then delivers notifications to the responsible duty officers or teams. The system implements duty schedules, rotations, shift overrides, incident acknowledgement, ACK/Resolve status, reminders, escalations, temporary duty officer replacements, scheduled maintenance times, and alert suppression.
The project's main advantage is complete control over the infrastructure and routing logic. IncidentRelay is deployed in its own environment, works with its own database, and allows for explicit separation of incoming routes, commands, rotations, and delivery channels. Incoming tokens belong to routes, not channels, making it easier to understand which external source has permission to send events to a specific command.
IncidentRelay supports receiving events from Prometheus Alertmanager, Grafana Alerting, Zabbix, Sentry, LibreNMS, RMON, AWS SNS/CloudWatch, and custom webhooks. Notifications can be sent through Mattermost, Slack, Telegram, Discord, Microsoft Teams, email, webhooks, browser/PWA push, and voice call providers. In Mattermost and Telegram, notifications can include actions to acknowledge and resolve the issue, allowing incidents to be handled without switching to a separate interface.
The project can be run via Docker Compose, an RPM package for Red Hat-like distributions, manually via systemd, or in Kubernetes using a Helm chart. SQLite can be used for small installations, while PostgreSQL is recommended for production environments and higher loads.
The new version proposes the following changes:
- Added multi-level on-call rotations with time limits, layer priorities, and consideration of temporary replacements;
- A duty calendar, CalDAV and ICS subscriptions for external calendars have appeared;
- Implemented incident escalation policies with multi-step escalation chains;
- Added alert groups, event grouping, delayed notifications and manual merging of related alerts;
- Windows for scheduled work and “silent” alerts have appeared;
- Added the ability to add comments to alerts;
- Added incident priorities (P1-P5) and automatic priority escalation based on importance level;
- Implemented service catalog, service dependencies, SLI/SLO, Service Impact History and Business Services;
- Explain Trace has been added for analyzing routing: why an alert was or was not included in a command, was grouped, suppressed, or sent to a specific channel;
- Added checks (Heartbeats/dead-man-switch) for watchdog, backup, ETL and other tasks where the problem is the absence of the expected signal.
Source: opennet.ru
