We’re choosing to share this incident publicly, although not without hesitation. Being transparent about security issues is uncomfortable and feels scary. But we believe it’s important to follow our own values. Trust is built not by hiding mistakes, but by owning them, learning from them, and showing our commitment to doing better. We communicated a more detailed version to our existing customers and we are working with them on any concerns they may have. We hope this openness reinforces our dedication to security, accountability, and the long-term trust of our customers and community.
Our customers can find a more detailed version of this report in their GitHub repository.
What Happened
A publicly exposed endpoint was discovered within our infrastructure automation setup, which integrates with version control systems to plan and apply changes. The endpoint was intended to be protected via IP-based access restrictions but due to a configuration error, those restrictions were not enforced.
The misconfiguration stemmed from the use of an outdated Kubernetes annotation for ingress access control, causing the IP whitelist to not be applied.
Resolution
Upon discovery:
- We corrected the misconfiguration and confirmed that the intended IP-based access control was functioning.
- We reviewed all other exposed endpoints across our systems to ensure proper access controls were in place.
- An internal investigation was conducted to assess potential exposure and identify improvements.
Impact
The exposed endpoint allowed read-only access to a web interface used to visualize infrastructure changes.
While no credentials, sensitive application data, or customer-owned data were exposed, some configuration metadata (such as cloud resource identifiers and IP ranges) was viewable. Logs indicate the endpoint was accessed by Internet bots, suggesting some external visibility.
Risk Assessment
Although the interface was read-only and could not trigger infrastructure changes, the exposed metadata could theoretically aid in identifying further vulnerabilities if malicious actors had context.
No evidence was found of active exploitation, and no changes were triggered via this endpoint.
What we learned from this
We are sorry this happened and acknowledge making mistakes leading up to this security incident. We’re taking the following actions to prevent similar incidents:
- Researching and implementing a solution to actively scan and test all our endpoints for (mis)configuration
- Improving our incident response process to ensure fast resolution and reporting
- Further engaging with security partners for advice and auditing our security practices
Skyscrapers Platform Circle