Principal Site Reliability Engineer

Clarity Innovations is a trusted national security partner, dedicated to safeguarding our nation’s interests and delivering innovative solutions that empower the Intelligence Community (IC) and Department of Defense (DoD) to transform data into actionable intelligence, ensuring mission success in an evolving world.

Our mission-first software and data engineering platform modernizes data operations, utilizing advanced workflows, CI/CD, and secure DevSecOps practices. We focus on challenges in Information Warfare, Cyber Operations, Operational Security, and Data Structuring, enabling end-to-end solutions that drive operational impact.

We are committed to delivering cutting-edge tools and capabilities that address the most complex national security challenges, empowering our partners to stay ahead of emerging threats and ensuring the success of their critical missions. At Clarity, we are people-focused and set on being a destination employer for top talent, offering an environment where innovation thrives, careers grow, and individuals are valued. Join us as we continue to lead innovation and tackle the most pressing challenges in national security.

Position Overview:
In this role, you will focus on ensuring the availability, reliability, and performance of our multi-tenant, microservices application suite. You will collaborate closely with cross-functional teams to troubleshoot issues, automate processes, and build scalable, resilient systems. You will learn the nuances of the entire KRADOS suite of applications and its infrastructure, which will facilitate your missions of 24/7/365 tier 2/3 outage response, and improving the efficiencies of KRADOS.

Key Responsibilities:

• Monitor system health, define Service Level Indicators (SLIs), and ensure adherence to Service Level Objectives (SLOs).


• Respond promptly to outages, conduct root cause analyses, and implement durable solutions to prevent recurrence.


• Collaborate with development and DevOps teams to optimize and maintain Kubernetes environments and CI/CD pipelines.


• Develop and refine automation scripts to enhance system reliability, including automated recovery and self-healing capabilities.


• Build and maintain observability frameworks, integrating metrics, logging, and tracing tools for proactive issue identification.


• Contribute to performance tuning and scalability improvements across the application stack.


• Document incident responses and contribute to a knowledge base to foster a culture of continuous improvement.


• Participate in an on-call rotation to provide 24/7/365 support for mission-critical systems.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.


• 6 years of experience in site reliability, systems engineering, or DevOps roles.


• Proficiency in one or more programming/scripting languages (e.g., Python, Go, Java, Bash).


• Strong understanding of distributed systems, microservices architecture, and RESTful API design.


• Hands-on experience with Kubernetes and container orchestration.


• Familiarity with monitoring, alerting, and logging tools (e.g., Prometheus, Grafana, ELK stack, or Datadog). Experience with Elastic will be highly helpful with this position.


• Hands-on experience with incident response, including designing and improving incident management processes.


• Expertise in Observability practices, including metrics, logs, traces, and understanding of distributed tracing tools (e.g., OpenTelemetry).


• Strong problem-solving skills with a focus on building resilient, fault-tolerant systems.


• Excellent communication skills and a collaborative mindset.


• Have to have SEC+ or higher certification or ability to obtain it within six months from hire.


• Must be willing to do shift work to provide 24/7/365 coverage.


• Must be within 45 minutes drive of an IL6 workstation location (e.g., SIPR café, SCIF)

Preferred Qualifications:

• Experience with cloud platforms (e.g., AWS) and their associated managed services.


• Knowledge of database management and optimization for systems (e.g., PostgreSQL)


• Familiarity with Infrastructure as Code (IaC) tools (e.g., Terraform, CloudFormation).


• Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

We are an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.