Senior Systems Engineer

Role Purpose

The Senior Systems Engineer builds, operates and supports Vumatel's integrations and payment integrations — the systems that connect our business platforms to the underlying fibre network infrastructure and to our external payment providers. These systems run 24/7 and are business-critical: an integration failure stops customer activations, service changes and payment collection, with immediate operational and revenue consequences. The role is therefore equally weighted between engineering new capability and keeping existing integrations reliable, observable, scalable and recoverable under load.

Reporting line and team structure

Reports to: Principal OSS Solutions Architect.

Works alongside: Systems Engineers on the Payments and NMS teams, the QA and BSS teams, Active Network team, together with the Platform/DevOps and Site Reliability functions who own the shared AWS and Kubernetes environments.

Key Responsibilities

Building and maintaining network integrations

• Write, test and review C# services that translate business-system events (new order, upgrade, downgrade, suspension, cease) into provisioning instructions issued to network element management and OSS platforms, and that feed the resulting state back to the originating system.
• Implement and maintain the API and messaging contracts between Vumatel systems and ISP partner systems, including request validation, retry and back-off behaviour, idempotency handling and duplicate suppression so that a repeated instruction never double-provisions a customer.
• Diagnose provisioning failures reported by the NOC or Service Desk: trace the transaction through application logs and message queues, identify whether the fault sits with the request, the integration or the downstream network platform, and either correct the data, replay the transaction or raise the defect with the responsible party.
• Monitor the integration platforms for exceptions and failed transactions, investigate the underlying fault, and drive each to resolution — whether by correcting data, replaying the transaction, patching the integration or escalating to the responsible system owner — working the outstanding exception backlog down on a scheduled basis rather than allowing it to accumulate.

Building and maintaining payments

• Develop and support the C# services that initiate payment instructions to payment service providers, receive and verify provider callbacks and webhooks, and record payment outcomes against customer accounts.
• Build and maintain reconciliation processes that compare Vumatel's records against provider settlement and statement files, and investigate variances with Finance until each is explained or corrected.
• Prepare the integrations for scheduled high-volume events, in particular month-end and debit order collection runs: confirm capacity headroom in advance, monitor throughput and rejection rates during the run, and remain available for issues arising from it.
• Implement and maintain secure handling of payment credentials and sensitive data, including secrets stored in AWS Secrets Manager, key rotation, encryption in transit and at rest, and restriction of sensitive values from application logs.

Data and persistence

• Design, review and evolve PostgreSQL schemas supporting the integration platforms, including indexing strategy, partitioning of high-volume transaction tables and archival of historical records.
• Write and tune SQL, review query execution plans for slow or lock-contending queries identified through monitoring, and remediate them before they affect transaction throughput.
• Author and apply versioned database migrations, and verify each migration against production-like data volumes in a lower environment before release.

Deployment, infrastructure and scalability

• Containerise services using Docker and maintain the Kubernetes deployment configuration for them, including resource requests and limits, readiness and liveness probes, horizontal pod autoscaling rules and rolling update strategies that allow deployment without dropping in-flight transactions.
• Perform load and soak testing against pre-production environments ahead of significant releases or expected volume growth, document the results, and adjust scaling configuration and connection pooling accordingly.
• Review capacity, error rate, latency and queue depth trends on a weekly basis and raise remediation work into the team backlog before thresholds are breached.

Reliability, monitoring and incident response

• Build and maintain dashboards, metrics, structured logging and alert rules for every integration the incumbent owns, ensuring each production failure mode produces an actionable alert routed to the correct responder.
• Respond to production alerts within the agreed response times, and lead technical triage during integration incidents affecting provisioning or payments, including after hours.
• Act as the technical point of contact during major incidents, providing regular status updates to the NOC, business stakeholders and management until service is restored.
• Conduct and document post-incident reviews, produce written root cause analyses, and implement the resulting corrective actions to completion rather than leaving them as recommendations.
• Write and keep current the runbooks, escalation paths and troubleshooting guides used by the NOC and by other engineers on call.

Testing, quality and change control

• Write unit, integration and contract tests for all new and amended code, and maintain the automated test suites and mock or sandbox provider environments that allow integrations to be tested without touching production systems.
• Maintain and improve the CI/CD pipelines for the services owned by the team, including automated test gates, container image builds and deployment promotion between environments.
• Review colleagues' pull requests, with attention to failure handling, backwards compatibility of integration contracts, observability and rollback safety.
• Prepare and submit change records for production releases, define and test the rollback plan for each, and execute changes during approved maintenance windows, which are frequently scheduled outside of business hours to protect customer-facing availability.

Collaboration, documentation and knowledge sharing

• Work with the OSS Solutions Architect to refine integration designs, and produce sequence diagrams, interface specifications and architecture decision records for the integrations built.
• Attend daily stand-ups, sprint planning, refinement and retrospective sessions, and provide realistic estimates and status on committed work.
• Mentor junior and intermediate engineers through pairing, code review and structured handover of on-call knowledge.
• Maintain accurate technical documentation in the team's knowledge base, and record all work against the appropriate tickets in the team's tracking system.

Minimum Requirements(essential)

Experience

• At least eight (8) years' experience in a comparable systems, software or integration engineering role.
• Demonstrable experience building and supporting production integrations between business systems and external third-party platforms via APIs and messaging.
• Demonstrable experience operating high-availability systems that run 24/7, including production incident response and root cause analysis.
• Experience designing for scale and resilience: retry and back-off strategies, idempotency, graceful degradation, circuit breaking and recovery from partial failure. Technical
• C# and the .NET ecosystem, to a level of independently designing, building and supporting production services.
• PostgreSQL, including schema design, indexing, query tuning and migration management.
• Sound understanding of REST APIs, webhooks, message queueing and asynchronous processing patterns.
• Practical experience with mon