Example: .NET console operational runbook

The operational runbook is the page an on-call engineer opens at 03:00 when a scheduled console application has failed. Doc gen produces one per application at docs/applications/<app-slug>/operational-runbook.md, covering invocation, schedule, console and log output, exit codes, failure scenarios, and alerting.

What reviewers should check

• The invocation block is a runnable named-argument command matching the scheduler definition, and the sentence about the alternate syntax forms is kept and marked † as inherited parser behaviour.
• ## Scheduling names a real job identifier and states upstream and downstream dependencies; an inferred schedule is a defect — "Not determined from source" is the correct value when nothing states one.
• The console output table keeps the four framework lines, including the final Exit Code: <n> line, so an engineer knows what a normal run looks like before comparing it to a failed one.
• Log files are described as the file0-file4 machine-level environment variables the framework scans, with retention per destination and unused ids marked "Not used".
• The exit code table covers the whole SaifConsoleException hierarchy and each row's operator action is specific to this application — "investigate" is not an action.
• Every error scenario names an observable symptom (an exit code, a log line, an alert), the business impact, and a recovery procedure that says whether a rerun is safe and what must be cleaned up first.
• ## Monitoring and alerting names a real recipient and severity per alert rather than a generic distribution list placeholder.

Example

The document below is a complete, correct console-operational-runbook.md for a fictional .NET Framework console application.

View rendered example — the same document as a real page, so you can compare rendering as well as source.

---
title: Operational runbook
summary: Invocation, scheduling, logs, exit codes, error scenarios, and recovery procedures.
status: draft
owner: Policy Systems Team
last_reviewed: 2026-09-10
---

# Operational runbook

`SAIF.Policy.RenewalNotice` runs unattended at 02:30 on BATCHPRD02 and queues
renewal notices for print and email delivery. A failed run is not immediately
customer-facing, but notices have statutory lead times, so a night that is missed
must be rerun before the print vendor pickup at 05:00. Reruns are safe with the
same arguments: a `POLICY.RENEWAL_NOTICE` row is written only inside the
transaction that queues the notice, so anything that failed is still eligible and
nothing is noticed twice.

## Invocation

```
SAIF.Policy.RenewalNotice.exe -NoticeDate 2026-09-18 -NoticeType Standard -BatchSize 500
```

Arguments are named and may also be supplied as `--<Argument>`, `/<Argument>`, `<Argument>:<value>`, or `<Argument>=<value>`.† See [Arguments](arguments.md) for the full contract.

The scheduled task on BATCHPRD02 runs the equivalent of:

```
D:\Apps\RenewalNotice\SAIF.Policy.RenewalNotice.exe -NoticeDate 2026-09-18 -NoticeType Standard
```

### Arguments

| Argument | Example | Required | Notes |
| --- | --- | --- | --- |
| `NoticeDate` | `-NoticeDate 2026-09-18` | Yes | Anchors the notice window. The scheduled task passes the current date. Cannot be a future date or more than 30 days in the past. |
| `NoticeType` | `-NoticeType Standard` | Yes | `Standard`, `NonRenewal`, or `Conditional`. The nightly task always passes `Standard`; the other two are operator-initiated runs. |
| `BatchSize` | `-BatchSize 250` | No | Defaults to `500`, valid range 1-5000. Lower it when working through a backlog during the business day. |
| `WhatIf` | `--WhatIf` | No | Generates and logs notices without writing print files or queue rows. Use it to validate a template change before a real batch. |

## Scheduling

| Attribute | Value |
| --- | --- |
| Frequency | Daily |
| Typical start time | 02:30 |
| Expected duration | 15-30 minutes |
| Scheduler job name | `SAIF-RenewalNotice-Nightly` (Windows Task Scheduler on BATCHPRD02) |
| Upstream dependencies | Policy term replication into the `POLICY` schema (completes by 02:00); `SAIF.Agency.Api` availability |
| Downstream dependents | Print vendor pickup of the drop folder (05:00) and the email delivery service that polls `POLICY.RENEWAL_NOTICE_QUEUE` (every 15 minutes) |

## Console output

| Output | Source | Notes |
| --- | --- | --- |
| `* Resolved argument - <ARG>: <value>` | Argument resolution | One line per resolved argument† |
| `Running as: <userId> on track: <track>` | `ProgramRunnerBase` | Identity and environment track echo† |
| `<ApplicationName> began at <time>` / `ended at <time>` | `ConsoleBase` header/footer | Includes total running time† |
| `Exit Code: <n>` | `SaifStandardProgramBase` | Final line of every run† |

A healthy nightly run shows four resolved arguments, `Running as: svc_batch on
track: PROD`, a notice window line naming the eligible date range, and
`Exit Code: 0`.

## Log files

Log destinations are configured through **machine-level environment variables** `file0`–`file4` on the host that runs the executable. Each variable's value is a log file path.†

| Environment variable | Content | Retention |
| --- | --- | --- |
| `file0` | Run control log: header block, resolved arguments, notice window, progress every 100 notices, totals, and fatal errors | 90 days on disk, then archived to `\\saifarchive\batchlogs` for 7 years |
| `file1` | Notice detail: one line per generated notice with policy number, renewal effective date, agency number, servicing agent, and delivery channel | 90 days on disk, then archived for 7 years |
| `file2` | Exception detail: one line per skipped or failed term with policy number, agency number, and exception message | 90 days on disk, then archived for 7 years |
| `file3` | Not used | — |
| `file4` | Not used | — |

On BATCHPRD02 these resolve to
`D:\BatchLogs\RenewalNotice\RenewalNotice_File0_yyyyMMdd.log` and the matching
`_File1_` and `_File2_` paths.

## Environment configuration

| Setting | Key | Purpose |
| --- | --- | --- |
| Policy database instance | `policy.db.instance` | Oracle instance behind the `policyDb` unit of work |
| Agency API address | `agency.api.baseUrl` | Base address of `SAIF.Agency.Api` used for agency and agent lookups |
| Agency API key | `agency.api.apiKey` | Secure value sent by the agency client on every request |
| Notice window | `renewalnotice.noticeWindowDays` | Default lead time in days applied to the notice window |
| Print drop path | `renewalnotice.printQueuePath` | UNC folder the print vendor collects generated notices from |
| Summary recipients | `renewalnotice.notificationRecipients` | Addresses for the run summary mail |

See [Configuration](configuration.md) for the complete settings catalog.

## Exit codes

| Code | Meaning | Operator action |
| --- | --- | --- |
| 0 | Success | None |
| 1 | Invalid or missing argument | Correct the `SAIF-RenewalNotice-Nightly` arguments field or the manual command line and re-run. A `NoticeType` with no active template row fails here too. |
| 2 | Database error | Confirm the Oracle instance for the current track is up and `POLICY_APP` can connect, then re-run with the same arguments. |
| 3 | Invalid data encountered | Read the `file2` and `file0` entries: either the notice template for the requested type is missing or a term lacks a required merge field. Raise the data correction with the Policy Systems Team, then re-run. |
| 4 | File read/write failure | Verify the print drop share in `renewalnotice.printQueuePath` is reachable and writable by `svc_batch`, and that the `file0`-`file2` paths exist, then re-run. |
| 5 | Unhandled exception | Escalate to Policy Systems on-call with the formatted exception from standard error and the last `file0` progress line. |
| 6 | No data found | No terms fall inside the window. Expected for `NonRenewal` and `Conditional` runs; for the nightly `Standard` run, confirm policy term replication completed before 02:30 and re-run once it has. |

## Error scenarios

| Scenario | Symptom | Impact | Recovery |
| --- | --- | --- | --- |
| Invalid arguments | Validation messages on stderr, exit code 1, empty `file0` | Job fails before any processing | Check the task's Arguments field reads `-NoticeDate <yyyy-MM-dd> -NoticeType Standard`. Correct it and re-run for the same notice date. |
| Database unavailable | `DatabaseErrorException`, exit code 2, `file0` stops after the header block | No notices generated; the statutory window shifts by a day if not recovered before 05:00 | Confirm the instance named by `policy.db.instance` is up and check for blocking on `POLICY.RENEWAL_NOTICE`. Re-run with the same arguments once connectivity is restored. |
| Environment API unavailable | Failure during dependency registration, exit code 5, no header block | Application never starts | Verify Environment API availability for the configured track and that `ApplicationHost` in `App.config` resolves from BATCHPRD02, then re-run. |
| Notice template missing | `DataErrorException`, exit code 3, `file0` records the notice type | Nothing generated for that notice type | Confirm an active `POLICY.NOTICE_TEMPLATE` row exists for the requested type and effective date. Template changes are released by the Policy Systems Team; re-run after the row is in place. |
| Print drop share unreachable | `FileInputOutputException`, exit code 4, `file0` names the UNC path | No notices queued; nothing partially delivered | Confirm the share is mounted and writable by `svc_batch`. No cleanup is needed — notices are recorded only when their transaction commits — so re-run with the same arguments. |
| Agency API outage mid-run | Exit code 0 with a high skipped count; repeated timeout entries in `file2` | Notices for the skipped terms were not generated and remain eligible | Confirm `SAIF.Agency.Api` is healthy for the track, then re-run with the same arguments; only the skipped terms are picked up. If the outage persists past 04:30, notify the Agency Systems Team and hold the print pickup. |
| Empty notice window on the nightly run | Exit code 6, `file0` shows zero terms read | No notices for the night; a day of statutory lead time is lost if not recovered | Check that policy term replication finished. Re-run once it has; if replication is genuinely late past 04:00, escalate to Policy Systems on-call so the notice date can be reconciled. |
| Partial batch after a fatal exit | Exit code 2, 3, or 4 with `file1` lines present but no totals block in `file0` | Some notices queued, the rest still eligible | Re-run with the same arguments. Do not delete queue rows or print files from the failed run: queued terms have a `POLICY.RENEWAL_NOTICE` row and are excluded from the rerun. Confirm the rerun's totals plus the failed run's queued count match the eligible count in `file0`. |

## Monitoring and alerting

| Alert | Trigger | Recipient | Severity |
| --- | --- | --- | --- |
| Renewal Notice batch failed | Scheduled task last result is non-zero (any exit code except 0) | Batch Operations (`batchops@example.saif`) | High |
| Renewal Notice produced no notices | Exit code 6 on the nightly `Standard` run | Policy Systems on-call (`policysystems-oncall@example.saif`) | High |
| Renewal Notice did not complete by 04:00 | No totals block in the `file0` stream for the current date by 04:00 | Batch Operations (`batchops@example.saif`) | High |
| Elevated skip rate | Summary mail reports skipped terms above 5 percent of terms read | Policy Systems Team (`policysystems@example.saif`) | Medium |
| Agency lookup failures | More than 25 agency or agent timeout entries in `file2` for one run | Agency Systems Team (`agencysystems@example.saif`) | Low |

## Related documentation
- [Application index](index.md)
- [Overview](overview.md)
- [Architecture](architecture.md)
- [Run lifecycle](run-lifecycle.md)
- [Arguments](arguments.md)
- [Dependencies](dependencies.md)
- [Configuration](configuration.md)
- [Operations and processes](operations-processes.md)
- [Business rules](business-rules.md)
- [Data model](data-model.md)
- [Tech debt](tech-debt.md)

## Sources
- `src/SAIF.Policy.RenewalNotice/Program.cs`
- `src/SAIF.Policy.RenewalNotice/RenewalNoticeApplication.cs`
- `src/SAIF.Policy.RenewalNotice/NoticeQueueWriter.cs`
- `src/SAIF.Policy.RenewalNotice/App.config`
- `deploy/scheduled-tasks/SAIF-RenewalNotice-Nightly.xml`