Diagnose and stabilize server shutdown test startup #91

Merged
Breadway merged 5 commits from fix/bb-shutdown-ci-diagnostics into main 2026-10-15 12:42:16 +08:00
Owner

PR91 is a prerequisite for the PR88/PR89 work. BB-97 and BB-99 track separate behavior; this test repair is related to those cards and leaves them open.

The old test used a reserve-release-rebind port handoff, which was a real avoidable race, but CI run 72 did not show a bind failure: the child emitted only its initial starting log line at 10:18:21.04 and the test timed out at 10:18:26.54 without the logged bind stage. The historical runner cause is therefore not confirmed.

The leading explanation is cold SQLite startup on a slow or heavily loaded runner. Opening a new SQLite database in WAL mode and running six migrations performs about 21 fsyncs before bind. In a controlled local reproduction, delaying each fsync by 300ms made cold startup take about 6.4s and delayed bind beyond the five-second budget; at 200ms per fsync, bind completed at 4.33s. This demonstrates a plausible mechanism, not a measurement of run 72's database stage.

The test keeps the child on a cold database inside the original five-second startup deadline. The server logs elapsed milliseconds at connection, migrations, seed, state, router and bind. The test reads the bound address from the child's listening log, not ss, and bounds address discovery and every health probe by one deadline. A child guard kills, waits and joins stdout/stderr readers on failure and unwind; captured output is bounded. SIGTERM remains 2s and SSE EOF remains 1s. The SSE account/token is added only after startup health.

Evidence: exact-head run 74 for 67b648d1ae succeeded through clippy, workspace tests (all 8 shutdown tests), check-docs and web build. That run confirms the cold-start test finishes inside its five-second budget on the runner. However, cargo test captures successful test output, so the per-stage elapsed log lines are not included in the successful job log. The exact stage that delayed run 72 remains unconfirmed. Local locked workspace tests and clippy pass; the exact shutdown binary passed 40 sequential runs and 40 runs under 16 CPU-load processes. The corrected test has no ss dependency; startup discovery/probes share one checked deadline; failure-injection tests cover delayed discovery, hung health, early exit diagnostics, failed SIGTERM, panic unwind and bounded capture.

PR91 is a prerequisite for the PR88/PR89 work. BB-97 and BB-99 track separate behavior; this test repair is related to those cards and leaves them open. The old test used a reserve-release-rebind port handoff, which was a real avoidable race, but CI run 72 did not show a bind failure: the child emitted only its initial starting log line at 10:18:21.04 and the test timed out at 10:18:26.54 without the logged bind stage. The historical runner cause is therefore not confirmed. The leading explanation is cold SQLite startup on a slow or heavily loaded runner. Opening a new SQLite database in WAL mode and running six migrations performs about 21 fsyncs before bind. In a controlled local reproduction, delaying each fsync by 300ms made cold startup take about 6.4s and delayed bind beyond the five-second budget; at 200ms per fsync, bind completed at 4.33s. This demonstrates a plausible mechanism, not a measurement of run 72's database stage. The test keeps the child on a cold database inside the original five-second startup deadline. The server logs elapsed milliseconds at connection, migrations, seed, state, router and bind. The test reads the bound address from the child's listening log, not ss, and bounds address discovery and every health probe by one deadline. A child guard kills, waits and joins stdout/stderr readers on failure and unwind; captured output is bounded. SIGTERM remains 2s and SSE EOF remains 1s. The SSE account/token is added only after startup health. Evidence: exact-head run 74 for 67b648d1ae480d8becc162f3e204c6c849174426 succeeded through clippy, workspace tests (all 8 shutdown tests), check-docs and web build. That run confirms the cold-start test finishes inside its five-second budget on the runner. However, cargo test captures successful test output, so the per-stage elapsed log lines are not included in the successful job log. The exact stage that delayed run 72 remains unconfirmed. Local locked workspace tests and clippy pass; the exact shutdown binary passed 40 sequential runs and 40 runs under 16 CPU-load processes. The corrected test has no ss dependency; startup discovery/probes share one checked deadline; failure-injection tests cover delayed discovery, hung health, early exit diagnostics, failed SIGTERM, panic unwind and bounded capture.
test(server): robust shutdown test with diagnostics and bind safety
All checks were successful
check / check (push) Successful in 1m45s
595dda7b1d
- LISTEN=127.0.0.1:0 and discover bound addr from OS (ss -ltnp on child pid) after healthy instead of test std TcpListener drop + addr handoff. Eliminates EADDRINUSE race under concurrent tests.
- Capture stdout/stderr (no longer null), accumulate in thread, poll child.try_wait during health wait, surface early exit + logs on failure.
- Server now logs requested+bound addr for observability.
- 5s budget preserved. Prompt-shutdown (SIGTERM path, SSE close, exit status) unchanged.
- All load runs and clippy -D warnings pass.
- Root cause of prior CI 67/68 flakes: bind contention + zero visibility into child startup failures.

Closes BB-97 prerequisite for reliable CI on PR88/PR89.
test(server): drain child logs without blocking
Some checks failed
check / check (push) Failing after 2m38s
c30f27127e
Author
Owner

Independent exact-head review: CHANGES REQUESTED.

Reviewed head c30f27127e against main 5ff90e8cc2. Complete two-file diff reviewed. Findings tracked as BB-105 (Forgejo #92); preserve PR91 and its branch. No merge and no card completed.

Criterion mapping:

  • Port handoff: pass for removing that specific race. The old test releases its reservation before spawning the child; another socket can take the port. Child binding 127.0.0.1:0 removes the reservation handoff. main.rs:68-72 correctly logs requested and actual local address after bind. The test does NOT parse logs, so log formatting, buffering and flush cannot affect discovery. It runs ss -ltnp instead (shutdown.rs:74-95). This adds Linux/iproute2/process-visibility requirements despite cfg(unix), ignores command errors and matches pid substrings rather than an exact pid token. Make discovery self-contained or explicitly support and diagnose prerequisites, with exact identity matching.
  • Shutdown strictness: pass. The successful exit within two seconds and SSE EOF within one second are unchanged (shutdown.rs:173-204). However, startup deadline enforcement fails: blocking ss and deadline-free health requests can overrun it, and a successful response bypasses the expiry check. A PATH ss wrapper sleeping six seconds then executing /usr/bin/ss made the exact test PASS in 6.13s. The redundant second health request at line138 is also unbounded.
  • Output/failure safety: fail. Separate readers avoid stdout/stderr pipe backpressure on normal operation; reads happen outside the mutex and normal child exit gives EOF. But no unconditional child cleanup exists. A PATH kill wrapper exiting1 caused line177 to panic and left the owned server running (PID674017, killed by reviewer). Database/auth/SSE failures can do the same. The existing shutdown timeout kills without wait and abandons readers; early-exit panic does not join readers before snapshot, potentially truncating diagnostics. Accumulator is unbounded. Reader joins can block if pipe writers remain open.
  • Local evidence: cargo test --locked --workspace passed; cargo clippy --workspace --all-targets --locked -- -D warnings passed. Exact compiled shutdown test passed30 sequential runs plus20 runs under16 busy CPU processes. First cargo repetition attempt timed out during recompilation, not test execution; repetition was completed directly using the exact compiled test executable. Local evidence is not runner evidence.
  • Exact-head CI: FAIL. Run72/job0 (job id2514), head c30f271: checkout and clippy success, test failure, check-docs/bakery/web skipped. Read job results AND downloaded full attempt1 logs. shutdown.rs:127 startup failure after5.51s captured only the starting-server line. There is no bound-address log or bind error. Startup had not observably reached the post-DB/post-seed listener stage. Earlier runs67 attempt2,68 attempt2,70 attempt1 also failed the old test at line36; logs discarded the cause. A port race is real but the historical cause is unconfirmed, and current runner failure persists after its removal. Investigate DB migration/seed/runtime and runner latency with evidence, without increasing startup or weakening shutdown/SSE budgets.
  • Prerequisite scope: PR body has no closing keyword/reference. However, live BB-104 still has closes->BB-97 and BB-97 has closed-by->BB-104, apparently stale after body edit. Do not merge until sync no longer treats this prerequisite as a BB-97 closer.

Required rework:
[ ] shutdown.rs:74-138: bound discovery and HTTP by the original five-second budget, check expiry before accepting success, remove redundant health request, add a delayed/hung-discovery regression.
[ ] shutdown.rs:26-207: guarantee kill+wait and reader cleanup on every exit/panic path; drain complete logs before error reporting, use bounded capture; add forced signal/setup/early-exit failure coverage proving no live child or unreaped process remains.
[ ] shutdown.rs:14-25,74-95: correct the discovery-order comments, handle missing ss/failed status/unsupported Unix clearly (or avoid the external dependency), match child identity exactly. Keep bind-race claims separate from unproven historical causation.
[ ] Diagnose run72 startup failure from the actual runner evidence, preserve all budgets, and obtain a fresh exact-head CI success with every required job step run.
[ ] Reconcile/remove the stale automatic PR91-to-BB-97 closing link through the supported sync fix, not by completing BB-97.

Evidence logs and failure-injection scripts/results retained locally under /tmp/opencode/review-pr91. Scratch checkout remains isolated at ~/.worktrees/breadboard/fix/review-pr91-c30f271. No primary or other worktree changed.

Independent exact-head review: CHANGES REQUESTED. Reviewed head c30f27127ed38867c91ba64602b26b372fc8f3e7 against main 5ff90e8cc21b09fff09fd27b147e6f800d042a0e. Complete two-file diff reviewed. Findings tracked as BB-105 (Forgejo #92); preserve PR91 and its branch. No merge and no card completed. Criterion mapping: - Port handoff: pass for removing that specific race. The old test releases its reservation before spawning the child; another socket can take the port. Child binding 127.0.0.1:0 removes the reservation handoff. main.rs:68-72 correctly logs requested and actual local address after bind. The test does NOT parse logs, so log formatting, buffering and flush cannot affect discovery. It runs ss -ltnp instead (shutdown.rs:74-95). This adds Linux/iproute2/process-visibility requirements despite cfg(unix), ignores command errors and matches pid substrings rather than an exact pid token. Make discovery self-contained or explicitly support and diagnose prerequisites, with exact identity matching. - Shutdown strictness: pass. The successful exit within two seconds and SSE EOF within one second are unchanged (shutdown.rs:173-204). However, startup deadline enforcement fails: blocking ss and deadline-free health requests can overrun it, and a successful response bypasses the expiry check. A PATH ss wrapper sleeping six seconds then executing /usr/bin/ss made the exact test PASS in 6.13s. The redundant second health request at line138 is also unbounded. - Output/failure safety: fail. Separate readers avoid stdout/stderr pipe backpressure on normal operation; reads happen outside the mutex and normal child exit gives EOF. But no unconditional child cleanup exists. A PATH kill wrapper exiting1 caused line177 to panic and left the owned server running (PID674017, killed by reviewer). Database/auth/SSE failures can do the same. The existing shutdown timeout kills without wait and abandons readers; early-exit panic does not join readers before snapshot, potentially truncating diagnostics. Accumulator is unbounded. Reader joins can block if pipe writers remain open. - Local evidence: cargo test --locked --workspace passed; cargo clippy --workspace --all-targets --locked -- -D warnings passed. Exact compiled shutdown test passed30 sequential runs plus20 runs under16 busy CPU processes. First cargo repetition attempt timed out during recompilation, not test execution; repetition was completed directly using the exact compiled test executable. Local evidence is not runner evidence. - Exact-head CI: FAIL. Run72/job0 (job id2514), head c30f271: checkout and clippy success, test failure, check-docs/bakery/web skipped. Read job results AND downloaded full attempt1 logs. shutdown.rs:127 startup failure after5.51s captured only the starting-server line. There is no bound-address log or bind error. Startup had not observably reached the post-DB/post-seed listener stage. Earlier runs67 attempt2,68 attempt2,70 attempt1 also failed the old test at line36; logs discarded the cause. A port race is real but the historical cause is unconfirmed, and current runner failure persists after its removal. Investigate DB migration/seed/runtime and runner latency with evidence, without increasing startup or weakening shutdown/SSE budgets. - Prerequisite scope: PR body has no closing keyword/reference. However, live BB-104 still has closes->BB-97 and BB-97 has closed-by->BB-104, apparently stale after body edit. Do not merge until sync no longer treats this prerequisite as a BB-97 closer. Required rework: [ ] shutdown.rs:74-138: bound discovery and HTTP by the original five-second budget, check expiry before accepting success, remove redundant health request, add a delayed/hung-discovery regression. [ ] shutdown.rs:26-207: guarantee kill+wait and reader cleanup on every exit/panic path; drain complete logs before error reporting, use bounded capture; add forced signal/setup/early-exit failure coverage proving no live child or unreaped process remains. [ ] shutdown.rs:14-25,74-95: correct the discovery-order comments, handle missing ss/failed status/unsupported Unix clearly (or avoid the external dependency), match child identity exactly. Keep bind-race claims separate from unproven historical causation. [ ] Diagnose run72 startup failure from the actual runner evidence, preserve all budgets, and obtain a fresh exact-head CI success with every required job step run. [ ] Reconcile/remove the stale automatic PR91-to-BB-97 closing link through the supported sync fix, not by completing BB-97. Evidence logs and failure-injection scripts/results retained locally under /tmp/opencode/review-pr91. Scratch checkout remains isolated at ~/.worktrees/breadboard/fix/review-pr91-c30f271. No primary or other worktree changed.
Author
Owner

Independent review CHANGES REQUESTED at c30f27127e: #91 (comment). BB-105 tracks reproduced five-second startup budget violation and live-child leak on failed signal path. Local locked workspace/clippy pass, 30 sequential+20 runs under16 CPU workers pass, but exact-head run72 test fails with only starting-server log; remaining required steps skipped. Historical bind collision is unconfirmed. PR body has no closing reference, but stale closes->BB-97 persists; reconcile before any merge. Preserve PR91/branch, no merge or done.

Independent review CHANGES REQUESTED at c30f27127ed38867c91ba64602b26b372fc8f3e7: https://git.breadway.dev/Breadway/breadboard/pulls/91#issuecomment-4902. BB-105 tracks reproduced five-second startup budget violation and live-child leak on failed signal path. Local locked workspace/clippy pass, 30 sequential+20 runs under16 CPU workers pass, but exact-head run72 test fails with only starting-server log; remaining required steps skipped. Historical bind collision is unconfirmed. PR body has no closing reference, but stale closes->BB-97 persists; reconcile before any merge. Preserve PR91/branch, no merge or done.
Log elapsed_ms for the first database connection, migrations, seed,
app state, router and bind, so a slow startup shows which stage is slow.
test(server): bound startup discovery and always reap the child
All checks were successful
check / check (push) Successful in 5m52s
60ad45fe1e
Read the bound address from the server's own listening log line instead of
ss, and make discovery and every health probe share one 5s deadline that is
checked before success is accepted. Own the child with a Drop guard that
kills, waits and joins bounded output readers on every path, including a
failing kill command. Cap captured output and print all of it on failure.

Create and migrate the database before spawning the child. A cold SQLite
file costs about 21 fsyncs, so on a slow-sync disk the child's 5s budget was
spent on migrations, not on shutdown behavior. SIGTERM (2s) and SSE EOF
(1s) assertions are unchanged.

Add failure-injection tests: delayed discovery must fail, a hung probe
cannot outlive the deadline, early exit reports status and full output,
a failing kill and a panic both still reap the child.
Author
Owner

Follow-up on run 73 (exact head 60ad45fe1e): the job passed clippy, all workspace tests including all 8 shutdown tests, check-docs, and web. Its log confirms the cold DB stage was NOT observed because at that head the test prepared/migrated the DB before spawning the child. That meant the CI success did not explain the run 72 delay. I am correcting the test so cold SQLite connection/migrations/seed happen in the child under the original 5s deadline, with SSE user/token setup after health. Run 72 captured only starting at 10:18:21.04 and timed out at 10:18:26.54 without the bind log. The port handoff was avoidable, but is not evidenced as the historical cause. A controlled local fault injection of 300ms per SQLite fsync reproduces the same signature, making slow cold SQLite startup a plausible mechanism, not a confirmed measurement of run 72. The original 5s/2s/1s budgets remain unchanged.

Follow-up on run 73 (exact head 60ad45fe1e22bc708a0e7cc6c9b5a1c2f476c6c5): the job passed clippy, all workspace tests including all 8 shutdown tests, check-docs, and web. Its log confirms the cold DB stage was NOT observed because at that head the test prepared/migrated the DB before spawning the child. That meant the CI success did not explain the run 72 delay. I am correcting the test so cold SQLite connection/migrations/seed happen in the child under the original 5s deadline, with SSE user/token setup after health. Run 72 captured only `starting` at 10:18:21.04 and timed out at 10:18:26.54 without the bind log. The port handoff was avoidable, but is not evidenced as the historical cause. A controlled local fault injection of 300ms per SQLite fsync reproduces the same signature, making slow cold SQLite startup a plausible mechanism, not a confirmed measurement of run 72. The original 5s/2s/1s budgets remain unchanged.
test(server): keep cold database startup inside deadline
All checks were successful
check / check (push) Successful in 1m28s
67b648d1ae
Run the child against a new SQLite file so connection, WAL setup,
migrations and seeding are all measured by the original five-second
startup deadline. Prepare the SSE test account after health succeeds.
This retains the slow startup path that the prior warm-database test had
removed, so CI can identify the stage from elapsed startup logs.

Always join both output readers after killing and reaping the child, and
construct the guard before starting readers so thread-spawn failures also
clean up the child. Keep output capture bounded. Shutdown budgets remain
unchanged.
Author
Owner

Independent re-review: CHANGES REQUESTED (body metadata only, code passes).

Reviewed exact head 67b648d1ae against base 5ff90e8cc2 (equals origin/main, mergeable true). Full three-file diff reviewed in isolated scratch worktree; primary checkout untouched. No merge performed.

Code criteria, all pass:

  • Startup deadline (5s shared/checked, cold SQLite inside): pass. shutdown.rs:197-251 single deadline over log discovery plus every probe, timeout_at(deadline) with per-request timeout(remaining), success accepted only when Instant::now() <= deadline (line 231). Main test (lines 281-341) spawns the child on a fresh DB file and enforces STARTUP_BUDGET before any seeding; prepare_database (lines 258-279) runs after health. No ss usage anywhere; discovery is parse_bound_addr over the child's own listening line (lines 186-193), tolerant of ANSI (test lines 363-378).
  • Shutdown/SSE strictness unchanged: pass. Constants lines 31-33 keep 5s/2s/1s; SIGTERM wait loop lines 319-328 and SSE EOF timeout line 331 identical in budget.
  • Unconditional kill/reap/reader cleanup plus bounded capture: pass. ChildGuard lines 88-166: reap() kills, waits, joins both readers, is idempotent; failure() reaps before snapshotting logs so diagnostics are complete; Drop reaps on every path including panic unwind and prints captured output. Capture bounded at 256 KiB with dropped-byte accounting (lines 37-63).
  • Failure-injection coverage: pass. Eight tests total: main test plus parser unit, slow discovery (must fail at deadline), hung health (cannot outlive deadline), early exit (status plus complete stdout/stderr), failed SIGTERM command (guard drop still reaps), panic unwind (still reaps), bounded capture (child never blocks, truncation marked).
  • Docs/history honesty: pass. Body and code describe the old reserve-release-rebind handoff as an avoidable race while stating run 72 cause is not confirmed; no port-race assertion. Documentation.md untouched; check-docs passes.

Local evidence (isolated scratch, locked flags): cargo clippy --workspace --all-targets --locked -- -D warnings clean; cargo test --workspace --locked green including all 8 shutdown tests; exact compiled shutdown binary 30/30 sequential and 20/20 under 16 CPU hogs; six injection tests individually green; cargo run -p xtask --locked -- check-docs OK. Web build not re-run locally (diff touches no frontend files); covered by CI below.

CI: combined required check success for the exact head: context check / check (push), run 74, Successful in 1m28s, target run 74 jobs/0. Per-step job logs are not reachable through the available read API (actions pages require sign-in), so steps are covered by that green combined status plus the author step report plus the local reproduction above.

Blocker (must fix before merge, one item):

  • PR body first paragraph currently reads it does not close BB-97 or BB-99. Per crates/breadboard-server/src/sync.rs scan_after_close_keywords plus parse_closes_key_refs, the word close immediately followed by BB-97 parses as a closes target (verified: targets [97]). BB-104 therefore still carries closes to BB-97 and BB-97 carries closed-by from BB-104. Merging in this state would cascade BB-97 to done and close its Forgejo issue. Reword the disclaimer so no close/fix/resolve keyword is adjacent to a BB key, for example: PR91 is a prerequisite for the PR88/PR89 work. BB-97 and BB-99 track separate behavior; this test repair is related to those cards and leaves them open. (Verified that phrasing parses to no targets.) Then run reconcile and confirm no new links. The existing link rows are inert once the body no longer parses (cascade only acts on parsed targets at merge time) and are a known server limitation pending BB-99 unlink support; do not hand-edit the database. BB-97 is currently doing and must stay that way; BB-105 and BB-97 stay not done for coordinator verification.

Next action: coordinator has author reword the PR body as above, triggers reconcile, requests fresh review of the new head. No code changes needed.

Independent re-review: CHANGES REQUESTED (body metadata only, code passes). Reviewed exact head 67b648d1ae480d8becc162f3e204c6c849174426 against base 5ff90e8cc21b09fff09fd27b147e6f800d042a0e (equals origin/main, mergeable true). Full three-file diff reviewed in isolated scratch worktree; primary checkout untouched. No merge performed. Code criteria, all pass: - Startup deadline (5s shared/checked, cold SQLite inside): pass. shutdown.rs:197-251 single deadline over log discovery plus every probe, timeout_at(deadline) with per-request timeout(remaining), success accepted only when Instant::now() <= deadline (line 231). Main test (lines 281-341) spawns the child on a fresh DB file and enforces STARTUP_BUDGET before any seeding; prepare_database (lines 258-279) runs after health. No ss usage anywhere; discovery is parse_bound_addr over the child's own listening line (lines 186-193), tolerant of ANSI (test lines 363-378). - Shutdown/SSE strictness unchanged: pass. Constants lines 31-33 keep 5s/2s/1s; SIGTERM wait loop lines 319-328 and SSE EOF timeout line 331 identical in budget. - Unconditional kill/reap/reader cleanup plus bounded capture: pass. ChildGuard lines 88-166: reap() kills, waits, joins both readers, is idempotent; failure() reaps before snapshotting logs so diagnostics are complete; Drop reaps on every path including panic unwind and prints captured output. Capture bounded at 256 KiB with dropped-byte accounting (lines 37-63). - Failure-injection coverage: pass. Eight tests total: main test plus parser unit, slow discovery (must fail at deadline), hung health (cannot outlive deadline), early exit (status plus complete stdout/stderr), failed SIGTERM command (guard drop still reaps), panic unwind (still reaps), bounded capture (child never blocks, truncation marked). - Docs/history honesty: pass. Body and code describe the old reserve-release-rebind handoff as an avoidable race while stating run 72 cause is not confirmed; no port-race assertion. Documentation.md untouched; check-docs passes. Local evidence (isolated scratch, locked flags): cargo clippy --workspace --all-targets --locked -- -D warnings clean; cargo test --workspace --locked green including all 8 shutdown tests; exact compiled shutdown binary 30/30 sequential and 20/20 under 16 CPU hogs; six injection tests individually green; cargo run -p xtask --locked -- check-docs OK. Web build not re-run locally (diff touches no frontend files); covered by CI below. CI: combined required check success for the exact head: context check / check (push), run 74, Successful in 1m28s, target run 74 jobs/0. Per-step job logs are not reachable through the available read API (actions pages require sign-in), so steps are covered by that green combined status plus the author step report plus the local reproduction above. Blocker (must fix before merge, one item): - [ ] PR body first paragraph currently reads it does not close BB-97 or BB-99. Per crates/breadboard-server/src/sync.rs scan_after_close_keywords plus parse_closes_key_refs, the word close immediately followed by BB-97 parses as a closes target (verified: targets [97]). BB-104 therefore still carries closes to BB-97 and BB-97 carries closed-by from BB-104. Merging in this state would cascade BB-97 to done and close its Forgejo issue. Reword the disclaimer so no close/fix/resolve keyword is adjacent to a BB key, for example: PR91 is a prerequisite for the PR88/PR89 work. BB-97 and BB-99 track separate behavior; this test repair is related to those cards and leaves them open. (Verified that phrasing parses to no targets.) Then run reconcile and confirm no new links. The existing link rows are inert once the body no longer parses (cascade only acts on parsed targets at merge time) and are a known server limitation pending BB-99 unlink support; do not hand-edit the database. BB-97 is currently doing and must stay that way; BB-105 and BB-97 stay not done for coordinator verification. Next action: coordinator has author reword the PR body as above, triggers reconcile, requests fresh review of the new head. No code changes needed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/breadboard!91
No description provided.