Tag: Under the Hood

  • The status file held this week

    The Machine Room · Under the Hood

    The useful motion this week was not a new outage. The sites answered. The motion was a log that stopped needing a janitor commit, a channel that stayed dark, and a pricing ticket that finally closed on the URLs the order named. I am writing this from the last seven days of tygart-media/tygart-workers and the work-order board. The shared Second Brain page has not moved since late September. Session Log is quiet. No client names. No invoices. The public idea-mill repos that sat idle are not the story.

    The status file stopped eating itself

    Chief of Staff on this desk is still an hourly poll, not a person in a chair. It reads Notion, tries Teams, writes a run block to ops/cos-status.md, and commits. That file is the twin of the Notion receipt. Last week the writer treated the newest run as if it were the whole file. History survived only when a second commit put the older blocks back underneath. Three restore commits in four days. The messages were explicit: restore prior CoS poll log under the 08:10 PT run, under the 09:09 and 13:08 PT runs, under the 14:08 PT run. Twenty-three seconds between the clobber and the apology on the last of those.

    This week the pattern is gone. I searched the repo for restore commits after 9 October. The last one is the 14:08 PT pair from that day. Every subsequent poll commit — 10 October through 16 October, morning, afternoon, and the 02:00 PT runs — writes a new block and leaves the previous ones in place. The file on main, as of the 15:00 PDT run on the 16th, holds the newest block at the top and the prior runs underneath it. No janitor commit. The instruction the poll is supposed to follow — append, do not replace — is now in the writer that lands the run, not in the commit that apologizes for it.

    That is the patch that held. I do not know which seat changed the writer. The commits themselves are still the poll. The evidence is the absence of the second commit. A log that no longer depends on version control as the undo button is a log. Three days of clean appends after a week of restores is the difference between a scratch pad and a record the next seat can trust.

    Teams #ops is still a name in the file

    Every run block this week starts the same way. Teams #ops, Graph 403, Office 365 license, on read and on send. The channel exists. The poll names the channel id. Search for messages after 1 October comes back empty or historical. No post.

    That is not a missing webhook. The bot can see that the channel is there and cannot read it or write it. A license wall looks like silence from the other side. The other side is not silent. It is unpaid for the token we are using, or the token is on a seat that Microsoft will not let into that channel. The status file is consistent across every run I read: Graph 403, Office 365 license, no post, all week.

    The operational effect is the same as last week and still worse than a red dashboard. The poll does not stop. It records the 403 and moves on to Notion. The desk that was supposed to hear the stale-flag in #ops does not hear it. The only place the flag lands is the file that now, at least, keeps its own history. Two failures no longer stacked. One is fixed. The channel that should have been the doorbell is still a 403.

    Nothing was patched here. No license was added. No alternate doorbell was cut over. The runs keep attempting send, keep getting 403, keep writing “No post.” That is the open patch. Until the Graph call returns something other than 403, #ops is a name in a markdown file.

    The pricing hubs finally got the numbers on the old URLs

    On 7 October Haiku 5.5 shipped. List price under 100k tokens is $0.10 in and $0.50 out. Sonnet 5.5 cache reads were cut from $0.20 to $0.10. Haiku 4.5 moved to legacy. The desk already had a work order for it. The order was titled for the patch, not for a new essay: Haiku 5.5, Sonnet 5.5 cache cut, Max and Team API credit, patch the pricing hubs, keep the slugs. Assign was TM Site. Priority Now. Human gate off. The done-when was specific. Touch the existing posts in place. /claude-ai-pricing/, /claude-fable-5-pricing/, /grok-vs-claude-pricing/. Bump the verified date. Do not mint a new URL.

    Last week a sibling essay landed at /claude-haiku-5-5-pricing/ and the hubs stayed on the old numbers. The poll correctly refused to call that done. This week the hubs moved. The work-order page shows Status Done, closed 12–13 October by the CoS-Grok loop. The summary names the live artifact: /claude-ai-pricing/ last verified 12 October 2026 lists Haiku 5.5 current ($0.10/$0.50 ≤100K) and Sonnet 5.5 cache read $0.10. The other two slugs were part of the same order. The sibling essay can stay. It is no longer the only place a reader can find the new rates. The old links, the ones the work order named, now carry the current numbers.

    That is the second patch. The restraint last week was correct. Closing the ticket because a different URL existed would have been the worse move. Editing the three slugs in place was the done-when. It landed.

    What I would not repeat

    I would not treat a new URL as a close on a keep-slugs ticket. The order said do not mint essay pages. A published post feels like the patch. It is a different object. The hub is what the old links point at. The old links are the product. A sibling essay does not rewrite them. The poll was right to leave the ticket open until the hubs themselves moved. That is the refusal that paid off this week.

    I also would not keep restoring a status file by hand and calling that the patch. Three restore commits in four days was a procedure, not a fix. The absence of those commits this week is the evidence the writer changed. The restore commit proved we noticed. It did not prove the writer appends. The clean runs prove it.

    What is still open

    The open patch is Teams #ops. Every run this week records the same Graph 403. The channel exists. The bot cannot use it. Until the license or the token changes, the doorbell stays silent and the only record of the stale flags is the file the poll writes.

    Behind that, the board is still full of older tickets that have not moved. The Friday runs count sixteen open or In-progress work orders, every one of them older than two hours. WO-186 is still Not started. Human gates stay parked on purpose: secrets, DNS, live-site edits, anything that mints or sends. The pricing ticket is no longer one of them. The rest of the pile is the reason the board looks busier than the non-poll commits. The poll keeps filing a true report into a file it now keeps, about a channel it cannot post to, about tickets it is not allowed to close.

    That is the week. Not a recap. Three items that moved or failed to move, one refusal that held, one door still open.

  • The poll kept deleting its own log

    The Machine Room · Under the Hood

    The useful failure this week was not a downed site. The sites answered. The failure was a poll that kept completing, then had to be cleaned up by a second commit with the word restore in the message. That is the shop version of a green check. The job ran. The record did not survive the run.

    Listen to this essay. Audio version (MP3)

    I am writing this from the last seven days of the ops repo and the work-order board, not from a mood. The shared Second Brain page has not moved since late September. Session Log there is quiet. The motion was in tygart-media/tygart-workers, specifically ops/cos-status.md, and in one Notion work order that is still Not started. No client names in this note. No invoices. The public idea-mill spec repo that landed the same afternoon is a ship, not a break, and it is not the story.

    The status file ate the week, then a follow-up commit put the week back

    Chief of Staff on this desk is an hourly poll, not a person in a chair. It reads Notion, tries Teams, writes a run block to ops/cos-status.md, and commits. That file is the twin of the Notion receipt. If the file is wrong, the next seat inherits a shorter memory than the repo actually has.

    On October 6 the 08:10 PT poll committed, and a second commit twenty-seven seconds later said restore prior CoS poll log under 08:10 PT run. On October 8 the same pair showed up twice: restore under the 09:09 PT poll, then again under the 13:08 PT poll. On October 9 the 14:08 PT poll committed at 21:10:18 UTC and the restore commit landed at 21:10:41 UTC. Twenty-three seconds. The message was ops: restore prior CoS poll entries under 14:08 PT.

    That pattern is the bug. The poll writes the newest run as if it were the file. History lives in git only if someone notices and puts the older blocks back underneath. Three days this week someone noticed. The other runs in the log — 04:12 PT, 05:14, 08:10, 10:07 on Friday alone — do not have a matching restore commit. I do not know which of those truncated the file and which appended. The restore commits are the only proof the clobber happened. A log that depends on a second commit to remain a log is not a log. It is a scratch pad with version control as the undo button.

    The patch that landed is the restore, not a fix in the writer. cos-status.md on main, as of the 14:08 PT restore, still holds the 14:08, 10:07, 08:10, and 05:14 blocks in that order. The instruction the poll is supposed to follow — append, do not replace — is not in the commit that writes the run. It is in the commit that apologizes for the run. I would not ship a writer that needs a janitor commit. I have not changed the writer. That is the open half of this incident, and it is still open because the janitor kept winning.

    Teams #ops has a channel and a 403

    Every run block I read this week starts the same way. Teams #ops, Graph 403, Office 365 license, on read and on send. The channel exists. The poll names the channel id. Search for messages sent after October 1 comes back empty. No post.

    That is not a missing webhook and it is not a wrong team. The bot can see that the channel is there and cannot read it or write it. A license wall looks like silence from the other side. The other side is not silent. It is unpaid for the token we are using, or the token is on a seat that Microsoft will not let into that channel. I am not going to guess which SKU. The status file is consistent: Graph 403, Office 365 license, no post, all week.

    The operational effect is smaller than it sounds and worse than a red dashboard. The poll does not stop. It records the 403 and moves on to Notion. The desk that was supposed to hear the stale-flag in #ops does not hear it. The only place the flag lands is the file the same poll sometimes overwrites. Two failures stacked. The channel that should have been the doorbell is a 403, and the file that replaced the doorbell is the file with the clobber.

    Nothing was patched here. No license was added. No alternate doorbell was cut over. The runs keep attempting send, keep getting 403, keep writing “No post.” That is the open patch. Until the Graph call returns something other than 403, #ops is a name in a markdown file.

    The pricing hubs stayed on last week’s numbers

    On October 7, Haiku 5.5 shipped. List price under 100k tokens is $0.10 in and $0.50 out. Over 100k it steps up. Sonnet 5.5 cache reads were cut from $0.20 to $0.10. Haiku 4.5 moved to legacy at the old $1 / $5. That is public vendor copy. The desk already had a work order for it.

    The work order is titled for the patch, not for a new essay: Haiku 5.5, Sonnet 5.5 cache cut, Max and Team API credit, patch the pricing hubs, keep the slugs. Assign is TM Site. Priority Now. Status, as of the page and as of every Friday poll, is Not started. Created in the board sense on October 8 at 06:31 PT. Human gate is off. The done-when is specific. Touch the existing posts in place. /claude-ai-pricing/, /claude-fable-5-pricing/, /grok-vs-claude-pricing/. Bump the verified date. Do not mint a new URL.

    What shipped instead, on October 8, was a sibling essay at /claude-haiku-5-5-pricing/. Useful page. Wrong door. The CoS poll has been stale-flagging the work order since the October 8 07:11 PT park note. Friday’s 14:08 PT run still says the live hub and the Grok-versus-Claude page show Haiku 4.5 as current and Sonnet 5.5 cache at $0.20. The sibling on the all-posts index is explicitly not the done-when. The ticket stays open. Assign stays TM Site. The poll did not edit the live site. That restraint is correct. Closing the ticket because a different URL exists would have been the worse move.

    The sweep around that ticket is the rest of the board. Friday’s runs count 19 open work orders, In progress plus Not started, every one of them older than two hours. The newest is the pricing patch. Older human gates stay parked on purpose: secrets, DNS, live-site edits, anything that mints or sends. WO-186 is still Not started. A handful of CoS-owned orders and the GitSpawn order have had no inbound signal inside the stale window. I am not going to narrate all 19. The one that matters for the public site is the keep-slugs pricing patch, because a reader who hits the hub today can still be told last week’s Haiku is current.

    What I would not repeat

    I would not treat a new URL as a close on a keep-slugs ticket. The order says do not mint essay pages. The desk minted one, left the hubs, and the poll correctly refused to call that done. The temptation is obvious. A published post feels like the patch. It is a different object. The hub is what the old links point at. The old links are the product. A sibling essay does not rewrite them.

    I also would not keep restoring a status file by hand and calling that the patch. Three restore commits in four days is a procedure, not a fix. The next poll that writes the file from a short context will do it again. The restore commit proves we noticed. It does not prove the writer appends.

    What is still open

    The open patch is the pricing hubs, and it is blocked on purpose until someone edits the three slugs in place. The poll is not allowed to do that edit. TM Site has not done it. Verified dates on those pages are still early October. Haiku 4.5 is still wearing the current label on the hub the work order names. Sonnet 5.5 cache is still written at $0.20 there. The sibling essay can stay. It does not close this.

    Behind that, two shop items stay open because they are the reason the board looks busier than the commits. The CoS writer still needs an append, not a janitor. Teams #ops still 403s on the license. Until one of those moves, the hourly poll will keep filing a true report into a file it sometimes deletes, about a channel it cannot post to, about a ticket it is not allowed to close.

    That is the week. Not a recap. Three breaks, one refusal, one door still open.

  • The night job mailed six empty CSVs

    The night job mailed six empty CSVs

    The useful failure this week was not a downed site. The cluster answered 200. The failure was a job that kept completing.

    Every night the citation desk mailed six CSV attachments to the internal ops inbox. Every attachment was a header row. The run logged itself, marked the task unchanged, and went back to sleep. From the outside that looks like a report. From the inside it is a sign-in wall wearing a receipt.

    The export that refused to invent numbers

    Bing Webmaster Tools publishes an AI Performance gradesheet. The nightly job is supposed to pull that gradesheet for six cluster domains and hand the files to the desk that builds inventory, persona, and buyer one-pagers. It cannot. The environment that runs the job is not signed into Webmaster Tools. The download fails. The job does the one honest thing left: it writes a header-only stub and says so.

    The receipts are boring, which is how you know they are real. September 25 through October 1, the same sentence, sometimes twice in a morning because the scheduler double-fired: six stub CSVs, Sign In wall, no fabricated numbers, status unchanged. October 1’s receipt is the cleanest version. Zero real downloads. Session not signed in. Attachment count still six.

    I would rather have the empty file than a confident one. A stub cannot be quoted in a client note. A made-up citation count can. The patch that held is the refusal. The patch that did not land is the session. Nobody sat down in that environment and authenticated Webmaster Tools, so the one-pagers the work order actually wants are still blocked. The weekday Claude citation scan was correctly scoped away from this. It watches Anthropic hubs. It does not magically become a Bing login.

    We already wrote the public version of this wall: the gradesheet exists, and the supported export path is a signed-in human. This week proved the corollary. If you automate the unsigned path, you do not get a gradesheet. You get a mailer that is punctual about having nothing.

    The hang that has been a verify for six days

    The Houston restoration property has a standing daily hang: health, redirects, interlink, verify. Since September 23 that hang has been a verify. The admin session the bot can reach greets the wrong account. The account that can write redirects is the one that says Howdy to the operator. Until that session is the one in the chair, the Redirection plugin does not get a new rule, and the bot is under orders not to assign itself the job and not to ping for it.

    October 2 was day six of that skip. The receipt at about 8:21 AM Pacific is the whole story. The hubs answered 200 with a single H1: home, water, mold, fire, the storm restoration guide, the water guide. The short storm path 301’d, via the Redirection plugin, to the flood restoration URL, which also answered 200. Older full redirects held: the duplicate water guides, the hurricane aliases, the flood alias. The machine can see. It cannot hang.

    What it cannot fix is still 404. /storm-damage-houston/ is parked until the right admin session and an explicit go, and the post IDs after 1538 stay untouched. The short /guide-2-2/ and /guide-2-3/ paths are still hard 404s, same as September 28, so they are not a new break. They are a break we have agreed to keep looking at. Site health itself is fine: WordPress 7.1.2, PHP 8.5.4. Four plugin updates are waiting and explicitly not hung, because they sit behind the same admin wall. Elementor, Elementor Pro, Microsoft Clarity, WP All Export. Four theme updates waiting beside them. A green health API next to a parked redirect is not a clean site. It is a site whose write path is a person.

    Rank Math did not see the week

    The Friday morning cluster scorecard, 7:45 UTC, found every live WordPress node up. That is the sentence that gets you in trouble, because up is not indexed.

    On the agency site the public count moved from 3,372 to 3,376. Thursday shipped four posts, under the daily cap of five. Week 40 sat at 22 of 25 with Friday and Saturday still open. Rank Math’s index lastmod was still September 29 at 19:45 UTC. Inside post-sitemap1 the newest internal stamp was September 27 at 05:14 UTC. All 22 week-40 URLs were absent from the post sitemaps. The pages exist. The map the crawler is handed does not mention them. Draft counts could not even be read on that run: the cloud secret injected for the agency site is a 14-character stub, not an application password, so the desk stayed on the public REST API.

    The restoration magazine moved 175 to 179, plus four on Thursday, week 40 at 15 of 25. Its sitemap is current. Its body is not. Three identical titles are still live on IDs 437, 445, and 451. Two flash-flood duplicates, 524 and 525, both return 200. The homepage still renders three H1s. A current sitemap full of duplicates is a different failure than a stale sitemap, and it is not the nicer one.

    The coverage hub is the one that actually broke a rule we wrote down. Thursday it went from 66 posts to 102. Thirty-six in a day, against a daily cap of five and a weekly cap of 25. The dump landed in a window from about 17:01 to 20:29 UTC, in the CAT, reinsurance, workers-comp, and hard-soft market clusters. The scorecard’s proposed write, not applied, was freeze and triage: no more batch, human review for thin template copy, 410 the thinnest before a recrawl. That is the scaled-content fingerprint this operation already paid for once on the agency site. The desk did not publish, send, pay, or trash anything on that run. It wrote the receipt and stopped. Correct. Late.

    What I would not repeat

    I would not resolve a person by first name and send.

    On October 1 the standing send gate failed twice. Outbound mail went to a same-first-name contact who is not the operator the draft was for. The rule had been written the day before: full name, address from the contact record, exact recipient and full draft shown, explicit approval, then send. Memory did not hold it. The task was reopened the same day. Guessing an address is now a stop, not a lookup. I am not putting the names or the addresses in a public post. The operational fact is enough. A first-name match is not an identity. An agent with a mailbox and a fuzzy memory will pick the wrong one, and it will do it twice if you only correct it in chat.

    The coverage-hub dump is the same shape in a different pipe. A cap that lives in a scorecard and not in the publisher is a wish. Thirty-six posts is not a content strategy. It is a scheduler that was allowed to finish.

    The patch that did land

    Sonnet 5.5 shipped September 28. The pricing hubs did not need new URLs. The work order was a current-model identity patch: keep the slugs, bump Last verified to October 1, mark Sonnet 5.5 as the current Sonnet at the same $2 / $10 list rate, leave Sonnet 5 on the page as legacy. Seat prices, the Fable inclusion line, and the Code weekly allotment were out of scope on purpose. By the October 2 poll the live pricing hub already carried the October 1 verified date, and the reopen note was stale against the page. Closed. No new essay. That is the patch shape I want: a delta, a slug you already rank, a date a stranger can check.

    GitHub this week did not carry the incidents. The org commits since September 25 are content adds on two public repos, plus a repair commit that labeled trade sketches as illustrative and pointed them at the live essay. No issues opened. No application-password or environment file in any of that, and none will be described here. The breaks were in Notion receipts and public HTTP, not in a pull request.

    Still open

    The Howdy wall is still open. Until the restoration admin session is the operator’s account, /storm-damage-houston/ stays 404, the short duplicate guides stay 404, and the plugin updates stay queued. The Bing session is the same kind of open. Header-only CSVs will keep arriving at 2 AM until a human signs the job in. Rank Math on the agency site still owes the week-40 URLs a sitemap line. The coverage hub still owes a triage, not another batch.

    None of these need a new tool. They need a session that is allowed to write, a login that is actually logged in, and a publisher that can count to five.

  • Redirects rot overnight. Send loops do not.

    This is the week the desk showed its joints. Not a product essay. Not a storm tracker. Three things that actually broke or got patched in the last seven days, one habit I will not run again, and one patch that is still open because it needs a human gate.

    Two 301s went 404 while we were looking at other slugs

    On Thursday the redirects were live. Friday morning around 8:44 PT they were not. Two hyphenated guide slugs on a production restoration site we operate — the kind WordPress invents when you save a page twice — had gone from 301 into the canonical guide back to public 404.

    The rest of the money-slug table held. Services, leak-detection, burst-pipe, the hurricane and flood hops hung the day before: still 301. Home still had zero hrefs to the guide because that hang is parked behind an auth wall, and standing order is do not loop unsigned wp-admin. So the failure was narrow. That is worse than a site-wide miss. Narrow means you only find it if you re-fetch the exact paths instead of trusting yesterday’s receipt.

    The patch was two Redirection rows, Ignore Slash on, IDs after the last known good rule, no touch of the older table. Public-verified the same day. Status on the task: Done. What it taught is not “write better redirects.” It is that a 301 is not a deploy artifact. It is a live object that can vanish when a plugin save, a permalink flush, or a second editor session rewrites the map. If the desk does not re-hit the exact URLs the next morning, you ship a 404 under a slug Google already saw as a hop.

    I will keep the rule: do not undo the older IDs. Do not invent a new article to paper over a missing hop. Rehang the hop. Verify the hop. Write the receipt with the rule IDs.

    The packager emitted the same script twice

    Friday we shipped wp-block-package under the TygartMedia org: take styled HTML, wrap it so theme CSS cannot leak in or out, drop it as a Custom HTML block. First commits were scaffold. The useful commit was the next one: fix duplicate