Stop Babysitting AI Agents: An Overnight Research Pipeline

Over 100 research papers, and no time to read them. Interactive AI agents helped to reduce the effort, but only with someone at the keyboard: approve this, answer that, re-prompt when context ran out. The agent was fast; the human was the bottleneck.

There is an easy fix - a pipeline that runs overnight. Papers go into a folder in the evening; by morning each one has a structured report, plus a cross-paper synthesis, waiting in the notes app.

The idea came from Mitchell Hashimoto’s “end-of-day agents”: spend the last 30 minutes of the day starting agents on research and exploration, and get a warm start the next morning. Well…the goal was the same, without leaving a laptop on and without trusting an unsupervised agent with anything it could break.

The runner, prompts and security settings are open source: github.com/azeeb/overnight-research-agents.

How it fits together

#

Three machines, one source of truth. The laptop is for thinking, a small cloud server is where agents work, and private GitHub repos sit in between.

Two repos matter. A context repo holds a one-page project brief, so every agent starts with the same picture of what’s being built. A research repo holds papers, prompts, a runner script and the reports.

Context as a repo, not a chat project

#

Project context used to live in a chat app’s “project” folder. That works for conversations, but a headless agent on a server can’t see it. So it moved into git:

Now every agent, today’s researcher or next month’s coder, starts from the same brief. Edit it in one place, push, and the next run picks it up.

The server: always on, always reachable

#

A small cloud VPS (in my case Hostinger) does the work, so nothing depends on my laptop being awake.

• herdr on the server. Terminal multiplexer built for coding agents. Sessions live on the server, so closing the laptop stops nothing, and a sidebar marks each agent as working, blocked or done.
• One window, two machines. The local herdr connects to the server over SSH, so local and remote agents sit side by side in one sidebar.
• Moshi on the phone. An iOS terminal with herdr support, connected over Tailscale. From bed, one glance shows whether a run finished or an agent is stuck, with a way to step in if needed.

The overnight loop

#

One command before bed starts it. The key design choice: a fresh, headless agent for every paper, never one long session.

Long sessions degrade as context fills up; fresh runs keep each report sharp, and one failure can’t sink the rest. Each agent reads the extracted text first, then opens only the PDF pages with key tables and figures. The final pass reads every report and ranks what to act on.

Every report follows the same skeleton: a one-line verdict (use, watch or skip), a TL;DR, method, results with page references, red flags, and how the idea applies to the project. The prompt tells the agent to be sceptical and never invent results.

A day in the pipeline

#

1. Evening, 5 minutes. Drop new PDFs into the research folder in Obsidian and push.
2. Before bed. In a herdr pane on the server, run ./run.sh and close the laptop.
3. Overnight. The runner works through every new paper, writes the synthesis, pushes to GitHub and sends a push notification to my phone.
4. Morning. Obsidian auto-pulls on startup. Read the synthesis first, then only the papers marked “use”.

The first time, run ./run.sh 1 to process a single paper and check the output before trusting it with a full batch. While it runs, tail -f on the log in a second pane shows each paper start and finish.

Guardrails that make it safe to walk away

#

An unattended agent needs tighter limits than one you are watching. The server also runs a production service, so the agents get a fenced-off corner of it.

• A dedicated, unprivileged user. No sudo, no password, no access to other users’ files.
• Resource caps. A systemd slice limits that user to a share of CPU and memory, so a heavy job slows down instead of starving the production service.
• Private network only. SSH is reachable only over Tailscale; the public firewall blocks everything.
• Repo-scoped deploy keys. Read-only for context, write for research. A breach exposes two repos, not the whole GitHub account.
• Headless, read-only tools. Each run allows only Read, WebSearch and WebFetch. Anything else is refused, so the agent never stalls at 2 a.m. waiting for an approval.

The hole that’s left: prompt injection

#

Read plus WebFetch is a risky pair. A paper could hide instructions telling the agent to read ~/.ssh/deploy_research and send it to an attacker’s URL. Every PDF is untrusted input, and the agent reads it with the same tools.

The setup limits the damage rather than eliminating it. The worst case is one leaked key that reaches one research repo: no sudo, no other users’ files, no GitHub account, nothing on the laptop. The next step closes most of the gap: deny rules in .claude/settings.json so the agent can’t read outside the project folder.

{ "permissions": { "deny": ["Read(~/.ssh/**)", "Read(~/.claude/**)"] } }

Even then, anything inside the project folder, including the project brief, could still leak. Treat the server as a place where nothing truly secret lives.

Gotchas along the way

#

• Tailscale SSH checks every user by name. A new Linux user gets “tailnet policy does not permit you to SSH” until you add it to the ACL’s users list.
• command not found inside herdr panes. Panes start non-login shells, which skip ~/.profile, where Ubuntu adds ~/.local/bin to PATH. One export PATH line at the top of ~/.bashrc fixed both herdr and claude.
• “Nested herdr is disabled.” Typing herdr inside a herdr pane tries to open herdr within herdr. Use its subcommands there instead.
• No password means no sudo, by design. System packages are installed from the admin account in a separate terminal, never from a pane the agents share.
• Interactive agents stall overnight. One approval prompt at 2 a.m. blocks until morning. Headless runs with an explicit tool allow-list refuse instead of waiting.
• Usage limits are real. A big batch can hit the plan’s limit mid-run. Because the runner skips papers that already have a report, the rest simply finish the next night.
• PDF text loses tables. Plain-text extraction is cheap and complete, but scrambles results tables. Reading the text first, then only the key PDF pages, gets both.

Results

#

Across 100 papers, the median time from start to saved report was 77 seconds, and 79 of the 100 finished in between one and two minutes. Only one took longer than three minutes.

Lessons

#

• Yes, it’s just a bash loop. No agent framework, no orchestration platform, no queue: a shell script, git and a cron-friendly command. That’s the point: the whole runner is 80 lines of bash. Every piece is boring, inspectable and replaceable, and when something breaks at 2 a.m., the log is one cat away.
• The prompt is the product. A fixed report structure with a one-line verdict lets you sort 100 reports by use, watch or skip before reading any of them in full.
• Context beats cleverness. Feeding every run the same short project brief turned generic summaries into specific, usable ideas.
• Failure should be boring. A paper that fails is simply retried the next night. Nothing to babysit.
• Keep the human at the merge. Agents research and draft; a human decides what to act on.

The shift is less about speed than about attention. Evenings now end with a five-minute hand-off instead of an hour of supervision.