A plain-English brief on what got built, how hard it was, and how good it looks. Three real projects, analyzed by the real engine — identities anonymized. The page is ours; the paper is yours.
A consumer app — website, phone agent, and server systems — built by an outside development agency
Owner paid an agency — 250 commits over seven months
Jan 2026 – Aug 2026
confidence: medium
237 commits52 of 214 active days+280,208−102,358 lines3 contributors
277,510 vendored/generated lines detected and excluded from scoring
period score72
difficulty68
quality52
red flags6section 04
all scores out of 100 · the tick marks 50 ≈ one competent professional
01 · the read
Over seven months this repository went from empty to a working AI answering and quoting service for moving companies. It now holds the screens customers use (dashboard, calendar, call and email inboxes, agent setup), a separate internal admin console for your own team, the behind-the-scenes systems that run everything, a public documentation site, and connections to ElevenLabs and Twilio for phone calls, Stripe for billing, Google and Microsoft for calendars and mailboxes, and HubSpot and SmartMoving for pushing leads into customers' existing sales tools. The biggest single pieces built were: a self-service signup and agent-training wizard; a quoting engine that prices moves from hourly, flat, mileage and hybrid rules including overtime, travel time and fuel; an AI agent that answers phone calls, texts and emails, books and reschedules jobs, and hands off to a human; and, in July, a system that lets you run the same phone agent on either of two voice providers and switch a customer between them safely while calls are in flight. About 280,000 lines of hand-written change landed, alongside roughly another 277,000 lines of generated or copied-in content (automatically produced database and routing files, a vendor's email templates, encoded logo images) that does not represent original work and is excluded from the assessment. Nearly all of this came from one person, working under two email addresses, in concentrated bursts across 51 active days. The last three months of work in particular is careful, well-tested engineering. The main concerns are that credentials and website certificate private keys were committed into the repository early on and remain in its history, that several very large single commits are effectively unreviewable, and that the project has no second pair of eyes on any of it.
02 · selected weeks
2026-W07score 85 / 100
The heaviest early week: a complete new dashboard (about 24,000 lines, part boilerplate), automatic synchronisation of the AI agent's instructions using a background job queue, fixes to social and email login, dashboard analytics built on database views, and deletion of the old 26,000-line web app. A run of trial-and-error deployment commits also sits here.
2026-W14score 85 / 100
Biggest week of the first half. The quoting engine landed with its own pricing rules (hourly, flat, mileage, hybrid, overtime, travel, fuel, minimum floors) and tests, metered access for outside software, usage reporting, service-type management screens in two applications, subscription plan management and a documentation site. Also a password reset fix and recovery of previously uncommitted database changes.
2026-W24score 88 / 100
The largest week by delivered scope: dedicated HubSpot and SmartMoving lead delivery with a central lead create-or-update pipeline (about 28,000 lines), mailbox synchronisation for Gmail and Microsoft with an AI email agent that can quote and reschedule, a hand-written calendar replacing the third-party one, and dynamic pricing plans in signup. Includes a deployment safety check for the production database.
2026-W31score 92 / 100
Exceptional week. One commit of about 81,000 lines introduced the switchable phone-call system with two voice providers, safeguards against processing the same call or charge twice, cost tracking, and real database and cache integration tests. On top of that: the admin organisations area rebuilt, staff impersonation for support, protection against a class of server-side request attacks, and repairs to lead delivery. Reviewers rated this the most demanding work of the period.
03 · who did what
Developer A213 commits · 37 active days · ≈850-1280 hours of work
about 99% of the difficulty-weighted work
Built essentially the entire product over seven months: the customer dashboard, a separate internal admin dashboard, the behind-the-scenes systems, the database, and the integrations with ElevenLabs, Twilio, Stripe, Google, HubSpot and SmartMoving. Delivered the self-service signup and agent-training wizard, the quoting engine with its own pricing rules and metered access for outside software, the email and SMS agents with mailbox synchronisation, and the switchable phone-call system with the safety controls that stop calls or payments from being processed twice. Also handled deployment, monitoring, backups, and a security hardening pass that added request throttling, tightened access to internal connection points, and signed the login handshakes with outside services. Roughly 26,000 lines of automated checks were added, most of them from June onwards.
⚑ Worth asking: have the database passwords, monitoring tokens, service secrets and web-certificate private keys that were committed in January through April been rotated? Deleting them from the current files does not remove them from the project's history, so they remain recoverable by anyone with repository access.
Developer B1 commits · 1 active days · ≈0-0 hours of work
negligible, under 1%
One single-line change to an image on the signup page, on 7 February.
04 · red flags
Committed credentials. Passwords and secret keys were saved into the project's files where anyone with repository access can read them, and they stay readable in the project's history even after being deleted. Reviewers found: a plaintext PostgreSQL password in docker-compose configuration (January, repeated later), a plausible real authentication secret in .env.example, an ElevenLabs webhook secret in backend source and in a WEBHOOK_TESTS.md file that was later deleted, Let's Encrypt certificate private keys (infra/nginx/ssl/privkey.pem, grafana-privkey.pem) committed and repeatedly replaced, a live quoting access key in scripts/quoting-test.md, provider-key-looking values and real phone numbers in examples/pooling-numbers.json, and a plaintext Prometheus authorization token in infra/monitoring/prometheus/prometheus.yml. All of these should be treated as compromised and rotated.
Session data written to logs. For a period, the system wrote its own diagnostic output into logs including complete request headers, which can contain login session cookies. If those logs were retained anywhere, they could be used to impersonate users. This appeared in the login/authorization layer and the shared client code in February and was cleaned up during the May hardening pass, but the earlier deployment window is worth confirming with the developer.
Probable live defect. Two of the AI agent's abilities may be calling the wrong address and therefore silently not working. A configuration change on 24 July appears to have copy-pasted one address into the appointment-booking and text-a-link tool definitions, so both point at /api/tools/tally instead of their own connection points. Quick to verify and fix.
Unreviewable commit sizes. Some changes are too large for a second person to check. Several commits bundle a month of work into one submission, including one of roughly 81,000 lines covering the voice-call system, billing, database changes, admin screens and infrastructure at once. Reviewers also noted duplicate identical commits on 30 July, which usually indicates a mistake in how branches were merged rather than extra work. This does not make the code wrong, but it means no one else has realistically reviewed it.
Volume inflated by generated and copied content. Reported line counts are much larger than the work behind them. Roughly 277,000 of the added lines are machine-generated or copied from elsewhere: automatically produced database type and routing files, a vendor's sample email templates complete with Notion/Stripe/Vercel images, duplicated AI assistant documentation stored three times over, and logo images encoded as text. These were excluded from the effort assessment.
Single-person dependency. One person holds all working knowledge of the system. Across 237 commits, all but one line came from a single developer using two email addresses. If that person becomes unavailable, there is no one with context on the phone-call handling, the payment flows or the deployment process.
05 · recommendation
The work is real and, from late May onwards, of good quality. Two things are worth acting on. First, commission a one-time security review focused on secrets: database passwords, a monitoring token, a live quoting access key, a service webhook secret and Let's Encrypt certificate private keys were all committed at various points between January and June. Removing them from current files does not remove them from history, so they should be rotated regardless of whether the repository is private. Second, address the single-person concentration. One developer holds all knowledge of the phone-call system, the payment handling and the deployment process, and several commits (one of about 81,000 lines) are too large for anyone else to review. Getting a second engineer to read through the voice-call and Stripe handling, and asking for smaller commits going forward, would reduce the risk cheaply. Beyond that, one small defect flagged by review is worth a five-minute check: two of the AI agent's tools appear to call the wrong address.
An AI-voice SaaS product built by an outside development agency
Owner paid an agency — five weeks, empty repo to deployed product
Jun 2026 – Jul 2026
confidence: medium
169 commits16 of 33 active days+53,038−14,612 lines3 contributors
48,531 vendored/generated lines detected and excluded from scoring
period score88
difficulty72
quality68
red flags6section 04
all scores out of 100 · the tick marks 50 ≈ one competent professional
01 · the read
Over about four and a half weeks, this repository went from an empty project to a substantially complete AI phone-verification product. One developer, Vinicius, built almost all of it. The system takes an employment verification request, places an automated phone call, navigates whatever it encounters (voicemail, call screening, hold music, press-1 menus, a human HR representative), runs a branching interview using an AI voice agent, records the outcome and transcript, and can also pursue the same verification by email, SMS or fax when a call fails. Alongside that sits a staff dashboard for verifications, calls, campaigns, contacts and access keys, plus PDF verification reports, per-call cost tracking, and a way for other software to connect and submit verification requests directly. The last week added a real-time voice mode that talks to the caller with much lower delay than the earlier approach, and a mode that lets the whole system run inside someone else's product without its own login screens. A significant share of the effort — roughly a week's worth — went into security and reliability rather than features: checking that incoming notifications from Twilio, Telnyx and Resend are actually from those companies, encrypting personal data and call transcripts in the database, preventing the same person from being called twice, preventing a call from being recorded as finished twice, blocking scanner bots, daily database backups, and automated checks that run on every change. That work is unglamorous and it is what separates a demo from something you can point at real customers.
02 · selected weeks
2026-W26score 88 / 100
Three active days but a huge amount of ground covered: the database design and its structured changes for around fifteen tables, sign-in and organization accounts, the first working API for contacts, campaigns and calls, background job workers, the Twilio phone integration with an AI voice agent, the dashboard skeleton, and local development setup. Reviewers estimated 60-85 focused hours across the two slices in this window and rated the integration complexity moderate-to-high. Discounted somewhat because several commits were large scaffolding drops and a few would not have built on their own.
2026-W27score 85 / 100
Three active days covering the employment-verification feature end to end, the staff dashboard for verifications, calls, campaigns, contacts and access keys, production deployment with automatic SSL certificates, and a long tail of real debugging against live phone behavior (signature checks behind the proxy, voicemail and call-screening detection, speech latency tuning). Reviewers put this at 50-70 hours of focused work.
2026-W28score 95 / 100
The heaviest week: 86 commits over five active days. Timezone-aware calling windows, campaign rate limits and duplicate suppression, CSV contact import, outcome tracking, email/SMS/fax delivery channels, PDF reports, multiple AI providers with fallback, then a broad security and data-integrity pass (signature verification on incoming notifications, encryption of personal data and transcripts, duplicate-call and double-finish prevention, backups, automated checks on every change). Reviewers rated the security and concurrency portions senior-level and estimated well over 100 hours across the slices. Some of the volume is automated reformatting and generated files, which is excluded.
2026-W29score 72 / 100
One active day but high-value: the real-time voice mode built and debugged against Twilio's live protocol, per-call cost tracking, and a mode letting the system run without its own login screens for embedded deployments, with automated verification that the two halves stay separable.
03 · who did what
Developer A167 commits · 14 active days · ≈270-390 hours of work
about 99% of the difficulty-weighted work
Built essentially the entire product: the database and its structured changes, sign-in and organization accounts, the phone-calling engine that conducts AI verification calls (including voicemail detection, hold/IVR menu navigation, branching question flows, and a newer real-time voice mode), the delivery of verifications by email, SMS and fax, PDF report generation, and the staff dashboard covering verifications, calls, campaigns, contacts and access keys. Also delivered the production deployment setup, daily database backups, automated checks that run on every change, and a large security and data-integrity pass covering signature checks on incoming notifications from Twilio, Telnyx and Resend, encryption of personal data and call transcripts, duplicate-call prevention, and protection against scanner bots. Closed the period with a real-time voice transport, per-call cost tracking, and a mode that lets the system run without the built-in login system for embedded deployments.
⚑ Worth asking: several reviewers noted that a few individual commits referenced code that did not yet exist at that point in history, and one early set of security gaps (unverified incoming email/SMS/fax notifications, test call-simulation routes exposed in production) shipped before being fixed days later. Is there any second pair of eyes on this work before it reaches production?
a production server account1 commits · 1 active days · ≈0-1 hours of work
well under 1% of the work
Made a single one-line change on a production server, switching the new real-time voice mode to be on by default when no explicit setting is present.
⚑ Worth asking: this flips a safety default that was deliberately set to off until the new voice mode had been validated on live calls, and it was committed directly from a server rather than from a developer machine. Was that intentional and coordinated?
the owner1 commits · 1 active days · ≈0-0 hours of work
negligible
Created the repository with an initial one-line description file.
04 · red flags
single point of knowledge. Almost the entire product was written by one person with no review: 167 of 169 commits, no merges, no second author. Concretely this means bugs and security gaps have only one set of eyes on them, and the project would be difficult for anyone else to take over quickly. The code does carry unusually thorough internal documentation, which reduces but does not remove the risk.
security gaps present before being fixed. For a few days in early July, incoming notifications from the email, SMS and fax providers were accepted without verifying they actually came from those providers, and those notifications could mark a verification as complete. Reviewers also flagged that call-simulation test routes were mounted in the live system and that the fax handler fetched documents from URLs supplied in the incoming message (an avenue for probing internal systems). All of these were closed on 2026-07-10 through 2026-07-11 with signature verification for Twilio, Telnyx and Resend, protections against redirect and DNS tricks, and hiding the test routes in production. Worth confirming nothing was exposed during the window.
history not reliably buildable. Reviewers found several points in the history where code referred to functions that did not exist yet, meaning the project at those specific moments probably would not have built or started. Examples cited: a campaign scheduling function referenced by the queue workers, and the webhook delivery queue used by the verifications service one commit before it was added. This is a symptom of large batches of work being split into commits after the fact, rather than a live defect, but it makes the history unreliable for rolling back to a known-good point.
production change made outside normal workflow. A one-line change on 2026-07-20 came from a server account (a production server account on a hosting provider machine) rather than a developer machine, and it flipped the new real-time voice mode to on-by-default. The deployment configuration still sets it to off, so the two now disagree. Small change, but production-adjacent edits made directly on a server are hard to audit.
credentials visible in the repository. Automated scanning found database passwords and one cloud access key inside files reviewed across several slices (reported as connection-string passwords and an AWS secret key, mostly in example and deployment configuration). Some of these may be placeholders, but they should be checked and rotated if real.
line counts inflated by generated and reformatted files. Two large commits (roughly 5,400 lines changed each, and several thousand more elsewhere) are mostly automated reformatting and generated database migration files rather than new work. These have been discounted in the scoring; the raw line counts overstate the amount of original code by a wide margin.
05 · recommendation
The delivered product is broad and the hard parts (live phone calls, concurrency, security) were treated seriously. Two things are worth acting on. First, this is a one-person system with no code review anywhere in the history; commission a one-time independent security and reliability review by a senior developer, focused on the phone-call handling and the incoming-notification paths, before onboarding real customers at volume. Second, resolve the real-time voice mode default: the code now turns it on when nothing says otherwise, while the deployment configuration and documentation say it should be off until validated. Pick one and make them agree.
A subscription revenue-recovery app built by a solo contract developer
Owner hired one contractor — idea to deployed application
Sep 2024 – May 2025
confidence: medium
52 commits10 of 227 active days+41,517−11,416 lines1 contributor
56,529 vendored/generated lines detected and excluded from scoring
period score26
difficulty65
quality28
red flags6section 04
all scores out of 100 · the tick marks 50 ≈ one competent professional
01 · the read
Over about seven and a half months, one developer built the product from an empty repository into a working subscription-recovery product. The first commit (September 2024) put up the screens and navigation with placeholder data. Then, after a five-month gap, the application was wired to real services: sign-in by emailed magic link, connecting a customer's Stripe account, importing their customer list with live progress feedback, and automated recovery campaigns with scheduled follow-up emails. March 2025 added a real analytics dashboard (recovery rate, churn, revenue retention computed from actual payment events) and then a complete invoicing module that generates PDF invoices, emails them to customers, and can produce them automatically when Stripe reports a payment. Early April was spent getting the application live in production, and the last two working days added switching between multiple connected businesses and finer control over campaign timing.
The work is real and the feature range is wide for one person. Two things temper it. First, the calendar: 52 commits landed on 10 working days, with a five-month gap between September and February and long quiet stretches after. Expert review of the code estimates roughly 150-230 hours of hands-on effort in total, so this reads as an intense part-time project rather than sustained full-time development. Second, quality hygiene is weak in a specific and urgent way. Live passwords and service keys were committed into the project files repeatedly, an encryption routine uses a key that is itself stored in the repository (so the encryption protects nothing), and reviewers flagged an email-sending connection point that appears to be open to anyone who finds it. Those need attention before anything else.
02 · selected weeks
2025-W09score 82 / 100
The heaviest week of the period. The application went from mock screens to working software: email-link sign-in, Stripe account connection, customer list import with live progress, automated campaigns with scheduled follow-ups, and then a dashboard rebuilt on real payment data with churn and recovery metrics. Reviewers estimated 44-72 hours of work landing in these four commits across three days.
2025-W11score 58 / 100
One working day (March 10) delivering the analytics and charting overhaul, expanded Stripe payment-notification handling, and protection-rule statistics, plus same-day corrections to the revenue-retention calculations. Substantive full-stack work compressed into a single day.
2025-W12score 70 / 100
Two days producing the invoicing module essentially from scratch: pulling payment events, generating invoices, producing PDFs, and emailing them through Postmark, with the supporting database structure and business-details settings. Fast and functional, though debug artifacts and an unauthenticated test-email route were left behind.
2025-W14score 45 / 100
23 commits on April 6 taking the application live: automatic invoicing triggered by Stripe notifications, internal route protection, moving the project to the repository root, deployment configuration, and a long run of one-line production fixes. Low line volume but real work; a meaningful share of it was deployment trial and error.
03 · who did what
the developer52 commits · 10 active days · ≈150-230 hours of work
100% of the period's difficulty-weighted work
Built the entire product single-handedly: the initial screens and scaffolding, then the connections to Stripe, the email service, and the scheduling service that turned mock screens into a working application. Later work delivered the analytics dashboard (recovery rates, churn, revenue retention), the invoicing module with PDF generation and emailed invoices, automatic invoicing triggered by Stripe payment notifications, and the production deployment. The final stretch added switching between multiple connected businesses and finer control over campaign timing.
⚑ Worth asking: live credentials (Stripe keys, database passwords, email service tokens) were committed into the project files in at least four separate stretches, and reviewers found an email-sending connection point that appears to require no authentication. What is the plan for rotating those keys and closing that route?
04 · red flags
Committed live credentials (urgent). Practical meaning: someone who finds these files could charge cards, read the customer database, or send email as the business. Reviewers found live-looking credentials committed in tracked environment files across at least four separate periods, including Stripe secret keys, a Supabase database password, Resend/Postmark email tokens, QStash scheduling tokens, and the application's auth secret. Files named `.env`, `.env.actual`, `.env.defunct`, and `.the owner.env` were all committed at various points. Removing the files now is not enough; they remain recoverable from project history, so every key must be rotated.
Encryption that does not protect anything. Practical meaning: the customer Stripe keys stored in the database are not actually protected. In `lib/crypto.ts` the encryption key is written directly into the source code and a fixed salt is used, and the same key also appears in the committed environment file. Anyone with repository access can decrypt every stored key. Separately, a Stripe notification-handling route was found writing decrypted secrets and API keys into server logs; some of that logging was later commented out, but not all of it.
Open email-sending route and leftover development shortcuts. Practical meaning: a stranger could potentially trigger emails from the business's account. Reviewers flagged `/api/invoicing/test-email` as appearing to have no authentication while being able to send mail through the configured Postmark token. A hardcoded development notification secret and a hardcoded temporary tunnel URL (ngrok) were also left in production-bound Stripe code.
Specific correctness bugs flagged in review. Practical meaning: connecting a Stripe account may not verify what the user typed, and could store the wrong key. In the original connection route, the code accepted an API key from the user but validated it against the server's own key instead. Reviewers also noted payment amounts stored in a database column type too small for realistic values, and a reference to the wrong data object in an email-alert function.
Database change history was rewritten mid-project. Practical meaning: the record of how the database has changed over time no longer matches reality, which makes future changes riskier and can cause a deployment to fail or corrupt data. Reviewers found older structured database changes deleted and later ones renumbered and recreated (migrations 0004-0009), leaving the code, the change journal, and the live database potentially out of step. This should be reconciled deliberately rather than left to be discovered during a deployment.
Very large single commits and almost no automated tests. Practical meaning: it is hard for anyone (including a future developer) to verify or safely revise this work. Two commits alone account for the bulk of the product: the September launch and the February commit labelled 'Customer Import feature', which in fact rewrote authentication, the database structure, Stripe connection and notifications, email domain handling, campaigns, and most screens. Across the whole period only one test file exists (406 lines), in a product that moves money and sends invoices. A read-only dashboard data route was also found writing corrections back into the database, which is a side effect that can surprise people later.
05 · recommendation
Two actions, in order. First, treat the committed credentials as compromised: rotate the Stripe keys, database password, and email service tokens, remove the secret files from the project's history, and replace the encryption routine whose key is stored alongside the data it protects. Second, commission a one-time review by a senior developer covering the security items and the specific correctness bugs reviewers flagged (the Stripe key check that ignores the key the user typed in, the unauthenticated email route, the payment-amount columns that may be too small for real amounts, and the rewritten database change history). Going forward, ask for smaller, single-purpose commits and at least basic automated tests around payments and invoicing; the current single test file gives no safety net for a product that handles money. The feature work itself is competent and delivered fast, so this is about hardening what exists rather than rebuilding it.