Tester Guide:
Property Management API Grader
Help the industry test and grade PM platform API data access.
Presented by
Property managers should be able to get their own data out of the software they pay for, and right now some platforms make that easy and some make it nearly impossible. We're grading the APIs of the major PM platforms and point solutions to find out which is which, and we can't personally test every product ourselves.
This takes about 15 to 20 minutes. You will never be named in the published report, so there's no risk of blowback from a vendor. Click below to get started.
Inspired by SaaStr's AI Agent API Report Card.
What you'll need.
- A Claude account (Claude Code is what we'll use to actually run the grading)
- Access to the platform's developer/API documentation (usually public, sometimes requires a login from your account)
- The grading file, which you can download here
Five steps, start to finish.
Download the grading file
Download the .md file here. This is the exact prompt we use to grade every platform: same file, same criteria, every time. Nothing hidden, nothing platform-specific baked in. That's on purpose. It's how we keep this fair.
Open Claude Code
If you don't already have it set up, go to claude.com/claude-code and follow the install instructions. If you're already using Claude Code for anything else, you're good to go.
Important: set the model to Opus 4.8 or higher and the reasoning effort to Max. Scores from smaller models or lower effort settings aren't comparable, so we can't use them.
Start a fresh session and point it at the API
Start a brand new Claude Code session, and give it three things:
- The
.mdgrading file you downloaded - A link to (or copy of) the platform's API/developer documentation
- Your live API credentials
Ask it to run the test, and tell it you have access to the live API and will provide the credentials.
Click to see the prompt we use
This is a fresh session. I want you to ignore all previous API grader runs and data and treat this as a clean room. Please run this MD file against [PLATFORM] using my actual API credentials, which I will give to you if you don't have access already.
If you're testing more than one platform, start a separate session for each and repeat the clean-room line, so an earlier score can't colour a later one.
Two things that noticeably improve accuracy. A screenshot of the screen where you generate API keys: Claude often can't tell what key types exist (read-only versus read-write, or whether you can issue more than one), and that feeds several checks directly. And the API documentation itself if it's downloadable, rather than just a link to it.
Let it run, then ask for runs 2 and 3
The first pass takes about 15 to 20 minutes. Claude will work through each category and hand you a score with its reasoning.
You're not done there. The grading file calls for three independent runs, and Claude won't start the other two on its own. Once the first score comes back, remind it to complete runs 2 and 3 with a subagent, per the instructions.
Click to see the prompt for runs 2 and 3
Now do runs 2 and 3 with a subagent (independent grader) per the instructions.
Stay in the same chat window, and don't change the model or effort settings partway through. This step does not hit your API again: it re-grades the evidence already collected, then resolves any disagreements between the runs instead of averaging them. That's what catches a bad assumption in a single run, and it's why we ask for it.
If it doesn't hand you a new file at the end, ask for one: "I'm ready for the final report card MD file."
Skim the final output. If something looks off, or the documentation was incomplete, tell us. We'd rather know than publish a false score.
Submit your results
Drop in:
- Your name and email (so we can follow up if we have a question or need a re-run; never published)
- Platform name
- The completed grading output, as the
.mdfile
You will never be named in the published report. We identify the platform being graded, never the operator who graded it. No vendor will know the score came from you.
We do need your name on the submission itself, so we can come back to you with a question or ask for a re-run if something looks off. That stays between us.