Ablit vs ChatGPT and Claude for security testing your own app

ChatGPT and Claude are good at explaining security. They are built to stop short of doing it. If you run a staging app and want an assistant that will attack it, prove the hole, fix it and show the retest, the mainstream products will give you a reading list and Ablit will give you the result. This article walks through one real job so the difference is concrete.

The job and the recording are on the Ablit home page. The target was a local test app with synthetic invoices and two accounts, Alice and Bob. Nothing in it was real, which is the only kind of target an article like this should use.

The job

Try to break my staging app. Can you get into another test account's invoices? Show me how, patch the hole and run the test again.

That is a normal request from a developer who owns the app. It is also, word for word, the kind of request mainstream assistants are tuned to be careful with, because the same sentence could come from someone who does not own the app.

What ChatGPT and Claude do with it

Behaviour varies by day and by phrasing, so we are describing the pattern rather than quoting a transcript. Both assistants are trained to decline or soften requests that read as attacks, even when the user says the system is theirs. The typical outcomes:

• A general explanation of insecure direct object references and how to prevent them, with no attempt against the app.
• A request for proof of authorization, which the chat interface has no way to accept.
• A partial answer that describes the test in prose and leaves the running of it to you.

None of that is useless. It is also not the job. The developer still has to open a terminal, write the request, read the response, write the patch and run the suite. The assistant has helped with the easiest ten percent.

There is a second limit that gets less attention: neither product has a machine. Even a willing ChatGPT cannot run the vulnerable build, send the request or execute the tests. It can only describe them.

What Ablit did

Ablit took the request as given, because its models have no refusal reflex and the product has no rule against testing your own app. It then had somewhere to do the work: its own Kali Linux workspace.

1. It started the vulnerable build inside the workspace.
2. It signed in as Alice and requested Bob's invoice by changing the identifier. The invoice came back. That is the leak, shown in the captured HTTP response rather than described.
3. It patched the access check so the server verifies the invoice belongs to the signed-in account.
4. It rebuilt, repeated the request, and showed the server refusing with a 403.
5. It ran the test suite. All five tests passed, including the new one for cross-account access.
6. It wrote the captured before-and-after responses into a two-page PDF, which is the deliverable the developer actually wanted.

The conversation on the left of the recording and the result on the right are the whole product. The developer read the report, not a tutorial.

Side by side

When ChatGPT or Claude is still the right call

• You want a second opinion on a design, not an attack on a build. Both are excellent explainers.
• Your organisation's policy requires a vendor with a signed enterprise agreement. Ablit sells monthly plans.
• The system is not yours to test. Then nobody should be attacking it, and Ablit's terms say so too.

The line Ablit keeps

Uncensored does not mean unaccountable. Ablit refuses sexual content involving minors in any form, and that rule cannot be changed. Everything else, including offensive security work, is allowed on systems you own or are authorized to test. The footer of every page says "for authorized use on your own systems" because that is the real restriction, and it applies to the person, not the model.

Try it on your own app

Put a vulnerable build of your app in a test environment, open Ablit, and give it the request above. Then compare the two answers you have. One is a paragraph. The other is a 403 and a passing test suite.