I burnt my antropic credits with 2 tests

[maximouton]

Hi,

I have been trying shortest and find it fantastic I would really push it in my company but I was shocked when I see the difference between estimation of costs from the logs and the actual consumption.

For my tests I used the youtube exemple and here is the corresponding log:

`📄 app/tests/youtube.test.ts
● Search for 'cute puppies' on YouTube and play the first video
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-00-17-404Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-00-32-435Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-00-47-251Z.png
⏳ Waiting for 3 seconds...
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-06-34-084Z.png
⏳ Waiting for 3 seconds...
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-06-53-846Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-07-28-132Z.png
⏳ Waiting for 3 seconds...
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-08-40-068Z.png
⏳ Waiting for 3 seconds...
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-09-13-532Z.png
✓ passed
↳ 8,858 tokens (≈ $0.03)
● Watch a YouTube video and change playback settings
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-09-26-438Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-10-21-390Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-10-48-267Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-11-14-241Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-12-27-178Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-12-59-311Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-13-20-650Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-14-12-871Z.png
✓ passed
↳ 8,958 tokens (≈ $0.03)
● Visit a YouTube channel and verify latest content
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-14-39-006Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-14-43-280Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-14-47-961Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-15-05-967Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-15-27-252Z.png
Screenshot saved to: /home/maximouton/shortest/shortest/.shortest/screenshots/screenshot-2025-02-03T21-15-36-646Z.png
✓ passed
↳ 4,900 tokens (≈ $0.02)
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Tests 1 failed | 4 passed (5)
Duration 1115.08s
Started at 10:00:03 PM
Tokens 33,406 tokens (≈ $0.11)`

But after the test I received a message from anthropic telling me my 6$ credits were already gone (it is the second time I run this test).

Here is the capture from anthropic site:

And a part of the logs:

I have more than 50 calls in the logs for the corresponding test.

Can someone explain me when happened?

[gladyshcodes]

Sorry for the inconvenience—this is definitely a bug

It looks like we have two issues here:

1. Incorrect token usage reporting
2. Too many actions required to complete the test. Could there be a network issue? A slow connection might trigger excessive wait actions, followed by a series of screenshots

A video recording of a test would be super helpful in diagnosing and fixing this quickly

Worth mentioning that we’re working on performance and cost improvements, but your case seems like an edge scenario

[gladyshcodes]

For usage, I've created issue #315. Additionally, to simplify recording the test execution process, I've started discussion #314—which will likely be moved to issues and addressed soon

The core issue will be investigated once you provide more input. Please feel free to create an issue for it

[rmarescu]

I've used https://github.com/anti-work/shortest-example referenced on our bug report page to create a sample repo to reproduce the issue: https://github.com/rmarescu/shortest-318

My attempt used 217,843 tokens in, and 2,722 tokens out, for a cost of $0.69. I've updated #315 with all the info needed to reproduce it (including usage & cost screenshots).

Too many actions required to complete the test. Could there be a network issue? A slow connection might trigger excessive wait actions, followed by a series of screenshots

@maximouton I got about half of your usage for the same tests, so there could be something that can be further investigated.
I'd like to look more into why your execution used double the number of tokens. For that, it would be best if you could use the bug report form, similar to #315.
I know these runs are not free. Happy to compensate for the AI usage cost to run these tests (and the ones that you already executed).