Testing: guidelines and best practices.

[aleeusgr]

Cardano Community should have guidelines and best practices for testing dApps and products.

The guiding questions are:

• At what stage of your project you introduce testing automation?
• Unit testing, Integration Testing, API, UI, smart contract business logic testing, penetration testing. What is missing?
• Developer performance, Smart Contract Optimization, Security. What are other goals of testing?
• does your team use Quviq, plutip, tasty, Quickcheck?
• What stack are you using? Do you use native testing tools in your stack?
• When you create transactions, do you do it on Backend or FrontEnd?
• How do you ensure correctness of Datum and Redeemer?
• Do you perform input type validation and what tools are you using?
• What black box testing tools and techniques do you use?
• What testing tools or features could be added that could make your life easier as a developer
• How are we testing the on-chain code?
• How existing tools can be integrated with each other?
• How do we facilitate development of the set of best practices for testing similar to Cardano Well-Architected Framework?

code and docs

[aleeusgr]

Tooling

TODO: add short descriptions. write a blog article.
Here is the list of libraries used so far:

https://github.com/tweag/plutus-libs/

https://github.com/mlabs-haskell/plutip

https://github.com/mlabs-haskell/plutus-simple-model

https://github.com/tweag/plutus-libs/tree/main/cooked-validators

https://github.com/Quviq?language=haskell

ply

https://docs.cardano.org/development-guidelines/node-tests

[ThomasArts]

Some input on how to test plutus contracts using a ContractModel. That is, use a QuickCheck model to automatically generate test cases for your plutus contract.

https://plutus-apps.readthedocs.io/en/latest/plutus/tutorials/contract-testing.html
https://plutus-apps.readthedocs.io/en/latest/plutus/tutorials/contract-models.html

Note that you do not necessary need to use the contract monad in plutus-apps, even though the tutorials do use this for their testing approach. Just ping Quviq if you want to get automatic test case generation in a different setting.

[aleeusgr]

Many Thanks.
I've heard many times about property based testing and formal verification in Cardano, it seems to be an important part of tooling for teams working in the ecosystem. However the learning curve is quite demanding, so it is likely that this remains not accessible to many teams.

The community doc for reference:
https://plutus-community.readthedocs.io/en/latest/#Plutus/Lectures/Cohort_03/Lecture_08/

https://www.cis.upenn.edu/~plclub/blog/2020-10-09-hs-to-coq/

[ThomasArts]

I would like to downplay the learning curve a bit. In our view, writing a contract property is far easier than writing a plutus contract. In fact, in a recent collaboration with Plank Ltd, we noticed that the Plank engineers got their first model running within a week and only thereafter got some questions on more detailed issues.

Sure, the Plank guys may be extraordinary clever, but possibly it's more a psychologic barrier than a real complexity in making a contract model.

[aleeusgr]

PPBL

Gimbalabs PBL presents a good framework to express best practices in development.

[aleeusgr]

MixAxiM, Ekival:
black box testing,
Input type validation,

Smart contract security concerns, cooked-validators
Pros and cons of (algorithmic) stablecoin implementations.

security | optimization |

languages and compilers:
plutus, plutarch, aiken.
one can write code in plutus, but use plutarch compiler to obtain smaller .cborhex, or both write transactions in plutarch and use its compiler for more efficiency.
Transaction: Datum, Redeemer, Core.

reference UTXO for the contract that handles the

Correctness of datum and redeemer.

Creating datum and redeemer: frontend, backend; testing for each.

[aleeusgr]

Interviews with community

[aleeusgr]

Fell-x27, Medusa wallet, adawallet.io:

We outsource testing. Our project passed closed beta phase with unit, integration and API tests, now we are in user testing phase for UI.

[aleeusgr]

Szist:

dapp contracts testing: ply, tasty as mostly unit tests, but
for actual integration testing doing tests on preview/preprod networks + context builder)
but for now most dapps will also need some tooling with that private chain - like ogmios/db-sync/etc.

TODO: format, reframe.

Yeah sure. So during testing you would multiple phases, right?:

1. unit testing contracts if there are any (e.g. stuff like tasty and property based testing with a context builder)
2. some integration tests - plutus has/had the playground and could emulate the blockchain transactions. This can be also used to estimate properly the exunit usages. But it didn't cover all edge-cases that you might want
3. ?

Now the 3rd is what I was aiming at. At this point you have some frontend/backend services that are interacting with the contracts, and in some way you need an environment, where you can test things.

I've seen multiple paths taken:

• jump straight to mainnet 😦
• let people test things on a testnet (preprod or preview) making the users QA
• if backend services do some load-testing on the public testnet
• Spin up a private testnet - some projects

There are a lot of issues with private testnets that are solved if you use a public one. Usually you need several components if you want to test a dapp:

• indexer (db-sync, kupo, carp)
• way to submit transactions (cardano-submit-api, ogmios)
• ?optionally some realtime connection (ouru, ogmios)
• wallet (to actually connect to the dapp) or a replacement library (e.g. lucid tries to be)

• there are some other components that you might need depending on use-case.

If you are on a private testnet, all this is a pain in the * to set up, so the easy way out is to use the public one 🤷‍♂️ . When you use a pubic one, usually you get support from blockfrost, koios, etc.

Circling around, you really want to test the not-so happy path. If you look at congestion issues handled by dripdropz droperatorz (https://dripdropz.medium.com/shrinking-the-interaction-window-bba8c39268c9), bridges tackling just now transaction chaining to get any reasonable throughput, utxo handling and blocking, retry logic for rollbacks and mempool access... it's clear a lot of unexpected things pop up when there is actual blockchain and not a playground. So doing proper testing on a testnet should be a must...

In an ideal case I think apart from the unit/playground integration tests, you might want to have some

• congestion and throughput testing - at least doing the math so we don't end up with contracts like heavy dexes (e.g. Sundae), which hog the blocks with large scripts and costly exunits.
• e2e tests properly - if frontend then some cypress with an emulated cip30 wallet and doing interactions with the app (haven't seen this though )
• load tests for apis (jmeter/artillery/etc.)
  Medium
  [10:17]
  All this for now without the security part. The key point from security standpoint are the contracts, that's where the real money needs to be safe. There for now, thanks to the design of Plutus smart contracts and the scope limited to a single transaction: thorough negative tests with a tx context builder for plutus validators should be the minimum. The challenge there is to have well documented invariants and assumptions. I think here might come into the picture the tool mentioned on the call by tweag.

example:

You have an nft marketplace, you want to make sure you never break your main flow:

1. # with wallet
2. Put up an nft up for sale
3. # with second wallet & buy the nft
4. Check that first wallet got the stuff.

Off course you don't want to do such tests on the mainnet. Alternative to automating this is to have manual QA, who would test always before release, that none of the main flows were broken. (pretty easy once you move past some trivial complexity and you have 15+ flows)
[14:09]
Also from my experience wallet integrations constantly break with every new release (different collateral handling, eternl introducing locked utxos, nami behaving differently if you have a collateral set vs when you don't, etc.)
[14:10]
So retesting dapps manually for 5+ wallets 10+ flows for every release can become quite costly down the line.

[aleeusgr]

Jingles: https://github.com/MartifyLabs/starter-next-ts-template

working e2e, cip30 wallet and doing interactions with the app
easily integration with plutus script with this documentation
for tests, help-needed;

plan to test from API to blockchain (i.e. check if something breaks due to network changes).

can we do end to end for every possible test cases?