Writing the test is only the beginning. Making the whole team trust is the real work.
Photo by Park Troopers on Unsplash
A Quick Note Before We Start
Hellooooooo… everyone. Last month, I had the opportunity to speak at a QA & Testers meetup at Thoughtworks Bangkok about our mobile E2E testing journey at Jitta.
The talk was called “Test to Ship, Not to Check. E2E Mobile Testing at Jitta”
Public speaking is just production testing — with an audience
I had not attended a meetup like this for quite a while since moving away from Bangkok, so getting the chance to come back was really fun.
I met old friends, caught up with people from the mobile development community, and talked with many QA engineers who had their own stories about frameworks, flaky tests, broken CI pipelines, and bugs that somehow disappear the moment someone starts debugging them.
It reminded me how much I missed being around people who understand why a test failure can ruin your mood before your first coffee.
The talk itself was around 40 minutes of lessons learned, mistakes, technical pain, and a surprising amount of emotional damage caused by E2E automation. People seemed to enjoy it, and several conversations afterward made me realize that almost every team has a flaky-test story.
- The tools may be different.
- The infrastructure may be different.
- The pain feels strangely familiar.
This article is based on that talk and one more disclaimer before we begin: English is not my first language, and I am using this article as a way to practice writing in English.
The experiences, opinions, and jokes are mine. AI only helped with the grammar. So if some sentences still sound a little unusual, that part is probably still me. 🙏 😅
Again… this is not a tutorial, a perfect architecture guide, or a list of universal best practices. It is simply a story about what we tried, what broke, and what we learned while trying to build mobile E2E tests that the team could actually trust.
More Tests Did Not Give Us More Confidence
Have you ever shipped a build and still felt nervous?
The build passed QA. The regression checklist was complete. Everything looked green but just before release, there was still that small voice saying:
“This should be fine… probably.”
I think many people who work on software releases know this feeling. The process says the build is ready, but emotionally, the team is still not completely convinced.
That feeling was one of the reasons we started taking E2E automation more seriously.
The problem was not that QA was doing a bad job. Manual regression was doing exactly what it was supposed to do. The problem was that our process still did not give the entire team enough confidence before shipping.
As we knows, the mobile releases are expensive to get wrong. On the web, some issues can be fixed and deployed quickly but mobile is different. There is App Store review, multiple application versions in the wild, and users who may not update immediately.
If a critical flow breaks, users do not care whether we use Appium, Robot Framework, XCUITest, Espresso, Maestro or some designed internal framework.
Our users care about one thing… “Why can’t I do what I came here to do?”
For a financial application, that could mean users cannot log in, set up an account, deposit money, or complete an investment journey.
At that point, it is no longer just a functional bug. It affects user trust and the confidence of everyone involved in the release.
Manual regression was necessary, but repetitive
Before automation, important flows had to be checked repeatedly across both iOS and Android.
- Login on iOS/Android.
- Deposit on iOS/Android.
- Onboarding on iOS/Android.
- Withdrawal on iOS/Android.
Then the product grew. More features meant more flows. More flows meant a longer regression checklist.
The difficult part was not always the complexity of the testing. It was the repetition. Repetitive checks consume time and attention.
They leave less energy for exploratory testing, unusual error states, UX issues, and risks that require human judgment.
This became our reason for investing in E2E automation. Not to replace QA. Not to remove manual testing but to let automation handle predictable repetition, so QA and Testers could spend more time on the kinds of problems that actually require thinking.
Then we tried to automate everything?
Once we decided to automate, we made the most obvious mistake:
“Why don’t we automate everything?”
It sounded reasonable at the time. More tests should mean more safety, right? ….. Not exactly.
We ended up with a test suite that took longer to run, required more maintenance, and failed in more interesting ways than we expected.
A larger suite can create:
- Longer feedback loops
- More test data dependencies
- More unstable selectors
- More environment-related failures
- More tests that nobody wants to investigate
It was like buying insurance for everything in your life. Eventually, the insurance costs more than the things you are trying to protect.
E2E automation can become the same.
If coverage grows faster than reliability, the maintenance cost arrives before the confidence does.
That was the lesson:
More tests do not automatically create more confidence.
You can have hundreds of tests and still be afraid to ship.
The important question is not only:
“How many tests do we have?”
It is:
“Does the team trust the result?”
A smaller set of stable flows can be more valuable than a huge suite that fails randomly and gets ignored.
I like to describe it this way:
Flows your team trusts are more valuable than tests your team ignores.
Photo by New York Said on Unsplash
We changed the question
Instead of asking:
“What can we automate?”
We started asking:
“What can we not afford to break?”
That question immediately made the scope smaller and the priority clearer.
- Can signup break?… No.
- Can login break?… No.
- Can a critical deposit or investment journey break? … Definitely not!
but can we survive a small visual inconsistency in a secondary settings screen?
Probably, at least while deciding what to automate first. That does not mean secondary features are unimportant.
It means E2E automation should begin where failure would hurt users and the business immediately.
So we started with happy paths but “happy path” does not mean an easy path. It means the main journey that users expect to complete successfully.
For example:
Signup -> Login -> Deposit -> Invest -> Portfolio
If one of these steps breaks, users feel it immediately. The “happy” part does not mean the path is easy. It means everyone is happier when it still works.
Our strategy became simple:
- Choose critical flows or important business flows.
- Make them stable.
- Run them consistently.
- Expand only after the team trusts what already exists.
The order matters. If you expand before stabilizing, you are not scaling confidence. You are scaling instability.
We tried that already. It hurt. ❤️🩹
The Test Was Not Always the Problem
We stopped trusting the test suite?
At one point, test failures became frequent enough that rerunning became a habit.
A test failed -> We reran it… again
If it passed, great. If it failed again, we reran it one more time.
This sounds ridiculous and in reality, it can happen gradually. The team starts seeing red builds that do not always mean the product is broken.
Then someone says:
“Probably flaky. Try rerunning it.”
Eventually, “probably flaky” becomes the default explanation and that is where trust starts disappearing.
Because when red does not reliably mean broken, green does not reliably mean safe either.
The team starts wondering:
“Did it pass because the product is working, or because the environment was lucky this time?”
At that point, the test suite is no longer a safety net. It becomes noise.
E2E failures are like an iceberg
The visible result is simple: TEST FAILED!
But the actual cause may be hiding somewhere below the surface:
- Timing
- Animation
- Asynchronous state
- Test data
- Environment instability
- Network behavior
- Brittle selectors
- Poor test design
- Device or simulator problems
Sometimes the product is broken. Sometimes the test script is broken. Sometimes neither is broken, but the environment is having a bad day.
One of the hardest parts of E2E testing is distinguishing between:
- A product bug
- A test script bug
- A test data issue
- An environment issue
- A test infrastructure issue
They may all produce the same red result, but they require completely different fixes.
- If the product is broken, fix the product.
- If the test is broken, fix the test.
- If the data is wrong, fix the data contract.
- If the environment is unstable, adding more sleep probably will not save you.
Sometimes the infrastructure is flaky
As you know, an E2E test does not run in isolation. It depends on a small universe of systems working together. A typical mobile E2E flow may require:
- A test-user creation API
- Predictable account state
- Staging services
- Authentication
- Network connectivity
- Feature flags
- External services
- Security or traffic-protection layers
- A device or simulator
- Stable backend response times
Any one of these can cause a failure.
For example, we may have an API that creates a test user.
That sounds great but if the generated user does not always have the same state, the test becomes unpredictable.
One run gives the expected account. The next run gives slightly different data. The test expects one thing and receives another. The report says the E2E flow failed.
Technically true but not especially useful.
This is not only an automation problem. It is a test data contract problem and if test data is unpredictable, every test that depends on it inherits that unpredictability.
Staging environments have personality
Staging is another source of entertainment.
Sometimes an API responds quickly. Sometimes it responds slowly.
Sometimes it times out. Sometimes it returns an error.
Sometimes the request appears to disappear into a black hole and comes back only after the test has already given up on life.
Then the test report says the flow failed and everybody looks at the automation but sometimes the test is fine.
The staging environment is simply having a character-development moment.
Security and traffic-protection systems can also create surprising failures. An automated test may send requests faster than a normal user.
A protection layer may decide that this behavior looks suspicious but from a security perspective, that makes sense.
From the test’s perspective, the situation feels more like:
“Excuse me, I am a test. I am not attacking you.”
The result is still a failure. Today the flow passes. Tomorrow it gets blocked. The next day it passes again.
Then everything gets grouped under the convenient label of “flaky test” but sometimes the script is not flaky. The infrastructure around it is.
Mobile applications are asynchronous systems
Timing is another major source of unstable E2E tests and mobile application may be handling several things at once:
- Screen transitions
- Animations
- API calls
- Loading indicators
- Background work
- Button state changes
- Data rendering
- Modal presentation
- and more…
Meanwhile, the test executes steps sequentially and often very quickly.
If the test moves faster than the application can respond, it may fail — not because the feature is broken, but because the test did not synchronize correctly with the application.
The easiest response is often:
Sleep for 5 seconds # GGEZ Right?Sleep feels useful because it can make the failure disappear temporarily but sleep is not a synchronization strategy.
It is hope written in code.
The better rule is “Do not wait for time. Wait for state”.
- Wait until the loading indicator disappears.
- Wait until the button is enabled.
- Wait until the expected element exists.
- Wait until the screen is actually ready.
The goal is not to wait longer. The goal is to wait for the correct condition.
Flaky tests change team behavior
A flaky test is not just annoying. It teaches people that failures can be ignored.
I often compare it to an alarm clock that rings at the wrong time every day.
- At first -> you react immediately.
- Later -> after enough false alarms, your brain learns not to care.
The same thing happens with a test suite.
- At first -> Red means investigate.
- Later -> Red means rerun.
- Eventually -> Red means nothing.
That is why flaky tests are so damaging. They do not only fail to catch bugs. They damage the credibility of the entire testing system.
Credibility is slow to build and very easy to lose.
A green build should mean something. A red build should trigger action.
If neither is true, the test suite is not providing confidence. It is only producing notifications.
Building a Test Suite the Team Can Trust
Reliability comes before coverage
Once we understood that the real problem was trust, our priorities changed.
We stopped celebrating test count by itself. Adding another test was no longer automatically considered progress.
A new test creates value only when it is:
- Stable
- Understandable
- Maintainable
- Run consistently
- Useful when it fails
This meant that flaky tests had to be treated as real engineering problems.
- Not something to ignore.
- Not something to rerun forever.
- Not something to hide behind a retry count of five and hope nobody notices.
Retries can sometimes help with temporary infrastructure instability, but they should not become a way to hide unreliable behavior.
A retry is a recovery mechanism. It is not a root-cause fix.
Classify failures instead of calling everything flaky
One practical improvement is to classify failures more clearly. Instead of putting everything into one bucket called “E2E failed,” separate failures into categories such as:
- Product failure
- Test script failure
- Test data failure
- Environment failure
- Network or security-layer failure
- Device or simulator failure
- Timeout or performance issue
This may sound like simple reporting, but it changes how the team reacts.
When every red result looks the same, people waste time investigating the wrong thing. When failure types are clearer, ownership and next actions become clearer too.
- A product bug goes to the product code.
- A selector problem goes to the test code or application testability.
- A test data problem goes to whoever owns the test data setup.
- An unstable staging service needs a different conversation entirely.
Classification turns random frustration into actionable information.
Make testability part of application development
Reliable E2E testing is not only the responsibility of the person writing the test.
The application itself has to be testable.
- Stable selectors matter.
- Accessibility identifiers matter.
- Predictable navigation matters.
- Clear loading states matter.
If the test identifies a button using visible text, a copy change can break it.
If it depends on a fragile UI hierarchy, a refactor can break it.
If the app does not expose a clear state for “ready,” the test may have to guess.
That is why E2E should be a team effort.
- QA understands the user journey and the risk.
- Developers help make the application testable.
- Infrastructure teams help provide reliable environments and data.
Failures are investigated together because the same red result can come from many different layers.
Without clear ownership, E2E rarely dies in a dramatic explosion. It dies slowly and a few tests are disabled.
Some failures are ignored. The suite runs less often.
One day, nobody remembers the last time it provided useful feedback.
Run it consistently
A test that runs only when someone remembers is not really a safety net. It is a suggestion.
A team may not have full CI automation on day one, and that is fine and hybrid process is still better than nothing.
QA may trigger the suite at agreed points. Developers may run critical flows before important merges. The team may run the suite before release.
The important thing is consistency.
Once the suite becomes part of CI, nobody has to remember to run it manually. It runs automatically. Feedback arrives earlier.
And earlier failures are usually cheaper to investigate than failures found after release.
Automation makes the test repeatable. Consistency makes it part of the engineering process.
E2E does not replace manual regression
One question I hear often is:
“If we have E2E automation, do we still need manual regression?” Yes!
E2E is useful for repeatedly checking critical business journeys but it is not good at everything. Humans are still better at evaluating:
- Visual details
- Device-specific behavior
- Unexpected error states
- Usability
- UX quality
- Exploratory scenarios
- Things that technically pass but still feel wrong
E2E should reduce repetitive manual checks. It should not remove the human judgment that makes QA valuable.
The goal is not fewer humans. The goal is fewer repetitive tasks for humans that gives QA more time to explore risks that automation does not understand.
Confidence is shared
The real outcome of E2E automation is not a pass-rate dashboard. It is shared confidence.
- Developers gain confidence that a new change did not break an existing critical flow.
- QA gains confidence that key journeys still work before release.
- Product gains confidence that important user journeys remain available.
- The business gains confidence in release readiness.
Confidence is not only a QA outcome. It is a team outcome.
That is why the title of my talk was Test to Ship, Not to Check.
The goal is not simply to confirm that a button can be tapped. The goal is to help everyone involved feel confident enough to ship.
Photo by Vlad Hilitanu on Unsplash
What changed for us
The most important change was not the number of tests. It was the workflow around them.
Before:
- Critical flows were manually repeated every release
- Feedback arrived late
- Some test results were difficult to trust
- The team could still feel nervous even after testing was complete
After improving the approach:
- Critical flows were checked more consistently
- Failures appeared earlier
- A green result started to mean something
- QA had more time for exploratory and risk-based testing
- Release confidence became more visible across the team
Automation handled more of the repetition. People had more room for judgment.
That, to me, is the real value of E2E testing.
What I would remember
If I had to reduce the whole experience to three lessons, they would be these.
1. Start with what you cannot afford to break
Do not begin with whatever is easiest to automate. Begin with the journeys that matter most to users and the business.
2. Earn trust before expanding coverage
- Make a small suite stable.
- Run it consistently.
- Let the team learn to trust it.
- Then expand.
Do not grow coverage faster than reliability.
3. Treat E2E as a team sport
QA cannot build a reliable E2E system alone.
Developers, QA, infrastructure, and product context all contribute to whether the result can be trusted.
A good E2E suite is not just a collection of scripts. It is a safety net maintained by the team.
Final Thought
At the beginning, we asked:
“How many tests do we have?”
It was easy to answer.
It fitted nicely in a spreadsheet.
It could even become a KPI.
But after dealing with flaky tests, unreliable environments, timing problems, and confusing failures, we started asking a different question:
“Can we trust what we are about to ship?”
That question is harder to measure but it is the better question.
More tests do not automatically create more confidence. Sometimes they create more failures, more reruns, and more notifications that everyone learns to ignore.
The tests that matter are the ones the team believes.
A green build should mean something. A red build should trigger action.
And when neither is true, the problem is no longer only testing.
It is trust. 👊
If you enjoyed it, hit Like — not Report. 😆
Thank you everyone, for sticking with me until the end.
I hope you all have fun writing code.
Be Lazy & Happy Coding! 🤤🖥️🤗