One of the main contributors of this project[0], was the core contributor (creator?) of Puppeteer[1], but then I guess left Google to join Microsoft and work on this[2][3].
I wonder what the story is there. Why wouldn't MS just have him continue to work on Puppeteer? They're both open source, so there's not much point in "owning" their own clone of it.
What do people thing of playwright vs cypress? I've been considering using playwright instead as it supports more browsers and I feel like it's easier to do production monitoring (by putting it in a aws lambda or using checkly)
My main issue with Cypress was the fact that it required a shift in mental model, i.e. queuing commands rather than executing them, thenables that aren't actually promises, running as client side JS as opposed to in a controller script. There were a ton of sharp edges and required a lot of education to adopt.
Meanwhile Playwright's API is just promises. Everybody knows how to compose promises.
The Cypress approach does in theory completely eliminate latency-induced flakiness, but in my experience, Playwright/Puppeteer's usage of bi-directional real-time protocols to communicate with the browser makes latency low enough that latency-induced flakiness is no longer a practical concern, especially when paired with the ability to selectively run test logic in client-side JS in the page with 0 latency.
Selenium did suffer from latency-induced flakes all the time due to its slow uni-directional request/response model. I personally believe the Cypress model is an over-correction for Selenium PTSD, and isn't making as good a set of tradeoffs compared to Playwright/Puppeteer.
Cypress was such a pain that I simply wasn't writing or running tests just to avoid it. It was incredibly slow, had major bugs that are till open after multiple years, and worst of all, in headless mode it was passing tests that should have been failing (and were failing in headed mode). The only reason I discovered it was because headless mode seemed to run unusually fast.
I switched to Playwright 6 months ago - I'm much happier now and the switch also allowed me to delete 3/4 of the helper code that I needed before.
On my team we evaluated Cypress and Playwright and landed on Playwright. Some features it has that Cypress didn’t was Safari support, support for cross domain tests, and support for tests that need to open multiple browser windows (for testing collaborative editing).
Cypress was terrible choice for us where flow in the system is managed by multiple users - as cypress can work only on single open window/tab, you had to effectively duplicate, triple etc. each test. Playwright also allows trivially control/crosscheck backend/whatever as you're in nodejs context, not browser context.
We recently picked playwright for a project because we could do more with the "mouse". With playwight you can click coordinates on the page. I did not find an easy way to do this using cypress. Aside from that, playwright seems to be so much faster.
Anyone have experience with Playwright compared to Selenium? I have a fairly large test suite and Selenium produces constant false positive errors, typically due to various timeouts that seem fundamentally unsolvable when running it from .NET. It's just very finicky.
I don't know if it's Selenium specifically or some problem with the .NET binding, but I figure Microsoft must have better .NET integration so it will at least eliminate that possible source of problems.
I'm not sure if any of these are pertinent to your tests, but these are the issues I see most often that cause flaky tests:
- Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this.
- Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test.
- No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage.
We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/
While I have not used Playwright (but have a lot of experience with Se), I would say the code style is refreshing:
// Expect an element "to be visible".
await expect(page.locator('text=Learn more').first()).toBeVisible();
Writing await for every action makes the timeout of the action seem more explicitly declared. There seems to be more granular control of timeouts as well https://playwright.dev/docs/test-timeouts
> I don't know if it's Selenium specifically or some problem with the .NET binding
If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it because the code execution is quicker than the page.
Large Se/Webdriver suites are often a PIA. I find it's nice to write them with Python or Ruby so they can be debugged interactively with the an interactive shell.
I have had similar issues with selenium via other languages too - it is generally pretty flaky. E.g. saying a button or some other element doesn't exist when it clearly does.
With great care and effort you can make your tests reliable (especially if you are happy to allow a "best of 3" type test strategy to allow for 1 flake and 2 passes) though. Prodigious use of the wait (i.e. stdlib polling) primitives seems to give you the most bang for your buck.
I am note sure if this is just the nature of web automation, or if selenium is just crap? My gut is to say it is selenium's fault since we never get the same issues when using javascript in the DOM or in an extension)...maybe it is the browser APIs I guess? U have no idea but if this playwright is any better than that would be superb.
On paper Playwright should be a LOT better - it's taken a similar approach to Cypress, where everything is designed around the need to reduce flaky tests.
I think it's possible to write tests in selinium which are time-independant... Eg. "Wait for element #foo to exist".
You can also give the browser a virtual clock so that you can use time based timeouts and give every test a timeout of 3 hours, but those 3 hours only take milliseconds in real time. That approach gets CPU expensive if your site has any background polling scripts or animation, because obviously the animation will end up animating a lot during the test!
I tried selenium then playwright for a .Net project, selenium wasn't easy to work with. Playwright was good but for some reason which I don't recall exactly (could have been because it had to redownload chromium everytime we deployed). I ended up switching to puppeteer and I ended up very happy with it.
I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev
It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it:
1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability
2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress.
3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box.
4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc...
5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful.
My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.
How hard is it for a site (server side or via JavaScript on the page) to tell that it is being accessed via a browser that is being automated with Playwright?
I've seen some sites that behave differently when the browser is being automated. E.g., if I access fanfiction.net from a browser being automated with Selenium it gets stuck in an endless Cloudflare CAPTCHA loop. Accordingly I've come to prefer automation methods that are less revealing to the site.
Playwright is great, especially if you are dealing with test cases that span multiple domains/contexts. I had to test some user flows which involved logging into two apps, each with three different users to perform and validate various actions. Playwright's context switching made it a breeze. Also, it offers a nice separation of browser automation and test runner API, so it can be used outside of E2E testing as well.
I’m working on a project that provides remote browsers, running on VMs/containers, capable of running Playwright tests (and Puppeteer scripts): https://headlesstesting.com/
We’ve seen a consistent growth of interest in people wanting to use Playwright for browser automation (and testing).
It's very simple to use either, there are loads of example implementations on GitHub.
I used one based on docker, and the bottleneck was actually sending the html, css you want to print (if it's not already served over http). I used a shared docker volume to write to from one process (python) and read from another (the node pupetter).
It all comes down to, load html, wait to load, save to pdf. Very simple, fast, and reliable. More so than weasyprint for example.
Playwright is a great tool. I was able to create a proof-of-concept stock screening tool using automation & screenshots of HTML elements to help me get swing trading ideas each morning/night when the market closed. It's a .NET Core console app using a CLI library called spectre.console based on rich(python) and playwright as the workhorse.
There's so much potential to use playwright in CI/CD with GitHub Actions cron jobs. Really enjoying it so far.
The only thing I wish we had was remote browser access - so I could run my tests on a VM (like within a docker image) and use a browser on the host.
We use TestCafe at work for this purpose. I personally hate TestCafe as it's is an absurd unfocused mess of a browser remote, but it lets me control my browser by navigating to a URL which no other browser remote system does.
That is the JavaScript spread operator. It takes all of the elements of an object or an array and adds them to another. In this case, the code posted is merging the iphone 13 pro device settings object into an object literal
Yes definitely, there's lots of products in the QA space trying to tackle the problem you're describing. I'm a co-founder of a no-code product in the space (https://reflect.run). Being no-code has the advantage of enabling all QA testers to build test automation, regardless of coding experience.
Is there an open source web testing tool which also integrates a dashboard, keeps track of test runs, creates reports, something that I can just install on a vm and run to test a web app?
Browser automation will occur by executing events in the DOM or by calling properties of the page/window. It’s all JavaScript designed for user interaction executed by a bot.
The one event that cannot be automated is cursor movement/position. Put a check into your event handlers that check that the cursor is actually over the event target.
Tell freemium users what is the acceptable rate for requests per second. Publish the allowable rate on the website. Ban freemium user IPs that exceed the allowable rate. This can be done using a proxy.
100 requests/second isn't that much, especially if you're fronting your website with Cloudflare. Do you have some unauthenticated endpoint(s) that eat up a ton of server CPU?
syspec|4 years ago
One of the main contributors of this project[0], was the core contributor (creator?) of Puppeteer[1], but then I guess left Google to join Microsoft and work on this[2][3].
[0] - https://github.com/aslushnikov
[1] - https://github.com/puppeteer/puppeteer/
[2] - https://github.com/microsoft/playwright/graphs/contributors
[3] - https://github.com/microsoft/playwright/graphs/contributors
vlunkr|4 years ago
pawelbylina|4 years ago
[deleted]
johnnypangs|4 years ago
- Cypress: https://www.cypress.io
- Playwright aws lambda: https://github.com/PauloGoncalvesBH/running-playwright-on-aw...
- Checkly: https://www.checklyhq.com
lewisl9029|4 years ago
Meanwhile Playwright's API is just promises. Everybody knows how to compose promises.
The Cypress approach does in theory completely eliminate latency-induced flakiness, but in my experience, Playwright/Puppeteer's usage of bi-directional real-time protocols to communicate with the browser makes latency low enough that latency-induced flakiness is no longer a practical concern, especially when paired with the ability to selectively run test logic in client-side JS in the page with 0 latency.
Selenium did suffer from latency-induced flakes all the time due to its slow uni-directional request/response model. I personally believe the Cypress model is an over-correction for Selenium PTSD, and isn't making as good a set of tradeoffs compared to Playwright/Puppeteer.
tnolet|4 years ago
Having said that, I would love to support Cypress if I had infinite time and focus.
Side note: Selenium is not on the menu, even with its large install base.
We are aiming for where the puck is going and it’s going to Playwright.
Full disclaimer: I’m CTO at Checkly.
nimbix|4 years ago
I switched to Playwright 6 months ago - I'm much happier now and the switch also allowed me to delete 3/4 of the helper code that I needed before.
cebert|4 years ago
dvngnt_|4 years ago
https://docs.cypress.io/guides/references/trade-offs#Permane...
CSDude|4 years ago
- Playwright is more lightweight. Can be good or bad on what you expect. But I definitely prefer Playwright.
mirekrusin|4 years ago
machiaweliczny|4 years ago
twohaibei|4 years ago
dgb23|4 years ago
cornedor|4 years ago
simonw|4 years ago
eatonphil|4 years ago
naasking|4 years ago
I don't know if it's Selenium specifically or some problem with the .NET binding, but I figure Microsoft must have better .NET integration so it will at least eliminate that possible source of problems.
tmcneal|4 years ago
- Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this.
- Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test.
- No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage.
We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/
i_like_apis|4 years ago
> I don't know if it's Selenium specifically or some problem with the .NET binding
If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it because the code execution is quicker than the page.
Large Se/Webdriver suites are often a PIA. I find it's nice to write them with Python or Ruby so they can be debugged interactively with the an interactive shell.
mattlondon|4 years ago
With great care and effort you can make your tests reliable (especially if you are happy to allow a "best of 3" type test strategy to allow for 1 flake and 2 passes) though. Prodigious use of the wait (i.e. stdlib polling) primitives seems to give you the most bang for your buck.
I am note sure if this is just the nature of web automation, or if selenium is just crap? My gut is to say it is selenium's fault since we never get the same issues when using javascript in the DOM or in an extension)...maybe it is the browser APIs I guess? U have no idea but if this playwright is any better than that would be superb.
simonw|4 years ago
In Playwright that manifests itself as the "auto-wait" feature: https://playwright.dev/docs/actionability
You can do this kind of thing with Selenium too but it wasn't designed in from the very start of that project.
londons_explore|4 years ago
You can also give the browser a virtual clock so that you can use time based timeouts and give every test a timeout of 3 hours, but those 3 hours only take milliseconds in real time. That approach gets CPU expensive if your site has any background polling scripts or animation, because obviously the animation will end up animating a lot during the test!
Bilal_io|4 years ago
forgotmyoldacc|4 years ago
machiaweliczny|4 years ago
vthommeret|4 years ago
I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it:
1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability
2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress.
3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box. 4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc...
5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful.
My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.
unknown|4 years ago
[deleted]
hoten|4 years ago
tzs|4 years ago
I've seen some sites that behave differently when the browser is being automated. E.g., if I access fanfiction.net from a browser being automated with Selenium it gets stuck in an endless Cloudflare CAPTCHA loop. Accordingly I've come to prefer automation methods that are less revealing to the site.
Karupan|4 years ago
defied|4 years ago
We’ve seen a consistent growth of interest in people wanting to use Playwright for browser automation (and testing).
ithrow|4 years ago
Pros of Playwright/Puppeteer:
Reuse existing HTML/CSS knowledge
Cons: Requires an external service or shelling out to an external process
Pros of using a pdf lib:
Probably better performance, simpler architecture by being in-process.
Cons: Ad-hoc language for designing the PDF.
rudasn|4 years ago
I used one based on docker, and the bottleneck was actually sending the html, css you want to print (if it's not already served over http). I used a shared docker volume to write to from one process (python) and read from another (the node pupetter).
It all comes down to, load html, wait to load, save to pdf. Very simple, fast, and reliable. More so than weasyprint for example.
unknown|4 years ago
[deleted]
thenerdhead|4 years ago
There's so much potential to use playwright in CI/CD with GitHub Actions cron jobs. Really enjoying it so far.
apatheticonion|4 years ago
We use TestCafe at work for this purpose. I personally hate TestCafe as it's is an absurd unfocused mess of a browser remote, but it lets me control my browser by navigating to a URL which no other browser remote system does.
kesor|4 years ago
tomcam|4 years ago
tycoon177|4 years ago
shp0ngle|4 years ago
https://github.com/microsoft/playwright/blob/0d277fa589e9508...
edit: ah puppeteer was a Google project. I forgot
jwithington|4 years ago
tmcneal|4 years ago
mesadb|4 years ago
Would love to help any of your testing needs
robstain|4 years ago
https://botcity.dev
petermd|4 years ago
[deleted]
cfrover|4 years ago
kundi|4 years ago
mxschmitt|4 years ago
tw20212021|4 years ago
hoten|4 years ago
https://github.com/GoogleChrome/lighthouse-ci
dzhiurgis|4 years ago
brimstedt|4 years ago
Br
headlessvictim2|4 years ago
The freemium service provides access to compute-heavy machine learning models running on GPUs.
Hackers blast 50-100 requests in the same second, which clog the servers and block legitimate users.
We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode" to thwart attacks, but the hackers still break through.
We don't require accounts, but could impose account requirements if this helps.
Any suggestions?
austincheney|4 years ago
The one event that cannot be automated is cursor movement/position. Put a check into your event handlers that check that the cursor is actually over the event target.
slig|4 years ago
1vuio0pswjnm7|4 years ago
cmeacham98|4 years ago
synergy20|4 years ago
forgotmyoldacc|4 years ago
rabi_molar|4 years ago
machiaweliczny|4 years ago
unknown|4 years ago
[deleted]
unknown|4 years ago
[deleted]
slimeboty|4 years ago
[deleted]