Selenium 2/WebDriver next alpha has been released recetnly. Although it's new kid on the market of open source web testing tools you can't argue that it's the most compelling web testing tool around at the moment.
For writing web automated tests I have been using first Selenium for over a year. And I am pretty satisfied. But there are several things those annoyed me. Above all is Selenium-RC server that has to be running befor execute your test. Actually it is a relatively slow and sometimes fickle process. Good news is that now within WebDriver this is no longer required. You just create new driver instance and it is ready to use.
So a month ago I started to refactore my tests removing Selenium 1 in favore of WebDriver. But had got the next problem. My tetst are written in python and I am using Python bindings for Selenium. A lot of the test are using the get_eval methode to execute javascript code. WebDriver has the execute_script methode to do the same and it does but within the "Unknown result type - 0" exception. Actually it means that it is not possible to return something from javascript back to selenium. Fortunatelly we were able to fix this issue. And espetial thanks to Vladimir Lavrechuk, a college of mine, for his huge contribution to solving the problem. Actually the problem was already reported to the developers and there is an issue 641. I have posted my solution there.
Now I continue to develope my tests using WebDriver and in my next posts I will try to share my experience.
Saturday, December 25, 2010
Wednesday, October 20, 2010
Hudson-Ant+Groovy: Fitnesse integration
Earlier I wrote that I started to use Hudson instead of CruiseControl for building projects and running our acceptance tests. After having it around for a week, I’ve tentatively switch our builds to use it. Since my CruiseControl's builds are mainly ant scripts so it was pretty easy. I just moved them to Hudson.
You know that Ant is a platform-independent scripting tool that lets you construct your build scripts in much the same fashion as the "make" tool in C or C++. You can use a large number of built-in tasks in Ant without any customization: scp, sshexec, copy, waitfor, etc.
Such built-in tasks were the main reason why I made the decision at the time to use ant. And it was going well until the scripts become big and complex. Simple builds are easy to express as a series of tasks and dependencies. For such builds the facilities of ant work well. But more complex builds require conditional logic, and that requires more general programming language constructs. So I have been extremely annoyed by trying to use ant and it's XML config files.
The reason for that is XML's verbosity. It's not too bad for data files but the moment you try writing reasonably complex code the amount of typing you have to do quickly starts to becoming unusable for any real project. Once the scripts get complex enough having to do it in XML becomes really annoying.
So I started tinkering around and discovered Gant that is a tool for scripting Ant tasks using Groovy instead of XML to specify the logic. It was exactly that I had been looking for and it is supported by Hudson very well. So I completely replaced the build process removing all the Ant in favor of Groovy. It was not so hard as I can reuse my favorite Ant tasks for declarative operations but use a real scripting language for procedural logic. And it is very handy to do things like loops, subroutines and list collection in my build files. And In order not to sound proofless, I want to give you a real example.
One of my favorite features of Hudson is the ability to watch the console output of a build in progress, like this one:
I've already mentioned in my posts that we use FitNesse for Acceptance testing. And it was a task to integrate FitNesse within Hudson. Actually there is a plugin that facilitate execution and reporting on Fitnesse. But I don't like it as there is no any progress reporting for running test suites and results is not integrated in Hudson. So I decided to use groovy for the integration. And here is a result:
Now I am having the verbose progress reporting for running tests. As soon as there are red tests I see them in the console and optionally can get messaging by e-mail. The results are published in Hudson's page so there is a really cool build history graph that shows both the time it took to run tests and whether they passed or failed — makes it easy to see our build trends on the team.
Here is a groovy script that I use for it:
You know that Ant is a platform-independent scripting tool that lets you construct your build scripts in much the same fashion as the "make" tool in C or C++. You can use a large number of built-in tasks in Ant without any customization: scp, sshexec, copy, waitfor, etc.
Such built-in tasks were the main reason why I made the decision at the time to use ant. And it was going well until the scripts become big and complex. Simple builds are easy to express as a series of tasks and dependencies. For such builds the facilities of ant work well. But more complex builds require conditional logic, and that requires more general programming language constructs. So I have been extremely annoyed by trying to use ant and it's XML config files.
The reason for that is XML's verbosity. It's not too bad for data files but the moment you try writing reasonably complex code the amount of typing you have to do quickly starts to becoming unusable for any real project. Once the scripts get complex enough having to do it in XML becomes really annoying.
So I started tinkering around and discovered Gant that is a tool for scripting Ant tasks using Groovy instead of XML to specify the logic. It was exactly that I had been looking for and it is supported by Hudson very well. So I completely replaced the build process removing all the Ant in favor of Groovy. It was not so hard as I can reuse my favorite Ant tasks for declarative operations but use a real scripting language for procedural logic. And it is very handy to do things like loops, subroutines and list collection in my build files. And In order not to sound proofless, I want to give you a real example.
One of my favorite features of Hudson is the ability to watch the console output of a build in progress, like this one:
I've already mentioned in my posts that we use FitNesse for Acceptance testing. And it was a task to integrate FitNesse within Hudson. Actually there is a plugin that facilitate execution and reporting on Fitnesse. But I don't like it as there is no any progress reporting for running test suites and results is not integrated in Hudson. So I decided to use groovy for the integration. And here is a result:
Now I am having the verbose progress reporting for running tests. As soon as there are red tests I see them in the console and optionally can get messaging by e-mail. The results are published in Hudson's page so there is a really cool build history graph that shows both the time it took to run tests and whether they passed or failed — makes it easy to see our build trends on the team.
Here is a groovy script that I use for it:
Friday, March 19, 2010
Fitnesse+SLIM+Selenium tandem
First time I met Finesse a couple years ago. That time my team was adopting Story Test Driven Development approach. We were using Fit Framework as the Wiki to write active specifications and run the acceptance tests. FitNesse has always had the ability to express tests as declarative tables. So we were trying to create a syntax for the application code that is readable by our analysts and business partners.
But actually the tests were not clear enough and maintaining the fixtures was hard and time consuming as the amount of code duplication was high for any standard. New tests required us writing more code and they were not descriptive enough to work as a live documentation. Because of the way Fit uses inheritance, its fixture code is cumbersome and hard to maintain. Then we started search for other tools that allow to test the software with lower costs and we neglected Fitnesse for some time.
Recently I was pointed out the relatively new Slim protocol for using Fitnesse tests with Systems Under Test. The primary benefit of Slim is reducing the footprint of the framework in the SUT, sharing a single implementation of the table interpreter, and a Scenario table that can be called from other tables. It seemed to be that Slim FitNesse holds more promise for less effort and I decided to try it.
By the way, my current project is a web application, so the biggest part of our acceptance tests are tests against UI. As you know user interface has traditionally been hard to integrate into an automated test process. So trying to solve this task just lately we've added Selenium RC to our toolbox. Frankly speeking I don't like Selenium IDE and I am an antagonist of a recording and playing back tests in a browser. So the idea was to use Fitnesse to create some kind a Domain Specific Language to write and execute Selenium tests. And I can say that it is succeeded. Now I really like how we are writing and maintaining our user stories, how they look and feel, and its low price.
We can write test according to zero emission principle. I mean it is possible do not write any fitnesse fixture to run tests against our web application. We develop in Python, so we use a python port of the fitnesse slim server - WaferSlim and Selenium Python Client Driver to control pages of our application. So if you have setup fitnesse, WaferSlim, and Selenium you could write the next story without any coding except fitnesse.
|import|
|selenium|
|script|selenium;|${SLIM_HOST}|4444|iexplore|!-http://google.com-!|
|run|
|open|!-http://google.com/ncr-!|
|type;|q|selenium server|
|click|name=btnG|
|allow_native_xpath|true|
|wait_for_page_to_load|10000|
|check not|get_text|id=resultStats|BLANK|
So, it is great! The only thing I had to do to run it is an adopting the 'selenium' class in the selenium.py module. I mean that the 'selenium' class is not ready for a while to be using within Fitnesse and Slim. First of all - it necessary to rename this class into 'Selenium' instead of 'selenium'. For some reason fitnesse tries to find exactly that case-sensitive name. Next - in the '__init__' method you have to convert the 'port' parameter, that is given by fitnesse as a string, into int - self.port = int(port). And the last - rename the 'start' method, for example with 'run' or so. It is necessary because the 'start' keyword has a special meaning for fitnesse. This is it.
Now just imagine that you need to do the same thing, I mean call google's search from different user stories. In this case you have to move this code into python and make a function to be called from the different places on fitnesse. But wait! Fitnesse offers other way. Just look.
!|scenario|google search|string|
|start|selenium;|${SLIM_HOST}|4444|iexplore|!-http://google.com-!|
|run|
|open|!-http://google.com/ncr-!|
|type;|q|@string|
|click|name=btnG|
|allow_native_xpath|true|
|wait_for_page_to_load|10000|
|check not|get_text|id=resultStats|BLANK|
|script|
|google search|selenium server|
|script|
|google search|fitnesse slim|
But actually the tests were not clear enough and maintaining the fixtures was hard and time consuming as the amount of code duplication was high for any standard. New tests required us writing more code and they were not descriptive enough to work as a live documentation. Because of the way Fit uses inheritance, its fixture code is cumbersome and hard to maintain. Then we started search for other tools that allow to test the software with lower costs and we neglected Fitnesse for some time.
Recently I was pointed out the relatively new Slim protocol for using Fitnesse tests with Systems Under Test. The primary benefit of Slim is reducing the footprint of the framework in the SUT, sharing a single implementation of the table interpreter, and a Scenario table that can be called from other tables. It seemed to be that Slim FitNesse holds more promise for less effort and I decided to try it.
By the way, my current project is a web application, so the biggest part of our acceptance tests are tests against UI. As you know user interface has traditionally been hard to integrate into an automated test process. So trying to solve this task just lately we've added Selenium RC to our toolbox. Frankly speeking I don't like Selenium IDE and I am an antagonist of a recording and playing back tests in a browser. So the idea was to use Fitnesse to create some kind a Domain Specific Language to write and execute Selenium tests. And I can say that it is succeeded. Now I really like how we are writing and maintaining our user stories, how they look and feel, and its low price.
We can write test according to zero emission principle. I mean it is possible do not write any fitnesse fixture to run tests against our web application. We develop in Python, so we use a python port of the fitnesse slim server - WaferSlim and Selenium Python Client Driver to control pages of our application. So if you have setup fitnesse, WaferSlim, and Selenium you could write the next story without any coding except fitnesse.
|import|
|selenium|
|script|selenium;|${SLIM_HOST}|4444|iexplore|!-http://google.com-!|
|run|
|open|!-http://google.com/ncr-!|
|type;|q|selenium server|
|click|name=btnG|
|allow_native_xpath|true|
|wait_for_page_to_load|10000|
|check not|get_text|id=resultStats|BLANK|
Now just imagine that you need to do the same thing, I mean call google's search from different user stories. In this case you have to move this code into python and make a function to be called from the different places on fitnesse. But wait! Fitnesse offers other way. Just look.
!|scenario|google search|string|
|start|selenium;|${SLIM_HOST}|4444|iexplore|!-http://google.com-!|
|run|
|open|!-http://google.com/ncr-!|
|type;|q|@string|
|click|name=btnG|
|allow_native_xpath|true|
|wait_for_page_to_load|10000|
|check not|get_text|id=resultStats|BLANK|
|script|
|google search|selenium server|
|script|
|google search|fitnesse slim|
I really like it! What about you? And one more great thing that I appreciate about fitnesse.
!|scenario|User is trying to search _ on google|string|
|google search|selenium server|
![ script
User is trying to search selenium server on google
]!
So, Fitnesse+SLIM+Selenium tandem is hightly recommended. Good lack in acceptance testing!
Tuesday, March 16, 2010
I have met Hudson!
Continuous Integration with automated tests execution is the main part of the dev process in my team. We used CruiseControl for a few years and it proved to be very helpful for us. We feel always in control and we make sure the quality stays at the top of priority all the time. But last time I started practicing to break the build up into smaller builds and chaining them together that let us identify problems earlier. It raised several problems:
- configuring new builds is a lot more work than it needs to be;
- multiple ant builds interfere with each other with multiple builds;
- multiple builds are some kind a mess on the dashboard;
- JUnit results processing - nice HTML, results trends, charts;
- The upstream and downstream projects;
Sunday, February 14, 2010
Integrating SLIM+Fitnesse with CruiseControl
On the current project we use SLIM framework within FitNesse for Acceptance testing. We rely quite heavily on the Fitnesse tests results to certify our builds. Hence it was important for us to automate running our SLIM tests and to properly publish the result in the cruise page.
I use the ant scripts to execute Fitnesse test runner from a command-line, parse the resulting XML and transform it using style sheet and finally write the results to the xml files in the cruise log directory. Сonsequently the CruiseControl dashboard with the results looks like here
To have such view each suite should have its own xml log in an appropriate format. I do this just running each suite separately. For this purpose I use the names fitnesse command to get a list of the suites and then run them using the suite Fitnesse command, so it is pretty easy to save the results in the different files. If I don't need to run the whole bunch of the suites (for example: only suites of a particular virtual machine) I use the suiteFilter parameter of the suite command.
As you see the 'exec fit suite' target is a procedure that runs fitnesse against the given suite and store it's output into the 'suite_result' property. After that there is a small trick. Unfortunately, the output contains not only test's result in the xml format, but also the fitnesse command output
As I already have said, the suiteFilter parameter of the suite command is used to run only particular suites. In this case fitnesse tries to execute each suite but if it does not fit for the given criterion it's tests do not start but we have the xml output as well. Such output does not have relativePageName tag, so in the 'save log as xml ' I just check it and write xml logs only for suites that were really ran.
Pay attention to the 'xslt' ant's task in the 'save log as xml'target. It is used here to write fitnesse test's results in the particular format by processing they via the XSLT.
That's it for today. Feel free to ask me questions.
I use the ant scripts to execute Fitnesse test runner from a command-line, parse the resulting XML and transform it using style sheet and finally write the results to the xml files in the cruise log directory. Сonsequently the CruiseControl dashboard with the results looks like here
To have such view each suite should have its own xml log in an appropriate format. I do this just running each suite separately. For this purpose I use the names fitnesse command to get a list of the suites and then run them using the suite Fitnesse command, so it is pretty easy to save the results in the different files. If I don't need to run the whole bunch of the suites (for example: only suites of a particular virtual machine) I use the suiteFilter parameter of the suite command.
As you see the 'exec fit suite' target is a procedure that runs fitnesse against the given suite and store it's output into the 'suite_result' property. After that there is a small trick. Unfortunately, the output contains not only test's result in the xml format, but also the fitnesse command output
FitNesse (v20100103) Started... port: 9125 root page: fitnesse.wiki.FileSystemPage at C:\FitNesse/FitNesseRoot logger: none authenticator: fitnesse.authentication.PromiscuousAuthenticator ... Exit-Code: 0So, it is necessary to filter it and here only strings between <testResults> and </testResults> are written to the logs.
As I already have said, the suiteFilter parameter of the suite command is used to run only particular suites. In this case fitnesse tries to execute each suite but if it does not fit for the given criterion it's tests do not start but we have the xml output as well. Such output does not have relativePageName tag, so in the 'save log as xml ' I just check it and write xml logs only for suites that were really ran.
Pay attention to the 'xslt' ant's task in the 'save log as xml'target. It is used here to write fitnesse test's results in the particular format by processing they via the XSLT.
That's it for today. Feel free to ask me questions.
Tuesday, February 9, 2010
Problems use runProgramInGuest in Windows 2008
I meet a problem when use runProgramInGuest method of vmrun to run Acceptance Tests for my product in Windows 2008 os on the ESXi 4.0.0 server. The problem is that I can't see any active window in the windows 2008 console and as a result it is hard to develop and support user interface acceptance tests. It happens because of a program invoked by RunProgramInGuest is running under session 0 that is non-interactive. It is made in order to better protect the system against security risks. But what if a session 0 service needs to interact with the user? For example : a backup program wants to inform the user that it cannot start an automated backup because the external USB hard drive is disconnected. For this purpose there is the Interactive Services Detection Services (UI0Detect). It can be invoked by the service that needs to interact with the user. UI0Detect in turn presents to the user a window telling him that a particular service needs some interaction. The user gets two choices : "Show me the message", or "Remind me in a few minutes".
Clicking "Show me the message" switches the display to the service which wanted to interact with the user. By default, the startup mode of this service is Manual. Just use command below in cmd to change this mode.
So this very gracefully solves my problem and I'm happy - my product will be tested in windows 2008.
Clicking "Show me the message" switches the display to the service which wanted to interact with the user. By default, the startup mode of this service is Manual. Just use command below in cmd to change this mode.
sc config UI0Detect start= auto
So this very gracefully solves my problem and I'm happy - my product will be tested in windows 2008.
Sunday, January 31, 2010
Replay on You do strive to create programmatically ...
That all sounds great in principle and I agree - it is certainly the way to go. But you know that sometimes "things needed for Acceptance Tests" might be too costly to create them programmatically. I mean it might slow down a development process. So I think we must take into account that someone might have eaten our "fruit" by then.
That all sounds great in principle and I agree - it is certainly the way to go. But you know that sometimes "things needed for Acceptance Tests" might be too costly to create them programmatically. I mean it might slow down a development process. So I think we must take into account that someone might have eaten our "fruit" by then.
Subscribe to:
Posts (Atom)