What's Mago? I like to call Mago a desktop testing "initiative", rather than a framework. In fact, it is heavily based on LDTP, a desktop testing framework, written in C.
With automated desktop testing we name all the tests that runs directly against the user interface (UI), just like a normal user would do: a script will run and you will start to see buttons clicking, menus poping up and down and things happening, automagically.
Mago tries to add consistency to the way we write, run and report results of this kind of scripts.
Before starting with this tutorial, please, follow the instructions at the getting started guide.
TestSuites & TestCases
With Mago, one of the things that we are building is a library with "testable" applications. If the application you want to test is already in the mago library, writing tests for it is easier.
First, some testing terminology:
- A test suite is a collection of test cases; with a test case being an scenario you want to test in your application.
- We also need to be able to determine whether a test is successful or not in order to report pass or fail. The "knowledge" we have that let us do that is an "oracle"
In Mago a test suite consists of 2 files: a PYTHON file and a XML file.
The .py file contains the code of the test. The things you want to do with the application. The .xml file is the description and arguments of the test suite. This file makes the .py file reusable in different test cases. Let's see how.
In your created mago folder (branching the code), open the gedit folder. We keep test suites ordered by application, to allow running test suites for only one of the applications.
Under the gedit folder you have gedit_chains.xml and a gedit_chains.py. They both have the same name, but this is not necessary. Open the Python file with your preferred text editor.
The .py file of a testsuite
The python file is only a simple python class inheriting from GeditTestSuite, which also inherits from TestSuite. All Mago test suites are classes that inherit, directly or inderectly from TestSuite. The main part:
if __name__ == "__main__": gedit_chains_test = GEditChain() gedit_chains_test.run()
is not necessary, because Mago will run the tests for you, but you can add it to your code for testing purposes.
So, we have a class, GEditChain, which contains a method. A test suite can contain as many methods as wanted. The only test method, "testChain", opens the application, writes on the first tab the string passed as the "chain" argument; saves it and compares the saved file with an oracle file (again oracle, in testing, means the "right" thing a test has to do. Something we know before hand that it is right, so we can check if our test is correct), and then closes the application.
As you can see in the code, there is not such a thing as "open" or "close" the application. This is done by the test suite for you. Mago leverage those sort of things, so you can concentrate on your test case. We will get back to that afterwards.
The .xml file of a testsuite
Now open the XML file with your preferred text editor. As you can see, it is a simple XML file. The root node, called "suite" allows setting a name for your suite. In this case, "gedit chains". The first child node, class, determines the python class of that test suite. In our case the class is the one we saw before. After the class node, we have a node call description. This is a text description of the suite and it will be included in the reports for your convenience. If you want your reports to be self explanatory, you have to include a nice description here
After that, there are as many "case" nodes as test cases included in the test suite. In our case we have two cases: "Unicode Tests" and "ASCII Tests". This is one of the advantages of separate the description and data, from the actual script. We can easily reuse the method for several test cases.
Each case has a "method", testChain in the example, a description, which will also be included in the report, and a set of arguments. These arguments need to match the arguments in the method.
Running the tests
So, let's try to run these tests using mago. If you are running any gedit session, please, close it if you don't want to lose your work
Go back to your mago folder and run:
$ PYTHONPATH=. ./bin/mago -f gedit_chains
That will run all the test cases in a test suite file name called gedit_chains. Once finished, you can check the test logs under ~/.mago/gedit
Under that folder, Mago have created two log files: the .log file is a XML, in case you want to parse for something else; the .html is a nice HTML report, with screenshots if something went wrong.
Adding a test case hacking the XML file
OK, let's continue by adding a test case for the same method. Open again the XML file and let's edit it. We will add it after the last one.
Add the following node:
<case name="Ubuntu Dev Week"> <method>testChain</method> <description>This is a test that everybody is going to try</description> <args> <oracle>data/udw.txt</oracle> <chain>Happy Ubuntu Developer Week!</chain> </args> </case>
before, the </suite> one, of course.
You can see the objective of the test case: open the application, write a text "Happy Ubuntu Developer Week!", save it, and compare it to the oracle.
We have to write the oracle file beforehand, so open a text editor, create a file with the text "Happy Ubuntu Developer Week!" (without no new lines) and save it as "udw.txt" in the gedit/data folder.
So lets's run again the test, always from the mago root folder:
$ PYTHONPATH=. ./bin/mago -f gedit_chains
This time, a new test case is also run for you, which will compare the new string to the newly created file.
Adding a test case hacking the mago library
OK, let's take it to the next level. Let's think that we want to do the opposite test case, opening a file, reading its contents and compare to the string we know it contains.
Let's create, under the gedit folder, a new file gedit_open.py with the following code:
import os from mago.test_suite.gnome import GEditTestSuite class GEditOpen(GEditTestSuite): def testOpen(self, oracle=None, filename=None): filename = os.path.join(self.get_test_dir(), filename) self.application.openfile(filename) textread = self.application.get_text() if textread != oracle: raise AssertionError, "Text read and oracle are different" if __name__ == "__main__": gedit_open_test = GEditOpen() gedit_open_test.run()
Also, let's create an XML file to run this (gedit_open.xml)
<suite name="gedit open"> <class>gedit_open.GEditOpen</class> <description> Tests which verify gedit's open file functionality. </description> <case name="UDW open test"> <method>testOpen</method> <description>Test open file.</description> <args> <filename>data/udw.txt</filename> <oracle>Happy Ubuntu Developer Week!</oracle> </args> </case> </suite>
In this case, the oracle is the string that we know the file contains.
Let's try to run this:
PYTHONPATH=. ./bin/mago -f gedit_open
The application opened, mago gave an error, and exited the application. That's expected, though.
The GEdit class does not contain a openfile method. We need to use LDTP functions to add new methods to the GEdit class. As we said at the beggining, one of the aims of Mago is reuse. Right now GEdit does not include a openfile method, but once added, anyone can benefit from this addition and use the method easily in their test scripts.
The GEdit class is under the mago library, application, gnome.py. Lets open it:
$ <editor> mago/application/gnome.py
Search for "class GEdit" and let's start editing.
Don't worry about LDTP syntax, that's another story. In this session we want to learn about the internals of Mago and how to contribute to it. LDTP has its own documentation and tutorials at http://ldtp.freedesktop.org/wiki/Docs
Going back to Mago. Mago library contains a set of resuable methods for testing applications. We want to avoid having ldtp functions in our scripts, and leave that in the library. If anything changes in the application, or we decide to change the framework, the scripts will remain the same.
In the GEdit class, add the followig two methods:
def openfile(self, filename): """ It opens a file name into gedit, by path. TODO: It fails if the file does not exists. The condition is not checked yet. @type filename: string @param filename: The path to the file we need to open """ gedit = ooldtp.context(self.name) mnuOpen = gedit.getchild("mnuOpen") mnuOpen.click() ldtp.waittillguiexist("dlgOpenFiles") dlgOpen = ooldtp.context("dlgOpenFiles") txtLocation = dlgOpen.getchild("txtLocation") txtLocation.settextvalue(filename) btnOpen = dlgOpen.getchild("btnOpen") btnOpen.click() def get_text(self): """ Given an application it gets the contents of the main buffer. """ gedit = ooldtp.context(self.name) txt_field = gedit.getchild(self.TXT_FIELD) text = txt_field.gettextvalue() return text
We are adding a method to open a file in Gedit, and another to get the contents of the main buffer.
All strings, as per Mago coding guidelines, should be set as constants of the class. Check the rest of the methods for an example. For the sake of simplicity of this tutorial, we have kept those as strings in the code. So you can start thinking about how LDTP recognizes the objects in an application.
OK, let's save the file and let's try to run it one more time now:
PYTHONPATH=. ./bin/mago -f gedit_open
Now, it runs the testcase properly.
Let's cover now the magic of opening the applications and closing them. As a told you, the GEdit test suite that we created, inherits from GEditTestSuite, which inherits itself from SingleApplicationTestSuite. Let's see what a TestSuite class and subclasses need to implement:
$ <editor> mago/test_suite/main.py
Every TestSuite class and subclasses need to reimplement, if needed, the setup, the teardown and the cleanup methods. The setup method is run before running any of the test cases, the clean up after every test case, and teardown, after the whole suite is run.
Let's take a look to the GEditTestSuite class:
$ <editor> mago/test_suite/gnome.py
The setup(), teardown() and cleanup() methods
What we do on the setup is opening the application. That's obvious. We close the application on the teardown method. The most complicated one in this case, is the cleanup method, run between test cases.
In this one we close the current gedit tab, ignore a "Save file" dialog if it appears, and create a new document; leaving gedit again, clean and ready for the next test case.
If you get errors about the setup, cleanup or teardown methods, it is here where you have to start debugging.
Questions & Answers
Q: Mago is like Rspec or TestUnit ara? A: No, those are unit test frameworks. Mago is for testing the UI directly. You don't need to have or to know the code of the application you're about to test
Q: Does it matter if the app is gnome or kde or something else? A: Right now, desktop testing is based in accessibility information, and KDE's is very poor. Right now you need to be running Gnome in order to test the application. That's going to change in a future, when at-spi (the communication layer for accessibility) gets ported to DBUS.
Q: Can mago be used to discover a11y problems in some apps, like missing description or invalid relationships A: It is the other way round, if an application has poor a11y information, it is going to be difficult to test (or impossible); but there are better ways to test if an application has good a11y information, like using Accerciser.
Q: Is it possible to run unattended on a head-less server? Like without X? A: I am afraid you can't. A full GNOME session is needed. Not only X, but also a gnome session