• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

QA: Test Automation Framework

InverseOfNeo

InverseOfNeo

Diamond Member
I just got a job as a QA Engineer for a software company. I have to find a good automation framework to start using. The software I am testing is a web app, written mostly in PHP, and AJAX. I would like the automation to test the content, function and links. I don't have that much guidance as far as requirements my manager/company has, but I would assume that free/cheap is preferrable, something that is easy to use and sustainable. Anyone have any recommendations? I'll try to answer any questions you might have.

Thank you!
 
The last framework I used for test automation was FitNesse. It's free, open-source, extensible, and at least used to have pre-written "fixtures" for web apps.
 
I have worked with a multitude of automation frameworks* in the past 5 or so years and I’ll try to condense my knowledge into a few recommendations...

* I have mostly worked in windows testing and in most cases, the windows OS is better supported for automation compared to Linux. If you are looking at Linux, only the role your own applies. Also, many of my answers are based upon .net solutions but many have reasonable ports (Example: WatiN [.NET] was made from WatiR [Ruby] and WatiJ [Java] is another port).

I’m going to steal a little from a presentation I gave to code camp earlier this year… Before investing too must time in the development of automation, I would consider the following:


  • How stable is the system or component (UI, DB, etc.)?
  • What are management’s expectations?
  • How long do you expect the product to last? What is the budget in time and money?
  • Who is writing the automation? Is it a full time job? How experienced in QA and in writing code is the automation developer?
  • How easy is it to create the automation? Are test hooks created (Unique IDs or names)? How many custom controls exist in the app (Treecontrols in HTML, for example)?
  • How stable is the test tool?

See: http://www.kaner.com/testarch.html [Link appears broken, here it the google cache of it: http://webcache.googleusercontent.c...tarch.html+http://www.kaner.com/testarch.html ] for more good advice. Also see http://www.io.com/~wazmo/papers/seven_steps.html. If you would like to see my slideshow, go to www.sisqa.com/Training/201002_Automation-1000_Tests.ppt. I don’t know where they have hidden my example source code, but I’d be glad to email it to you if you are interested.

Now to try to answer your question…

Paid frameworks:
Visual Studio Test Prof. [2010], QTP, SilkTest: While both of these tools can be helpful I don't think the automated web testing by itself is enough to be worth the cost. Both have entire systems built around them to run tests, associate to requirements/stories, associate to manual tests, and a whole lot more. If you are willing to spend the money (I’ve heard rumors of rather large #s, but I simply don’t know) but are unwilling to spend the time to write a custom solution for some of the problems you face. I do want to note that VS test tools before 2010 were a joke and should not be used for anything but load testing. Of the three solutions, I think VS is probably the best, newest and most reasonable long term.

Free Non-Test Frameworks:
AutoIt, AutoHotKey: IMHO these can be useful tools in your tool belt, but should never be your primary tool.

Role Your Own:
Nunit, MSTest, FitNesse, etc: Personally I would suggest you hijack a unit testing framework to run your tests and not use FitNesse which I think is a bit heavy to start with and for my current company did not make sense. That being said, I don’t have much experience with the system.

Some Test Runner, Reporting Agent:
I have yet to see a really good test runner except possibly the one built into selenium which is enough to get you started, but not the ideal solution. We ended up writing our own runner. I have considered writing my own open source system, but have not had the time.

Framework:
  • WatiN: This is what I use at work, and it is the best system I have ever worked with, if you design it right. The biggest issue I have with WatiN is handling dialog popups (alerts, document uploads, etc). One of only a few tools that will work with other tools without any problems. It has very active support. While firefox is mostly supported, there are no dialog handlers and I have found with fat pages it doesn’t work so well. [4.5 / 5 for IE, 3 / 5 for firefox]
  • Selenium: Not a bad system, but from our evaluation it did not handle regular expressions for IDs (or name or any other way you might need to find an element) nearly as well as WatiN. In an ASP site, having the while _ID001_Blah_Blah_RealID can be a bit painful. Selenium is by no means a bad tool. I should note my last major evaluation was about 2 years ago, so they may have fixed some of the pain points. [4 / 5 from my previous evaluation, no opinion for firefox as that was not evaluated]
  • Art Of Test: I did only a little work with them after having a significant framework in WatiN. I found Art of Test to be controlling (not willing to share with WatiN) and there were several times when I wanted to see the source, but couldn’t since they don’t provide it… [3 / 5]
  • Slick Test Developer: Written by me 🙂. Currently there is only beta support for IE and no support for Firefox. You would have to create your own build at this point as web support doesn’t exist in the released version. I refuse to give comment on its functionality other than to say it includes many ideas of things I felt were good ideas from other systems and attempts to provide reasonable functionality. If you do use this system, I would be willing to give some support as good beta testers are hard to find 🙂.

IDE:
I really do feel a proper IDE with autocomplete can’t be overlooked . I suggest Visual Studios or even Sharp Develop if you want a free solution. If you can get the money, invest in resharper*.
* I received a free license from them for working on open source projects.

I hope this helps. I probably could write a book on the subject, so I'm sure I've skipped over some important points in just a page and half 🙂. Feel free to ask more questions in detail.
 
Thanks for the info! I think I have a fairly big mountain in front of me as management hasn't given me any input for what they want. I'll try out WatiN. Have you used the Telerik WebUI tool? It's not free. It seems pretty easy to use but may not have the capabilities that a truly code-based automation would have.
 
Yes, I have done some evaluation of Telerik’s WebUI test. If you are using Telerik controls, it certainly adds some value. To give it credit, where credit is due, I really like how they start with a non-code recording and then provide the ability to convert to code. By having this intermediate step, you can play with the recordings before they are code. This has great value if most of the automation development is being done by those who have limited coding knowledge. The down side to recorded tests is that they are generally not nearly as maintainable and the user typically doesn’t understand the system. This is OK if you plan on having ~50 automated tests (say smoke tests)*, but if you plan on creating 100s of tests. In particular, I don’t care for Telerik’s code output, which did not seem to be easily maintained. I should note that I was looking at the original Telerik release, where as Telerik has released at least one major revision since I looked. If you weren’t planning on using the recorder, I can’t give you much of an opinion other than to say they are actually using Art Of Test, which I already commented on.

* According to Microsoft, ~70% of organizations do either no automated testing or very minimal testing. From that basis, this maybe good enough for what management wants.

I should note that I feel that recordings are not the ideal way to automate in most cases where the testing effort is going to last longer than a year. Recordings tend to encapsulate UI information per test or a small set of tests rather than globally defining the UI. The reason for this design choice is because it is very hard to design code generally for all situations. Of all the solutions I have seen, Silk probably does the best with a global xml like document, but even that solution made it hard to maintain. Recordings also have a very hard time growing with the UI. For example, say you verify a cart total as “$10.41” and then the product changes price, a new "discount" module is added or the cost of shipping changes. Every single location you have that verification will need to be fixed as you will not have the data dynamically created. Furthermore, when you support France, you will have to create a second recording to be able to verify “10,41” (they use commas, not decimals; not to mention currency symbols).

At the risk of repeating my presentation, no matter what framework you choose, you need to find a good way to separate your concerns. Ideally you want to separate out the automation layer from the test cases to the point where you could literally replace one tool for another, replace the backend and all of your tests would continue to work. Here is an example:

[TestFixture]
public class SampleTest : Pages.SampleTestContext
{
[Test]
public void TextBox_AtLimit()
{
Test.Verify.SampleTestPage.IsTextCorrect(null);
TestData.TextFieldString = Test.Data.TextData.RandomString(25);//Text field limits user to 25 characters
Test.Step.SampleTestPage.EnterText(TestData.TextFieldString);
Test.Verify.SampleTestPage.IsTextCorrect(TestData.ExpectedTextFieldString);
​
}
​
}

As you can see, it is not clear what test framework I am using other than the NUnit attributes. In fact, I wrote this using WatiN, but I could just have easily have used Slick Test Developer. The other big advantage to this style is that a new automation tester can easily “discover” what tests can be written with whatever framework currently exist by simply typing “Test.” and use auto complete for discovery. In fact, they would find they had Steps, Data and Verifications they could use, and then they could discover what pages they could work with.
 
We use Selenium. We run ~425 functional tests against the test environment every hour (a total of ~5 hours of tests, spread across 18 testing machines). For the most part it works pretty well. We use CruiseControl.NET to drive it.

At the time we chose it, we needed it to be (a) free, since the tests are created/run by a large developer base, and (b) simple, since we also had a non-developer running the effort. I like WatiN and may revisit it in the near future but obviously we have a pretty large investment in our existing solution already.
 
Noobsa, can you confirm that the current release of WatiN cannot utlize <th>? Many of my tables are coded with header (<TH>) and WatiN cant find it. I have looked and I found a blog that describes it but it doesn't work anymore. I assume that I have the right idea/code:
Code: 
Table clusterTable = ie.Table(Find.ById("cluster_info"));
string CellText = clusterTable.TableRows[0].TableCells[0].Text;
Just to clarify, if I replace the 0 in the TableRows[0] with a 1, I get the row below the header.
The blog suggested to do the following, but ElementTags is no longer supported:
Code: 
TableCell.ElementTags.Add(new ElementTag("th"));
 
Last edited:
Unfortunately, it appears for historical reasons WatiN does not by default support th tags. Here is what I think is a reasonable workaround…

using System;
using WatiN.Core;
using WatiN.Core.Constraints;
using System.Collections;
namespace WatiNSample
{

class Sample
{

[STAThread]
public static void Main(string[] args)
{
using (WatiN.Core.Browser IE = new WatiN.Core.IE(@".\WebPages\TableSample.html"))
{

var tr = IE.TableRow(Find.ById("HeadRow"));
Console.WriteLine("Count: " + tr.TableCells.Count);//TableCells which only support td.
Console.WriteLine("Count: " + tr.ElementsWithTag("th").Count);//default watin functionality.
Console.WriteLine("Count: " + tr.AllTableCells(Find.Any).Count);//user created functionality...
Console.WriteLine("Count: " + tr.ThTableCells(Find.Any).Count);//user created functionality...
Console.ReadKey();
IE.Close();​
}
}
​
}
public class THTableCells
{

private TableRow tableRow;
private Constraint findBy;
private System.Collections.Generic.List<ElementTag> tags;
public THTableCells(TableRow element, Constraint findByFilter)
{

tableRow = element;
findBy = findByFilter;
tags = new System.Collections.Generic.List<ElementTag>();
tags.Add(new ElementTag("th"));
tags.Add(new ElementTag("td"));
​
}
public ElementCollection TableCells()
{
return tableRow.ElementsWithTag(tags);
}
public ElementCollection ThTableCells()
{
return tableRow.ElementsWithTag("th",null);
}
public Element ThTableCell()
{
return tableRow.ElementWithTag("th", findBy, null);
}​
}
public static class WatiNExtensions//provides nifty autocomplete, but you must have the a using statement if it isn't in the same file.
{

public static ElementCollection AllTableCells(this TableRow domContainer, Constraint findByContraint)
{
return new THTableCells(domContainer, findByContraint).TableCells();
}
public static ElementCollection ThTableCells(this TableRow domContainer, Constraint findByContraint)
{
return new THTableCells(domContainer, findByContraint).ThTableCells();
}
public static Element FirstTHTableCell(this TableRow domContainer, Constraint findByContraint)
{
return new THTableCells(domContainer, findByContraint).ThTableCell();
}
​
}
​
}
 
Umm wow, that's a lot for a seemingly small amount of functionality. I might just figure out if I can validate another part of the table. On the other hand, I'm still thinking if it is really worth it to automate at all. I'm somewhat on the fence about it but I have to come up with a good argument to present to my manager. The UI might not be stable enough.
 
I believe if you examine the code a little closer you will see that it is not that long, but instead meant to demonstrate a few ways to solve the problem. The absolute shortest method is to write it something like this:
var text = clusterTable.TableRows[0].ElementsWithTag("th")[0].Text;

I don't suggest you write it that way. The disadvantage to this method is it does not encapsulate the logic nor is it particularly obvious. The alternate method looks like this:

var text = clusterTable.TableRows[0].AllTableCells(Find.Any)[0].Text; The nifty thing about this is that if the system changed from a th to a td, this would handle it, as the tag is abstracted. Also, I think this is a little more clear in reading. This method however requires that you have both the THTableCells and the WatiNExtensions classes in the code base. One other advantage to this method is if the WatiN API changed, you only have to repair the THTableCells class and not every place you get a th element.

As for the value of automation, I think it depends on how much development is going to be done on the product. From what SearchMaster said, their company has roughly 2000 tests. At my company, we have roughly 1000 tests [ + ~2000 unit tests]. In order to get a tester to run through 1000 tests, I would estimate it would take 2.5-5 days, rather than the approximate 14-18 hours (12 hours at night to run + 2-6 hours to review results) it takes us to run the tests, review the results and file bugs. The significant value in this is not that the results were cheaper to get, but that the results were much faster (time = money) and much more consistent.
 
We have about 3400 unit tests, and as said before about 425 functional tests. The functional tests are complex and most test multiple functions (login, test a few pages and functions, logout), so we're pretty test heavy. Every one of the tests runs hourly.

On the question of value...well, as you said it would be impossible to test that much without automation. I implemented the system when we started to grow to the point that people were changing code to implement functionality in one area (or fix bugs) but inadvertently broke another area. I realized it was impossible to ask the developer to understand 100&#37; of the implications for every code change so I created this safety net. Because automated testing only does what you tell it to do, it will not "find bugs", just prevent you from breaking existing functionality with your changes.

We are a pretty agile shop - our code base is ~1.2M LOC and we deploy it daily. We absolutely could not do it without some sort of automated testing framework at our current size (>100 developers worldwide).

We have zero QA testers.
 
Oh, and with regard to this statement:

Selenium: Not a bad system, but from our evaluation it did not handle regular expressions for IDs (or name or any other way you might need to find an element) nearly as well as WatiN. In an ASP site, having the while _ID001_Blah_Blah_RealID can be a bit painful.

Selenium has good support for XPath so we write our tests like "clickAndWait //*[contains(@ID, 'RealID')]" and it works pretty well. Unfortunately the default IDE uses the full path so you have to manually modify the tests though I think there's a plugin that helps with that (we haven't tried it yet).

My biggest issue with Selenium is that support is....spotty. That's expected with a freeware tool but can be frustrating. Fortunately we've been able to solve most of our issues and are pretty stable now. Selenium 2 should be out pretty soon (it's in alpha now) and we're just now upgrading our platform to be able to support it. From what I understand, Google has a sizable investment in Selenium so hopefully they will continue to innovate and make it even better.
 
Noobsa, you are very right, that code is not bad at all. I just didn't look at it when I was fully awake. It works wonderfully.

However, I have hit a snag. There are multiple tables on each page, but not all of them have unique identifiers. I have submitted a feature request to fix it. I know it would be a big effort to change all of the tables. I talked to the UI developer and he said that I shouldn't expect much support for my request from management. He has been pushing for a page template management system (currently he writes all of the html/php for each page) and they don't see the need for it. I'm putting my automation effort on hold until I get more support. I greatly appreciate your input though. It was very helpful.
 
InverseOfNeo said:
However, I have hit a snag. There are multiple tables on each page, but not all of them have unique identifiers.
Click to expand...
SearchMaster said:
Selenium has good support for XPath so we write our tests like "clickAndWait //*[contains(@ID, 'RealID')]" and it works pretty well.
Click to expand...
I'm not at all familiar with Selenium, but there are many more ways to select an element in XPath than just by ID. The simplest would be by index: "div[contains(@ID, 'KnownID')]/table[47]", say. That's by no means the best way to select an element, but sometimes it might be a good way.

Evidently there are many other ways (like this one) to select an element relative to another element. Usually I would prefer something more specific over simply selecting by index, as Javascript or a developer might insert or remove a table, but I don't know how many of those methods Selenium supports.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top