Java vs PHP for Testing

 

I screwed up by using Behat, and its elegant syntax, as a testing tool rather than what it actually is — an acceptance tool. Rather than using it only to validate Lightning’s user stories (i.e., “does Lightning deliver what users want?”), I used it to test virtually everything, including bug fixes, obscure corners of functionality, edge cases, and regressions. Bad move.

We ended up with ugly, complicated tests...

Why can’t I write regression tests in Behat?

Well, technically, you can. (I did it in Lightning for 2+ years.) You can also technically throw a thousand monkeys into a building, give them two thousand keyboards, and wait for them to crack prime factorization. But recall the wisdom of Jeff Goldblum in Jurassic Park: just because you can doesn’t mean you should, and although you can write regression and deep functionality tests in Behat, it is an express train to hell. I know because I (and others) have been there.


 

 

Conclusions

No matter you are using Codeception or not it is a good idea to understand how to perform browser based acceptance testing using just the php-webdriver by itself. Php-webdriver library provides very clean and flexible APIs you will enjoy working with.

PHP is a language popular for web development, but not for web testing. Test automation engineers prefer Java and Ruby over it. And there is a serious reason for that. There is no such thing like “official Selenium bindings” for PHP, i.e. there is no Selenium client library for PHP created by Selenium team. Developers of php-webdriver get very close to the official Selenium client APIs. That’s why you should use php-webdriver - it really feels and works like native and official Selenium libraries. That is especially important if you have an experience writing acceptance tests in Java or Ruby. Moving to PHP is not that hard, when all the APIs are the same.


PHP is not used outside context of a webserver

PHP

  • Uses
    • PHP is designed to be ran on a webserver...and encapsulated in HTML for a product
      • PHP is not seen by the end user..ever.. because of the SSI or server-side-include
      • Javascript can be seen by the end user, but not always
    • While PHP is growing into other uses & products, it lags decades behind Java for ambiguous run-time environments
  • Selenium
    • the best selenium.jar PHP language binding is the Facebook\php-webdriver api
    • this is simply a PHP library that makes calls to the java.jar
    • php-webdriver is NOT maintained by the selenium UG so will not only lag behind selenium api
      • but no guarrantee ANY api call will be available
  • DEV
    • PHP should be used for whitebox testing of PHP back-end logic
    • Not only can PHPUnit Tests be created simultaneously with dev work...
    • ...but good frameworks invite TDD & BDD adoption
      • making life SO MUCH EASIER on the developer by automating execution
      • tweaking code & running unit/bdd test gives immediate feedback and reduces dev time
    • access to native methods allows for precise & controlled debugging by using mocks or
      • PHPUnit to produce issues with 1000s of iterations or permutations of test data
      • can circumvent javascript & browser interfaces to isolate symptoms and their causes
    • IN SAME REPO

 

Java

  • Uses
    • Java runs everything from the web, to enterprise back-end, to dishwashers and wrist watches.
    • Test products are not black-box extensions of whitebox frameworks
    •  
  • Selenium
    • Selenium JAR is simply a zipped archive of java classes with a main() entry point
    • Java will ALWAYS have the complete API available to it
    • DIRECT calls to the jar library can even be made
    • the JAR can be extended in its native language
  • QA
    • Compiled languages are the pro's choice for QA Automation - Java & C# have massive lead
    • Test products have more 'real-world' interface demands and might have to drive business applications & 3rd party tools in addition to web pages
    • Reporting & Test Data requirements are more demanding
    • Task types are infinitely more diverse
    • community is arguably the largest, making for an environment conducive & inviting to QA testers that are early in their automation career
      • learning curve for new staff is easier for same reason
      • availability of staff greater
      • design patters are more numerous & better vetted
    • QA is not testing php code.. testing browser interactions with a product including
      • php backend
      • javascript front end
      • java business application APIs
      • php not well suited for all these activities
    • SEPARATE REPO/OWNER
    •  

 

  Java PHP
strict typing
  • variable typing enforced immediately & logic errors caught well in advance of runtime
  • variables can re-assigned on the fly with no data-type validation.
     
 
  • intellisense provides immediate feedback & prevents compiling assignment errors
  • assignment errors won't provide intellisense notifications

 

 
  • logic errors prevent compilation so are found before run-time: they virtually can not make it to the build server and literally cannot be included in a compiled test product
  • logic errors only caught at runtime
 
  • logic errors will prevent compilation and so are detected for the entire testframework project & are flagged immediately - before they can be used by other objects
  • logic errors only occur when the code is used, so errors can go unnoticed for long periods of time - with major implications if the code is imported/used by other framework objects and/or tests.
 
  • since logic errors are caught immediately at the point of assignment, troubleshooting occurs immediately & at the exact point the erroneous assignment is made
  • logic errors are incredibly hard to debug as the errors are only seen in the symptom, not necessarily where the erroneous assignment are made - requiring digging through the stack to locate the point of failure
selenium webdriver
  • selenium is a java-native tool so api calls are made available and made in the native language
  • php-java bindings are required & available api calls necessarily lag behind the native language api.
  • This could be weeks or months depending on the php-binding project owners
  • many times the complete api is NEVER made available for various reasons either because the receiving language is lacking capability of similar functions or the implementation is so complex that it is left out for years - or permanently
  • design patterns in one language aren't always easily achieved in another - either causing delayed support or no support at all
language hooks
  • language maturity equates to a substantially larger library to draw from
  • language originally designed for server-side interactions & support, not client side & not really for local execution... libraries are becoming more extensive and mature
 
  • language has grown up with test frameworks in mind
  • language has been mutated to perform functions it was never designed to, making for an awkward syntax and notation.
  • ie not only does the language have to 'work magic' but it has to reside happily inside HTML - and inherits an entire ecosystem of limitations to avoid HTML validation errors.

 

runtime libs
  • java jvm is extremely portable
  • php library is also extremely portable, though php.ini needs much more tweaking to stand up a test development environment
language community
  • java's test community is arguably the largest resulting in an abundance of documentation, user Q&A, examples, and tutorials.
  • a large portion of the design patterns and best practices fall out of this community into other language communities.
  • most language tools are ports from Java - PHPUnit and Behat to name a couple
  • php's web community is massive and vibrant but lacks an  equal counterpart for testing
  • php is not used to test java applications, but java is certainly used to test php applications
whitebox testing
  • native language MUST be used for whitebox testing
  • TDD demands the same, though exceptions apply when a native-language api is created to allow for being driven by larger test ecosystems in other languages - particularly when these ecosystems pre-date the software product.
 
Audience
  • if developers are the owners & maintaners of the framework, it makes sense to use the native language - even at the expense of capability.
 
 
  • often Development teams press hard for native language test frameworks so they are comfortable with them, yet NEVER write a single test, nor have to support havoc from a poorly selected language for the job.
  • QA is largely seen as a payscale below Development and so frequently get railroaded into bad tools simply because they are perceived as naive or ignorant with creating test software.
  • Testers will both be the creators and end-users of the tools & are in the best position to decide. Developers know design patterns for the products they create - test frameworks design patterns are a completely different type of software product that doesn't present itself in normal product design.
  • Exceptions to this exist where Dev teams have build up & actively use unit test frameworks & understand the consequences of doing things wrong or inefficiently. Updating 5000 tests due to poor abstraction in test fixtures motivates poor souls to invest time & discovery with best practices.
 
 
  • Choosing a language for QA Automation that is native to the developers makes sense until QA is tasked with testing other products or another product shows up using a completely different dev stack.
  • Nowhere is found a php framework for testing nodejs or c++ applications... php frameworks normally are only designed for php products.
  • java and c#, conversely are equally adept at testing just about any product from websites to firmware.

 

 
browser driver
  • QA Automation deals largely with integration testing & UAT.
  • As such, testing needs to occur using the complete stack & on multiple environments
  • Multiple Browsers must be supported
  • mapping out tests, browsers, & filtering tests by browser capability in a spreadsheet allows for a single document to be used for test planning, tracking and data input.
  • Ideally QA Automation should be integrated into CI/CD, but also needs to be accessible by Testers for running specific suites on-the-fly, not just on a CI server.
  • Dedicated build servers should be used to closely approximate the end-user experience
  • headless is great for back-end testing, but does little to reduce risk for the user - given browsers are the UI & each have their own implementations.
  • given headless bypasses a browsers ui rendering, timing will be different & latency issues may go unnoticed.
    • headless is great for post-build tests, but what about UAT? Is the E2E result ever looked at? how about testing Chrome vs IE vs FireFox?
  • At some point the regression test solution must use the complete solution stack & this means running tests in at least one browser in an environment identical to production.
  • goutte is a simple curl wrapper with get & post functions.
  • phantomjs crashes sporadically
  • bugs exist in different places with different browsers
   

 

 

 

 

 

Weak Typing Hell

Great for developers, worst case scenario for testers

Even worse for distributed dev groups

Comparison of Language Handing for compiled & interpreted languages

  • one will never see php error until the code runs.. which could be months or years...
    • the error will only be found at runtime
  • triaging a bug that has been there long past the designers availability makes for a miserable and expensive excursion into debugging.
  • likely that bugs like this will have dependencies, each relying on the other's conflicting data type
  • large dev teams increase the possibility of encountering confusing code and bad logic, taking debugging into reverse-engineering mode with horrific consequences
  •  

Precision instrumentation is critical to precision testing...

Strongly typed languages force better coding practices and are more self descriptive

  • More easily understand the intention of author & what is taking pace
  • unable to commit or build these types of logic errors
  • developer can address immediately, not after a JIRA ticket is opened & deliberated & possibly assigned to another developer for remedy.
  • 30s can easily turn into 8hrs
  •  
<?php
namespace App\Models;


class WeakTypingHell
{
    private $text;
    private $element;
    private $driver;
    
    public function __construct(){
        
        
    }
    
    public function stringOrInteger()
    {
        //make assignments
        $a = 1;
        $b = '2';
        $c = $a . $b; // NEED COMPILE ERROR HERE NOW!
        $d = '1' . '2';
        $e = 1 . 2;
        
        
        
        //print out the text & eval results
        print "\n";
        print "==============================\n";
        print "==============================\n";
        print "a: " . $a . "\n";
        print "b: " . $b . "\n";
        print "c: " . $c . "\n";
        print "d: " . $d . "\n";
        print "e: " . $e . "\n";
        print "so.....\$c = \$a . \$b". "\n";
        print "and....\$d = '1' . '2'". "\n";
        print "and....\$e = 1 . 2". "\n";
        
        print "then \$d + \$c should cause compile error and NOT evaluate as EQUALS '='". "\n";
        print "and what should \$e be?....";
        
        
        if ($c == $d ) { //
            print "PASS?....How does c equal d?". "\n";
        } else {
            print ("FAIL!.. GREAT... PHP caught that c doesn't equal d");
        }


        

package com.cs.cmsauto.testcases;

public class StrongTypingHeavenTest {

    
    public void strongTypingHeaven() {
        int a = 1;
        String b = "2";
        String c = a + b;
        String d = "1" + "2";
        String e = 1 + 2;
        
    }
}

 


        
    }
}

 

 

 

 

 

 

 

 

 

 

 

 

 

 

SDLC

  • Requirements
  • Architect
  • Developer
  • QA
  • User

Role

  • DevOps
    • Automation with native language developers use
    • whitebox & unit testing
      • api calls
      • integration tests
      • call methods directly
    • BDD/TDD
    • CI/CD
  • QA Automation
    • QA Automation with compiled language: Java/C#
    • blackbox
    • headfull
    • UAT
    • Regression
    • Functional
    • Sanity
    • Smoke
    • BDD ( BDD can handle Sanity/Smoke/UAT/E2E & nothing else )
      • BDD is for Behavior Driven Development and meant for Biz/Dev/QA to understand a how a feature to work
        • Dev creates test FIRST & uses it as an automated test runner for fast iterations as feature implemented
      • QA creates independent counterpart that serves as the contract with Dev & Biz on feature requirements
        • QA will then create test suites to address 'all' possibilities and error conditions - possibly 1000's of tests(ie DataValidation for api calls or Forms )
        • BDD cites one example
      • Also known as "specification by example"
  • QA Test Plan
    • test suites
    • test cases, steps, & data
    • scheduling
    • environment specs

-----blocking issues------

At a cross roads
Mink won't cut it: basically curl + HTTP GET & POST
    Sophisticated Drupal implemenation, not a simple out-of-box CMS
    Phantomjs crashes
    No content creation,
    selector catch/transform: no CSS, hovering flaksy, limited options
    [FIX] phpunit/php-webdriver framework that behat calls
    Carousel NEXT/PREV inaccessible for click()
Behat ok for limited step defs, but needs test framework under it
    Already have 30 defs for 100 tests
    300 methods for 1000 tests
    !!!
        when meganav changes.....
        when element changes?
        when landing page changes?
    !!!!
    Each Feature/FeatureContext method is a framework unto itself
    [FIX] phpunit/pom framework required
Jenkins not ready for prime time
    pulling demo data-base which has redirects not in prod - test fail en masse
    workaround is a manual process - can't do 3x daily
    writing tests for unknown environment/product
Tests
    No Coverage Matrix, visibility
    Tests have cross pollination
    Test make reference to different environments
    Tests need tagging overhaul
        Plan needs to be in Excel first
    Tests call outdated content - 10 content items gone....
    Unnecessary setup() steps
Reporting Output adequate for 100 tests: 1000?
    [FIX] phpunit test base should provide logging & report handling

 

  • Already moving past php into javascript-driven functionality....is php a good choice for this?
  • who's using & maintaining
  • what features first
  • what is generating the features
  • visual & presentation?
  • "do an excellent job, but let's remove 20% of the tools available
  •  


    

-----blocking issues------