Drupal Testing: Hazards with Using 'onboard' Test Engine

It could be that these hazards NEVER arise during a project, but when they do - they are show stoppers

• Using built-in Drupal test engines might sound like a good idea to developers, but it breaks most QA principles of abstraction, flexibility, and minimum impact to dev workflows
  • Makes sense if developers are managing qa functions independently
  • Makes sense for unit tests, particularly white-box testing
  • Developing a self-testing Drupal product?
    • or a Test Product to test Drupal?
• In CI/CD Pipelines
  • QA environment down multiple times a day during CI/CD pipeline execution
  • If using local environment for test, must pull tip down multiple times a day to work on new feature tests
  • if build fails OR
  • if all tests fail, must decipher if it is product or QA codebase
  • QA changes the codebase as often ( or more ) than DEV teams
  • Build failures by DEV or QA impact both teams
  • Enslaves QA test tools to supported DEV tools
  • Test Server same as Product Server
    • lag & timing issues occur
    • does not simulate real world interaction
  • Test execution on product server does not emulate internet connections
    • even if vanity urls are used, performing tests on product server does not authentically test end-user experience
      • requests must go through corporate LAN out to DNS, then back in
    • testing on localhost ( laptop environment ) also bypasses user experience
      • server response differs from live environment
  • Jenkins/BitBucket are java ( for JAVA solutions )
    • JDK already on build server
    • thus ALL other dependencies are managed in the maven pom.xml
    • easy segue to custom module dev for Pipelines
• Troubleshooting Framework on multiple environments impossible
  • when tests fail, there's no way to verify if it is
    • a build issue
    • long-term code issue
    • newly introduced code issue
    • test framework issue
  • tests can only target machine they reside on
  •  
• Framework codebase vulnerable at all times