When developing applications for use it is important to test against physical devices, especially if you need to support a variety of technologies. Test on as many as is feasible, that is our view!
The reason this is so important is the fact that there will always be differences between simulators (where most developers test) and the physical device which has many additional applications running, settings unique to the device itself, and most importantly hardware and operating system specifications. Releasing applications based on simulator tests alone is not advised, which is why in the various App Stores it is often a mandatory requirement to test on physical devices.
Testing on the physical device for us is mostly for usability and stability/performance checks. This requires quite a few tests as you need to take all factors of the device into account, in all orientations and all conditions (low light, sunlight etc.). What might look great on a tablet could be unusable on on a phone. What works well using an Android device may make no sense on an Apple device. What is really fast on your desktop simulator performs terrible on a phone and so on.
Writing test scripts, taking snapshots of screen issues, making sure the tests are performed can be painful to do, and take considerable time. For us, if we develop across Android, iOS and Windows (a typical Mobile app) it requires us to test against 14 devices. If we are developing for Windows and OS X, the device count drops to 4 devices, the same for Web Based applications although we test across all major browsers on each device. In any case it is a lot of testing!
So we try to minimise this hassle by using a platform that allows the use of video/audio and questionnaire style feedback.
The image below shows our preparation to start testing an Application built to be used on Windows and Android devices. The configuration is made up of a Windows 8.1 based test computer (the white Acer mini), the LitmusX Professional Package – web site here (the software and hardware we used for testing physical devices), a 7 port USB hub to power and allow loading of Apps (some are loaded using wireless), and finally a professional camera on a tripod – not visible in the picture to do the recording and which is then used by the LitmusX software.