Approved ARM Compiler test suites include:
- The Perennial Validation Suite for C
- In-House Validation Suites
- Edison Design Group (EDG) Front-End Test Suite
- Roguewave C++ Library Self Tests
- Standard Benchmark Tests
- ABI Test Suite
- Defect Management
- ISO Certification
- TÜV Certification
- Keil Product Integration
The Perennial Validation Suite for C is the industry standard for the verification of C compilers. The Perennial Validation Suite uses both positive and negative test techniques to accomplish the following:
- Test basic conformance to the ISO c90 and c99 C language standards.
- Generate self-checking C programs that test permutations of operators and data types.
- Determine the size of certain compile time limits, e.g., significant length of identifiers or how deeply include files may be nested.
- Test for legal expressions by generating self-checking expressions of arbitrary complexity.
In-House Validation Suites
- Regression Suite - When defects are fixed, tests are added to our regression tests suite to ensure the defect is not re-enabled by future compiler changes.
- Language Extensions - ARM maintains an in-house test suite for testing product language extensions which are not covered in the ANSI/ISO standard.
- C-Header Tests - Verifies the internal consistency of the standard C header files. They work by running the compiler over each header and having it dump out its internal state. The result is then examined to ensure consistency.
- Math Library Tests - The Floating-Point and Math libraries experience directed tests of special cases (NaNs, infinites, underflow, etc.) as well as tests associated with known difficult values.
- Convergence Tests - This suite tests that the ARM Compiler can build itself. This is done in two stages, first using the ARM Compiler to build a semi-hosted version of itself targeting ARM, then using the semi-hosted version to build the second version, and comparing the final images and objects.
- Code Database (CodeDB) - CodeDB is a collection of open source applications and approved customer code used to verify that complex real-world applications can be built with the latest compiler version.
- Product Integration Tests - These tests ensure that the different product components have been successfully integrated and produce the preferred user experience. New tests are generated for each release based on new features in that release. In addition to new tests, product integration testing consists of more than 1,200 regression test cases.
EDG Front End software is tested using Gimpel's FlexeLint, Plum Hall, and Perennial commercial test suites. Additionally, the front-end software is subjected to ~25,000 local regression tests developed over the past 20 years.
A self-test suite supplied with the RogueWave C++ Standard Template Library (STL).
Standard Benchmark Tests
Standard benchmarks such as EEMBC and Dhrystone are used to verify that the benchmark suites compile and execute properly.
This suite tests conformance to the ARM-THUMB Embedded Application Binary Interface (EABI) Procedure Call Standard (ATEPCS).
Defects are numbered, documented, and prioritized in our defect database. Defects are regularly reviewed and targeted for correction based on priority. Defect data is secured via nightly backups and offsite storage.
ARM maintains ISO9001:2008 certification (number LRQ 0963114) which is applicable to design, development, and quality management of software solutions at ARM.
Some versions of the ARM Compiler have been certified by TÜV SÜD as meeting the toolchain requirements for safety- related development up to SIL3 (IEC 61508) or ASILD (ISO 26262) without further qualification activities.
The ARM Compiler has been used in all versions of the Keil MDK-ARM (Microcontroller Development Kit) since its release in 2006. A full MDK-ARM version history is available for reference.