Login
Register
Actian.comNavigation
Learn About
Developing With
Ingres Talk
Information
Toolbox
Views
Test Suite
From Ingres Community Wiki
Getting Started: Ingres Test Suite
The latest version of Ingres comes with a large number of SEP tests for validating database operation.
Note: Code examples use '/ingres' for the ingres root, 'T1' (Test 1) for the installation ID; you should replace these with the actual location of your code and actual installation code used for your installation.
Note for Ubuntu users
Ubuntu Linux from 8.0 onwards links the bourne shell to "dash", you can see this if you run "ls -l /bin/sh". "dash" is a lightweight shell that does not fully support all bourne shell syntax; some bourne shell syntax used in the Ingres scrips will not work in "dash"; Ubuntu users are therefore advised to re-link "sh" to "bash". Also, on some Ubuntu distributuins the program "basename" is not found in /bin, bit in /usr/bin; the Ingres scripts expect it to be in /bin, so it should be linked there. As root you should:
sudo dpkg-reconfigure dash ln -s /usr/bin/basename /bin/basename
Note for users of GCC version 4.3
As of (about) 10.0 build 106, Ingres builds created with this version of the compiler should pass the test suite. However, experience with gcc 4.3 is still somewhat limited as of this writing (Sep 09). If your build with gcc 4.3 does not pass, and you cannot find any other reason for the problems, double-check for function return type mismatches. (For example, function f() returns bool, but it's not declared at the point of call, making the caller implicitly expect an int return. This no longer works with gcc 4.3.) Please see the warning at the bottom of this page:
http://community.ingres.com/wiki/Building_Ingres_FAQ
Testing From Source (buildAndTest)
The Ingres database source code is frequently updated, so failed tests on old code may no longer be relevant. If your source is out of date, or you don't have any yet, it is best to start fresh from the subversion repository.
mkdir /ingres svn co http://code.ingres.com/ingres/main /ingres
From there, a single script can compile, install, and test the Ingres database for you. This process takes several hours and several GB of hard drive space. When running the complete build and test script you should do so as root. Since the script sets up the environment, in order to avoid issues with residual environment setting from previous installations or test runs you should start with a "clean" shell; that is, one that has never had an Ingres environment set for it, ideally you would create a new terminal window as root, or sudo or su to root and immediately cd to /ingres/buildtools and run the script.
(as root) ./ingres/buildtools/buildAndTest.sh T1
Continue to Reading Output
Customising 'buildAndTest.sh'
If you're doing this for a while, it wouldn't be a bad idea to examine the contents of 'buildAndTest.sh' at some point. Simple changes might include removing the line that resets file permissions, so you won't have to chown all the files back to the 'ingres' user to run subsequent tests.
What are some other good ideas?
Testing After Compile (for developers who wish to test their code changes)
If you have already compiled a recent source revision, or you have made code changes and re-built already, and want to run the test suite without recompiling, you can follow these steps in the order specified to test your already built code.
1. Re-setting permissions
Since the 'buildAndTest.sh' script resets permissions after execution, you'll need to give everything back to the 'ingres' user. You will only need to do this if you have previously run buildAndTest.sh.
(as root) chown -R ingres:ingres /ingres
2. Creating Users needed for Ingres testing
If you have not previously run buildAndTest.sh on this machine you will need to create the accounts that Ingres uses for testing. Using your system tools (either useradd or some other GUI tool) create the users ingres, pvusr1, pvusr2, pvusr3, pvusr4, and testenv. The password for testenv must be set to "ca-testenv" for the tests to complete successfully.
3. Creating an Ingres installation to test with
If you have not already done so, you will need to make an Ingres installation to test the build. We will assume from here on in that the Ingres installation code is T1, if you have an existing installation that uses a different installation code, then replace "T1" with that code in future commands in this page.
(as ingres) cd /ingres/buildtools source set_env.sh createdbms T1 source test_env.sh T1
4. Re-setting the test output area when running tests again on an existing Ingres installation
If you have a pre-existing Ingres installation on which you have already run tests, you'll also want to reset the output folder, as previous test output would still be there.
(as ingres) rm -rf $II_SYSTEM/output mkdir $II_SYSTEM/output chmod 777 $II_SYSTEM/output
5. Copying the test suite to your Install area
If you have not previously run buildAndTest.sh using this Ingres installation you will need to copy the test suite to the root location of the installation
(as ingres) mkdir $II_SYSTEM/tst cp -r $TST_LOC $II_SYSTEM/tst
6. Running the "root" setup script
If this is the first time you are running tests on this installation, you will need to run the root setup script as root
(as root) cd /ingres/buildtools source root_setup.sh
7. Running the tests
Now you are ready to run the test suite, you will do this as user testenv; since the script sets up your environment, you will not need to do that beforehand, simply switch user to testenv and run the script.
(as testenv) cd /ingres/buildtools source runtest.sh T1
Reading Output
Output is saved in four places:
The compile log displays thorough output from the compile process. Search this for "failed". If something failed during the compile, then it will probably be the cause for errors in subsequent tests.
- /ingres/logs/jam_<timestamp>.log
The test automation script log displays timestamps for when each set of tests started and finished.
- /ingres/install/T1/output/my_test_run.log
The test logs are stored in a very straightforward manner. Keep in mind there will only be log files if something went wrong, but other files will always be present regardless of errors.
- /ingres/install/T1/output/<category>/<subcategory>/<testid>.log
- example - /ingres/install/T1/output/be/access/am26.log
The Ingres database error log will contain a lot of errors, because some tests are for what the database should not do.
- /ingres/install/T1/ingres/files/errlog.log
How do I know that the tests are successful
As a developer, in order to submit your changes back to the code management system you will need to demonstrate that you have run the acceptance tests and have not caused any additional issues (regressions). Because of the nature of a development code-line some tests will produce output that is not identical to the canonical files (the default expected output). If you are running tests for code submission, it is recommended that you use a revision of the code from one of the Ingres "tags". Approximately every 2 weeks and new "checkpoint" of the code is made, at this time the latest revision of the code is pulled, built, and tested. The results of the tests are posted on this page (see below), and a new "tag" is created. If you use one of the "tags" listed below you can be sure that any deviation from the baseline results posted below are a result of your code changes or your environment.
Quick sanity check of results
Once your testing is complete the results are located in the output directory below the root of your installation (that is, $II_SYSTEM/output). You should first obtain the "baseline" results for your tag from the link below and extract the file out to another location (e.g. in /tmp). If any tests differ from the expected results the log file for that test is retained, otherwise the log file is deleted after the test completes. The following commands will tell you which tests produced different results than expected.
cd $II_SYSTM/output find . -name \*.log -print
You should also run this command in the area where you extracted the baseline results from below. If you have the same log files as the baseline you are almost done. If you do not, you should check those log files to find out why.
Even if you do have the same log files as the baseline, you should check them anyway to make sure that your diffs are the same as those in the baseline.
Checking log files for diffs
For each log file you find above, you should edit the file and find the string "DIFF" (this may be repeated several times in the log if there are several diffs). The log will display the output from your test run, the output expected from the canonical file, and the difference between them. This may be enough to find the problem. If you need to know more about what is being run at the time you can look at the original "sep" file that generated the log, these can be found below the "src/tst" directory in the root of your code line. look for a file with the same name as the log, but with extension ".sep".
If you have the same log files as the baseline, and your diffs are the same as those in the baseline your tests have completed successfully and you can submit your change (after following the correct submission procedure). If you have log files that the baseline does not, or if your log files contain different results to the baseline, you should investigate the diffs and work out why. If you are unsure of what is causing your diffs you should post to the forums and the Ingres developers will help you.
Test Results
The test logs contain directories that correspond to individual test suites. The files that need to be examined after the tests have completed are those with the .rpt filetype. Each test suite directory will have one .rpt file. The .rpt file contains a summary of all the problems found in the suite and a count of the number of problems found in each test. For each test that detected problems, there will also be a separate .log file which contains the details of the differences that were identified.
Since the Community version of Ingres is constantly being updated, there are bound to be problems and differences that have not been corrected yet. In order for Open Source Ingres developers to know whether the test differences that they encounter are caused by their changes or are pre-existing, there are reference test output logs which can be used for comparison.
The following output logs have been created for each checkpoint build and corresponds to the source revision in the specified subversion Tag.
- The release has been renamed from 9.4 to 10.0
- Ingres 9.3 build 151 is the last Community build that will be made available for 9.3. The next Ingres 9.3 build will be a Commercial GA version.
Troubleshooting
My compile/install/test keeps stopping! What's wrong?
The hard drive or partition is full
"It's always a matter of space!" ~vande02
Especially if you're running in a VM, free up some space by deleting old junk or unused ingres instances.
Your computer was locked or went to sleep
Sometimes the script fails to recover from sleep mode, so make sure to disable that. It has also been reported that the (Fedora Core) screensaver lock interrupts script execution, so consider disabling it temporarily.
There's a bug
Yes, it happens. If the script consistently stops at the same place, where it had never stopped on previous code revisions, either rollback or wait for an update to a newer revision of source code.
Isolating Segmentation Violation Errors
If you notice any SEGV errors in 'errlog.log' (see reading output) after running the test suite, it may be necessary to track down the cause. The procedure for this is to compare the time the error occurred with the timestamps for the beginning of each test in the SEP output (*.out) files. Since there are a number of SEP output files, it can get tedious to search through them all manually. You can quickly find the necessary file with the following command:
grep -lirc "Mon Oct 27 06:27" /ingres/install/T1/output/*
In this case, the error occurred at "Mon Oct 27 06:27:07 2008", but the search only compared up to minutes just in case there are any discrepancies. Open the file with your favourite text editor, and take note of which test (or series of tests if they have the same timestamp) that could possibly have caused the error.
Re-run each test individually (don't forget to set SEPPARAMDB!), until you are fairly certain which test is producing the SEGV. In a separate console, run "tail -f /ingres/install/T5/ingres/files/errlog.log", then continue with the test execution to see the SEGV happen in realtime.
If for any reason you want a complete list of all output files, you can find them with this command:
find /ingres/install/T1/output -name "*.out" -type f
TODO
- testing on Windows?
