Scalable Test Platform and Patch Lifecycle Manager
Milestone 0: Release Cleanup
* (DONE) Redesign & document Test Parameters feature [judith] * (DONE) Redesign & document Clean Reboot feature [cliffw] * (DONE) Add diagrams for Clean Reboot [cliffw] * (DONE) Web-med: Default the host type selection to first one in the list [bryce] * (DONE) Web-med: Add some docs to the search page regarding Hosts, etc. [bryce] * (DONE) Web-med: Add examples for the Lilo, Environment, and Sysctl boxes. [bryce] * (DONE) Web-med: Sync up the param & test pages from osdl_extranet to zope [bryce] * (DONE) Web-low: Determine & document procedure for test authors to update test descriptions [bryce] * (DONE) Web-low: Review error msg color - darker/tanner [bryce] - leaving as-is for now * (DONE) Web-low: Completed tests link is slow to load [bryce] * (DONE) Web-low: Consider alternative to 'Command' as header for results column [bryce] * (DONE) Web-low: Move the ":" inside the so it doesn't line break for sysstat link [bryce] * (DONE) Identify updates to developer websites we need to do in
Milestone 1 [bryce]
Milestone 1: New Tests, Analysis, and Test Performance Summary Reports
* (DONE) Get Open POSIX Testsuite working [bryce] * (DONE) Create PTS report script [bryce] * (DONE) Initiate monthly Test Performance Summary Reports [bryce] * Plan to review all existing tests scripts - remove hand installs * (DONE) Look at http://www.linuxbase.org/test/ & determine how to do LSB testing [cliff] * Plan adding debian sid virgin image to STP as new distro * (DONE) Meeting to discuss -mm kernels for doing interim kernel testing * stp-plm-web: Add info about Advanced Search tool to FAQ [judith] * (DONE) stp-plm-web: Remove NewFormMockup [bryce] * stp-plm-web: Update screenshots & add indexpage with thumbnails [bryce] * stp-plm-web: Add the Automated Testing presentation [bryce] * stp-plm-web: Add the roadmap presentation [bryce] * (DONE) stp-plm-web: Merge remaining tasks from STP Codebase Cleanup into this Roadmap and remove old page [bryce] * (DONE) stp-plm-web: Archive off the WBS Plan & Test Plan; Add link to this Roadmap instead [bryce] * (DONE) stp-plm-web: Add links & pages for Downloads, Discussion, Report Bugs, Request Features, etc. * (DONE) Determine which network performance tools to add & add to plan [bryce] * Web interface: Remove parts from test request page * Web interface: Remove parameter_columns if not needed * Web interface: aio-stress, loses checkbox status when re-editing * Create an add_parameters tool [bryce]
Milestone 2: Testdev Renovation & Documentation
* Design/documentation for distributed hosts / registration (see MS 4) * Add run time calculation based on started_date * System environment used in STP must be documented so outside user can reproduce the complete environment and replicate tests. * CGL/DCL tests need the concept of a 'test environment' Kernel plus user-space libraries/tools plus test code. * Design for CVS PULL into PLM * Design for Build meta-data in PLM * (DONE) Break out the power script into a separate package; generalize it as possible and post to freshmeat [kees] * Identify additional capabilities needed from SystemManagement module * Review the scripts in build:/home/robot/osdl-scripts and move valuable ones into the stp package
Milestone 3: RPC Core
* Rename stp/src/ to stp/scripts/ * Create a Perl module for host management (reboot, install, kick, reimage, results, activate/deactivate, etc.) * Create a Perl module for emailing user with test execution status * Create a Perl module for test request management (insert, cancel, etc.) * Create a Perl module for results management (compress / uncompress, upload, etc.) * Revise STP::Client::DiskHandler to be OOP and generalize for wider reusability, then post to CPAN * Revise STP::Client::OprofHandler with POD, then post to CPAN * Revise STP::Client::Tools to generalize for wider reusability * Revise STP::Util::machine with POD and generalize for wider reusability * WebService::TestSystem * Contact LWN about including report on linux performance [bryce] * Commandline tools * Check out GnuR (r-project.org) * Create capability to search on a range of dates * Add ability to sort results by any column * Enable searching on a set of patch id's
Milestone 4: Multi-Node Functionality
* Network testing * Tests Can Run at Multiple Sites and Multiple Machines * Tests Can Have a Stop Point for Testing (manual mode) * Create rough design document for multi-node testing.
Milestone 5: Analysis Tools
* Add a two-tr results comparison feature * Add a stock graphing capability * Add GCOV tools option
Milestone 6: Test Packages Improvements
* Update all tests to be installable in a standalone fashion like LTP is. * Consistency * Results upload * Environment should be built on the machine under test. Environment + kernel should be verified and failure reported before test is run. * Use PLM to maintain test code source trees. This requires a new PLM repository for test code. Each test will have a base version. Additions/alterations to the test will appear as PLM patches against the base version. Major changes to the test will result in the creation of a new base version.
Milestone 7: Minimize Host Turn-Around Time
* Pre-compiled kernel packages deliverable from PLM to STP * Editing of priorities for test requests * Use PLM to maintain a repository of 'test environment' code. This respository would hold source trees for any feature code needed by a test/environment. (ex. POSIX thread libraries ) Each feature code would submit a base version. Changes or variations to the feature code would appear as PLM patches against the base version.
Milestone 8: Server Daemon (Finite State Machine)
* FSM machine state report
Milestone 9: Expansion
* More tests * More distros * More software * Display lilo, env, and sysctl on web, but protect against malicious scripting attacks
Milestone Future:
* Analyze Samba BuildFarm for reusable portions * Add a capability to group test results together into a 'study'