11:00:36 <E-P> #startmeeting Mer QA meeting 3/4/2012
11:00:36 <MerBot> Meeting started Tue Apr  3 11:00:36 2012 UTC.  The chair is E-P. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings.
11:00:36 <MerBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
11:01:17 <lbt> Good afternoon all o/
11:01:18 <timoph> o/
11:01:36 <E-P> Welcome to Mer's initial QA meeting, I propose that we have following agenda: current status, QA goal, tools & tests and next meeting
11:01:59 * Stskeeps is here, on laggy gprs, and mostly listening
11:02:08 <E-P> great :)
11:02:24 <iekku> i'm here too
11:03:16 <E-P> We can start with the current status and if say if you want to discuss about some specific topic
11:03:27 <lbt> I'll add in 'scope'
11:03:47 <lbt> by which I mean I thought of 4 areas : SDK & Tools; Core; Hardware adaptations and other vendor issues; System
11:03:59 <lbt> s
11:04:50 <E-P> ok
11:05:09 <E-P> #topic Current QA status
11:05:53 <E-P> I have been following Mer about couple of weeks, so I don't have much of knowledge yet
11:06:23 <lbt> well, Mer Core has a CI system in place; each change to a core package is submitted for review before acceptance
11:06:54 <lbt> as a part of that review, we use BOSS to trigger some builds in the OBS
11:07:29 <lbt> that then uses the OBS build dependency capability to trigger other packages that depend on the package in question
11:07:40 <lbt> this happens over all architectures
11:07:59 <E-P> good, that is already a lot
11:08:06 <lbt> if there are any failures then BOSS rejects the submission automatically
11:08:44 <lbt> BOSS is not a QA system - it 'just' provides process automation. Typically it says what test-suites to run and what to do based on the results.
11:09:36 <E-P> I am familiar with BOSS, but I haven't used it before
11:09:37 <lbt> for example. we'd like to extend the process so that when a build finishes succesfully, it (BOSS) builds an image
11:10:45 <E-P> and if I have understood the BOSS correctly, it is easy to extend?
11:10:52 <lbt> very much so
11:11:08 <E-P> great
11:11:15 <lbt> think of it as robust shell-scripting for systems
11:11:43 <lbt> so it can do things like update bugzilla. make commits to git, trigger flashing of images
11:12:07 <lbt> anything you can do using an api really :)
11:12:48 <E-P> what is the status of the signaling the changes to the vendors?
11:13:18 <lbt> pipedream :)
11:13:51 <lbt> we need to figure out what to say, to whom, when and how to handle responses and non-responses
11:14:21 <lbt> but this is all process definition - when we know what we want to do, BOSS will be able to do it
11:14:31 <E-P> nod
11:14:47 <lbt> eg in Apps for MeeGo we implement a voting system
11:15:01 <lbt> in Mer Core we may grant some vendors veto rights
11:15:12 <lbt> depending on their contributions to QA
11:15:38 <lbt> that's all human level design of how to interact as a group
11:16:05 <lbt> My gut reaction is that we provide a "please test" signal
11:16:33 <lbt> wait for a few hours and then prompt a human reviewer to make a decision based on collated results
11:17:17 <lbt> ie we automate rejections for some criteria (eg core build fail) and ask for intervention on others
11:17:36 <lbt> as we learn we should be able to define and apply heuristics
11:17:42 <lbt> .
11:18:09 <E-P> that was one of issues, what I posted to the mailing list is that, how we can be sure that the results are valid and the tests are valid
11:18:31 <lbt> *nod* ... we have to make a call
11:19:11 <lbt> collaborative testing like this seems to be unusual
11:19:20 <E-P> yes, I think there will be many issues and we will solve them then
11:20:43 <E-P> there is a image building "process" added to the BOSS?
11:20:55 <E-P> (or how do you call the BOSS's plugins)
11:21:25 <lbt> BOSS has process definitions
11:22:17 <lbt> http://autodoc.meego.com/boss/processes/CE/MW/MTF/
11:22:25 <lbt> lets not get too deep into them :)
11:23:00 <lbt> we have a plugin in OBS which triggers a named 'obs' process on certain events
11:23:25 <lbt> the process calls a plugin (participant) to look at the event and decide how to handle it
11:23:43 <lbt> typically it uses a lookup based on the project name
11:23:58 <lbt> and then runs a project specific process
11:24:12 <lbt> each participant is a bit of python
11:24:19 <E-P> ok
11:24:29 <lbt> and it can do anything python can do
11:24:33 <E-P> we can discuss the details later in #mer channel
11:25:11 <lbt> So... proper testing is done with something like testrunner lite
11:25:31 <timoph> it's just an executor
11:25:48 <timoph> to help to get unified result format, etc.
11:26:05 <E-P> yes, the testrunner doesn't specify any test framework
11:26:26 <E-P> let me wrap up the current status, and then we can move on
11:26:43 <E-P> #info Mer Core has a CI system in place; each change to a core package is submitted for review before acceptance
11:27:31 <E-P> #info Changed package is compiled to all architectures in OBS, and rejected if any of the build fails
11:28:02 <E-P> #info BOSS is used to handle this process
11:28:24 <E-P> #info Signaling system is not implemented and requires planning
11:28:33 <E-P> anything else to add to current status?
11:29:38 <lbt> #info automated image generation after each build and before acceptance is almost ready
11:30:59 <E-P> thanks
11:31:22 <E-P> let's move on
11:31:48 <E-P> #topic QA Scope
11:32:25 <E-P> lbt you mentioned the 4 areas
11:32:52 <lbt> yes, I thought of: SDK & Tools; Core; Hardware adaptations and other vendor issues; Systems
11:33:28 <lbt> the HA/vendor covers the bit we mentioned about triggering some kind of distributed QA
11:33:56 <lbt> SDK & Tools is probably the easiest since they can run in a VM :)
11:34:11 <E-P> yep
11:34:15 <lbt> Systems is very hard since there are complex interactions and very large datasets
11:34:43 <E-P> can you open the Systems a bit, what do you mean by that?
11:34:43 <lbt> eg testing a change to a BOSS process needs a test using OBS, IMG, bugzilla, testrunner
11:36:01 <lbt> usually we end up testing on production but using test projects - not always ideal
11:36:50 <lbt> Right now I'm working on OBS deployment testing
11:37:08 <lbt> so creating VMs, running a deployment, building a package
11:37:30 <lbt> but this needs to work with latest scratchbox2
11:37:43 <E-P> ok
11:37:46 <lbt> and vice-versa - we need to test sb2 before deploying to production
11:37:48 <lbt> all nasty
11:38:22 <lbt> so ... Systems are in scope mainly to say they're nasty and we should special-case them :)
11:39:15 * lbt wonders again why he actually *chose* to do systems...
11:39:42 <E-P> I wouldn't separate the system testing as its own, it could be part of the tools
11:40:20 <lbt> there are defintely parts that can be tested in isolation
11:40:29 <E-P> sure
11:40:34 <lbt> OBS has some kind of test suite that should be run
11:41:15 <E-P> we could define dependencies between the tools, and when we change a tool, we would test the tool + all its dependencies
11:41:31 <E-P> meaning, in the system wide testing
11:41:39 <E-P> just an idea
11:42:42 <E-P> how do you see the core testing, like Stskeeps posted to the mailing list, there is no reference hardware or adaptation where to run tests for Mer core
11:42:44 <lbt> *nod* ... I feel it's a different area to 'device testing' which is what the other 3 relate to
11:43:13 <E-P> that is true if you think that from that point of view :)
11:43:37 <lbt> I feel we should setup a VM based reference HA which is not part of Core and also acts as a template for Vendors
11:44:01 <lbt> Nemo and Vivaldi could support this too
11:44:19 <E-P> and that could be in the early notification list
11:45:05 <lbt> possibly
11:45:24 <lbt> since it's a VM it may give false failures
11:45:48 <lbt> I'd make it a peer at 'level 1' notification
11:46:19 <lbt> any vendors who want to hold back until some sanity checks are done can wait for level 2 notices
11:46:41 <lbt> but I'd hope to see some participate at level 1 aswell
11:47:08 <E-P> would be good
11:47:24 <lbt> eventually if the VM turns out to actually catch a lot of problems we can optimise
11:47:36 <lbt> but we may find it's no substitute for hardware
11:47:43 <E-P> that kind of VM HA could be the way how we start to build the QA system
11:47:51 <lbt> 100%
11:48:27 <lbt> I'd like us to have a step called "flash the VM" :)
11:48:44 <lbt> to reinforce that it's not special - just a virtual device
11:49:01 <E-P> I have built something like that :)
11:49:11 * lbt notices time ...
11:49:12 <E-P> using OTS and KVM
11:49:19 <E-P> uh, we are running out of time sono
11:49:22 <lbt> yep - perfect
11:49:37 <E-P> #info 4 areas : SDK & Tools; Core; Hardware adaptations and other vendor issues; System
11:49:55 <lbt> are there any other areas BTW?
11:50:16 <E-P> we can start with those and define more later if needed
11:50:28 <E-P> or change them
11:50:35 <lbt> OK
11:50:57 <E-P> let's skip the tools topic for today
11:51:08 <E-P> we can use the tools from meego pretty well
11:51:22 <E-P> and define what we have to do next
11:51:49 <E-P> #topic QA tools
11:52:34 <E-P> #info Using QA tools from MeeGo, like testrunner-lite, OTS, test-definition, Testplanner, QA-Reports
11:52:55 <E-P> timoph: something to add briefly?
11:54:58 <E-P> for the last, lets discuss what to do next
11:55:29 <lbt> Deciding on a per-package basis what 'test plan' to run.
11:56:09 <E-P> we can choose couple of packages first and start the automation with them
11:56:17 <lbt> yep - I'd like to pick a package, setup some tests for it and run them
11:56:26 <E-P> #topic Next in QA
11:56:49 <lbt> so what do we need to achieve that?
11:57:02 <lbt> A Mer VM HA ?
11:57:07 <Stskeeps> wouldnn't it be better to start with some of the mcts/blts/mwts?
11:57:40 <E-P> mwts requires Qt, and the tests are not tested with Qt5
11:58:08 <Stskeeps> not an issue atm
11:58:09 <lbt> (please expand acronyms in writeup)
11:58:45 <E-P> mcts (meego core test suite)
11:58:54 <E-P> blts (basic layer test suite)
11:59:01 <E-P> mwts (middleware test suite)
11:59:17 <E-P> or something like that
11:59:21 <lbt> do these need a 'device' to run on
11:59:23 <lbt> ?
11:59:30 * Stskeeps has to go, bbl
11:59:36 <lbt> ie the Mer VM HA ?
11:59:46 <E-P> not all of them, you can run them in the chroot
11:59:54 <E-P> but results might be weird :)
12:00:47 <lbt> it feels like there are some parallel activities we can have
12:01:31 <E-P> I have tried the mwts and blts tests, and they are working on Mer
12:01:48 <lbt> in the SDK ?
12:01:51 <E-P> yes
12:02:02 <lbt> could you document how to do that?
12:02:09 <lbt> action?
12:02:14 <E-P> I have done it
12:02:29 <E-P> http://wiki.merproject.org/wiki/Quality/ExecuteTests
12:02:53 <iekku> :D
12:02:54 <lbt> info  then :D
12:02:59 <E-P> #info http://wiki.merproject.org/wiki/Quality/ExecuteTests
12:03:00 <E-P> :)
12:03:35 <E-P> one task could be to investigate what is needed for mer VM HA
12:04:09 <lbt> so that comment on the test plan...
12:04:25 <lbt> "sudo zypper in blts-bluetooth-tests"
12:04:48 <lbt> how do I know that blts package exists?
12:04:58 <lbt> and is that the only bluetooth test package?
12:05:30 <E-P> that is one problem, how we add the tests to the image
12:05:50 <lbt> so I would like to see some kind of outline off the test:package mappings
12:06:18 <lbt> so that's a design/document task
12:06:31 <E-P> yes
12:06:51 <Stskeeps> lbt. test:package usuaully doesn't work, one test usually goes through multiple layers
12:07:07 <Stskeeps> test coverage etc
12:07:22 <Stskeeps> and one change can affect many  packages
12:07:25 <lbt> *nod* ... comes back to the same point... how BOSS makes a test-plan
12:09:28 <E-P> we need to define somewere what is needed to execute when a package changes
12:10:04 <E-P> example in the spec file or in another xml or similar
12:10:54 <lbt> *nod*
12:11:07 <E-P> but already over time
12:11:21 <lbt> I think this is an action/info area for discussion in #mer/ml
12:11:39 <E-P> lets have it as action
12:12:07 <E-P> #action We need to define how the BOSS makes a test-plan
12:12:24 <E-P> so much issues to discuss and so less time
12:12:33 <lbt> yep - it's a crucial area
12:12:56 <E-P> I will propose next meeting time to mailing list, ok?
12:13:07 <lbt> yep - this has been useful
12:13:17 <Stskeeps> thanks for a good meeting
12:13:23 <E-P> #action Propose next meeting time
12:13:36 <E-P> thanks for everyone
12:13:48 <lbt> ty
12:13:56 <E-P> lets continue the chat at #mer
12:14:06 <E-P> #endmeeting