11:00:56 <E-P> #startmeeting Mer QA meeting 19/4/2012
11:00:56 <MerBot> Meeting started Thu Apr 19 11:00:56 2012 UTC.  The chair is E-P. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings.
11:00:56 <MerBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
11:01:11 <E-P> #topic Follow up and current status
11:01:35 <E-P> The last meeting minutes are available at http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.html
11:02:09 <E-P> We had two actions from the last meeting, both for me
11:03:01 <E-P> I wrote up some basic stuff how the QA was done in Maemo
11:03:37 <E-P> #info Maemo QA process described in http://wiki.merproject.org/wiki/Quality/Test_processes
11:04:34 <E-P> #info timoph submitted first QA tools to Mer:Tools:Testing
11:04:52 <E-P> no git repositories created yet
11:05:15 <lbt> I can help with them if Stskeeps is busy
11:05:53 <E-P> lbt: nice, we would need repositories for testrunner-lite and test-definition
11:06:03 <E-P> under the mer-tools I assume?
11:06:11 <lbt> makes sense
11:07:13 <E-P> should we have an action about that?
11:08:26 <lbt> no, doing it
11:08:48 <E-P> thanks
11:08:58 <lbt> http://gitweb.merproject.org/gitweb?p=mer-tools/testrunner-lite.git;a=summary
11:09:33 <E-P> lbt: how we should move packages from the Mer:Tools:Testing to Mer:Tools?
11:09:42 <lbt> good question
11:09:56 <lbt> ideally we follow some kind of process
11:10:09 <lbt> in practice we copypac them when we feel it's sensible
11:10:25 <lbt> or SR
11:11:04 <lbt> the SDK/Tools space is not as far forward as Core in terms of release process
11:11:27 <lbt> or even patch/review
11:11:43 <lbt> dcthang: http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-19-11.00.log.txt
11:11:44 <E-P> nod, so me or timoph can create SR for those
11:11:48 <lbt> yes
11:12:04 <E-P> ok
11:12:16 <dcthang> lbt: thanks
11:12:33 <lbt> and we should work as part of a Tools group to get ourselves (ie me) organised
11:13:05 <E-P> #info Git repositories created for testrunner-lite and test-definition under mer-tools
11:13:26 <E-P> I agree
11:14:13 <E-P> is Stskeeps or timoph here?
11:14:24 <E-P> they could say something about the eat-package
11:14:45 <iekku> Stskeeps is having limited connectivity to internet today, not sure if he can participate to meeting
11:14:53 <E-P> as far as I know, it is done and submitted to Mer:Tools:Testing
11:16:24 <E-P> https://bugs.merproject.org/show_bug.cgi?id=308
11:16:59 <E-P> #info Initial eat package has been submitted to Mer:Tools:Testing
11:17:24 <E-P> Anything else?
11:19:20 <E-P> if not then moving on
11:20:05 <E-P> #topic Test package mapping
11:20:14 <E-P> #info http://wiki.merproject.org/wiki/Quality/Test_processes#Test_mapping
11:20:44 <E-P> we had little chat about this one yesterday
11:21:07 <E-P> and I tried to add all the stuff to the wiki
11:22:00 <E-P> Do we agree that we should do the mapping to a file and store the file in the git?
11:22:12 <E-P> at least for now
11:22:17 <mdfe_> This mapping is needed to create a testplan?
11:22:24 <E-P> mdfe_: yes
11:22:29 <mdfe_> :)
11:22:54 <mdfe_> looks good to me
11:24:10 <niqt> testplan is good thing
11:24:13 <lbt> seems like a sensible start
11:24:15 * Stskeeps is sortof here now
11:24:43 <mdfe_> it is a adaption from a allready working  test mapping?
11:25:20 <E-P> no, as far as I know
11:25:38 <mdfe_> so we have write all from scratch?
11:25:54 <Stskeeps> the idea is that test 'planner' will then walk the dependency tree and then give a test plan?
11:26:02 <Stskeeps> so far example if glibc changes, everything will test
11:26:40 <Stskeeps> mdfe_: there's already some test cases, but proper linux tests are in demand
11:26:41 <mdfe_> like the OBS works to calculate the rebuild tree?
11:26:48 <E-P> do we want to write the package dependencies to the mapping file or can we get them ex. from OBS?
11:27:08 <lbt> on the mapping side - it's important to be able to easily access the metadata about a test. Considering some of them have lots of test data I'd like some way to externalise that
11:27:34 <lbt> err, sorry, was going to wait to post that , hit return by mistake
11:28:37 <Stskeeps> so what we are trying to do with this test mapping, is how to decide what to test when X,Y,Z packages change?
11:28:43 <Stskeeps> just to fit it in context
11:28:48 <lbt> E-P: I think dependencies should be specified in the test, not the package
11:29:27 <lbt> OBS can act as a tool to report that
11:29:35 <lbt> so can any other .spec analyser
11:29:51 <lbt> Stskeeps: yes
11:30:14 <E-P> lbt: I meant that, if glibc changes, will the test planner get only glibc package name or list of package where the glibc is effecting?
11:30:37 <lbt> Stskeeps: and we need both dependency (ie it's a useful test) and constraint (ie only runnable on device or takes 3hrs to run)
11:31:24 <lbt> E-P: I'd say all packages that got rebuilt
11:32:01 <E-P> lbt: ok, I would like to have it like that too
11:32:37 <E-P> then the mapping file describes what tests to run, in what order and constraints?
11:33:03 <E-P> or should the constrains be in the test package, in the tests.xml?
11:33:23 <E-P> s/constrains/constraints
11:33:45 * Sage agrees with lbt that all package that got rebuilt
11:34:03 <E-P> the tests.xml has the basic information about the test case, like domain, timeout etc
11:34:05 <Sage> however, that isn't always enough, because some packages have runtime (install time) dependencies that are not causing rebuilds.
11:34:08 * Stskeeps makes a casual remark that sometimes behaviour in a lib can change while the user does not..
11:34:17 <lbt> Sage: yes, exactly
11:34:47 <lbt> we need API dependency too
11:35:10 <Sage> hmmp... install all packages that got rebuild and run tests to all installed packages?
11:35:21 <E-P> we can define them in the mapping file?
11:35:22 <lbt> well, this is perfection
11:35:55 <lbt> E-P: I *think* constraints go in the XML - I could be wrong
11:35:55 <Stskeeps> i guess one of the downsides we have atm is that we don't have a sane featurezila
11:35:57 <Sage> I would prefer some automatic way or otherwise there is more maintenance in test mappings.
11:35:59 <E-P> if the package x changes, and it has runtime dependencies to package y, then having in the mapping file that when x package changes, run tests for y
11:36:14 <Stskeeps> zilla, as to understand what we need to test
11:36:38 <lbt> E-P: not sure that's in the mapping file
11:37:33 <Stskeeps> from an entirely architectual pov, we might be going into too much specifics if we're doing something on a package level just - mapping would be an optimization?
11:37:49 <Stskeeps> to tests based on how architecture looks, let's say, 'test connectivity domain'
11:38:08 <lbt> Stskeeps: I wondered if tests should be pkgconfig based
11:38:38 <Stskeeps> http://wiki.merproject.org/wiki/Architecture as a reference
11:39:25 <E-P> the tests have domain and feature areas attribute, those can be used for
11:39:27 <Stskeeps> also, our first priority should be to test run-time architecture, not build-time
11:39:34 <Stskeeps> which narrows our scope a lot
11:40:02 <E-P> yes
11:40:18 <lbt> build-time is (only?) used to indicate which tests need running
11:40:43 <E-P> I would see it like that for now
11:40:46 <lbt> ie package in donnectivity domain changed - run those tests
11:41:05 <Stskeeps> right, just saying that i couldn't care less from a test pov if LaTeX fails to produce correct documentation ;)
11:41:08 <Stskeeps> just as an example
11:41:59 <lbt> so at the mer level we'd do capability tests
11:42:17 <lbt> at the vendor level they'd do feature/functional stuff
11:42:19 <mdfe_> +1 for to test run-time architecture
11:43:25 <Stskeeps> also, i'm welcome to input on what i need to specify in architectual documentation in order to help QA along
11:44:04 <lbt> well, to change the focus somewhat
11:44:11 <lbt> what tests and specs do we have?
11:44:36 <lbt> (and what do we want)
11:45:10 <Stskeeps> we have http://wiki.meego.com/Quality/TestSuite/MCTS  for example
11:45:35 <Stskeeps> and http://wiki.meego.com/Quality/TestSuite/MWTS_Asset_Descriptions
11:45:54 <E-P> those are good to start from
11:46:14 <Stskeeps> we also have free hands to do things right :P
11:46:19 <mdfe_> do anyone knows the meego hudson 'fft' aka 'fast-feedback-testing' plugin?
11:46:35 <E-P> I think timoph knows something about that
11:46:57 <mdfe_> I also worked with it for some time
11:47:03 <E-P> so trying wrap up some kind of conclusion
11:47:17 <Stskeeps> on top of that, we have a lot of build-time test cases for components
11:47:19 <mdfe_> for me it was nice usefull tool
11:47:29 <Stskeeps> and sometimes test cases for runtime
11:48:13 <E-P> the mapping file defines what tests to run (test packages) when package changes?
11:48:23 <E-P> or when making a release
11:48:54 <Stskeeps> a release can also be seen as a delta of package changes, so it's up to us to delimit it based on it being integration instead
11:50:14 <E-P> that is pretty much the same as the 'feature set' what I wrote to the wiki
11:50:20 <Stskeeps> :nod:
11:50:23 <lbt> E-P: correct - the release being a collection of packages
11:51:13 <lbt> realistically the main trigger for a test is a package change - possibly at the rpm level, not source level
11:51:27 <E-P> right
11:51:42 <lbt> the decision about what tests to run must be made by the person writing the tests
11:51:47 <E-P> the next step is to define a first version of the mapping file
11:51:52 <lbt> and hence that gets put in the mapping
11:52:14 <lbt> but that should be a first-order thing
11:52:35 <lbt> ie they should not worry about testing indirect dependencies
11:53:02 <E-P> lbt: like you said, the test packages might have hundreds of tests, so in the mapping file we can specify which one to execute
11:53:30 <E-P> example, using tr-lite filtering
11:53:30 <lbt> OK - so mapping is done at the low level (like pkgconfig again - makes sense)
11:53:44 <Stskeeps> when we have a moment, i'd like to quickly talk about EAT as i came a bit late to meeting
11:53:48 <lbt> and at some level we know which pkg provides a test
11:54:00 <lbt> as a seperate matter, the decision to run the test is based on constraints
11:54:16 <lbt> if tests were instant and free we'd run them all
11:54:31 <E-P> Stskeeps: I will wrap this up, then we can go back to EAT
11:54:37 <lbt> so that's not part of the mapping
11:54:37 <Stskeeps> ok
11:54:54 <lbt> but it is used, with the mapping, to make a decision
11:55:16 <lbt> what I'm not sure about is whether the constraint is written into the test
11:55:20 <E-P> #info Test mapping defines what tests to run and in what order
11:55:32 <lbt> not sure about order?
11:55:40 <lbt> I'd say not actually
11:55:42 <Stskeeps> order can matter, we might want to sanity-check first
11:55:48 <Stskeeps> before running longer tests
11:55:54 <lbt> I'd say that constraints would ...
11:55:56 <E-P> lbt: mainly like top-to-down order
11:55:59 <lbt> yes, +1 stsks
11:56:28 <lbt> yeah - so order comes from constraint, not mapping
11:56:51 <E-P> ok, feel free to info
11:57:05 <lbt> #info order comes from constraint, not mapping
11:57:09 <lbt> doorbell
11:57:46 <E-P> #info changed package list is delivered to the 'test planner'
11:58:16 <E-P> or is it better to call rebuild list
11:58:54 <lbt> change is what we care about I think
11:59:06 <E-P> I can take an action to define a draft from the mapping file
12:00:56 <E-P> #action E-P create a mapping file draft
12:01:40 <E-P> if something missing, feel free to #info
12:02:48 <lbt> #info Tests need to provide constraints which are used to decide when to run them (eg need certain time, network connectivity, run on a real device etc)
12:04:23 <E-P> thanks, anything else
12:04:33 <Stskeeps> quick EAT summary?
12:04:39 <E-P> Stskeeps: go ahead
12:05:28 <Stskeeps> #info re-architected EAT -- to add test enablement to a device, add Mer:Tools:Devel, add eat-device in %packages and su -c 'eat-add-device-key' - root and for each of the users needing test login
12:06:01 <Stskeeps> #info logging in with EAT key, if a X session, will give you environment as inside the X session, Xauth + dbus sesssion etc
12:06:21 <Stskeeps> X session user, that is
12:06:43 <E-P> sounds good
12:06:55 <Stskeeps> #info on host (sdk) you need to install eat-host and /usr/bin/eat-install-host-key to install the private EAT key
12:07:16 <Stskeeps> #info tested with testrunner-lite, testrunner-ui needs a UI part for selecting SSH private key
12:07:54 <E-P> could you file a bug about that?
12:08:10 <Stskeeps> ok
12:08:32 <Stskeeps> it's a good start for starting to play with tests on our devices
12:08:48 <Stskeeps> as it's really easy to test enable a device
12:08:56 <Stskeeps> no matter the UI
12:08:57 <mdfe_> sounds good
12:09:16 <E-P> is the eat-device in the Mer:Tools:Testing?, you wrote Mer:Tools:Devel
12:09:31 <Stskeeps> yes, that's what i meant
12:09:44 <E-P> nod
12:10:16 <E-P> last thing
12:10:37 <E-P> is it ok that we will have regular QA meeting at this time every week?
12:10:45 <Stskeeps> i'm ok with that
12:10:54 <lbt> +1
12:11:10 <mdfe_> +1
12:11:11 <jbos> +1
12:11:20 <timoph> could be an hour later?
12:11:25 * timoph just joined
12:11:42 * lbt is OK with that too
12:12:04 <jbos> fine with me.
12:12:09 <Stskeeps> ok with me too
12:12:13 <mdfe_> +1
12:12:16 <E-P> fine for me too
12:12:24 <niqt> +1
12:12:34 <E-P> ok, I will add the time to wiki and post the timing to ml, so 12 UTC it will be
12:12:50 <E-P> are we done?
12:12:53 <Stskeeps> #info bug 315 (testrunner-ui ssh key)
12:12:53 <lbt> so E-P, you're doing mappings... is anyone doing anything on constraints?
12:12:54 <timoph> cool. thanks for being flexible
12:13:41 <lbt> I'd like to find some way to say which tests are run at build time in just a chroot, and which on a VM and which need a device
12:14:37 <lbt> if not I'll just dig in when I get around to it :)
12:15:09 <timoph> are we collectiong requirements somewhere? wiki, etc.
12:15:16 <timoph> that sounded like one :)
12:15:35 <lbt> close to http://wiki.merproject.org/wiki/Quality/Test_processes#Test_mapping  ?
12:15:53 <E-P> we could have those as task bugs
12:16:10 <timoph> E-P: true. easier to track
12:16:41 <E-P> lbt: could you please create a task bug?
12:16:45 <lbt> ok
12:17:03 <E-P> we can continue the discussion and see what could be good way of doing that
12:17:19 <E-P> +later
12:17:37 <lbt> *nod*
12:18:21 <E-P> ok
12:18:28 <E-P> I think we are done for now
12:18:40 <iekku> :)
12:18:48 <E-P> thanks for everyone
12:19:18 <lbt> ty
12:19:28 <iekku> E-P, you will send the meeting minutes?
12:19:39 <iekku> should the new time be #info ?
12:19:49 <E-P> oh, true
12:20:25 <E-P> #info The weekly QA meeting will be on Thursdays at 12 UTC
12:20:30 <E-P> now good? :)
12:20:54 <E-P> I will send minutes too and notifying about the time
12:21:42 <E-P> #endmeeting