12:01:21 <E-P> #startmeeting Mer QA meeting 12/4/2012
12:01:21 <MerBot> Meeting started Thu Apr 12 12:01:21 2012 UTC.  The chair is E-P. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings.
12:01:21 <MerBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
12:01:55 <Stskeeps> o/
12:01:59 <E-P> Welcome all to Mer's second QA meeting
12:03:04 <E-P> Meeting agenda and last meeting's actions can be found from here http://www.mail-archive.com/mer-general@lists.merproject.org/msg00414.html
12:03:24 <iekku> thanks :)
12:03:37 <E-P> any additions/changes to agenda?
12:03:42 <Stskeeps> looks fine to me
12:03:52 <iekku> +1
12:04:26 <timoph> o/
12:04:59 <vgrade> +1
12:05:12 <E-P> ok, let's start with the process
12:05:19 <Stskeeps> http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, for those late joining
12:05:38 <E-P> #topic QA Process
12:05:52 <E-P> we had good discussion last time with lbt
12:06:54 <E-P> and one action was filed about the how BOSS makes the test plan
12:07:13 <Stskeeps> :nod: there hasn't moved much on that side sadly to my knowledge
12:07:35 <Stskeeps> the current process is fairly simplified and done on a per-change basis
12:08:05 <Stskeeps> and a lot of manual work done to make a release
12:09:12 <Stskeeps> which on the other side means we can do things properly with QA :)
12:09:16 * Sage starts following
12:09:30 <E-P> when a change is made (via gerrit), that now triggers automatically package rebuild in OBS?
12:09:50 <E-P> or does that also require some manual work?
12:10:55 <Stskeeps> when a change is made in gerrit, it (automatically) will first check if the changed package itself will build, then run a build-test with the change package with all the other packages there as well
12:11:06 <Stskeeps> so it will rebuild any packages that depend on the package
12:11:39 <Stskeeps> when that's done, it'll report back to gerrit
12:12:14 <Stskeeps> an example can be http://review.merproject.org/314 , click "expand all"
12:12:42 <Stskeeps> when we speak of QA process, is there older models that were in use in meego we can use as reference to understand it better?
12:13:11 <Stskeeps> to gain terminology, etc
12:13:57 <E-P> there are some terminology and process description in the wiki.meego.com
12:14:15 <timoph> sadly the meego process never really took flight. the pieces were there but not really put together so I'd say we can agree on the terminology as we go
12:14:25 <E-P> yep
12:15:35 <lbt> sorry I'm late
12:15:57 <Stskeeps> http://wiki.merproject.org/wiki/Quality/Terminology helps at least
12:16:01 <E-P> lbt: welcome
12:16:04 <lbt> to be clear on the test plan thing...
12:16:09 <Stskeeps> http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, for those late joining , mdfe
12:17:17 <lbt> the meaning is "what logic should BOSS use to decide what tests to run"
12:17:33 <timoph> good question?
12:17:43 <timoph> tests depending on the package?
12:17:57 <lbt> so that's something that should be designed and expressed by the group and (much) later, implemented in BOSS code
12:18:16 <lbt> yes - it relates to the package naming conventions mentioned on the ml
12:18:36 <lbt> personally I thing we should not use package naming - it's too simplistic
12:18:41 <Stskeeps> i think we need to possibly look at higher level view first, ie, change and 'release' testing
12:18:51 <lbt> exactly
12:18:57 <E-P> agree
12:18:57 <timoph> package groups?
12:19:09 <timoph> or meta
12:19:15 <Stskeeps> we do have a mer architecture too, you know
12:19:15 <lbt> lets just say mapping
12:19:15 <Stskeeps> :P
12:19:25 <timoph> :D
12:19:28 <Stskeeps> http://wiki.merproject.org/wiki/Architecture
12:20:05 <lbt> *nod* ... and that's another area of testing
12:20:39 <timoph> test-definition lets you define component and other information to it
12:20:40 <mdfe_> May I ask how the BOSS is connected to a device, and how make sure the Device OS is in sync with the latest release?
12:21:05 <lbt> mdfe_: lets leave BOSS out of it for now
12:21:10 <mdfe_> k
12:21:29 <lbt> we should say what steps we would ideally like to take... then we choose tools to execute steps
12:21:30 <Stskeeps> mdfe_: the general idea is that something would be creating an image, which then gets installed on your device, a test pc will then connect to device and run tests
12:21:37 <lbt> BOSS will orchestrate those steps
12:21:39 <E-P> like Stskeeps said, let's first define the higher level view
12:21:53 <E-P> what kind of testing we want to have, and when they are executed
12:21:56 <Stskeeps> at the moment my biggest weakness is that i can't verify that a change will cause regressions
12:21:57 <mdfe_> +1
12:22:08 <timoph> E-P: yep
12:22:19 <lbt> so mdfe_'s point is that there should be a process step that says "put the right OS on a device"
12:22:21 <Stskeeps> and it gives very late breaking problems
12:22:55 <E-P> from the Mer core point of view, we would have: a change and release testing?
12:23:32 <Stskeeps> right, change to justify it's OK to merge something, release testing to justify the combination of changes is working?
12:23:51 <lbt> E-P: the main reason for differentiation is the resource demands of testing
12:24:10 <lbt> eg we can't run a week-long soak test before each commit :D
12:24:44 <timoph> yep. that should be clear
12:24:50 <lbt> so realistically change tests are lighter than release tests but IMHO only for that reason
12:25:10 <E-P> do we want to test in the release testing something else than only changed packages?
12:25:28 <lbt> ideally yes
12:25:39 <lbt> the reason is that changes only represent build dependencies
12:25:44 <E-P> lbt: yes, change tests should be light and fast to execute
12:25:56 <lbt> API dependencies are not easily seen in this way
12:26:12 <mdfe_> there is a need for both
12:26:13 <lbt> eg if a service changes its dbus API
12:26:56 <E-P> in meego we had a static tests for every release
12:27:06 <E-P> and then "feature pack" set for changes
12:27:11 <lbt> note: we must be careful to define some larger scope but deliver small steps
12:27:29 <lbt> yep - that makes sense E-P
12:29:05 <E-P> the static tests, eg. core tests, will test the overall status of the release
12:29:39 <Stskeeps> http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, notmart
12:29:41 <lbt> I'd like tests (test packages?)  to be able to define test dependencies
12:29:53 <Stskeeps> in which sense?
12:30:04 <Stskeeps> 'don't bother running these if the other ones don't pass'?
12:30:09 <lbt> no
12:30:20 <lbt> if any package that the test TDs on changes, run the tests
12:31:29 <mdfe_> the test package should contain related unit tests for device deployment?
12:31:30 <lbt> (yes, it's just a build-dep in one sense)
12:31:46 <Kaadlajk> in maemo we had XB-Maemo-CI-Packages and XB-Maemo-CI-Stage fields in control file of the test package
12:32:14 <Kaadlajk> which defined what packages the tests are for
12:32:18 <Kaadlajk> and on what stage they should be run
12:32:19 <notmart> Stskeeps: thx, sorry i was away ;)
12:32:26 <E-P> Kaadlajk: I remember that, but that required a big database to work :)
12:32:30 <lbt> Kaadlajk: yep, sounds reasonable
12:32:46 <lbt> E-P: that also sounds right :/
12:33:08 <Stskeeps> that's however a bit of optimization though, ie, optimization on top of 'what we ought to run if it was a full QA run'
12:34:09 <lbt> so I can see a need for a service which does something like : Input=changed package list Output=ks of image with tests to run
12:34:30 <Stskeeps> from mer process POV, we need to strike at change (earliest we can check problems), and the various 'builds'/releases of full mer
12:34:44 <lbt> yep
12:34:57 <Stskeeps> in mer, we have prereleases and full releases
12:35:18 <Stskeeps> each of these being a 'build', perhaps with some smaller releases leading up to it
12:36:02 <Stskeeps> so that also gives input to how we can test/make tests
12:36:06 <Stskeeps> +when
12:36:26 <lbt> OK - but there is no reason to treat each commit any differently from a final build other than to filter which tests to run
12:36:34 <Stskeeps> :nod:
12:37:42 <lbt> Maybe more detail : Input=changed package list & position in overall CI process (DBC, snap, pre, release) ;   Output=ks of image with tests to run
12:37:51 <Stskeeps> E-P: btw, just a random note: if you see anything that for example me, or others's areas haven't done yet / not doing yet / should be doing to advance QA ability, you're more than welcome to tell us to do that :)
12:38:10 <Stskeeps> i know that our current process definition / release methods may be blocking things
12:39:30 <E-P> Stskeeps: I definitely will
12:40:21 <Stskeeps> so since we've spent 40 minutes on this discussion, what can we conclude for this topic for now?
12:40:27 <Stskeeps> just to get moving on other agenda items
12:40:41 <E-P> #info Mer Core would have change and release testing
12:41:24 <E-P> #info Change testing is lighter than release testing
12:41:52 <E-P> #info Release testing contains test for changed packages + Core test set
12:42:10 <Stskeeps> :nod:
12:42:16 <lbt> is there agreement that the test:package mapping needs more detail than <pkgname>-tests:<pkgname>
12:43:07 <Stskeeps> i think it needs to be done on architectual domains instead, tbh, as sometimes one test will cover many packages
12:43:18 <lbt> exactly
12:43:32 <lbt> it's not just arch though
12:43:37 <lbt> sometimes it will be 1:1
12:43:58 <Stskeeps> :nod:
12:44:13 <E-P> we definitely need some kind of mapping
12:44:14 <lbt> #info The test:package mapping needs more sophistication than <pkgname>-tests:<pkgname>
12:44:20 <E-P> thanks
12:44:23 <Stskeeps> have you peeked at mwts-mcts-blts setup yet, lbt?
12:44:30 <Stskeeps> just to understand how that was done back in meego
12:44:57 <Stskeeps> setup as in, how they were laid out
12:44:59 <mdfe_> does this mean the test package name is to contain the sense of it?
12:45:40 <Stskeeps> mdfe_: we're talking a bit on how we'd know what needs testing / what test suites we need to run, from a core perspective
12:46:23 <Stskeeps> (if i misunderstood, please rephrase question)
12:46:26 <E-P> the mcts test packages has some faults, but we can discuss about them some other day
12:46:29 <Stskeeps> :nod:
12:46:29 <mdfe_> ah, I got it now
12:46:34 <mdfe_> mapping
12:46:39 <mdfe_> ok
12:46:41 <Stskeeps> next topic?
12:46:49 <E-P> anything to add to this topic?
12:47:27 <lbt> not here
12:47:34 <E-P> lets move on then
12:47:40 <E-P> #topic Tools and tests
12:48:03 <E-P> Like mentioned in the last meeting, Meego project provides many good tools
12:48:50 <E-P> one issue is that where do we get tests?
12:49:08 <mdfe_> yepp
12:49:24 <E-P> we can use mcts (Meego Core test suite) tests, but they don't cover all the core packages
12:49:58 <E-P> I think first using packages own tests if they have
12:50:17 <E-P> for example qt has good and wide unittests
12:50:39 <Stskeeps> but also very big
12:50:56 <Stskeeps> part of the problem is that most tests from packages are meant to be run inside the source compilation tree
12:51:01 * lbt raises a question for AOB or next time : how do we handle version control of tests when they're not part of upstream
12:51:01 <mdfe_> for cmake based packages I tried to seperate and package ctest based unit tests for target devices
12:51:21 <lbt> I'm not in favour of running tests at build time
12:51:23 <Stskeeps> and for ARM targets, well, we don't run on arm hardware in practice
12:51:29 <lbt> exactly
12:51:45 <Stskeeps> i'd be interested in innovative ideas on how we can extract those during build process, and run later on actual hardware
12:52:05 <lbt> hack "make" and intercept make test?
12:52:22 <lbt> or innovative *and* sane?
12:52:34 <E-P> very good questions
12:52:41 <Stskeeps> i think one way we can approach all this is a 'week' goal, where we make it possible to add testability enabled images we can actually run simple tests against
12:52:50 <Stskeeps> that gives a better basis for all this work
12:53:15 <Stskeeps> and sets for what tools we need to package, work with
12:54:05 <E-P> that is good idea
12:54:07 <lbt> as another aside... it seems reasonable to be able to say "hmm. let me easily install run the tests for this package on this random image"
12:54:16 <Stskeeps> lbt: yes
12:54:27 <mdfe_> +1
12:54:34 <Stskeeps> so after goal is accomplished, we can add test enablers to existing images such as Nemo, virtual machine images, Plasma Active, etc
12:54:46 <Stskeeps> and run tests from PC onto the devices
12:55:20 <lbt> yup ... <innocent>we could detect memory leaks and things like that</innocent>
12:55:22 <timoph> yep
12:55:25 <E-P> I would like that the tests are own packages and you can just install them with zypper and execute them
12:55:31 <Stskeeps> agreed
12:55:46 <Stskeeps> i'm working a bit on 'eat' today to cover that side
12:55:48 <mdfe_> +1
12:56:54 <iekku> +1 for E-P
12:56:59 <lbt> should these tests live in 'Core' ? I tend to say not (cf tools and -cross ... some comes from core but much doesn't and it's moved to a discrete area)
12:57:20 <E-P> (I have to go soon, but you can continue the discussion without me)
12:57:28 <Stskeeps> :nod: i think we can wrap up soon
12:57:44 <Stskeeps> we have some goals and requirements/ideas, can we #info them?
12:58:00 <Stskeeps> lbt: i think external, as to make tests flexible to include new tools
12:58:10 <Stskeeps> they're obviously built against mer and released alongside
12:58:15 <lbt> *nod*
12:58:20 <E-P> +1 for that
12:58:24 <mdfe_> lbt: In case they are in Core they are always available
12:58:46 <Stskeeps> that's another angle, yes
12:58:54 <lbt> mdfe_: I see that as a slight negative
12:59:39 <E-P> Stskeeps: can we use Mer OBS and git for storing some of the tools and tests?
12:59:39 <mdfe_> In case someone has a bug, he is able to run the test if he like
12:59:52 <lbt> mdfe_: +1
12:59:57 <Stskeeps> E-P: yes, mer-tests/* git is all yours if you need it
13:00:01 <Stskeeps> OBS we can make Mer:QA perhaps
13:00:13 <Stskeeps> we can put it on community obs for now
13:00:19 <Stskeeps> and then more officially release it later
13:00:25 <E-P> great
13:00:28 <lbt> #info tests should ideally be run at a discrete point in the process, not just/always inside the build environment
13:00:57 <lbt> I'd like to package any QA tools into Mer:Tools
13:01:01 <lbt> Tests into QA ?
13:01:12 <Stskeeps> Mer:Tools would be available in SDK?
13:01:14 <E-P> #info tests should be easy to execute, eg using zypper to install them and then just 'executing'
13:01:18 <lbt> yes
13:01:21 <Stskeeps> and Tests not necessarily ?
13:01:28 <lbt> correct
13:01:37 <Stskeeps> though that might be a practicality, if we want to use it as a host
13:01:42 <Stskeeps> the tests come in .xml format, so
13:01:50 <lbt> zypper ar
13:01:55 <Stskeeps> ok
13:01:59 <Stskeeps> or just enabling a repo
13:02:07 <E-P> yep, that is good
13:02:27 <E-P> #info QA tools should go into Mer:Tools
13:02:39 <E-P> #info QA tests should go into Mer:QA
13:03:06 <Stskeeps> :nod:
13:03:37 <E-P> anything to add?
13:03:40 <Stskeeps> anybody want to state their plans within QA area for the next week or so?
13:03:51 <lbt> checking for more info points
13:04:08 <Stskeeps> #info work on bringing 'eat' up to mer/systemd ability
13:04:44 <E-P> I can move some tools and tests to mer git and obs
13:04:50 <lbt> #info innovative ideas welcome on how to extract tests made during the build process, to be run later on actual hardware
13:05:28 <E-P> #action Create git and OBS projects for QA tools and tests
13:05:58 <lbt> so ... can anyone (Kaadlajk?) take actions to write up either proposals or "how it's done elsewhere" notes on the test:package mapping
13:06:12 <lbt> mdfe_: or requirements
13:06:40 <lbt> actually I think we should refrain from proposals for a while yet
13:06:53 <Stskeeps> #info goal for nextg week: we make it possible to add testability enabled images we can actually run simple tests against
13:07:32 <lbt> you have no idea of the stupid rabbitholes that involves 8D
13:07:43 <E-P> :)
13:07:48 <lbt> (but yeah)
13:08:02 <E-P> are we good?
13:08:14 <lbt> *cough* .... volunteers?
13:08:30 <Stskeeps> think so, else we can continue conversation in #mer - if you'd like to do anything, or doing it, just say :)
13:08:44 <E-P> I can write something down how the CI testing was done in Maemo
13:08:58 <lbt> E-P: thanks
13:09:00 <Stskeeps> that can help, gives a vendor perspective
13:09:21 <E-P> hopefully timoph, iekku and Kaadlajk can fill that up then
13:09:21 <lbt> #action E-P to document a little about how the CI testing was done in Maemo
13:09:52 <iekku> don't count on me, i don't have any idea :(
13:10:12 <lbt> #info OBS Publish process should distribute test packages to the right repos (cf -cross)
13:10:59 <E-P> I have to go now
13:11:00 * lbt is all done ... ty
13:11:14 <Stskeeps> thank you
13:11:20 <Stskeeps> (all for coming)
13:11:21 <E-P> let's continue the discussion in #mer and mailing list
13:11:34 <E-P> #endmeeting