12:07:39 <Stskeeps> #startmeeting Release management meeting 20/3/2012
12:07:39 <MerBot> Meeting started Tue Mar 20 12:07:39 2012 UTC.  The chair is Stskeeps. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings.
12:07:39 <MerBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
12:07:45 * Sage_ stops the calendar reminder
12:07:51 <Stskeeps> so, long story short last release was a nightmare :)
12:08:02 <Stskeeps> and has shown why we need properly automated processes
12:08:04 <Stskeeps> .. and more disk space
12:08:33 <Sage_> how much does one release take disk space btw?
12:08:39 <lbt> ~7Gb
12:08:56 <lbt> and transient processing takes more
12:09:07 <Sage_> ok
12:09:43 <Sage_> Should we make policy that after a 1 month the snaphots/prereleases are removed and only releases are kept?
12:10:11 <lbt> https://bugs.merproject.org/show_bug.cgi?id=175
12:10:39 * Sage_ cc's
12:10:50 <lbt> so yes, I agree completely
12:11:02 <Stskeeps> makes sense
12:11:13 <Stskeeps> but either way, with CI coming in to play, we need to get more space for the release builder
12:11:28 <lbt> we actually have a lot of disk space
12:11:35 <lbt> it's just not well used
12:11:39 <Stskeeps> right
12:11:54 <Sage_> don't some of the hots have like 2x3TB disks?
12:11:58 <Sage_> hosts*
12:12:02 <lbt> some of that is down to me not cleaning up infra
12:12:14 <Stskeeps> so, to the point -- how do we best get started implementing a saner release process?
12:12:25 <Stskeeps> oh, before that
12:12:40 <Stskeeps> i managed to screw up Core:armv7hl on sunday and i successfully resurrected it with latest mer release
12:12:52 <Stskeeps> just by building
12:13:03 <Stskeeps> which was a good experience as we know Mer is resurrect-able in case we get shut down for whatever reason
12:13:07 <Sage_> REL & CI counts ok?
12:14:04 <Stskeeps> yes, to my surprise, however, i think merds delivers wrong information
12:14:15 <Stskeeps> http://releases.merproject.org/~prjfetcher/ver.txt , CI_CNT should definately not be 1 at all times
12:15:00 <lbt> #info We made a mistake and lost a part of the Mer release and rather than just restore we were able to verify that a rebuild worked as expected. This is very useful from a DR perspective.
12:15:53 <Stskeeps> ish, yeah
12:16:29 <Stskeeps> lbt: do you know how far phaeron got with putting his copy-whole-project patches on github?
12:16:47 <lbt> I know it was close
12:17:17 <lbt> couldn't say if it was finished though
12:19:21 <lbt> I would like to tidy up the infra as a high prio task
12:19:38 <lbt> we shouldn't really be running on the phosts
12:19:47 <Stskeeps> running?
12:19:57 <Stskeeps> ah, point
12:20:15 <Stskeeps> so, task, move prjfetcher 'stuff' to another vm?
12:20:23 <lbt> yes
12:20:33 <Stskeeps> makes sense
12:20:50 <Stskeeps> and releases.* hosting too
12:20:58 <lbt> so probably a release staging VM
12:21:08 <lbt> and a release distribution VM
12:21:30 <Stskeeps> ok
12:21:42 <Stskeeps> #info move prjfetcher releasing scripts to release staging VM
12:21:53 <Stskeeps> #info move releases.* www+rsync to release distribution VM
12:22:10 <Stskeeps> just keep in mind releases.* has gerrit pushing to it
12:22:36 <lbt> *nod*
12:23:14 <lbt> I need to look at where source/git lives too
12:23:39 <Stskeeps> git lives on gerrit and mirrors to releases.* automatically
12:24:18 <Stskeeps> ok, so, process
12:24:22 <lbt> I'm also keeping http://airy:5001/ as the primary docs place
12:24:47 <Stskeeps> i'm going to be a difficult person and ask if we can get that behind a htpasswd'ed url
12:24:58 <Stskeeps> i don't always have my ssh forwarding on
12:25:01 <lbt> Sage_: I'm not sure what access you have to the infra - if you ever want to see anything then just yell
12:25:29 <Stskeeps> so it ends up with me not utilizing that wiki properly
12:25:34 <lbt> Stskeeps: sure, I just never needed it
12:25:39 <Sage_> lbt: atm. I don't think I have access but I'm not really wanting one eithe atm.
12:26:10 <Stskeeps> #info task: airy wiki accessible through password protected url
12:26:35 <Sage_> I'm a bit busy as is anyway so wouldn't benefit anyone. If I don't need the access I don't want one. I'll inform if situation changes.
12:26:36 <Stskeeps> (ideally ssl)
12:26:37 <lbt> Stskeeps: my main concern is that we need to be careful about recording credentials on there
12:26:43 <Stskeeps> lbt: naturally
12:26:45 <lbt> ssl mandatory
12:29:17 <Stskeeps> lbt: i don't even mind ssl client certificates
12:29:18 <Stskeeps> :P
12:29:36 <Stskeeps> either way
12:30:46 <Stskeeps> process.. http://wiki.merproject.org/wiki/Process#New_process_2
12:31:22 <Stskeeps> the idea is that for CI checks we copy from 'master' to a change-specific project, then be able to release that as a full repo
12:31:30 <Stskeeps> so we can run automated tests on it, have vendors test it, etc
12:32:14 <lbt> So in general we are moving all non-OBS-worker stuff off the phosts
12:32:14 <lbt> I may have to take up vgrade's offer of a new phost
12:32:14 <lbt> I'll wait until this afternoon's OBS meeting though
12:32:35 <Stskeeps> ok
12:35:50 <Stskeeps> i guess i should work more on MDS2
12:36:10 <lbt> hmm
12:36:43 <lbt> right now I'm more interested in a release process with more QA
12:37:08 <lbt> I'd say to leave process v2 for a while - it depends on unwritten/deployed code anyway
12:37:24 <lbt> and focus on automation on the existing CI and release process
12:37:36 <lbt> installed BOSS is old apparently
12:37:39 <Stskeeps> i can only say that you need v2 in order to do this
12:37:49 <Stskeeps> i wouldn't be talking about it else
12:38:11 <lbt> fair enough
12:38:29 <Stskeeps> we can't do QA image builds properly against partial repos
12:39:09 <lbt> ah
12:40:06 <lbt> I got this working just fine in Nokia...
12:40:21 <Stskeeps> yeah, well, we're not nokia
12:40:22 <Stskeeps> :P
12:40:47 <lbt> so I'm trying to figure out what's blocking us now that didn't then
12:41:10 <lbt> ks handling ?
12:42:01 <Stskeeps> no, MDS is one part of why
12:42:26 <Stskeeps> and the fact we're exporting things to vendors
12:43:11 <lbt> I see 3 areas
12:43:28 <lbt> CI img/rootfs tests as part of commit review
12:43:55 <lbt> img/rootfs tests as part of release verification
12:44:11 <lbt> possibly img/rootfs tests as part of vendor early-sharing
12:45:40 <Stskeeps> i think img/rootfs is too narrow a focus right now, we need to look ahead to running tests as well
12:46:05 <lbt> on what though?
12:47:11 <lbt> because I said "img/rootfs _tests_"  ... ie create rootfs/img and run a (possibly null) set of tests on it
12:47:21 <Stskeeps> okay, if you don't mind i'll just run through what i see we need, get some terminology on same page, etc, without discussing specific technology
12:47:45 <lbt> sure
12:47:52 <Sage_> creating an image is very nice test already so we avoid fiasco that was done with previuos release :)
12:48:09 <lbt> exactly
12:48:36 <Sage_> tests is next step after that but at least we should do images (no need to boot either) just do image and check that they build ok.
12:48:38 <Stskeeps> a release = project-core consisting of packages, which are pointers to specific sha1's in packages git, + binaries matching these sources
12:49:00 <Stskeeps> you can run image builds against this, or build packages
12:49:08 <Stskeeps> any objections to that definition?
12:49:26 <lbt> it's broad
12:49:27 <Sage_> ps. create image after each submit might be a bit too heavy in process but we should do it before (pre)release at least.
12:49:54 <lbt> it encompasses both the MDS and the http://../releases
12:50:00 <Stskeeps> right
12:50:41 <lbt> so if we split it to rpm release and mds release
12:50:56 <lbt> then we can 'easily' test the rpm release wrt images
12:51:04 <lbt> that's useful and incremental
12:51:15 <Stskeeps> a change = one or more modifications to packages and/or project core description
12:51:30 <lbt> then we can move on to MDS testing (bearing in mind MDS2 isn't even written yet)
12:51:51 <lbt> yep - that's OK
12:51:53 <Stskeeps> oi, MDS2 is written fine for the basic parts
12:51:55 <Stskeeps> :P
12:52:08 <lbt> OK
12:52:29 <Stskeeps> so, old release + change = new release, right
12:52:34 <lbt> Sage_: it's quicker than you imagine to make an image on a fast machine
12:53:16 <lbt> Stskeeps: OK
12:53:40 <Sage_> lbt: under 5mins? I mean that there is need to do at least one image per arch.
12:53:44 <lbt> (but old release + change doesn't mean you have to make a new release)
12:53:52 <lbt> Sage_: worker farm
12:54:38 <Sage_> ok, I know it can be done, but as we don't have too much resources atm. AFAIK I meant that we should put those resources do images if we have better things for them.
12:54:48 <Sage_> if we already have the capacity why not
12:54:56 <Stskeeps> when a change comes in through gerrit, for example, we take an old release, add the change, and we make a 'release' based on that
12:55:20 <lbt> Sage_: we have quite some spare capacity for IMG on the SLX machine
12:55:21 <Stskeeps> to test if it would be valid still
12:55:51 <lbt> Stskeeps: OK - now you're using release in a way that I think will confuse
12:55:56 <lbt> I know what you mean though
12:56:18 <lbt> old release + change =  release candidate (rc)
12:56:31 <Stskeeps> right
12:56:41 <lbt> ie "we could release anytime if we wanted too but we'll only do it every other thursday"
12:56:58 <Stskeeps> that release candidate is then what is sent for testing around the place
12:57:12 <Stskeeps> ABI tests, image creation tests, automated test cases, etc
12:57:33 <Stskeeps> vendor checks..
12:58:43 <lbt> yes
12:59:03 <lbt> vendor-checks is ... interesting
12:59:21 <lbt> and here is where rpm vs mds gives us 2 deliverales
12:59:25 <Sage_> what doe vendor-checks mean?
12:59:42 <Stskeeps> Sage_: we signal to vendors there's a change that they want to QA
12:59:54 <Sage_> ok
13:00:01 <Stskeeps> for example, to the x86 adaptation vendor, please verify that this change doesn't break your systems
13:00:05 <lbt> I'd like to focus initially on getting those rpms available in a state that permits img
13:00:16 <Stskeeps> now, just to tie this in with what i meant with process v2, in order for us to do old release + change = release candidate, we need to use project-copy
13:00:29 <lbt> well
13:00:43 <lbt> we need to ensure access to all those rpms
13:01:28 <lbt> which may be as horrid as running create-rc-release.sh against the transient OBS project
13:01:45 <lbt> so project-copy is certainly a solution
13:01:55 <lbt> and I agree it's a good solution
13:02:08 <lbt> but it blocks us until it's deployed
13:03:10 <Stskeeps> it does, but we can't work with 'blob' obs project practically -- you saw how the sync script nuked our armv7hl stuff as OBS protocol doesn't relay information on added packages when doing linked projects
13:03:31 <Stskeeps> ie, right now, i manually have to linkpac and rdelete packages upon update
13:03:34 <Stskeeps> it's not sustainable
13:03:49 <Stskeeps> update=upon new package, or delete package i mean
13:04:56 <Stskeeps> lbt: ok, instead of release, read "build"
13:05:49 <lbt> yeah - just being careful on meanings
13:07:10 <Stskeeps> and project-core should be managed by process instead of manual work
13:07:37 <lbt> so ... I was just noticing the insertion of a solution into the "no specific tech" part
13:07:52 <Stskeeps> i just wanted to tie that in in order to explain why it's a problem
13:08:06 <Stskeeps> and why we need this in order to continue
13:08:28 <lbt> yep - so that part of the process (ie create an image after a trial build) needs to handle new+deleted packages
13:08:48 <lbt> that's an issue and one solution is project copy
13:09:06 <Stskeeps> on top of that, we need to support topic branches
13:09:09 <lbt> (and I think we've discussed that it's the best solution)
13:09:23 <Stskeeps> ie, changes that require other changes
13:09:46 <lbt> new terminology alert
13:11:28 <lbt> also .... looking at the time and wondering if we should go back to the RM meeting and then continue this discussion in #mer straight after?
13:12:05 <Stskeeps> this is fairly RM :P but we should close RM meeting soon anyway
13:12:47 <lbt> agreed - just worried it may overshadow other things - if there are any?
13:12:58 <Stskeeps> we have plans from last meeting at least
13:13:00 <lbt> and it's a huge and really important topic :)
13:13:08 <lbt> (releases and stuff I mean)
13:15:05 <Stskeeps> hmm, where was the MINT github?
13:15:14 <lbt> So Qt5 and Python2.7 next release
13:15:27 <Stskeeps> right, if possible
13:15:33 <Stskeeps> eglibc 2.15 is getting tested through atm
13:15:47 <lbt> https://github.com/Merproject ?
13:16:15 <Stskeeps> https://github.com/MeeGoIntegration was looking at
13:16:31 <lbt> oh yes
13:17:35 <Stskeeps> let's wrap up now and continue discussion after you've done the VM moves
13:18:05 <Stskeeps> as that's prerequistes for anything anyway
13:18:47 <lbt> OK - that'll take a while too
13:18:55 <Sage_> I would like to see qt5 packaging already in review so that could be commented
13:18:58 <lbt> both volume of data and planning
13:19:01 <Sage_> otherwise it goes quite close
13:19:02 <Stskeeps> Sage_: yes
13:19:13 <Sage_> at least the qt5 core or what ever
13:19:16 <Stskeeps> at least qtbase as a start
13:19:20 <Sage_> yes
13:19:29 <Sage_> IMO should be qt5-base or something
13:19:46 <lbt> I will get an SB2 SDK out before starting the VM moves too
13:19:58 <Stskeeps> Sage_: qt5base
13:20:15 <lbt> I also have to prep for this afternoons OBS meeting
13:20:22 <Stskeeps> yep
13:20:26 <Stskeeps> thank you all for coming
13:20:53 <lbt> tt
13:21:02 <lbt> ty even :)
13:21:13 <Stskeeps> #endmeeting