09:00:37 #startmeeting CalDAV/CardDAV Test Server Planning 09:00:37 Meeting started Mon Sep 19 09:00:37 2016 UTC. The chair is chriadam|windows. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings. 09:00:37 Useful Commands: #action #agreed #help #info #idea #link #topic. 09:00:46 Hi dr_gogeta86, occirol, lbt 09:00:53 I will stay out of the way as much as possible in this meeting 09:01:02 since I don't know anything about system administration or the Mer infra ;-) 09:01:18 I guess from my POV I would like to see: 09:01:33 1) concrete plan for setting up the first test server instance (preferably owncloud or nextcloud) within Mer Infra 09:01:46 2) a timeline for us to aim for (doesn't have to be concrete timeline, but best-effort ETA) 09:02:01 lbt: I guess I'll hand over to you? 09:02:39 I suppose the first step is to find out what you can provide on the Mer infra, and by when. And then we can ask dr_gogeta86 and occirol whether they can use those resources / VMs to set up what we need to do the caldav/carddav testing? 09:03:42 in terms of infra we use kvm VMs 09:04:21 typically I restrict access pretty heavily for obvious reasons 09:04:41 seems fair :) 09:04:47 so shell access and suchlike to most infra is just not available - never mind root :) 09:05:19 however I think we can set up a more isolated system for this kind of thing 09:05:55 I'd want to use a different VPN to connect these machines together and then setup a gateway/firewall for monitoring and potentially remote control 09:06:41 ie so the community machines wouldn't be able to reach into the mer infra but we could deploy eg a web api to reach out to the community machines to control them 09:07:06 sounds sensible. are there turnkey OSS solutions to do that sort of thing? 09:07:11 or is it going to be painful to set up? 09:07:48 we already have a vpn for all the infra so I 'just' need to setup a new one and then do some iptables on a router/firewall 09:08:18 the point is to get a publically reachable server? 09:08:26 larstiq: yes 09:08:30 k 09:08:39 and one which can have some kind of reset done on it 09:08:56 and, to some degree, share the admin of the service 09:09:07 right 09:09:54 so a community guy would manage the owncloud system and probably have root access (ideally just a user but ...) to do updates, DB setup etc 09:10:14 they'd certainly have admin access but I'd expect shell to be needed too 09:10:29 this model would replicate for every sync service 09:10:55 as I say, we use kvm at the moment to spawn VMs 09:11:02 it's a bit hard to say at this point what kind of access is needed after setting it up 09:11:03 other solutions include docker 09:11:07 you can mod some vagrant scripts to use kvm for reproducable/updatable vm images 09:11:09 as an example of that: https://github.com/tomav/docker-mailserver 09:11:10 larstiq: agreed 09:11:29 which is probably a better idea than using docket 09:11:32 *docker 09:11:55 I'm thinking that we should probably aim to have resetable services rather than resetting VMs 09:11:59 it feels more sane ? 09:12:01 occirol: from your experience with apache+owncloud services, do you have any comments on what sort of access is required in these vms? 09:12:02 lbt: yeah 09:12:45 r0kk3rz: I'd like to discuss your viewpoints on vagrant, later today? 09:12:49 larstiq: do you think this is a good solution for docker-isation? 09:12:55 * larstiq knows of it but not much experience 09:12:56 candidate even 09:13:11 lbt: in my personal opinion, yes 09:13:31 the main thing I can see is pre-population with a known set of data 09:13:44 and I'd *love* to see this as a non-privileged service 09:13:50 * larstiq nods 09:14:16 there are IP issues 09:14:18 lbt: both easily doable I think 09:14:29 network, or intellectual property? 09:14:32 haha 09:14:32 larstiq: yeah if you want, ive not really used it for anything but it seems like a good thing 09:14:33 nw 09:14:47 chriadam|windows: well, setting up apache, owncloud, etc would require a certain level of access to the VMs: installing packages, configuring apache files, and a db access 09:15:07 the whole point of it is generating reproducable environments without causing a maintenance nightmare 09:15:08 r0kk3rz: occirol: dr_gogeta86.... experience with docker ? 09:15:19 owncloud provides https://hub.docker.com/_/owncloud/ 09:15:33 occirol: yeah - that's what I'd like to see inside the docker image 09:15:47 lbt: nop, no experience yet 09:15:52 lbt: i did read a really good docker rant, i should find it 09:16:06 r0kk3rz: ooh, reminds me I have nice one for that too 09:16:33 r0kk3rz: https://circleci.com/blog/its-the-future/ 09:17:17 but if you're just going to use the provided docker things verbatim its probably not so bad 09:17:22 docker is a well known name - I'm happy to look at other solutions but we need to be reasonable - whatever tech we use we must have the skills and they must commit to working on this 09:18:05 sure 09:18:13 * lbt asks chriadam|windows -- can we use IPv6 here? 09:18:34 è1 09:18:37 back on track 09:18:50 shouldn't matter so long as it has an externally resolvable host name 09:18:51 I think 09:18:56 i wanna focus on what we need 09:19:01 or would you need v4 ? ... I'm thinking that we'd need to share a very limited number of v4 addresses to all these services (like 1-4) 09:19:01 like a matrix 09:19:19 lbt we can expose those services behind a reverse proxy 09:19:22 chriadam|windows: yeah - if we can use an aliasing reverse-proxy - but that does bring it's own issues :D 09:19:22 without problems 09:19:33 lbt like ? 09:19:42 dr_gogeta86: well - try doing streaming of large files :) 09:19:51 eg git 09:19:54 of course 09:19:59 or obs logs 09:20:04 but for caldav purpose 09:20:05 or http push 09:20:19 yeah - I'm just mentioning that some stuff could hurt 09:20:22 caldav/carddav "should" only need to support simple GET/POST/PUT 09:20:35 etc 09:20:54 is pretty simple imho 09:20:57 *nod* ... but owncloud gui and ajax and clever bugger stuff 09:21:08 adding a reverse proxy *could* bring some problems 09:21:37 occirol: yeah - just mentioning it - I use that approach for *all* mer services so I know it hurts occasionally 09:21:44 otoh I like it :) 09:21:54 we have not to forget that it will used for testing purposes, so we need to avoid as much as possible "interferences", imho 09:22:09 lbt: I like it too :) 09:22:19 occirol: yes. 09:22:35 even weird stuff like chunking or content negotiation 09:23:28 so .. back to the virtualisation ? 09:23:47 so, we are talking about how many services right now ? 3 (owncloud, nextcloud, radicale) ? 09:24:04 occirol: lets aim for 10+ eventually 09:24:11 also different versions 09:24:32 eventually 09:24:38 also we need to consider that a test needs a dedicated instance doesn't it ? 09:24:48 well, not necessarily. 09:24:51 what happens if 2 users run a transactional test at once? 09:25:00 ie add/modify/delete a contact 09:25:05 same test contact 09:25:05 they'll fail. but I'm not seeing these as CI / automated at this stage 09:25:20 it's more like "developer can run a suite of tests, and get results" 09:25:44 yeah - I'd like to just consider this for the future 09:25:48 sure 09:26:17 ie it would be nice to have a 'give me a hostname to a "personal" caldav server' 09:26:58 it's not too hard as long as we realise it's a requirement 09:28:03 right, it depends on the effort required, I guess. I mean, it would be a 10x improvement to have just a single caldav server which I have full control over (e.g. resetable, which I can build some system tests against) 09:28:29 chriadam|windows: yeah - that should fall out as stage 1 09:28:44 and I'd avoid doing anything fancy so as not to delay it 09:29:10 just some personal thoughts on naming and auto-adding vhosts to the rev proxy 09:29:36 there was also clever mapping scheme 09:29:40 hmm and if we use letsencrypt then throwaway certs... 09:29:43 like map files 09:29:48 dr_gogeta86: ? 09:29:59 ah rev proxy 09:30:01 *nod* 09:30:24 larstiq: r0kk3rz: so ... docker ... any more discussion on that? 09:30:37 any serious alternatives? 09:30:39 i don't think so 09:30:44 lbt: as far as I'm concerned I can start on that as soon as you have a vm and a vpn going 09:31:03 the advantage of Docker is the reproducible env 09:31:06 could I run up a single VM and put docker on it and have many user accounts and have them all manage docker images ? 09:31:09 I' can work with it locally 09:31:29 lbt: yes 09:31:50 so that really sounds like a jfdi for docker then :) 09:31:51 lbt: need to take some care perhaps with clashing ports usage, but with a proxy that becomes trivial 09:31:57 *nod* 09:32:32 * lbt is thinking of an automated on-demand image deployment at some point 09:32:33 lbt first we need the image tailored 09:32:39 yep 09:33:05 We need at least container for different owncloud versions out there 09:33:13 from 7 to 9 ( nextcloud ) 09:33:18 what OS on the VM ... Debian OK ? (please) 09:33:29 better without systemd 09:33:33 :-D 09:33:37 really? 09:33:46 systemd is fine :P 09:33:50 systemd inside container can be a pain in the but 09:33:51 I can believe that from a cgroups PoV 09:33:56 dr_gogeta86: we can start with https://github.com/docker-library/owncloud as a base 09:34:16 dr_gogeta86: but outside systemd is fine 09:34:21 sure 09:34:35 but i think to another problem for people like chriadam|windows 09:34:36 lbt: Debian on the VM please :) 09:34:43 a simple way to check logs 09:34:45 good 09:34:51 and enter into the machne 09:34:55 dr_gogeta86: right 09:35:13 we are going to build an infra for developers 09:35:19 not for sysadmins 09:35:20 dr_gogeta86: do you have a solution or do we need to think about that a bit? 09:35:25 keep in minnd 09:35:27 *mind 09:35:27 * larstiq nods 09:35:36 i need to check those docker files 09:35:51 there are docker organisers like kubernetes 09:35:54 usually i do it myself 09:36:31 r0kk3rz, temptation is hard but KISS 09:36:55 r0kk3rz: it feels to me like we first need our team to develop docker images 09:37:16 worst comes to worst there is `tail -F /var/lib/docker/containers/**/*.log` ;) 09:37:18 then we look at image deployment solutions 09:37:20 docker comes with theis logs 09:37:25 chriadam|windows, got a testbed plans ? 09:37:25 does kubernetes fall into the deployment side ? 09:37:25 r0kk3rz: that's overkill here 09:37:27 abranson_, too 09:37:37 larstiq, not at certain point 09:37:37 'cos I'm OK with manual-ish deployment in phase 1 09:37:37 lbt: yes, but not going to need that at the start 09:37:55 good 09:37:56 chriadam|windows: will be happy 09:37:56 dr_gogeta86: to begin with, the tests will be simple (e.g., trigger a sync, inspect db, ensure content is expected) 09:38:01 chriadam|windows: and get logs if it's not 09:38:16 but that should be part of the image really 09:38:19 from bugzilla pov ... got reports from which caldav servers ? 09:38:35 lbt: let's jfdi 09:38:41 ROFL 09:38:47 was about to say that 09:39:34 ok, so, concretely, what's the plan? 09:39:37 so setup a VM with Debian and give it ports 80/443 on an external IP ... more on demand 09:39:47 larstiq: yes, i said 'like' :P turns out theres a few simpler ones that just have a webui of whats running 09:40:13 larstiq: install docker from debian packages or ? 09:40:13 lbt, lgtm 09:40:17 lbt: depends on version 09:40:19 lbt: we can figure that out offline 09:40:29 r0kk3rz: can you setup a docker image as a user with such a thing ? 09:40:29 larstiq: +1 09:40:29 we don't need fancy things for now 09:40:42 lbt go head for debian/devuan 09:41:17 r0kk3rz: first let's just get 1 owncloud instance up. Then it's just "is it running yes/no" ;) 09:41:20 r0kk3rz: because I'm happy if you experiment and demo docker managers within your docker images 09:41:36 but yeah - as a team we focus on what larstiq just said :) 09:41:39 lbt: docker/docker-compose are really plenty at the start 09:42:07 * lbt is looking forward to playing with this tech too 09:42:26 OK .. I'm good I can set this up over the next couple of days 09:42:49 I want to put up a new VPN and a firewall/router 09:43:03 I'll play with LDAP accounts 09:43:21 I'll post requirements for shell ssh keys etc too 09:43:22 i hope not openvpn 09:43:36 we won't be exposing it 09:43:44 this is a management VPN 09:43:58 (and it's tinc for now - OVS soon probably) 09:44:20 ssh shell access via a port (probably 2222) 09:44:47 some minor ssh key management requirement stuff too 09:45:25 so ... are we done ? 09:45:31 just so sum up: 09:45:48 1) lbt will create the VM and install docker in it, get LDAP accounts set up, and give details to dr_gogeta86 and occirol 09:46:01 2) dr_gogeta86 and occirol will then ... get a docker image up and running with OwnCloud on it? 09:46:09 3) someone will let me know when that's done and how to use it 09:46:11 ? 09:46:12 * LarstiQ inserts his name in there somewhere 09:46:22 LarstiQ: does stuff I don't understand to make it work ;-) 09:46:31 * lbt will share setup with LarstiQ 09:46:59 dr_gogeta86: occirol: ok, I will ping you guys this time next week to see what our status is 09:47:03 if that makes sense to everyone? 09:47:08 ok for me 09:47:51 chriadam|windows: another meeting this time next week ? 09:47:57 it might take a couple of weeks for lbt + LarstiQ to get the VM etc setup done - depends on what crises occur in Jolla ;-) 09:48:09 but if we weekly track progress, that might be helpful / keep everyone on same page 09:48:19 *nod* ... I'm happy to do weekly updates.... yeah 09:48:19 lbt: I think so, would be good, if everyone is ok with it 09:48:26 great 09:48:31 ok, thanks everuone - much appreciated! 09:48:36 ending meeting in 5... 09:48:37 ok, I think :) 09:48:47 3 09:48:50 2 09:48:51 1 09:48:53 #endmeeting