Talk:SAC Meeting 2018-05-27
Jump to navigation
Jump to search
06:55:15 robe2: https://wiki.osgeo.org/wiki/SAC_Meeting_2018-05-27 06:55:16 sigabrt: Title: SAC Meeting 2018-05-27 - OSGeo (at wiki.osgeo.org) 06:59:52 MartinSpott: Voilaaaa ! ;-) 06:59:59 astrodog: Ha. 07:00:06 MartinSpott: Oh my, it's Sunday morning .... 07:00:45 robe2: Everyone ready for meeting? 07:00:52 robe2: I'm sure strk is snoozing 07:00:54 astrodog: I'm around at the moment on a Marketing Committee set of questions I've got. Cool to see the new hardware progressing, though! 07:01:32 robe2: astrodog should I add to list? 07:01:47 robe2: or is it on already. At a glance don't see marketing in there 07:01:54 MartinSpott: no 07:02:04 astrodog: robe2: Nah, just an informal thing. Making sure some things are realistic before I propose them. 07:02:23 robe2: astrodog great :) 07:02:41 MartinSpott: indeed 07:03:10 astrodog: Better to have that discussion ahead of time than get everyone all hyped up, then realize there's no way we can support it. :P 07:03:32 robe2: First on list hardware - well it's shipped, but not sure if OSUOSL has got it yet 07:04:14 robe2: TemptorSent did you and wildintellect ever get around to outlining the plans for new server? 07:04:35 MartinSpott: moreover I'd expect a few days delay for racking, connecting some remote console 07:05:13 robe2: Okay seems TemptorSent is asleep to move on to next topic 07:05:39 robe2: osgeo6 I lost track of what's going on with that - we had disk issues and that coin minining thing 07:05:58 robe2: hard disk 07:06:10 MartinSpott: disk issues on osgeo6 itself ? 07:06:13 robe2: MartinSpott what were your thoughts on that 07:06:27 robe2: sorry not disk more how we run out of disk and doing net whatever on it 07:06:35 astrodog: I do have a thought on osgeo6... did we ever determine if the compromised account was also capable of delivering or altering content? 07:07:14 robe2: astrodog no clue -- it was calling out to germany some where was all I could surpmise 07:07:15 MartinSpott: you mean the geotools account ? 07:07:21 astrodog: Yeah. 07:07:23 robe2: yes geotools 07:08:42 MartinSpott: as far as can tell: no 07:08:42 astrodog: Assuming the account has the ability to deliver content to users... we should probably write something up. Even if there's no indication that it actually did anything, someone just stumbling onto that issue browsing the lists and not seeing anything anywhere else is bad times. 07:09:15 robe2: who owns the geotools account? 07:09:54 MartinSpott: no idea 07:10:21 astrodog: Even if it's just a page or so of "This account was compromised, we have not seen any indication that it delivered malicious binaries or content to users, or extracted any user information. We've fixed it by doing 'X'." etc, etc. 07:10:22 robe2: did we disable it already guess someone will complain when we do :) 07:10:23 MartinSpott: the account says: geotools Build User Boundlessgeo 07:11:06 astrodog: Have we identified how it was compromised, or is the guess a bad PHP script? 07:11:18 robe2: so jive[m] might know something about that account 07:12:08 robe2: astrodog no not sure how it was compromised. I don't even guess a bad PHP script 07:12:12 MartinSpott: looks like an account to run some builds, the command history mentions Confluence 07:12:39 astrodog: What / who do we need to get what happened there run down? 07:14:04 robe2: astrodog well it was Markus Neteler that discovered it 07:14:09 MartinSpott: From naive view I'd say this account is being used to update some geotools public web page from SVN using an automated process 07:15:20 robe2: I'll send Jody a note he's the only one I know on that team 07:15:32 MartinSpott: But I have to admit I didn't see the mysterious processes which have been observed by others 07:15:53 MartinSpott: simply because I wasn't around in time 07:15:59 robe2: MartinSpott it was a process running in a folder " " 07:16:04 astrodog: Since we do actually deliver a fair number of binaries and the like to people... we really just need to figure out what we're doing with it and say that publicly. 07:16:06 MartinSpott: ah 07:16:10 robe2: so essentially hidden folder in tmp folder 07:16:19 robe2: and it was called d or something like that 07:16:41 robe2: researching it was a bitcoin harvester executable I think released in march or so 07:16:46 robe2: I forget the exact date 07:17:11 MartinSpott: ok 07:17:56 robe2: So what would be the best course of action - disable the account entirely and have geotools setup a new one? Do nothing? just reset the password? 07:18:04 astrodog: Martin: Do you happen to recall what OS and kernel version are running on that host? 07:18:06 MartinSpott: in this case I'd agree with setting a deadline for password renewal 07:18:30 robe2: I guess the other concern is if that is the only one compromised on that server isnce it was discovered by accident 07:18:32 MartinSpott: astrodog: It's Debian8 and kernel 4.7 07:19:27 robe2: MartinSpott is there any way to enforce password renewal? I'll ask jody if he can get someone to reset the geotools one 07:19:34 MartinSpott: robe2: geotools is not in the sudoers list 07:19:49 astrodog: Martin: Any indication that it's spread beyond that account? 4.7 is old enough that I'd expect some random local escalation exploits to be floating around. 07:19:53 robe2: so we don't care so much about it 07:20:04 MartinSpott: well, we can simply set a new one and the the user ask for it :-) 07:20:13 MartinSpott: and *let* the user 07:20:40 astrodog: Before we bring the account back online, we kinda need to figure out how it got there in the first place. :\ 07:21:18 MartinSpott: Most communication in this account is authorized by public keys, I'd say 07:21:21 robe2: MartinSpott not following *let* the user? 07:21:38 robe2: ah okay so resetting shouldn't be an issue 07:21:42 astrodog: robe: I think Martin is saying, break the account, they'll complain. 07:21:47 robe2: resetting the password I mean 07:22:07 robe2: MartinSpott okay can you break it? 07:22:18 robe2: I can give Jody a heads up at least :) 07:22:33 MartinSpott: astrodog: correct 07:22:43 astrodog: If the account is primarily driven by SSH keys, that may be how it got in, too. Could be someone with access to it had an issue. 07:23:44 MartinSpott: Let's reset the password plus remove the SSH keys and announce their removal in advance to some list which is linked to geotools 07:24:21 robe2: Sounds good 07:24:21 MartinSpott: I'm sure they'll understand if we tell them why 07:24:27 astrodog: Putting on my... TrustedBSD contributor hat for a moment, my inclination would be to disable the account, and work with the geotools folks to figure out what happened to it. Just resetting its keys and passwords doesn't buy us much if we don't know what happened. 07:25:11 MartinSpott: that's fine with me as well 07:25:48 astrodog: I'd probably try and coordinate that off-list, we can do some kind of announcement when we've got meaningful details to share. As it stands, it seems like a very minor intrusion and someone stumbled on to the machine. 07:25:50 MartinSpott: who knows where to find these geotools folks ? 07:25:57 astrodog: They're on IRC, I think? 07:26:14 astrodog: Jody is, anyway. 07:26:38 robe2: I'm about to send Jody an email. the list looks like sourceforge 07:26:49 MartinSpott: ok, let's check mark this item 07:26:56 astrodog: I'll ping him on Skype, too. 07:27:17 astrodog: Martin: Do you want me to coordinate running it down with them? 07:27:26 MartinSpott: please ;-) 07:27:40 robe2: so no do and ask questions later :) 07:27:49 robe2: you guys are wusses 07:28:21 astrodog: robe: Nuke osgeo6 from orbit, it's the only way to be sure. Better? 07:28:22 astrodog: :P 07:28:29 robe2: yes 07:28:32 robe2: that's more like it 07:28:50 robe2: I'm sending Jody a note saying we are shutting it down and talk to his people 07:28:59 robe2: cc'ing Martin and Alex 07:29:03 astrodog: Martin: Do we have any sort of binary integrety checking on osgeo6? 07:29:03 MartinSpott: great 07:29:30 MartinSpott: as far as I can tell, we don't 07:29:48 MartinSpott: you men tripwire or the like ? 07:29:56 astrodog: Yeah. 07:30:05 MartinSpott: pretty sure we don't 07:30:22 astrodog: You mind doing a basic runthrough? Statically linked sha binary, and just compare $PATH and lib to whatever is in the distribution. 07:31:07 MartinSpott: yes, I'll fetch one to check the current system binary set for known suspects 07:32:05 astrodog: That'd be great. There are a few 1-2 line scripts using find -x that can crank through it pretty quickly. 07:32:43 robe2: MartinSpott sent note to Jody and cc'd you 07:34:03 MartinSpott: astrodog: Noted in the ticket so I don't forget .... 07:35:19 MartinSpott: robe2: Next item ? 07:36:05 robe2: Next item is funtoo but we were waiting for board to test the nextcloud thing 07:36:08 robe2: we can skip that 07:36:24 robe2: oh I know the question I had 07:36:35 MartinSpott: next one is download's disk extension vi NFS 07:36:36 robe2: MartinSpott you know if there is an ldap group for board members? 07:36:47 robe2: I couldn't find one 07:36:49 MartinSpott: ldap for board, no idea - wait a second 07:37:09 robe2: that was on my list to just limit to board but couldn't find such a group 07:38:11 MartinSpott: no, no group whose name relates to board 07:38:25 MartinSpott: or resembles with 07:38:39 robe2: seems like we should have one. I'll put in a ticket to add one 07:38:43 astrodog: Propose we make a board LDAP group! 07:38:44 astrodog: :P 07:38:46 MartinSpott: fine 07:41:29 robe2: https://trac.osgeo.org/osgeo/ticket/2170 07:41:30 sigabrt: Title: #2170 (Create LDAP group for board) – OSGeo (at trac.osgeo.org) 07:42:50 robe2: grrh I've been editing the wrong sac meeting 07:43:04 astrodog: These kinds of meetings are quick when you only have 3 attendees and one of them only shows up once every year or two. :P 07:43:47 MartinSpott: I'll set it to Milestone Sysadmin Contract for better visibility 07:44:23 robe2: next topic nfs mount 07:44:45 robe2: how much disk space is qgis and (what is the other one, I forget) taking up on nfs mount? 07:44:59 robe2: oh osgeo4w 07:45:11 astrodog: robe2: Sorry if I'm rehashing something that's been discussed before, but is there a particular reason to use NFS for this versus an iSCSI target? 07:45:35 MartinSpott: both approx 30 GByte 07:45:37 MartinSpott: each 07:45:38 astrodog: (Or, is that just how it's set up, and there's no particular need / interest in poking at it?) 07:45:47 robe2: astrodog no clue, I think it was one of those in the moment decisions "This will work" so it was done 07:45:59 astrodog: Gotcha. 07:46:06 MartinSpott: yup 07:46:20 MartinSpott: what make me so upset is: 07:46:23 robe2: I'm not familiar with iSCSI target 07:46:46 astrodog: There are some... performance and management gains to be had in changing that, but I don't know where that would fall on the list in terms of SAC priorities. 07:46:54 astrodog: robe: Block device over the network, basically. 07:46:58 robe2: MartingSpott sorry for interrupting 07:47:10 MartinSpott: a) people are excercising root permissions while having no idea of security concerns 07:47:22 MartinSpott: b) simply because it's convenient 07:47:53 MartinSpott: c) even though it adds maintenance burdeb 07:48:00 MartinSpott: d) period 07:48:13 robe2: people will be people - take the easiest road to immediate satisfaction 07:48:24 robe2: gratification I should say 07:48:35 MartinSpott: NFS isn't bad in general, but apparently there are too many users having sudo permission 07:48:43 astrodog: One of the advantages of using the iSCSI target thing is that it blocks the... permissions flow through. 07:48:51 MartinSpott: yup 07:49:11 astrodog: You can have the filesystem be r/o on the downloads server, and let the actual file management happen somewhere else. 07:49:14 robe2: so is it hard to set up or even worth it if some of that will be offloaded to osgeo7 07:49:30 astrodog: It's pretty easy, but I'd probably defer it to the new server setup. 07:49:33 robe2: astrodog so sounds like a good long term solution 07:50:50 robe2: I aslo saw ftp running on osgeo6 which surprised me 07:50:57 robe2: looked like plain old ftp 07:51:15 astrodog: I think some things download over that, osgeo4w comes to mind. 07:51:27 robe2: over ftp instead of http? 07:51:32 astrodog: Yeah. 07:51:47 astrodog: You can definitely feed the installer FTP mirrors, anyway. 07:51:59 robe2: okay that explains it 07:52:23 astrodog: Might be worth seeing if anyone uses it, though. 07:52:23 robe2: and I think someone said we have webdav too presumably for maven 07:52:58 robe2: it'd be nice if download was just downloads 07:53:37 robe2: next topic wiki / ldap 07:53:39 astrodog: We could do that as part of the iSCSI change. We'd be doing some cleanup on file layouts anyway. 07:53:51 robe2: MartinSpott did you ever create pgbackup sorry if I missed it 07:54:09 MartinSpott: err, pgbackup ? 07:54:14 astrodog: Oh god, wiki and LDAP is still a thing? I know I dropped the ball on that... but jeez. 07:54:41 robe2: astrodog a thing you want it to not be a thing? 07:55:23 astrodog: robe2: I want to light the entire pile on fire, push it on a boat into the sea, and we can honor it once a year by drinking heavily. 07:55:26 MartinSpott: I've created dumps to be sent to TemptorSent 07:55:30 robe2: MartinSpott pg dump of wiki database 07:55:58 robe2: MartinSpott can i have access to them too? 07:55:59 MartinSpott: MySQL plus filesystem backup of the Wiki dir 07:55:59 astrodog: Is TemptorSent coordinating that one now? 07:56:38 MartinSpott: astrodog: He suggested to set up a copy of the Wiki for test with LDAP authentication and user migratio 07:56:43 robe2: MartinSpott is it running on MySQL? I thought it was on postgres. 07:57:03 MartinSpott: MySQL, unfortunately, no Postgres 07:57:09 robe2: :( 07:57:31 robe2: are we planning to run the new one on Postgres or we going to try a two phase migration? 07:57:39 astrodog: Martin: Ah, gotcha. I'll track down my notes and pass them along. As I recall, where things kinda fell apart was trying to figure out what the relationship of wiki and LDAP users might be. 07:57:48 MartinSpott: People more familiar with the topic than I am suggested to stick with MySQL because some extentions would probably not work on Postgresql 07:58:17 robe2: MartinSpott yah I was afraid of that 07:58:30 MartinSpott: sounds like some adhere to the MySQL dialect of SQL 07:58:42 MartinSpott: or the like 07:58:59 astrodog: Do we use any extensions like that? (Or, do we know?) 07:59:32 robe2: astrodog I have no clue. Doesn't seem like we use wiki for much aside from wiki 07:59:42 robe2: so I would think we are minimalist on extensions or should be 07:59:55 MartinSpott: we're using quite a few extensions and I don't have in-depth knowledge of the respective implications 08:00:07 astrodog: Maybe that'd be something else to do in the test... see if it breaks if we switch it, 08:00:31 MartinSpott: there's a list on the Wiki I took as a reference during the last upgrade 08:00:32 robe2: LDAP is already a big mouthful though 08:00:59 robe2: as much as I hate MySQL I'd stick with it (well MariaDB at anyrate) 08:01:13 MartinSpott: same here 08:01:51 robe2: MartinSpott still I'd like to have access to the mysql backups 08:02:08 MartinSpott: where sould I put it ? 08:02:12 astrodog: I suppose I'm agnostic on it, really. It seems strange that we'd be using something that would preclude a change, but exactly what database sits behind the wiki seems, ultimately, pretty immaterial. 08:02:37 robe2: I actually have access to the wiki server I think 08:02:41 astrodog: (Also, my opinion on it doesn't count for much. I'm not stuck maintaining it, you guys are.) 08:02:51 robe2: so if you put it somewhere there or even on osgeo6 that would be fine 08:03:07 MartinSpott: robe2: it already is 08:03:10 robe2: on osgeo6 I have sudo but wiki I think I just have regular access 08:03:18 MartinSpott: because disk space on the Wiki is getting tight .... 08:03:36 MartinSpott: robe2: wait a second, I'll move it into your home dir 08:04:26 MartinSpott: there's a wiki page on the wiki plugins we use for the wiki (hah, three times "wiki" in a single, short sentence) 08:04:42 MartinSpott: where is it ? .... 08:05:02 robe2: MartinSpott I don't have a home directory on wiki server 08:05:18 MartinSpott: on osgeo6 08:05:25 robe2: oh okay 08:06:26 robe2: weird I don't seem to have a home on osgeo6 either 08:06:37 MartinSpott: we'll make one 08:07:14 robe2: okay make one and put it in there 08:08:33 MartinSpott: next topic ? 08:09:30 robe2: MartinSpott I think we are done other topics involve people not here 08:09:40 robe2: astrodog wanted to raise your questions 08:09:51 robe2: marketing stuff I guess is the next topic :) 08:10:13 astrodog: My marketing related question is: How much of a burden would it be if Marketing wanted to work with the constituent projects to set up some coordinated demos? 08:10:41 robe2: astrodog burden for who? 08:10:43 astrodog: A stripped down version of something like maps.esri.com, or some of their examples. 08:10:45 astrodog: SAC. 08:10:51 robe2: sounds like more of a burden for marketing :) 08:11:19 robe2: astrodog depends how much active content and stuff needs installing 08:11:23 astrodog: You say that, until you try to run QGIS's WMS server in production, and realize you're tracking Qt dependencies on a webserver. :P 08:11:25 robe2: setting up a domain would be easy 08:12:13 robe2: astrodog too bad TemptorSent isn't here 08:12:30 robe2: I was thinking our funtoo server might be a good fit for that 08:12:45 robe2: we could setup containers for each of those 08:12:54 robe2: lxc containers 08:13:10 astrodog: My guess would be a domain plus services. We can keep the datasets small, so the actual memory, data and CPU footprint stays managable... but I think we're really missing something by not showing how the OSGeo projects work together. 08:13:46 robe2: astrodog I agree I like the idea 08:14:11 MartinSpott: I'd say it depends on how much self contained the setup would be 08:14:58 astrodog: I think we can keep it very self-contained... it'd mostly be showing how the various pieces combined with OGC standards let you do spatial stuff. 08:15:11 MartinSpott: if someone is around to put everything into a container, then I'd say go for it 08:15:13 astrodog: Ideally, we'd also work out some... combined installation documentation at the same time. 08:15:21 astrodog: I don't want to suggest it to marketing if it's going to drown SAC, though. How would you guys feel about trying to do a quick proof of concept when the new server is up? 08:15:25 MartinSpott: but probably you're reaching out for such person right now 08:15:54 astrodog: Martin: Getting it up and running isn't too bad, but SAC would probably be stuck with keeping it up to date. 08:16:05 astrodog: (In terms of software versions) 08:16:30 robe2: astrodog so this would be an all encompassing thing showing all osgeo goodies 08:16:35 MartinSpott: does it rely on software installed on the "host" system or does it bring its own ? 08:16:51 robe2: so something like maps.osgeo.org with all the different pieces interoperating 08:17:06 astrodog: robe: That's what I have in mind, yeah. 08:17:07 MartinSpott: if it's self contained, than replacing an old bundle with a new one is cheap 08:17:26 robe2: I think we should run it in a container for sure 08:17:31 astrodog: Martin: I think we'd need to bring our own in the container. The dependency hell on the host would escalate quickly. 08:17:36 MartinSpott: maps.osgeo.org is dead 08:17:46 MartinSpott: I mean not existent 08:18:17 astrodog: Are you guys okay with me posting to marketing saying SAC is looking into the viability of the idea? 08:18:22 robe2: that's good so we can use the domain :) 08:18:46 robe2: astrodog I am but might want to wait for wildintellect's blessing on that one 08:19:05 astrodog: Marketing seems a little... downbeat these days, after pushing through the rebranding, and I think it'll be exciting for people. 08:19:48 robe2: well sounds exciting to me I know postgis we always wanted to do something like that 08:20:14 robe2: but it would be more of a try your queries kinda thing 08:20:39 robe2: okay meeting adjourned 08:20:41 MartinSpott: I think a long-term agenda we can call "containerize it" 08:21:07 astrodog: I have some cool PostGIS/QGIS/R use cases we could show. Pretty sure some clients would be fine with throwing their data in if we need that. 08:21:11 robe2: yah I think wildintellect and TemptorSent were supposed to flesh that out with Osgeo7 08:21:20 MartinSpott: robe2: what is "something like that" ? A site to compare diferent map renderers ? 08:21:22 robe2: I thought the plan was to run lxc containers on that 08:22:09 robe2: MartinSpott was talking about just containerize in general so we don't have to worry about dependencies clubbering each other 08:22:56 robe2: astrodog would we use something like Jupyter Notebooks 08:23:30 robe2: I see everyone talking about those things and saying how great they are especially for demoing and showcasing, but I haven't found the energy to learn yet another thing 08:26:48 MartinSpott: robe2: you're "robe" on osgeo6, correct ? 08:27:00 robe2: yes 08:28:50 MartinSpott: voila 08:29:08 MartinSpott: open topics to be discussed today ? 08:29:57 robe2: don't think any other topics to be discussed 08:30:09 MartinSpott: close the meeting ? 08:30:22 robe2: took us 1.5 hrs (well a little under) -- I'm pooped 08:30:51 MartinSpott: sleep well, thanks for tacing care of the meeting schedule - in particular as well as in general 08:31:17 robe2: MartinSpott you are welcome and thanks for all that system stuff you do that I don't understand :) 08:31:20 MartinSpott: my family woke up, time to play with the kids 08:31:36 robe2: by that I mean that we have someone as capable on our team as you 08:31:46 robe2: have fun with the kids ttyl 08:32:23 robe2: meeting closed 08:32:27 robe2: ttyl