Talk:SAC Meeting 2018-05-10
Jump to navigation
Jump to search
Transcript
20:00:23 robe2: Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10 20:00:24 sigabrt: Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org) 20:01:03 robe2: First topic is status of hardware as wildintellect noted still waiting for shipment 20:01:10 robe2: anything to add to that? 20:02:07 wildintellect: thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship 20:02:32 wildintellect: osuosl is aware of the order and expecting it 20:02:44 robe2: wildintellect great 20:03:04 robe2: next topic - osgeo6 coin mining issue 20:03:04 wildintellect: we should probably start discussing the setup plan 20:03:34 robe2: wildintellect I'll add that to the end of agenda today 20:03:41 wildintellect: so I'll not this isn't the 1st time we've caught a miner on an osgeo system 20:03:47 robe2: I think that might take a bit of discussion and flow into after party 20:04:06 wildintellect: martin found one once, I can't recall which machine, I think adhoc 20:04:17 wildintellect: that was clearly injected into a website 20:04:49 markusN: hi sorry for late 20:05:04 robe2: markusN I wasn't paying attention too closely were you saying j was running under geotools account? 20:05:51 markusN: np 20:06:03 robe2: np? 20:07:08 robe2: anyway can we disable geotools LDAP account or at very least remove for ldap_shell group? 20:07:21 robe2: ping strk you around? 20:09:54 TemptorSent: Check crontab entries. 20:10:53 wildintellect: there was a note that removing users from the ldap_shell group doesnt' work 20:10:54 TemptorSent: Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools. 20:11:08 markusN: I'm still convinced of resetting all accounts 20:11:19 wildintellect: TemptorSent, do you have access to that machine to poke around? 20:11:31 TemptorSent: No idea, and I'd rather not try. 20:12:03 markusN: (and I'm in Germany with totally crappy mobile connection... on and off) 20:12:05 TemptorSent: It's asking for a compromise of passwords. 20:12:26 markusN: mhh 20:12:27 TemptorSent: Anyone logging in with a password should subsequently reset their passwords. 20:12:45 wildintellect: ya that's part of the greater need to move to key based 20:12:57 TemptorSent: Trojaning SSH is a time-honored tradition., 20:13:01 wildintellect: Martin will have a way to key based login as root 20:13:06 wildintellect: I believe I have that too 20:13:10 robe2: TemptorSent didn't see any jobs running under geotools account 20:13:14 wildintellect: so I could add more keys 20:13:15 robe2: that was first thing I checked 20:13:47 TemptorSent: depending on how good the hackere/kit, they may be cloaked as 'nobody' even. 20:14:18 TemptorSent: A good trick is to pick the name of a running process, clone it, and restart yourself periodically. 20:14:49 robe2: wildintellect you know if Martin has used up his contract yet? 20:14:59 TemptorSent: To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against. 20:15:01 robe2: or can we assign him to look into this issue further 20:15:02 wildintellect: no idea, strk was overseeing that 20:15:20 robe2: and strk appears to be asleep :) 20:15:57 robe2: as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so 20:16:09 robe2: he was going to start putting in more time this coming week. 20:16:19 robe2: So I take that to mean he's still got some unspent time 20:16:20 TemptorSent: Without identifying the vector, we must work on the presumption that they have gained privleged access. 20:17:40 robe2: TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on. 20:17:56 robe2: and of course change the none ldap ones 20:18:26 robe2: does that sound like a reasonable start. Guess we also need to scan the whole system for trojans 20:18:50 TemptorSent: Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against. 20:19:30 robe2: off hand anyone knows what's running on osgeo6 20:19:45 TemptorSent: No idea.... 20:19:53 robe2: was thinking maybe those should be candidates to be moved 20:19:57 TemptorSent: Probably on the wiki somewhere. 20:20:21 TemptorSent: Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles... 20:20:55 wildintellect: martin setup most of what's on osgeo6 20:21:05 wildintellect: fyi the list server is on there 20:21:15 TemptorSent: Oh, joy. 20:22:20 TemptorSent: I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials... 20:22:51 wildintellect: https://wiki.osgeo.org/wiki/Osgeo6 20:22:52 sigabrt: Title: Osgeo6 - OSGeo (at wiki.osgeo.org) 20:22:55 TemptorSent: But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring. 20:24:55 robe2: Okay guess we should move on. I'll add a task for martin to look into the issue further. 20:25:00 TemptorSent: I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect. 20:25:31 wildintellect: quick look the geotools sites are all static sites 20:26:24 robe2: I'm actually more concerned at this point at relying too much on Martin's knowledge . I think we need a bit more knowledge coverage 20:27:05 wildintellect: well thats my note about new server, and how we can plan to avoid some issues 20:27:46 TemptorSent: True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry. 20:29:33 robe2: next topic FunToo container 20:29:41 robe2: and nextcloud 20:29:45 wildintellect: snapshotting, I know we didn't since it's Debian on ext4 20:30:00 wildintellect: checksum yes, the backups should have checksums 20:30:05 robe2: we have nextcloud running with ldap auth. Need to narrow down groups 20:30:09 TemptorSent: Ouch, yeah, unless backups were done at a low level, it'll be hard. 20:30:18 wildintellect: we use bacula 20:30:24 TemptorSent: robe2 Do we have a group setup for it yet? 20:30:27 wildintellect: it's file based 20:31:03 TemptorSent: I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered. 20:31:11 robe2: that's one reason I prefer VMs and try to keep the base very locked down 20:31:42 robe2: TemporSent I highly suspect bacula isn't capturing the rogue things 20:31:46 TemptorSent: VMs don't offer as much protection as you might think unfortunately. 20:32:01 robe2: I think it is set to only capture some subfolders of which for example /tmp is not a member of 20:32:14 TemptorSent: but we can explicitly compare the state BEFORE and determine what has been changed. 20:32:15 wildintellect: there's a newer type of container more focused on security than docker 20:32:24 robe2: TemporSent but they are easier to snapshot and destroy 20:33:08 TemptorSent: Not really easier to snapshot, and come with a lot of overhead. 20:33:56 TemptorSent: Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources. 20:34:04 robe2: TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot 20:34:18 wildintellect: this is conflation of container & VM 20:34:24 TemptorSent: Yeah, the containers work great with snapshotting :) 20:34:44 TemptorSent: Yes wildintellect. 20:34:44 robe2: VMs provide simple snapshotting too :) 20:35:21 wildintellect: yes some of them do (qcow base ones, or lvm snapshots) 20:35:34 robe2: the only ones worth using :) 20:35:49 TemptorSent: But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed. 20:36:13 robe2: or a cloud provider where you have a snapshot every day or as you need it 20:36:39 TemptorSent: With zfs, snapshots every 15 minutes are no problem. 20:36:40 robe2: True anyway lets move on 20:36:49 TemptorSent: Just age them out 20:38:17 TemptorSent: ... 20:39:10 robe2: for the ldap groups we don't have one set up specifically for nextcloud 20:39:25 TemptorSent: Okay, we might want to do that. 20:39:26 robe2: markusN you know if board has a ldap group 20:39:34 jive[m]: okay, I am here! 20:39:39 robe2: I think we asked that and I forget if the question was answered 20:39:54 robe2: jive[m] hi 20:40:14 robe2: jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board? 20:40:30 jodygarnett: I do not know if there is an LDAP group for the board 20:40:49 wildintellect: isn't there an ldap query webpage that lists all the groups? 20:41:01 jodygarnett: we are doing our best trying to track member status in the new website, rather than a series of wiki pages ... 20:41:52 robe2: wildintellect was looking for that but can't find it 20:42:08 TemptorSent: Hmm, sounds like some 'member_of_*' groups are needed. 20:42:08 robe2: and too lazy to look up ldapsearch. There is no group called board though 20:42:28 markusN: sorry for disconnected 20:42:34 robe2: TemportSet yah right now we have it set to allow any osgeo member to share 20:42:37 markusN: what was the question? 20:43:24 TemptorSent: Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group. 20:44:03 robe2: TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to? 20:44:56 TemptorSent: Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container. 20:44:58 robe2: My internet has been acting flaky today so could be my internet connection 20:47:17 robe2: I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects 20:47:21 TemptorSent: Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that. 20:47:56 TemptorSent: I'll be looking into service supervision at some point. 20:48:39 TemptorSent: Okay, you should be able to get to nextcloud.osgeo.org fine now :) 20:49:19 robe2: jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org 20:49:21 sigabrt: Title: Nextcloud (at nextcloud.osgeo.org) 20:50:05 robe2: I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet 20:51:23 robe2: next topic wiki ldap integration 20:52:05 TemptorSent: Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well 20:52:53 robe2: TemtorSent site-wide ldap auth engine? 20:53:20 robe2: TemptorSent typo not clear what that is 20:53:35 jodygarnett: sorry lost connection 20:53:38 robe2: is tht site-wide as in specific to wiki or even more encompassing 20:54:03 robe2: jodygarnett no problem my connection has been pretty flaky today too 20:54:11 TemptorSent: All of funtoo.org uses a single signon auth essentially. 20:54:25 jodygarnett: (what adgenda topic are we on please) 20:54:36 robe2: we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere 20:54:40 delawen[m]: Thanks! 20:54:51 TemptorSent: So you login and it provides the auth tokens to each service, rather than having to login to each individually. 20:55:59 robe2: TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc. 20:56:09 robe2: doesn't that still need to work with those 20:56:33 TemptorSent: Yes, it provides the auth-token to the individual applications. 20:56:48 TemptorSent: I'll talk to drobbins on details. 20:57:15 robe2: okay would be interesting to see that in action like if I have a funtoo.org account 20:58:08 TemptorSent: Yeah, it works on all the funtoo.org services. 20:58:12 robe2: jodygarnett I still owe you the proper setup of wordpress git in staging 20:58:18 TemptorSent: the wiki, the bug tracker, etc. 20:58:51 robe2: then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early 20:59:07 jodygarnett: I have a more serious short term website issue, further down in the meetin adgenda 20:59:54 jodygarnett: And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability" 20:59:56 TemptorSent: Okay, sounds like we're still waiting on status of DB for examination and plotting the migration. 21:00:09 robe2: jodygarnette we might be there in the agenda already 21:00:39 robe2: TemptorSent yah I was going to look at the db to see how crazy the user setup is 21:01:01 robe2: jodygarnett so what is your pressing issue? 21:01:37 robe2: oh info@osgeo.org 21:01:43 jodygarnett: The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket... 21:02:17 jodygarnett: https://trac.osgeo.org/osgeo/ticket/2158 21:02:18 sigabrt: Title: #2158 (sponsor logos are taken down too soon) – OSGeo (at trac.osgeo.org) 21:02:21 robe2: can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email 21:03:03 jodygarnett: because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug. 21:04:22 jodygarnett: updated the title to reflect recent testing, captured in the ticket 21:04:38 jodygarnett: I was hoping vicky could help, as she worked on a related issue 2071 21:05:16 robe2: jodygarnett I think vicky is traveling she's on some crazy worldish tour 21:05:45 robe2: she wrote me saying she'll be out of commission until the May 14th 21:05:59 jodygarnett: okay cool 21:06:17 jodygarnett: I will engage with vendor then, use some of our support hours. 21:06:26 jodygarnett: as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159 21:06:27 sigabrt: Title: #2159 (Concern expressed over awstats file) – OSGeo (at trac.osgeo.org) 21:06:50 robe2: jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea 21:07:57 robe2: which sites do we use awstats on? 21:08:33 robe2: the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D 21:08:34 sigabrt: Title: Index of /logs (at download.osgeo.org) 21:09:17 robe2: oh wait that one is just for downloads.osgeo.org not sure why we would publish those 21:09:57 jodygarnett: The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example 21:10:58 TemptorSent: That should be the least of our worries... 21:11:42 robe2: I did notice one had webdav for geotools 21:11:44 TemptorSent: Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list. 21:11:47 robe2: why are we using webdav 21:12:10 TemptorSent: We may not be intentionally... 21:12:33 TemptorSent: SVN uses it, so perhaps that bit of kit was piggybacked in using it. 21:14:16 robe2: oh 21:14:37 robe2: okay looks like we are out of time - start of after party if anyone wants to hang around 21:14:51 jodygarnett: we are using it as a poor-mans maven repository 21:14:56 TemptorSent: Thank you robe2. 21:14:58 jodygarnett: alternative is to deploy something like artifactory 21:15:22 robe2: artifactory? what's that 21:16:57 robe2: wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it 21:17:03 robe2: felt like we were at a standstill 21:17:12 jodygarnett: A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/ 21:17:13 sigabrt: Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com) 21:17:36 jodygarnett: no need to look into that suff at present, just answering the question on why we are using webdav 21:17:52 jodygarnett: thanks for running the meeting robe2 21:18:00 robe2: Too Integrated to Fail :) 21:18:24 robe2: great pitch 21:19:06 jodygarnett: (If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple) 21:20:48 TemptorSent: robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed. 21:21:31 TemptorSent: Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily. 21:23:14 TemptorSent: canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO. 21:25:05 robe2: TemptorSent glad someone has a memory for this 22:28:55 wildintellect: robe2, maven built java products rely on webdav to pull artifacts 22:29:08 wildintellect: sorry I had another meeting I had to go to 22:30:35 wildintellect: TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7