Talk:SAC Meeting 2018-05-10

Transcript
20:00:23	robe2:	Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10 20:00:24	sigabrt:	Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org) 20:01:03	robe2:	First topic is status of hardware as wildintellect noted still waiting for shipment 20:01:10	robe2:	anything to add to that? 20:02:07	wildintellect:	thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship 20:02:32	wildintellect:	osuosl is aware of the order and expecting it   20:02:44	robe2:	wildintellect great 20:03:04	robe2:	next topic - osgeo6 coin mining issue 20:03:04	wildintellect:	we should probably start discussing the setup plan 20:03:34	robe2:	wildintellect I'll add that to the end of agenda today 20:03:41	wildintellect:	so I'll not this isn't the 1st time we've caught a miner on an osgeo system 20:03:47	robe2:	I think that might take a bit of discussion and flow into after party 20:04:06	wildintellect:	martin found one once, I can't recall which machine, I think adhoc 20:04:17	wildintellect:	that was clearly injected into a website 20:04:49	markusN:	hi sorry for late 20:05:04	robe2:	markusN I wasn't paying attention too closely were you saying j was running under geotools account? 20:05:51	markusN:	np 20:06:03	robe2:	np? 20:07:08	robe2:	anyway can we disable geotools LDAP account or at very least remove for ldap_shell group? 20:07:21	robe2:	ping strk you around? 20:09:54	TemptorSent:	Check crontab entries. 20:10:53	wildintellect:	there was a note that removing users from the ldap_shell group doesnt' work 20:10:54	TemptorSent:	Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools. 20:11:08	markusN:	I'm still convinced of resetting all accounts 20:11:19	wildintellect:	TemptorSent, do you have access to that machine to poke around? 20:11:31	TemptorSent:	No idea, and I'd rather not try. 20:12:03	markusN:	(and I'm in Germany with totally crappy mobile connection... on and off) 20:12:05	TemptorSent:	It's asking for a compromise of passwords. 20:12:26	markusN:	mhh 20:12:27	TemptorSent:	Anyone logging in with a password should subsequently reset their passwords. 20:12:45	wildintellect:	ya that's part of the greater need to move to key based 20:12:57	TemptorSent:	Trojaning SSH is a time-honored tradition., 20:13:01	wildintellect:	Martin will have a way to key based login as root 20:13:06	wildintellect:	I believe I have that too 20:13:10	robe2:	TemptorSent didn't see any jobs running under geotools account 20:13:14	wildintellect:	so I could add more keys 20:13:15	robe2:	that was first thing I checked 20:13:47	TemptorSent:	depending on how good the hackere/kit, they may be cloaked as 'nobody' even. 20:14:18	TemptorSent:	A good trick is to pick the name of a running process, clone it, and restart yourself periodically. 20:14:49	robe2:	wildintellect you know if Martin has used up his contract yet? 20:14:59	TemptorSent:	To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against. 20:15:01	robe2:	or can we assign him to look into this issue further 20:15:02	wildintellect:	no idea, strk was overseeing that 20:15:20	robe2:	and strk appears to be asleep :)   20:15:57	robe2:	as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so    20:16:09	robe2:	he was going to start putting in more time this coming week.    20:16:19	robe2:	So I take that to mean he's still got some unspent time    20:16:20	TemptorSent:	Without identifying the vector, we must work on the presumption that they have gained privleged access.    20:17:40	robe2:	TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on.    20:17:56	robe2:	and of course change the none ldap ones    20:18:26	robe2:	does that sound like a reasonable start. Guess we also need to scan the whole system for trojans    20:18:50	TemptorSent:	Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against. 20:19:30	robe2:	off hand anyone knows what's running on osgeo6 20:19:45	TemptorSent:	No idea.... 20:19:53	robe2:	was thinking maybe those should be candidates to be moved 20:19:57	TemptorSent:	Probably on the wiki somewhere. 20:20:21	TemptorSent:	Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles... 20:20:55	wildintellect:	martin setup most of what's on osgeo6 20:21:05	wildintellect:	fyi the list server is on there 20:21:15	TemptorSent:	Oh, joy. 20:22:20	TemptorSent:	I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials... 20:22:51	wildintellect:	https://wiki.osgeo.org/wiki/Osgeo6 20:22:52	sigabrt:	Title: Osgeo6 - OSGeo (at wiki.osgeo.org) 20:22:55	TemptorSent:	But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring. 20:24:55	robe2:	Okay guess we should move on. I'll add a task for martin to look into the issue further. 20:25:00	TemptorSent:	I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect. 20:25:31	wildintellect:	quick look the geotools sites are all static sites 20:26:24	robe2:	I'm actually more concerned at this point at relying too much on Martin's knowledge. I think we need a bit more knowledge coverage 20:27:05	wildintellect:	well thats my note about new server, and how we can plan to avoid some issues 20:27:46	TemptorSent:	True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry. 20:29:33	robe2:	next topic FunToo container 20:29:41	robe2:	and nextcloud 20:29:45	wildintellect:	snapshotting, I know we didn't since it's Debian on ext4 20:30:00	wildintellect:	checksum yes, the backups should have checksums 20:30:05	robe2:	we have nextcloud running with ldap auth. Need to narrow down groups 20:30:09	TemptorSent:	Ouch, yeah, unless backups were done at a low level, it'll be hard. 20:30:18	wildintellect:	we use bacula 20:30:24	TemptorSent:	robe2 Do we have a group setup for it yet? 20:30:27	wildintellect:	it's file based 20:31:03	TemptorSent:	I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered. 20:31:11	robe2:	that's one reason I prefer VMs and try to keep the base very locked down 20:31:42	robe2:	TemporSent I highly suspect bacula isn't capturing the rogue things 20:31:46	TemptorSent:	VMs don't offer as much protection as you might think unfortunately. 20:32:01	robe2:	I think it is set to only capture some subfolders of which for example /tmp is not a member of   20:32:14	TemptorSent:	but we can explicitly compare the state BEFORE and determine what has been changed. 20:32:15	wildintellect:	there's a newer type of container more focused on security than docker 20:32:24	robe2:	TemporSent but they are easier to snapshot and destroy 20:33:08	TemptorSent:	Not really easier to snapshot, and come with a lot of overhead. 20:33:56	TemptorSent:	Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources. 20:34:04	robe2:	TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot 20:34:18	wildintellect:	this is conflation of container & VM   20:34:24	TemptorSent:	Yeah, the containers work great with snapshotting :)    20:34:44	TemptorSent:	Yes wildintellect.    20:34:44	robe2:	VMs provide simple snapshotting too :) 20:35:21	wildintellect:	yes some of them do (qcow base ones, or lvm snapshots) 20:35:34	robe2:	the only ones worth using :)   20:35:49	TemptorSent:	But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed.    20:36:13	robe2:	or a cloud provider where you have a snapshot every day or as you need it    20:36:39	TemptorSent:	With zfs, snapshots every 15 minutes are no problem.    20:36:40	robe2:	True anyway lets move on    20:36:49	TemptorSent:	Just age them out    20:38:17	TemptorSent:	...    20:39:10	robe2:	for the ldap groups we don't have one set up specifically for nextcloud    20:39:25	TemptorSent:	Okay, we might want to do that.    20:39:26	robe2:	markusN you know if board has a ldap group    20:39:34	jive[m]:	okay, I am here!    20:39:39	robe2:	I think we asked that and I forget if the question was answered    20:39:54	robe2:	jive[m] hi    20:40:14	robe2:	jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board? 20:40:30	jodygarnett:	I do not know if there is an LDAP group for the board 20:40:49	wildintellect:	isn't there an ldap query webpage that lists all the groups? 20:41:01	jodygarnett:	we are doing our best trying to track member status in the new website, rather than a series of wiki pages ... 20:41:52	robe2:	wildintellect was looking for that but can't find it   20:42:08	TemptorSent:	Hmm, sounds like some 'member_of_*' groups are needed. 20:42:08	robe2:	and too lazy to look up ldapsearch. There is no group called board though 20:42:28	markusN:	sorry for disconnected 20:42:34	robe2:	TemportSet yah right now we have it set to allow any osgeo member to share 20:42:37	markusN:	what was the question? 20:43:24	TemptorSent:	Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group. 20:44:03	robe2:	TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to? 20:44:56	TemptorSent:	Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container. 20:44:58	robe2:	My internet has been acting flaky today so could be my internet connection 20:47:17	robe2:	I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects 20:47:21	TemptorSent:	Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that. 20:47:56	TemptorSent:	I'll be looking into service supervision at some point. 20:48:39	TemptorSent:	Okay, you should be able to get to nextcloud.osgeo.org fine now :)   20:49:19	robe2:	jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org    20:49:21	sigabrt:	Title: Nextcloud (at nextcloud.osgeo.org)    20:50:05	robe2:	I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet    20:51:23	robe2:	next topic wiki ldap integration    20:52:05	TemptorSent:	Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well    20:52:53	robe2:	TemtorSent site-wide ldap auth engine?    20:53:20	robe2:	TemptorSent typo not clear what that is    20:53:35	jodygarnett:	sorry lost connection    20:53:38	robe2:	is tht site-wide as in specific to wiki or even more encompassing 20:54:03	robe2:	jodygarnett no problem my connection has been pretty flaky today too 20:54:11	TemptorSent:	All of funtoo.org uses a single signon auth essentially. 20:54:25	jodygarnett:	(what adgenda topic are we on please) 20:54:36	robe2:	we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere 20:54:40	delawen[m]:	Thanks! 20:54:51	TemptorSent:	So you login and it provides the auth tokens to each service, rather than having to login to each individually. 20:55:59	robe2:	TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc.   20:56:09	robe2:	doesn't that still need to work with those 20:56:33	TemptorSent:	Yes, it provides the auth-token to the individual applications. 20:56:48	TemptorSent:	I'll talk to drobbins on details. 20:57:15	robe2:	okay would be interesting to see that in action like if I have a funtoo.org account 20:58:08	TemptorSent:	Yeah, it works on all the funtoo.org services. 20:58:12	robe2:	jodygarnett I still owe you the proper setup of wordpress git in staging 20:58:18	TemptorSent:	the wiki, the bug tracker, etc.   20:58:51	robe2:	then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early 20:59:07	jodygarnett:	I have a more serious short term website issue, further down in the meetin adgenda 20:59:54	jodygarnett:	And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability" 20:59:56	TemptorSent:	Okay, sounds like we're still waiting on status of DB for examination and plotting the migration. 21:00:09	robe2:	jodygarnette we might be there in the agenda already 21:00:39	robe2:	TemptorSent yah I was going to look at the db to see how crazy the user setup is   21:01:01	robe2:	jodygarnett so what is your pressing issue? 21:01:37	robe2:	oh info@osgeo.org 21:01:43	jodygarnett:	The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket... 21:02:17	jodygarnett:	https://trac.osgeo.org/osgeo/ticket/2158 21:02:18	sigabrt:	Title: #2158 (sponsor logos are taken down too soon) â€“ OSGeo (at trac.osgeo.org) 21:02:21	robe2:	can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email 21:03:03	jodygarnett:	because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug. 21:04:22	jodygarnett:	updated the title to reflect recent testing, captured in the ticket 21:04:38	jodygarnett:	I was hoping vicky could help, as she worked on a related issue 2071 21:05:16	robe2:	jodygarnett I think vicky is traveling she's on some crazy worldish tour 21:05:45	robe2:	she wrote me saying she'll be out of commission until the May 14th 21:05:59	jodygarnett:	okay cool 21:06:17	jodygarnett:	I will engage with vendor then, use some of our support hours. 21:06:26	jodygarnett:	as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159 21:06:27	sigabrt:	Title: #2159 (Concern expressed over awstats file) â€“ OSGeo (at trac.osgeo.org) 21:06:50	robe2:	jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea 21:07:57	robe2:	which sites do we use awstats on? 21:08:33	robe2:	the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D 21:08:34	sigabrt:	Title: Index of /logs (at download.osgeo.org) 21:09:17	robe2:	oh wait that one is just for downloads.osgeo.org not sure why we would publish those 21:09:57	jodygarnett:	The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example 21:10:58	TemptorSent:	That should be the least of our worries... 21:11:42	robe2:	I did notice one had webdav for geotools 21:11:44	TemptorSent:	Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list. 21:11:47	robe2:	why are we using webdav 21:12:10	TemptorSent:	We may not be intentionally...   21:12:33	TemptorSent:	SVN uses it, so perhaps that bit of kit was piggybacked in using it. 21:14:16	robe2:	oh 21:14:37	robe2:	okay looks like we are out of time - start of after party if anyone wants to hang around 21:14:51	jodygarnett:	we are using it as a poor-mans maven repository 21:14:56	TemptorSent:	Thank you robe2. 21:14:58	jodygarnett:	alternative is to deploy something like artifactory 21:15:22	robe2:	artifactory? what's that 21:16:57	robe2:	wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it   21:17:03	robe2:	felt like we were at a standstill 21:17:12	jodygarnett:	A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/ 21:17:13	sigabrt:	Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com) 21:17:36	jodygarnett:	no need to look into that suff at present, just answering the question on why we are using webdav 21:17:52	jodygarnett:	thanks for running the meeting robe2 21:18:00	robe2:	Too Integrated to Fail :)   21:18:24	robe2:	great pitch    21:19:06	jodygarnett:	(If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple)    21:20:48	TemptorSent:	robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed.    21:21:31	TemptorSent:	Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily.    21:23:14	TemptorSent:	canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO.    21:25:05	robe2:	TemptorSent glad someone has a memory for this    22:28:55	wildintellect:	robe2, maven built java products rely on webdav to pull artifacts    22:29:08	wildintellect:	sorry I had another meeting I had to go to    22:30:35	wildintellect:	TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7