Talk:SAC Meeting 2018-05-10

Transcript

   20:00:23	robe2:	Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10
   20:00:24	sigabrt:	Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org)
   20:01:03	robe2:	First topic is status of hardware as wildintellect noted still waiting for shipment
   20:01:10	robe2:	anything to add to that?
   20:02:07	wildintellect:	thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship
   20:02:32	wildintellect:	osuosl is aware of the order and expecting it
   20:02:44	robe2:	wildintellect great
   20:03:04	robe2:	next topic - osgeo6 coin mining issue
   20:03:04	wildintellect:	we should probably start discussing the setup plan
   20:03:34	robe2:	wildintellect I'll add that to the end of agenda today
   20:03:41	wildintellect:	so I'll not this isn't the 1st time we've caught a miner on an osgeo system
   20:03:47	robe2:	I think that might take a bit of discussion and flow into after party
   20:04:06	wildintellect:	martin found one once, I can't recall which machine, I think adhoc
   20:04:17	wildintellect:	that was clearly injected into a website
   20:04:49	markusN:	hi sorry for late
   20:05:04	robe2:	markusN I wasn't paying attention too closely were you saying j was running under geotools account?
   20:05:51	markusN:	np
   20:06:03	robe2:	np?
   20:07:08	robe2:	anyway can we disable geotools LDAP account or at very least remove for ldap_shell group?
   20:07:21	robe2:	ping strk you around?
   20:09:54	TemptorSent:	Check crontab entries.
   20:10:53	wildintellect:	there was a note that removing users from the ldap_shell group doesnt' work
   20:10:54	TemptorSent:	Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools.
   20:11:08	markusN:	I'm still convinced of resetting all accounts
   20:11:19	wildintellect:	TemptorSent, do you have access to that machine to poke around?
   20:11:31	TemptorSent:	No idea, and I'd rather not try.
   20:12:03	markusN:	(and I'm in Germany with totally crappy mobile connection... on and off)
   20:12:05	TemptorSent:	It's asking for a compromise of passwords.
   20:12:26	markusN:	mhh
   20:12:27	TemptorSent:	Anyone logging in with a password should subsequently reset their passwords.
   20:12:45	wildintellect:	ya that's part of the greater need to move to key based
   20:12:57	TemptorSent:	Trojaning SSH is a time-honored tradition.,
   20:13:01	wildintellect:	Martin will have a way to key based login as root
   20:13:06	wildintellect:	I believe I have that too
   20:13:10	robe2:	TemptorSent didn't see any jobs running under geotools account
   20:13:14	wildintellect:	so I could add more keys
   20:13:15	robe2:	that was first thing I checked
   20:13:47	TemptorSent:	depending on how good the hackere/kit, they may be cloaked as 'nobody' even.
   20:14:18	TemptorSent:	A good trick is to pick the name of a running process, clone it, and restart yourself periodically.
   20:14:49	robe2:	wildintellect you know if Martin has used up his contract yet?
   20:14:59	TemptorSent:	To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against.
   20:15:01	robe2:	or can we assign him to look into this issue further
   20:15:02	wildintellect:	no idea, strk was overseeing that
   20:15:20	robe2:	and strk appears to be asleep :)
   20:15:57	robe2:	as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so
   20:16:09	robe2:	he was going to start putting in more time this coming week.
   20:16:19	robe2:	So I take that to mean he's still got some unspent time
   20:16:20	TemptorSent:	Without identifying the vector, we must work on the presumption that they have gained privleged access.
   20:17:40	robe2:	TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on.
   20:17:56	robe2:	and of course change the none ldap ones
   20:18:26	robe2:	does that sound like a reasonable start. Guess we also need to scan the whole system for trojans
   20:18:50	TemptorSent:	Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against.
   20:19:30	robe2:	off hand anyone knows what's running on osgeo6
   20:19:45	TemptorSent:	No idea....
   20:19:53	robe2:	was thinking maybe those should be candidates to be moved
   20:19:57	TemptorSent:	Probably on the wiki somewhere.
   20:20:21	TemptorSent:	Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles...
   20:20:55	wildintellect:	martin setup most of what's on osgeo6
   20:21:05	wildintellect:	fyi the list server is on there
   20:21:15	TemptorSent:	Oh, joy.
   20:22:20	TemptorSent:	I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials...
   20:22:51	wildintellect:	https://wiki.osgeo.org/wiki/Osgeo6
   20:22:52	sigabrt:	Title: Osgeo6 - OSGeo (at wiki.osgeo.org)
   20:22:55	TemptorSent:	But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring.
   20:24:55	robe2:	Okay guess we should move on. I'll add a task for martin to look into the issue further.
   20:25:00	TemptorSent:	I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect.
   20:25:31	wildintellect:	quick look the geotools sites are all static sites
   20:26:24	robe2:	I'm actually more concerned at this point at relying too much on Martin's knowledge . I think we need a bit more knowledge coverage
   20:27:05	wildintellect:	well thats my note about new server, and how we can plan to avoid some issues
   20:27:46	TemptorSent:	True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry.
   20:29:33	robe2:	next topic FunToo container
   20:29:41	robe2:	and nextcloud
   20:29:45	wildintellect:	snapshotting, I know we didn't since it's Debian on ext4
   20:30:00	wildintellect:	checksum yes, the backups should have checksums
   20:30:05	robe2:	we have nextcloud running with ldap auth. Need to narrow down groups
   20:30:09	TemptorSent:	Ouch, yeah, unless backups were done at a low level, it'll be hard.
   20:30:18	wildintellect:	we use bacula
   20:30:24	TemptorSent:	robe2 Do we have a group setup for it yet?
   20:30:27	wildintellect:	it's file based
   20:31:03	TemptorSent:	I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered.
   20:31:11	robe2:	that's one reason I prefer VMs and try to keep the base very locked down
   20:31:42	robe2:	TemporSent I highly suspect bacula isn't capturing the rogue things
   20:31:46	TemptorSent:	VMs don't offer as much protection as you might think unfortunately.
   20:32:01	robe2:	I think it is set to only capture some subfolders of which for example /tmp is not a member of
   20:32:14	TemptorSent:	but we can explicitly compare the state BEFORE and determine what has been changed.
   20:32:15	wildintellect:	there's a newer type of container more focused on security than docker
   20:32:24	robe2:	TemporSent but they are easier to snapshot and destroy
   20:33:08	TemptorSent:	Not really easier to snapshot, and come with a lot of overhead.
   20:33:56	TemptorSent:	Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources.
   20:34:04	robe2:	TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot
   20:34:18	wildintellect:	this is conflation of container & VM
   20:34:24	TemptorSent:	Yeah, the containers work great with snapshotting :)
   20:34:44	TemptorSent:	Yes wildintellect.
   20:34:44	robe2:	VMs provide simple snapshotting too :)
   20:35:21	wildintellect:	yes some of them do (qcow base ones, or lvm snapshots)
   20:35:34	robe2:	the only ones worth using :)
   20:35:49	TemptorSent:	But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed.
   20:36:13	robe2:	or a cloud provider where you have a snapshot every day or as you need it
   20:36:39	TemptorSent:	With zfs, snapshots every 15 minutes are no problem.
   20:36:40	robe2:	True anyway lets move on
   20:36:49	TemptorSent:	Just age them out
   20:38:17	TemptorSent:	...
   20:39:10	robe2:	for the ldap groups we don't have one set up specifically for nextcloud
   20:39:25	TemptorSent:	Okay, we might want to do that.
   20:39:26	robe2:	markusN you know if board has a ldap group
   20:39:34	jive[m]:	okay, I am here!
   20:39:39	robe2:	I think we asked that and I forget if the question was answered
   20:39:54	robe2:	jive[m] hi
   20:40:14	robe2:	jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board?
   20:40:30	jodygarnett:	I do not know if there is an LDAP group for the board
   20:40:49	wildintellect:	isn't there an ldap query webpage that lists all the groups?
   20:41:01	jodygarnett:	we are doing our best trying to track member status in the new website, rather than a series of wiki pages ...
   20:41:52	robe2:	wildintellect was looking for that but can't find it
   20:42:08	TemptorSent:	Hmm, sounds like some 'member_of_*' groups are needed.
   20:42:08	robe2:	and too lazy to look up ldapsearch. There is no group called board though
   20:42:28	markusN:	sorry for disconnected
   20:42:34	robe2:	TemportSet yah right now we have it set to allow any osgeo member to share
   20:42:37	markusN:	what was the question?
   20:43:24	TemptorSent:	Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group.
   20:44:03	robe2:	TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to?
   20:44:56	TemptorSent:	Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container.
   20:44:58	robe2:	My internet has been acting flaky today so could be my internet connection
   20:47:17	robe2:	I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects
   20:47:21	TemptorSent:	Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that.
   20:47:56	TemptorSent:	I'll be looking into service supervision at some point.
   20:48:39	TemptorSent:	Okay, you should be able to get to nextcloud.osgeo.org fine now :)
   20:49:19	robe2:	jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org
   20:49:21	sigabrt:	Title: Nextcloud (at nextcloud.osgeo.org)
   20:50:05	robe2:	I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet
   20:51:23	robe2:	next topic wiki ldap integration
   20:52:05	TemptorSent:	Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well
   20:52:53	robe2:	TemtorSent site-wide ldap auth engine?
   20:53:20	robe2:	TemptorSent typo not clear what that is
   20:53:35	jodygarnett:	sorry lost connection
   20:53:38	robe2:	is tht site-wide as in specific to wiki or even more encompassing
   20:54:03	robe2:	jodygarnett no problem my connection has been pretty flaky today too
   20:54:11	TemptorSent:	All of funtoo.org uses a single signon auth essentially.
   20:54:25	jodygarnett:	(what adgenda topic are we on please)
   20:54:36	robe2:	we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere
   20:54:40	delawen[m]:	Thanks!
   20:54:51	TemptorSent:	So you login and it provides the auth tokens to each service, rather than having to login to each individually.
   20:55:59	robe2:	TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc.
   20:56:09	robe2:	doesn't that still need to work with those
   20:56:33	TemptorSent:	Yes, it provides the auth-token to the individual applications.
   20:56:48	TemptorSent:	I'll talk to drobbins on details.
   20:57:15	robe2:	okay would be interesting to see that in action like if I have a funtoo.org account
   20:58:08	TemptorSent:	Yeah, it works on all the funtoo.org services.
   20:58:12	robe2:	jodygarnett I still owe you the proper setup of wordpress git in staging
   20:58:18	TemptorSent:	the wiki, the bug tracker, etc.
   20:58:51	robe2:	then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early
   20:59:07	jodygarnett:	I have a more serious short term website issue, further down in the meetin adgenda
   20:59:54	jodygarnett:	And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability"
   20:59:56	TemptorSent:	Okay, sounds like we're still waiting on status of DB for examination and plotting the migration.
   21:00:09	robe2:	jodygarnette we might be there in the agenda already
   21:00:39	robe2:	TemptorSent yah I was going to look at the db to see how crazy the user setup is
   21:01:01	robe2:	jodygarnett so what is your pressing issue?
   21:01:37	robe2:	oh info@osgeo.org
   21:01:43	jodygarnett:	The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket...
   21:02:17	jodygarnett:	https://trac.osgeo.org/osgeo/ticket/2158
   21:02:18	sigabrt:	Title: #2158 (sponsor logos are taken down too soon) – OSGeo (at trac.osgeo.org)
   21:02:21	robe2:	can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email
   21:03:03	jodygarnett:	because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug.
   21:04:22	jodygarnett:	updated the title to reflect recent testing, captured in the ticket
   21:04:38	jodygarnett:	I was hoping vicky could help, as she worked on a related issue 2071
   21:05:16	robe2:	jodygarnett I think vicky is traveling she's on some crazy worldish tour
   21:05:45	robe2:	she wrote me saying she'll be out of commission until the May 14th
   21:05:59	jodygarnett:	okay cool
   21:06:17	jodygarnett:	I will engage with vendor then, use some of our support hours.
   21:06:26	jodygarnett:	as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159
   21:06:27	sigabrt:	Title: #2159 (Concern expressed over awstats file) – OSGeo (at trac.osgeo.org)
   21:06:50	robe2:	jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea
   21:07:57	robe2:	which sites do we use awstats on?
   21:08:33	robe2:	the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D
   21:08:34	sigabrt:	Title: Index of /logs (at download.osgeo.org)
   21:09:17	robe2:	oh wait that one is just for downloads.osgeo.org not sure why we would publish those
   21:09:57	jodygarnett:	The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example
   21:10:58	TemptorSent:	That should be the least of our worries...
   21:11:42	robe2:	I did notice one had webdav for geotools
   21:11:44	TemptorSent:	Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list.
   21:11:47	robe2:	why are we using webdav
   21:12:10	TemptorSent:	We may not be intentionally...
   21:12:33	TemptorSent:	SVN uses it, so perhaps that bit of kit was piggybacked in using it.
   21:14:16	robe2:	oh
   21:14:37	robe2:	okay looks like we are out of time - start of after party if anyone wants to hang around
   21:14:51	jodygarnett:	we are using it as a poor-mans maven repository
   21:14:56	TemptorSent:	Thank you robe2.
   21:14:58	jodygarnett:	alternative is to deploy something like artifactory
   21:15:22	robe2:	artifactory? what's that
   21:16:57	robe2:	wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it
   21:17:03	robe2:	felt like we were at a standstill
   21:17:12	jodygarnett:	A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/
   21:17:13	sigabrt:	Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com)
   21:17:36	jodygarnett:	no need to look into that suff at present, just answering the question on why we are using webdav
   21:17:52	jodygarnett:	thanks for running the meeting robe2
   21:18:00	robe2:	Too Integrated to Fail :)
   21:18:24	robe2:	great pitch
   21:19:06	jodygarnett:	(If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple)
   21:20:48	TemptorSent:	robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed.
   21:21:31	TemptorSent:	Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily.
   21:23:14	TemptorSent:	canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO.
   21:25:05	robe2:	TemptorSent glad someone has a memory for this
   22:28:55	wildintellect:	robe2, maven built java products rely on webdav to pull artifacts
   22:29:08	wildintellect:	sorry I had another meeting I had to go to
   22:30:35	wildintellect:	TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7