Difference between revisions of "Talk:SAC Meeting 2018-05-10"

From OSGeo
Jump to navigation Jump to search
(Replaced content with "== Transcript ==")
 
Line 1: Line 1:
 
== Transcript ==
 
== Transcript ==
 +
 +
    20:00:23 robe2: Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10
 +
    20:00:24 sigabrt: Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org)
 +
    20:01:03 robe2: First topic is status of hardware as wildintellect noted still waiting for shipment
 +
    20:01:10 robe2: anything to add to that?
 +
    20:02:07 wildintellect: thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship
 +
    20:02:32 wildintellect: osuosl is aware of the order and expecting it
 +
    20:02:44 robe2: wildintellect great
 +
    20:03:04 robe2: next topic - osgeo6 coin mining issue
 +
    20:03:04 wildintellect: we should probably start discussing the setup plan
 +
    20:03:34 robe2: wildintellect I'll add that to the end of agenda today
 +
    20:03:41 wildintellect: so I'll not this isn't the 1st time we've caught a miner on an osgeo system
 +
    20:03:47 robe2: I think that might take a bit of discussion and flow into after party
 +
    20:04:06 wildintellect: martin found one once, I can't recall which machine, I think adhoc
 +
    20:04:17 wildintellect: that was clearly injected into a website
 +
    20:04:49 markusN: hi sorry for late
 +
    20:05:04 robe2: markusN I wasn't paying attention too closely were you saying j was running under geotools account?
 +
    20:05:51 markusN: np
 +
    20:06:03 robe2: np?
 +
    20:07:08 robe2: anyway can we disable geotools LDAP account or at very least remove for ldap_shell group?
 +
    20:07:21 robe2: ping strk you around?
 +
    20:09:54 TemptorSent: Check crontab entries.
 +
    20:10:53 wildintellect: there was a note that removing users from the ldap_shell group doesnt' work
 +
    20:10:54 TemptorSent: Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools.
 +
    20:11:08 markusN: I'm still convinced of resetting all accounts
 +
    20:11:19 wildintellect: TemptorSent, do you have access to that machine to poke around?
 +
    20:11:31 TemptorSent: No idea, and I'd rather not try.
 +
    20:12:03 markusN: (and I'm in Germany with totally crappy mobile connection... on and off)
 +
    20:12:05 TemptorSent: It's asking for a compromise of passwords.
 +
    20:12:26 markusN: mhh
 +
    20:12:27 TemptorSent: Anyone logging in with a password should subsequently reset their passwords.
 +
    20:12:45 wildintellect: ya that's part of the greater need to move to key based
 +
    20:12:57 TemptorSent: Trojaning SSH is a time-honored tradition.,
 +
    20:13:01 wildintellect: Martin will have a way to key based login as root
 +
    20:13:06 wildintellect: I believe I have that too
 +
    20:13:10 robe2: TemptorSent didn't see any jobs running under geotools account
 +
    20:13:14 wildintellect: so I could add more keys
 +
    20:13:15 robe2: that was first thing I checked
 +
    20:13:47 TemptorSent: depending on how good the hackere/kit, they may be cloaked as 'nobody' even.
 +
    20:14:18 TemptorSent: A good trick is to pick the name of a running process, clone it, and restart yourself periodically.
 +
    20:14:49 robe2: wildintellect you know if Martin has used up his contract yet?
 +
    20:14:59 TemptorSent: To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against.
 +
    20:15:01 robe2: or can we assign him to look into this issue further
 +
    20:15:02 wildintellect: no idea, strk was overseeing that
 +
    20:15:20 robe2: and strk appears to be asleep :)
 +
    20:15:57 robe2: as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so
 +
    20:16:09 robe2: he was going to start putting in more time this coming week.
 +
    20:16:19 robe2: So I take that to mean he's still got some unspent time
 +
    20:16:20 TemptorSent: Without identifying the vector, we must work on the presumption that they have gained privleged access.
 +
    20:17:40 robe2: TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on.
 +
    20:17:56 robe2: and of course change the none ldap ones
 +
    20:18:26 robe2: does that sound like a reasonable start. Guess we also need to scan the whole system for trojans
 +
    20:18:50 TemptorSent: Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against.
 +
    20:19:30 robe2: off hand anyone knows what's running on osgeo6
 +
    20:19:45 TemptorSent: No idea....
 +
    20:19:53 robe2: was thinking maybe those should be candidates to be moved
 +
    20:19:57 TemptorSent: Probably on the wiki somewhere.
 +
    20:20:21 TemptorSent: Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles...
 +
    20:20:55 wildintellect: martin setup most of what's on osgeo6
 +
    20:21:05 wildintellect: fyi the list server is on there
 +
    20:21:15 TemptorSent: Oh, joy.
 +
    20:22:20 TemptorSent: I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials...
 +
    20:22:51 wildintellect: https://wiki.osgeo.org/wiki/Osgeo6
 +
    20:22:52 sigabrt: Title: Osgeo6 - OSGeo (at wiki.osgeo.org)
 +
    20:22:55 TemptorSent: But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring.
 +
    20:24:55 robe2: Okay guess we should move on. I'll add a task for martin to look into the issue further.
 +
    20:25:00 TemptorSent: I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect.
 +
    20:25:31 wildintellect: quick look the geotools sites are all static sites
 +
    20:26:24 robe2: I'm actually more concerned at this point at relying too much on Martin's knowledge . I think we need a bit more knowledge coverage
 +
    20:27:05 wildintellect: well thats my note about new server, and how we can plan to avoid some issues
 +
    20:27:46 TemptorSent: True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry.
 +
    20:29:33 robe2: next topic FunToo container
 +
    20:29:41 robe2: and nextcloud
 +
    20:29:45 wildintellect: snapshotting, I know we didn't since it's Debian on ext4
 +
    20:30:00 wildintellect: checksum yes, the backups should have checksums
 +
    20:30:05 robe2: we have nextcloud running with ldap auth. Need to narrow down groups
 +
    20:30:09 TemptorSent: Ouch, yeah, unless backups were done at a low level, it'll be hard.
 +
    20:30:18 wildintellect: we use bacula
 +
    20:30:24 TemptorSent: robe2 Do we have a group setup for it yet?
 +
    20:30:27 wildintellect: it's file based
 +
    20:31:03 TemptorSent: I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered.
 +
    20:31:11 robe2: that's one reason I prefer VMs and try to keep the base very locked down
 +
    20:31:42 robe2: TemporSent I highly suspect bacula isn't capturing the rogue things
 +
    20:31:46 TemptorSent: VMs don't offer as much protection as you might think unfortunately.
 +
    20:32:01 robe2: I think it is set to only capture some subfolders of which for example /tmp is not a member of
 +
    20:32:14 TemptorSent: but we can explicitly compare the state BEFORE and determine what has been changed.
 +
    20:32:15 wildintellect: there's a newer type of container more focused on security than docker
 +
    20:32:24 robe2: TemporSent but they are easier to snapshot and destroy
 +
    20:33:08 TemptorSent: Not really easier to snapshot, and come with a lot of overhead.
 +
    20:33:56 TemptorSent: Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources.
 +
    20:34:04 robe2: TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot
 +
    20:34:18 wildintellect: this is conflation of container & VM
 +
    20:34:24 TemptorSent: Yeah, the containers work great with snapshotting :)
 +
    20:34:44 TemptorSent: Yes wildintellect.
 +
    20:34:44 robe2: VMs provide simple snapshotting too :)
 +
    20:35:21 wildintellect: yes some of them do (qcow base ones, or lvm snapshots)
 +
    20:35:34 robe2: the only ones worth using :)
 +
    20:35:49 TemptorSent: But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed.
 +
    20:36:13 robe2: or a cloud provider where you have a snapshot every day or as you need it
 +
    20:36:39 TemptorSent: With zfs, snapshots every 15 minutes are no problem.
 +
    20:36:40 robe2: True anyway lets move on
 +
    20:36:49 TemptorSent: Just age them out
 +
    20:38:17 TemptorSent: ...
 +
    20:39:10 robe2: for the ldap groups we don't have one set up specifically for nextcloud
 +
    20:39:25 TemptorSent: Okay, we might want to do that.
 +
    20:39:26 robe2: markusN you know if board has a ldap group
 +
    20:39:34 jive[m]: okay, I am here!
 +
    20:39:39 robe2: I think we asked that and I forget if the question was answered
 +
    20:39:54 robe2: jive[m] hi
 +
    20:40:14 robe2: jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board?
 +
    20:40:30 jodygarnett: I do not know if there is an LDAP group for the board
 +
    20:40:49 wildintellect: isn't there an ldap query webpage that lists all the groups?
 +
    20:41:01 jodygarnett: we are doing our best trying to track member status in the new website, rather than a series of wiki pages ...
 +
    20:41:52 robe2: wildintellect was looking for that but can't find it
 +
    20:42:08 TemptorSent: Hmm, sounds like some 'member_of_*' groups are needed.
 +
    20:42:08 robe2: and too lazy to look up ldapsearch. There is no group called board though
 +
    20:42:28 markusN: sorry for disconnected
 +
    20:42:34 robe2: TemportSet yah right now we have it set to allow any osgeo member to share
 +
    20:42:37 markusN: what was the question?
 +
    20:43:24 TemptorSent: Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group.
 +
    20:44:03 robe2: TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to?
 +
    20:44:56 TemptorSent: Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container.
 +
    20:44:58 robe2: My internet has been acting flaky today so could be my internet connection
 +
    20:47:17 robe2: I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects
 +
    20:47:21 TemptorSent: Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that.
 +
    20:47:56 TemptorSent: I'll be looking into service supervision at some point.
 +
    20:48:39 TemptorSent: Okay, you should be able to get to nextcloud.osgeo.org fine now :)
 +
    20:49:19 robe2: jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org
 +
    20:49:21 sigabrt: Title: Nextcloud (at nextcloud.osgeo.org)
 +
    20:50:05 robe2: I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet
 +
    20:51:23 robe2: next topic wiki ldap integration
 +
    20:52:05 TemptorSent: Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well
 +
    20:52:53 robe2: TemtorSent site-wide ldap auth engine?
 +
    20:53:20 robe2: TemptorSent typo not clear what that is
 +
    20:53:35 jodygarnett: sorry lost connection
 +
    20:53:38 robe2: is tht site-wide as in specific to wiki or even more encompassing
 +
    20:54:03 robe2: jodygarnett no problem my connection has been pretty flaky today too
 +
    20:54:11 TemptorSent: All of funtoo.org uses a single signon auth essentially.
 +
    20:54:25 jodygarnett: (what adgenda topic are we on please)
 +
    20:54:36 robe2: we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere
 +
    20:54:40 delawen[m]: Thanks!
 +
    20:54:51 TemptorSent: So you login and it provides the auth tokens to each service, rather than having to login to each individually.
 +
    20:55:59 robe2: TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc.
 +
    20:56:09 robe2: doesn't that still need to work with those
 +
    20:56:33 TemptorSent: Yes, it provides the auth-token to the individual applications.
 +
    20:56:48 TemptorSent: I'll talk to drobbins on details.
 +
    20:57:15 robe2: okay would be interesting to see that in action like if I have a funtoo.org account
 +
    20:58:08 TemptorSent: Yeah, it works on all the funtoo.org services.
 +
    20:58:12 robe2: jodygarnett I still owe you the proper setup of wordpress git in staging
 +
    20:58:18 TemptorSent: the wiki, the bug tracker, etc.
 +
    20:58:51 robe2: then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early
 +
    20:59:07 jodygarnett: I have a more serious short term website issue, further down in the meetin adgenda
 +
    20:59:54 jodygarnett: And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability"
 +
    20:59:56 TemptorSent: Okay, sounds like we're still waiting on status of DB for examination and plotting the migration.
 +
    21:00:09 robe2: jodygarnette we might be there in the agenda already
 +
    21:00:39 robe2: TemptorSent yah I was going to look at the db to see how crazy the user setup is
 +
    21:01:01 robe2: jodygarnett so what is your pressing issue?
 +
    21:01:37 robe2: oh info@osgeo.org
 +
    21:01:43 jodygarnett: The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket...
 +
    21:02:17 jodygarnett: https://trac.osgeo.org/osgeo/ticket/2158
 +
    21:02:18 sigabrt: Title: #2158 (sponsor logos are taken down too soon) – OSGeo (at trac.osgeo.org)
 +
    21:02:21 robe2: can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email
 +
    21:03:03 jodygarnett: because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug.
 +
    21:04:22 jodygarnett: updated the title to reflect recent testing, captured in the ticket
 +
    21:04:38 jodygarnett: I was hoping vicky could help, as she worked on a related issue 2071
 +
    21:05:16 robe2: jodygarnett I think vicky is traveling she's on some crazy worldish tour
 +
    21:05:45 robe2: she wrote me saying she'll be out of commission until the May 14th
 +
    21:05:59 jodygarnett: okay cool
 +
    21:06:17 jodygarnett: I will engage with vendor then, use some of our support hours.
 +
    21:06:26 jodygarnett: as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159
 +
    21:06:27 sigabrt: Title: #2159 (Concern expressed over awstats file) – OSGeo (at trac.osgeo.org)
 +
    21:06:50 robe2: jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea
 +
    21:07:57 robe2: which sites do we use awstats on?
 +
    21:08:33 robe2: the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D
 +
    21:08:34 sigabrt: Title: Index of /logs (at download.osgeo.org)
 +
    21:09:17 robe2: oh wait that one is just for downloads.osgeo.org not sure why we would publish those
 +
    21:09:57 jodygarnett: The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example
 +
    21:10:58 TemptorSent: That should be the least of our worries...
 +
    21:11:42 robe2: I did notice one had webdav for geotools
 +
    21:11:44 TemptorSent: Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list.
 +
    21:11:47 robe2: why are we using webdav
 +
    21:12:10 TemptorSent: We may not be intentionally...
 +
    21:12:33 TemptorSent: SVN uses it, so perhaps that bit of kit was piggybacked in using it.
 +
    21:14:16 robe2: oh
 +
    21:14:37 robe2: okay looks like we are out of time - start of after party if anyone wants to hang around
 +
    21:14:51 jodygarnett: we are using it as a poor-mans maven repository
 +
    21:14:56 TemptorSent: Thank you robe2.
 +
    21:14:58 jodygarnett: alternative is to deploy something like artifactory
 +
    21:15:22 robe2: artifactory? what's that
 +
    21:16:57 robe2: wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it
 +
    21:17:03 robe2: felt like we were at a standstill
 +
    21:17:12 jodygarnett: A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/
 +
    21:17:13 sigabrt: Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com)
 +
    21:17:36 jodygarnett: no need to look into that suff at present, just answering the question on why we are using webdav
 +
    21:17:52 jodygarnett: thanks for running the meeting robe2
 +
    21:18:00 robe2: Too Integrated to Fail :)
 +
    21:18:24 robe2: great pitch
 +
    21:19:06 jodygarnett: (If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple)
 +
    21:20:48 TemptorSent: robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed.
 +
    21:21:31 TemptorSent: Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily.
 +
    21:23:14 TemptorSent: canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO.
 +
    21:25:05 robe2: TemptorSent glad someone has a memory for this
 +
    22:28:55 wildintellect: robe2, maven built java products rely on webdav to pull artifacts
 +
    22:29:08 wildintellect: sorry I had another meeting I had to go to
 +
    22:30:35 wildintellect: TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7

Latest revision as of 05:58, 12 May 2018

Transcript

   20:00:23	robe2:	Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10
   20:00:24	sigabrt:	Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org)
   20:01:03	robe2:	First topic is status of hardware as wildintellect noted still waiting for shipment
   20:01:10	robe2:	anything to add to that?
   20:02:07	wildintellect:	thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship
   20:02:32	wildintellect:	osuosl is aware of the order and expecting it
   20:02:44	robe2:	wildintellect great
   20:03:04	robe2:	next topic - osgeo6 coin mining issue
   20:03:04	wildintellect:	we should probably start discussing the setup plan
   20:03:34	robe2:	wildintellect I'll add that to the end of agenda today
   20:03:41	wildintellect:	so I'll not this isn't the 1st time we've caught a miner on an osgeo system
   20:03:47	robe2:	I think that might take a bit of discussion and flow into after party
   20:04:06	wildintellect:	martin found one once, I can't recall which machine, I think adhoc
   20:04:17	wildintellect:	that was clearly injected into a website
   20:04:49	markusN:	hi sorry for late
   20:05:04	robe2:	markusN I wasn't paying attention too closely were you saying j was running under geotools account?
   20:05:51	markusN:	np
   20:06:03	robe2:	np?
   20:07:08	robe2:	anyway can we disable geotools LDAP account or at very least remove for ldap_shell group?
   20:07:21	robe2:	ping strk you around?
   20:09:54	TemptorSent:	Check crontab entries.
   20:10:53	wildintellect:	there was a note that removing users from the ldap_shell group doesnt' work
   20:10:54	TemptorSent:	Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools.
   20:11:08	markusN:	I'm still convinced of resetting all accounts
   20:11:19	wildintellect:	TemptorSent, do you have access to that machine to poke around?
   20:11:31	TemptorSent:	No idea, and I'd rather not try.
   20:12:03	markusN:	(and I'm in Germany with totally crappy mobile connection... on and off)
   20:12:05	TemptorSent:	It's asking for a compromise of passwords.
   20:12:26	markusN:	mhh
   20:12:27	TemptorSent:	Anyone logging in with a password should subsequently reset their passwords.
   20:12:45	wildintellect:	ya that's part of the greater need to move to key based
   20:12:57	TemptorSent:	Trojaning SSH is a time-honored tradition.,
   20:13:01	wildintellect:	Martin will have a way to key based login as root
   20:13:06	wildintellect:	I believe I have that too
   20:13:10	robe2:	TemptorSent didn't see any jobs running under geotools account
   20:13:14	wildintellect:	so I could add more keys
   20:13:15	robe2:	that was first thing I checked
   20:13:47	TemptorSent:	depending on how good the hackere/kit, they may be cloaked as 'nobody' even.
   20:14:18	TemptorSent:	A good trick is to pick the name of a running process, clone it, and restart yourself periodically.
   20:14:49	robe2:	wildintellect you know if Martin has used up his contract yet?
   20:14:59	TemptorSent:	To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against.
   20:15:01	robe2:	or can we assign him to look into this issue further
   20:15:02	wildintellect:	no idea, strk was overseeing that
   20:15:20	robe2:	and strk appears to be asleep :)
   20:15:57	robe2:	as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so
   20:16:09	robe2:	he was going to start putting in more time this coming week.
   20:16:19	robe2:	So I take that to mean he's still got some unspent time
   20:16:20	TemptorSent:	Without identifying the vector, we must work on the presumption that they have gained privleged access.
   20:17:40	robe2:	TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on.
   20:17:56	robe2:	and of course change the none ldap ones
   20:18:26	robe2:	does that sound like a reasonable start. Guess we also need to scan the whole system for trojans
   20:18:50	TemptorSent:	Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against.
   20:19:30	robe2:	off hand anyone knows what's running on osgeo6
   20:19:45	TemptorSent:	No idea....
   20:19:53	robe2:	was thinking maybe those should be candidates to be moved
   20:19:57	TemptorSent:	Probably on the wiki somewhere.
   20:20:21	TemptorSent:	Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles...
   20:20:55	wildintellect:	martin setup most of what's on osgeo6
   20:21:05	wildintellect:	fyi the list server is on there
   20:21:15	TemptorSent:	Oh, joy.
   20:22:20	TemptorSent:	I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials...
   20:22:51	wildintellect:	https://wiki.osgeo.org/wiki/Osgeo6
   20:22:52	sigabrt:	Title: Osgeo6 - OSGeo (at wiki.osgeo.org)
   20:22:55	TemptorSent:	But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring.
   20:24:55	robe2:	Okay guess we should move on. I'll add a task for martin to look into the issue further.
   20:25:00	TemptorSent:	I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect.
   20:25:31	wildintellect:	quick look the geotools sites are all static sites
   20:26:24	robe2:	I'm actually more concerned at this point at relying too much on Martin's knowledge . I think we need a bit more knowledge coverage
   20:27:05	wildintellect:	well thats my note about new server, and how we can plan to avoid some issues
   20:27:46	TemptorSent:	True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry.
   20:29:33	robe2:	next topic FunToo container
   20:29:41	robe2:	and nextcloud
   20:29:45	wildintellect:	snapshotting, I know we didn't since it's Debian on ext4
   20:30:00	wildintellect:	checksum yes, the backups should have checksums
   20:30:05	robe2:	we have nextcloud running with ldap auth. Need to narrow down groups
   20:30:09	TemptorSent:	Ouch, yeah, unless backups were done at a low level, it'll be hard.
   20:30:18	wildintellect:	we use bacula
   20:30:24	TemptorSent:	robe2 Do we have a group setup for it yet?
   20:30:27	wildintellect:	it's file based
   20:31:03	TemptorSent:	I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered.
   20:31:11	robe2:	that's one reason I prefer VMs and try to keep the base very locked down
   20:31:42	robe2:	TemporSent I highly suspect bacula isn't capturing the rogue things
   20:31:46	TemptorSent:	VMs don't offer as much protection as you might think unfortunately.
   20:32:01	robe2:	I think it is set to only capture some subfolders of which for example /tmp is not a member of
   20:32:14	TemptorSent:	but we can explicitly compare the state BEFORE and determine what has been changed.
   20:32:15	wildintellect:	there's a newer type of container more focused on security than docker
   20:32:24	robe2:	TemporSent but they are easier to snapshot and destroy
   20:33:08	TemptorSent:	Not really easier to snapshot, and come with a lot of overhead.
   20:33:56	TemptorSent:	Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources.
   20:34:04	robe2:	TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot
   20:34:18	wildintellect:	this is conflation of container & VM
   20:34:24	TemptorSent:	Yeah, the containers work great with snapshotting :)
   20:34:44	TemptorSent:	Yes wildintellect.
   20:34:44	robe2:	VMs provide simple snapshotting too :)
   20:35:21	wildintellect:	yes some of them do (qcow base ones, or lvm snapshots)
   20:35:34	robe2:	the only ones worth using :)
   20:35:49	TemptorSent:	But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed.
   20:36:13	robe2:	or a cloud provider where you have a snapshot every day or as you need it
   20:36:39	TemptorSent:	With zfs, snapshots every 15 minutes are no problem.
   20:36:40	robe2:	True anyway lets move on
   20:36:49	TemptorSent:	Just age them out
   20:38:17	TemptorSent:	...
   20:39:10	robe2:	for the ldap groups we don't have one set up specifically for nextcloud
   20:39:25	TemptorSent:	Okay, we might want to do that.
   20:39:26	robe2:	markusN you know if board has a ldap group
   20:39:34	jive[m]:	okay, I am here!
   20:39:39	robe2:	I think we asked that and I forget if the question was answered
   20:39:54	robe2:	jive[m] hi
   20:40:14	robe2:	jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board?
   20:40:30	jodygarnett:	I do not know if there is an LDAP group for the board
   20:40:49	wildintellect:	isn't there an ldap query webpage that lists all the groups?
   20:41:01	jodygarnett:	we are doing our best trying to track member status in the new website, rather than a series of wiki pages ...
   20:41:52	robe2:	wildintellect was looking for that but can't find it
   20:42:08	TemptorSent:	Hmm, sounds like some 'member_of_*' groups are needed.
   20:42:08	robe2:	and too lazy to look up ldapsearch. There is no group called board though
   20:42:28	markusN:	sorry for disconnected
   20:42:34	robe2:	TemportSet yah right now we have it set to allow any osgeo member to share
   20:42:37	markusN:	what was the question?
   20:43:24	TemptorSent:	Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group.
   20:44:03	robe2:	TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to?
   20:44:56	TemptorSent:	Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container.
   20:44:58	robe2:	My internet has been acting flaky today so could be my internet connection
   20:47:17	robe2:	I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects
   20:47:21	TemptorSent:	Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that.
   20:47:56	TemptorSent:	I'll be looking into service supervision at some point.
   20:48:39	TemptorSent:	Okay, you should be able to get to nextcloud.osgeo.org fine now :)
   20:49:19	robe2:	jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org
   20:49:21	sigabrt:	Title: Nextcloud (at nextcloud.osgeo.org)
   20:50:05	robe2:	I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet
   20:51:23	robe2:	next topic wiki ldap integration
   20:52:05	TemptorSent:	Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well
   20:52:53	robe2:	TemtorSent site-wide ldap auth engine?
   20:53:20	robe2:	TemptorSent typo not clear what that is
   20:53:35	jodygarnett:	sorry lost connection
   20:53:38	robe2:	is tht site-wide as in specific to wiki or even more encompassing
   20:54:03	robe2:	jodygarnett no problem my connection has been pretty flaky today too
   20:54:11	TemptorSent:	All of funtoo.org uses a single signon auth essentially.
   20:54:25	jodygarnett:	(what adgenda topic are we on please)
   20:54:36	robe2:	we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere
   20:54:40	delawen[m]:	Thanks!
   20:54:51	TemptorSent:	So you login and it provides the auth tokens to each service, rather than having to login to each individually.
   20:55:59	robe2:	TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc.
   20:56:09	robe2:	doesn't that still need to work with those
   20:56:33	TemptorSent:	Yes, it provides the auth-token to the individual applications.
   20:56:48	TemptorSent:	I'll talk to drobbins on details.
   20:57:15	robe2:	okay would be interesting to see that in action like if I have a funtoo.org account
   20:58:08	TemptorSent:	Yeah, it works on all the funtoo.org services.
   20:58:12	robe2:	jodygarnett I still owe you the proper setup of wordpress git in staging
   20:58:18	TemptorSent:	the wiki, the bug tracker, etc.
   20:58:51	robe2:	then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early
   20:59:07	jodygarnett:	I have a more serious short term website issue, further down in the meetin adgenda
   20:59:54	jodygarnett:	And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability"
   20:59:56	TemptorSent:	Okay, sounds like we're still waiting on status of DB for examination and plotting the migration.
   21:00:09	robe2:	jodygarnette we might be there in the agenda already
   21:00:39	robe2:	TemptorSent yah I was going to look at the db to see how crazy the user setup is
   21:01:01	robe2:	jodygarnett so what is your pressing issue?
   21:01:37	robe2:	oh info@osgeo.org
   21:01:43	jodygarnett:	The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket...
   21:02:17	jodygarnett:	https://trac.osgeo.org/osgeo/ticket/2158
   21:02:18	sigabrt:	Title: #2158 (sponsor logos are taken down too soon) – OSGeo (at trac.osgeo.org)
   21:02:21	robe2:	can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email
   21:03:03	jodygarnett:	because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug.
   21:04:22	jodygarnett:	updated the title to reflect recent testing, captured in the ticket
   21:04:38	jodygarnett:	I was hoping vicky could help, as she worked on a related issue 2071
   21:05:16	robe2:	jodygarnett I think vicky is traveling she's on some crazy worldish tour
   21:05:45	robe2:	she wrote me saying she'll be out of commission until the May 14th
   21:05:59	jodygarnett:	okay cool
   21:06:17	jodygarnett:	I will engage with vendor then, use some of our support hours.
   21:06:26	jodygarnett:	as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159
   21:06:27	sigabrt:	Title: #2159 (Concern expressed over awstats file) – OSGeo (at trac.osgeo.org)
   21:06:50	robe2:	jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea
   21:07:57	robe2:	which sites do we use awstats on?
   21:08:33	robe2:	the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D
   21:08:34	sigabrt:	Title: Index of /logs (at download.osgeo.org)
   21:09:17	robe2:	oh wait that one is just for downloads.osgeo.org not sure why we would publish those
   21:09:57	jodygarnett:	The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example
   21:10:58	TemptorSent:	That should be the least of our worries...
   21:11:42	robe2:	I did notice one had webdav for geotools
   21:11:44	TemptorSent:	Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list.
   21:11:47	robe2:	why are we using webdav
   21:12:10	TemptorSent:	We may not be intentionally...
   21:12:33	TemptorSent:	SVN uses it, so perhaps that bit of kit was piggybacked in using it.
   21:14:16	robe2:	oh
   21:14:37	robe2:	okay looks like we are out of time - start of after party if anyone wants to hang around
   21:14:51	jodygarnett:	we are using it as a poor-mans maven repository
   21:14:56	TemptorSent:	Thank you robe2.
   21:14:58	jodygarnett:	alternative is to deploy something like artifactory
   21:15:22	robe2:	artifactory? what's that
   21:16:57	robe2:	wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it
   21:17:03	robe2:	felt like we were at a standstill
   21:17:12	jodygarnett:	A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/
   21:17:13	sigabrt:	Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com)
   21:17:36	jodygarnett:	no need to look into that suff at present, just answering the question on why we are using webdav
   21:17:52	jodygarnett:	thanks for running the meeting robe2
   21:18:00	robe2:	Too Integrated to Fail :)
   21:18:24	robe2:	great pitch
   21:19:06	jodygarnett:	(If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple)
   21:20:48	TemptorSent:	robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed.
   21:21:31	TemptorSent:	Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily.
   21:23:14	TemptorSent:	canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO.
   21:25:05	robe2:	TemptorSent glad someone has a memory for this
   22:28:55	wildintellect:	robe2, maven built java products rely on webdav to pull artifacts
   22:29:08	wildintellect:	sorry I had another meeting I had to go to
   22:30:35	wildintellect:	TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7
Retrieved from "http://wiki.osgeo.org/w/index.php?title=Talk:SAC_Meeting_2018-05-10&oldid=114901"