Transcript
20:00:23 robe2: Everyone ready to meet - https://wiki.osgeo.org/index.php?title=SAC_Meeting_2018-05-10
20:00:24 sigabrt: Title: SAC Meeting 2018-05-10 - OSGeo (at wiki.osgeo.org)
20:01:03 robe2: First topic is status of hardware as wildintellect noted still waiting for shipment
20:01:10 robe2: anything to add to that?
20:02:07 wildintellect: thats all I know it usually takes 1-2 weeks for them to build and test the components before they ship
20:02:32 wildintellect: osuosl is aware of the order and expecting it
20:02:44 robe2: wildintellect great
20:03:04 robe2: next topic - osgeo6 coin mining issue
20:03:04 wildintellect: we should probably start discussing the setup plan
20:03:34 robe2: wildintellect I'll add that to the end of agenda today
20:03:41 wildintellect: so I'll not this isn't the 1st time we've caught a miner on an osgeo system
20:03:47 robe2: I think that might take a bit of discussion and flow into after party
20:04:06 wildintellect: martin found one once, I can't recall which machine, I think adhoc
20:04:17 wildintellect: that was clearly injected into a website
20:04:49 markusN: hi sorry for late
20:05:04 robe2: markusN I wasn't paying attention too closely were you saying j was running under geotools account?
20:05:51 markusN: np
20:06:03 robe2: np?
20:07:08 robe2: anyway can we disable geotools LDAP account or at very least remove for ldap_shell group?
20:07:21 robe2: ping strk you around?
20:09:54 TemptorSent: Check crontab entries.
20:10:53 wildintellect: there was a note that removing users from the ldap_shell group doesnt' work
20:10:54 TemptorSent: Try to determine what the means of CnC is, because backdoors or reentry ports are common with such tools.
20:11:08 markusN: I'm still convinced of resetting all accounts
20:11:19 wildintellect: TemptorSent, do you have access to that machine to poke around?
20:11:31 TemptorSent: No idea, and I'd rather not try.
20:12:03 markusN: (and I'm in Germany with totally crappy mobile connection... on and off)
20:12:05 TemptorSent: It's asking for a compromise of passwords.
20:12:26 markusN: mhh
20:12:27 TemptorSent: Anyone logging in with a password should subsequently reset their passwords.
20:12:45 wildintellect: ya that's part of the greater need to move to key based
20:12:57 TemptorSent: Trojaning SSH is a time-honored tradition.,
20:13:01 wildintellect: Martin will have a way to key based login as root
20:13:06 wildintellect: I believe I have that too
20:13:10 robe2: TemptorSent didn't see any jobs running under geotools account
20:13:14 wildintellect: so I could add more keys
20:13:15 robe2: that was first thing I checked
20:13:47 TemptorSent: depending on how good the hackere/kit, they may be cloaked as 'nobody' even.
20:14:18 TemptorSent: A good trick is to pick the name of a running process, clone it, and restart yourself periodically.
20:14:49 robe2: wildintellect you know if Martin has used up his contract yet?
20:14:59 TemptorSent: To be honest, I wouldn't trust much of anything without having proper logs and and audit list to check against.
20:15:01 robe2: or can we assign him to look into this issue further
20:15:02 wildintellect: no idea, strk was overseeing that
20:15:20 robe2: and strk appears to be asleep :)
20:15:57 robe2: as I recall I think we asked Martin in last meeting and he said he still had time but got tied up with other emergencies in past 2 weeks or so
20:16:09 robe2: he was going to start putting in more time this coming week.
20:16:19 robe2: So I take that to mean he's still got some unspent time
20:16:20 TemptorSent: Without identifying the vector, we must work on the presumption that they have gained privleged access.
20:17:40 robe2: TemptorSent agree so at very least everyone in ldap_access should reset their passwords and we must make sure to only log in with ssh keys from now on.
20:17:56 robe2: and of course change the none ldap ones
20:18:26 robe2: does that sound like a reasonable start. Guess we also need to scan the whole system for trojans
20:18:50 TemptorSent: Yes. And presume that the machine has been rootkitted, which we don't have a means of detecting unless we took a snapshot before that we can diff against.
20:19:30 robe2: off hand anyone knows what's running on osgeo6
20:19:45 TemptorSent: No idea....
20:19:53 robe2: was thinking maybe those should be candidates to be moved
20:19:57 TemptorSent: Probably on the wiki somewhere.
20:20:21 TemptorSent: Yeah, let's not move anything without having a way of verifying we're not transporting zebra muscles...
20:20:55 wildintellect: martin setup most of what's on osgeo6
20:21:05 wildintellect: fyi the list server is on there
20:21:15 TemptorSent: Oh, joy.
20:22:20 TemptorSent: I will say that condsideing we found a cryptominer that wasn't well masked, we can HOPE that it was a script-kiddy, not someone more sophisticated running a slurp of addresses, ips, and credentials...
20:22:51 wildintellect: https://wiki.osgeo.org/wiki/Osgeo6
20:22:52 sigabrt: Title: Osgeo6 - OSGeo (at wiki.osgeo.org)
20:22:55 TemptorSent: But the later are worth big money in the black-hat world, so I wouldn't bet against the cryptomining being a red-herring.
20:24:55 robe2: Okay guess we should move on. I'll add a task for martin to look into the issue further.
20:25:00 TemptorSent: I've had such layered attacks carried out against targets I saw after the fact -- clever, and very, very hard to detect.
20:25:31 wildintellect: quick look the geotools sites are all static sites
20:26:24 robe2: I'm actually more concerned at this point at relying too much on Martin's knowledge . I think we need a bit more knowledge coverage
20:27:05 wildintellect: well thats my note about new server, and how we can plan to avoid some issues
20:27:46 TemptorSent: True, but unless someone throught to run a checksum over the whole thing at the beginning and running periodic full snapshotting, we'll probably never know for sure when or how they gained entry.
20:29:33 robe2: next topic FunToo container
20:29:41 robe2: and nextcloud
20:29:45 wildintellect: snapshotting, I know we didn't since it's Debian on ext4
20:30:00 wildintellect: checksum yes, the backups should have checksums
20:30:05 robe2: we have nextcloud running with ldap auth. Need to narrow down groups
20:30:09 TemptorSent: Ouch, yeah, unless backups were done at a low level, it'll be hard.
20:30:18 wildintellect: we use bacula
20:30:24 TemptorSent: robe2 Do we have a group setup for it yet?
20:30:27 wildintellect: it's file based
20:31:03 TemptorSent: I'll have to see what bacula captures, if we can get a delta from before/after the compromise, we might be able to say something about what was altered.
20:31:11 robe2: that's one reason I prefer VMs and try to keep the base very locked down
20:31:42 robe2: TemporSent I highly suspect bacula isn't capturing the rogue things
20:31:46 TemptorSent: VMs don't offer as much protection as you might think unfortunately.
20:32:01 robe2: I think it is set to only capture some subfolders of which for example /tmp is not a member of
20:32:14 TemptorSent: but we can explicitly compare the state BEFORE and determine what has been changed.
20:32:15 wildintellect: there's a newer type of container more focused on security than docker
20:32:24 robe2: TemporSent but they are easier to snapshot and destroy
20:33:08 TemptorSent: Not really easier to snapshot, and come with a lot of overhead.
20:33:56 TemptorSent: Running one container-per-service is quite reasonable, while running a vm-per-service quickly eats all resources.
20:34:04 robe2: TemptorSent you'll have to educate me on that sometime maybe it's just cause I'm used to all the container stuff providing a quick command snapshot
20:34:18 wildintellect: this is conflation of container & VM
20:34:24 TemptorSent: Yeah, the containers work great with snapshotting :)
20:34:44 TemptorSent: Yes wildintellect.
20:34:44 robe2: VMs provide simple snapshotting too :)
20:35:21 wildintellect: yes some of them do (qcow base ones, or lvm snapshots)
20:35:34 robe2: the only ones worth using :)
20:35:49 TemptorSent: But they are very ham-fisted in how they snapshot, and it's not at all easy to see what changed.
20:36:13 robe2: or a cloud provider where you have a snapshot every day or as you need it
20:36:39 TemptorSent: With zfs, snapshots every 15 minutes are no problem.
20:36:40 robe2: True anyway lets move on
20:36:49 TemptorSent: Just age them out
20:38:17 TemptorSent: ...
20:39:10 robe2: for the ldap groups we don't have one set up specifically for nextcloud
20:39:25 TemptorSent: Okay, we might want to do that.
20:39:26 robe2: markusN you know if board has a ldap group
20:39:34 jive[m]: okay, I am here!
20:39:39 robe2: I think we asked that and I forget if the question was answered
20:39:54 robe2: jive[m] hi
20:40:14 robe2: jive[m] perhaps you can answer the board question you are on board. Is there an ldap group for board?
20:40:30 jodygarnett: I do not know if there is an LDAP group for the board
20:40:49 wildintellect: isn't there an ldap query webpage that lists all the groups?
20:41:01 jodygarnett: we are doing our best trying to track member status in the new website, rather than a series of wiki pages ...
20:41:52 robe2: wildintellect was looking for that but can't find it
20:42:08 TemptorSent: Hmm, sounds like some 'member_of_*' groups are needed.
20:42:08 robe2: and too lazy to look up ldapsearch. There is no group called board though
20:42:28 markusN: sorry for disconnected
20:42:34 robe2: TemportSet yah right now we have it set to allow any osgeo member to share
20:42:37 markusN: what was the question?
20:43:24 TemptorSent: Right robe2, we probably want to at least split up access rights, as well as have a 'nextcloud_admin' role or similar as a group.
20:44:03 robe2: TemptorSent I don't seem to be able to get to nextcloud.osgeo.org are you able to?
20:44:56 TemptorSent: Nope -- server was restarted earlier for kernel upgrade, lemme see if we forgot to set something to autostart in the container.
20:44:58 robe2: My internet has been acting flaky today so could be my internet connection
20:47:17 robe2: I don't think I have access to create new groups -- I presume I need to be in this list - https://id.osgeo.org/ldap/group?group=admin&ou=projects
20:47:21 TemptorSent: Back up, nginx had failed to start, but had no problem starting manually -- I'll look into that.
20:47:56 TemptorSent: I'll be looking into service supervision at some point.
20:48:39 TemptorSent: Okay, you should be able to get to nextcloud.osgeo.org fine now :)
20:49:19 robe2: jive[m] markusN delawen[m] if you want to take a test drive while we are sorting out the permissions the link is - https://nextcloud.osgeo.org
20:49:21 sigabrt: Title: Nextcloud (at nextcloud.osgeo.org)
20:50:05 robe2: I haven't finished setting up the ssh via ldap on osgeo.host@funtoo yet
20:51:23 robe2: next topic wiki ldap integration
20:52:05 TemptorSent: Oh, any issue there? If so, I'm sure drobbins could help -- also, has a pretty functional site-wide ldap auth engine that he's releasing that may help as part of the solution for our wiki issues as well
20:52:53 robe2: TemtorSent site-wide ldap auth engine?
20:53:20 robe2: TemptorSent typo not clear what that is
20:53:35 jodygarnett: sorry lost connection
20:53:38 robe2: is tht site-wide as in specific to wiki or even more encompassing
20:54:03 robe2: jodygarnett no problem my connection has been pretty flaky today too
20:54:11 TemptorSent: All of funtoo.org uses a single signon auth essentially.
20:54:25 jodygarnett: (what adgenda topic are we on please)
20:54:36 robe2: we were just talking about wiki ldap. I recall we left off with Martin getting us a backup of the database. I forget if he did and just put it somewhere
20:54:40 delawen[m]: Thanks!
20:54:51 TemptorSent: So you login and it provides the auth tokens to each service, rather than having to login to each individually.
20:55:59 robe2: TemptorSent still a bit lost how that integrates with specific apps like wordpress, nextcloud, drupal, wiki etc.
20:56:09 robe2: doesn't that still need to work with those
20:56:33 TemptorSent: Yes, it provides the auth-token to the individual applications.
20:56:48 TemptorSent: I'll talk to drobbins on details.
20:57:15 robe2: okay would be interesting to see that in action like if I have a funtoo.org account
20:58:08 TemptorSent: Yeah, it works on all the funtoo.org services.
20:58:12 robe2: jodygarnett I still owe you the proper setup of wordpress git in staging
20:58:18 TemptorSent: the wiki, the bug tracker, etc.
20:58:51 robe2: then we can do all the crazy changes in the pages and split up of month sponsors without worrying about pushing things to production too early
20:59:07 jodygarnett: I have a more serious short term website issue, further down in the meetin adgenda
20:59:54 jodygarnett: And although I did not add it to the adgenda, a info@osgeo.org email came in a couple days ago with a "possible security vulnerability"
20:59:56 TemptorSent: Okay, sounds like we're still waiting on status of DB for examination and plotting the migration.
21:00:09 robe2: jodygarnette we might be there in the agenda already
21:00:39 robe2: TemptorSent yah I was going to look at the db to see how crazy the user setup is
21:01:01 robe2: jodygarnett so what is your pressing issue?
21:01:37 robe2: oh info@osgeo.org
21:01:43 jodygarnett: The sponsors logo page is "busted", I have been adding new sponsors and they are not shown. I have a ticket...
21:02:17 jodygarnett: https://trac.osgeo.org/osgeo/ticket/2158
21:02:18 sigabrt: Title: #2158 (sponsor logos are taken down too soon) – OSGeo (at trac.osgeo.org)
21:02:21 robe2: can you send me the info@osgeo.org email (I don't think I'm on that list) not sure who gets that email
21:03:03 jodygarnett: because we are close to event season many organizations are sponsoring, 4 in the last week, .... so this ends up being a very visible bug.
21:04:22 jodygarnett: updated the title to reflect recent testing, captured in the ticket
21:04:38 jodygarnett: I was hoping vicky could help, as she worked on a related issue 2071
21:05:16 robe2: jodygarnett I think vicky is traveling she's on some crazy worldish tour
21:05:45 robe2: she wrote me saying she'll be out of commission until the May 14th
21:05:59 jodygarnett: okay cool
21:06:17 jodygarnett: I will engage with vendor then, use some of our support hours.
21:06:26 jodygarnett: as for the info email, reported here: https://trac.osgeo.org/osgeo/ticket/2159
21:06:27 sigabrt: Title: #2159 (Concern expressed over awstats file) – OSGeo (at trac.osgeo.org)
21:06:50 robe2: jodygarnett I am planning to resetup dev tonight (I'll restore latest prod backup) so will be ready for testing and automatic pulling from gitea
21:07:57 robe2: which sites do we use awstats on?
21:08:33 robe2: the logs here haven't been updated since Feb - https://download.osgeo.org/logs/?C=M;O=D
21:08:34 sigabrt: Title: Index of /logs (at download.osgeo.org)
21:09:17 robe2: oh wait that one is just for downloads.osgeo.org not sure why we would publish those
21:09:57 jodygarnett: The bug report indicates concerns over publishing the contents of those files, they show internal directory structure for example
21:10:58 TemptorSent: That should be the least of our worries...
21:11:42 robe2: I did notice one had webdav for geotools
21:11:44 TemptorSent: Granted, there is no reason to expose them, but as vulnarabilities go, that's reasonably low on the list.
21:11:47 robe2: why are we using webdav
21:12:10 TemptorSent: We may not be intentionally...
21:12:33 TemptorSent: SVN uses it, so perhaps that bit of kit was piggybacked in using it.
21:14:16 robe2: oh
21:14:37 robe2: okay looks like we are out of time - start of after party if anyone wants to hang around
21:14:51 jodygarnett: we are using it as a poor-mans maven repository
21:14:56 TemptorSent: Thank you robe2.
21:14:58 jodygarnett: alternative is to deploy something like artifactory
21:15:22 robe2: artifactory? what's that
21:16:57 robe2: wildintellect you wanted to discuss plans for new server. I forget where we left off with what kind of container/ vm thingy we were going to put on it
21:17:03 robe2: felt like we were at a standstill
21:17:12 jodygarnett: A fancy artifact repository, speaks a couple kinds of protocols not just maven. https://jfrog.com/artifactory/
21:17:13 sigabrt: Title: Artifactory - Universal Artifact Repository Manager - JFrog (at jfrog.com)
21:17:36 jodygarnett: no need to look into that suff at present, just answering the question on why we are using webdav
21:17:52 jodygarnett: thanks for running the meeting robe2
21:18:00 robe2: Too Integrated to Fail :)
21:18:24 robe2: great pitch
21:19:06 jodygarnett: (If the time comes it is not hard to migrate from webdav to artifactory or nexas, webdav is just nice and simple)
21:20:48 TemptorSent: robe2 Last I recall was ubuntu + zfs + lxd + kvm/qemu vms as needed.
21:21:31 TemptorSent: Ideally it shouldn't matter too much as long as it's stable, as all the actual work is done inside containers, which can be managed easily.
21:23:14 TemptorSent: canonical offers support for both zfs and lxd directly, including paid support contracts if needed, and everyone else is already comfortable with debian semantics it seems, so that's a good choice IMHO.
21:25:05 robe2: TemptorSent glad someone has a memory for this
22:28:55 wildintellect: robe2, maven built java products rely on webdav to pull artifacts
22:29:08 wildintellect: sorry I had another meeting I had to go to
22:30:35 wildintellect: TemptorSent, we should probably make a new wiki page for the incoming machine osgeo7