Difference between revisions of "SAC Meeting 2018-02-15"
(10 intermediate revisions by 3 users not shown) | |||
Line 38: | Line 38: | ||
* Discussion pending on list (moderator: could not find the mail thread) | * Discussion pending on list (moderator: could not find the mail thread) | ||
* Needs: | * Needs: | ||
− | ** trac is painfully slow (now seems better with Chris | + | ** trac is painfully slow (now seems better with Chris Giorgi's recent changes to apache config) |
** osgeo6 is running an ancient Debian, needs to be updated | ** osgeo6 is running an ancient Debian, needs to be updated | ||
Line 44: | Line 44: | ||
of moving to ganeti next year when we get more hardware. | of moving to ganeti next year when we get more hardware. | ||
− | DONE: Alex sent quote to mailing list. Chris Gorgi has some ideas, may provide enhancements | + | DONE: Alex sent quote to mailing list, [https://drive.google.com/file/d/1X-z66jXXBUZuPqh6EP0d43g2NUCL7xcL/view?usp=sharing Updated Feb 15] ~$6300-7000. Chris Gorgi has some ideas, may provide enhancements |
− | + | * DONE: provide clarification of new hardware purchase options suitability for hosting type. (wildintellect) | |
+ | ** [https://linuxcontainers.org/lxd/ LXD] was suggested as a drop-in alternative to full Virtualization with [https://www.linux-kvm.org/page/Main_Page KVM] for most service; this more similar to the feel of a virtual machine than say Docker. | ||
+ | ** We can mix, nest, and layer both containers and virtualization interchangeably. | ||
+ | ** Easy candidates for containers include Downloads and Webextra (FOSS4G) which are static files. | ||
+ | * Suggestions: | ||
+ | ** Optane card for extra disk caching beyond memory - Question how do we configure this: | ||
+ | *** 25% is reserved as unallocated to reduce write-wearing and maintain speed over the expected life of the server. | ||
+ | *** A portion (~25-50%) would be allocated to the ZFS L2ARC (or equivalent) to keep warm FS blocks instantly accessible. | ||
+ | *** The remainder is available to be used essentially as a large persistent ramdisk. | ||
+ | ** Fill RAM, mostly caches requests, in particular file downloads | ||
+ | *** This will happen automatically as files are accessed and added to the filesystem's cache. | ||
+ | *** Data can be pre-cached simply by accessing the files and directories -- a simple script can run periodically to ensure they are kept marked as hot. | ||
+ | ** Larger DWPD rating for SSDs to better handle writes - Question, we didn't do this for OSGeo6, anything we should look out for? Should we estimate life of OSGeo6 drives and replace pre-failure next couple of years. | ||
+ | *** Total write volume can be checked with smart-tools and drives approaching the stated limit should be replaced or move to storage-only applications. | ||
+ | *** Write endurance and long-term speed can be improved by leaving 20-50% of each drive unallocated. | ||
+ | ** Mirrored pair of SATA SSDs for write-caching and high io loads. | ||
+ | *** Reserve 25-33% unallocated. | ||
+ | *** Provide ZFS ZIL SLOG with 32-64GB to minimize write latency for data stored on HDDs. | ||
+ | *** The remainder can be used for loads with high mixed read/write transactional loads, such as active databases, mail, etc. | ||
+ | ** 4 TB Spinning disks, still plenty of space (7+ usable), takes less time to rebuild on failure, cost diverted to other features. | ||
+ | *** (Please note - an active storage pool should never be filled to more than 2/3 of capacity to avoid serious performance degradation and fragmentation - C.G.) | ||
+ | |||
− | |||
Any other feedback from others to be discussed | Any other feedback from others to be discussed | ||
=== Others === | === Others === | ||
+ | |||
+ | * Formally ask Chris Giorgi if he'd like to officially join SAC | ||
'''<Fill report here>''' | '''<Fill report here>''' | ||
Line 57: | Line 79: | ||
* Regina looked at postgres load and see if anything amiss there and if any query bottlenecks | * Regina looked at postgres load and see if anything amiss there and if any query bottlenecks | ||
Concluded apache is our main issue. | Concluded apache is our main issue. | ||
− | * Chris | + | * Chris Giorgi made changes to apache config which helped and suggests adding a [https://trac.osgeo.org/osgeo/ticket/1693 caching nameserver with DNSSec] |
+ | * Martin's plans to upgrade OS of trac and other VMS | ||
==== Ticket Triage ==== | ==== Ticket Triage ==== | ||
* More https setup (got a couple of these, last our downloads which seems critical) | * More https setup (got a couple of these, last our downloads which seems critical) | ||
* [https://trac.osgeo.org/osgeo/ticket/2115 Bas says we need downloads.osgeo.org under SSL] | * [https://trac.osgeo.org/osgeo/ticket/2115 Bas says we need downloads.osgeo.org under SSL] | ||
− | + | ** Add Let's Encrypt | |
+ | ** Question, force https by default, I recall some maven stuff breaks on that. | ||
+ | ** Also related there was a bug requiring newer Debian to get correct algorithms for some https services. | ||
* [https://trac.osgeo.org/osgeo/ticket/2116 Add support for registering public user SSH keys in LDAP] | * [https://trac.osgeo.org/osgeo/ticket/2116 Add support for registering public user SSH keys in LDAP] | ||
− | |||
==== GeoForAll DNS==== | ==== GeoForAll DNS==== | ||
Jeff to report on status of GeoForALL and other DNS issues he's been fixing | Jeff to report on status of GeoForALL and other DNS issues he's been fixing | ||
+ | TODO: Keep nudging GeoForAll folks | ||
== Minutes == | == Minutes == | ||
=== Attendance === | === Attendance === | ||
+ | Regina Obe (robe2) | ||
+ | |||
+ | Alex Mandel (wildintellect) | ||
+ | |||
+ | Chris Giorgi (TemptorSent) | ||
+ | |||
+ | Jeff Mckenna (jmckenna) | ||
+ | Sandro Santilli (strk zzz asleep and then woke up at the end) | ||
− | + | Vicky Vergara (cvvergara too busy fighting queries to concentrate) | |
+ | === Full transcript === | ||
+ | [https://wiki.osgeo.org/wiki/Talk:SAC_Meeting_2018-02-15#Transcript Transcript] | ||
= Details = | = Details = |
Latest revision as of 14:36, 15 February 2018
Agenda
Where and When
- in IRC on Freenode, channel: osgeo-sac Feb 15 2018 UTC: 8 PM
- Web IRC client: https://webchat.freenode.net/
Go over status of SSLs - FOSS4G2018
- Website code: https://github.com/foss4g2018/foss4g2018
- The code for the payments is also very simple HTML - we'd be happy to host in a single point
- 3rd party payment service: Pesapal
- post-meeting. To be moved to webextra, with ssl in front
- merge them and have the payment site be something like https://2018.foss4g.org/registration
- ragnvald: https://github.com/timlinux/foss4g2018
- Ensure we have continuity within our payment systems
<Fill report here>
New Website status report
Report:
- [Vicky & Jody] Experimented with what can be done on the site.
- Role Incubator & Role Project PSC to handle Projects.
- See the note: https://trac.osgeo.org/osgeo/ticket/2075
- Please decide if those roles can be created on main site
- [robe] Status of staging.www.osgeo.org now setup on web18a.osgeo.osuosl.org (plans to migrate production to there as well)
Eventual shutoff of cloudvps.com hosting
Sys Admin Contract
- [martin]:
<Fill report here>
Martin to report on what he's been working on.
Migration off old hard-ware AND Virtualization, Containerization, or None
- Discussion pending on list (moderator: could not find the mail thread)
- Needs:
- trac is painfully slow (now seems better with Chris Giorgi's recent changes to apache config)
- osgeo6 is running an ancient Debian, needs to be updated
Leaning to libvrt since it can be done with one server (less resource intensive than ganeti), with possibility of moving to ganeti next year when we get more hardware.
DONE: Alex sent quote to mailing list, Updated Feb 15 ~$6300-7000. Chris Gorgi has some ideas, may provide enhancements
- DONE: provide clarification of new hardware purchase options suitability for hosting type. (wildintellect)
- LXD was suggested as a drop-in alternative to full Virtualization with KVM for most service; this more similar to the feel of a virtual machine than say Docker.
- We can mix, nest, and layer both containers and virtualization interchangeably.
- Easy candidates for containers include Downloads and Webextra (FOSS4G) which are static files.
- Suggestions:
- Optane card for extra disk caching beyond memory - Question how do we configure this:
- 25% is reserved as unallocated to reduce write-wearing and maintain speed over the expected life of the server.
- A portion (~25-50%) would be allocated to the ZFS L2ARC (or equivalent) to keep warm FS blocks instantly accessible.
- The remainder is available to be used essentially as a large persistent ramdisk.
- Fill RAM, mostly caches requests, in particular file downloads
- This will happen automatically as files are accessed and added to the filesystem's cache.
- Data can be pre-cached simply by accessing the files and directories -- a simple script can run periodically to ensure they are kept marked as hot.
- Larger DWPD rating for SSDs to better handle writes - Question, we didn't do this for OSGeo6, anything we should look out for? Should we estimate life of OSGeo6 drives and replace pre-failure next couple of years.
- Total write volume can be checked with smart-tools and drives approaching the stated limit should be replaced or move to storage-only applications.
- Write endurance and long-term speed can be improved by leaving 20-50% of each drive unallocated.
- Mirrored pair of SATA SSDs for write-caching and high io loads.
- Reserve 25-33% unallocated.
- Provide ZFS ZIL SLOG with 32-64GB to minimize write latency for data stored on HDDs.
- The remainder can be used for loads with high mixed read/write transactional loads, such as active databases, mail, etc.
- 4 TB Spinning disks, still plenty of space (7+ usable), takes less time to rebuild on failure, cost diverted to other features.
- (Please note - an active storage pool should never be filled to more than 2/3 of capacity to avoid serious performance degradation and fragmentation - C.G.)
- Optane card for extra disk caching beyond memory - Question how do we configure this:
Any other feedback from others to be discussed
Others
- Formally ask Chris Giorgi if he'd like to officially join SAC
<Fill report here>
Trac SVN status
- Regina looked at postgres load and see if anything amiss there and if any query bottlenecks
Concluded apache is our main issue.
- Chris Giorgi made changes to apache config which helped and suggests adding a caching nameserver with DNSSec
- Martin's plans to upgrade OS of trac and other VMS
Ticket Triage
- More https setup (got a couple of these, last our downloads which seems critical)
- Bas says we need downloads.osgeo.org under SSL
- Add Let's Encrypt
- Question, force https by default, I recall some maven stuff breaks on that.
- Also related there was a bug requiring newer Debian to get correct algorithms for some https services.
- Add support for registering public user SSH keys in LDAP
GeoForAll DNS
Jeff to report on status of GeoForALL and other DNS issues he's been fixing TODO: Keep nudging GeoForAll folks
Minutes
Attendance
Regina Obe (robe2)
Alex Mandel (wildintellect)
Chris Giorgi (TemptorSent)
Jeff Mckenna (jmckenna)
Sandro Santilli (strk zzz asleep and then woke up at the end)
Vicky Vergara (cvvergara too busy fighting queries to concentrate)
Full transcript
Details
Next Meeting
Proposed Time: UTC: Thursday, March 1st, 2018 at 8:00 pm