Difference between revisions of "SAC Meeting 2018-02-15"

From OSGeo
Jump to navigation Jump to search
 
(10 intermediate revisions by 3 users not shown)
Line 38: Line 38:
 
* Discussion pending on list (moderator: could not find the mail thread)
 
* Discussion pending on list (moderator: could not find the mail thread)
 
* Needs:
 
* Needs:
** trac is painfully slow (now seems better with Chris Gorgi's recent changes to apache config)
+
** trac is painfully slow (now seems better with Chris Giorgi's recent changes to apache config)
 
** osgeo6 is running an ancient Debian, needs to be updated
 
** osgeo6 is running an ancient Debian, needs to be updated
  
Line 44: Line 44:
 
of moving to ganeti next year when we get more hardware.   
 
of moving to ganeti next year when we get more hardware.   
  
DONE: Alex sent quote to mailing list.  Chris Gorgi has some ideas, may provide enhancements
+
DONE: Alex sent quote to mailing list, [https://drive.google.com/file/d/1X-z66jXXBUZuPqh6EP0d43g2NUCL7xcL/view?usp=sharing Updated Feb 15] ~$6300-7000.  Chris Gorgi has some ideas, may provide enhancements
'''<Fill report here>'''
+
* DONE: provide clarification of new hardware purchase options suitability for hosting type. (wildintellect)
 +
** [https://linuxcontainers.org/lxd/ LXD] was suggested as a drop-in alternative to full Virtualization with [https://www.linux-kvm.org/page/Main_Page KVM] for most service; this more similar to the feel of a virtual machine than say Docker.
 +
** We can mix, nest, and layer both containers and virtualization interchangeably.
 +
** Easy candidates for containers include Downloads and Webextra (FOSS4G) which are static files.
 +
* Suggestions:
 +
** Optane card for extra disk caching beyond memory - Question how do we configure this:
 +
*** 25% is reserved as unallocated to reduce write-wearing and maintain speed over the expected life of the server.
 +
*** A portion (~25-50%) would be allocated to the ZFS L2ARC (or equivalent) to keep warm FS blocks instantly accessible.
 +
*** The remainder is available to be used essentially as a large persistent ramdisk.
 +
** Fill RAM, mostly caches requests, in particular file downloads
 +
*** This will happen automatically as files are accessed and added to the filesystem's cache.
 +
*** Data can be pre-cached simply by accessing the files and directories -- a simple script can run periodically to ensure they are kept marked as hot.
 +
** Larger DWPD rating for SSDs to better handle writes - Question, we didn't do this for OSGeo6, anything we should look out for? Should we estimate life of OSGeo6 drives and replace pre-failure next couple of years.
 +
*** Total write volume can be checked with smart-tools and drives approaching the stated limit should be replaced or move to storage-only applications.
 +
*** Write endurance and long-term speed can be improved by leaving 20-50% of each drive unallocated.
 +
** Mirrored pair of SATA SSDs for write-caching and high io loads.
 +
*** Reserve 25-33% unallocated.
 +
*** Provide ZFS ZIL SLOG with 32-64GB to minimize write latency for data stored on HDDs.
 +
*** The remainder can be used for loads with high mixed read/write transactional loads, such as active databases, mail, etc.
 +
** 4 TB Spinning disks, still plenty of space (7+ usable), takes less time to rebuild on failure, cost diverted to other features.
 +
*** (Please note - an active storage pool should never be filled to more than 2/3 of capacity to avoid serious performance degradation and fragmentation - C.G.)
 +
 
  
* DONE: provide clarification of new hardware purchase options suitability for hosting type. (wildintellect)
 
 
Any other feedback from others to be discussed
 
Any other feedback from others to be discussed
  
 
=== Others ===
 
=== Others ===
 +
 +
* Formally ask Chris Giorgi if he'd like to officially join SAC
  
 
'''<Fill report here>'''
 
'''<Fill report here>'''
Line 57: Line 79:
 
* Regina looked at postgres load and see if anything amiss there and if any query bottlenecks
 
* Regina looked at postgres load and see if anything amiss there and if any query bottlenecks
 
Concluded apache is our main issue.  
 
Concluded apache is our main issue.  
* Chris Gorgi made changes to apache config which helped and suggests [https://trac.osgeo.org/osgeo/ticket/1693 changes to DNSSec]
+
* Chris Giorgi made changes to apache config which helped and suggests adding a [https://trac.osgeo.org/osgeo/ticket/1693 caching nameserver with DNSSec]
 +
* Martin's plans to upgrade OS of trac and other VMS
  
 
==== Ticket Triage ====
 
==== Ticket Triage ====
 
* More https setup (got a couple of these, last our downloads which seems critical)  
 
* More https setup (got a couple of these, last our downloads which seems critical)  
 
* [https://trac.osgeo.org/osgeo/ticket/2115 Bas says we need downloads.osgeo.org under SSL]
 
* [https://trac.osgeo.org/osgeo/ticket/2115 Bas says we need downloads.osgeo.org under SSL]
 
+
** Add Let's Encrypt
 +
** Question, force https by default, I recall some maven stuff breaks on that.
 +
** Also related there was a bug requiring newer Debian to get correct algorithms for some https services.
 
* [https://trac.osgeo.org/osgeo/ticket/2116 Add support for registering public user SSH keys in LDAP]
 
* [https://trac.osgeo.org/osgeo/ticket/2116 Add support for registering public user SSH keys in LDAP]
 
  
 
==== GeoForAll DNS====
 
==== GeoForAll DNS====
  
 
Jeff to report on status of GeoForALL and other DNS issues he's been fixing
 
Jeff to report on status of GeoForALL and other DNS issues he's been fixing
 +
TODO: Keep nudging GeoForAll folks
  
 
== Minutes ==
 
== Minutes ==
  
 
=== Attendance ===
 
=== Attendance ===
 +
Regina Obe (robe2)
 +
 +
Alex Mandel (wildintellect)
 +
 +
Chris Giorgi (TemptorSent)
 +
 +
Jeff Mckenna (jmckenna)
  
 +
Sandro Santilli (strk zzz asleep and then woke up at the end)
  
=== Full transcript ===
+
Vicky Vergara (cvvergara too busy fighting queries to concentrate)
  
 +
=== Full transcript ===
 +
[https://wiki.osgeo.org/wiki/Talk:SAC_Meeting_2018-02-15#Transcript Transcript]
  
 
= Details =
 
= Details =

Latest revision as of 14:36, 15 February 2018

Agenda

Where and When

Go over status of SSLs - FOSS4G2018

January 4th 2018 Meeting

<Fill report here>

New Website status report

Report:

  • [Vicky & Jody] Experimented with what can be done on the site.
  • [robe] Status of staging.www.osgeo.org now setup on web18a.osgeo.osuosl.org (plans to migrate production to there as well)
 Eventual shutoff of cloudvps.com hosting

Sys Admin Contract

  • [martin]:

<Fill report here>

Martin to report on what he's been working on.

Migration off old hard-ware AND Virtualization, Containerization, or None

  • Discussion pending on list (moderator: could not find the mail thread)
  • Needs:
    • trac is painfully slow (now seems better with Chris Giorgi's recent changes to apache config)
    • osgeo6 is running an ancient Debian, needs to be updated

Leaning to libvrt since it can be done with one server (less resource intensive than ganeti), with possibility of moving to ganeti next year when we get more hardware.

DONE: Alex sent quote to mailing list, Updated Feb 15 ~$6300-7000. Chris Gorgi has some ideas, may provide enhancements

  • DONE: provide clarification of new hardware purchase options suitability for hosting type. (wildintellect)
    • LXD was suggested as a drop-in alternative to full Virtualization with KVM for most service; this more similar to the feel of a virtual machine than say Docker.
    • We can mix, nest, and layer both containers and virtualization interchangeably.
    • Easy candidates for containers include Downloads and Webextra (FOSS4G) which are static files.
  • Suggestions:
    • Optane card for extra disk caching beyond memory - Question how do we configure this:
      • 25% is reserved as unallocated to reduce write-wearing and maintain speed over the expected life of the server.
      • A portion (~25-50%) would be allocated to the ZFS L2ARC (or equivalent) to keep warm FS blocks instantly accessible.
      • The remainder is available to be used essentially as a large persistent ramdisk.
    • Fill RAM, mostly caches requests, in particular file downloads
      • This will happen automatically as files are accessed and added to the filesystem's cache.
      • Data can be pre-cached simply by accessing the files and directories -- a simple script can run periodically to ensure they are kept marked as hot.
    • Larger DWPD rating for SSDs to better handle writes - Question, we didn't do this for OSGeo6, anything we should look out for? Should we estimate life of OSGeo6 drives and replace pre-failure next couple of years.
      • Total write volume can be checked with smart-tools and drives approaching the stated limit should be replaced or move to storage-only applications.
      • Write endurance and long-term speed can be improved by leaving 20-50% of each drive unallocated.
    • Mirrored pair of SATA SSDs for write-caching and high io loads.
      • Reserve 25-33% unallocated.
      • Provide ZFS ZIL SLOG with 32-64GB to minimize write latency for data stored on HDDs.
      • The remainder can be used for loads with high mixed read/write transactional loads, such as active databases, mail, etc.
    • 4 TB Spinning disks, still plenty of space (7+ usable), takes less time to rebuild on failure, cost diverted to other features.
      • (Please note - an active storage pool should never be filled to more than 2/3 of capacity to avoid serious performance degradation and fragmentation - C.G.)


Any other feedback from others to be discussed

Others

  • Formally ask Chris Giorgi if he'd like to officially join SAC

<Fill report here>

Trac SVN status

  • Regina looked at postgres load and see if anything amiss there and if any query bottlenecks

Concluded apache is our main issue.

  • Chris Giorgi made changes to apache config which helped and suggests adding a caching nameserver with DNSSec
  • Martin's plans to upgrade OS of trac and other VMS

Ticket Triage

GeoForAll DNS

Jeff to report on status of GeoForALL and other DNS issues he's been fixing TODO: Keep nudging GeoForAll folks

Minutes

Attendance

Regina Obe (robe2)

Alex Mandel (wildintellect)

Chris Giorgi (TemptorSent)

Jeff Mckenna (jmckenna)

Sandro Santilli (strk zzz asleep and then woke up at the end)

Vicky Vergara (cvvergara too busy fighting queries to concentrate)

Full transcript

Transcript

Details

Next Meeting

Proposed Time: UTC: Thursday, March 1st, 2018 at 8:00 pm

SAC main page