Monthly Archives: July 2014

Moodle Journey

Posted on by

Late spring term, 2014, Moodle would periodically reach a load level that made it effectively unusable.  During the early part of summer term, the OS load levels would go from a reasonable level of 2 and would spike at 50 or 60.  During these high load events, both CPU utilization and memory utilization would spike, along with disk caching. From a user perspective the system was down.

In the early part of the diagnostics, Munin, a systems monitoring tool, collected detailed systems performance data from both the Moodle server and the MySQL back-end database.  It showed that there was memory paging occuring between RAM and disk.  After identifying the paging issue, the system admin increased memory(from 16G to 64G), CPUs(from 2 to 6) and increased the buffer size of the MySQL server.  This fixed the paging issue, but the same performance symptoms were observed.

Something drastic needed to occur to determine the cause of the underlying performance issue.  Moodle was split into two instances, one running fall, winter and spring, while the second ran summer only.  Performance issues were still observed.  At this point, a focus was placed on large embedded video files and large course backups, both of which caused excessively large load levels.  The large embedded video files were identified as a possible cause, moved out of Moodle and replaced with a link to the videos.  At the same time a memory leak was identified within Flowplayer.  Flowplayer was updated to the version used in Moodle 2.7 and the system admin put a cron job in place that frees unused memory every 3 – 5 minutes.  At this point, CPU, load average, memory and network utilization were greatly reduced.

What is the current status of Moodle?

  • Moodle 2.5, with upgraded Flowplayer has not performed poorly since week 28, 2014
  • Moodle 2.7 in currently being configured for fall term
  • The memory cleanup script will remain in place for Moodle 2.7
  • Load testing will be performed, utilizing LoadStorm.com and results will be posted here
  • The location of large video files will be dependent upon the results of load testing
  • Test performance utilizing SSD system drives

August 8, 2014 Update

  • Moodle 2.7, the lastest version, is available for Fall term course development. It is located at http://ginger.wou.edu — its final name will be http://moodle.wou.edu
  • All Moodle data on 2.7 have been moved to SSD drives (solid state disk)
  • A performance test was run, utilizing spinning disk followed by SSD drives.  The graph can be seen here:  (the large double spikes represent a 580 user load test utilizing spinning drives. The small bump in the afternoon is a 5,125 user load test utilizing solid state drives.)  What this means to you is, the system can handle 10 times the users at 1/20 the processing time.
  • A second Moodle web server was added, doubling the capacity.  Testing will commence shortly.
  • online3.wou.edu and online2.wou.edu are meant for archival purposes only.  New courses must be developed on moodle.wou.edu  For help exporting from old versions of Moodle into the 2.7 version of Moodle, call Elayne Kuletz.

August 28, 2014

Today, there are 4 web servers residing behind the load balancer.  The load balancer hands out initial web-requests utilizing a round-robin algorithm.  This means that if there are a total of 100 active users utilizing Moodle, each web-server will be handling 25 users.  To access the load-balanced Moodle, go to http://moodle.wou.edu  Each web-server can handle about 410 users and still maintain a good user experience.  This will give us a capacity of 1,240 concurrent users with a good user experience.  This past academic year, 250 concurrent users were the maximum number of users that were observed.  All load testing was performed while playing large videos.  As our load grows, additional web servers can be easily cloned from the existing web servers and placed behind the load balancer.  By the end of next week, we will have a consultant available to provide systems administration support.

The load-balanced schematic can be found here.

Final hardware configuration:

    • NetApp EF550 solid state storage array
    • Cisco UCS server platform
    • Cisco ACE 4710 load balancers
    • 10Gb network infrastructure at the core
    • Four load balanced web servers running on RedHat
    • One database server running on RedHat
    • Moodle version 2.7.1

Best / Worst Analysis

  • What is the best outcome that can occur, continuing with Moodle?
  • What is the worst outcome that can occur, continuing with Moodle?
  • What is the best outcome than can occur, by not continuing with Moodle?
  • What is the worst outcome that can occur, by not continuing with Moodle?

WOU Computing Infrastructure

Posted on by
Reply

Where to start…?  Maybe a summary from the ten thousand foot level, then expand out from there, yet keep a rein on going too far into details.  🙂

 

Summary:

WOU supports a variety of computer labs, smart classrooms, faculty and staff workstations and residential computing.  A variety of software packages are supported, both in computer labs, virtually and on faculty/staff workstations.  High speed internet is provided to campus as well as the residence halls through both wired and wireless connections.  NERO is our internet provider.  There is both a primary and redundant data center.  The network infrastructure includes routing, switching, firewalls, intrusion detection, network access control, wireless access points, etc.  A variety of enterprise applications are supported, including purchased, open source and developed.

Detail:

There are 25+ computer labs in a variety of locations on campus.  This includes Hamersly Library, APSC, Education, DeVolder Family Science Center, Natural Science, Creative Arts and Werner University Center, along with many others.  There are approximately 400 lab stations available for students.  This includes a mixture of PC’s, Mac’s and VDI (virtual) stations.  VDI can also be accessed remotely by a resident hall student from their own computer, providing access to a variety of software titles.  There are 40 Xerox 8870 color cube computers spread across all of these labs.

The lab workstations contain a variety of software including the Adobe Creative Suite, MS Office, Google Apps, SPSS, ARC GIS, Autodesk suite, Matlab, SAS, MSDN, Sophos and many others.  Not all applications are found at all locations.  All the applications contained on VDI, can be accessed remotely from anywhere, utilizing your WOU Pawprint account credentials.

There are 110 smart classrooms on campus.  Every room has one or two installed video projectors.  The instructor station has a computer, document camera, touch panel control system, AV interface, Blu-Ray DVD play and a sound system.  In addition some rooms contain an annotating or touch screen monitor, lecture capture system and IP-video conferencing.  Some of the larger rooms are dual-platform, PC, Mac or VDI.  Several lab-classrooms include the ability to display any of the lab stations on the projection screen.  About 20 of the classrooms have Apple-TV capability, giving you the ability to display the contents of an iOS device on the projection screen.

There are approximately 800 faculty and staff workstations on campus.  The current workstation life cycle is four years. There is a combination of PC’s, Mac’s and VDI stations, depending on need.  Access to Google Apps and MS-Office is provided on each station.  Other software is provided on an as-need basis.  In addition there are approximately 200 general-use stations.

The network backbone is a switched 10 Gb/sec network.  Network from the core to a building is typically two 10Gb/sec network ports that are aggregated, effectively providing 20 Gb/sec of bandwidth, plus redundancy.  In most cases, there is 1 Gb/s to the desktop.  The connection from WOU to the internet is a 500 Mb/sec connection.  NERO provides WOU internet services.  There is a redundant core project under way.  Both the campus and residence halls has wireless connectivity provided through 600 Cisco wireless access points, centrally managed by redundant wireless lan controllers.

The data center includes UPS power, backed up by a diesel generator.  The core server infrastructure resides on Cisco UCS Blade servers.  NetApp is our primary storage vendor, while SuperMicro provides secondary storage.  The data center includes both traditional spinning hard drives and solid-state drives.  The majority of the servers are virtualized.  There are 100 plus servers installed in the virtualized environment, including Windows Server 2008, 2012, RedHat, Oracle Linux and OpenIndiana.   More information on the redundant data center can be found here.

Examples of enterprise software run in the data center include, DNS, DHCP, Moodle course management system, Astra course and event scheduling, HVAC control system, Dimensional Data Warehouse, Portal, Oracle Enterprise Database, Oracle ODI, Cognos Connection and Framework manager, wouTV and many more.

Digital Media Productions resources include a studio with LED lighting, a TriCaster, 4 Sony HD video cameras, a Mac Pro editing station running Final Cut Pro X.  DMP broadcast its’ projects on the local community cable channel and wouTV as well as campus digital signage.

 

E-mail Security

Posted on by

E-mail is not secure.  E-mail can be intercepted on the path from sender to receiver.  If the e-mail is sent to/from an open list-server, Google can scan it and post the results in Google searches.  These are just a couple of the many ways your e-mail can be viewed.  If you don’t want the contents of your conversations showing up in the newspaper, don’t put it in an e-mail.

If you need to send HIPPA, FERPA or other confidential information using e-mail, then encrypt the e-mail.  In the subject line use the keyword encryption or #secure#.  This only works when sending e-mail to an e-mail address outside of @wou.edu.  Only the body of the e-mail will be encrypted.  The subject line will be in clear text, available to the public.

More on encryption can be found here.

 

Desktop Support

Posted on by

A new web site has been created, dedicated to providing information regarding Desktop Support Services.  Lori, Megan and John will each provide frequent updates to the Desktop Support site.

I would like to introduce you to John Rushing.  He is the latest addition to our desktop support team.  Most recently he was a Analyst Programmer / Support Technician for Dining Services and Resident Halls.  Previously John provided staff and student support services for Dining and the Resident Halls, including the Landers student lab.  Previous to coming to WOU in 2005, John was a Programming Engineer at McAfee.com, providing virus programming support.  John started his first position at WOU in 1997.  He provided hardware support for the computer labs, managing student workers and ordering of supplies and inventory.  John was in this position through 1999.  John brings a vast level of knowledge to this position.  John will be combining both his Desktop Support experience and programming experience to automate many of the current manual processes.

Megan provides both printer and desktop support.  As noted in a previous entry, she is a certified Xerox printer technician.  Megan will continue to provide services in all areas of desktop and network support services.  Megan is managing the Service Request and Computer Lab student employees.

Lori continues to provide Desktop Services, Network Access Control(NAC) services and Windows Server administration.  Lori also manages the Desktop Support student employees.  Lori is responsible for re-imaging all the labs over the summer, preparing them for fall term classes.  Lori is Dell hardware certified.

Each technician will have primary responsibility for specific buildings.  After analyzing Desktop support data, each technician will be assigned a specific area of campus.  The assignment process will be completed in August.

Each technician will be responsible for the following tasks and more…

  • imaging new computers for faculty, staff and student computer labs
  • re-imaging student computer labs
  • configuring and monitoring Windows update server
  • configuring and monitoring Sophos anti-virus server
  • researching and deploying OSX and iOS deployment server
  • updating Windows applications
  • research and deploy new technology
  • maintaining Network Access Control (NAC appliance)
  • support MathTest, CompTest, IME, SOAR, MelBrown, etc.
  • provide useful information on the Desktop Support blog site
  • provide web-site with recent top 10 service requests, with solutions
  • face-to-face communication with faculty/staff on a daily basis
  • follow-up with faculty/staff after tasks have been resolved
  • provide training for lab attendants
  • backup Nathan when he is not available for classroom support tasks
  • support all aspects of computer replacement deployments, including imaging, bios changes, software installs, old data migration, installation, etc.
  • support printer deployments, maintenance and repair
  • maintain inventory
  • support PDR process, including DoD wipe, storage and disposal
  • interview and hire student employees, including Desktop Support, Service Request Desk and Student Computer Labs
  • maintain Active Directory (AD), including changes, moves and adds
  • maintain folder permissions in LDAP
  • research, recommend and implement streamlining and optimization of services processes

A summary of what is new:

  1. Move John into Desktop Support services, increasing support by one FTE. (equipment systems specialist)
  2. Distributed primary responsibility of new computer deployments to all three technicians
  3. All three technicians will be able to perform the full complement of duties required of an Equipment Systems Specialist
  4. Each technician will provide primary support for a specific area of campus
  5. Provide users with additional information through Desktop Support web site