Ack! http://webmail.artifex.org was working, but the encrypted version, https://webmail.artifex.org, was not. The vagaries of SSL termination.
So, webmail is fixed and properly tested now. Please let me know if you have any problems. I have reset several passwords today already and will continue as messages come in.
POP (via SSL), IMAP (via SSL), and webmail (at https://webmail.artifex.org) are all up, and in the process the SSL certificate for both mail.artifex.org and webmail.artifex.org has changed. You can download the new Artifex root certificate [1] from:
http://artifex.org/certs/cacert_2009-08-30.crt
and import it into your browser/mail client/Keychain as a certificate authority, or you can just add a new exception for the certificate you get when you connect.
If you haven't written yet with a number to call for a new password, please do. (You can also call my (215) or (409) number directly if you've got it; please note that I am on the East Coast).
| [1] | Interestingly, the former Artifex root certificate wasn't compromised. So although the individual certificates for webmail and mail.artifex.org had to be regenerated, regenerating the root cert was actually done just out of expedience. (The machine with the former root certificate authority is in Illinois and has been inaccessible for the past several months.) |
Internet mail (SMTP) and most websites continue to stay up, and I'm finally turning my attention to POP, IMAP, SSL, and webmail later tonight or tomorrow morning. I've spent the past couple evenings sorting out the right way to do file integrity monitoring -- which regrettably was something that really needed to go in before we turn on the remaining services like POP and IMAP. Nothing out there is quite right for supervising virtual instances, but I finally settled on one of the more modern, simple systems and have it running.
Thank you for your patience. I've got a number of password requests in from folks and will work my way through them on Sunday.
You may now SSH to the server, provided your password/SSH key is set up.
Incoming mail (SMTP) came up last night (8/25), so we are now receiving mail for all addresses hosted with the Artifex. I'll be setting up POP/IMAP/SSH access for the userbase tonight -- and after that's done, folks will be able to read/send mail and update websites. I may get to webmail as well.
All passwords have been reset, so please contact me (hblanks@artifex.org) or Mike Hugo to update the password on your account. Please remember that we do not set up accounts by e-mail, so you will need to speak with one of us to update your password.
If you include your phone number in your e-mail, I should be able to call you and update your password within two days.
The first round of web services are back up on a newly installed & virtualized Terra (which is to say, static files, out of the box PHP -- but not http://artifex.org or anything using Python/Ruby). The Python end won't be hard -- it's just late here and time to turn in. I'll mop up the remainders this coming Tuesday evening and turn to mail then as well.
Shell and file access is still only set up for administrators but we will can begin resetting passwords Thursday, I think. A list of administrators you can contact will be posted here when that begins.
Dear All,
The primary Artifex server was brought down this Thursday morning due to a hack that occurred (by our estimate) around 1PM on 8/18 (PDT), and that was detected later that day. All user data was backed up, and no permanent damage was done.
Mike and I are in the process of rebuilding the existing server and upgrading its security measures. I ordered new server hardware today as well; in addition to providing a second machine with more storage/processor, it will also allow us to run at least four virtual servers in isolated security contexts.
Our efforts in the recovery at this point are:
- Bring up all websites on a temporary box.
- Complete a forensic analysis of the break-in, if possible. (Our success in determining the hole so far has been limited; although a recent fix to the Linux kernel is all but certainly part of the problem.)
- Reinstall the primary server with further security precautions.
- Begin resetting user passwords.
I'm confident that Mike and I can make it through most of this work before the weekend ends. Nevertheless, this does make for the longest downtime we'll have had in several years -- so if any sysadmin has time to lend, there is no doubt that we could use it. At this moment, we are painfully short for extra hands on the admin team.
With only a couple days' downtime, it's highly unlikely that we'll lose any incoming e-mail. Nevertheless, I will look into setting up a secondary mailserver to queue up mail on the temporary web services box.
Please don't hesitate to write with any questions, or to make use of the Artifex group on Google at:
http://groups.google.com/group/artifex-org
The downtime, new server, and great assortment of internet services available to non-profits these days are all subjects worthy of discussion, comment, and action.
Thank you for your patience,
Hunter Blanks