A Riddle, Wrapped in a Mystery; Inside an Enigma

Do you ever see inconsistent results when running unit tests in a sandbox versus running them in production?  Or, inconsistent results when running unit tests between sandboxes?  Well, the black-box that is the Salesforce unit test, just became blacker and boxier to me.

I was trying to deploy a patch to production and was going through our normal build path that passes through our full sandbox.  When attempting to deploy to the full sandbox, our automated build process failed on a test method that we had not changed in any way.

The error: ... System.TypeException: Invalid date/time: 05/05/2010 00:00 AM stack...

A colleague and I inspected the method and the class and determined that they were identical in our sandbox and production.  When we ran the individual test in our full sandbox, it failed, and when we ran it in our production environment, it passed.

What the?!

The error pointed to this fragment of code in the test: datetime.parse('05/05/2010' + ' 00:00 AM');

Not sure why it was 00:00 AM so I updated it to 12:00 AM and was able to proceed with the deployment.

I was unsettled by experience because we had been able to deploy to this sandbox two weeks earlier and had not had this test method fail.  So, we opened a case with Salesforce and asked their "premier" support to explain it.  Our first support rep reproduced the results but gave us a canned response that it wasn't a good practice to use 00:00 AM.  Fine, we said, but tell us why it passes production tests but not tests in our full sandbox and what exactly changed in our sandbox?  We went on and on like this for a couple days, escalated the case, and are now doing the same dance with another "premier" support rep.

For those of you who read this, please try this at home and let me know if you can replicate this in your sandboxes:

Create a test object called TestDate__c.  Add one custom field called SomeDate (type = date/time).

Create an apex class (api version 27, for consistency sake) with the following code:

global with sharing class TestingDate
{
    public static testMethod void testMyObj()
    {
        TESTDate__c myobj = new TESTDate__c();
        myObj.SomeDate__c = datetime.parse('05/05/2010' + ' 00:00 AM');
        insert myobj;
    }
}


Run your test.  The results we got were as follows:

NA14 (developer org): Pass
CS15: Fail
CS12: Pass

If nothing changed on CS15, how can this be?  Still waiting for an answer...

Salesforce Sandbox Instances

A while back, I was on a client site that was affected by a sandbox outage that lasted over 3 days. Unfortunately, both of our full copy sandboxes were on this instance and a good number of our developer/configuration sandboxes were as well.  Needless to say, we were greatly affected by the outage and when we asked about migrating some of our sandboxes to another instance to reduce the risk for our team, we were told that it was not possible.  Today, again, there is another major outage, this time on CS15, and again, most of our development, testing, and full copy sandboxes are affected. It seems crazy to me that customers who seek the cloud for protection against this type of outage do not have the ability to specify that they want their sandboxes distributed across multiple instances.  If we have a break/fix situation, we are in a difficult situation given that our development path is dependent on CS15 being operational.  Customers should be allowed to spread their developer sandboxes across instances to reduce the impact of another outage.

Vote for the idea here.