A Riddle, Wrapped in a Mystery; Inside an Enigma

Do you ever see inconsistent results when running unit tests in a sandbox versus running them in production?  Or, inconsistent results when running unit tests between sandboxes?  Well, the black-box that is the Salesforce unit test, just became blacker and boxier to me.

I was trying to deploy a patch to production and was going through our normal build path that passes through our full sandbox.  When attempting to deploy to the full sandbox, our automated build process failed on a test method that we had not changed in any way.

The error: ... System.TypeException: Invalid date/time: 05/05/2010 00:00 AM stack...

A colleague and I inspected the method and the class and determined that they were identical in our sandbox and production.  When we ran the individual test in our full sandbox, it failed, and when we ran it in our production environment, it passed.

What the?!

The error pointed to this fragment of code in the test: datetime.parse('05/05/2010' + ' 00:00 AM');

Not sure why it was 00:00 AM so I updated it to 12:00 AM and was able to proceed with the deployment.

I was unsettled by experience because we had been able to deploy to this sandbox two weeks earlier and had not had this test method fail.  So, we opened a case with Salesforce and asked their "premier" support to explain it.  Our first support rep reproduced the results but gave us a canned response that it wasn't a good practice to use 00:00 AM.  Fine, we said, but tell us why it passes production tests but not tests in our full sandbox and what exactly changed in our sandbox?  We went on and on like this for a couple days, escalated the case, and are now doing the same dance with another "premier" support rep.

For those of you who read this, please try this at home and let me know if you can replicate this in your sandboxes:

Create a test object called TestDate__c.  Add one custom field called SomeDate (type = date/time).

Create an apex class (api version 27, for consistency sake) with the following code:

global with sharing class TestingDate
{
    public static testMethod void testMyObj()
    {
        TESTDate__c myobj = new TESTDate__c();
        myObj.SomeDate__c = datetime.parse('05/05/2010' + ' 00:00 AM');
        insert myobj;
    }
}


Run your test.  The results we got were as follows:

NA14 (developer org): Pass
CS15: Fail
CS12: Pass

If nothing changed on CS15, how can this be?  Still waiting for an answer...