Upgrading from VMFS-3 to VMFS-5 – Online

You can upgrade a datastore from VMFS3 to VMFS5 online and it is non-disruptive. VMs will continue to run on the datastore and the host does not need to be restarted.

I just tested this without issue by using the vmware vSphere Web Client, clicking on my server, related objects, Datastores, and right clicking on the datastore that had type VMFS3 and selected upgrade to VMFS5. It worked just fine.