There can be a number of reasons for this error. Check the following:
1. Do you have the correct serial cable connected to the correct ports (host side and target side)? Is it a 'straight-through' cable, i.e. not null modem (as supplied with some Phytec hardware packages).
2. Is the boot code installed correctly? If other applications have been executed via BDM and produce the expected behavior then this indicates that the installation is complete.
3. Have you reset the hardware immediately after initiating the download from MATLAB? Normally it is necessary to do this so as to execute the boot code which responds to the host request to initiate a download.
4. Check to make sure JUMPER 6 is open for serial download.