We're struggling with an issue where a couple of our projects are failing consistently, but there is no error logged at all to explain why. After contacting support we were basically told the situation was not possible, so I figured I'd throw something on the forums here to see if anyone else has encountered a similar issue.
We use FB7 (7.0.0.1535) and the FB Server product to automate nightly builds. We have a parent 'build manager' project that serves as an engine for many different builds; each build can have multiple stages, e.g. view creation, compile, binary packaging, etc. The build manager project can take default actions for each stage, or a separate project/library can override a particular stage by defining that action list (i.e. the project would have a 'compile' action list to override the default compile stage). We do this by checking to see if a particular .fbp7 file exists in the source directory and running the action list for the stage from that file (via 'Include Project') if the file exists. A build consists of whichever stages are defined being run until completion of the last stage. This scheme has worked for us since FB5.
Now, we have a couple of tiny builds that are failing consistently with no logged error. They override the compile and packaging stages and do the following for the compile stage:
- set a variable value
- ssh open connection
- ssh execute command - set an environment variable
- ssh execute command - execute a perl script
- ssh close connection
- set a variable value to an expected filename
- read the file
- fail if the contents contain an error condition
The binary packaging stage is similar:
- set a variable value
- ssh open connection
- ssh execute command - set an environment variable
- ssh execute command - execute a perl script
- ssh close connection
- set a variable value to an expected filename1
- set another variable to another expected filename2
- try - fail if filename1 exists
- catch - read filename1, send email to developer
- end
- read file2
- fail if the contents contain an error condition
We've recently switched to the SSH actions, we used to use Telnet. This behavior occurs with Telnet and with SSH. We do not save any output to a variable from the SSH actions, and we have set really high timeouts (3600000ms) on each action. As far as other troubleshooting attempts go, we've tried running this on different machines and we've tried recreating our project files from scratch, i.e. creating a new file, copying in the actions from the old file and setting up all the variables again.
What we're seeing is that when we run this project with the defaults (i.e. all stages), we never get past the compile stage. If we run just the compile stage, it fails. If we run just the binary packaging stage, it fails. In both cases, the log shows an error for the 'Include Project' action, but the child items (i.e. every action taken from the included file) show up as completing successful.
Example:
Note that we've checked the child actions and they did actually execute and complete successfully, i.e. in the above example, the ssh commands were executed and did finish successfully. We ran the above in the FB IDE, but the same behavior is seen when running our project through FB Server.
Has anyone seen anything like this before? Does anyone have suggestions as to what we should try to solve this?