This week ended up being highly productive as it culminated into the completed functionality of the file upload system with Base64 decoder and CRC support. However, it was not always plain sailing during the week and much time, effort and attention to detail was needed to debug the existing code. By the end of the week, all aspects of the file upload were debugged and performance tests were run, the results of which can be seen below.
Debugging the existing code
A couple of weeks ago, the file upload mechanism was redesigned to incorporate Base64 encoded files to be decoded and the corresponding binary data to be checked using the CRC-16 available on the SD cards (https://blog.aeste.my/archives/2909). By the end of the design process for the new system, the current SD stopped functioning and the bugs in the code could not be rectified as the code itself could not be tested on a SD card. The availability of a Kingston 8gb SDHC card proved fruitful as it was able to initialize successfully and was function with the PICTail. Thus, the work in the initialization stack was put on hold to first debug the current code and get the file upload up and running.
Upon the first few attempts, it was found that the SD card was able to write the data on some occasions and on other occasions, only partial data was written. In some cases, especially for smaller files, the data could not be written at all. This was particularly difficult to debug as there were no obvious reason as to why this would occur. After spending much time and effort to try and figure out what was going wrong, it was found that the Hardware debugger was not the best tool to use for debugging this particular stack of code, because the debugger can halt the PIC from processing the data, but the ethernet side tends to get timed out while waiting for the program to resume. So, a new approach was taken to understand why the problem occurred. This was to breakdown the code and look at each part’s functionality before proceeding.
CRC functionality
It was highly likely that the CRC checksum on the SD card was not working as theorized and that there were some issues with that part which caused and unexpected CRC error which caused the problem int writing the data. The previous code for File Upload (without the Base64 decoder) was used to check whether the CRC works as expected by feeding it a binary file of 512 bytes and sending a known CRC. This was useful as it was confirmed that the SD card responded with a CRC error when the given CRC was incorrect and accepted the data when the given CRC was correct.
Base64 decoder functionality
During the initial debugging of the Base64 decoder (and corresponding write operations) didn’t seem to reveal any significant bugs. However a closer look at the data written itself revealed that at some random locations, the same bytes were being repeated. This was hard to see thus far, as random characters were used to create the test files. This bug was confirmed using a file with data that was predetermined. The cause for this bug was found to be a misplaced “}” in the code which caused the write function to repeat the same 3 bytes just written as it went to fetch more data from the TCP socket. This was rectified but when the program was run, a CRC error was returned every time. So, the CRC was turned off and the data written on the SD card was checked block by block to see where the CRC error occurs.
After some painstaking work, the anomaly was found to be an extra three bytes of data of hex values “fc”. This was true for all tested files but the cause for these extra bytes was not very obvious. Fortunately, the documentation on the SD card commands and responses came to the rescue and it was found that the hex value for “Write multi block” data token is indeed “fc”. Armed with this knowledge, a quick review of the code revealed that unnecessary Data Start Tokens were being sent to the SD card at wrong times which caused both a CRC error and incomplete writes. The reason for sending these extra data tokens was that, after a block is written completely by the SD card, it goes into busy state to give the SD card some time to prepare the next block to be written. Thus when the SD card responds that it is ready to write more data (that is still waiting at the TCP socket), a Data Start token is sent and the remaining bytes in the socket are written first before the program loops back to fetch more data. The problem occurs when there is insufficient bytes to be written to the card (i.e.<4 bytes). In this instance, the Base64 decoder does not have 3 bytes prepared to be written to the SD card but a Data Start Token has already been issued. After the program loops back to fetch more data, another Data Start Token is being issued by the SD card, as the block is still not written into, causing the error.
This was rectified by using a status check, which notifies the code whether a Data Start token has been sent or not. If sent, the status indicated that a new data start token is not needed to be sent although the block has not been written into. This solved the problem, but upon activation of the CRC, the program was able to run, but for some files, failed to write any data. Further digging revealed a big error, which was that a proper Data Sent Token was not being sent to the card which caused the SD card to fail to write the data. Using proper conditioning in the code, this was rectified and the proper amount of Data Start Token was sent to the SD card.
One more condition was needed to ensure that the proper Data Stop Token was sent to the SD card. After making all these changes, the code was tested and found to be working. Correspondingly, the CRC was turned back on and that was found to be working as well. The progress was reported to the supervisor who advised to display a “HTTP 500 series” error if a CRC error does occur which was easily incorporated into the code. The following image is an example case of the HTTP error displayed when a CRC error occurs during the file upload.
CRC error page
File Upload Performance
Following the success of the functionality, performance of the code was tested and the results obtained are discussed in this section.
A sample of 30 test files were created within a range of 1kB to 1MB to test the performance of the file upload. Out of the 30 sample files, 20 were within the range of 1kB to 350kB (our expected file size for the bitstream). The file sizes implemented were non-linear.
Two separate time measurement systems were incorporate to measure the time taken for the file upload, the first was a simple stopwatch and the other was the in browser Time Measurement tools available. The two times recorded proved very similar so an average of the two times were taken to calculate the average file upload speed. The following table and graph displays the results:
Graph of File upload performance
0 Comments