Week 05 – Extremely Unfavorable Speeds & Minicom

Uploading Large Files

Since small files can be uploaded, now it is time to test with uploading large file (350 KB). Firstly, I try to generate a random text file by using the dd command. It turns out that the file generated by it has a lot of weird characters, I can see Chinese characters, Japanese characters, Korean characters and a lot other unknown symbols. To be safe, I created another file by typing random characters from the keyboard and duplicate it through copy paste to get a file sized at 350 KB.

First, I try with the file generated by the dd command. It turns out that the uploading process is never ending and I forcefully stopped the process and check the SD card content.  The result is only the first line of the text is uploaded nicely. Then, I try with the file I typed, the uploading process is never ending as well, I again stopped the process and check the SD card content. It turns out that the a good chuck of data is written into the file correctly but it is incomplete since I stopped the upload process. For both of the uploading process, it is noticeable that the blinking of the LED, which should be blinking every second, is slowing down more and more as the process goes on.

Next, I removed all the unnecessary operations within the while loop which copies and writes the data. I also increase the number of byte written in per loop from 16 bytes to 111 bytes. An error stating “HTTP2.c:1721:Error [1300] stack frame too large” will show up during compilation if I increase it to 112 bytes. But I notice that the bytes allowed is higher after I comment out codes in the loop. I did not test with whether removing unnecessary codes outside the loop would help to increase the number of bytes.  I also increase the RAM buffer for the TCP sockets from 200 bytes for TX, 200 bytes for RX to 800 bytes for both while commenting out all the other TCP sockets for other demo applications to save memory for the HTTP server TCP sockets. After all these changes, the 350 KB file can be uploaded in 7 minutes (which is very very very slow).

After removing the part related with the SD card and testing with only uploading the file through the TCP/IP stack, the upload is done within a minute. The problem is narrowed down to the file system functions.

Minicom (Download bitstream)

Things to note:

1. If the Minicom is getting something, means the USART is working. If it is garbage characters, then it is the baud rate problem.
2. The code should only send character once, if it keeps on sending something is wrong with the code. My supervisor, Dr. Shawn, inserted a line “while(1);” at the end of the code to stop it from sending characters over and over again.
3. The function puts1USART() is a repeating process of putc1USART() and it will only stop the loop only if the array is ended with a null character.
4. To solve the problem stated in 3, my supervisor, again, create a new for loop in the code similar to the puts1USART() function but the putc1USART() will stop when the counter reaches 100. So, it puts 100 characters. This for loop is later modified to put the exact amount of byte read by the FSfread() function to prevent unnecessary characters being send through the UART.
5. The baud rate is also changed, because garbage characters appear in between the correct data.
6. There is a line feed and carriage return issue where the text we used initially only has a carriage return and no line feed. It results in that the data displayed in the Minicom does not moves the first column of the new line and just start from the same column that the previous line ended but on the next line. This issue is solved by using another software to convert the text file to have both carriage return and line feed.

After these issues are solved, doing some minor modification to the code. Downloading bitstream from SD card and output it through the UART is working fine.

Speed Issues

I wrote a code to read from the SD card and then write into another file in the SD card. Testing with a 350 KB file, the time required is 2 minutes, which is still very slow.

I tested with decreasing the bytes read/write per loop to 64 bytes, there is no significant improvement. Unless the loop is decreased to read/write 1 byte per loop, then there is a significant slowdown in speed.

I also tried to remove those codes in the FSfread() and FSwrite() functions which test for invalid read and write conditions, but there is still no significant improvement in speed.

Timer1

My supervisor told me that if I am to work with optimization then there should be a method of measuring how much improvement in speed I am getting with the changes I made. A stopwatch is a fair enough method if the time is long but the target time for the PIC18F97J60 to read a 350 KB from the SD card is 0.5 second. For this case, a stopwatch is not suitable. He suggested that I use the Timer1 module on the PIC to create a timer that which will time the speed of the operations.

For the Timer1, I decided to use the on-chip crystal oscillator which is rated for 32 kHz. Then, the prescaler is 1 : 8. By calculation, it will be 0.25 ms per tick, and by dividing the tick with 40, I should get a value with 10 ms per unit. The value will be displayed through the LED.

I tried the Timer1 with FSfread(), reading from a 35 KB file took 2.2 s. Testing with larger files, I used an online stopwatch instead because a 175 KB file took 7.389 s and a 350 KB file took 14.509 s.