Week 04 – Modifying the HTTP Source Code

It turns out that the TCP/IP MDD Demo given by Microchip is not that useful. It mainly redefined all the functions to open webpages stored in SD cards instead. They did not modify the part where the original used to upload a file.But a file named “FileSystem.c” is particularly useful because it shows that which MPFS-related functions should be replaced with which FSIO-related functions. As a conclusion, I did not use the TCP/IP MDD Demo and put my focus back to the original TCP/IP Demo and try to modify the source code from there.

The part of the code that is used for the uploading process is located in the “HTTP2.c” file and in the function named “static HTTP_IO_RESULT HTTPMPFSUpload(void)“.  I changed the MPFS-related functions to FSIO-related functions but I couldn’t upload a file or even create a file on the SD card. So, I shifted my focus in the main function and try to do a simple create and write file process on the SD card.  It turns out it doesn’t work as well. Then, I decided to check the “HardwareProfile.h”, and it turns out that some pins are defined twice for two different purposes. One of the pin is defined for one of the push button on the PICDEM.net 2, thus, I removed that definition. The other pins are the SPI pins which are used originally to interface with the EEPROM on the PICDEM.net 2. So I commented those definitions as well since the SD card interface will also be using the SPI. After removing of this pins definitions, a lot of other functions in the source code needs to be modified. I change the functions which used the push button to another push button instead and I comment out most function that is related to EPPROM. Finally the code was able to compile, but still couldn’t read and write.

It turns out that the PIC only able to run FSInit() correctly, but couldn’t actually open the file even if the file is created beforehand. Thus, I replace the functions from FSfopen to FSfopenpgm, but to use this function I would need to add a definition, which is “#define ALLOW_PGMFUNCTIONS“. By the way, I also added a line to include “FSIO.h” to import the FSIO functions that I am using. By adding the line “#define ALLOW_PGMFUNCTIONS“, there is a little problem in the “FSIO.c” code, the function “int rmdirhelper (BYTE mode, char * ramptr, const rom char * romptr, unsigned char rmsubdirs)” does not compile. In the end, I commented that function, since I will not be using it right now. Finally, the PIC can open the file properly and write data that I defined in the code into it.

I actually copied the code to write onto a SD card from the my previous project and a lot of pins assignment is copied along as well. Because of this problem, the PIC actually does not continue to operate as a HTTP server after those pins assignments. After some testing, I found that the pin that is causing the problem is the line “TRISJ = 1;“, so I commented this line. The code now can create a file on SD card, write data into the file and also continue to function as a HTTP server.

Next thing I did is copied the codes related to the SD card back into the function “static HTTP_IO_RESULT HTTPMPFSUpload(void)“. The exact case in switch structure of the function that I will be modifying with is the “case HTTP_MPFS_OK:“, but I did a little bit of modification on a previous case because that case was suppose to check for the MPFS format. I modified it so it always says that the file uploaded is at the correct format. The case is named “case HTTP_MPFS_UP:“. I soon realize that the data was not written into the file instead it remains empty. It shows that the code successfully created a new file but did not write into in it. In the end, I added a FSfclose after every write to make sure the data are not gone after it is written into the file and reopened it in append mode to write the next packets of data.

Now, finally the code can write something into the SD card. The new problem appeared is that the code is writing extra lines after the actual data, which appears to lines sent by the internet browser to define the boundaries of different data. The other problem is that the first few bytes of the data is not written into the file. So, I looked into the “case HTTP_MPFS_UP:“. It turns out that there is this function which actually reads the first few bytes of data and did some checking to make sure it is a MPFS file. I commented it, all the data is written onto the SD card but with a few extra empty lines before the actual data. Oh god!

I checked the packet sent by the internet browser through Wireshark. It turns out that my code writes a “rnrn” before the actual data into the .txt file. “rnrn” was actually used by the code to determine where the data actually starts but the problem now is that my program writes it into my .txt file. So, I made use of the previous function which reads the first few bytes of the program to read only the first 4 bytes of the data. Finally there is no more empty lines appearing before the actual data, but there is still extra lines with characters appearing after the actual data. Interesting to note that both Mozilla Firefox and Google Chrome used different lines, so it ends up that I am getting a line filled with random numbers at the end when I upload my file using Firefox and a line starting with “WebKitFormBoundary” followed by a bunch of random characters when using Chrome.

It is much more trickier to solve this problem. I wrote a new section of codes to scan every string that is received by the PIC for the character ‘r‘ because ‘r‘ is actually the character after the last character of the actual data. I also changed the definition “BYTE c[16];” to “char c[16];” because I am using the strchr function to pinpoint the location of the ending character and subtracting it with c to get the actual length of data I need to write into the .txt file on the SD card. I also added a line to break from the loop so that the code does not write anymore unnecessary data. Finally, it works! I and I added a FSfopenpgm function at the start of the upload process to ensure that a clean file is created every time a file is upload.

But, come to think of it, using a check for the character ‘r‘ might not be safe if the character is a part of the data. But there is a pattern of  ‘rn‘ which is used to mark the end of the actual data can be used as a check. To implement this function, it would be necessary to check for two characters instead of one. It might be also possible to simplify this function using the function similar to “lenA = TCPFindROMArray(sktHTTP, (ROM BYTE*)”rnrn”, 4, 0, FALSE);” which returns the location of where that particular string is located. But unfortunately I was unable to fully verify the exact process and capability of this function. There is an argument in the function which is used to define which point will the function start its scan, if I did got it correctly, but it seems to me it always scans the ROM array from the start even if I changed the argument. I left it because I am not sure whether is the function that is not working or is it my portion of code that is causing the problem.

I moved onto my next task to implement a new text field on the html page that I was using to upload files. The code that I added to the original html text on the code is “<br> Any Text: <input type=text name=atxt><p>“. This will simply add another text field on the html page so I can send another type of data instead of the file data.

From Wireshark, it is seen that the file data and the text data was separated by another line of boundary line, which is the line generated by the web browser which consist of either random characters or the “WebKitFormBoundary” starting for the Chrome. So I modified the code to try to use the TCPFindROMArray function to scan for “rnrn” but it turns out that it did not work well. I then used the function to scan for “atxt” instead, which is the name I gave for that text field input. It works well but it reads the following unnecessary data. So I create a new char which equals to 5, to get rid of all the data up until the next five bytes of data after the string “atxt” is found. After that, I loop the code back to receive more data by ending with the line “return HTTP_IO_NEED_DATA;” until the last boundary line is found. In the end, the code is able to read the data from the uploaded file and the text field and write it into the same .txt file on the SD card.

In conclusion, the code was capable of reading data from POST and also writing the data into a .txt file on a SD card. But, I am a bit concern about the way I verified the boundary lines of the received data. I believe that if I could get the TCPFindROMArray to scan for “rnrn” for the boundary lines, it would be a lot safer than using the characters or strings that I am currently using for scanning now.