I must admit that I misunderstood the task when I first saw it. \u00a0The vendor had changed how they provide the data. \u00a0We will no longer be able to retrieve the data using ftp – the data should be picked up with sftp.<\/p>\n
I immediately thought that this would be quite the trick as as far as I was aware you cannot script sftp the same way you can ftp. \u00a0I was discussing this with my boss to get yet another dose of information. \u00a0My boss said that this seemed an odd choice of technology, that what would be a better choice would be to use ftps.<\/p>\n
It was about this time that I realized that I made the first mistake of thinking about the programs not the protocols. \u00a0Indeed there are three different protocols, FTP, SFTP and FTPS and there are two programs with the same name as the protocol. \u00a0The second mistake was that I was not familiar with the FTPS protocol. \u00a0First a bit about the protocols.<\/p>\n
The FTP protocol is to connect to the server with two connections. \u00a0The first is the command connection which controls the transfers, and the second connection is the data connection. \u00a0The process is controlled over one while the data is transferred over the second\u00a0connection.<\/p>\n
Although the setup of the ftp server can be anything the typical setup uses ports 20 and 21.<\/p>\n
The ftp client\u00a0connects from the client computer on port “X” to the server using TCP on port 21. \u00a0Once the connection has been made a second connection will be used to for transferring the data. \u00a0The way this second connection is made depends on the type of mode that is used – active or passive.<\/p>\n
The second connection is actually made by the server from port 20 to port X + 1 on the client.<\/p>\n
It is really simple, but this won’t usually work in practice at most serious companies\u00a0as usually there are firewalls protecting the server from the internet. \u00a0The firewalls only job is to prevent unknown computers to connect to the machine and thus would prevent the active ftp from\u00a0working.<\/p>\n
This problem was foreseen and thus the passive mode was also created.<\/p>\n
Passive mode is exactly the same as the active mode except that the client opens the second connection\u00a0instead of the server. \u00a0It is because the\u00a0client opens the second connection that usually\u00a0eliminates the problem of the firewall interfering with the creation of the connection.<\/p>\n
The coordination of how to keep the connections connected is pretty simple. \u00a0When the ftp client initiates the passive mode with the command PASV, it receives the number of the port to connect to for the data connection.<\/p>\n
Advantages<\/strong><\/p>\n Disadvantages<\/strong><\/p>\n The big disadvantage to the ftp protocol is that the user and password is communicated as clear text. \u00a0It is possible for anyone sniffing packets to get this information.<\/p>\n <\/p>\n The ftp program allows the user to transfer files and change directories on the remote computer. \u00a0It is really useful. \u00a0When the only real weakness of the file transfer protocol is that the users credentials are passed over in clear text, it seems small enough to correct.<\/p>\n Indeed that is exactly what was attempted with FTPS extensions to the file transfer protocol. \u00a0The change was to simply add encryption to plug this particular weakness. \u00a0So the change that was done was adding Transport Layer Security<\/a> and Secure Sockets Layer<\/a> encryption protocols.<\/p>\n Advantages<\/strong><\/p>\n Disadvantages<\/strong><\/p>\n <\/p>\n The SFTP protocol is also sometimes referred to as SSH File Transfer Protocol. \u00a0The SFTP protocol is a network protocol that provides file access, file transfer and file management over a network connection.<\/p>\n All data that is transferred between the client and server, including login credentials, are encrypted. This is usually done through the user of public and private keys but can be done in addition to a user and password.<\/p>\n The file transfer protocol uses only a single connection over port 22 on the server. \u00a0Both the commands and data transfer take place over this single connection.<\/p>\n Advantages<\/strong><\/p>\n Disadvantages<\/strong><\/p>\n <\/p>\n The file transfer protocol actually is not the most secure protocol due to the fact that the user credentials are sent over as clear text. \u00a0This is actually important if you actually think that someone may be sniffing your packets (not so likely) but depending on the service this might not be so important.<\/p>\n If the data that is transferred to the FTP server is encrypted then it may not be as important if the username and password is captured. \u00a0If on the destination server the data is processed and then removed this may not be a problem or if the data is really well encrypted this might not be a problem. \u00a0Perhaps the information is public information and if the data escapes it is not important (ie. which days are public holidays for a specific trading calendar, what is the trading price of a stock or what is a company’s PE ratio)<\/p>\n A simple scripted solution using ftp<\/p>\n This is a small script, which despite not being very secure, makes a small connection and puts a data file. \u00a0It is only possible because the input to the ftp command can be piped in. \u00a0This is done in this clever little script be redirecting the data from the script itself.<\/p>\n It is not possible to do this same trick using the sftp program, but it is possible to create a script using secure copy<\/a>.<\/p>\n This script is even smaller than the ftp script and is more clear to the casual reader.<\/p>\n The scp command\u00a0actually still requires a password and is not “scriptable” in the same way that the ftp client was. \u00a0However, it is possible to setup the user setup so the public\/private key is used for authentication and password isn’t necessary.<\/p>\n <\/p>\n For the really paranoid who must continue to use ftp, you might not want to connect directly to a server over the internet but instead connect via a proxy server. \u00a0It is possible to do that using ftp. \u00a0Simply pass all the information necessary for the proxy server.<\/p>\n Although it is possible to add this level of complexity to buffer your server from the ills of the world, it really wouldn’t be more secure than to use secure copy (scp) for transferring the data files. \u00a0Secure copy would have encrypted credentials and the key used as a “password” would most likely be considerably longer and more secure than any normal password.<\/p>\n Even if you picked a small 256bit key, it would on average still be better than some 16 or 20 character \u00a0login. \u00a0Yet, a more reasonable choice of a 1024 or 2048 bit key would be massively more secure than any password selected.<\/p>\n Nice explanation of active \/ passive ftp<\/b> I must admit that I misunderstood the task when I first saw it. \u00a0The vendor had changed how they provide the data. \u00a0We will no longer be able to retrieve the data using ftp – the data should be picked … Continue reading \n
\n
FTPS (or FTPES or FTP-SSL)<\/h2>\n
\n
\n
Secure File Transfer Protocol (SFTP)<\/h2>\n
\n
\n
Putting it all to good use<\/h2>\n
#!\/bin\/bash\r\nUSER=richard\r\nPASS=secretpasscode\r\nDATA=\/var\/tmp\/datafile.tar\r\n\r\nftp -i -n ftp.somedomain.co.uk <<MARK\r\nuser $USER $PASS\r\npwd\r\ncd data\r\nbin\r\nhash\r\nput $DATA\r\nls -ltr\r\nMARK\r\n\r\necho file transferred<\/code><\/pre>\n<\/div>\n#!\/bin\/bash\r\nUSER=richard\r\nMACHINE=192.168.178.57\r\nDATA=\/var\/tmp\/datafile.tar\r\n\r\nscp $DATA $USER@$MACHINE:\/tmp \r\n<\/code><\/pre>\n<\/div>\nftp Scripting – Extra credit edition<\/h2>\n
#!\/bin\/bash\r\n\r\n#our destination machine\r\nUSER=dick\r\nPASS=secretpasscode\r\nDEST=ftp.somedomain.co.uk\r\n\r\n#our proxy server\r\nPROXYUSER=bob\r\nPROXYPASS=secret\r\nPROXYMACH=myproxy.mydomain.com\r\nDATA=\/var\/tmp\/datafile.tar\r\n\r\nftp -i -n $PROXYMACH <<MARK\r\nuser $USER@PROXYUSER@DEST $PASS@$PROXYPASS\r\ncd data\r\nbin\r\nhash\r\nput $DATA\r\nls -ltr $DATA\r\nMARK\r\n\r\necho file transferred<\/code><\/pre>\n<\/div>\n
\nhttp:\/\/www.slacksite.com\/other\/ftp.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"