Understanding FTP using raw FTP commands and telnet

Both FTP and SMTP are simple text based protocols. A previous article showed how to check if an email address exists using SMTP commands from the terminal. Here I would like to show you how you can use raw FTP commands to connect to an FTP server, login, traverse directories and even download files. But before we do this we need to understand how FTP is different from the other protocols. 

Firstly FTP (File Transfer Protocol) uses two channels, the data channel and the control channel.  This is called out-of-band control. The control channel sends commands to the FTP server and the data channel is used for data (to retrieve files from the server, etc).

Secondly there are two major modes of FTP operation, the active mode and the passive modes. The difference lies in the way the data channels are opened. In Active FTP, the FTP server will connect to the client port and send data to it. In Passive FTP, the FTP server will tell the client which port to connect to for retrieving data. Firewalls can complicate the process on both sides. 

In our example, we will use Passive FTP (avoiding firewall issues on client) to download a file using anonymous FTP login to the IETF servers. There are a lot of files on this server by some estimates it is about 4GB. We will pick up a small file called ftpext-charter.txt located in the /ietf/ftpext/ folder on the server.

Open the terminal/command prompt (On windows, Go to Start > Run > type cmd). Once you are on the command prompt, type this command to connect to the FTP server and issue commands

C:\> telnet ftp.ietf.org 21

220 ProFTPD 1.3.1 Server (ProFTPD) [64.170.98.33]
USER anonymous
331 Anonymous login ok, send complete email address as your password
PASS blogger@webdigi.co.uk
230 Anonymous access granted, restrictions apply
CWD ietf/ftpext/
250 CWD command successful
PASV
227 Entering Passive Mode (64,170,98,33,151,31).
RETR ftpext-charter.txt
150 Opening ASCII mode data connection for ftpext-charter.txt (6060 bytes)
226 Transfer complete
QUIT
221 Goodbye. 

                                     Commands/Response on control channel

We issued these five commands in the following order at lines 2, 4, 6, 8, 10 and 13. 
USER – Send username to the FTP server
PASS  - Send the password (Anonymous servers need email address)
CWD  - Change the working directory on the server
PASV – To enter the passive mode (To let client connect to the server)
RETR – To retrieve a remote file from the server
QUIT – To terminate the connection to the server 

Between line 10 and 12, you will notice that the file was downloaded. To start the download, I had to open up another telnet window to open the data channel. To figure out to which IP address and port I had to connect to, we have to look at line number 9. We received a set of numbers (64,170,98,33,151,31) from the server in response to the PASV command. The first four related to the IP address 64.170.98.33 and the last two 151 and 31 help us identify which port to connect to.  Multiply the first by 256 and add it to the second. So, 151 * 256 + 31 which is equal to 38687.  Now that we have the IP address and port number, all we have to do is to open a second terminal and telnet to IP:Port as shown below:

C:\> telnet 64.170.98.33 38687 

This will now show you all the contents of the file ftpext-charter.txt being thrown into your second terminal window. Once this is done, you can proceed to type further commands on the control channel (the first terminal window).

Notes:
- The anonymous FTP server on IETF has a 60 second timeout on its control channel connection. Please connect to your own FTP servers they might be more forgiving to humans on terminals.
- FTP is not very secure as you can see the password and username are sent in plain text! Also, there is no encryption as you saw on file downloads or uploads.
- Type HELP once you send your password to see what commands you can issue the server.
- Here is a list of raw FTP commands and the parameters
- Here is a list of anonymous FTP servers
- This is the FTP sequence diagram which explains stuff at DNS and TCP level
- On windows there is a built in command line FTP tool (called ftp). It is useful but it does not show us how to use raw commands and communicate to an FTP server.
- SFTP (SSH File Transfer Protocol), FTPS (FTP over SSL) are more secure ways of using FTP.

The FTP HELP command via terminal

Hope this helps!

Tags: FTP, telnet, terminal

Run PHP on the Google App Engine

Google launched their Google App Engine (GAE) a year ago. The free hosting in App Engine is allocated 500 MB of persistent storage and enough CPU and bandwidth for about 5 million page views a month. Also, if you really want more you can see pricing plans.

GAE will support Java going forward. Unfortunately PHP support on the App Engine is still left as the top item in the wishlist. So until Google announces their official PHP support we have a workaround to run PHP using Quercus. Quercus is basically a 100% Java implementation of the PHP language (requires JDK 1.5).  Since the App Engine now supports Java this means we can use Quercus to run PHP scripts on the App Engine.

So all you need to use the GAE and run PHP
1) Register a free account.
2) Download this file to your computer.
3) Edit application XML tag in the file war\WEB-INF\appengine-web.xml to the name of the application you have registered.
4) Finally upload your application. I downloaded Google App Engine SDK for Java and use the following command in windows.
appcfg.cmd update C:\projects\phpwithjava\war

To see this in action just visit:
http://phpwithjava.appspot.com/webdigi.php and http://phpwithjava.appspot.com/info.php

NOTE: phpwithjava is my app name with GAE. Image by Aral Balkan.

Tags: appengine, google, PHP Development, quercus

Avoid Javascript blocking content download on your website during page load

Around 80% of the end-user response time is spent on the front-end. A fair share of this time is spent on downloading components of the page like scripts, Flash, stylesheets, images etc.  Javascript takes majority of the loading time of a webpage because scripts block parallel downloading and rendering in the page. Even if you do not have a lot of Javascript files to load on your webpage they can still block loading other content on your page while they load.  Lets have a look at how the standard javascript file include method and the script DOM element method in detail below.

Standard Javascript file include method

<SCRIPT src="A.JS" language="JavaScript/text"></SCRIPT>
<SCRIPT src="B.JS" language="JavaScript/text"></SCRIPT> 

<IMG src="1.GIF" />
<IMG src="2.GIF" />
<IMG src="3.GIF" />

Javascript blocks the other elements from loading (Example)

Script DOM element method

var p = g.getElementsByTagName("HEAD")[0];
var c = g.createElement("script");
c.type= "text/javascript";
c.onreadystatechange = n;
c.onerror = c.onload = k;
c.src = e;
p.appendChild(C);

Javascript executed without blocking any element from loading

This method creates a DOM element for each Script and then adds the element to the HTML. 

NOTE: 

1) Only when the appendChild function is called the Javascript will be executed.
2) For inlined code that depends on the Javascript and also for a similar method of Asynchronous Script Loading – See Steve Souders blog
3) If you go to MSN.com (Alexa top 10 site based on traffic) and hit view source you can see the javacript elements are included by using the script dom element to load the web pages faster. This is a good example of where you can use the script dom element method for certain js files that do not have inlined code dependency. Let us run a Pagetest waterfall report and you can see the following for MSN.COM

Pagetest Waterfall report of MSN.COM shows no blocking during page load

Tags: Javascript Development, performance, Web Development

What is new in PHP 5.3 for PHP amateurs?

In a recent PHP conference in London some great speakers spoke about new features in PHP to be released in PHP 5.3. PHP 5.3 contains functionality that was scheduled for PHP 6, which takes PHP 5.3 from being a minor release to a significant and huge release. A release that no PHP developer should ignore. Most of these features are pretty complicated additions for novice PHP programmers. I have listed some features and some ways to use them.

1) Namespaces for classes and functions
This feature will help us shorten the class names and function names. To appreciate this feature, we need to go back to the days before there was Object Oriented Programming in PHP. Imagine all the function names with name save(). How would you differentiate if the call save() was to save a blogs or save comments? The solution was to use blog_save() or comment_save() before the introduction of classes in which we could write the save() function within the Blog class or the Comment class. Using classes is obviously a much more elegant solution.

We now have the same situation with the large number of classes and functions. Using namespaces, we could simply separate the two functions above in the code below:

<?php
namespace Blog;
function save()
{
echo "Now saving the blog!";
}

namespace Comment;
function save()
{
echo "Now saving the comment!";
}

// To invoke the functions
Blog\save();    // This prints - Now saving the blog!
Comment\save(); // This prints - Now saving the comment!
?>

EDIT: A final decision was made on October 2008.  Developers will have to use \ backslash operator to dereference namespaces.

2) MySQL Native Driver
PHP 5.3 has a native driver specific to PHP, optimised for the ZEND engine. It is an alternative to connect to MySQL server versions newer than 4.1. Being a native driver we should be able to get much faster execution times. The native driver will also be licensed under the PHP license. If you are like most users, you are currently using libmysql (A MySQL database client library) you will be able to easily switch over to mysqlnd without making any changes to your existing PHP Scripts!

3) phar – PHp ARchive
This is a cool new feature.  Think of it like an archive, like a .zip file or a .tar file. Besides just being able to group all the files into one simple file, we will be able to deliver and run an entire PHP application from a single file!

We will also be able to use phar archives within PHP, so the following will work in PHP 5.3 and above

<?php
include "singlefilelibrary.phar"
?>

Obviously, there will be a performance hit but the possibilities are endless, imagine being able to upload phpMyAdmin to the server as a single phar file instead of hundreds of small files.

4) Closures & Lambdas
This gets into the list because this is something most web developers would have been familiar with while working on Javascript. A lambda can be declared anywhere and they can be assigned to a variable. A closure on the other hand are lambda funcions but have access to the variables where they were declared. This is something called lexical scoping. To see this in action take a look at this example.

<?php
$hellolambda = function () {
echo "Hello world via Lambda";

}
$hellolambda(); // Outputs Hello world via Lambda
?>

5) All of the rest!
There are a lot of other things in PHP 5.3 which I thought are nice, I have just described all of them very succinctly.
Functors: This allows an object to be invoked as a function.
Traits: This is a new unit of reuse, traits can be incomplete, provides reusability, modularity and structure. In short it is copy-paste glorified!
Magic functions: We have a couple of new magic functions for classes (interceptors) __callstatic() and invoke()
Ternary operator: You can now display the a value that exists $value1 or $value2 using this simple statement echo $value1?:$value2;
There are many more things added like Late Static Binding, Variable Static Calls, Changes to PHP Error Levels, new PHP functions, improvements to help with OpenID, Command line and many more.

Final Thought
Well, this gives us much more to play with. It is definitely a lot to include into PHP 5.3 and I would have expected so many changes to go into PHP 6. I sometimes wonder if there will be anything new left to add into PHP 6 given the fact that so much has been released already.  If you are interested in PHP 5.3, do give it a try here, it is in beta at the time of the writing.

Tags: closures, lambdas, namespaces, phar, PHP Development

Add Google Adsense to WordPress without plugins

I was just trying to add Adsense to this blog and found several plugins available to add Google adsense to wordpress. Finally, I managed to add Google Adsense to this blog without any plugins – this solution takes around 5 mins to setup and it only uses the text widget available in wordpress by default.

Step 1: Login to your blog as administrator
Step 2: Click on Appearance
Step 3: Select Widgets
Step 4: Select the sidebar you want (Default Sidebar 1)
Step 5: Add Text widget
Step 6: Click on Edit widget
Step 7: Enter Caption. I have used “Useful Links”
Step 8: Paste your Google Adsense code in the text area. See example code & screenshot

Example (Copy and Paste below into Widget):

<script type="text/javascript">
<!--
google_ad_client = "pub-9990819195828769";
google_ad_slot = "7810166087";
google_ad_width = 200;
google_ad_height = 200;
// --></script>
<script src="http://pagead2.googlesyndication.com/pagead/show_ads.js"
type="text/javascript"></script>

Screenshot

Wordpress Text Widget - Add Google Adsense code

Step 9: Click done & Save changes.

You should now be able to see Ads from Google showing on the left side of your sidebar. Just like we have on the top right of this page.

Tags: adsense, google, wordpress

Hello World! – Adobe Air on Ubuntu and Windows

This article is to help fellow web developers get started with working on Adobe AIR! Using Adobe AIR, you will be able to create desktop applications which are cross-operating system using just HTML, JS and a bit of XML.

Adobe Air works great on Windows, Linux, Mac and hopefully a lot of future operating systems and mobile devices. If you get an application working, it should work and display exactly as it does across all the other operating systems as all AIR installations use WebKit. This article will give you an insight into how I got it working on Windows and Linux. We need Abobe AIR SDK to develop and build AIR applications. You can get Adobe AIR SDK from this link here.

Setting up AIR SDK on Ubuntu
1) Get Adobe AIR SDK from http://adobe.com/go/getairsdk (Choose Linux)
2) Extract contents of the file into a location that you prefer
3) Pick the adl and adt files and place them in your bin directory on your machine OR set a path to the bin folder so that these files are accessible
4) Once you have it all setup, you should be able to run adl on your terminal and get a message back instead of the standard “command not found”
5) You will have to save both of the files (AIRHello.xml and AIRHello.html into a folder)
6) On the terminal, go into the folder where the files are saved and run “adl AIRHello.xml” and that’s it you

Setting up AIR SDK on Windows
1) Get Adobe AIR SDK from http://adobe.com/go/getairsdk (Choose Windows)
2) Extract contents of the file into a location that you prefer
3) Go to Start > My Computer (Right click Properties) > Advanced Tab > Environment Variables > Path > Edit and add the path to your bin folder
4) Once you have it all setup you should be able to run adl on your command line and get a message back instead of the standard “command not found”
5) Make sure that on the command line if you type “path” you get to see the new entry.
6) You will have to save both of the files (AIRHello.xml and AIRHello.html into any folder)
6) On the command line browse to the folder where files are saved and run “adl AIRHello.xml” and that’s it!

AIRHello.xml

<?xml version="1.0" encoding="utf-8" ?>
<application xmlns="http://ns.adobe.com/air/application/1.0">
<id>uk.co.webdigi.AIRHello</id>
<filename>AIRHello</filename>
<name>Hello World</name>
<description>This is a sample Adobe AIR application.</description>
<version>1.0</version>
<initialWindow>
<content>AIRHello.html</content>
<title>Hello by Webdigi</title>
<systemChrome>standard</systemChrome>
<transparent>false</transparent>
<visible>true</visible>
<minimizable>true</minimizable>
<maximizable>true</maximizable>
<resizable>true</resizable>
<width>500</width>
<height>500</height>
<x>150</x>
<y>150</y>
<minSize>300 300</minSize>
<maxSize>800 800</maxSize>
</initialWindow>
</application>

AIRHello.html

AIRHello
<div>
<h2>Hello World</h2>
</div>

This should get you started on working with Adobe AIR! You can easily create AIR applications using just HTML, Javascript and XML, all tools that web developers regularly use. You can do more advanced stuff like system tray alerts, drag and drop, sounds, file access and many more stuff that you can’t do on a traditional web browser.

Tags: adobe air

Reddit vs Digg – A collaborative test for users.

I recently decided to run a test to see how well users can collaborate using social bookmarking sites like reddit & digg. This was assuming that users will collaborate and can accomplish something together!

How the test works

1) Each visitor to the site will be given any ONE random letter of a phrase (one per IP address).
2) They have to use the comments or any method to share with other users know the letter they got.
3) All users have only two attempts to guess the phrase. After they have used their attempts they can share their guesses in the comments section.

EG: If the phrase was “wild goose chase”. Visitors to the page might get any letter from ‘w’ through to ‘e’ and so on. Users use the comment section to share their letters and work together and find out the phrase. This is how a test looked on reddit.

Result of test on Reddit 

Phrase / Word          Section posted       Time to finish

Observatory              Programming        14.5 minutes
Starbucks Coffee       Entertainment     107.3 minutes 
Wild Goose Chase      WTF                   225.6 minutes 

Result of test on DIGG 

Phrase / Word          Time to finish

Spill the beans           580.5 minutes

Observations on the test results

1) Although more than 5000 visitors saw each page. There were around 105 comments and around 20 – 30 users who actively participate to guess the phrase.
2) Thanks to proxy servers used – Some users (esp in programming section) could get more letters from the system.
3) Reddit edit comments helps a lot as users can change their comments any number of times.
4) Another interesting observation was that some people lie that they got a letter which was not actually given to them. It was good to see users upvote the letters in the comments that they have also got to confirm a letter.
5) Reddit programming users managed to find the word in just under 15 minutes of starting the test. WOW!

Reddit users win the collaboration test!

I am looking for any other fun ideas of a collabortive test? Looking forward to hear your ideas!

Tags: collaborative, digg, fun, reddit, test

Google store does not use google checkout instead uses Worldpay

The Official Google Store which sells a range of Google labelled products doesn’t use Google Checkout for credit card processing but instead uses Worldpay.

Google Store uses Oscommerce (popular open source PHP ecommece store management program) and free open source plugins are available to integrate google checkout with oscommece in minutes (eg: http://code.google.com/p/google-checkout-oscommerce/). I wonder why they cant simply use Google checkout to deal with credit cards on their own site?

Screenshots
1) Google store without Google checkout option for payment.
2) Worldpay message when checkout starts.

Another question
OK, now we know that Google store doesnt have Google Checkout support at the moment. Then how does the store have the google checkout badge on their sponsored listing?

Google search result shows the Google Checkout badge

Try a Google search for “Google Store” and you can see the sponsored link has Google checkout badge on it.

NOTE: I love Google and use most of their products. Just don’t understand why they cant use their own checkout in their official store.

Tags: google, google checkout, google store, oscommerce, worldpay

PHP on Windows Server 2008 : 500 – Internal Server Error on IIS

Recently we had to work on a Windows server 2008 (web edition) machine with IIS 7.0 and we ran into the strangest of errors. The server stops running the script with this message :  

Server Error 
500 – Internal server error.
There is a problem with the resource you are looking for, and it cannot be displayed.
Cause:

This error happens on the default configuration of windows server 2008. The server logs do not say a lot except for the fact that the php file is causing an error. This happens for the following reasons on the server:

• PHP Fatal error (Something that you did or didn’t do caused the server to run into a fatal error)
• PHP error_reporting is on and there are some errors (Anything from a NOTICE or WARNING can cause this)

 
How to rectify:

Obviously the best way to resolve the error is find out what is wrong with the PHP code. But how do you do this if the server won’t show you what the error is? One of the best ways to find what is wrong with your PHP script is to try to login to the machine via RDP. This works only if you have remote desktop access. Once in RDP try to go to the same URL but using the server’s Internet explorer. Accessing the server locally will avoid the 500 Internal Server Error and show you what is wrong with the PHP file as on a development environment. The server just shows you the PHP error messages and the rest of your script output is not shown.

Change IIS 7.0 Setting
Alternatively to the above, you can always change the configuration using the Configuration Editor of IIS 7.0. You will find this under the Section > system.webServer/httpErrors. You have to change errorMode to Detailed from the usual DetailedLocalOnly and then click on Apply.

This is a good security measure as it will not accidentally show your PHP errors to the users of your system.

Tags: IIS 7.0, Server Error, Windows Server 2008

How to detect if your webserver is hacked and get alerted

We all do our best to write excellent code and also keep our installations of popular open source tools like WordPress, Joomla, Oscommerce, Drupal, phpmyadmin and all its plugins always updated to prevent any attack or hackers using known exploits on them. This article is not aimed at going through all those methods to help you secure your website BUT focuses on how to send you an alert once your website is hacked and running “hidden” code that you didnt write.

Read more…

Tags: hash, PHP Development, Security, Web Development, webserver