Skip to content

HTTP: Requests and responses

In the previous article we introduces HTTP, the protocol used for the majority of communication between clients and servers on the web. In this article we examine how a client requests data from a server using HTTP, and the format of the response the server replies with.

The diagram below gives an overview of the interaction between the client and the server, when requesting a webpage.

Example HTTP Request

This request - response flow is used for almost all web based communication, whether browsing a static website, searching google, or getting data on dynamic websites using AJAX and REST (Representational state transfer)

HTTP Requests

HTTP Requests form the basis for all communication, between devices on the web. The client will request a given resource (or page) using a URL. The Request should contain all the data the server requires to properly serve the response.

Lets talk HTTP

We can open a manual connection to a server using a TCP communication tool like netcat or telnet.

Lets make a connection to the cueh.coventry.ac.uk

dang@DESKTOP-KJDVQ2J:$ ncat -v -C cueh.coventry.ac.uk 80
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to 194.66.34.56:80.

dang@DESKTOP-KJDVQ2J:$ telnet cueh.coventry.ac.uk 80
Trying 194.66.34.56...
Connected to cueh.coventry.ac.uk.
Escape character is '^]'.

Important

Depending on your version of netcat you may not need to use the -C flag. In my case it is needed otherwise it sends the wrong line endings, meaning the request is mangled.

You will notice that at this point the Connection hangs. This is because the HTTP protocol states that the client needs to make a request for data (Bob asking Alice for the file in the example above).

We can tell the Server we are talking HTTP, by sending the following message. You may need to hit enter a couple of times after typing it.

GET / HTTP/1.1
host: cueh.coventry.ac.uk

dang@DESKTOP-KJDVQ2J:~$ ncat -v -C cueh.coventry.ac.uk 80
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to 194.66.34.56:80.
GET / HTTP/1.1
host: cueh.coventry.ac.uk

HTTP/1.1 200 OK
Date: Wed, 04 Aug 2021 15:30:04 GMT
Server: Apache/2.4.18 (Ubuntu)
... SNIP ...

dang@DESKTOP-KJDVQ2J:~$ telnet cueh.coventry.ac.uk 80
Trying 194.66.34.56...
Connected to cueh.coventry.ac.uk.
Escape character is '^]'.
GET / HTTP/1.1
host: cueh.coventry.ac.uk

HTTP/1.1 200 OK
Date: Wed, 04 Aug 2021 15:24:24 GMT
Server: Apache/2.4.18 (Ubuntu)
... SNIP ...

So what happened here?

  1. We opened a connection to a host running a HTTP compliant service.
  2. As the HTTP protocol states the client will initiate the request, (there is no acknowledgement that the connection is open as part of HTTP),
  3. We made the HTTP request by sending the following text to the server. 
    GET / HTTP/1.1
host: cueh.coventry.ac.uk
  4. The Server responded with some data.

The Request itself can be broken down into.

  • GET is the request method
  • / is the resource we are requesting. In this case its the root of the website (usually index)
  • HTTP/1.1 tells the server we are using version 1.1 of the HTTP protocol
  • host is an additional header, telling the server which site want to look at3

Easytask

Using your choice of Netcat / Telnet (or both) make a request for a web page from cueh.coventry.ac.uk

  • Try sending the request without the host field. What happens?
  • Try sending a HTTP 1.0 version of the request, do we need to send the same information to get the data?

Now we have made a request, lets look at these elements in more detail.

The Request Method

The web would be a pretty boring place if we could only request single static pages. Request methods allow us to interact with the server in different ways, and do more than just request a page.

Common HTTP Request methods include:

  • GET requests are used to request content and send small amounts of data.
  • POST requests are used to send data send larger amounts of data to the server (they also retrieve the page)
  • HEAD requests returns just the HTTP headers

Other request methods such as TRACE, PUT, DELETE and OPTIONS are also implemented, although we will not use them frequently here (although REST based APIs and the OPTIONS header can provide a good source of amusement and information disclosure2, and other methods may be useful for information disclosure).

The Resource Requested

We also need a way of telling the server which piece of information we are interested in.

The resource element of a HTTP request is the URL that we are interested in. URL's are a unique identifier given to each item on the server. For example /index.html or /auth/login.html

Note

When making a request the Resource identifier is the part of the URL following the domain.

Most user-agents will automagically split whatever URL we provide into the host and resource parts before sending, but if we are making a manual request we will need to do it ourselves.

For example with the a URL of http://cueh.coventry.ac.uk/people/index.html we get

  • Host of cueh.coventry.ac.uk representing the host (or server) we send the request to
  • Resource of people/index.html representing the page we want to get from this server

The HTTP Version

We discussed HTTP versions in the previous article on HTTP The HTTP version is generally sent as HTTP/1.1 or HTTP/2

HTTP Request Headers

Request headers allow us to send extra information to the server to help it to complete the task we want. This is intended to help the host respond to the request in an appropriate way.

In our example request above, we send the host field, telling the web-server that we expect a the page for the cueh.coventry.ac.uk subdomain.

Important

Depending on the server implementation we may get a different response if we are missing the host header.

  • The Server may return the default subdomain
  • The Server may default back to HTTP 1.0
  • An error could be raised

In the case of cueh, Apache raises an error if we don't send the host field.

Lets examine some request headers to see the types of data that are sent. We will need to request a page from a website, we will use httpbin, which is designed for testing the data sent in HTTP requests.

All of the major request methods (GET, POST, HEAD etc) will send the same core set of request headers, with the different functionality encapsulated in either optional headers, or inside the request body.

Request headers sent by a browser can be viewed without any plugins, on:

  • Chrome using developer tools (Network Bar, click on the file to see its requests)
  • Firefox developer tools (Network, click the file)

If we visit http://httpbin.org/get/ and examine the request headers in the browser we get this:

Get Request headers for httpbin.org

Host: httpbin.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

The only mandatory line is:

  • Host The host name of the server we wish to talk to. This is used if there are multiple websites on the same server, and is mandatory when making HTTP 1.1 and above requests.

Note

While this is not mandatory for HTTP 1.0 we can consider this version of the protocol obsolete

Optional, but interesting lines are:

  • Referer tells the server where the request came from (this is not a typo; it was misspelled in the original HTTP specification and has become standard since)
  • User-Agent provides information about the browser making the request. More history here, as most browsers will include the Mozilla string (thank Netscape for that)1
  • Accept tells the server the types of data this request is expecting
  • Cookie isn't shown here, but it can be used to submit additional cookie parameters to the server

Video Looking at Request headers

HTTP Responses

Each request will be served with a response. Usually this will be an HTML- based web page, however protocols such as REST may return things like JavaScript Object Notation (JSON) formatted data. Looking at a request outside of the browser can be done with tools like Netcat (or python). For example, grabbing the CUEH website gives the following response (truncated for brevity):

HTTP/1.1 200 OK
Date: Mon, 06 Mar 2017 13:01:22 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Tue, 10 May 2016 15:48:20 GMT
ETag: "2446-5327edab5ac1c"
Accept-Ranges: bytes
Content-Length: 9286
Vary: Accept-Encoding
Content-Type: text/html

&lt!DOCTYPE html&gt
&lthtml lang="en"&gt

... (HTML DATA) ...

The first part of the response is the Header, which contains information such as:

  • Response Status Code (HTTP/1.1 200 OK) -- defines how to handle the response:
    • HTTP version
    • HTTP status code
    • Textual version of status code
  • Server -- gives us information on the server itself
  • Last Modified -- date page was last modified
  • E-Tag -- hash of the page (used for caching)
  • Content-Type -- the type of data that is being returned in the body
  • Content-Length -- the amount of data being returned in the response body

This actually leaks quite a bit of information. For example, the type (and version) of the server that is responding, and a good idea of when the page was created or modified -- more stuff that can be included in your recon phase.

Note

This is one of the times that having a good understanding of the underlying protocols is good. Sometimes having a good think about the data returned can be really useful, and applying a bit of cunning can help provide more information for the recon phase, or save you time. For example, the server type can be useful for finding exploits; last modified can tell you if the page has been changed recently, or what types of technology to expect.

HTTP Response Status Codes

The first element of a HTTP response is its status code. Response codes let us know the status of the request, and gives some information on how to handle the response.

Response codes are broken into categories, (the hundreds parts of the code), consisting of subcategories (the tens and units parts of the code)

The core response code categories, and common response codes include:

  • 1xx Information
  • 2xx Success
    • 200 OK -- The page was returned without error
  • 3xx Redirection
    • 303 Redirect -- The page has moved, attempt to automatically redirect to the correct address.
  • 400 Bad Request
    • 401 Unauthorised -- There has been some issue authenticating with the server.
    • 403 Forbidden -- the server actively rejects the request
    • 404 Not Found (no such page)
  • 500 Internal Server error

HTTP Response body

The response body will include the data being returned by the web server. This will be either the HTML representing the page itself, or in the cases of a REST service, some representation of the data (usually in JSON, but it can be XML).

Summary

In this article we have discussed methods of sending and receiving data using the HTTP protocol.

HTTP requests are used by a client to request information from a service. To allow the service to respond dynamically to a request, we can also include parameters to the server.

The server returns this information though a HTTP response. If successful, this will return the data from the server to the client.

In our next article we will look at how data is sent to the web server as part of a request.

Video "Lecture"

Further reading

  1. History of the user agent string https://webaim.org/blog/user-agent-string-history/ ↩

  2. Bill Sempf on testing REST API's ↩

  3. HTTP > 1.1 requires this host field. If we were to send a HTTP 1.0 request, it is optional. ↩

Back to top