.: Sarah's Blog :.

An Corran - Dinosaur Beach

27/07/2010 11:09AM

Steven and I went for a three day trip to the Isle of Skye last weekend, and went to a beach near the village of Staffin called An Corran, made famous for not only being the oldest Mesolithic human hunter-gatherer site (~6500BC) on Skye, but also a place to see dinosaur footprints embedded in the rocks!

The footprints are around 160 million years old, and probably belong to a large predatory dinosaur similar to a Megalosaurus. A hilarious aside about Megalosaurus: part of a bone was recovered from Oxfordshire in 1676, where it was incorrectly assumed to be from a giant human. It was then described again in 1763 and called a Scotum Humanum when comparing its appearance to a pair of human testicles. Funnily enough the label was not considered to be a proper name for an animal..! In the 1990s a petition formally suppressed the genus name Scrotum in favour of Megalosaurus. (Read more at Wikipedia)

Here is a photo of one of the footprints at An Corran:

From http://www.skyecomuseum.co.uk

An here is a photo I took of what I believe to maybe be a footprint - we unfortunately arrived at high tide, covering most of the footprints. Confirmation bias or dinosaur footprint?

Get Demand Five working in Windows 7

08/07/2010 12:32PM

Demand Five is Channel Five's online TV player, and it claims not to work on Windows 7 when you want to purchase a program to download/watch. When you try and buy the programs, the website does a check for "system requirements" and if you are running Windows 7, this will fail, like in Screenshot 1. I have found a way around this, and was able to purchase my program and watch it on my Windows 7 laptop.

Screenshot 1: What happens in Windows 7

Disclaimer: if this doesn't work for you, please don't blame me. Your purchase is tied to your account, so if you log into a Windows XP/Vista machine, you will be able to download your program (and not pay twice).

You will need:

Firefox (download)
Internet Explorer 8 (download)
The latest Windows Media Player (download)
The Firefox plug-in Firebug (download)

Go to the Demand Five website in Firefox, and proceed with picking your programs and buying them. You will get to the screen like in Screenshot 1, and the website will not allow you to proceed. Luckily, Five does their validation client-side in JavaScript, which means using the Firebug plug-in you can edit the values it is validating and circumvent the checks.

Firebug has a little bug icon at the bottom-right of Firefox which you should click on. This will bring up a Firebug console. Reload the Demand Five page, and then click on the Script tab in Firebug. Scroll down until you get to line 120, and press just to the left of the line number, which should produce a red dot. This should look like Screenshot 2.

Screenshot 2: Line 120 is the line stopping Windows 7 users. Clicking to the left of the line number will produce a red dot.

This will pause the JavaScript on the webpage. You can now change the value that gets passed in! Go to the DOM tab, and scroll down until you see a bold b on the left hand side. Click the plus sign, and you should see something like Screenshot 3.

Screenshot 3: You can see an entry called OS with "Windows Storage Server" in red. This may be different for you and might say "Windows 7" or similar.

Double click on the text in red with the label OS, and change this to "Windows Vista" with quotes. Now you can press the blue play button in the top-right of Firebug (Screenshot 4).

Screenshot 4: Press the button to allow the JavaScript to continue. Now the variable has been changed to Vista, the check will allow you to continue.

Voila! You can now proceed. You may have to do this a few times if they redirect you back to this page, but eventually you will be allowed to pay for the downloads. When you are finally on the page where you can click your downloads to actually download/play them, move to Internet Explorer, otherwise odd error alerts appear and the downloads never happen. Just copy and paste the URL into IE8 and proceed with the downloads. Accept all the ActiveX things the site wants to install.

Happy TV watching! Many thanks to Steven for coming up with this ingenious get-around :)

How Internet Explorer stores web history

15/06/2010 04:52PM

Internet Explorer stores files downloaded from the internet in a cache called Temporary Internet Files (e.g. html pages, images, CSS files). Each cached file is assigned an alphanumeric cache name. Some index.dat files serve to map the cached name with the filename and URL it came from. Other index.dat files store the user’s cookies or web browser history (by default 20 days’ worth). index.dat files are in binary format, and need to be viewed using a hex editor.

There are numerous index.dat files kept on Windows machines. Assuming the computer is running Windows XP, the locations of the main index.dat files are:

C:\Documents and Settings\<UserName>\Local Settings\History\History.IE5\index.dat

(Older history index.dat files can be found in C:\...\History.IE5\MSHist[18digits])

C:\Documents and Settings\<UserName>\Local Settings\Temporary Internet Files\Content.IE5\index.dat

For Windows Vista and Windows 7, the corresponding paths are:

C:\Users\<UserName>\Local\Microsoft\Windows\History\History.IE5\index.dat

C:\Users\<UserName>\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5\index.dat

The index.dat files all have the same format, and comprise of a header followed by a series of records. There are four types of records: HASH, REPR, URL and LEAK. HASH records are indexes to the other three record types, and can be ignored as they are only used internally by Internet Explorer. REPR, URL and LEAK are called activity records, since they each contain information about some sort of online browser activity.

There are a few differences between the various index.dat files. The one stored in the Temporary Internet Files folder (the "cache index.dat" file) is used to relate web files to those cached on the computer, so this additionally stores the names of the cached folders in the file header, and a reference to a corresponding cache folder within each activity record. Other differences will be explained below.

INDEX.DAT FILE HEADERS

The headers contain a small amount of information about the file and, for a cache index.dat, an array of cache folder names.

The image above shows the header of an example cache index.dat file. All index.dat files start with "Client UrlCache MMF" followed by the version number, which is shown in red. Next, in blue, is the size of the file. All numbers are stored little-endian. Following on, in yellow, is a pointer to the start of the first record. In this example the next part of the header names four subfolders where the cached files are located – shown in green. In non-cache index.dat files, these would be 0x00 (null) values.

INDEX.DAT FILE CONTENTS

There are three types of activity records. These contain URL information and have the following common structure, illustrated by the image below:

TYPE: 4 bytes, either URL, LEAK or REDR. Shown in yellow.
LENGTH: 4 bytes, contains the length of the record in 128 byte (0x80) sized blocks.
DATA: variable length, the data we are interested in. Shown in grey. The end of every record is given by a 0x00 character, which can be seen in blue. The rest of the record is just filled with junk.

REDR ACTIVITY RECORDS

REDR records contain just a URL and indicate a redirect to a different location.

URL ACTIVITY RECORDS

These are the important records and an example can be seen in the image below. The information held in the DATA section is dependent on the type of index.dat file. They all start with the last modified time (in blue) followed by the last accessed time (in green). Time is stored in Windows FILETIME format (100-nanosecond intervals since 1st January, 1601 UTC).

If the index.dat is a cache file, like that of the image below, the structure follows that of Table 1. If the index.dat is a history file, the structure follows that of Table 2, and looks like the final image.

Location	Meaning
38 bytes in	Reference to the cache folder the file is located in. This is just one byte long and is an index into the array of cache folders given in the file header. Shown in dark grey (second to last image).
96 bytes in	The URL the file came from (shown in purple). This is followed by the name of the corresponding cached file stored on disk (orange) and finally the HTTP headers (dark blue). Each part starts on a new 16 byte boundary. The Windows username is attached to the end of the HTTP headers.

Table 1 - The DATA structure of a URL activity record in a cache index.dat file

Location	Meaning
96 bytes in	A URL starting with "Visited: <user>@". This is a URL the user with the login name <user> has visited using their Internet Explorer browser (shown in purple in the last image).

Table 2 - The DATA structure of a URL activity record in a History.IE5 index.dat file

LEAK ACTIVITY RECORDS

LEAK activity records look the same as URL activity records, and are essentially a Microsoft term for an error.

References

Forensic analysis of internet explorer activity files, K Jones, 2003
The meaning of LEAK records, Mike Murr, 2010
Understanding index.dat Files, S Bunting, 2010
Delete index.dat files, Mil Incorporated, 2009

How Opera stores web history

14/06/2010 03:12PM

Out of all the popular browsers, Opera leaves behind the least amount of useful information for investigators. Not only is the data stored in plain text format, but it does not record every URL visited, only the latest one. Therefore it is impossible to tell how often someone has visited a particular website. Even when viewing web history from within the browser only the latest entries are shown, giving a false impression of the actual history. For example if someone went to exactly the same websites two days in a row, the first day would have no history associated with it, since each entry would be overridden by the latest visit.

If the computer is running Windows XP, the Opera default path to user data is:

C:\Documents and Settings\<User Name>\Application Data\Opera\Opera

For Windows Vista and Windows 7, the default path is:

C:\Users\<User Name>\AppData\Roaming\Opera\Opera\

There are two important files in this directory, global_history.dat and typed_history.xml.

GLOBAL_HISTORY.DAT

global_history.dat is a plaintext file which stores details for each URL visited. Each entry takes up four lines as described in the table below.

Line	Meaning
1	The title of the website.
2	Website URL.
3	Time of the last visit. This is the number of seconds after the Unix Epoch UTC (1st January 1970).
4	An integer representing the ‘popularity’ of the website. This number seems to be set to -1 initially, and rises to several million if the website has been visited more than once. It is not clear how this relates to the popularity column when you view web history via the Opera browser.

The lines below show two entries in global_history.dat and the image underneath shows the same entries in Opera’s history viewer. Popularity has been made bold. It is unclear what the popularity field in the file and the viewer mean, and how 1936804 corresponds to 2 and -1 to 1. They do not correspond to number of visits.

http://cuteoverload.com 
http://cuteoverload.com
1276255696
1936804   
BBC NEWS | News Front Page
http://news.bbc.co.uk
1276255696
-1

TYPED_HISTORY.XML

typed_history.xml is an XML file that has an entry for each URL that was typed in manually. Each entry is in the format

<typed_history_item content="url" type="type" last_typed="date" />

Type is either ‘text’ or ‘selected’, where ‘selected’ means the URL was chosen from Opera’s autocomplete and ‘text’ means it was typed in manually. Again, this only stores the latest URL and not all of them.

And that's really all I could find that Opera stores!

References

IR and forensic talk blog post

How Safari stores web history

11/06/2010 11:41AM

Safari has a very simple method of storing browser history compared to those that use SQLite databases.

If the computer is running Windows XP, the Safari default path to user data is:

C:\Documents and Settings\<User Name>\Application Data\Apple Computer\Safari

For Windows Vista and Windows 7, the default path is:

C:\Users\<User Name>\AppData\Roaming\Apple Computer\Safari\

For Apple Macintosh computers, the default path is:

/Users/<User Name>/Library/Safari

History is stored in a binary XML file named History.plist. History is only stored for a month. The data can easily viewed in a hex editor, but can be converted to a structured text file on Macintosh computers by typing in the command:

$ defaults read /Users/<username>/Library/Safari/History > history.txt

This produces output like below:

{
WebHistoryDates = (
       {
            "" = "http://www.twitter.com/";
            D =             (
                1
            );
            lastVisitWasFailure = 1;
            lastVisitedDate = "297678774.0";
            redirectURLs =             (
                "http://twitter.com/"
            );
            title = "Twitter / Home";
            visitCount = 2;
        },
        {
            "" = "http://www.apple.com/startpage/";
            D = (
                1,
                3,
                3,
                1,
                1,
                2,
                0
            );
            W = (
                12,
                18,
                15,
                31
            );
            lastVisitedDate = "297678768.7";
            title = "Apple - Start";
            visitCount = 87;
        },           
    );
    WebHistoryFileVersion = 1;
}

The format for each entry appears to have the URL visited, followed by a list of “D”, an optional list of “W”, an optional last visit was a failure, the date this URL was last visited, an optional redirect URL, the URL page title and the total visit count. Safari stores the date as the number of seconds since 1st January 2001 GMT. It is not clear what the D and W lists are for, as the only reference to History.plist on the internet is to an older version of Safari which stored fewer fields. If anyone has any ideas on what they mean, I'd love to know. Even programs that convert web log files into nice tables (Such as Net Analysis) ignore the D and W fields.

Other files that are useful are Bookmarks.plist and Form Values.plist. Bookmarks.plist contains a list of the user’s bookmarks, and Form Values.plist the user’s form inputs – however this file appears to be store data in an encrypted format.

References

Mac Forensics: Mac OS X and the HFS+ File System, Philip Craiger & PK Burke
Analyzing Safari 2.x Web Browser Artifacts using SFT, Jacob Cunningham
MacScriptor Forum

How Google Chrome stores web history

10/06/2010 03:45PM

In 2008 Google released most of Chrome’s source code as a project called Chromium under a BSD license. Chromium is essentially the same browser as Chrome, but lacks built-in automatic updates and Google branding [Chromium Developer Website].

Assuming the computer is running Windows XP, the Chrome default path to user data is:

C:\Documents and Settings\<User Name>\Local Settings\Application Data\Google\Chrome\User Data

For Windows Vista and Windows 7, the default path is:

C:\Users\<User Name>\AppData\Local\Google\Chrome\User Data

Chrome uses SQLite databases similar to Firefox but with notable differences explained below. All databases are stored in the directory Default, with history and web data being the most important databases stored here. Chrome does not add extensions to its files; but fortunately SQLite databases have a file header of “SQLite format 3” making them easy to recognise.

Chrome is not consistent with which datetime format it uses, some tables use standard Unix Epoch time (microseconds since 1st January 1970) whilst in others it uses its own variation of Windows Filetime (100 nanosecond intervals since 1st January, 1601 UTC divided by 10).

HISTORY

Similar to places.sqlite in Firefox, history contains a table of unique URLS visited called urls, and a table for each unique visit called visits. Also like Firefox, it is possible to trace a user’s exact path on the internet as Chrome also stores the referring URL and the visit type, which Chrome names transition. Tables below show the fields in urls and visits respectively.

Field	Meaning
id	The table’s primary key. This is used in a lot of other tables to reference this table.
url	Stores a unique visited URL.
title	Stores the URLs page title.
visit_count	Stores the total visit count for this URL.
typed_count	Stores the number of times the URL was typed manually.
last_visit_time	Stores the last time the URL was visited. This is stored in Google’s own variation of time.
hidden	Indicates if the URL will be displayed by the autocomplete function. A value of 1 will keep it hidden and 0 will display it.
favicon_id	A foreign key to the favicon table which stores the favicon for each URL.

Table urls

Field	Meaning
id	The table’s primary key.
url	Stores a foreign key to the urls table.
visit_time	Stores the time the URL was visited in Google’s own variation of time.
from_visit	Stores the id from where the URL came from originally. If the URL does not have a referring URL this value is 0.
transition	Shows how the URL has been accessed. Unlike Firefox which just gives a number to represent the type of transition, Chrome has some additional information such as whether or not this was a client or server redirect and if this is a beginning or end of a navigation chain. The transition is a long number, so to extract just the transition type number the variable CORE_MASK (0xFF) has to be used to AND with the transition value. There are 11 page transition types, such as 0 – user followed a link, 1 – user typed in the URL and 6 – this is the user’s homepage/startpage.
segment_id	Stores the segment id. It is not clear what ‘segments’ are. There are tables called segments and segment_usage also in history, and they appear to store the domain names of accessed URLs along with a total visit count.

Table visits

To see how these tables relate, see my earlier blog post on how Firefox stores web history.

WEB DATA

web data stores logins and autofill data. The table logins stores usernames and passwords for URLs the user has asked to save data for. Only the password is stored encrypted. The table autofill stores values the user has used in forms. This is joined to autofill_dates which stores the last used date for the data. There are also tables called autofill_profiles and credit_cards which contain the user’s details and credit card information so they can be automatically added to web forms.

HISTORY INDEX FILES

In the same folder are files named History-Index-YYYY-MM which stores the text-based contents of any website visited. This is used when the user searches through web history with key words. Chrome will search over the history index files to find any webpages that match. Over time these files can grow very large, to tens of Gigabytes. This has been issued as a bug in the browser, but can be of importance to a forensic investigator who can also do key word searches to get an idea of the contents of the pages the user was visiting.

References

SANS Google Chrome Forensics
Chromium Documentation: page_transition_types.h, time.cc
Chrome Bug report: Issue 16705: User Data Grows Too Large (~2GB)

How Firefox stores web history

09/06/2010 06:05PM

Firefox version 3 (first released in 2008) employs a different system of storing browser history than its predecessor Firefox 2. Since only 2.75% of Firefox users still use version 2 or smaller, only Firefox version 3 will be explored here and will hence just been known as Firefox.

Firefox uses SQLite database files to store browser history, bookmarks, cookies, downloads, form field entries and web site logins. Assuming the computer is running Windows XP, the Firefox default path to the databases is:

C:\Documents and Settings\<user>\Application Data\Mozilla\Firefox\Profiles\<profile folder>\

For Windows Vista and Windows 7, the default path is:

C:\Users\<user>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile folder>\

The Firefox databases are stored in several different .sqlite files, which can be viewed using a SQLite viewer such as SQLite Database Browser. Several of these files hold important forensic data, the most useful being places.sqlite and formhistory.sqlite. Firefox stores history for a default of 90 days, whereas Internet Explorer for only 20 days and Firefox version 2 for only 9 days.

PLACES.SQLITE

places.sqlite is the main web history database and stores URLs accessed and user bookmarks. The database contains 11 tables, two important ones are moz_places and moz_historyvisits. The fields in moz_places can be found in the table below.

Field	Meaning
id	The table’s primary key. This is used in a lot of other tables to reference this table.
url	Stores a unique visited URL.
title	Stores the URLs page title.
rev_host	Stores the reverse host name.
visit_count	Stores the total visit count for this URL.
hidden	Indicates if the URL will be displayed by the autocomplete function. A value of 1 will keep it hidden.
typed	Indicates if the URL was typed into the address bar or not. A value of 1 means it was manually entered.
favicon_id	A foreign key to the favicon table which stores the favicon for each URL.
frecency	Amalgamation of the words frequency and recency. Frecency is "a score given to each unique URI in Places, encompassing bookmarks, history and tags. This score is determined by the amount of revisitation, the type of those visits, how recent they were, and whether the URI was bookmarked or tagged", Mozilla Developer Center. This value is used by Firefox’s autocomplete. URLs start with a value of -1, and the higher the frecency the higher in the autocomplete the URL will appear. Values of 0 are ignored (and have a value of 1 for hidden).
last_visit_date	Stores the last time the URL was visited. This is a 64bit integer storing number of microseconds since 1st January 1970 UTC called PRTime.

Another important table in places.sqlite is moz_historyvisits which stores all accessed URLs. The fields can be found in the table below.

Field	Meaning
id	The table’s primary key.
from_visit	Stores the id from where the URL came from originally. If the URL does not have a referring URL this value is 0.
place_id	Stores a foreign key to the moz_places table.
visit_date	Stores the time the URL was visited in PRTime.
visit_type	Shows how the URL has been accessed. This is one of seven possible values, the most common being: 1 – the user followed a link and got a new top-level window; 2 – the user typed in the URL or selected it from autocomplete results; or 3 – the user clicked on one of their bookmarks to get to the page.
session	Stores the session ID that the URL belongs to.

Using from_visit and place_id it is possible to retrace a user’s steps and see how they got to a particular page. Using the two images below, it can be shown that an example user accessed three additional pages on the website http://last.fm after they accessed it for the first time. In between these visits, the user also searched on Google for lyrics and followed a link – this can happen because they were using tabbed browsing or had two instances of Firefox open to access multiple websites at the same time. This is confirmed by the session being different for both sets of URLs. moz_places only stores the unique URLs accessed, but combined with moz_historyvisits a full account of the user’s online history can be made. Every URL impression is stored in moz_historyvisits, so the number of entries will be considerably more if the user visits a URL more than once.

Figure 1- Part of moz_historyvisits table showing the user clicking on links

Figure 2 - Part of moz_places table showing the corresponding URLs in Figure 1

FORMHISTORY.SQLITE

Form history can provide useful information such as usernames, email addresses, postal addresses and search engine queries. Firefox stores this data in formhistory.sqlite which has a singular table called moz_formhistory. The fields can be found in the table below.

Field	Meaning
id	The table’s primary key.
fieldname	Stores the name of the field on the form
value	Stores the value the user entered on the form
timeused	Stores the number of times this value was submitted.
firstused	Stores the time the value was submitted for the first time in PRTime.
lastused	Stores the time the value was submitted for the last time in PRTime.

Google queries appear in here with the fieldname as “q”. Other possible searches will have fieldnames such as “query”, “search” and “search_terms”. Some web mail use forms to send email, so email subjects and email address recipients will be available too. Usernames and passwords to websites can be found in signons.sqlite, but the username and password fields are both stored encrypted.

References

Mozilla developer center pages on nsNavHistory.cpp, PRTime and Frecency.
Forensic analysis of the Firefox 3 Internet history and recovery of deleted SQLite records by Pereira, Murilo Tito.

Webscavator

08/06/2010 05:17PM

So I started my thesis 'officially' yesterday, after being told we all were allowed to proceed to masters following the exam board meeting. No actual results posted up yet, but hopefully I did well!

Unofficially, I've been working on my thesis for about 2 weeks, just making the skeleton application. I'm making a program called Webscavator (an amalgamation of website and excavator) and it will be a web application that visualizes web browser history. It accepts CSV files from a couple of common web history analyzer programs (so far Web Historian, Pasco and Net Analysis) and will eventually visualize them with timelines, graphs, pictures etc.

I've made it very easy to add other web history programs - you just have to add a class for that program and implement a processRow() method, which takes in a row of the CSV file, and returns the normalized output Webscavator expects. It was much easier to hard code each web history program converter like this than to come up with a generic CSV parser - each program produces very different files and it would get very messy. This also has the advantage that when I open source Webscavator after my thesis is done, other developers can quickly add in other web browser programs converters and even different kinds of parsers (i.e. not CSV).

If you want a sneaky peek at Webscavator then email me and I’ll give you the username and password to view the online current beta version of it. If anyone is feeling creative I’ve love a logo for it! I’m thinking along the lines of Indiana Jones meets the Internet, and also it’s written in Python, so maybe an adventurer/snake theme?

The next week or so I will be investigating the inner workings of browser history files, i.e. index.dat files for Internet Explorer, places.sqlite for Firefox, global_history.dat for Opera, history for Chrome and history.plist for Safari. There are some very detailed papers on IE, Firefox and Safari, but no one has written an academic paper on the forensics of Chrome and Opera history. I so desperately want to get published – is this an opportunity?? Expect a blog post about each one soon!

Gullane

24/05/2010 03:55PM

Gullane is a village about 20 miles east of Edinburgh which has an awesome beach called the Gullane Bents. It was quite a hassle getting to it when I went yesterday as I forgot the Edinburgh marathon was on so all the main roads to Gullane were shut. The beach is about a mile long and has lovely golden sand, with big rocks and rock pools at either end. It was surprisingly clean given the number of people and lack of bins.

Gullane clearly knows how to attract people like me: the village sports a German cafe/patisserie which does awesome cake and icecream, perfect after a long stroll on the beach! Their website informs me they also have a branch in Bruntsfield, Edinburgh - yay!

Tips to make a good CV

10/05/2010 02:50PM

I've been asked six times now to help people out in making their CVs better or helping them produce their first CV since leaving university. I quite enjoy doing it and hope my suggestions help people get further in the interview process.

I have uploaded a CV template that I use - feel free to use this for your CVs. It has lots of different sections, some of which won't apply to you, so just get rid of them. It's specifically designed for those (like me) who haven't had a 'proper' job yet, and rely heavily on things they did at university and possibly high school. There are a couple of things I think are very important when it comes to CVs:

Maximum two pages. A CV is a snapshot of your life and why you should be hired. An employer does not want your whole life history and will get bored if it is longer than 2 pages
Highlight the relevant bits. Employers often get hundreds of CVs and will only glance through each quickly. Make it easier for them by putting important parts in bold or a different colour (e.g. any awards you've won)
Only put in what is either relevant or impressive. If you have an entry that is neither, why do you want someone to read about it? You might have had 20 part time jobs during uni, but if only 2 are relevant to the job at hand, then just mention those two and talk more about them
Layout. Make sure the CV is not too compressed with information and it is neatly sectioned. Some employers are looking for very specific details, and if you organise your CV well they will be able to find what they are looking for. Use headings like "Education", "Work Experience" and "Skills"
Spell check! This document is supposed to represent you at your best, make sure everything is spelt right and makes sense
Add soft skills. Employers want to know what your soft skills are (team player, leadership, time management, communication, organisation etc) so add these to your work experience as things you've learnt. E.g. an IT retail job may mean you learnt to communicate effectively, and explain technical details to a non-technical audience.