what savory currently knows about urls
February 10th, 2010You could go on and on and on collecting data about any particular URL. The amount of contextual data relevant to these things is near limitless, especially when they are involved in malware distribution or other hosting nastiness.
Savory includes somewhere around 100 or so different bits of information for each URL it records. More information is added when new, reputable, data sources are found on the net. savory’s design makes it easy to extend the list of information about any particular URL, so as more new sources of context are uncovered, they are easily added to new or existing URLs.
So what does savory currently know?
I picked out a recent URL that was reported as having been defaced recently. Here’s what savory will show you when you view the general information about the URL.
Remember that almost all of this information is retrieved automatically. There is very little user-interaction with savory aside from just seeing “what’s there”
The first tab here shows the URL in question and the value of the <title> tag at the time it was submitted (or if no title was found, a copy of the URL)
In addition to that are a list of notes and tags. Notes can be added by automated processes or human processes to log actions or info about the URL. Tags can be used to allow future narrowing of search results, or the building of various forms of metrics or statistics.
From here we can move on to the “meat and potatoes” of the URI context; the document.
The document contains the majority of the URL information and it can be big, so I’ll show it in pieces. First, the “document details”. This information is specific to the document itself, not to the URL which the document is associated with.
Each document has a unique ID; the Document ID. This is a UUID that can be used to directly pull the full document details out of savory. This ID is used in the context of the backend document database. Savory also uses a relational database which includes a “savory ID”. These two IDs are different, but stored in both databases so that tools that use one database can also access the other database if they need to.
Each time savory updates the details of a URL’s document, it creates a revision of the document. Using the drop down menu, admins can view the context of the URL as it has changed over time and new fields have been added, removed, or changed. It should be noted that the database can, and is, compressed periodically. The end result of this is that old revisions are lost. This is not a particularly big concern for us though because at this time we are not as concerned with document history as we are with what is current.
Moving on down the page we get to the URI context
So here is what savory knows. This document has 72 fields of context (we’ll see more in a second).
You can see some fields that may be particularly useful to you as a human, but we also include information that is more useful to robots, scripts, or other automated tools.
Moving on, here is more context.
Among other things, savory GeoIP maps the IP address of the URL in question, so you can map the location of the URL.
Another automated process that savory runs is page sourcing. Often we’re interested to know what sort of badware is actually hosted at the URL in question. Sometimes it’s a script, other time’s it’s an exe, or a malformed webpage, you get the idea.
We use resources that we get from page sourcing to re-inforce our training and awareness programs at work. We’ve found admins to be much more interested in seeing the actual crap that is used to infect their machines than to just be told about it.
Part of the page sourcing is hashing of the content, show in the next couple pics
Alright, I can here you asking yourself “isn’t that a little excessive? Do you really need that many hashes?”
You’d be surprised how often we come across a new tool that finds some new weird way of doing things. Most of the time you’ll see MD5 or some form of SHA used by these tools but then you’ll find a tool that doesn’t. We include all the hashes we can for the URL to ensure that we don’t have to worry about the next new-fangled hash that people start using.
Finally, a little bit more information after the hashes.
You get the idea.
savory includes one more stub of information on the document page. I will grant you that it is probably useless information. However, it makes for stimulating discussions between upper management during their multiple hour long meetings. So in that case, we refer to the following data as “printing money”; completely irrelevant to the Techs, only cared for by the Suits.
Management like pictures of things, especially ones that look alien like the above QR code. When you take a bunch of these and put them on a single page, people begin to ask questions and your applications get more air-time among the population. QR codes are almost entirely useless in the context of this application. But we did it to scratch an itch (ok I was just wasting time and having fun with technology I admit it)
But it does make for an interesting conversation starter.
