Parsing server log files are a common task in server administration REF. Historically R would not be well suited to this task and it would be better performed using a scripting language such as perl. However rex makes even this task easy to do and allows you to perform both the data cleaning and analysis in R!

Common server logs consist of space separated fields.

198.214.42.14 - - [21/Jul/1995:14:31:46 -0400] “GET /images/ HTTP/1.0” 200 17688

lahal.ksc.nasa.gov - - [24/Jul/1995:12:42:40 -0400] “GET /images/USA-logosmall.gif HTTP/1.0” 200 234

The logs used in this vignette come from two month’s worth of all HTTP requests to the NASA Kennedy Space Center WWW server in Florida and are freely available for use. 1

parsed <- scan('NASA.txt', what = "character", sep = "\n") %>%
  re_matches(
    rex(

      # Get the time of the request
      "[",
        capture(name = "time",
          except_any_of("]")
        ),
      "]",

      space, double_quote, "GET", space,

      # Get the filetype of the request if requesting a file
      maybe(
        non_spaces, ".",
        capture(name = 'filetype',
          except_some_of(space, ".", "?", double_quote)
        )
      )
    )
  ) %>%
  mutate(filetype = tolower(filetype),
         time = as.POSIXct(time, format="%d/%b/%Y:%H:%M:%S %z"))

This gives us a nicely formatted data frame of the time and filetypes of the requests.

time filetype
1995-07-21 14:31:46
1995-07-24 12:42:40 gif
1995-07-02 02:30:34 gif
1995-07-05 13:51:39
1995-07-10 23:11:49 gif
1995-07-15 11:27:49 mpg
1995-07-13 11:02:50 xbm
1995-07-23 09:11:06
1995-07-14 10:38:04 gif
1995-07-25 09:33:01 gif

We can also easily generate a histogram of the filetypes, or a plot of requests over time.

x_angle <- theme(axis.text.x = element_text(size = 7,
      hjust = 0,
      vjust = 1,
      angle = 310))

ggplot(na.omit(parsed)) + geom_histogram(aes(x=filetype)) + x_angle
ggplot(na.omit(parsed)) + geom_histogram(aes(x=time)) + ggtitle("Requests over time")