Almost since it’s inception, Kismet has had a handful of log types that are generated: nettxt, netxml, gpsxml, and so on. These logs had to be correlated - to get detailed location info about a network, you had to also parse the gpsxml and correlate by BSSID, to get packet information related to an event time you had to process a pcap file and extract the packets.
With the massive changes to the new object model in git-master, many of these logs either became defunct, showed major flaws and shortcomings under the new system, or were just finally ripe to be replaced entirely.
Flaws with the old log formats
There are any number of problems, or at least, limitations, which come with the old log formats:
- Many of the logs are only generated intermittently, making it difficult for tools to read data from them. The XML based logs are only generated at fixed intervals.
- Some logs cannot be read while Kismet is running at all: the state of the log was either non-deterministic or the file was incomplete.
- Some log formats are very sensitive to errors during creation, such as the gpsxml, which could easily become invalid and required special care to parse.
- Parsing XML is terrible in pretty much all of its forms.
- Correlation across log files was a pain, if not impossible. Very few tools, if any, correlated the GPS and network files.
- Anything logged to a file became inaccessible to Kismet; there was no reasonable capability for reading historic data from logs.
- Streaming logs could become corrupt. The gpsxml file was particularly vulnerable to ending without proper closing XML tags when running on battery powered systems.
The new Kismetdb log
The new log system attempts to address most, if not all, of these problems, and more.
The Kismetdb log file is based on sqlite3. A Kismetdb log is a single file containing everything Kismet could collect while it was running; instead of multiple logs per run which have to be correlated, all the data is in a single file, with searchable normalized fields.
Several basic data types are logged:
- Device data, as JSON objects. These records are basically equivalent to the old netxml and nettxt data, and can be trivially transformed to text, reports, or even XML, and can trivially be decoded as native objects in almost any language.
- Packet data, as original frame format. Every packet has a low level link type, which may contain the packet or may contain optional metadata not actually part of the packet format. Kismet now logs the original packet as seen from the data source, preserving this metadata (such as the antenna the signal was received on). These records can be easily transformed into traditional pcap files of several formats. By preserving the original packet with no changes, all possible metadata is retained for processing as well as display in tools such as Wireshark - the original packet headers (typically radiotap) are kept unmodified, including all antenna reception headers, encoding headeres, and similar; as the radiotap standard expands to encompass larger arrays of MIMO antennas, the data will be preserved in the kismetdb log.
- Packet location data. Replacing the old gpsxml format, the GPS location is now stored as normalized fields with each packet. It is no longer necessary to parse fragmented XML files and attempt to correlate data between them.
- Text messages and server output normally shown on the server console are included, with time and location data.
- WIDS alerts are logged with the event, location, and a full JSON record of the alert object.
- Datasource records, with full capture definitions of the original datasources and statistics.
- Arbitrary data objects. Not every data source generates traditional packets; sources may log JSON objects tagged with location and time.
- Sapshot data such as channel utilization and system health over time.
REST API access to historical packets
Combining a database style log with the new REST interface lets Kismet provide historical data on demand, giving a whole new degree of contextual access to packet data.
With all packets stored in a random-access database file, Kismet is able to provide dynamic pcapng streams, including:
- All packets from a specific datasource.
- All packets from a specific device.
- Packets within a time window.
- Packets within a geographic area.
- Packets with a minimum (or maximum) signal level.
- Packets within a range of frequencies.
- Packet size windows.
The filters can be combined; for instance:
- Fetch all packets 30 seconds before and after an alert
- Fetch packets with a stronger signal than -30dB
- Fetch packets within 500 feet of a target. From a specific BSSID. With signal greater than -20db. Within the past hour.
- Fetch all packets from a BSSID and feed them to an external tool, like Aircrack
More to come
Over time the historic API will grow to encompass other data, but the initial work for packets is available today!