5. Advanced

5.1 The .dmat/ Format

Directory containing compressed data.

Contains:

• 0.data, 1.data, 2.data
• indexfile
• fieldnames

5.2 The VPL language

VPL (vmat processing language) is a home brewed mini-language in postfix notation. As of today, it is used is the {PRE,POST}FILTERING and PROCESSING sections of a .vmat file. It can handle reals as well as dates (format is: CYYMMDD, where C is 0 (1900-1999) or 1 (2000-2099). The language will not be extensively described here. For more info, you can look at plearn/vmat/VMatLanguage.*.

A VPL code snippet is always applied to the row of a VMatrix, and can only refer to data of that row. The result of the execution will be a vector, which is the execution stack at code termination.

When you use VPL in a PROCESSING section, each field you declare must have its associated fieldname declaration. The compiler will ensure that the size of the result vector and the number of declared fieldnames match. This doesn't apply in the filtering sections since the result is always a single value.

To declare a fieldname, use a colon with the name immediately after. To batch-declare fieldnames, use eg.:myfield:1:10. This will declare fields myfield1 up to myfield10.

There are two notations to refer to a field value: the @ symbol followed by the fieldname, or % followed by the field number.

To batch-copy fields, use the following syntax: [field1:fieldn] (fields can be in @ or % notation).

Here's a real-life example:

@lease_indicator 88 == 1 0 ifelse :lease_indicator
@rate_class 1 - 7 onehot :rate_class:0:6
@collision_deductible { 2->1; 4->2; 5->3; 6->4; 7->5; 
[8 8]->6; MISSING->0; OTHER->0 }
7 onehot :collision_deductible:0:6
@roadstar_indicator 89 == 1 0 ifelse :roadstar_indicator

5.3 The Metadata Directory

A metadata directory is associated with each dataset. For the datasets corresponding to a file (.amat, .pmat, .vmat) or directory (.dmat/) the associated metadata directory is obtained by appending .metadata/ to the file or directory name.

A metadata directory will typically contain the following cache directories to avoid recomputing costly things

• STATSCACHE/ contains cached statistics
• MODELCACHE/<classname>/ contains any pertinent cached data computed on this dataset by objects of class <classname>

In addition, the .metadata directory associated with a .vmat may contain

• precomputed.dmat/ or precomputed.pmat if the .vmat description specified <PRECOMPUTE>
• source.index containing row indexes in the source (resulting from <PREFILTER>, <POSTFILTER>, <SHUFFLE>)