StelsCSV JDBC Driver v5.2 Documentation

 

 

Contents 

 

Installation

Driver Classes

URL Syntax

Driver Properties

Database Schema

Data Types

Supported SQL Syntax

Connection Example

Driver Modes

User-defined SQL functions

Performance and other hints

 

Installation

 

Add the file csvdriver.jar to your classpath or extract the jar file in the directory of the application.

 

Driver Classes

 

Description

Classes

Driver class (JDBC API v1.0)

jstels.jdbc.csv.CsvDriver2

Data Source class (JDBC API v2.0)

jstels.jdbc.csv.CsvDataSource2

Connection Pool Data Source class (JDBC API v2.0)

jstels.jdbc.csv.CsvConnectionPoolDataSource2

 

URL Syntax

 

The connection URL is jdbc:jstels:csv:csvdir, where csvdir may be the following:

 

Note: a path to some particular CSV/text file can be specified directly in a SQL query, e.g.:

 

Driver Properties

 

The driver supports a number of parameters that change default behavior of the driver.

These properties are the following:

 

charset is used to specify a different than the default charset encoding for a CSV/text file (Default is the JVM default charset

 

commentLine is used to specify a string denoting comment line (By default is not-defined)

 

dateFormat is used to specify a format for date/time values. A value of this property is a sequence of date/time formats separated by the '|' character, e.g: "dd.MM.yy | dd.MM | dd". (Default is "yyyy-MM-dd HH:mm:ss.SSS | yyyy-MM-dd HH:mm:ss | yyyy-MM-dd |  HH:mm:ss.SSS |  HH:mm:ss"). For more details about date/time format used in the driver please see the chapter "time format syntax" in the java.util.SimpleDateFormat class documentation. 

 

defaultColumnType is used to specify the default data type for columns for a CSV/text file (Default is "Varchar"). 

 

decimalFormatInput,  decimalFormatOutput are used to specify input and output formats for floating point values in a CSV/text file. For instance, you can use these formats to specify currency values, e.g.: "###,###.##$". For more information about patterns used in "decimalFormat" please see the documentation for java.text.DecimalFormat class.

 

dbInMemory, dbPath are used to set a driver mode. For more details see driver modes.

 

fileExtension is used to specify a different file extension (Default is ".txt")

If extension is set to ".txt", then both "myTable.txt" and myTable are valid.

Note: You should stick to one of these table naming types in the SQL queries, i.e. either "myTable.txt" or myTable.

 

ignoreCase. The property sets the case sensitivity for text columns. By default the driver are not case sensitive, i.e. the default value for this property is true.

 

rowDelimiter is used to specify a custom row delimiter for a CSV/text file. (Default is a line separator specified by JVM environment variable "line.separator", e.g. for Windows - "\r\n", for Linux - "\r")

 

schema is used to specify a path to the schema file. It can be absolute or relative to a CSV directory path, e.g.: "c:/schemas/schema1.xml" or "schemas/schema2.xml". (Default is "schema.xml")

 

separator is used to specify a different column separator (Default is '\t' (tab))

 

suppressHeaders is used to specify if the first line contains column header information (Default is false; column header are on first line).

 

watchFileModifications tells the driver to start an additional thread that checks regularly (every second once by default) if the file was modified by some external process. If the file was modified, the driver refreshes it in the cache. (Default value is false). This property is recommended, if your files are modified by an external program / process and you want the changes to be immediately available to the driver. Also, use the property "checkPeriod" to set regularity of the file check. By default it is 1000 (1000 ms)

 

 

Advanced Properties

 

escapeEOLInQuotes is used to escape row delimiters within double quotes. (Default is false).

 

escapeSeparatorInQuotes is used to escape column separators within double quotes. (Default is true).

 

logPath is used to set a file path to the log file.

 

emptyStringAsNull. If 'emptyStringAsNull' is set to 'true', empty strings (i.e. "") are treated as NULL values. (By default is true).

 

nullString is a string value that is treated as the NULL value. Not case sensitive. (By default is "NULL").

 

paddingChar - padding char for fixed-length files. (By default is space).

 

preSQL is used to specify a path to a SQL script that will be executed after creating a driver connection, e.g. c:/sql_script.txt. It may be useful for creating table indexes and other preparations.

 

singletonConnection. If "singletonConnection" is set to "true", the driver creates only one java.sql.Connection instance for each unique JDBC URL and reuses that connection for all threads. It allows the driver to increase up performance and reduce memory usage in multithreading environments like web/application servers Tomcat, GlassFish, WebSphere, etc. For example JDBC URL with this property may look like the following: jdbc:jstels:csv:c:/mydir/csvfiles?property1=value1&property2=value2&singletonConnection=true. Default value is false.

 

trimBlanks. If 'trimBlanks' is set to 'true', the driver trims leading and trailing spaces for string values when reading a text file. (By default is true).

 

useWebParam is used to specify the name of the web parameter that will be used to pass a CSV/text file name to the dynamic server page. For instance, if you specify the following value: 'tablename', the driver will access server page using the following HTTP URL:

http://www.site.com/out.jsp?tablename=sometable (where 'sometable' is the table specified in the SQL query or the schema file)
Note: If you specify driver properties directly in the driver URL and server page has its own parameters as well, you should separate them with '??':

jdbc:jstels:csv:http://www.site.com/out.jsp?param1=value1??useWebParam=tablename&suppressHeaders=true

 

quoteString is used to enable/disable writing of double quotes for string values while inserting and updating records in a CSV/text file (Default is true).
 

To set the driver properties, you can use one of the following ways:

1)  using Properties class:

java.util.Properties props = new java.util.Properties();
 
props.put("separator", "|");              // separator is a bar
props.put("suppressHeaders", "false");    // column headers are on the first line
props.put("fileExtension", ".txt");       // file extension is .txt
props.put("charset", "ISO-8859-2");       // file encoding is "ISO-8859-2"
props.put("commentLine", "--");           // string denoting comment line is "--"
// date/time format
props.put("dateFormat", "yyyy-MM-dd HH:mm | dd/MM/yyyy");       
 
Connection conn = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles", props);
 
2) using a data source class jstels.jdbc.csv.CsvDataSource2 class:
 
CsvDataSource2 csvDS = new CsvDataSource2();
 
csvDS.setPath("c:/csvfiles");        // path to the CSV directory
csvDS.setSeparator("|");             // separator is a bar
csvDS.setSuppressHeaders(false);     // column headers are on the first line
csvDS.setFileExtension(".txt");      // file extension is .txt
 
Connection conn = csvDS.getConnection();
 
3) appending the properties to the URL (multiple properties should be separated by '&' or '!' character):
 
Connection conn = 
DriverManager.getConnection(
"jdbc:jstels:csv:c:/csvfiles?suppressHeaders=true&dateFormat=yyyy-MM-dd HH:mm | dd/MM/yyyy&dbInMemory=false");
 
4) using local properties specified in the schema for some particular file(s).
 
 

 

Database Schema

 

The database schema file is called "schema.xml". It is intended to specify data types and local properties for your CSV/text files. By default it should be located in the same directory where CSV/text files are contained. To specify another path use the driver property "schema".

 

The schema has the following format:

<schema>

  <table name = "my_table1.txt">

    <column name = "my_field1" type = "Integer"/>

    <!-- this tag assigns the SQL data type to the column by its name in the text file-->

    <column pos = "2" type = "Integer"/>

    <!-- this tag assigns the SQL data type to the column by its position in the text file-->

    <column name = "my_field3" pos = "3" type = "Integer"/>

    <!-- if you do not use the column header in the text file, you should set column names by using both the 'pos' and the 'name' attributes-->

  </table>

 

  <!-- you can also use file templates, if your files have the same format-->

  <table name = "*.csv">

    <!-- all files with the template "*.csv" -->

    <!-- the wildcard '*' denotes any string of zero or more characters -->

    <column name = "col1" type = "Varchar"/>

    <column name = "col2" type = "Integer"/>

  </table>

 

  <table name = "file????.*">

    <!-- all files with the template "file????.*" -->

    <!-- the wildcard '?' denotes any single character -->

    <column name = "id" type = "Integer"/>

    <column name = "descr" type = "Varchar" size = "50"/>

    <column name = "num" type = "Decimal" size = "15" decimalCount = "2"/>

    <!-- the 'size' attribute specifies the maximum number of characters for the VARCHAR type or total number of digits that can be stored for the DECIMAL type -->

    <!-- the 'decimalCount' attribute specifies the maximum number of digits that can be stored to the right of the decimal separator -->

  </table>

 

  <!-- absolute path to CSV/text files with the template '*' " -->

  <table name = "c:/csvfiles/*.txt">

    <column name = "id" type = "Integer"/>

    <column name = "name" type = "Varchar" size = "50"/>

  </table>

 

  <!-- absolute path to a particular CSV/text file" -->

  <table name = "zip://c:/csvfiles.zip/dir/test.txt" constraint = "PRIMARY KEY(product_id, region_id), UNIQUE (min_price), CHECK min_price > 100">

    <!-- the 'constraint' attribute specifies a table constraint like a primary key, unique value, etc -->

    <column name = "product_id" type = "Integer"/>

    <column name = "region_id" type = "Integer"/>

    <column name = "min_price" type = "Integer"/>

    <column name = "max_price" type = "Integer"/>

  </table>

 

  <!-- you can also set local properties for each table by using attributes of the same name " -->

  <table name = "my_table2.txt" charset = "ISO-8859-2" suppressHeaders = "true" commentLine = "--" separator = "," dateFormat = "dd/MM/yyyy">

    <column name = "my_field1" pos = "1" type = "Integer"/>

    <column name = "my_field2" pos = "2" type = "Integer"/>

    <column name = "my_field3" pos = "3" type = "Date"/>

  </table>

 

  <!-- If you do not use the column header in a text file ("suppressHeaders" is true), you should set column names by using both the 'pos' and the 'name' attributes -->

  <table name = "file_without_header.txt" suppressHeaders = "true">

    <column name = "my_field1" pos = "1" type = "Integer"/>

    <column name = "my_field2" pos = "2" type = "Integer"/>

    <column name = "my_field3" pos = "3" type = "Date"/>

  </table>

 

  <!-- The driver also supports the fixed-length files. For this, you should set the "fixed" value for the "separator" parameter. -->

  <table name =" fixed_length_file.txt" separator = "fixed">

    <column name = "id" begin = "1" end = "5" type = "Integer"/>

    <!-- The "begin" attribute sets the initial position of a column in a text file and the "end" attribute sets its final position. -->

    <column name = "name" begin = "6" end = "20" type = "Varchar"/>

    <column name = "birthdate" begin = "21" end = "28" type = "Date"/>

  </table>

 

</schema>

 

Notes:

 

Data Types

 

The driver supports the following data types:

 

Data Type

JDBC returned type

(java.sql.Types.*)

Java class used

AUTOINCREMENT

java.sql.Types.BIGINT

java.lang.Long

Int, Integer, Tinyint, Smallint, SHORT

java.sql.Types.INTEGER

java.lang.Integer

Long, Bigint

java.sql.Types.BIGINT

java.lang.Long

Float, Real

java.sql.Types.FLOAT

java.lang.Float

Double

java.sql.Types.DOUBLE

java.lang.Double

BIGDECIMAL, DECIMAL, NUMERIC (recommended for storing currency values)

java.sql.Types.DECIMAL

java.math.BigDecimal

String, Char, Varchar

java.sql.Types.VARCHAR

java.lang.String

Datetime, Date, Time, Timestamp

java.sql.Types.TIMESTAMP

java.util.Date

BOOLEAN, LOGICAL, BIT

java.sql.Types.BOOLEAN

java.lang.Boolean

 

Notes:

Supported SQL Syntax

 

StelsCSV version 5.x uses H2 database as an SQL engine and supports the most part of ANSI/ISO SQL grammar like SELECT, INSERT, UPDATE, DELETE and CREATE statements.

 

An SQL query must meet the following conditions:

Query examples:

// ---- SELECT queries ---

SELECT SUM(a) AS col1, MAX(b) / MAX(c) AS col2 FROM "test.txt" GROUP BY a HAVING AVG(a) > 30;

SELECT name FROM "salesreps.txt" WHERE (rep_office IN ( 22, 11, 12 ))  OR (manager IS NULL AND hire_date >= PARSEDATETIME('01-05-2002','dd-MM-yyyy') OR (sales > quota AND NOT sales > 600000.0);

SELECT city, target, sales FROM "offices.txt" WHERE region = 'Eastern' AND sales > target ORDER BY city;

 

// ---- SELECT queries with join ----

SELECT * FROM "prices.txt" ps JOIN regions regs ON ps.regionid = regs.id JOIN "products.txt" prod ON prod.prodid = ps.prodid;

SELECT * FROM "prices.txt" ps, "products.txt" prod WHERE prod.prodid = ps.prodid;

 

// ---- INSERT, UPDATE and DELETE commands ----

INSERT INTO "salesreps.txt" (name, age, empl_num, sales, title) VALUES ('Henry Smith', 35, 111, NULL, 'Sales Mgr');

UPDATE "customers.txt" SET credit_limit = 50000.00 WHERE company = 'Acme Mfg.';

DELETE FROM "salesreps.txt" WHERE NAME LIKE 'Henry%';

 

// ---- CREATE TABLE command ----

CREATE TABLE "new_table.txt" (int_col INT, long_col LONG, float_col REAL, double_col DOUBLE, str_col VARCHAR(20), date_col DATETIME, bool_col BOOLEAN, num_col DECIMAL(15,2));
 

// ---- CREATE INDEX command ----

CREATE INDEX i_1 ON "new_table.txt" (int_col);

 

 

See also:

 
Connection Example
 

This example code shows how the driver is used. You can download it here.

import java.sql.*;
 
public class DriverTest
{
  public static void main(String[] args)
  {
    try
    {
      // load the driver into memory
      Class.forName("jstels.jdbc.csv.CsvDriver2");
 
      // create a connection. The first command line parameter is assumed to
      // be the directory in which the .csv files are held
      Connection conn = DriverManager.getConnection("jdbc:jstels:csv:" + args[0] );
 
      // create a Statement object to execute the query with
      Statement stmt = conn.createStatement();
 
      // execute a query
      ResultSet rs = stmt.executeQuery("SELECT * FROM \"test.txt\"");
 
      // read the data and put it to the console

      for (int j = 1; j <= rs.getMetaData().getColumnCount(); j++) {

        System.out.print(rs.getMetaData().getColumnName(j) + "\t");

      }

      System.out.println();

 

      while (rs.next())
      {
            for(int j=1; j <= rs.getMetaData().getColumnCount(); j++){
                   System.out.print(rs.getObject(j)+ "\t");
            }
            System.out.println();
      }
 
      // close the objects
      rs.close();
      stmt.close();
      conn.close();
    }
    catch(Exception e)
    {
      e.printStackTrace();
    }
  }
}

 

 

Driver Modes

 

The driver works in the following way: it loads data from files to an intermediate database (further referred to as "synchrobase") that is used for running SQL queries and synchronizing changes between this database and external CSV/text files.
 


The driver can work in the following three modes:

1) With a temporary synchrobase in RAM. The synchrobase is created in RAM and is removed from it after the connection is closed. It is a default mode.

2) With a temporary synchrobase on the hard drive. The synchrobase is created on the HDD every time a connection is opened and deleted after it’s closed. To use this mode, set the driver property dbInMemory to false.

Connection conn = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles?dbInMemory=false&tempPath=c:/tempfiles");


3) With a persistent synchrobase on the hard drive. The synchrobase is created just once and is re-used afterwards. To use this mode set the property dbPath to the file path where is synchrobase will be stored.

Connection conn = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles?dbPath=c:/synchrobases/syncro_db_name&tempPath=c:/tempfiles ");


In the first mode, all synchrobase data are stored in the system RAM, which ensures maximum performance. Keep in mind that Java Virtual Machine must have enough free memory allotted for storing large tables (use -Xms and -Xmx JVM options). You can also use the DROP TABLE <table name> FROM CACHE command to force tables to be removed from the cache.
 

In the second case, a temporary synchrobase is created on the hard drive. This mode is used for processing large files, since it uses a minimum of RAM. The synchrobase is deleted after the connection is closed

In the third mode, the synchrobase is created once and is reused afterwards. This is the optimal mode, but it requires additional files to store a synchrobase on the hard drive.
 

Note: if your files are modified by an external program / process and you want the changes to be immediately available to the driver, consider using watchFileModifications property.
 

There are also some properties for configuring these modes:
 

tempPath - directory where temporary files will be created (by default it is a OS temporary directory, specified by JVM environment variable "java.io.tmpdir"). It is recommended to set your own value.

 

Example:

Properties props = new java.util.Properties();

props.setProperty("dbInMemory", "false");         // switch to the second mode (a temporary synchrobase on the hard drive)
props.setProperty("tempPath", "c:/temp");     
Connection conn = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles", props);

 

 

User-defined SQL functions

 

You can use your own SQL functions in the driver. To use this feature, you should do the following: 

 

1) Create a static method that will act as an SQL function. Mind that:

For example:

package my_pack;

public class MyFuncs{

// user-defined SQL function that formats the given argument into

// a date/time string with specified format

public static String format_date(java.util.Date d, String format) {
    // process the null values

    if (d == null || format == null)
      return null;
    java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat(format);
   
// return a function result with java.lang.String type

    return sdf.format(d);
}
}

 

2) Register the function by executing CREATE ALIAS ... FOR command. Use an unique name for each function.

For example:

CREATE ALIAS IF NOT EXISTS format_date FOR "my_pack.MyFuncs.format_date(java.util.Date, java.lang.String)"

 

Also, you can use the driver property function:<my_func>.

For example:

Properties props = new java.util.Properties();
props.put("function:formate_date","my_pack.MyFuncs.format_date");
Connection conn = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles",props);
// or append this property to the URL
Connection conn2 = DriverManager.getConnection("jdbc:jstels:csv:c:/csvfiles"
 + "?function:formate_date=my_pack.MyFuncs.format_date");

 

3) Call the function in an SQL query

For example:

Statement st = conn.createStatement();

st.execute("SELECT format_date(date_column, 'yyyy-MM-dd') FROM \"test.txt\"");

 

 

Performance and other hints

 

 

 

[HOME]   [TOP]