Vector datatype support #2634

muskan124947 · 2025-03-18T17:57:12Z

The VECTOR data type is designed to store multi-dimensional arrays of numerical data, which are essential for various machine learning and data analysis applications. By introducing the VECTOR type, we aim to enhance the capabilities of SQL Server and Azure SQL Database, enabling users to efficiently store, retrieve, and manipulate vector data.

Vector Design doc -> https://microsoft.sharepoint.com/:w:/t/SQLAI-Projects/Ef632IF6mp1Mqj0OyKSrI1cBWNwxtM29eM31TZ3wUPgf4w?e=Ztgtq2&wdOrigin=LOOP-WEB.PREVIEW.NT&wduihid=ea3cf057-9d25-4ea5-bc4f-518a7ff343a7&web=1&ct=1741863974397

codecov · 2025-03-18T18:18:47Z

Codecov Report

Attention: Patch coverage is 22.53521% with 110 lines in your changes missing coverage. Please review.

Project coverage is 51.40%. Comparing base (a0ea23a) to head (7ede77d).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/main/java/microsoft/sql/Vector.java	0.00%	43 Missing ⚠️
...oft/sqlserver/jdbc/SQLServerPreparedStatement.java	4.54%	19 Missing and 2 partials ⚠️
...rc/main/java/com/microsoft/sqlserver/jdbc/dtv.java	5.55%	16 Missing and 1 partial ⚠️
...m/microsoft/sqlserver/jdbc/SQLServerResultSet.java	0.00%	11 Missing and 1 partial ⚠️
...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java	0.00%	7 Missing ⚠️
.../microsoft/sqlserver/jdbc/SQLServerConnection.java	75.00%	5 Missing ⚠️
...n/java/com/microsoft/sqlserver/jdbc/DataTypes.java	83.33%	3 Missing ⚠️
...n/java/com/microsoft/sqlserver/jdbc/Parameter.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2634      +/-   ##
============================================
- Coverage     51.59%   51.40%   -0.20%     
+ Complexity     3999     3994       -5     
============================================
  Files           147      148       +1     
  Lines         33706    33841     +135     
  Branches       5631     5648      +17     
============================================
+ Hits          17391    17396       +5     
- Misses        13866    13991     +125     
- Partials       2449     2454       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…o decode/encode data, along with additional features like dimensionType, and dimensionCount.

…roved constructor parameter positioning for clarity, removed VectorInputStream

…nsionCount for vector, fixed datatype and tdsType mappings along with setting correct typeDefinition in Parameter class.

…ector handling to ensure insertion of vector data.

…a, and measured performance for max-length vectors at varying scales — 100, 1K, 10K, 100K, and 1M records.

… performance testing.

…ull insertion correctly, added test case to validate null insertion and test vectorsupport feature enabled. Removed getVector, setVector and updated existing setObject() along with adding helper functions in Vector class.

…getObject, and setObject APIs.

…CopyForBatchInsert and added support for vector datatype in SQLServerBulkCSVFileRecord along with updating strategy for Vector type in dtv.java

muskan124947 · 2025-04-16T16:56:56Z

src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java

+                JDBCType.Category.TIMESTAMP, JDBCType.Category.NCHARACTER, JDBCType.Category.GUID)),
+
+        VECTOR(SSType.Category.VECTOR, EnumSet.of(JDBCType.Category.BINARY, JDBCType.Category.LONG_BINARY,
+                JDBCType.Category.CHARACTER, JDBCType.Category.LONG_CHARACTER, JDBCType.Category.BLOB, JDBCType.Category.VECTOR));    


Need to clean getter, setter and update conversions in DataTypes class

Can we add test cases to validate all data types in this list, including their conversions?

… for read, insert, bulkCopy, bulkCopyCSV operation and corrected typeInfo and Parameter.java along with adding test cases for BulkCopy

…ated insertion and retrieval of VECTOR data, and updated stored procedure and TVP definitions to use scale as dimensionType and precision as dimensionCount.

…CopyForBatchInsert along with updating getObject() when data is null

…d functions (UDFs); fixed previously failing test case.

… by returning normalised vector, global and local temp tables

…ta information for vector data type

… varbinary combinations -vector and varchar combinations

…ios along with adding test for setObject(getObject())

…peration is performed for mismatched source and destination tables

src/main/java/com/microsoft/sqlserver/jdbc/Column.java

divang · 2025-04-29T11:42:11Z

src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java

+        JDBCType.Category.TIME, JDBCType.Category.TIMESTAMP, JDBCType.Category.CHARACTER,
+        JDBCType.Category.LONG_CHARACTER, JDBCType.Category.NCHARACTER, JDBCType.Category.LONG_NCHARACTER,
+        JDBCType.Category.BINARY, JDBCType.Category.LONG_BINARY, JDBCType.Category.BLOB, JDBCType.Category.GUID,
+        JDBCType.Category.SQL_VARIANT, JDBCType.Category.VECTOR));


Can we add test cases to validate all data types in this list, including their conversions? It would also help to understand how this mapping is utilized in the execution flow.

divang · 2025-04-29T11:43:08Z

src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java

+        SSType.Category.CHARACTER, SSType.Category.LONG_CHARACTER, SSType.Category.NCHARACTER,
+        SSType.Category.LONG_NCHARACTER, SSType.Category.XML, SSType.Category.BINARY,
+        SSType.Category.LONG_BINARY, SSType.Category.UDT, SSType.Category.TIMESTAMP, SSType.Category.GUID,
+        SSType.Category.SQL_VARIANT, SSType.Category.VECTOR));


[As above] Can we add test cases to validate all data types in this list, including their conversions? It would also help to understand how this mapping is utilized in the execution flow.

divang · 2025-04-29T11:48:05Z

src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java

+                    }
+                }
+                break;
+


Can we write in more concise way:

case VECTOR:
byte[] bValue = (currentObject == null) ? null : (byte[]) currentObject;
writeShort((short) (bValue == null ? -1 : bValue.length));
if (bValue != null) {
writeBytes(bValue);
}
break;

divang · 2025-04-29T11:54:50Z

src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java

+            writeShort((short) (8 + (scale * precision))); //maxLength
+            byte scaleByte = (byte) (scale == 2 ? 0x01 : 0x00);
+            writeByte((byte) scaleByte); //dimension type
+            writeShort((short) -1); // actual len


It looks like the writeTVPColumnMetaData method contains duplicate code. If possible, consider moving the common logic to a utility method to improve reusability and maintainability.

divang · 2025-04-29T12:00:36Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCSVFileRecord.java

+                            }
+
+                            String vectorData = data[pair.getKey() - 1].trim();
+                            microsoft.sql.Vector.VectorDimensionType vectorDimensionType = microsoft.sql.Vector.getVectorDimensionType(cm.scale);


Is it possible to import the microsoft.sql.Vector package?

divang · 2025-04-29T12:11:03Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerDataTable.java

@@ -318,6 +318,21 @@ private void internalAddrow(JDBCType jdbcType, Object val, Object[] rowValues,
                    internalAddrow(internalJDBCType, val, rowValues, pair);
                    break;

+                case VECTOR:
+                    nValueLen = ((microsoft.sql.Vector) val).getActualLength();
+                    int scale = ((microsoft.sql.Vector) val).getScale() == 0x01 ? 2 : 4;


Could you move the hardcoded values 2 and 4 to constant variables with meaningful names?

divang · 2025-04-29T12:13:01Z

src/main/java/com/microsoft/sqlserver/jdbc/dtv.java

+                typeInfo.displaySize = typeInfo.maxLength;
+                typeInfo.ssType = SSType.VECTOR;
+                int scaleByte = tdsReader.readUnsignedByte(); // Read the dimension type (scale)
+                typeInfo.scale = (scaleByte == 0) ? 4 : 2;


Please use the constant that will be created as suggested in the comment above.

divang · 2025-04-29T12:25:22Z

src/main/java/microsoft/sql/Vector.java

+    public VectorDimensionType vectorType;
+    public int dimensionCount;
+
+    public float[] data;


Can those variables be made private?

divang · 2025-04-29T12:27:36Z

src/main/java/microsoft/sql/Vector.java

+            return null;
+        }
+        if (bytes.length < 8) {
+            throw new IllegalArgumentException("Byte array length must be at least 8 bytes.");


Can we add the below error message in SQLServerResource.java
{"R_InvalidVectorLength", "Byte array length must be at least "{0}" bytes."}

divang · 2025-04-29T12:27:47Z

src/main/java/microsoft/sql/Vector.java

+            throw new IllegalArgumentException("Byte array length must be at least 8 bytes.");
+        }
+        if (bytes.length % 4 != 0) {
+            throw new IllegalArgumentException("Byte array length must be a multiple of 4.");


Can we add the below error message in SQLServerResource.java
{"R_InvalidVectorLength", "Byte array length must be at least "{0}" bytes."}

Vector feature extension

7465df6

github-project-automation bot added this to MSSQL JDBC Mar 18, 2025

github-project-automation bot moved this to In progress in MSSQL JDBC Mar 18, 2025

Ananya2 and others added 13 commits March 21, 2025 20:38

Added initial skeleton code for VECTOR support in JDBC

96d6743

Added Vector support and strategy in DataTypes and dtv

9e3481d

Added changes which got removed from previous commit

23ece9f

Added support to read vector object using getObject

7faea8f

Updated read API for Vector support and created custom Vector class t…

7e81804

…o decode/encode data, along with additional features like dimensionType, and dimensionCount.

Updated getObject() API to skip 8-byte header in Vector class and imp…

ec86a56

…roved constructor parameter positioning for clarity, removed VectorInputStream

Added support for setVector() API, updated setObject() to handle dime…

3651596

…nsionCount for vector, fixed datatype and tdsType mappings along with setting correct typeDefinition in Parameter class.

Updated the setObject API to support vector insertion and fixed RPC V…

7ede77d

…ector handling to ensure insertion of vector data.

Validated insert and read for sample and max-length (1998) vector dat…

a801490

…a, and measured performance for max-length vectors at varying scales — 100, 1K, 10K, 100K, and 1M records.

Updated insert logic to make batch size configurable for single-batch…

88cb711

… performance testing.

Stored Procedure support for VECTOR Type using registerOutParameter, …

297fab9

…getObject, and setObject APIs.

Added support for SQLServerBulkCopy, prepared statement using useBulk…

46ab7d8

…CopyForBatchInsert and added support for vector datatype in SQLServerBulkCSVFileRecord along with updating strategy for Vector type in dtv.java

muskan124947 assigned Ananya2 and muskan124947 Apr 16, 2025

muskan124947 requested review from divang and machavan April 16, 2025 16:55

muskan124947 commented Apr 16, 2025

View reviewed changes

muskan124947 and others added 8 commits April 18, 2025 16:00

Updated to use scale as dimensionType and precision as dimensionCount…

5dff4e7

… for read, insert, bulkCopy, bulkCopyCSV operation and corrected typeInfo and Parameter.java along with adding test cases for BulkCopy

Added Table-Valued Parameter (TVP) support for the VECTOR type, valid…

dde875a

…ated insertion and retrieval of VECTOR data, and updated stored procedure and TVP definitions to use scale as dimensionType and precision as dimensionCount.

Added performance testing tests for bulkCopy and insert using useBulk…

76b868a

…CopyForBatchInsert along with updating getObject() when data is null

Added test case to validate vector as input and output in user-define…

69c7f6f

…d functions (UDFs); fixed previously failing test case.

Added vector as non-searchable and Validated select*into, UDF and SVF…

e768d4f

… by returning normalised vector, global and local temp tables

Added test to validate transaction rollback, database view and Metada…

7c0815a

…ta information for vector data type

Validate bulk copy scenarios with source and destination as -vector &…

87cadf0

… varbinary combinations -vector and varchar combinations

POC-BulkCopy usign CSV file and added tests to check different scenar…

cfbb2ed

…ios along with adding test for setObject(getObject())

Added test scenarios to validate failures when insert and bulk copy o…

67bc647

…peration is performed for mismatched source and destination tables

divang reviewed Apr 29, 2025

View reviewed changes

src/main/java/com/microsoft/sqlserver/jdbc/Column.java Show resolved Hide resolved

divang reviewed Apr 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vector datatype support #2634

Vector datatype support #2634

muskan124947 commented Mar 18, 2025 •

edited

Loading

codecov bot commented Mar 18, 2025 •

edited

Loading

muskan124947 Apr 16, 2025 •

edited

Loading

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

divang Apr 29, 2025

Vector datatype support #2634

Are you sure you want to change the base?

Vector datatype support #2634

Conversation

muskan124947 commented Mar 18, 2025 • edited Loading

codecov bot commented Mar 18, 2025 • edited Loading

Codecov Report

muskan124947 Apr 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muskan124947 commented Mar 18, 2025 •

edited

Loading

codecov bot commented Mar 18, 2025 •

edited

Loading

muskan124947 Apr 16, 2025 •

edited

Loading