syntax for call to java library function with byte[] (reference) parameter?

Sockets, again.

Where I am:

I have cast & typecast to make my data look like uint8.

I have socket & getInputStream & BufferedInputStream.

I have seen references to using java.nio.channels.Channels.newChannel(stream) instead & socket inputstreams.

I can get individual bytes from any of these with byteReturn = thing.read(). They are the uint version of my data sent. So connectivity & such are ok.

.

Issue:

Clearly I'd like to move to the more efficient looking

 thing.read(byte[]) forms.

methodsview('java.io.InputStream') ==> read(byte[]) & read(byte[],int,int)

K>> methodsview('java.net.SocketInputStream') ==> read(... the same

.

 javaByteArray = javaArray('java.lang.Byte',classStrLen);
 count = configS.in.read(javaByteArray);

No method 'read' with matching signature ...blah, blah

 javaNioByteBuffer = java.nio.ByteBuffer.allocate(1024);
 count = configS.channel.read(javaNioByteBuffer,0,classStrLen);

No method 'read' with matching signature found in class ...blah, blah

I've combed the Matlab help (clearly inadequately) & not found it.

Can anyone enlighten me to the syntax for a call to a java library file with a reference parameter? ie make int = read(byte[]) work or other information that will make it viable?

Or how to get data out of a java.nio.ByteBuffer if it does arrive?

I'm guessing that I can just write a java function that returns the array & call it, but it feels as if it should be possible with just a few library calls.

Thanks in anticipation.

.

PS: pnet, yes I know: 2 days wasted trying to get SDK to install & failed, so no compiler. (Also pnet docs seemed so much more complicated than typecast & write() --> read(byte[]) ...)

[Erik]

Bumping, since I have the same issue. Put in my words:

A Byte[] array can be created like this:

target_buffer = javaArray('java.lang.Byte', 10);
for i=1:length(target_buffer); target_buffer(i)=java.lang.Byte(0); end

Then Matlab holds a reference to the array, so if a Java function would modify it to return data I could access that from Matlab afterwards.

A byte[] array can be created, I think, like this:

target_buffer = java.util.Arrays.copyOf(int8(zeros(10,1)),10);

However, upon receiving the result Matlab converts it back to int8 instead of holding a reference. Thus it is useless to call

f = java.io.FileInputStream(filename); f.read(target_buffer, 0, 10);

because I don't have a reference to the temporary Java-buffer that was created upon sending the Matlab int8 target_buffer.

Like the original poster, I wonder if anyone is aware of a way of using methods like read(byte[]) that return by reference into a a Java primitive array? It would would be OK if it was Byte instead of byte, but for byte the autoboxing/autoconversion seems to destroy the functionality.

I found no solution in the Passing data to Java methods-documentation.

Answer 1

[Jacob Lynch August]

A SOLUTION

I haven't figured out how to pass the byte reference, but I have found a way to read a fixed number of bytes from a -Stream object. org.apache.commons.io.IOUtils comes with MATLAB (2017b, unsure when started) and toByteArray is a function that will pass the bytes directly out.

This bit of code works for me without errors.

% Importing Java classes/methods to simplify code
import java.io.FileInputStream
import java.util.zip.GZIPInputStream
import org.apache.commons.io.IOUtils.toByteArray
% User settings
TWO_KB              = 2048;%bytes;%"DATA_LENGTH = 2kB"
GZ_PATH_CHAR        = 'X:\dir\file.dat.gz';
% Ordered cleanup tasks are necessary for more complicated operations
ON_CLEANUP_TASKS    = cell(1,2)
% Start file input stream
FILE_INPUT_STREAM   = FileInputStream( GZ_PATH_CHAR ); 
ON_CLEANUP_TASKS{2} = onCleanup(@()FILE_INPUT_STREAM.close);% close AFTER Gzip
% Start gzip input stream
GZIP_INPUT_STREAM   = GZIPInputStream( FILE_INPUT_STREAM ); 
ON_CLEANUP_TASKS{1} = onCleanup(@()GZIP_INPUT_SRTEAM.close);% close first
% read three chunks of data
DATA_2KB_INT8_1 = toByteArray( GZIP_INPUT_STREAM, TWO_KB );
DATA_2KB_INT8_2 = toByteArray( GZIP_INPUT_STREAM, TWO_KB );
DATA_2KB_INT8_R = toByteArray( GZIP_INPUT_STREAM ); % remainder of data

I've tried this with GZIPInputStream and InflaterInputStream.

[David Laurenson]

Works really well, and is fast even for large files. Thanks for this!

Answer 2

[Jacob Lynch August]

Bob Gilmore, a software engineer for Mathworks at the time of this posting, indicates:

[S]ince MATLAB arrays are passed to Java by value (not
by reference), Java code that tries to change an input argument (like
FileInputStream.read()) won't work. The only way for you to get answers
back from Java is if the method you call actually *returns* a value.

So I think that you have three options available:
(1) Call read(no arguments) and get the data out a byte at a time, looping
until you get back a -1, and stringing them together in your M-File, or 
(2) Write a small "pure Java" wrapper that takes the file name (or the
FileInputStream) as an argument and returns the already-constructed byte 
array as its output. 
(3) Use the MATLAB file I/O methods.

He references a document I can't find on Mathworks anymore, but did find here.

Since MATLAB arrays are passed by value, any changes that a Java method 
makes to them will not be visible to your MATLAB code. If you need to 
access changes that a Java method makes to an array, then, rather than 
passing a MATLAB array, you should create and pass a Java array, which is 
a reference.

[Steven Lord]

That posting is over 16 years old and so may be out-of-date.

I'm not a Java expert so I'm not certain, but this page in the documentation may be relevant.

[Jacob Lynch August]

Nothing in Pass Data to Java Methods is useful to passing the byte reference to Java methods. Mathworks' Support contacted me yesterday to confirm that data is not passed by reference to Java.

Answer 3

[Jacob Lynch August]

What is a byte[] buf in MATLAB, and how do I pass it to read from the GZIP_INPUT_STREAM to retrieve a fixed number of bytes?

I'm suffering from the same problem. I have large files (+12GB over 4 GZ files) to dynamically stream in, dependent on the next header, sometimes up to 64MB chunks. I'm aware I could gunzip all the files, but this read in VERY slow.

Using the Java objects:

FILE_INPUT_STREAM = javaObject( 'java.io.FileInputStream', DATA_PATH_CHAR );
GZIP_INPUT_STREAM = javaObject( 'java.util.zip.GZIPInputStream', FILE_INPUT_STREAM );

I can use the standard read method one byte at a time. This is insignificantly faster than reading the files from the hard drive.

DATA_BYTES_UINT8 = zeros( DATA_LENGTH, 1, 'uint8' ); 
DATA_BYTES_UINT8(:) = arrayfun( @(b)read(GZIP_INPUT_STREAM), DATA_BYTES_UINT8 );

The fastest method, however, requires the most memory, which isn't feasible for these files.

BYTE_ARRAY_OUTPUT_STREAM = javaObject( 'java.io.ByteArrayOutputStream' ); 
INTERRUPTIBLE_STREAM_COPIER = com.mathworks.mlwidgets.io.InterruptibleStreamCopier.getInterruptibleStreamCopier();
copyStream( INTERRUPTIBLE_STREAM_COPIER, GZIP_INPUT_STREAM, BYTE_ARRAY_OUTPUT_STREAM ); 
DATA_BYTES_INT8 = toByteArray( BYTE_ARRAY_OUTPUT_STREAM );

The Oracle documentation describes that the read method for a GZIPInputStream accepts a byte [buffer], integer offset, and integer length of bytes to read. What is a byte[] buf in MATLAB, and how do I pass it to read from the GZIP_INPUT_STREAM to retrieve a fixed number of bytes? I'll document my failed attempts, in hopes it will draw more searches toward an eventual solution.

• I tried MATLAB Java Arrays of bytes.

JAVA_ARRAY = javaArray( 'java.lang.Byte', DATA_LENGTH, 1 );
BYTES_READ = read( GZIP_INPUT_STREAM, JAVA_ARRAY, 0, DATA_LENGTH ); 
% No method 'read' with matching signature found for class 'java.util.zip.GZIPInputStream'.

• I've tried Java Byte Buffers.

% BYTE_BUFFER = javaMethod( 'allocateDirect', 'java.nio.ByteBuffer', DATA_LENGTH ); % alt
BYTE_BUFFER = javaMethod( 'allocate', 'java.nio.ByteBuffer', DATA_LENGTH );
BYTES_READ = read( GZIP_INPUT_STREAM, BYTE_BUFFER, 0, DATA_LENGTH );
% No method 'read' with matching signature found for class 'java.util.zip.GZIPInputStream'.

• I've tried passing an array, knowing this would not work. Oddly enough, the BYTES_READ was NOT the same as DATA_LENGTH nor was it the current position in the GZIP file. Someone correct me, but I believe the property to describe the MATLAB array would "immutable?" Is there a way to make a (u)int8 array not immutable?

DATA_IN = zeros(DATA_LENGTH,1,'uint8');
BYTES_READ = read( GZIP_INPUT_STREAM, DATA_IN, 0, DATA_LENGTH ); 
% I tried passing 0 and DATA_LENGTH as java.lang.Integer and received: 
% No method 'read' with matching signature found for class 'java.util.zip.GZIPInputStream'.

• I've tried passing in a pointer.

% DATA_PTR = libpointer( 'int8Ptr', zeros( DATA_LENGTH, 1, 'int8' ) ); % alt
DATA_PTR = libpointer( 'uint8Ptr', DATA_IN ); 
BYTES_READ = javaMethod( 'read', GZIP_INPUT_STREAM, DATA_PTR, 0, DATA_LENGTH ); 
% No read method with appropriate signature exists in Java class java.util.zip.GZIPInputStream

• I've tried another Buffered input stream, and used the interruptible stream copier. While I received ONLY the remaining data, it was not bounded to DATA_LENGTH.

BUFFERED_INPUT_STREAM = javaObject( 'java.io.BufferedInputStream', GZIP_INPUT_STREAM, DATA_LENGTH );
BYTE_ARRAY_OUTPUT_STREAM = javaObject( 'java.io.ByteArrayOutputStream', DATA_LENGTH );
copyStream( INTERRUPTIBLE_STREAM_COPIER, BUFFERED_INPUT_STREAM, BYTE_ARRAY_OUTPUT_STREAM );
DATA_IN(:) = toByteArray( BYTE_ARRAY_OUTPUT_STREAM ); 
% In an assignment  A(:) = B, the number of elements in A and B must be the same.

Edit 1: cleared up confusing statements Edit 2: added links to GZIPInputStream Edit 3: asked if passing MATLAB array was useless because it is "immutable"

Answer 4

[Benjamin Davis]

This is quite an old post, but I wanted to document that I found a way to pass primitive data type array references using the reflection API and Collections. It is not pretty, but can be wrapped up into a function. The example here applies to a BufferedInputStream, but the technique can easily be adapted for any general function.

EDIT: I created an FEX submission to assist with the use of this technique.

https://www.mathworks.com/matlabcentral/fileexchange/74308-javamethodwrapper

With this submission one can simply do

fis = java.io.FileInputStream(filename)
methodObj = JavaMethodWrapper(fis, 'read(byte[], int, int)');
buf = zeros(1,1024, 'int8');
[count buf] = methodObj.invoke(fis, buf, int32(0), int32(1024));

PossiblyWrong provides a similar solution as well at:

https://possiblywrong.wordpress.com/2019/07/05/calling-java-from-matlab-with-pass-by-reference-arrays/

The key is that when using invoke() in the reflection API, an Object[] is passed in with the arguments to the function. It is not possible to construct an Object[] {byte[], int, int} directly, but it IS possible to do this through ArrayList.toArray(). The following function demonstrates the technique.

function out = read_BufferedInputStream(input_stream)

num_available = input_stream.available();

%short circuit out if none available
%do not try to read, or it will block
if num_available == 0
    out = '';
    return;
end

%save the reflection method object between calls
persistent m_read

if isempty(m_read)
    %build the reflection object
    %we are going to lookup BufferedInputStream.read(byte[], int, int)
    %using reflection API
    getMethod_args = javaArray('java.lang.Class',3);
    
    %this rather cryptic syntax is used for byte[]
    %since it is not possible to use java.lang.Byte[].TYPE
    %See:
    %https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.3.2
    byteArrayName = '[B';
    
    %these are the vararg list to getMethod
    getMethod_args(1) = java.lang.Class.forName(byteArrayName);
    getMethod_args(2) = java.lang.Integer.TYPE;
    getMethod_args(3) = java.lang.Integer.TYPE;
    
    %use reflection to get the method object
    m_read = input_stream.getClass().getMethod('read', getMethod_args);
end

%save the current buffer size and array of arguments to read()
persistent buf_size read_args
if isempty(buf_size)
    MIN_SIZE = 1024; %you could set this to whatever you want
    %make the buffer large enough to eat all available characters in the
    %stream
    buf_size = max(MIN_SIZE, num_available);
    
    %Note that this will fail:
    %   read_args = javaArray('java.lang.Object', 3)
    %   read_args(1) = zeros(1,buf_size,'int8')
    %So we instead use an ArrayList which will then be converted to
    %Object[] for the call to invoke.
    read_args = java.util.ArrayList();
    %this will become a byte[] in the ArrayList
    read_args.add(zeros(1,buf_size,'int8'));
    %arg for read start offset
    read_args.add(int32(0));
    %arg for read length
    read_args.add(int32(buf_size));
end

%Update the buffer to be larger if the input stream content grows
if num_available > buf_size
    buf_size = num_available;
    read_args.set(0, zeros(1,buf_size,'int8'));
    read_args.set(2, int32(buf_size));
end

%Here is the magic, when read_args is unpacked, the byte[] reference in the first
%element is passed
n_read = m_read.invoke(input_stream, read_args.toArray());

%so now we can go back to the original ArrayList and read out the contents
out = char(read_args.get(0));
out = out(:)'; %make row vector
out = out(1:n_read); %trim to indicated size

This can be demoed with:

jf = java.io.File('path\to\myfile.txt');
fis = java.io.FileInputStream(jf);
bis = java.io.BufferedInputStream(fis);
my_file_text = read_BufferedInputStream(bis);

[J. T. Holmi]

Well done Benjamin Davis! Your discovery of java.util.ArrayList's toArray() has opened various Java-usage doors to us! I post below an example function that converts MATLAB's variables to Java objects and classes and classnames.

% This uses java.util.ArrayList to convert varargin to (1) java.lang.Object
% array, and (2) java.lang.Class array. Output (1) can be used i.e. as
% second input to the Java method's invoke-call. It is noteworthy that this
% way any modifications to its arrays, i.e. primitive type arrays, can be
% collected and used at the MATLAB side after the invoke-call! Output (2)
% can be used i.e. to find a method with the matching signature! Output (3)
% can be used i.e. to quickly inspect what class types Java side has
% obtained in human-readable form. Read the reference [1] to understand how
% MATLAB types are mapped to Java types. Idea for this code was inspired by
% the reference [2].
% [1] https://www.mathworks.com/help/matlab/matlab_external/passing-data-to-java-methods.html
% [2] https://www.mathworks.com/matlabcentral/answers/66227-syntax-for-call-to-java-library-function-with-byte-reference-parameter#answer_416021
function [jObjects, jClasses, classnames] = java_objects_from_varargin(varargin),
    % Convert varargin to a java.lang.Object array via java.util.ArrayList
    jArrayList = java.util.ArrayList();
    for ii = 1:nargin,
        jArrayList.add(varargin{ii}); % Benefit from Java's autoboxing feature!
    end
    jObjects = jArrayList.toArray(); % Get constructed java.lang.Object array
    
    % Exit here if second or higher outputs are not used
    if nargout < 2, return; end
    % Get varargin classes by reflecting getClass-method of java.lang.Class
    % in order to force java.lang.Object. Direct call to getClass will fail
    % for primitive types.
    jC_Class = java.lang.Class.forName('java.lang.Class');
    jM_getClass = jC_Class.getMethod('getClass', []);
    jClasses = java.lang.reflect.Array.newInstance(jC_Class, nargin); % Backward-compatible, because javaArray cannot initialize zero length array in older MATLAB versions.
    for ii = 1:nargin,
        if ~isempty(jObjects(ii)), % Continue if not null
            % Although SCALAR primitive types (byte, short, int, long,
            % float, double, boolean, char in Java) are correctly autoboxed
            % by their corresponding Java object wrappers on the creation
            % of java.util.ArrayList, their classes cannot be reliably
            % obtained due subsequent unboxing/autoboxing and MATLAB
            % interference. Their classes must be manually set.
            if numel(varargin{ii}) == 1, % Continue only if scalar value
                matlab_class = {'logical', 'char', 'int8', 'uint8', 'int16', 'uint16', 'int32', 'uint32', 'int64', 'uint64', 'single', 'double'};
                java_class = {'Boolean', 'Character', 'Byte', 'Byte', 'Short', 'Short', 'Integer', 'Integer', 'Long', 'Long', 'Float', 'Double'};
                B_match = strcmp(matlab_class, class(varargin{ii}));
                if any(B_match),
                    java_wrapper_classname = ['java.lang.' java_class{B_match}];
                    jClasses(ii) = eval([java_wrapper_classname '.TYPE']); % Call wrapper's static TYPE field
                    continue; % Done
                end
            end
            % Arrays of primitive types require indirect call to getClass()
            jClasses(ii) = jM_getClass.invoke(jObjects(ii), []); % Subsequent unboxing/autoboxing event
        end
    end
    
    % Exit here if third or higher outputs are not used
    if nargout < 3, return; end
    classnames = cell(nargin, 1);
    for ii = 1:nargin,
        if ~isempty(jClasses(ii)), % Continue if not null
            classnames{ii} = char(jClasses(ii).getName());
        end
    end
end

Other advanced Java functions I have developed can be found here. I am hoping that they will inspire all us to benefit more from Java + MATLAB combo.

One more thing:

In my decompression code, I have used jByteBuffer = java.nio.ByteBuffer.allocate(int); in combination with jChannel = java.nio.channels.Channels.newChannel(InputStream); to replace any N_read = jStream.read(byte [], int, int); with N_read = jChannel.read(ByteBuffer);. This ByteBuffer object can then be converted back to bytes by buffer = jByteBuffer.array();. Its main benefit is that I do not need to use these ArrayList and Java reflection tricks, which may also have performance penalty, but I can instead directly call read of java.nio package, which has existed since Java 4! For instance, R2011a uses Java 6 and R2019b uses Java 8.