Parallel processing remote and server logs collection / scp

I would like to stream a file using HTTP endpoint / scp so that I can collect the logs from a remote server.

Is there a way ,parallel.internal.logging.enableClientLogging or parallel can help me connecting the input and output model. I have model where I use ugetdir to collect logs and then process the logs.

But now I have larger system where logs should be downloaded and processed.

I accept a HTTP POST request which will contain the following JSON in its body:

{ 
      event, 
      entity, 
      actor, 
      machines, 
      tests, 
      options 
} 

I'II call back the HTTP domain name mentioned in the actor, so that it will give back the JSON of all the log files generated for that job

Can you please suggest me how (

JobStorageLocation

) can do this ?

Thank you!

[Edric Ellis]

I'm not sure I've understood quite what's going on here. Are you asking for a way to have the workers write out files in JobStorageLocation, and then get the client to pick them up afterwards? If so, Parallel Computing Toolbox doesn't have any APIs to help with that. If the client and workers can see any sort of shared filesystem, your best bet is to put the files there.

[Life is Wonderful]

Thanks @Edric Ellis ,

Probably I did not explain well,I'II try again

I want MATLAB to read test results produced by NodeJS ,

your best bet is to put the files there ?

Question and Help: I need matlab pseudo code using below attributes from Nodejs to put the files to JobStorageLocation and let workers pick them afterwards

At the moment, I am looking how Matlab can run HTTP server with the following endpoint.

Accept a HTTP POST request which will contain the following JSON in its body:

    {       event,       entity,       actor,       machines,       tests,       options       } 

I am interested in the event (‘completed’) and entity (‘job’).

Then I'II call back the HTTP domain name mentioned in the actor with the following call:

URL = actor + /api/logs/:id

Where id will be options.id that received in the previous callback.

This will give back the JSON of all the log files generated for that job.

I can then generate the full path for any log that I want:

URL = actor + logs/:id/:path

I can then process the log and create final log.

Once I am are ready,

I should do a HTTP post request to: URL = actor + logs + :id + :name

With Nodejs support , the file should be streamed via the body

Thanks a lot!

[Edric Ellis]

I'm afraid I still don't understand where (i.e. on which host, caused by which process) the files are being generated. There is no portable way to store arbitrary files inside JobStorageLocation, the only documented API is to use properties of the Job and Task objects. If the data needs to be transferred from client to worker, you could read the files and put the resulting data structure into some property of the Job or Task; if the data needs to be transferred from worker to client, you could read the file(s) and return the value from your task's function. So, you could do something like this:

% Example sending JSON data to a worker for processing
jsonFile = 'somePathAtClient.json';
jsonData = jsondecode(fileread(jsonFile));
job = batch(@processJsonData, 1, {jsonData});
function out = processJsonData(jsonData)
    out = doStuff(jsonData);
end

Or, in the other direction:

job = batch(@returnJsonData, 1, {args..});
function out = returnJsonData(args..)
    % do stuff ...
    jsonFile = 'somePathOnTheWorker.json';
    jsonData = jsondecode(fileread(jsonFile));
    out = doStuff(jsonData);
end

I don't know if that helps.

[Life is Wonderful]

Hi Edric Ellis

Thanks a lot for the help.

I want to listen to requests by running an HTTP server with MATLAB.

The way is to create a server that listens to requests which can be done using MATLAB compiler SDK/ MATLAB Production server and I have to create API endpoints

Here I would like HTTP requests to trigger via MATLAB and capture responses.

Sample example

I found API matlab.net.http.xxx which is useful to construct HTTP POST. Can you please indicate full structure to get below fields ( I am missing -xxxx) elements

 event,
 entity,
 actor,
 machines,
 tests,
 options
 

Pseudo code sample

request = matlab.net.http.RequestMessage;
request.Method = matlab.net.http.RequestMethod.POST;
request.Body = matlab.net.http.MessageBody('username=myusername&password=xxxx');
request.Header = matlab.net.http.field.ContentTypeField('xxxxyyyyzzzz-UTF-format');
response = request.send('https://my_url/Edric_Ellis')

Cheers!

[Edric Ellis]

Sorry, I don't know anything about MATLAB Production Server or matlab.net.http. You might be better off either asking a separate question about that piece.

[Life is Wonderful]

Sure ! Thanks a lot . I will close the question