Ways to finesse HTTP HeaderField constraints?

Hey folks,

Webwrite isn't meeting my needs for POSTing data to a RESTful web service, so I decided to go with the lower level matlab.net.http.RequestMessage functionality, which sometimes works, but setting the Header with my authorization token sometimes fails depending on the token I get.

URL = 'https://api.tdameritrade.com/v1/accounts/1234567890/orders';
r = matlab.net.http.RequestMessage;
r.Method = 'Post';
r.Header(1).Name = "Authorization";
r.Header(1).Value = "Bearer " + token; % Token is long and changes periodically. **Usually** Matlab throws error

The problem is in setting the Header, which requires a field Name called "Authorization" and the Value of "Bearer " + token. The "token" value is very lengthy, around 1,200 characters (example at bottom of this post) and I have to get a new one every 30min to keep accessing this service. Sometimes Matlab is okay and accepts the token, but most of the tokens are rejected when I try to set the value, giving me an error like the one below:

The Matlab class HeaderField is clearly doing a validation routine on the Value and determining it is not valid.

Here's what I've tried:

1. Remove space after "Bearer", or using "Bearer%20" or "Bearer+". Matlab accepts this, but then web server returns "unauthorized"
2. I've noticed that if I take the first 1,144 characters in token then Matlab accepts (e.g. "Bearer " + token(1:1144), but of course the web server returns "unauthorized" since I'm not providing the full token

Any ideas for how to enter "Bearer " + token to avoid the above error?

Here's an example token (I changed one character just to be sure no one can use it):

token = "zgzMa4kOwPmVxbQ3tp0unIp4JIXv1Hzh7fzRs3JmQbTnPVFeXX7oF2pJH+3gL4KYEfXFIuiL7IKHOJRUdxe0HXnU0yN+Gn9cZUyU/eovZVyG4kVTqu47hyDnNkq0C1gShWhBtWcvI8uKjzHjGMJP8Ot2Gjhbw3/l6YfQe3q3PBQ+6wLsSV7gHBm406PukLdql7schTpvrqLi8h308G7OIVrTJB2oXgqfIVGGDYbQNuw6iGamxJUqOZ8Fv8PeRapG92iT8B1p1sW5LFq5Vpu6N2pFgkhoSilmW0aZ+aq03R9V8FYFsZ30/jrNqSxllf3WFELkVYlJlGNOvudzIzJwANcfcl4UzvDYpduN33lOarpIK3a17tRlsjIimtgiLhyxEYlUnVLiuwI5l1+9tkJ0HjC3asEL6kWj+ysW5gx36YWyb8tbhOz5I5wFkqGkcjZNJjUQ12Ou4+DVG/tCwhCHwJOR+x8imWbZqTcDwNRMKyQkn1CXSY463Ycxf3XE6boRBkK2OtAQ/m4Q276i26HhRGjuMFfDW1fzCrrXy23zERVAkAo9t100MQuG4LYrgoVi/JHHvlwCIoz+tGIgiaHeuLUSTR/PGs5RLwaXaMkRXixdeWNw5qOODbq9fsFHe0zBCweEkN8CK5Uxa3+FXv3OtlJUDZ8Pytum96pRkgDPvib1n2sDmtV0DgHc1q2zJqfitAcfS3lmIYPx7RcGBHJV/jn1Ojp6mcN9tu0C8S7u/Mk4gqV+dwkdLIICRGe3u2ASbsX6j16Nrbv+XHy9sxQRVh0eWpvLppM0ICBlSxrZ6qr/d6pYqn505fB6gICX0D+326pFZF/fgcyLnfGjCaHv9v5qMv0LTNooGYZ5BguM4IZGOCts4h+/zm3wGCchBczi3BWh/AXw+fNro1TOAkGDdzshZs6GyBoo52vziYR319E/pep9qDs0talrh1OJGnOOquBhnqY9ej2kzQdDtf3sBxQjSXn3JEIlAcszSCQbPDwwrr5sPDMWuhHsgbuU1+CIGOPEKiL0zi57fne5zTjAvMBov8/FWmCULZ+dEGRiBfdqK4OJMyZn8Kpcq+oN2JYSOBcE+c/6Y3XrOjW/8D1U+Hvb2sYhWk/f1hYh2AHDO2dwlbCoRtPipQ==212FD3x19z9sWBHDJACbC00B75E"

Note that I can set the header with webwrite and successfully use the RESTful service that way, but unfortunately the response when using webwrite is empty and I need the info that gets returned from the non-webwrite approach described above.

Thanks,

Paul S

[Ken]

Paul,

I have been spending the day searching for anything on JSON formats for endpoint conditional study orders. I use conditional studies all the time in TOS and must use them as well when using TDA endpoints.

This is because when an order is placed, I want to exit based on studies; either standard or my own custom studies resident on the TDA server. I cannot count on my home PC to exit trades based on studies. I'm sure you know why: power failures, internet service interruptions, PC failures, . . . I don't mind losing entry opportunities when bad things happen, but I sure don't want to be in a trade when bad things happen locally! We are all counting on TDA to have servers that are well maintained with lots of back-up power, redundant processing, virus protection, etc. That forces exit orders to be processed and placed on TDA's servers after entry orders. I want to do more than simple stop loss, trailing stop, or time stop orders.

Have you uncovered the formats for conditional orders based on studies for endpoint orders?

Ken

[Paul Shoemaker]

I will admit to doing this level of exit condition checking locally on my PC. There are a few reasons for this:

1. I have not discovered robust endpoints for making use of custom conditions and/or studies
2. While there is always the chance of a local machine failure (e.g. power, internet, hardware, etc), my software sends me emails on a regular basis so I'll know if it's down if I don't receive communications from it, and none of my trades/trading is so crucial as to threaten my account in a meaningful way should things go offline for a couple hours. Most of my positions are held on the order of multiple days to several weeks.

[Ken]

I suspected this.

The real advantage to robo-trading is entering a position on unexpected moves at all times of the day or night.

It seems necessary to engineer trades that have server exits that can be cancelled/changed by an endpoint preemption.

There are many shock moves . . . and this just might be the way to harvest them!

Happy trading!

Answer 1

[Paul Shoemaker]

Hey folks,

I think I may have fixed the issue. The trick seems to be creating a GenericHeader object, which does not validate supplied values. Then assign the GenericHeader field to the RequestMessage Header.

Here's how it looks:

URL = 'https://api.tdameritrade.com/v1/accounts/1234567890/orders';
r = matlab.net.http.RequestMessage;
r.Method = 'Post';
genericHeader = matlab.net.http.field.GenericField('Authorization',"Bearer " + token); % This bypasses Matlab value validation
r.Header = genericHeader; % Assign to RequestMessage. Fortunately, the RequestMessage class accepts generic header objects
r.Body(1).Data = order; % trade/order to POST
response = r.send(URL); % Works like a charm and output contains the info I need.

I hope this helps someone else out there.

Paul S

[Herman R]

Thanks!

[Joe R]

Works great on older version of matlab that will not let you use the "Authorization Bearer token" header with matlab.net.http.HeaderField.

[Nickolas LaSorte]

Paul,

Could you share more code for you TD Ameritrade API call?

Thanks

Nick

[Ken]

I cannot get proper response. I think it is a format problem with the order string. I am following JSON format in TDA examples. Could you show what your order string looks like?

[Ken]

I assume r.send takes care of urlencode. True?

[Paul Shoemaker]

Kenneth,

Happy to help!

TDA has several examples for various order types in the API developer guide. Examples here.

I'm attaching an example .mat file that contains a very simple order to buy three shares of Apple Inc. ("Buy 3 APPL"). The variable contained in the .mat file is called "orderExample."

Then, you can submit this order like so:

r = matlab.net.http.RequestMessage; % Create generic request object and populate below
URL = ['https://api.tdameritrade.com/v1/accounts/' ACCOUNT '/orders']; % Account is your TDA trading account number (char format)
r.Method = 'Post';
header = matlab.net.http.field.GenericField('Authorization',['Bearer ' ACCESS_TOKEN]); % must use generic header to bypass validation
r.Header = header; % Assign header to RequestMessage
r.Body(1).Data = orderExample; % Insert order into Data field (Matlab structure format)
response = r.send(URL); % Submit order. ****IMPORTANT**** This is REAL, not paper money, so only execute when ready to affect real-world brokerage transaction

[Ken]

Hi Paul,

I just noticed your answer now and it's past my bedtime! I see that I thought orderExample would be a JSON string defining the order! Obviously not as I see in your example.mat variables. I will set up a structure and then it should work. Thanks so much. TDA never responds to my emails!

Ken

[Ken]

Paul,

I seem to be setting part of the structure incorrectly. Here's my program:

order.orderStrategyType = 'SINGLE';

order.duration = 'DAY';

order.orderType = 'LIMIT';

order.price = 100;

order.orderLegCollection.instruction = 'BUY';

order.orderLegCollection.quantity = 1;

order.orderLegCollection.instrument.symbol = 'SPY';

order.orderLegCollection.instrument.assetType = 'EQUITY';

URL=['https://api.tdameritrade.com/v1/accounts/' accountNum '/orders'];

r = matlab.net.http.RequestMessage;

r.Method = 'POST';

genericHeader = matlab.net.http.field.GenericField('Authorization',['Bearer ' accessToken]); % This bypasses Matlab value validation

r.Header = genericHeader; % Assign to RequestMessage. Fortunately, the RequestMessage class accepts generic header objects

r.Body(1).Data = order; % trade/order to POST;

response = r.send(URL) % Works like a charm and output contains the info I need.

What am I still doing wrong?

Ken

[Ken]

Here is the response for both header and headerExample:

response =

ResponseMessage with properties:

StatusLine: 'HTTP/1.1 401 Unauthorized'

StatusCode: Unauthorized

Header: [1×15 matlab.net.http.HeaderField]

Body: [1×1 matlab.net.http.MessageBody]

Completed: 0

[Paul Shoemaker]

You're close, but there are a couple errors. First, you need to define the trading session ("normal") in the order. Secondly (and this one is prickly), you need to submit the orderLegCollection as a cell containing the structure.

order.orderStrategyType = 'SINGLE';
order.duration = 'DAY';
order.orderType = 'LIMIT';
order.price = 100;
order.orderLegCollection.instruction = 'BUY';
order.orderLegCollection.quantity = 1;
order.orderLegCollection.instrument.symbol = 'SPY';
order.orderLegCollection.instrument.assetType = 'EQUITY';

% Add session field
order.session = 'NORMAL';

% Put orderLegCollection into a cell format
order.orderLegCollection = {order.orderLegCollection};

[Ken]

Hi Paul,

Yeah! It works at long last!

order.orderStrategyType = 'SINGLE';

order.duration = 'DAY';

order.orderType = 'LIMIT';

order.price = 100;

order.orderLegCollection.instruction = 'BUY';

order.orderLegCollection.quantity = 1;

order.orderLegCollection.instrument.symbol = 'SPY';

order.orderLegCollection.instrument.assetType = 'EQUITY';

order.orderLegCollection = {order.orderLegCollection};

order.session = 'NORMAL';

URL=['https://api.tdameritrade.com/v1/accounts/' accountNum '/orders'];

r = matlab.net.http.RequestMessage;

r.Method = 'Post';

header = matlab.net.http.field.GenericField('Authorization',['Bearer ' accessToken]);

r.Header = header;

r.Body(1).Data = order;

response = r.send(URL)

Thanks so much for your help! I hope that this thread helps others who have also been banging their head against the wall. Do you have any suggestions for how I can get more enlightened on EndPoint communications using Matlab? Up 'til now I was just doing system calls to cURL and not getting very far.

Best regards and enjoy the super bowl. I'll be watching while writing code to talk to the mother ship!

Ken

[Paul Shoemaker]

Wish I could help, but I've figured it all out through trial and error in my own code.

Glad your above commands worked to place an order though. BTW, I don't have my code handy at the moment, but as I recall the line where you put orderLegCollection in a cell ({order.orderLegCollection}) is not needed if you have multiple legs in the collection, such as for an option credit spread or other multi-leg position. In those cases, omit this line of code.

[Ken]

Paul,

I see what you mean about trial and error! Now I can place an order, but to cancel, I need the order number. When I inspect the response structure, I see it shows an empty data field. So I have to ask, is there a trick to read the response structure to get the order number?

Ken

[Ken]

The frustration builds. I managed to overwrite my refresh token. Must get a new pair ot tokens. I must email tda to figure this one out. What a mess! Will I ever get this thing going?

[Paul Shoemaker]

You'll find the order number in the response returned from r.send(URL);

response is an object and one of its methods is "show." So, if you execute response.show you'll see the available fields. In our case, we want the location field, which contains the order number. How do you get this field? Use "methods(response)" to see the available methods for this object and you'll see there's one called "getFields." Use "help response.getFields" to see how to use it. See code below for how to put it to use.

response = r.send(URL); % response is a "responseMessage" object
location = response.getFields('location'); % Use getFields method to retrieve order info (i.e. "location")
% location is a structure with two fields: Name & Value. Order number will
% be at the tail end of location.Value.

[Ken]

Thanks. But looks like I'm done for the weekend without a refresh token. Has this ever happened to you. Is there a straight forward way to get back in business with a new refresh token? Can you feel my pain?

[Paul Shoemaker]

You shouldn't have to bother TDA for the refresh token or access token. You can login to the TDA developer API site, go to your apps, and grab the consumer key for the app you defined for use in Matlab. With your client ID, you can get your refresh token that'll last for 90 days. With the refresh token, you can get a valid access token that is then used for interacting with TDA api. This is all laid out in the TDA Developer API guide and help.

[Paul Shoemaker]

Go to TDA Developer API and login. Click "My Apps" at the top and identify your Matlab app you setup. Grab the client ID. Follow instructions here to get a refresh token. Then get access token using instructions here.

[Ken]

I followed the instructions, substituting my client ID for EXAMPLE in the string:

https://auth.tdameritrade.com/auth?response_type=code&redirect_uri=https%3A%2F%2F127.0.0.1&client_id=EXAMPLE%40AMER.OAUTHAP

but instead of getting 404 redirect, it says:

A third-party application may be attempting to make unauthorized access to your account. For help with your account, contact us. For information on securing your account, visit the TD Ameritrade Security page.

What could possibly be wrong?

[Ken]

Well, here I should have used localhost instead of 127.0.0.1 so now I get the code, but still having trouble after following all the details before SEND. Could they make this any easier?

[Ken]

Wow, I finally got SEND to work and received my new refresh token! What a job. For some reason I could not use Matlab's urldecode() function and had to go the website:

https://www.url-encode-decode.com/

The URL decode was my final stumbling block. I will keep a history cache of refresh tokens so I don't need to run the guantlet again! Thanks for your help Paul. Growth through pain!!!

[Paul Shoemaker]

Glad you got it working! There are definitely some hurdles when trying to do this type of stuff in Matlab and TDA documentation doesn't cater to our small niche crowd of Matlab uses.

Yes, definitely keep the token around. In my own code, I keep the refresh token and date in a Matlab file, my code then checks it periodically and asks me to get a new one when the token gets close to 90 days old. It helps me through the process too, reminding me of the steps, opening a web browser for me automatically, then I copy/paste the resulting address into a Matlab box and it parses the token and saves it to file. Whole process takes maybe 30sec once you've done it enough times, and that's only needed every 90days so it's not too bad.

[Ken]

For now I have a push button for UPDATE ACCESS TOKEN and another for UPDATE REFRESH TOKEN. Later I'll put them in a timer job so I won't think about tokens ever again. I now can see account status, view prices, place orders, and cancel orders. My next challenge will be conditional orders, OTO orders, OCO orders . . .

Then it's onward and upward to robotic trading! My computer becomes the screen zombie instead of me!

Your help was pivotal to this effort. I am grateful for your help.

[Paul Shoemaker]

Awesome! Really glad I could help. I've been using TDA API for about 10yrs now and have really enjoyed and made use of it in a major way for me. Hopefully, you have a similar experience.