Getting sporadic 404 Bad request and 429 Too Many Requests when posting bulk data

Have 10 IoT ESP32 devices which posts status every 30 minutes.

Sometime it works well, some other times the same log fails and returns with either HTTP error code 404 or 429.

From my end it looks exactly the same.

Would like to research how a bad request looks from Thingsspeak's end and why they are rejected.

Channel ID: 1023335.

thanks in advance.

Answer 1

[Vinod]

This is most likely with a badly constructed request to ThingSpeak. It will be difficult to debug because it is likely that some values of parameters the request is badly encoded. One way to debug this is to log to the serial monitor the exact request being made to ThingSpeak when you get a 404 or 429 response code.

Ideally you are not constructing the request yourself and are using the library here: https://github.com/mathworks/thingspeak-arduino to update ThingSpeak.

[avner gideobi]

Thank you Vinod.

Problem is this:

1. AFAIK, the library doesn't support bulk update.
2. the messages i am sending are identical as far as i can tell, while some of them are rejected.

for example, this request was rejected with ERROR: Unanticipatred response recived: HTTP/1.1 429 Too Many Requests:

ERROR BODY SENT: write_api_key=XXXXXXX&time_format=absolute&updates=2020-6-3T19:29:17-0000,,9104118,361,22,2020060101,,448,448,0.00,0.00,0,This is boot No. 22|2020-6-3T19:29:18-0000,,9104118,362,22,2020060101,,2990,2541,0.20,20.30,0,Configuration server is now: raw.githubusercontent.com|2020-6-3T19:29:19-0000,507,9104118,363,22,2020060101,,2993,3,0.20,20.30,0,Device found. Device name: 507|2020-6-3T19:29:20-0000,507,9104118,364,22,2020060101,,2997,3,0.20,20.30,0,Participating in 4 plans.|2020-6-3T19:29:21-0000,507,9104118,365,22,2020060101,,3013,16,0.10,20.30,0,Network configuration loaded and parsed succesfully|2020-6-3T19:29:22-0000,507,9104118,366,22,2020060101,,3021,8,0.20,20.30,0,This is device 507|2020-6-3T19:29:23-0000,507,9104118,367,22,2020060101,,3022,1,0.10,20.30,0,Wakeup caused by touchpad|2020-6-3T19:29:24-0000,507,9104118,368,22,2020060101,23.40,3046,24,0.10,20.30,0,Temperature: 23.40 Humidity: 58.00|2020-6-3T19:29:25-0000,507,9104118,369,22,2020060101,23.40,3047,1,0.00,20.30,0,Initialization Completed.|2020-6-3T19:29:26-0000,507,9104118,370,22,2020060101,23.40,3047,0,0.20,20.30,0,Going to wake in 1800|2020-6-3T19:29:27-0000,507,9104118,371,22,2020060101,23.40,3048,0,0.20,20.30,0,Going to sleep now for 1800 secs....This cycle took: 3048

[Vinod]

You're right, Avner. The library does not support bulk-update yet, but keep your eyes open for an update :)

Regarding the problem of 429s, are all devices posting to the same channel?

[avner gideobi]

Yes, all 20 devices report on the same channel.... Truly hope this is supported, otherwise pulling data for reports could be a huge pain....

here is how the dashbaord looks like:

https://docs.google.com/spreadsheets/d/1SyZzk1fjAfYyT4cHCbjSA_L9iymSBKd3CWMHx7ZhsJ0/edit?usp=sharing

[Christopher Stapels]

It is supported, but not suggested to have all 20 devices posting to the same channel, but it would be easy for them to have collisions, as you are seeing. To overcome this, you would have to scynchronize the clocks on all 20 devices.

Have a look at Error Codes , a 429 is too many requests. This means your devices are somethings posting within the 15 second required gap betweeen bulk updates.

The suggested model is one channel per device. You can then run an aggregation script with a time control to regularly place the data from all 20 sub channels into a 21 derived channel that contains all the data, assuing there are no overlapping timestamps.

Can you describe your method for pulling data? I think you could easily generate a MATLAB script that would do the effort for you from 20 seperate channels.

[avner gideobi]

Thank you Christopher for the clear answer.

It doesn't seem scalable to open a channel per any IoT device..

Reporting aggregation and configuration with different URLs and keys doesn't seem scaleable.

Thanks anyway,

[Vinod]

If you need to have 20 devices sending data to the same channel, I'd recommend each device update the channel no more frequently than once every (15 seconds between updates * 20 devices per channel = 300 sec) 3 minutes or so and they stagger their inputs 15 seconds apart.

[Christopher Stapels]

I would like to suggest that scalability is quite possible with ThingSpeak. The channels API gives you a lot of control.

You can use the Create Channel API call to create channels in real time from a device. You can store information in the metadata or tags section of a channel that allows you to group devices. You can write code in MATLAB analysis that scans your channels and returns the api keys for the devices that have the right metadata key or tag.Then you can selectivelty read those channels. Then in ThingSpeak, you can even schedule it to run automatically with a TimeControl. I can help you with pseudo code for each of those parts if you are interested.

I think this weatherstation example shows how to apportion a channel from a device.

[avner gideobi]

Thanks much Vinod.

Problem is these devices are sleeping and wake up occsionally, so I can't really control timing.