Bambu Lab MQTT Limitations

Announcements
1

Dear Bambu Lab Users,

Last week we received an abnormal connection alert from the MQTT service, which indicated that a small number of Client accounts had generated a large number of failed subscription actions in a short period of time. After a detailed analysis, we found that more than 10 user accounts had established over 2000 MQTT Client connections in client_id starting with different “nodered_”, with the most accounts having over 1000 simultaneous connections. While maintaining a high number of connections, these Clients also repeatedly disconnected/reconnected/subscribed in cycles of nearly 1 minute. In the following days, these abnormal Clients and accounts maintained a daily increase of over 2000 connections.

Through further research, we suspect that such abnormal connections come from Home Assitant applications developed based on NodeRed low-code tools, which are developed by unofficial third-party organizations or individuals. Due to our limited understanding of NodeRed’s technology, we can only speculate that there are bugs in the MQTT connection logic, which led to the occurrence of the exception. As we are unsure of the specific application that caused the exception, we cannot directly notify the relevant developers for modification.

Finally, considering the threat of high-frequency connections/reconnections and the ultra-high number of connections per account to the service carrying capacity, in order to ensure the rights and interests of the vast majority of regular users, we temporarily adopted a short-term ban measure for accounts with more than 50 concurrent connections, with a short-term ban period of 24 hours to 7 days.

During the ban period, the corresponding account cannot connect to the MQTT server through official or third-party Client software. The corresponding Client will not be able to remotely control the printer or view printer status information through the cloud, but can still initiate printing through Banbu Studio or Banbu Handy. After the ban period ends, the account will resume normal connection. If an exception occurs again, it will still be banned.

As a long-term MQTT service security measure, we are developing a single account MQTT service maximum connection limit function, which is expected to be launched around August.
Until then, we recommend everyone to avoid using third-party apps or services that integrate in the Bambu Lab MQTT service, to avoid potential temporary bans caused by them.

We are unable to give more insight on how the ban will look like in third-party software, but in Bambu Handy, there will be an infinite loading circle in the Devices tab.
SD Card or LAN Mode printing is unaffected by this temporary ban and you will still be able to send prints via Bambu Studio, but no additional remote interaction, camera access or metrics reporting will work.

This incident once again reminds us that there are still shortcomings in cyber security features implemented and the way connections work.
In the future, we will continue to enhance cyber security and ensure the safe and stable use of our products and services by users through technical means.

Thank you for your understanding in this matter

22 Likes
2

I don’t think it’s intentionally to cause a high amount of requests. Below is the node-red flow I use, maybe you can contact the developer of this flow.

Or this is another one.

4 Likes
3

I’m the sole developer of the popular NodeRed integration that was linked above. I am more than willing to chat about the behaviour found with the clientId’s that started with “nodered_” to diagnose if it is related to my cloud connection flow or if it is another individual/group’s nodered integration.

The cloud connection flow of my integrations are (thankfully) uncommonly used, and the vast majority of my users use the LAN connection. I have been unable to replicate it having multiple concurrent sessions after substituting in a local mqtt broker, but as per the internal logic I have setup, it attempts to cleanly disconnect and reconnect/subscribe no less than every 10 minutes only if the user is cloud-mqtt connected.

This was due to a behaviour I found over a year ago where the data from the cloud mqtt connection was not sending updates but staying alive for any client subscribed to it, causing stale-connections only from the cloud mqtt. I suppose it is possible another user or group may have made their own flow that connects every minute to avoid this “stale connection” behaviour, and does not cleanly disconnect.

If the clientId’s start with “nodered_” then it is definitely related to the use of NodeRed, and the HACs integration is not a cause.

Again, I am more than willing to discuss this to see if it is a result of my cloud flow or if it is an isolated user/group unrelated to my flows. I have already put a notice on my site where the integration is hosted to avoid using the cloud login flow and that it may result in a temporary ban - and if it is found to be the case my cloud flow is the cause, I will it down or fix as needed.

29 Likes
4

Just wanted to say that I like this collaboration between producer and community! Let’s make things better together!

9 Likes
5

here the ioBroker integration, which is using the mqtt protocol as well
many Users mainly from germany, like me, use this integration.

3 Likes
6

I have not yet been in contact with Bambu, however I have identified a very specific scenario which could cause some issues.

Specifically, if a user of my integration is using the cloud connection, it reconnects in a 10 minute loop due to the cloud mqtt sometimes making it a stale connection and not receiving data. If during a reconnect attempt, there is instability in the bambu cloud mqtt, it will try a second reconnect. However, if this second reconnection also experiences instability during the process and does not successfully connect, then it will create a non-ending loop of reconnection attempts that will run in parallel. These loops will increase with cloud instability, but after about 3-4 loop stacks, it will cause its own instability and just keep creating new loops.

To fix this, I published version 2.1.6 of my basic flow integration with the following fixes:

  • Add a delay of 30 seconds after mqtt status is detected as down, after which, check if it has recovered before attempting a reconnection.
  • Before each delay node in the reconnection logic, I reset the queue, such that only one reconnection loop can ever exist at one time. Combined with the above fix, this means it will only ever have the 10 minute reconnection loop attempt.
  • Previously, the mqtt client had a randomly generated client-id. This is now hardset to be nodered_wws_<printer_serial>. This way it can be easier to track.
  • Finally, during all dynamic connection logic, it will force overwrite a previous connection. This likely was not needed, as a disconnect and reset is sent before reconnecting each time, but this way it forces NodeRed to always clear it just in case. Combined with the hard-set client id, this guarantees no sneaky clients connect multiple times.

I have only been able to find this scenario that causes the reconnect loop as mentioned in the original post, by introducing my own broker with simulated instability. I have been unable to recreate having NodeRed keep up multiple concurrent connections though, as it is very good at closing down all previous connections connecting to a broker. I hope that if it were my flows/integrations to cause this, that this fixes it.

Less than 1% of my users have ever downloaded the cloud-login flow required to activate the cloud connection logic. In the past 6 months, it is roughly 30-50 downloads. The range is because I myself have downloaded it at least 10 of those times in testing, so it is very likely many of these downloads are non-unique.

For users of my flow/integration, if you use the cloud connection at all, please update to 2.1.6 by reconfiguring a new flow.

12 Likes
7

Might be a while to get a response, they are in the opposite timezone if you are in NA.

1 Like
8

I’ll be watching this thread, I haven’t implemented HomeAssistant yet, but am gearing up to. I hope it can be worked out.

9

Thank you for the quick fix.

We will monitor the system and we will share an update by the end of the week if we see an improvement.

7 Likes
10

Thank you for the helpful summary at the start!

I do not have a good way to push out notifications that I have an update available if users are on much older versions of my integration, but anything from within the half year should notify that an update is available, but it again will require that the users do update.

Either way, I’ll have my printer on and connected to the cloud MQTT during this time to test as well, and you should hopefully start seeing a couple unique client id’s in the format of nodered_wws_<printer_serial>.

It is probably a good idea to still maintain some sort of temp ban system for excess connections (I don’t see how anyone would need more than 10-15 concurrent connections, so the 50 you have set would be a good limit). This will also (hopefully) get the users using my cloud login approach to look into updating or switching to LAN.

2 Likes
11

Hey! I’m the developer of OctoEverywhere, the remote access and AI failure detection plugin for Bambu Lab 3D printers.

I’m adding logic to support connecting to MQTT via the Bambu Cloud instead of directly connecting to the printer locally. While I understand the need to protect the limited computing resources of the local printer, it’s a much better user experience to allow 3rd party plugins to connect directly over the LAN. It would be awesome if similar rate limiting and anti-abuse protections could be added to the local firmware so the local LAN connection can still be used.

Do you think that could be a possibility?

1 Like
12

If I’m not mistaken, P1 and A1 series already have connected client limits (5 for P1, and A1 is less iirc), just not X1 series.

13

Yeah, that’s a good point, but I think even for the average user, only 1 or 2 MQTT connections are all that’s required. It’s so much more reliable, cleaner, and simpler to do a local connect over the cloud relay connection.

14

I was one of the people temp banned, I suspect. I had an old integration in my HA from my old A1. I sent it back due to the recall and purchased a P1s. Last week I started noticing I would get a login failure with Bambu Studio. It would log me in then immediately tell me my session expired.

Meanwhile I worked on updating my HA integration to the P1 and it dawned on me it might be the culprit. After disabling the integration, after days of no access via Bambu Studio or Bambu Handy my access was restored within about a day.

I use the cloud connection, so @WolfwithSword I am probably one of the few that used it that way.

I would never want to run something that negatively impacted the service or other users.

I’ll look to updating everything and trying again in the future as I really like the HA integration. Thanks to everyone collaborating on this, it makes for a better overall user experience.

2 Likes
15

Let me know if you need any help with updating the integration/flow, but hopefully v2.1.6 will prevent the reconnect loop/stack that could happen with the cloud connection. Worst case scenario, it should still work just fine with LAN MQTT (it’s easy to switch between without reimporting).

I have been running my X1 in the cloud connection for the past couple days on v2.1.6, and so far no temp ban so finger’s crossed!

2 Likes
16

Thanks for all of your efforts in not only putting this integration together, but continually improving it.

2 Likes
17

I’m not an expert in the MQTT protocol, but could you put the integration version in as data in the client-id after the ‘nodered_’ portion? Maybe ‘noderedv020106_’?

Bambu might be able to then just block client-ids with the old non versioned client-id leader, to protect their service and encourage people to upgrade to a non-blocked version.

Also, embedding the version would be a good way to allow them to filter any issues going forward if same things occur.

Reading through the MQTT spec, client name and version might also go better as an optional user properly (similar to user-agent in http.) but there is no convention or standard for that.

19

At the moment v2.1.6 of my flow it has a non-versioned clientid, and future updates I will likely put a version in. However, they probably shouldn’t just block ‘non-versioned’ nodered clientId’s as there could be others using NodeRed to connect who don’t use my integration, for example.

(Hit the wrong reply button, whoops)

20

Howdy, is the Implementation and code generation on your site v2.1.6 still ok? I am not afraid of a temp ban and can use LAN mode but my issuee seems independant of this current topic.

Rather than in these forums do you have your own discord or way to connect re your amazing work.

I have spent about yesteerday 7 hours rebuilding HA, rebuilding NodeRed, regenerating the yaml etc and I just cannot get all the “sensors” to work correctly. Its as iff the entiies have new name or some no longer exposed. I get
Ams_tray_0-3 is now like AMS_tray_1-4 and ‘humidy_level’ is now ‘index’ -thus the dashboard yaml generator is making a huge mess as it cant reach those entiies. Others like Chamber, Nozzel temp - work fine.

NB. Bambu X1C

21

I’ve yet to hear from Bambu if things seem “better”, at least for clients connected using the new clientId scheme intended for monitoring. However in my testing, 2.1.6 seems to fix the issues.

Though keep in mind, currently you do not need to even use the cloud login part of my integration. The basic flow works with a LAN connection even if the printer is in cloud mode. Using a LAN connection for my flows, you are at 0 risk for a cloud temp ban.

On the site I do have various places mentioned to contact me, but the easiest is to just DM me on discord (@wolfwithsword).

Your issues sound awfully like you setup the HACs integration (ha-bambulab) instead of my nodered integration and using the YAMLS intended for my integration. DM me on discord and I can help resolve it further.