Are you using IPS Cloud?

[JoelR]

IPS has been positioning cloud as its primary set of options for several years now.  There are different tiers of cloud, with more features on higher tier.  The pricing page prioritizes cloud (and makes it difficult to find self hosted / on premise option). 

Are you already on cloud? 

[bernhara]

Yes, we are hosted on the Cloud solution and we are really happy with it.

 

However, I think that the proposed technical solution still needs some enhancements to be considered as a Cloud service like the ones provided by AWS or Google for example.

• Since I'm using rather often the API, sometimes the front infrastructures are responding to my requests (throttling, server not reachable, ...) while they should be completely transparent. As a consequence, I'm getting responses which are not documentent in the API.
• It's not always easy to deal with system errors, since we don't have access to the full file system. This makes debugging and advanced configuration a bit harder. So, some specific "cloud helpers" would make things more convenient.

[JoelR]

On 12/2/2024 at 1:56 AM, bernhara said:

Yes, we are hosted on the Cloud solution and we are really happy with it.

 

However, I think that the proposed technical solution still needs some enhancements to be considered as a Cloud service like the ones provided by AWS or Google for example.

• Since I'm using rather often the API, sometimes the front infrastructures are responding to my requests (throttling, server not reachable, ...) while they should be completely transparent. As a consequence, I'm getting responses which are not documentent in the API.
• It's not always easy to deal with system errors, since we don't have access to the full file system. This makes debugging and advanced configuration a bit harder. So, some specific "cloud helpers" would make things more convenient.

These are all very valid (and fairly advanced) concerns.  Have you brought this up to IPS?  

[bernhara]

12 hours ago, JoelR said:

Have you brought this up to IPS?

Not yet.

We are coming form a vBulletin migration and we are still facing other details which make this question less urgent. We were using the API to correct migration failures (bad accents, disappeared images, ...). So we are facing the API problems from time to time, when we need to launch a global action. We are also mostly ready to migrate to the V5 and we will probably wait for being in V5 to address this kind of issues.

If we had a constantly running API client, this would for sure generate a much bigger problem form us.

 

I forgot to mention that I observed some other side effects resulting from a flood of API calls.

Sometimes, the server became unavailable during the API calls, even for the standard end user GUI. Since my API calls are simple "curl" commands wrapped in a shell script, my client can be classified in the "turtle scripts". And a "turtle script" is able to overload the Cloud service😉

[Matt]

On 12/10/2024 at 9:28 AM, bernhara said:

Not yet.

We are coming form a vBulletin migration and we are still facing other details which make this question less urgent. We were using the API to correct migration failures (bad accents, disappeared images, ...). So we are facing the API problems from time to time, when we need to launch a global action. We are also mostly ready to migrate to the V5 and we will probably wait for being in V5 to address this kind of issues.

If we had a constantly running API client, this would for sure generate a much bigger problem form us.

 

I forgot to mention that I observed some other side effects resulting from a flood of API calls.

Sometimes, the server became unavailable during the API calls, even for the standard end user GUI. Since my API calls are simple "curl" commands wrapped in a shell script, my client can be classified in the "turtle scripts". And a "turtle script" is able to overload the Cloud service😉

You are welcome to contact us, but generally speaking we have to scrub traffic to stop bad actors and bots. We block millions of requests daily and it may mean that some API calls are captured in that because they behave like bots (lots of calls in a short space of time, missing user-agent, etc, etc).

Also running an API that generates a lot of traffic means more hits to origin which costs more for us to host, so there has to be a balance. 

In terms of a turtle script taking down cloud, that wouldn't be possible given we use AWS. What's more likely is that WAF blocked your IP for a short time as it thought you were a bot.

[bernhara]

3 hours ago, Matt said:

What's more likely is that WAF blocked your IP for a short time as it thought you were a bot

Here is an example of what I got some times:

Quote

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error.
Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by revie
wing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ADnYXAtE5l9dn9sflS_t4YdEP835Y2vXvSV4fSlleDEOj7D_lVkBBQ==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

What I want to underline is that this kind of behavior is not documented nor specified. In fact it's a bit hard to write reliable code, since you may get an error you never saw before.

However, I understand the need to protect the API service.