Solved

Archive offload to AWS S3 Glacier error "no proxy appliances are available"


Userlevel 2

Hello

Hope you're all doing well these days.

I work for a company in Spain and I would like to ask you about an issue I’m facing with archive offload to AWS S3 storage.

We are currently performing backups on a capacity tier (AWS S3) without issues but archive tier offload is failing. Here is what the proxy appliance log says:

[...]

02.12.2021 15:02:46] <43> Info         [Ssh] Granados '617c81c8-8c61-404a-86c0-1f00ecf6ee18' connected to . Session: [SSH Session; Local: ; Remote: ]
[02.12.2021 15:02:46] <43> Info         [Ssh] Connected to x.x.x.x while other addresses () are unavailable.
[02.12.2021 15:03:07] <43> Error        Failed to connect by SSH. RetryCount: '0'. MaxRetryCount: '10'.
[02.12.2021 15:03:07] <43> Error        Failed to login to host: 'x.x.x.x', port: 2500, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [x.x.x.x], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]. The destination would not be a correct SSH server. (System.Exception)

[...]

x.x.x.x is our Veeam B&R server located in our internal network.

  • We currently have that Veeam B&R internal server selected as a gateway server in the S3 Glacier repository.
  • I have already configured both TCP 22 and TCP 443 ports in the security group configured under "Proxy Appliance" on the S3 Glacier Veeam side.
  • I’ve been able to verify that EC2 appliances are being created on AWS side.
  • I have already configured a firewall rule to reach AWS.

What am I missing?

I've already opened a case with Veeam support. I'm waiting for their answer.

Thank you very much for your time.

icon

Best answer by jmaluna 28 December 2021, 13:14

View original

11 comments

Userlevel 7
Badge +20

To me just glancing at the errors you posted it seems to be credential related but that is just a guess based on this line -

[02.12.2021 15:03:07] <43> Error        Failed to login to host: 'x.x.x.x', port: 2500, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [x.x.x.x], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]. The destination would not be a correct SSH server. (System.Exception)

I would also look under the logs to see if those give you any other details.  They are located at - C:\ProgramData\Veeam\Backup -- then the folder related to the Archive Tier offload.

Userlevel 7
Badge +9

Hello @jmaluna , 

Here are some of my suggestions. You may have configured the SGs and external facing FW. You can see the errors in the log you appended are failed logins.

Is your x.x.x.x (Veeam B&R) a Windows Server or Linux machine? 🤔 If you are able to answer this correctly, you should be able to resolve this issue. 

  • As you can see  from the log, the destination server is not a correct SSH server. (System.Exception)

If this comment helps in resolving your issue, please let us know!

Userlevel 2

To me just glancing at the errors you posted it seems to be credential related but that is just a guess based on this line -

[02.12.2021 15:03:07] <43> Error        Failed to login to host: 'x.x.x.x', port: 2500, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [x.x.x.x], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]. The destination would not be a correct SSH server. (System.Exception)

I would also look under the logs to see if those give you any other details.  They are located at - C:\ProgramData\Veeam\Backup -- then the folder related to the Archive Tier offload.

Those error lines are from the logs you mentioned. From one of the proxy appliances log, actually. No more useful information is present.

Hello @jmaluna , 

Here are some of my suggestions. You may have configured the SGs and external facing FW. You can see the errors in the log you appended are failed logins.

Is your x.x.x.x (Veeam B&R) a Windows Server or Linux machine? 🤔 If you are able to answer this correctly, you should be able to resolve this issue. 

  • As you can see  from the log, the destination server is not a correct SSH server. (System.Exception)

If this comment helps in resolving your issue, please let us know!

That Veeam B&R is a Windows server. I’ve tried to uncheck the gateway box to see what happened.

It failed with the same message but different IP address:

[03.12.2021 03:03:33] <53> Info         [LinFLR] Retrying to connect by SSH ...
[03.12.2021 03:03:33] <53> Info         [Ssh] Creating new connection ce5249ea-770d-4776-9096-5401ba993281 [host: '54.229.168.74', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [54.229.168.74], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]].
[03.12.2021 03:03:33] <53> Info         [Ssh] Creating SSH connection ce5249ea-770d-4776-9096-5401ba993281 to server 54.229.168.74
[03.12.2021 03:03:33] <53> Info         [Ssh] Creating Granados SSH connection 'ce5249ea-770d-4776-9096-5401ba993281' (unknown protocol)
[03.12.2021 03:03:33] <53> Info         [Ssh] logon, host: '54.229.168.74', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [54.229.168.74], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]
[03.12.2021 03:03:33] <53> Info         [Ssh] Granados 'ce5249ea-770d-4776-9096-5401ba993281' connected to . Session: [SSH Session; Local: ; Remote: ]
[03.12.2021 03:03:54] <53> Error        Failed to connect by SSH. RetryCount: '1'. MaxRetryCount: '10'.
[03.12.2021 03:03:54] <53> Error        Failed to login to host: '54.229.168.74', port: 22, elevation to root: 'no', autoSudo: no, use su if sudo fails: no, host name: , IPs: [54.229.168.74], AuthenticationData: [UserName: ubuntu, AuthTypes: [PublicKey]]. Unable to establish connection to host 54.229.168.74 on any IP address.

Looking at the proxy appliance logs, it looks like every appliance tried to reach a different IP (probably its own IP or another appliance’s IP address, not sure)

Thank you both for your answers.

Userlevel 7
Badge +20

So it seems there might be a routing issue?  Unsure since the IP changed but now wondering if it might be DNS possibly?  Can you resolve hosts by both IP and FQDN?  I know with other Veeam technologies if DNS is broke that breaks things.  Would continue to push Support to see what they find though.

Userlevel 7
Badge +9

Hello @jmaluna, kindly pay attention to some recommendations from @Chris.Childerhose above. Also, the Proxy Appliance requires a connection via SSH and this is correct! Please take a look at this article for a fix to similar issue: 

I hope this helps resolve your issue! 

Userlevel 2

Hi all,

I finally was able to solve that. EC2 instances created by the proxy appliance were not able to reach the Internet :sob:

After that I was facing a certificate download issue that I solved as well. Now I’m facing a permissions issue :rofl: 

Thank you all!!

Jose Manuel

Userlevel 7
Badge +9

Hi all,

I finally was able to solve that. EC2 instances created by the proxy appliance were not able to reach the Internet :sob:

After that I was facing a certificate download issue that I solved as well. Now I’m facing a permissions issue :rofl: 

Thank you all!!

Jose Manuel

Great to know that you were able-to configure the rules to allow connectivity! Could you please post the error you are having?

Userlevel 7
Badge +20

Hi all,

I finally was able to solve that. EC2 instances created by the proxy appliance were not able to reach the Internet :sob:

After that I was facing a certificate download issue that I solved as well. Now I’m facing a permissions issue :rofl: 

Thank you all!!

Jose Manuel

Great to know that you were able-to configure the rules to allow connectivity! Could you please post the error you are having?

Yeah let us know the permission message and we might be able to assist.

Userlevel 2

Hi all, solved as well.

I was getting some access denied 403 error when trying to access the archive tier repository.

Our IAM user had S3 full access and EC2 full access permissions but I had to create a custom IAM policy to finally be able to perform the archiving offload.

I’ve used the JSON for inmutable buckets that it is detailed here https://www.veeam.com/kb3151

Regards,

Jose Manuel

Userlevel 7
Badge +20

Hi all, solved as well.

I was getting some access denied 403 error when trying to access the archive tier repository.

Our IAM user had S3 full access and EC2 full access permissions but I had to create a custom IAM policy to finally be able to perform the archiving offload.

I’ve used the JSON for inmutable buckets that it is detailed here https://www.veeam.com/kb3151

Regards,

Jose Manuel

Glad to hear you were able to fix it.  Be sure to mark one of the many posts as the answer to your question to help others.

Userlevel 7
Badge +9

Hi all, solved as well.

I was getting some access denied 403 error when trying to access the archive tier repository.

Our IAM user had S3 full access and EC2 full access permissions but I had to create a custom IAM policy to finally be able to perform the archiving offload.

I’ve used the JSON for inmutable buckets that it is detailed here https://www.veeam.com/kb3151

Regards,

Jose Manuel

Great to hear! Merry Christmas

Comment