Experimenting with External Authentication Option

[orware]

I'm going to be running some tests with sftpgo over the next few days since I'd like to try and replace our existing CrushFTP usage with sftpgo if I can so I need to run through a couple of scenarios to make sure things would work.

I'll be running sftpgo on a Windows server and already have it installed and running at the moment and experimented with some basic situations with some manually created users earlier today.

Now, I'm progressing to trying to experiment with the External Authentication option, however I have been running into a few issues with it in my testing so far.

Since I'm on Windows, a PowerShell script would be the obvious choice to use, but in my initial tests with a PowerShell script I wasn't sure how sftpgo would be executing the script (I looked up some options to potentially make the PowerShell script able to be executed directly by creating a shortcut as suggested here: https://stackoverflow.com/questions/10137146/is-there-a-way-to-make-a-powershell-script-work-by-double-clicking-a-ps1-file).

In this scenario, it wasn't clear if sftpgo had executed the PowerShell script at all (I need to expand on my simple script I was utilizing and add some more logging info to see if it is getting triggered), but the sftpgo.log file only had minimal output like this: {"level":"warn","time":"2021-03-09T18:39:40.222","sender":"dataprovider_sqlite","message":"error authenticating user \"exampleuser\": not found: sql: no rows in result set"} which made it seem like it had gone straight to the database to check if the user already existed (from what I read in the documentation though, it does seem like it should automatically create/update a user record based on the output received by the External Authentication program, but that wasn't happening for this set of tests with a PowerShell script).

Next, I went ahead and did something similar, but this time with a simple PHP script which I tried on two different servers. I ended up trying it on two different servers because on the first one I received the x509 error shown below:
{"level":"warn","time":"2021-03-09T18:48:44.007","sender":"dataprovider_sqlite","message":"error getting external auth hook HTTP response: Post \"https://api.example.com/simple-sftpgo-test.php\": x509: certificate signed by unknown authority"}

And I thought that might be due to the first server having a Let's Encrypt SSL certificate being used. But I tried against a second server, which uses a DigiCert SSL certificate, and I received the same error message, so I'm not really sure what that error is referring to at the moment since both of these servers have valid SSL certificates in place, but I believe the error is an indication that SFTPGo is not even proceeding to the next step and actually POSTing the data so that it can be verified by the script.

---

I'm planning on sharing the details of my setup once I have things working properly to hopefully provide as an example for others. Our main use case is to be able to provide a space for staff to be able to upload files via SFTP, and then in combination with Caddy, those files will be able to be served by Caddy publicly. In addition, as part of my script above, I'm hoping that a "private" virtual folder will also be able to be assigned to each staff member, and any files in there will be stored separately from the location that Caddy is serving, so those won't be publicly accessible.

At the moment, I'm not sure if I'm going to be able to have the virtual folder included immediately (I'd like it to be created on the fly if possible, but the current design seems to point to needing to create folders separately from the users and then associating them with a user...and then in addition to that, the virtual folders need to also be created manually...while SFTPGo handles creating the user's home directory folder automatically from what I saw earlier today, it won't do the same thing when it comes to the virtual folders so I may have to create a separate process that assists with that, since it seems to be an explicit design choice...basically though, I do want to have things organized specifically by each user's username...in CrushFTP there was a specific option flag that could be enabled that allowed for that, but so far I'm not seeing something equivalent available as an option, but I was hoping that the External Authentication + REST API functionality would allow me to create something equivalent, but today was just my first day experimenting with SFTPGo, so I still have a fair amount more experimenting to go, but hopefully these details are helpful for anybody that might already have some more experience and could provide some tips for me.

Thank you!

[drakkan]

Hi,

there are several questions here:

1. External auth as ps script. I'm not a windows expert, from a quick search it seems that you cannot use something like a shebang to run a ps script on Windows. You probably need to convert the ps script to an exe. It seems there are several converter available. Maybe this can be improved on SFTPGo side, I have to search for other projects that do this.
2. x509: certificate signed by unknown authority. Please try the settings here you can both import your root CA or skip TLS validation (not recommended). If you expect a lot of concurrent logins, an HTTP hook will give you better performance
3. Virtual folder auto creation on user add/update. This is supported via REST API/External auth but you have to create the folder on disk yourself. I will evaluate folder checking / automatic creation on login after finished refactoring the virtual folders currently in progress. If you only need to create the virtual folders you could also evaluate dynamic user modification

[orware]

Thank you Nicola for the response!

1. Regarding the PS script, I'll run some more tests with that method today to see if I can get a sense for a potential method that might work with that approach for myself/others and report back if I make any progress there.
2. For the x509: certificate signed by unknown authority error, I am seeing that http part of the config you shared, but I'm still confused as to why that would be required? Normally, I would expect that if a program is making an https request that it would be able to POST to a remote URL without a problem (without the need to explicitly import root CAs and whatnot) so I feel like I'm not understanding the reason why filling out that http part of the config would be required in this case? (I'm receiving that error after updating the hook config option to point to a URL endpoint it can POST to: "external_auth_hook": "https://api.example.com/simple-sftpgo-test.php",. I could try out enabling the skip TLS validation as a temporary workaround though just to at least see if my test PHP script is working as expected.
3. The virtual folder automatic creation part of things feels like it might be the trickiest piece at the moment due to that need to have already created the folder on disk (plus I think I would need to also be able to create a Folder mapping ahead of time too, since I don't think that would be able to be created as part of a user's initial login automatically?). I would be interested in understanding a bit further what you shared at the end about being able to create virtual folders via the dynamic user modification capabilities. (The way I'm having to do things in the web interface currently...it doesn't look like I can add a virtual folder like this: /private::private-example-folder::-1::-1 unless the folder already exists separately in the "Folders" part of the web admin area...I'm guessing due to a database record not being available in the folders table so that a new folders_mapping entry can be added?)

I am technically going to be using LDAP (and possibly some user database access on our end) via the External Authentication option to try and achieve something like what we are doing in CrushFTP currently, so I'm mainly working my way through the capabilities SFTPGo currently offers to try and work within the functionality available. If you don't mind, I'm going to send you an email separately just with some additional details I can't as easily share publicly here that might also help explain our current SFTP usage/setup a little better as well, but overall I'm really liking the capabilities I'm seeing at the moment so I think a lot of what I'm experiencing right now is simply the learning curve of experimenting with and adopting a new system :-).

[drakkan]

Thank you Nicola for the response!

1. Regarding the PS script, I'll run some more tests with that method today to see if I can get a sense for a potential method that might work with that approach for myself/others and report back if I make any progress there.

Please let me know if you can get it to work

2. For the x509: certificate signed by unknown authority error, I am seeing that http part of the config you shared, but I'm still confused as to why that would be required? Normally, I would expect that if a program is making an https request that it would be able to POST to a remote URL without a problem (without the need to explicitly import root CAs and whatnot) so I feel like I'm not understanding the reason why filling out that http part of the config would be required in this case? (I'm receiving that error after updating the hook config option to point to a URL endpoint it can POST to: "external_auth_hook": "https://api.example.com/simple-sftpgo-test.php",. I could try out enabling the skip TLS validation as a temporary workaround though just to at least see if my test PHP script is working as expected.

It should work without any additional configuration and on Linux I cannot replicate the issue. I have to test it on Windows

3. The virtual folder automatic creation part of things feels like it might be the trickiest piece at the moment due to that need to have already created the folder on disk (plus I think I would need to also be able to create a Folder mapping ahead of time too, since I don't think that would be able to be created as part of a user's initial login automatically?). I would be interested in understanding a bit further what you shared at the end about being able to create virtual folders via the dynamic user modification capabilities. (The way I'm having to do things in the web interface currently...it doesn't look like I can add a virtual folder like this: /private::private-example-folder::-1::-1 unless the folder already exists separately in the "Folders" part of the web admin area...I'm guessing due to a database record not being available in the folders table so that a new folders_mapping entry can be added?)

from web admin this is not possible (my web skills are very poor ...) but you can do it via API/external auth, take a look at these test cases:

• https://github.com/drakkan/sftpgo/blob/main/sftpd/sftpd_test.go#L2384
• https://github.com/drakkan/sftpgo/blob/main/httpd/httpd_test.go#L2068

you basically need to pass the whole folder structure, not only the name, and it will be created or updated.

Full details for users, folders, admins and other resources are documented in the OpenAPI schema. If you want to render the schema without importing it manually, you can explore it on Stoplight.

I am technically going to be using LDAP (and possibly some user database access on our end) via the External Authentication option to try and achieve something like what we are doing in CrushFTP currently, so I'm mainly working my way through the capabilities SFTPGo currently offers to try and work within the functionality available. If you don't mind, I'm going to send you an email separately just with some additional details I can't as easily share publicly here that might also help explain our current SFTP usage/setup a little better as well, but overall I'm really liking the capabilities I'm seeing at the moment so I think a lot of what I'm experiencing right now is simply the learning curve of experimenting with and adopting a new system :-).

For LDAP there are same examples here.

One of the features that I would like to add in future is a plugin system over RPC using this library. LDAP support could be added via a plugin, so we don't add too many dependencies to the SFTPGo main binary. I don't have a definite schedule, it could take a while. Meantime external auth is the only option.

I'm now refactoring virtual folders so you can mount an S3 bucket as virtual folder or a local virtual folder to an S3 account. I'll evaluate folder auto creation after completing this feature.

[orware]

Thank you for the extra resources Nicola! I'll continue running some tests and reading through the Stoplight reference and see what I can figure out today :-).

[orware]

For anybody in a similar boat as myself that wants to add in LDAP authentication into their environment (specifically with Active Directory), then please take a look at the following mini-project:

• https://github.com/orware/sftpgo-ldap/releases

If you already have a PHP web server with LDAP installed you can use the code to connect to one or more Active Directory servers, and set things up in such a way that each user gets their own folder automatically, which is the primary use case I have for my environment.

In addition to being able to work from a web server, I made the code do double-duty and it can also function from the CLI. It's not quite as performant as a Golang solution at the moment, but there's probably still some room for improvement with the CLI option I'm using right now (I used the ExeOutput for PHP tool to help generate the EXE, which embeds the index.php from the project above, and the PHP runtime, but the rest of the files can still be modified separately, allowing for customization for your environment if needed).

I'm happy to help any others that might want to try using it and end up with some questions, but a lot of thanks must be shared with Nicola as well for helping to answer my questions (both here and via email) over the past few days.

Thank you!

[drakkan]

@orware thank you for sharing this!

Regarding your other questions/issues:

• the Windows specific "x509: certificate signed by unknown authority" was fixed in this commit
• the virtual folders are now auto-created if missing

These fixes have been backported to 2.0.x branch and they will be available in v2.0.3 that I plan to release the next weekend.

If you'll use Caddy for file serving please share. This could be useful for others too, for example see #339.

I hope SFTPGo will be useful for you!