Generally speaking, UOX3 is compatible with UO client versions in the ~4.0.11c to 188.8.131.52 range. See UO client versions/expansions and compatibility with UOX3 for more detailed information.
Since UOX3 does not accept incoming connections from encrypted UO clients, you must find a way of removing the encryption from the client before you can login. To do that, you can use one of the many tools that exist for just this purpose.
Razor is one such program, and the one recommended for use with UO clients connecting to UOX3 shards. It also includes a lot of bonus features that can make playing less of a hassle.
After downloading and installing Razor, start it up. You will be greeted by a welcome screen. Leave "Load Client:" on "2D Client (Auto Detect" unless you encounter a problem with auto-detecting the client, or you have multiple clients installed. Make sure the "Patch client encryption" flag is checked, then enter the IP and port of your UOX3 shard. If you're running the UO client on the same computer as your UOX3 shard, enter 127.0.0.1 as IP and 2593 as port.
Once you've completed the initial setup of Razor, click on the Okay button to launch the UO client. You should now be ready to log onto your UOX3 server by entering the username and password you specified during the UOX3 setup process.
Razor also includes a whole host of extra options you can enable or disable as you see fit.
When the UO client has loaded, type in the username and password which you specified during the Installation/Configuration-process for UOX3, and you should be good to go!
One thing you might notice when you log in for the first time, is that your world is void of any Non-Player Characters (NPCs) as well as any decorative/usable items. You can fix this by either populating your world manually, or by downloading premade spawn and/or worldfiles from the UOX3 Downloads-page.
For more information on how to run/administer your shard, check out the following pages:
UOX3 Server Settings (Local copy)
UOX3 Command List (Local copy)